scikits.timeseries: tsfromtxt

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

scikits.timeseries: tsfromtxt

Timmie
Administrator
Hello Pierre & Matt,
I have seen that you are expanding timeseries.

How robust is the function tsfromtext?
Can I use it or do you (better we all) need more testing?

Is this a rewrite of np.loadtxt?

You recently said, that the extras.py functions are not yet for common use.
May you indicate for those functions that are still under development
the status in the docstring?

Thanks and kind regards,
Timmie

The links:
http://scipy.org/scipy/scikits/browser/trunk/timeseries/scikits/timeseries/extras.py

http://scipy.org/scipy/scikits/browser/trunk/timeseries/scikits/timeseries/_preview.py

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Pierre GM-2

On Jan 13, 2009, at 3:19 PM, Tim Michelsen wrote:
>
> How robust is the function tsfromtext?
> Can I use it or do you (better we all) need more testing?

It's fairly robust, but testing is always needed of course. I'm pretty  
sure we can run into some nasty corner cases.


> Is this a rewrite of np.loadtxt?

It's actually an adaptation of numpy.genfromtxt, the rewrite of  
np.loadtxt/mlab.csv2rec that I have implemented last month and that  
still don't know where to put in the numpy distribution. As I copied  
the necessary code in scikits.timeseries, you won't need to install  
anything else.


> You recently said, that the extras.py functions are not yet for  
> common use.

Did I ? I was probable exaggerating a bit. The functions (should)  
work. tsfromtxt does most certainly and replaces trecords.fromtextfile  
that was badly broken.

>
> May you indicate for those functions that are still under development
> the status in the docstring?

Will do. But once again, feel fere to try tsfromtxt and to send some  
feedback.
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Timmie
Administrator
Hi,

> Will do. But once again, feel fere to try tsfromtxt and to send some  
> feedback.
I guess I need some help on dateconverter.

I used:

data = ts.tsfromtxt('test.csv', datecols=(0,1), skiprows=1)


Then got the error:
TypeError: <lambda>() takes exactly 1 argument (2 given)


A sample column of my data:

2009-01-14 12:00; 23; 46

How would I read the such data in?

Kind regdards,
Timmie




_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Pierre GM-2


On Jan 13, 2009, at 6:29 PM, Tim Michelsen wrote:

>
> I used:
>
> data = ts.tsfromtxt('test.csv', datecols=(0,1), skiprows=1)
>
>
> Then got the error:
> TypeError: <lambda>() takes exactly 1 argument (2 given)
>
>
> A sample column of my data:
>
> 2009-01-14 12:00; 23; 46


I will assume that you mean row...
* First, your separator isn't a comma, but a semicolon. Use  
delimiter=";"
* Second, your date is actually only in the first column, so you  
should use datecols=0;
* Last, you don't need to define a converter for the dates in that  
case, as it should be recognized by the date parser. However, you  
should provide a freq argument, such freq="H"
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Timmie
Administrator
> I will assume that you mean row...
> * First, your separator isn't a comma, but a semicolon. Use  
> delimiter=";"
> * Second, your date is actually only in the first column, so you  
> should use datecols=0;
> * Last, you don't need to define a converter for the dates in that  
> case, as it should be recognized by the date parser. However, you  
> should provide a freq argument, such freq="H"

I tried on a small random data set (see below).
Here the ipython script and output:

In [2]: import scikits.timeseries as ts

In [3]: series = ts.tsfromtxt('test_ts.csv', delimiter=';', freq='H',
datecols=0, skiprows=1)
/usr/lib/python2.5/site-packages/numpy/ma/core.py:1383: UserWarning:
MaskedArray.__setitem__ on fields: The mask is NOT affected!
   warnings.warn("MaskedArray.__setitem__ on fields: "\

In [4]: series
Out[4]:
timeseries([(10,) (1,) (13,) (7,) (17,) (1,) (4,) (15,) (11,) (15,)
(15,) (6,) (1,)
  (16,) (3,) (19,) (11,) (16,) (12,) (8,) (11,) (19,) (15,) (10,) (6,) (0,)
  (14,) (6,) (12,) (1,) (13,) (12,) (2,) (12,) (16,) (18,) (9,) (5,) (19,)
  (5,) (14,) (14,) (18,) (1,) (14,) (20,) (13,) (11,)],
    dtype = [('f1', '<i4')],
    dates = [15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00
  15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00 15-Jan-2009 00:00],
    freq  = H)

Here the data:

"datetime";"temp"
01.01.07 00:00;10
01.01.07 01:00;0
01.01.07 02:00;14
01.01.07 03:00;6
01.01.07 04:00;12
01.01.07 05:00;1
01.01.07 06:00;13
01.01.07 07:00;12
01.01.07 08:00;2
01.01.07 09:00;12
01.01.07 10:00;16
01.01.07 11:00;18
01.01.07 12:00;9
01.01.07 13:00;5
01.01.07 14:00;19
01.01.07 15:00;5
01.01.07 16:00;14
01.01.07 17:00;14
01.01.07 18:00;18
01.01.07 19:00;1
01.01.07 20:00;14
01.01.07 21:00;20
01.01.07 22:00;6
01.01.07 23:00;10
02.01.07 00:00;15
02.01.07 01:00;15
02.01.07 02:00;1
02.01.07 03:00;13
02.01.07 04:00;7
02.01.07 05:00;17
02.01.07 06:00;1
02.01.07 07:00;4
02.01.07 08:00;15
02.01.07 09:00;11
02.01.07 10:00;15
02.01.07 11:00;6
02.01.07 12:00;19
02.01.07 13:00;1
02.01.07 14:00;16
02.01.07 15:00;3
02.01.07 16:00;19
02.01.07 17:00;11
02.01.07 18:00;16
02.01.07 19:00;12
02.01.07 20:00;8
02.01.07 21:00;11
02.01.07 22:00;13
02.01.07 23:00;11

How do I read the data from above into a time series?

Thanks in advance,
Timmie

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Pierre GM-2
Tim,
It works on my machine: you do end up with a series with a structured  
dtype [('f1',int)], and the date is correctly processed...

 >>> import StringIO
 >>> data="""datetime;test
... 01.01.07 00:00; 10
... 01.01.07 01:00; 15
"""
 >>> ts.tsfromtxt(StringIO.StringIO(data), delimiter=';', skiprows=1,  
datecols=0, freq='H')
timeseries([(10,) (15,)],
    dtype = [('f1', '<i4')],
    dates = [01-Jan-2007 00:00 01-Jan-2007 01:00],
    freq  = H)

You can then take a view of the result as int to get a series without  
named fields.

So:
It looks like you're using an old version of numpy (older than mine  
anyway...): which one is it ?
FYI, I only tested the module on 1.3.x (the SVN version). 1.3.0 will  
be a requirement as soon as it is released anyway.
Explanation: there's been a significant effort to improve support of  
structured/flexible dtype arrays in numpy.ma between 1.2 and 1.3. The  
code underlying tsfromtxt is based on these modifications. I can't/
won't backport the modifications from 1.3.0dev to 1.2.2. I could try  
to put some of the modifications to numpy.ma in the _preview.py file  
of the package, but that'd be just a hack. It's far easier to require  
1.3 only.
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Timmie
Administrator
> It looks like you're using an old version of numpy (older than mine  
> anyway...): which one is it ?
I am using 1.2.1.1 from PythonXY.

I will test again and report back.

Thanks so far.
Timmie


_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: scikits.timeseries: tsfromtxt

Timmie
Administrator
Timmie schrieb:
>> It looks like you're using an old version of numpy (older than mine  
>> anyway...): which one is it ?
> I am using 1.2.1.1 from PythonXY.
>
> I will test again and report back.
Yes, it doesn't work with version 1.2.1.1.
Too bad that I have to wait now.

Do you know when Numpy 1.3.x will be released?
I didn't find a roadmap site.

Kind regards,
Timmie

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user