[SciPy-User] How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[SciPy-User] How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

Propadovic Nenad
Hello,

bear with me for the long post that follows: it took me more than a week to get this far, and I tried to compress all the relevant information into the post.

There seems to be a bug in scipy.io.loadmat; I'll present it by a short piece of code and it's output.

I create file x.mat with the following:

import scipy.io

d = {'CanData':
    {
    'msg': {
            'RPDO2': {
                'timest': [0.0, 0.0039679999899817631, 0.0079779999941820279],
                'sig': {
                    'VelAct': {
                        'Values': [-0.050000000000000003, -0.10000000000000001, 0.29999999999999999, ],
                        'Name': 'VelAct'
                    },
                    'PosAct': {
                        'Values': [61.960000000000001, 61.960000000000001, 61.960000000000001, ],
                        'Name': 'PosAct'
                    }
                }
            }
        }
    }
}
scipy.io.savemat("x.mat", d)

Matlab is happy with the file and handles it the way I expect.

When I read in the data stored in the file and print it out:

import scipy.io
y = scipy.io.loadmat("x.mat")
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

The output is:
>C:\Anaconda2\pythonw -u "test1.py"
[[ array([[ ([[(array([[ ([[(array([[ 61.96,  61.96,  61.96]]), array([u'PosAct'],
      dtype='<U6'))]], [[(array([[-0.05, -0.1 ,  0.3 ]]), array([u'VelAct'],
      dtype='<U6'))]])]],
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0.      ,  0.003968,  0.007978]]))]],)]],
      dtype=[('RPDO2', 'O')])]]
object
None

Now  I've read the manual, and as I see it I have no way for me to access the deeper layers of data I just put in the file x.mat, although they are obviously right there in the data read in. Access via msg['RPDO2'] gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices.

If I use parameter squeeze_me=True:

scipy.io.savemat("x.mat", d)
y = scipy.io.loadmat("x.mat", squeeze_me=True)
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

I get output:
>C:\Anaconda2\pythonw -u "test1.py"
((array(((array([ 61.96,  61.96,  61.96]), u'PosAct'), (array([-0.05, -0.1 ,  0.3 ]), u'VelAct')),
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0.      ,  0.003968,  0.007978])),)
object
None
>Exit code: 0

All well, but the name 'RPDO2' disapeared from the data!

Now I need this information; in future I won't handle what's put into x.mat, so I need a way to access through the data all the way down (and handle the variations that will come).

I have found a workaround at:
http://stackoverflow.com/questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/

The problem is, the workaround uses struct_as_record=False in loadmat, and which boils down to using scipy.io.matlab.mio5_params.mat_struct, and when you read the docstring of class mat_struct, it says:

'''
...
We deprecate this method of holding struct information, and will
soon remove it, in favor of the recarray method (see loadmat
docstring)
'''
So my questions:
1) Did I miss something? Is there a way to access the data in 'RPDO2' by using this name, without using parameter struct_as_record=False in loadmat?
2) If not, where do I file a bug? The workaround is five years old, so the issue seems to be in scipy for ages...

(For the records, I use scipy within Anaconda2 1.4.1, under Windows, but this does not seem to matter).

Thanks a lot for the answers, in advance.

Nenad



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

Gregor Thalhammer-2

Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <[hidden email]>:

Hello,

bear with me for the long post that follows: it took me more than a week to get this far, and I tried to compress all the relevant information into the post.

There seems to be a bug in scipy.io.loadmat; I'll present it by a short piece of code and it's output.

I create file x.mat with the following:

import scipy.io

d = {'CanData':
    {
    'msg': {
            'RPDO2': {
                'timest': [0.0, 0.0039679999899817631, 0.0079779999941820279],
                'sig': {
                    'VelAct': {
                        'Values': [-0.050000000000000003, -0.10000000000000001, 0.29999999999999999, ],
                        'Name': 'VelAct'
                    },
                    'PosAct': {
                        'Values': [61.960000000000001, 61.960000000000001, 61.960000000000001, ],
                        'Name': 'PosAct'
                    }
                }
            }
        }
    }
}
scipy.io.savemat("x.mat", d)

Matlab is happy with the file and handles it the way I expect.

When I read in the data stored in the file and print it out:

import scipy.io
y = scipy.io.loadmat("x.mat")
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

The output is:
>C:\Anaconda2\pythonw -u "test1.py"
[[ array([[ ([[(array([[ ([[(array([[ 61.96,  61.96,  61.96]]), array([u'PosAct'],
      dtype='<U6'))]], [[(array([[-0.05, -0.1 ,  0.3 ]]), array([u'VelAct'],
      dtype='<U6'))]])]],
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0.      ,  0.003968,  0.007978]]))]],)]],
      dtype=[('RPDO2', 'O')])]]
object
None

Now  I've read the manual, and as I see it I have no way for me to access the deeper layers of data I just put in the file x.mat, although they are obviously right there in the data read in. Access via msg['RPDO2'] gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices.

For historic reasons, in Matlab everything is at least a 2D array, even scalars. By sprinkling some [0,0] in your code you should get what you want, e.g.

msg[0,0]['RPDO2'][0,0]['timest'][0,0]
array([[ 0.      ,  0.003968,  0.007978]])

Gregor



If I use parameter squeeze_me=True:

scipy.io.savemat("x.mat", d)
y = scipy.io.loadmat("x.mat", squeeze_me=True)
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

I get output:
>C:\Anaconda2\pythonw -u "test1.py"
((array(((array([ 61.96,  61.96,  61.96]), u'PosAct'), (array([-0.05, -0.1 ,  0.3 ]), u'VelAct')),
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0.      ,  0.003968,  0.007978])),)
object
None
>Exit code: 0

All well, but the name 'RPDO2' disapeared from the data!

Now I need this information; in future I won't handle what's put into x.mat, so I need a way to access through the data all the way down (and handle the variations that will come).

I have found a workaround at:
http://stackoverflow.com/questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/

The problem is, the workaround uses struct_as_record=False in loadmat, and which boils down to using scipy.io.matlab.mio5_params.mat_struct, and when you read the docstring of class mat_struct, it says:

'''
...
We deprecate this method of holding struct information, and will
soon remove it, in favor of the recarray method (see loadmat
docstring)
'''
So my questions:
1) Did I miss something? Is there a way to access the data in 'RPDO2' by using this name, without using parameter struct_as_record=False in loadmat?
2) If not, where do I file a bug? The workaround is five years old, so the issue seems to be in scipy for ages...

(For the records, I use scipy within Anaconda2 1.4.1, under Windows, but this does not seem to matter).

Thanks a lot for the answers, in advance.

Nenad


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

Jason Sachs
This looks familiar, I ran into this a few years ago, and if I recall correctly, there is an option to loadmat to reduce array dimensions appropriately. There is a "squeeze_me" option (unfortunately named... should probably be deprecated in favor of  "squeeze") which I think does this.


On Wed, Feb 22, 2017 at 9:02 AM, Gregor Thalhammer <[hidden email]> wrote:

Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <[hidden email]>:

Hello,

bear with me for the long post that follows: it took me more than a week to get this far, and I tried to compress all the relevant information into the post.

There seems to be a bug in scipy.io.loadmat; I'll present it by a short piece of code and it's output.

I create file x.mat with the following:

import scipy.io

d = {'CanData':
    {
    'msg': {
            'RPDO2': {
                'timest': [0.0, 0.0039679999899817631, 0.0079779999941820279],
                'sig': {
                    'VelAct': {
                        'Values': [-0.050000000000000003, -0.10000000000000001, 0.29999999999999999, ],
                        'Name': 'VelAct'
                    },
                    'PosAct': {
                        'Values': [61.960000000000001, 61.960000000000001, 61.960000000000001, ],
                        'Name': 'PosAct'
                    }
                }
            }
        }
    }
}
scipy.io.savemat("x.mat", d)

Matlab is happy with the file and handles it the way I expect.

When I read in the data stored in the file and print it out:

import scipy.io
y = scipy.io.loadmat("x.mat")
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

The output is:
>C:\Anaconda2\pythonw -u "test1.py"
[[ array([[ ([[(array([[ ([[(array([[ 61.96,  61.96,  61.96]]), array([u'PosAct'],
      dtype='<U6'))]], [[(array([[-0.05, -0.1 ,  0.3 ]]), array([u'VelAct'],
      dtype='<U6'))]])]],
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0.      ,  0.003968,  0.007978]]))]],)]],
      dtype=[('RPDO2', 'O')])]]
object
None

Now  I've read the manual, and as I see it I have no way for me to access the deeper layers of data I just put in the file x.mat, although they are obviously right there in the data read in. Access via msg['RPDO2'] gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices.

For historic reasons, in Matlab everything is at least a 2D array, even scalars. By sprinkling some [0,0] in your code you should get what you want, e.g.

msg[0,0]['RPDO2'][0,0]['timest'][0,0]
array([[ 0.      ,  0.003968,  0.007978]])

Gregor



If I use parameter squeeze_me=True:

scipy.io.savemat("x.mat", d)
y = scipy.io.loadmat("x.mat", squeeze_me=True)
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

I get output:
>C:\Anaconda2\pythonw -u "test1.py"
((array(((array([ 61.96,  61.96,  61.96]), u'PosAct'), (array([-0.05, -0.1 ,  0.3 ]), u'VelAct')),
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0.      ,  0.003968,  0.007978])),)
object
None
>Exit code: 0

All well, but the name 'RPDO2' disapeared from the data!

Now I need this information; in future I won't handle what's put into x.mat, so I need a way to access through the data all the way down (and handle the variations that will come).

I have found a workaround at:
http://stackoverflow.com/questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/

The problem is, the workaround uses struct_as_record=False in loadmat, and which boils down to using scipy.io.matlab.mio5_params.mat_struct, and when you read the docstring of class mat_struct, it says:

'''
...
We deprecate this method of holding struct information, and will
soon remove it, in favor of the recarray method (see loadmat
docstring)
'''
So my questions:
1) Did I miss something? Is there a way to access the data in 'RPDO2' by using this name, without using parameter struct_as_record=False in loadmat?
2) If not, where do I file a bug? The workaround is five years old, so the issue seems to be in scipy for ages...

(For the records, I use scipy within Anaconda2 1.4.1, under Windows, but this does not seem to matter).

Thanks a lot for the answers, in advance.

Nenad


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

Jason Sachs
ah, yes, here it is:


----

So, in MATLAB, the struct array must be at least 2D, and we replicate that when we read into Scipy. If you want all length 1 dimensions squeezed out, try this:

>>>
>>> mat_contents = sio.loadmat('octave_struct.mat', squeeze_me=True)
>>> oct_struct = mat_contents['my_struct']
>>> oct_struct.shape
()

On Wed, Feb 22, 2017 at 9:11 AM, Jason Sachs <[hidden email]> wrote:
This looks familiar, I ran into this a few years ago, and if I recall correctly, there is an option to loadmat to reduce array dimensions appropriately. There is a "squeeze_me" option (unfortunately named... should probably be deprecated in favor of  "squeeze") which I think does this.


On Wed, Feb 22, 2017 at 9:02 AM, Gregor Thalhammer <[hidden email]> wrote:

Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <[hidden email]>:

Hello,

bear with me for the long post that follows: it took me more than a week to get this far, and I tried to compress all the relevant information into the post.

There seems to be a bug in scipy.io.loadmat; I'll present it by a short piece of code and it's output.

I create file x.mat with the following:

import scipy.io

d = {'CanData':
    {
    'msg': {
            'RPDO2': {
                'timest': [0.0, 0.0039679999899817631, 0.0079779999941820279],
                'sig': {
                    'VelAct': {
                        'Values': [-0.050000000000000003, -0.10000000000000001, 0.29999999999999999, ],
                        'Name': 'VelAct'
                    },
                    'PosAct': {
                        'Values': [61.960000000000001, 61.960000000000001, 61.960000000000001, ],
                        'Name': 'PosAct'
                    }
                }
            }
        }
    }
}
scipy.io.savemat("x.mat", d)

Matlab is happy with the file and handles it the way I expect.

When I read in the data stored in the file and print it out:

import scipy.io
y = scipy.io.loadmat("x.mat")
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

The output is:
>C:\Anaconda2\pythonw -u "test1.py"
[[ array([[ ([[(array([[ ([[(array([[ 61.96,  61.96,  61.96]]), array([u'PosAct'],
      dtype='<U6'))]], [[(array([[-0.05, -0.1 ,  0.3 ]]), array([u'VelAct'],
      dtype='<U6'))]])]],
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0.      ,  0.003968,  0.007978]]))]],)]],
      dtype=[('RPDO2', 'O')])]]
object
None

Now  I've read the manual, and as I see it I have no way for me to access the deeper layers of data I just put in the file x.mat, although they are obviously right there in the data read in. Access via msg['RPDO2'] gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices.

For historic reasons, in Matlab everything is at least a 2D array, even scalars. By sprinkling some [0,0] in your code you should get what you want, e.g.

msg[0,0]['RPDO2'][0,0]['timest'][0,0]
array([[ 0.      ,  0.003968,  0.007978]])

Gregor



If I use parameter squeeze_me=True:

scipy.io.savemat("x.mat", d)
y = scipy.io.loadmat("x.mat", squeeze_me=True)
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

I get output:
>C:\Anaconda2\pythonw -u "test1.py"
((array(((array([ 61.96,  61.96,  61.96]), u'PosAct'), (array([-0.05, -0.1 ,  0.3 ]), u'VelAct')),
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0.      ,  0.003968,  0.007978])),)
object
None
>Exit code: 0

All well, but the name 'RPDO2' disapeared from the data!

Now I need this information; in future I won't handle what's put into x.mat, so I need a way to access through the data all the way down (and handle the variations that will come).

I have found a workaround at:
http://stackoverflow.com/questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/

The problem is, the workaround uses struct_as_record=False in loadmat, and which boils down to using scipy.io.matlab.mio5_params.mat_struct, and when you read the docstring of class mat_struct, it says:

'''
...
We deprecate this method of holding struct information, and will
soon remove it, in favor of the recarray method (see loadmat
docstring)
'''
So my questions:
1) Did I miss something? Is there a way to access the data in 'RPDO2' by using this name, without using parameter struct_as_record=False in loadmat?
2) If not, where do I file a bug? The workaround is five years old, so the issue seems to be in scipy for ages...

(For the records, I use scipy within Anaconda2 1.4.1, under Windows, but this does not seem to matter).

Thanks a lot for the answers, in advance.

Nenad


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Loading...