common storage between matlab and python

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

common storage between matlab and python

Roger Herikstad
Hi list,
  Does anyone know of a matlab i/o interface beyond that of
scipy.io.loadmat /savemat? I know these routines will handle scalars
and vector, but what about matlab structures? My problem is that we
have a substantial amount of code written in matlab that makes use of
matlab's object oriented programming, storing the results of various
calculations in objects. What I would like to do is to interface with
these objects in python, that is read them from disk, do some
calculations, and write them back in a format consistent with what the
matlab object expects. So, for instance for the matlab object obj, the
data is stored as obj.data.field1, obj.data.field2, etc. Is there a
way for me to read the matlab file, do something to field1 and field2,
in python, then store the modified fields back into the structure for
matlab to read? Thanks!

~ Roger
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Albert Strasheim
Hello,

On Tue, May 6, 2008 at 5:26 AM, Roger Herikstad
<[hidden email]> wrote:

> Hi list,
>   Does anyone know of a matlab i/o interface beyond that of
>  scipy.io.loadmat /savemat? I know these routines will handle scalars
>  and vector, but what about matlab structures? My problem is that we
>  have a substantial amount of code written in matlab that makes use of
>  matlab's object oriented programming, storing the results of various
>  calculations in objects. What I would like to do is to interface with
>  these objects in python, that is read them from disk, do some
>  calculations, and write them back in a format consistent with what the
>  matlab object expects. So, for instance for the matlab object obj, the
>  data is stored as obj.data.field1, obj.data.field2, etc. Is there a
>  way for me to read the matlab file, do something to field1 and field2,
>  in python, then store the modified fields back into the structure for
>  matlab to read? Thanks!

If you don't find a solution specific to MATLAB, I'd recommend using HDF5.

Using HDF5 allows me to move my "objects" seamlessly between MATLAB,
Python and Java.

You'll have to write a bit of MATLAB code to take a struct and inspect
it using setfield, getfield and fieldnames, and then use hdf5write to
write it to disk.

obj might be stored in the HDF5 file obj.h5, with a "data" group, and
two datasets ("field1" and "field2").

Then you can load it up in PyTables, which should give you an
interface very similar to what you have in MATLAB. I think
obj.data.field1 might just work thanks to the magic that is PyTables
(where field1 could be a returned as a NumPy array).

Hope this helps.

Cheers,

Albert
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Roger Herikstad
Hi,
 Thanks alot! I was looking at hdf5 as an alternative, and by your
description I think it might suit my needs. I've been considering
using PyTables for a while, but never had the initiative to do so, but
I guess this is it... My one concern is to make this as invisible to
pure matlab users as possible. For the time being, we are using both
languages, and I was hoping there was a way for both python and matlab
to coexist. I'll look into it.. Thanks again!

 ~ Roger

On Tue, May 6, 2008 at 2:10 PM, Albert Strasheim <[hidden email]> wrote:

> Hello,
>
>
>
>  On Tue, May 6, 2008 at 5:26 AM, Roger Herikstad
>  <[hidden email]> wrote:
>  > Hi list,
>  >   Does anyone know of a matlab i/o interface beyond that of
>  >  scipy.io.loadmat /savemat? I know these routines will handle scalars
>  >  and vector, but what about matlab structures? My problem is that we
>  >  have a substantial amount of code written in matlab that makes use of
>  >  matlab's object oriented programming, storing the results of various
>  >  calculations in objects. What I would like to do is to interface with
>  >  these objects in python, that is read them from disk, do some
>  >  calculations, and write them back in a format consistent with what the
>  >  matlab object expects. So, for instance for the matlab object obj, the
>  >  data is stored as obj.data.field1, obj.data.field2, etc. Is there a
>  >  way for me to read the matlab file, do something to field1 and field2,
>  >  in python, then store the modified fields back into the structure for
>  >  matlab to read? Thanks!
>
>  If you don't find a solution specific to MATLAB, I'd recommend using HDF5.
>
>  Using HDF5 allows me to move my "objects" seamlessly between MATLAB,
>  Python and Java.
>
>  You'll have to write a bit of MATLAB code to take a struct and inspect
>  it using setfield, getfield and fieldnames, and then use hdf5write to
>  write it to disk.
>
>  obj might be stored in the HDF5 file obj.h5, with a "data" group, and
>  two datasets ("field1" and "field2").
>
>  Then you can load it up in PyTables, which should give you an
>  interface very similar to what you have in MATLAB. I think
>  obj.data.field1 might just work thanks to the magic that is PyTables
>  (where field1 could be a returned as a NumPy array).
>
>  Hope this helps.
>
>  Cheers,
>
>  Albert
>  _______________________________________________
>  SciPy-user mailing list
>  [hidden email]
>  http://projects.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Barry Wark
Have you considered mlabwrap (a scikits project)?  It allows bridged
calls into the matlab runtime (bridging sclaars, strings, numpy
arrays, etc.; I don't remember if it bridges structs). Although it's,
in some ways, a half-way solution, it would allow the issue to remain
more transparent to matlab users at the expense of a bit of overhead
for the python users.

On Tue, May 6, 2008 at 1:42 AM, Roger Herikstad
<[hidden email]> wrote:

> Hi,
>   Thanks alot! I was looking at hdf5 as an alternative, and by your
>  description I think it might suit my needs. I've been considering
>  using PyTables for a while, but never had the initiative to do so, but
>  I guess this is it... My one concern is to make this as invisible to
>  pure matlab users as possible. For the time being, we are using both
>  languages, and I was hoping there was a way for both python and matlab
>  to coexist. I'll look into it.. Thanks again!
>
>   ~ Roger
>
>
>
>  On Tue, May 6, 2008 at 2:10 PM, Albert Strasheim <[hidden email]> wrote:
>  > Hello,
>  >
>  >
>  >
>  >  On Tue, May 6, 2008 at 5:26 AM, Roger Herikstad
>  >  <[hidden email]> wrote:
>  >  > Hi list,
>  >  >   Does anyone know of a matlab i/o interface beyond that of
>  >  >  scipy.io.loadmat /savemat? I know these routines will handle scalars
>  >  >  and vector, but what about matlab structures? My problem is that we
>  >  >  have a substantial amount of code written in matlab that makes use of
>  >  >  matlab's object oriented programming, storing the results of various
>  >  >  calculations in objects. What I would like to do is to interface with
>  >  >  these objects in python, that is read them from disk, do some
>  >  >  calculations, and write them back in a format consistent with what the
>  >  >  matlab object expects. So, for instance for the matlab object obj, the
>  >  >  data is stored as obj.data.field1, obj.data.field2, etc. Is there a
>  >  >  way for me to read the matlab file, do something to field1 and field2,
>  >  >  in python, then store the modified fields back into the structure for
>  >  >  matlab to read? Thanks!
>  >
>  >  If you don't find a solution specific to MATLAB, I'd recommend using HDF5.
>  >
>  >  Using HDF5 allows me to move my "objects" seamlessly between MATLAB,
>  >  Python and Java.
>  >
>  >  You'll have to write a bit of MATLAB code to take a struct and inspect
>  >  it using setfield, getfield and fieldnames, and then use hdf5write to
>  >  write it to disk.
>  >
>  >  obj might be stored in the HDF5 file obj.h5, with a "data" group, and
>  >  two datasets ("field1" and "field2").
>  >
>  >  Then you can load it up in PyTables, which should give you an
>  >  interface very similar to what you have in MATLAB. I think
>  >  obj.data.field1 might just work thanks to the magic that is PyTables
>  >  (where field1 could be a returned as a NumPy array).
>  >
>  >  Hope this helps.
>  >
>  >  Cheers,
>  >
>  >  Albert
>  >  _______________________________________________
>  >  SciPy-user mailing list
>  >  [hidden email]
>  >  http://projects.scipy.org/mailman/listinfo/scipy-user
>  >
>  _______________________________________________
>  SciPy-user mailing list
>  [hidden email]
>  http://projects.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Pete Forman
In reply to this post by Roger Herikstad
"Roger Herikstad" <[hidden email]> writes:

 > I was looking at hdf5 as an alternative, and by your description I
 > think it might suit my needs.

MATLAB has read and write capabilities for HDF5 though I do not know
how well your MATLAB classes are done.  MATLAB's save function will
take a -v7.3 option to use the HDF5-based version of the MATLAB
MAT-file.  I've not tested any of this, please let us know how you get
on.  Here is a relevant thread:

http://www.mathworks.fr/matlabcentral/newsreader/view_thread/159346
--
Pete Forman                -./\.-  Disclaimer: This post is originated
WesternGeco                  -./\.-   by myself and does not represent
[hidden email]    -./\.-   the opinion of Schlumberger or
http://petef.22web.net           -./\.-   WesternGeco.

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Roger Herikstad
Hi,
 Thanks! I wasn't aware of this capability, but I think this solves my
problem. All I need to do is add the -v7.3 option to the save function
in my objects and all should be ok. Thanks again!

~ Roger

On Wed, May 7, 2008 at 7:32 PM, Pete Forman <[hidden email]> wrote:

> "Roger Herikstad" <[hidden email]> writes:
>
>   > I was looking at hdf5 as an alternative, and by your description I
>   > think it might suit my needs.
>
>  MATLAB has read and write capabilities for HDF5 though I do not know
>  how well your MATLAB classes are done.  MATLAB's save function will
>  take a -v7.3 option to use the HDF5-based version of the MATLAB
>  MAT-file.  I've not tested any of this, please let us know how you get
>  on.  Here is a relevant thread:
>
>  http://www.mathworks.fr/matlabcentral/newsreader/view_thread/159346
>  --
>  Pete Forman                -./\.-  Disclaimer: This post is originated
>  WesternGeco                  -./\.-   by myself and does not represent
>  [hidden email]    -./\.-   the opinion of Schlumberger or
>  http://petef.22web.net           -./\.-   WesternGeco.
>
>
>
>  _______________________________________________
>  SciPy-user mailing list
>  [hidden email]
>  http://projects.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Neal Becker
I've been interested in looking at pytables, but the doc makes it look like
there's a fair learning curve.  Anyone have a _simple_ example, maybe a
simple numpy vector, or a few vectors?

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Robert Cimrman
Neal Becker wrote:
> I've been interested in looking at pytables, but the doc makes it look like
> there's a fair learning curve.  Anyone have a _simple_ example, maybe a
> simple numpy vector, or a few vectors?

I used to use the code below, not sure if it works with the current
versions, but anyway, it can give you an idea how simple it is.

best regards,
r.

def writeSparseMatrixHDF5( fileName, mtx, name = 'a sparse matrix' ):
     """Assume CSR/CSC."""
     fd = pt.openFile( fileName, mode = "w", title = name )
     try:
         info = fd.createGroup( '/', 'info' )
         fd.createArray( info, 'dtype', mtx.dtype.str )
         fd.createArray( info, 'shape', mtx.shape )
         fd.createArray( info, 'format', mtx.format )

         data = fd.createGroup( '/', 'data' )
         fd.createArray( data, 'data', mtx.data )
         fd.createArray( data, 'indptr', mtx.indptr )
         fd.createArray( data, 'indices', mtx.indices )

     except:
         print 'matrix must be in SciPy sparse CSR/CSC format!'
         print mtx.__repr__()
         raise

     fd.close()

def readSparseMatrixHDF5( fileName, outputFormat = None ):
     import scipy.sparse as sp
     constructors = {'csr' : sp.csr_matrix, 'csc' : sp.csc_matrix}

     fd = pt.openFile( fileName, mode = "r" )
     info = fd.root.info
     data = fd.root.data

     format = info.format.read()
     if not isinstance( format, str ):
         format = format[0]

     dtype = info.dtype.read()
     if not isinstance( dtype, str ):
         dtype = dtype[0]

     if outputFormat is None:
         constructor = constructors[format]
     else:
         constructor = constructors[outputFormat]

     if format in ['csc', 'csr']:
         mtx = constructor( (data.data.read(),
                             data.indices.read(), data.indptr.read()),
                            dims = info.shape.read(), dtype = dtype )
     elif format == 'coo':
         mtx = constructor( (data.data.read(),
                             nm.c_[data.rows.read(), data.cols.read()].T),
                            dims = info.shape.read(), dtype = dtype )
     else:
         print format
         raise ValueError
     fd.close()

     if outputFormat in ['csc', 'csr']:
         mtx.sort_indices()

     return mtx
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Gael Varoquaux
In reply to this post by Roger Herikstad
On Tue, May 06, 2008 at 04:42:06PM +0800, Roger Herikstad wrote:
>  Thanks alot! I was looking at hdf5 as an alternative, and by your
> description I think it might suit my needs. I've been considering
> using PyTables for a while, but never had the initiative to do so, but
> I guess this is it... My one concern is to make this as invisible to
> pure matlab users as possible.

I have written some matlab code to do the saving to/loading from hdf5 as
a drop-in replacement of the vanilla matlab save/load functions. The code
might not work on every matlab structure, they work on the ones we used.
The code is available on http://www.scipy.org/Cookbook/hdf5_in_Matlab .
Feel free to improve it and put a new version there.

Cheers,

Gaël
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: common storage between matlab and python

Gael Varoquaux
In reply to this post by Pete Forman
On Wed, May 07, 2008 at 12:32:25PM +0100, Pete Forman wrote:
> MATLAB has read and write capabilities for HDF5 though I do not know
> how well your MATLAB classes are done.

They used to be really bad.

> MATLAB's save function will take a -v7.3 option to use the HDF5-based
> version of the MATLAB MAT-file.

That, on the contrary, is fantastic news. I am no longer a heavy user of
matlab (it must be a couple of years since I last coded anything in
matlab), but it is very handy to know this. Thanks for keeping us
informed. This means the death of my matlab scripts, and that's good
news: the less code to maintain the better.

Cheers,

Gaël
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user