HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

John [H2O]
Hello all,

I'm writing to ask what people are generally relying on as a
'toolchain' for reading/writing netcdf and hdf4/5 files. Also perhaps
grib? I would believe that PyNIO could solve most my problems, but
I've always encountered problems building it. I have now tried the
binaries and get a foolish error that it is looking for
libgfortran.so.1, when my system has libgfortran.so.3 (I have no admin
rights). These types of errors / headaches seem to plague me often
when trying to build hdf4 and link it to netcdf, etc. I've encountered
similar challenges when trying to build CDAT from source. I've always
managed to get close, but it seems something in the end causes a hang
up.

I have finally gotten netcdf4-python built, and it seems to work, but
it does not seem to be able to read hdf4 files. I have been able to
read hdf5, however which is terrific.

Mostly, this is a call for comments / thoughts on the tools that are
out there, and what people generally recommend. Perhaps, if anyone has
any pro/con feelings on the investment in learning CDAT vs.
PyNIO/PyNGL, that would also be welcomed.

Regards,
john
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Gerrit Holl-2
On 1 November 2010 00:13, John <[hidden email]> wrote:
> I'm writing to ask what people are generally relying on as a
> 'toolchain' for reading/writing netcdf and hdf4/5 files.

I use ScientificPython to read NetCDF and pytables to read and write
HDF5. I've been quite impressed by the speed of the latter,
particularly when using indexed searches (as in the pro version).

There is a bug in the NetCDF 4 library when using HDF-5 at the same
time. This will affect you unless you use pure Python bindings, but
there is an easy workaround. See
http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00419.html

regards,
Gerrit.
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Zachary Pincus-2
I use h5py for hdf5 files: it's a lot thinner a wrapper around the  
file format. Basically, h5py is a ctypes interface to the hdf  
libraries, and presents a rather pythonic and "numpy-ish" view of the  
data.

Where pytables tries to present its own interface, h5py just gives you  
the hdf5 file. This means that pytables can do a lot of neat things  
(like the indexed searching), but it also means that (at least last I  
checked) pytables isn't the best tool for reading in hdf5 files not  
created by pytables -- for that, you'd want h5py.

Zach


On Nov 1, 2010, at 4:30 AM, Gerrit Holl wrote:

> On 1 November 2010 00:13, John <[hidden email]> wrote:
>> I'm writing to ask what people are generally relying on as a
>> 'toolchain' for reading/writing netcdf and hdf4/5 files.
>
> I use ScientificPython to read NetCDF and pytables to read and write
> HDF5. I've been quite impressed by the speed of the latter,
> particularly when using indexed searches (as in the pro version).
>
> There is a bug in the NetCDF 4 library when using HDF-5 at the same
> time. This will affect you unless you use pure Python bindings, but
> there is an easy workaround. See
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00419.html
>
> regards,
> Gerrit.
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

dav (Bugzilla)
On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:

> Where pytables tries to present its own interface, h5py just gives you  
> the hdf5 file. This means that pytables can do a lot of neat things  
> (like the indexed searching), but it also means that (at least last I  
> checked) pytables isn't the best tool for reading in hdf5 files not  
> created by pytables -- for that, you'd want h5py.

Every time I've had an issue with pytables reading a non-pytables created file, I've submitted a bug and it got fixed usually in a few days. At the time, I was using HDF5 as a transfer layer between matlab's rudimentary hdf5 support and python w/ pytables. (Thanks Francesc!)

PyTables will automatically add it's own metadata, which is something h5py won't do - if you care.

Best,
Dav
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Robert Kern-2
On Mon, Nov 1, 2010 at 15:01, Dav Clark <[hidden email]> wrote:
> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>
>> Where pytables tries to present its own interface, h5py just gives you
>> the hdf5 file. This means that pytables can do a lot of neat things
>> (like the indexed searching), but it also means that (at least last I
>> checked) pytables isn't the best tool for reading in hdf5 files not
>> created by pytables -- for that, you'd want h5py.
>
> Every time I've had an issue with pytables reading a non-pytables created file, I've submitted a bug and it got fixed usually in a few days. At the time, I was using HDF5 as a transfer layer between matlab's rudimentary hdf5 support and python w/ pytables. (Thanks Francesc!)

I just wanted to add that in my experience, you can read just about
any HDF5 file with PyTables except for a few with some more exotic
features. If you absolutely need to write an HDF5 file according to a
strict standard without any extra bits, you may need h5py. However,
many other readers of your standard probably won't care about the
extra bits PyTables includes. You just have to be a little bit careful
to make sure that you aren't relying on any PyTables features, like
Python-pickled attributes.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Francesc Alted-2
A Monday 01 November 2010 21:19:02 Robert Kern escrigué:

> On Mon, Nov 1, 2010 at 15:01, Dav Clark <[hidden email]> wrote:
> > On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
> >> Where pytables tries to present its own interface, h5py just gives
> >> you the hdf5 file. This means that pytables can do a lot of neat
> >> things (like the indexed searching), but it also means that (at
> >> least last I checked) pytables isn't the best tool for reading in
> >> hdf5 files not created by pytables -- for that, you'd want h5py.
> >
> > Every time I've had an issue with pytables reading a non-pytables
> > created file, I've submitted a bug and it got fixed usually in a
> > few days. At the time, I was using HDF5 as a transfer layer
> > between matlab's rudimentary hdf5 support and python w/ pytables.
> > (Thanks Francesc!)
>
> I just wanted to add that in my experience, you can read just about
> any HDF5 file with PyTables except for a few with some more exotic
> features.

Let me chime in just to try to clarify couple of things.  First, both
PyTables and h5py can read most of the HDF5 files out there, but none of
them has *complete* support for HDF5 files (implementing complete
support for the whole HDF5 standard is really a tough task).  In
addition, the last time that I checked this (about one year ago, so
things might have changed since then), PyTables can read (and create)
HDF5 files that h5py cannot; and the contrary is true too.

> If you absolutely need to write an HDF5 file according to a
> strict standard without any extra bits, you may need h5py. However,
> many other readers of your standard probably won't care about the
> extra bits PyTables includes.

I suppose that the 'extra bits' you are referring to are the HDF5
attributes that complement HDF5 nodes as metainfo.  Let me say that most
of these attributes are not PyTables-specific, but those used in the
high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).  Anyway,
as I said many times, if these attributes are causing some trouble to
the user (they should not), you can always disable its creation by
setting the PYTABLES_SYS_ATTRS parameter to false during the opening of
a file (or, if you like this to be permanent, in the
tables/parameters.py).  For more info about this, see:

http://www.pytables.org/docs/manual/apc.html#id364726

> You just have to be a little bit
> careful to make sure that you aren't relying on any PyTables
> features, like Python-pickled attributes.

PyTables only uses pickle when trying to save attributes that are not
supported by HDF5 (with the exception of unicode strings that should be
implemented soon in PyTables).  For example, if you try to save a list
as an attribute:

node.attrs.my_attr = [1,2,[3,4]]

as such a list cannot be represented by HDF5 natively, PyTables chooses
to pickle it and save it.  During retrieval, the pickle is automatically
detected and unpickled before being returned to the user.  Of course,
you will not be able to read such attributes with a non-Python
application.  And, although I consider this like a feature, I can
understand that this might be considered as a bug by others (but I have
to say that very few PyTables users, if any at all, has ever complained
about this 'feature'/'bug').

Hope this helps clarifying some points,

--
Francesc Alted
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Zachary Pincus-2
Ack -- I didn't mean to set off a big back and forth! I'd only wanted  
to convey that h5py and pytables seem to serve different purposes: one  
a "simple and thin" pythonic wrapper around the official hdf5  
libraries, and the other seeking to provide some value-add on top of  
that.

I guess I got something of the rationale for using h5py over pytables  
wrong -- but there is some rationale, right? What is that?


On Nov 1, 2010, at 6:02 PM, Francesc Alted wrote:

> A Monday 01 November 2010 21:19:02 Robert Kern escrigué:
>> On Mon, Nov 1, 2010 at 15:01, Dav Clark <[hidden email]> wrote:
>>> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>>>> Where pytables tries to present its own interface, h5py just gives
>>>> you the hdf5 file. This means that pytables can do a lot of neat
>>>> things (like the indexed searching), but it also means that (at
>>>> least last I checked) pytables isn't the best tool for reading in
>>>> hdf5 files not created by pytables -- for that, you'd want h5py.
>>>
>>> Every time I've had an issue with pytables reading a non-pytables
>>> created file, I've submitted a bug and it got fixed usually in a
>>> few days. At the time, I was using HDF5 as a transfer layer
>>> between matlab's rudimentary hdf5 support and python w/ pytables.
>>> (Thanks Francesc!)
>>
>> I just wanted to add that in my experience, you can read just about
>> any HDF5 file with PyTables except for a few with some more exotic
>> features.
>
> Let me chime in just to try to clarify couple of things.  First, both
> PyTables and h5py can read most of the HDF5 files out there, but  
> none of
> them has *complete* support for HDF5 files (implementing complete
> support for the whole HDF5 standard is really a tough task).  In
> addition, the last time that I checked this (about one year ago, so
> things might have changed since then), PyTables can read (and create)
> HDF5 files that h5py cannot; and the contrary is true too.
>
>> If you absolutely need to write an HDF5 file according to a
>> strict standard without any extra bits, you may need h5py. However,
>> many other readers of your standard probably won't care about the
>> extra bits PyTables includes.
>
> I suppose that the 'extra bits' you are referring to are the HDF5
> attributes that complement HDF5 nodes as metainfo.  Let me say that  
> most
> of these attributes are not PyTables-specific, but those used in the
> high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).  
> Anyway,
> as I said many times, if these attributes are causing some trouble to
> the user (they should not), you can always disable its creation by
> setting the PYTABLES_SYS_ATTRS parameter to false during the opening  
> of
> a file (or, if you like this to be permanent, in the
> tables/parameters.py).  For more info about this, see:
>
> http://www.pytables.org/docs/manual/apc.html#id364726
>
>> You just have to be a little bit
>> careful to make sure that you aren't relying on any PyTables
>> features, like Python-pickled attributes.
>
> PyTables only uses pickle when trying to save attributes that are not
> supported by HDF5 (with the exception of unicode strings that should  
> be
> implemented soon in PyTables).  For example, if you try to save a list
> as an attribute:
>
> node.attrs.my_attr = [1,2,[3,4]]
>
> as such a list cannot be represented by HDF5 natively, PyTables  
> chooses
> to pickle it and save it.  During retrieval, the pickle is  
> automatically
> detected and unpickled before being returned to the user.  Of course,
> you will not be able to read such attributes with a non-Python
> application.  And, although I consider this like a feature, I can
> understand that this might be considered as a bug by others (but I  
> have
> to say that very few PyTables users, if any at all, has ever  
> complained
> about this 'feature'/'bug').
>
> Hope this helps clarifying some points,
>
> --
> Francesc Alted
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

John [H2O]
Back and forth can be good, as it brings out the 'juicy' details...
let's just not throwing darts ;)

I've used pytables, and it's been really useful. I guess, I was more
or less trying to find out from the Scipy users who are potentially
working with satellite data, what their tool chains are. Because there
are several different formats distributed HDF-EOS, HDF4, HDF5, etc.
It's a pain, honestly.

Luckily, the support of folks like Francesc and Jeff Whitaker and the
crew at PyNIO seem to making things easier...

The feedback so far has been really helpful. Thanks!

On Mon, Nov 1, 2010 at 11:13 PM, Zachary Pincus <[hidden email]> wrote:

> Ack -- I didn't mean to set off a big back and forth! I'd only wanted
> to convey that h5py and pytables seem to serve different purposes: one
> a "simple and thin" pythonic wrapper around the official hdf5
> libraries, and the other seeking to provide some value-add on top of
> that.
>
> I guess I got something of the rationale for using h5py over pytables
> wrong -- but there is some rationale, right? What is that?
>
>
> On Nov 1, 2010, at 6:02 PM, Francesc Alted wrote:
>
>> A Monday 01 November 2010 21:19:02 Robert Kern escrigué:
>>> On Mon, Nov 1, 2010 at 15:01, Dav Clark <[hidden email]> wrote:
>>>> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>>>>> Where pytables tries to present its own interface, h5py just gives
>>>>> you the hdf5 file. This means that pytables can do a lot of neat
>>>>> things (like the indexed searching), but it also means that (at
>>>>> least last I checked) pytables isn't the best tool for reading in
>>>>> hdf5 files not created by pytables -- for that, you'd want h5py.
>>>>
>>>> Every time I've had an issue with pytables reading a non-pytables
>>>> created file, I've submitted a bug and it got fixed usually in a
>>>> few days. At the time, I was using HDF5 as a transfer layer
>>>> between matlab's rudimentary hdf5 support and python w/ pytables.
>>>> (Thanks Francesc!)
>>>
>>> I just wanted to add that in my experience, you can read just about
>>> any HDF5 file with PyTables except for a few with some more exotic
>>> features.
>>
>> Let me chime in just to try to clarify couple of things.  First, both
>> PyTables and h5py can read most of the HDF5 files out there, but
>> none of
>> them has *complete* support for HDF5 files (implementing complete
>> support for the whole HDF5 standard is really a tough task).  In
>> addition, the last time that I checked this (about one year ago, so
>> things might have changed since then), PyTables can read (and create)
>> HDF5 files that h5py cannot; and the contrary is true too.
>>
>>> If you absolutely need to write an HDF5 file according to a
>>> strict standard without any extra bits, you may need h5py. However,
>>> many other readers of your standard probably won't care about the
>>> extra bits PyTables includes.
>>
>> I suppose that the 'extra bits' you are referring to are the HDF5
>> attributes that complement HDF5 nodes as metainfo.  Let me say that
>> most
>> of these attributes are not PyTables-specific, but those used in the
>> high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).
>> Anyway,
>> as I said many times, if these attributes are causing some trouble to
>> the user (they should not), you can always disable its creation by
>> setting the PYTABLES_SYS_ATTRS parameter to false during the opening
>> of
>> a file (or, if you like this to be permanent, in the
>> tables/parameters.py).  For more info about this, see:
>>
>> http://www.pytables.org/docs/manual/apc.html#id364726
>>
>>> You just have to be a little bit
>>> careful to make sure that you aren't relying on any PyTables
>>> features, like Python-pickled attributes.
>>
>> PyTables only uses pickle when trying to save attributes that are not
>> supported by HDF5 (with the exception of unicode strings that should
>> be
>> implemented soon in PyTables).  For example, if you try to save a list
>> as an attribute:
>>
>> node.attrs.my_attr = [1,2,[3,4]]
>>
>> as such a list cannot be represented by HDF5 natively, PyTables
>> chooses
>> to pickle it and save it.  During retrieval, the pickle is
>> automatically
>> detected and unpickled before being returned to the user.  Of course,
>> you will not be able to read such attributes with a non-Python
>> application.  And, although I consider this like a feature, I can
>> understand that this might be considered as a bug by others (but I
>> have
>> to say that very few PyTables users, if any at all, has ever
>> complained
>> about this 'feature'/'bug').
>>
>> Hope this helps clarifying some points,
>>
>> --
>> Francesc Alted
>> _______________________________________________
>> SciPy-User mailing list
>> [hidden email]
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



--
Configuration
``````````````````````````
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Python 2.6
PIL 1.1.6
Mailman 2.1.9
Postfix 2.4.5
Procmail v3.22 2001/09/10
Basemap: 1.0
Matplotlib: 1.0.0
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Francesc Alted-2
In reply to this post by Zachary Pincus-2
A Monday 01 November 2010 23:13:56 Zachary Pincus escrigué:
> Ack -- I didn't mean to set off a big back and forth! I'd only wanted
> to convey that h5py and pytables seem to serve different purposes:
> one a "simple and thin" pythonic wrapper around the official hdf5
> libraries, and the other seeking to provide some value-add on top of
> that.

I think the above is a good way to express the main difference between
h5py and PyTables.  But, unfortunately, many wrong beliefs about
packages that are similar in functionality extend in Internet without a
solid reason behind them.  I suppose this is a consequence of the
propagation of information in multi-user channels.  Unfortunately,
fighting these myths is not always easy.

> I guess I got something of the rationale for using h5py over pytables
> wrong -- but there is some rationale, right? What is that?

As you said above, both PyTables and h5py serve similar purposes in
different ways, and expressing a simple rational on using one or another
is not that easy.  If you just need HDF5 compatibility, then h5py
*might* be enough for you.  If you want more advanced functionality,
*might* be PyTables can offer it to you.  Also, people may like the API
of one package better than the other.  And some people may not like the
fact that there exist a Pro version of PyTables (although others may
appreciate it).

Frankly, I think the best rational here is more a matter of trying out
the different packages and choose the one you like the most.  This is
one of the beauties of free software: easy trying.

--
Francesc Alted
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Charles R Harris


On Mon, Nov 1, 2010 at 4:59 PM, Francesc Alted <[hidden email]> wrote:
A Monday 01 November 2010 23:13:56 Zachary Pincus escrigué:
> Ack -- I didn't mean to set off a big back and forth! I'd only wanted
> to convey that h5py and pytables seem to serve different purposes:
> one a "simple and thin" pythonic wrapper around the official hdf5
> libraries, and the other seeking to provide some value-add on top of
> that.

I think the above is a good way to express the main difference between
h5py and PyTables.  But, unfortunately, many wrong beliefs about
packages that are similar in functionality extend in Internet without a
solid reason behind them.  I suppose this is a consequence of the
propagation of information in multi-user channels.  Unfortunately,
fighting these myths is not always easy.

> I guess I got something of the rationale for using h5py over pytables
> wrong -- but there is some rationale, right? What is that?

As you said above, both PyTables and h5py serve similar purposes in
different ways, and expressing a simple rational on using one or another
is not that easy.  If you just need HDF5 compatibility, then h5py
*might* be enough for you.  If you want more advanced functionality,
*might* be PyTables can offer it to you.  Also, people may like the API
of one package better than the other.  And some people may not like the
fact that there exist a Pro version of PyTables (although others may
appreciate it).


I find PyTables quite a bit faster than h5py and more convenient for my own uses. However, when I need to exchange files with folks running IDL or Matlab on Windows it is generally safer to use h5py.

Chuck


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Andrew Collette-2
In reply to this post by Francesc Alted-2
Hi everyone,

I'm the author of h5py (although I haven't posted here in a while).

> I think the above is a good way to express the main difference between
> h5py and PyTables.  But, unfortunately, many wrong beliefs about
> packages that are similar in functionality extend in Internet without a
> solid reason behind them.  I suppose this is a consequence of the
> propagation of information in multi-user channels.  Unfortunately,
> fighting these myths is not always easy.

I think this is partially my fault for not making h5py's purpose
clearer in the beginning.  From my perspective, h5py is trying to be a
"native" (as close as possible) Python/NumPy interface to the HDF5
library, while adding as little as possible.  That means it doesn't
have any of the advanced indexing features of PyTables, or the
database metaphor (Francesc, reel me in if I'm getting out of bounds
here or below).  There are also some types which are unsupported, like
the NumPy unicode type, because I couldn't think of a way to map them
correctly.  You can find a complete list of supported/unsupported
types in the h5py FAQ
(http://code.google.com/p/h5py/wiki/FAQ#What_datatypes_are_supported?).

However, h5py provides a number of nifty things, including support for
object and region references, automatic exception translation between
HDF5 and Python (i.e. HDF5 itself can raise IOError, etc.), thread
support, and a very broad *low-level* interface to HDF5, in addition
to the NumPy-like high level interface:

http://h5py.alfven.org/docs/api/index.html

This interface is mainly of interest if you're an HDF5 weenie, or have
very, very, very specific requirements for how to write your files.
It's also the foundation on which the friendlier high-level interface
is built.

As far as compatibility, I would be very surprised if PyTables files
are much "better" or "worse" than h5py files.  Generally that sort of
thing is due to changes in HDF5 itself, for example going from HDF5
1.6 to 1.8, or the various knobs, features and anti-features in the
various releases.  The attributes thing is also a bit of a red
herring, although to toot my own horn I should point out that one of
the explicit design goals for h5py is to never touch "user-owned
spaces" like attributes or group entries.  I can't imagine it having a
practical effect.  In any case, as Francesc points out, PyTables lets
you control what gets written, or turn it off completely.

> Frankly, I think the best rational here is more a matter of trying out
> the different packages and choose the one you like the most.  This is
> one of the beauties of free software: easy trying.

Well said!  I should also point out that Francesc and I have shared
code and suggestions in the past, which is another great thing about
free software.  In fact, h5py started off using the PyTables Pyrex
definitions!  It certainly saved me lots of typing. :)

Andrew
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Francesc Alted-2
In reply to this post by Charles R Harris
A Tuesday 02 November 2010 02:28:01 Charles R Harris escrigué:
> On Mon, Nov 1, 2010 at 4:59 PM, Francesc Alted <[hidden email]>
wrote:

> > A Monday 01 November 2010 23:13:56 Zachary Pincus escrigué:
> > > Ack -- I didn't mean to set off a big back and forth! I'd only
> > > wanted to convey that h5py and pytables seem to serve different
> > > purposes: one a "simple and thin" pythonic wrapper around the
> > > official hdf5 libraries, and the other seeking to provide some
> > > value-add on top of that.
> >
> > I think the above is a good way to express the main difference
> > between h5py and PyTables.  But, unfortunately, many wrong beliefs
> > about packages that are similar in functionality extend in
> > Internet without a solid reason behind them.  I suppose this is a
> > consequence of the propagation of information in multi-user
> > channels.  Unfortunately, fighting these myths is not always easy.
> >
> > > I guess I got something of the rationale for using h5py over
> > > pytables wrong -- but there is some rationale, right? What is
> > > that?
> >
> > As you said above, both PyTables and h5py serve similar purposes in
> > different ways, and expressing a simple rational on using one or
> > another is not that easy.  If you just need HDF5 compatibility,
> > then h5py *might* be enough for you.  If you want more advanced
> > functionality, *might* be PyTables can offer it to you.  Also,
> > people may like the API of one package better than the other.  And
> > some people may not like the fact that there exist a Pro version
> > of PyTables (although others may appreciate it).
>
> I find PyTables quite a bit faster than h5py and more convenient for
> my own uses. However, when I need to exchange files with folks
> running IDL or Matlab on Windows it is generally safer to use h5py.

I'd appreciate if you can send to me some samples of IDL/Matlab files
that cannot be read by PyTables.  I always try to improve compatibility
with other apps, and perhaps these cases can be solved easily.

--
Francesc Alted
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

Francesc Alted-2
In reply to this post by Andrew Collette-2
A Tuesday 02 November 2010 04:40:47 Andrew Collette escrigué:
> I think this is partially my fault for not making h5py's purpose
> clearer in the beginning.  From my perspective, h5py is trying to be
> a "native" (as close as possible) Python/NumPy interface to the HDF5
> library, while adding as little as possible.  That means it doesn't
> have any of the advanced indexing features of PyTables, or the
> database metaphor (Francesc, reel me in if I'm getting out of bounds
> here or below).

Well, I recognize that I like to use the "database metaphor" because I
really think that PyTables is kind of database.  You know, the
boundaries between a simple format and a database are always fuzzy, but
definitely, the fact that PyTables can perform fast queries (I mean,
something typical in RDBMS like ("((lat>.45)|
(lon<.55))&(temp1+temp2)>23)" on very large tables in an efficient way
(using numexpr & compression internally and, when using Pro, column
indexes too), the support for metadata and its hierarchical nature,
makes me speak this way.

> > Frankly, I think the best rational here is more a matter of trying
> > out the different packages and choose the one you like the most.
> >  This is one of the beauties of free software: easy trying.
>
> Well said!  I should also point out that Francesc and I have shared
> code and suggestions in the past, which is another great thing about
> free software.  In fact, h5py started off using the PyTables Pyrex
> definitions!  It certainly saved me lots of typing. :)

You are welcome.  I also stole quite a few of code from h5py, as you
know (the fancy indexing code).  I certainly always thought that h5py is
a big contribution to HDF5 being more adopted in the scientific arena,
among other reasons because trusting only in one single package with
basically only one developer is not really that "trusty" :-)

Cheers,

--
Francesc Alted
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user