[SciPy-User] Microscopy TIFF files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[SciPy-User] Microscopy TIFF files

Chris Weisiger-2
Apologies for the somewhat off-topic nature of this post, but I'm not aware of any more-pertinent communities. Please feel free to redirect me elsewhere.

Our microscope control software currently outputs files in MRC format. In the interests of interoperability with the rest of the microscope community, it would be nice if we had TIFF support as well. There exist programs that convert MRC to TIFF and vice versa, but a native solution is always preferable if only to reduce the number of steps involved. Personally I have next to no experience with the TIFF format as used in microscopy, so I'm starting out by doing some research.

There exist Python libraries for writing TIFF files, of course; my concern with them is that they perform writes "atomically" (that is, the entire image data is written at once). This requires you to *have* all of the image data at the time you perform the writing, and means that for large datasets you need lots of RAM (to hold the dataset in memory) and lots of time (to write the data to disk all at once). I prefer to stream data to disk as it comes in, which greatly reduces RAM requirements and also improves performance by allowing writes to happen in parallel with data acquisition. However, I'm not aware of a Python/TIFF library that allows for this.

I've taken a brief look at the TIFF spec ( http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf ). Is this actually the same format that is used for microscopy data? I've seen OME-TIFF as well ( http://www.openmicroscopy.org/site/support/ome-model/ome-tiff/ ) but I don't know how widely it is supported (though I do see that Micro-Manager supports it). The Adobe TIFF standard specifies a limit in filesize of 4GB, which could be potentially troublesome. I don't know if OME-TIFF has similar limitations.

I'd appreciate any advice on this topic. I'm missing a lot of context, I'm sure, which will make it a lot harder to implement a working and useful solution.

-Chris

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Microscopy TIFF files

Robert Kern-2
On Fri, Jan 24, 2014 at 5:13 PM, Chris Weisiger <[hidden email]> wrote:
>
> Apologies for the somewhat off-topic nature of this post, but I'm not aware of any more-pertinent communities. Please feel free to redirect me elsewhere.
>
> Our microscope control software currently outputs files in MRC format. In the interests of interoperability with the rest of the microscope community, it would be nice if we had TIFF support as well. There exist programs that convert MRC to TIFF and vice versa, but a native solution is always preferable if only to reduce the number of steps involved. Personally I have next to no experience with the TIFF format as used in microscopy, so I'm starting out by doing some research.
>
> There exist Python libraries for writing TIFF files, of course; my concern with them is that they perform writes "atomically" (that is, the entire image data is written at once). This requires you to *have* all of the image data at the time you perform the writing, and means that for large datasets you need lots of RAM (to hold the dataset in memory) and lots of time (to write the data to disk all at once). I prefer to stream data to disk as it comes in, which greatly reduces RAM requirements and also improves performance by allowing writes to happen in parallel with data acquisition. However, I'm not aware of a Python/TIFF library that allows for this.

These are perfectly good reasons to save to your own format, MRC, and have the conversion take place as a post-processing step. The cost of that extra step is not that great, IMO.

> I've taken a brief look at the TIFF spec ( http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf ). Is this actually the same format that is used for microscopy data? I've seen OME-TIFF as well ( http://www.openmicroscopy.org/site/support/ome-model/ome-tiff/ ) but I don't know how widely it is supported (though I do see that Micro-Manager supports it). The Adobe TIFF standard specifies a limit in filesize of 4GB, which could be potentially troublesome. I don't know if OME-TIFF has similar limitations.

I don't know the microscopy field, but the answer is "sort of". TIFF is kind of a framework for building an image file format than an image format per se. Sure, you can stick to the defaults, write out one image with standard metadata, and most general image programs will be able to read it. However, there are a bunch of standards for extra, domain-specific (and application-specific!) metadata that build on top of the general TIFF spec to specify the semantics of certain metadata fields and what kind of raster data can go in, etc. OME-TIFF is, essentially, an instantiation of the general class of the TIFF standard and not a different standard altogether. The 4GB limitation is probably inherited.

--
Robert Kern

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Microscopy TIFF files

Hazen Babcock-2

On Fri, Jan 24, 2014 at 5:13 PM, Chris Weisiger <[hidden email]>
wrote:

> There exist Python libraries for writing TIFF files, of course; my concern
> with them is that they perform writes "atomically" (that is, the entire
> image data is written at once). This requires you to *have* all of the
> image data at the time you perform the writing, and means that for large
> datasets you need lots of RAM (to hold the dataset in memory) and lots of
> time (to write the data to disk all at once). I prefer to stream data to
> disk as it comes in, which greatly reduces RAM requirements and also
> improves performance by allowing writes to happen in parallel with data
> acquisition. However, I'm not aware of a Python/TIFF library that allows
> for this.

On the off chance that this is helpful, I wrote a TIFF writer that works
as you describe (i.e. it can write frame-by-frame as the images become
available). It is available here:

https://github.com/ZhuangLab/storm-control/blob/master/hal4000/halLib/tiffwriter.py

It is designed for 16bit images.

-Hazen

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Microscopy TIFF files

François-Xavier Thomas
In reply to this post by Chris Weisiger-2

On Jan 24, 2014 6:14 PM, "Chris Weisiger" <[hidden email]> wrote:
>
> Apologies for the somewhat off-topic nature of this post, but I'm not aware of any more-pertinent communities. Please feel free to redirect me elsewhere.
>
> Our microscope control software currently outputs files in MRC format. In the interests of interoperability with the rest of the microscope community, it would be nice if we had TIFF support as well. There exist programs that convert MRC to TIFF and vice versa, but a native solution is always preferable if only to reduce the number of steps involved. Personally I have next to no experience with the TIFF format as used in microscopy, so I'm starting out by doing some research.
>
> There exist Python libraries for writing TIFF files, of course; my concern with them is that they perform writes "atomically" (that is, the entire image data is written at once). This requires you to *have* all of the image data at the time you perform the writing, and means that for large datasets you need lots of RAM (to hold the dataset in memory) and lots of time (to write the data to disk all at once). I prefer to stream data to disk as it comes in, which greatly reduces RAM requirements and also improves performance by allowing writes to happen in parallel with data acquisition. However, I'm not aware of a Python/TIFF library that allows for this.

If I remember correctly, PyLibTiff can write streaming content using the low-level primitives from the C LibTiff (using WriteStrip for instance), the only issue being knowing how, because the documentation is spartan.

It is mostly a wrapper around the C version though, so you might find something in its documentation or re-use existing C code.

> I've taken a brief look at the TIFF spec ( http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf ). Is this actually the same format that is used for microscopy data? I've seen OME-TIFF as well ( http://www.openmicroscopy.org/site/support/ome-model/ome-tiff/ ) but I don't know how widely it is supported (though I do see that Micro-Manager supports it). The Adobe TIFF standard specifies a limit in filesize of 4GB, which could be potentially troublesome. I don't know if OME-TIFF has similar limitations.

Baseline Tiff does have the 4GB limit (which OME-TIFF must have inherited), but an extension called BigTiff does not. I am not sure if you can use it easily using Python though.

Well, you can use GDAL, but it is a pretty large library for such a simple goal, especially since you mentioned limited RAM.

Cheers,
François-Xavier


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Microscopy TIFF files

Sturla Molden-3
In reply to this post by Chris Weisiger-2
I have found the tifffile module from Christoph Gohlke to be sufficient to
read TIFF fikes from Olympus microscope software. It also writes TIFF
files:

http://www.lfd.uci.edu/~gohlke/

Also note that a number of Python image processing packages have i/o
routines for various image formats. Look around in OpenCV, scikit-image,
PIL, and scipy.ndarray.  

Personally I prefer to store image data in a Pytables database (hdf5).
Using an image format is only needed to get images into e.g. Adobe
Illustrator when preparing a figure. I would advice against using multiple
image files as a file format for storing microscope image data.

Sturla


Chris Weisiger <[hidden email]> wrote:

> Apologies for the somewhat off-topic nature of this post, but I'm not
> aware of any more-pertinent communities. Please feel free to redirect me elsewhere.
>
> Our microscope control software currently outputs files in MRC format. In
> the interests of interoperability with the rest of the microscope
> community, it would be nice if we had TIFF support as well. There exist
> programs that convert MRC to TIFF and vice versa, but a native solution
> is always preferable if only to reduce the number of steps involved.
> Personally I have next to no experience with the TIFF format as used in
> microscopy, so I'm starting out by doing some research.
>
> There exist Python libraries for writing TIFF files, of course; my
> concern with them is that they perform writes "atomically" (that is, the
> entire image data is written at once). This requires you to *have* all of
> the image data at the time you perform the writing, and means that for
> large datasets you need lots of RAM (to hold the dataset in memory) and
> lots of time (to write the data to disk all at once). I prefer to stream
> data to disk as it comes in, which greatly reduces RAM requirements and
> also improves performance by allowing writes to happen in parallel with
> data acquisition. However, I'm not aware of a Python/TIFF library that allows for this.
>
> I've taken a brief look at the TIFF spec ( <a
> href="http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf">http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf</a>
> ). Is this actually the same format that is used for microscopy data?
> I've seen OME-TIFF as well ( <a
> href="http://www.openmicroscopy.org/site/support/ome-model/ome-tiff/">http://www.openmicroscopy.org/site/support/ome-model/ome-tiff/</a>
> ) but I don't know how widely it is supported (though I do see that
> Micro-Manager supports it). The Adobe TIFF standard specifies a limit in
> filesize of 4GB, which could be potentially troublesome. I don't know if
> OME-TIFF has similar limitations.
>
> I'd appreciate any advice on this topic. I'm missing a lot of context,
> I'm sure, which will make it a lot harder to implement a working and useful solution.
>
> -Chris
>
> _______________________________________________ SciPy-User mailing list
> [hidden email] <a
> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Microscopy TIFF files

Jerome Kieffer
In reply to this post by Chris Weisiger-2

Hi Chris,

We have developped a library (FabIO) to open images from various
sources of X-ray detectors in read/write mode and includes a bit of
conversion between formats (while not obvious: see publication:
http://journals.iucr.org/j/issues/2013/02/00/kk5124/ )

This library supports MRC & Tiff among other formats:
https://github.com/kif/fabio
or:
http://sourceforge.net/projects/fable/files/fabio/0.1.3/

Hope this helps.
--
Jérôme Kieffer
Data analysis unit - ESRF
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user