[SciPy-User] low memory labeled component clustering

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[SciPy-User] low memory labeled component clustering

Edward Richards
I would like a LCC algorithm that produces sparse results. What I have
in mind would be similar to MATLAB's bwconncomp and return a sparse data
structure instead of a dense label array. I am new to image processing,
and have only a surface understanding of the LCC algorithm. I am hoping
that the experts out there can let me know if my goal is tenable (or
hopefully already implemented), before I dive too deeply into the
ndimage.label source code.

I am working on a detection algorithm, and the data is 3D and can be
very large in shape but sparse. I would like to process a bunch of data
at once to avoid edge effects. Currently I am maxing out my computers
RAM (32 GB) processing a (50, 200, 523264) element array. This array has
only ~3 million non-zero elements, or 0.00056 sparsity. For each cluster
I only need; the max value, it's location, and the number of elements.

Currently, the data is stored as a list of sparse matrices. Is it
possible to have the only 3D representation of the data be a Boolean
array that I pass to a clustering algorithm? For my problem, it is
better to take reasonable performance hits than hold temporary arrays in

Thank you,
SciPy-User mailing list
[hidden email]