Hi,
I've been using the new interpolation routines (scipy.interpolate.CloughTocher2DInterpolator) quite happily in my project but I'm wondering if there is a way to save the output from the interpolation function to disk. I'm using rather large datasets (200,000+ points) and it takes an appreciable time to recalculate the interpolant every time that I run my program. I'd like it if I could dump the baked interpolant to disk and then restore it on execution. Of course I probably need to generate the interpolant per machine but I can deal with that. Is there any cool way to do this? Thanks, Sloan Lindsey Technical University of Vienna _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Fri, 25 Feb 2011 12:42:47 +0100, Sloan Lindsey wrote:
> I've been using the new interpolation routines > (scipy.interpolate.CloughTocher2DInterpolator) quite happily in my > project but I'm wondering if there is a way to save the output from the > interpolation function to disk. I'm using rather large datasets > (200,000+ points) and it takes an appreciable time to recalculate the > interpolant every time that I run my program. I assume most of the time is taken by constructing the Delaunay triangulation (scipy.spatial.Delaunay(points)). > I'd like it if I could > dump the baked interpolant to disk and then restore it on execution. Of > course I probably need to generate the interpolant per machine but I can > deal with that. Is there any cool way to do this? Ideally, this would work: import pickle f = open('file.pck', 'wb') pickle.dump(interpolator, f) f.close() but it doesn't work, since there's a small unnecessary technical snatch (scipy.spatial.interpnd has the wrong __name__). This will be fixed in the next version of Scipy, so a small workaround is needed in the meantime: class PickleableInterpolator(CloughTocher2DInterpolator): def __getstate__(self): return (self.tri, self.values, self.grad, self.is_complex, self.fill_value, self.values_shape) def __setstate__(self, data): self.tri, self.values, self.grad, self.is_complex, \ self.fill_value, self.values_shape = data self.points = self.tri.points This kind of mucking around might break in future Scipy versions, so I suggest you put it in 'if scipy.__version__ == "0.9.0":' Note that pickling objects like this is quite brittle --- the internal details of what needs to be pickled may change, so do not expect the pickle files to work across different Scipy versions. -- Pauli Virtanen _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Fri, 25 Feb 2011 20:40:26 +0000, Pauli Virtanen wrote:
[clip] > pickle.dump(interpolator, f) For most stuff, one should do pickle.dump(interpolator, f, protocol=2) so that all Numpy arrays get dumped as binary data rather than first converted to text in the pickle stream. -- Pauli Virtanen _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Fri, Feb 25, 2011 at 08:42:02PM +0000, Pauli Virtanen wrote:
> On Fri, 25 Feb 2011 20:40:26 +0000, Pauli Virtanen wrote: > [clip] > > pickle.dump(interpolator, f) > For most stuff, one should do > pickle.dump(interpolator, f, protocol=2) > so that all Numpy arrays get dumped as binary data rather than first > converted to text in the pickle stream. Or use joblib's pickler subclass: http://packages.python.org/joblib/generated/joblib.dump.html which can only be read using joblib's load function, but will pickle numpy arrays as .npy, as thus be fast at save and load. G _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
Free forum by Nabble | Edit this page |