Here is an update for the shared memory arrays that Gaël and I wrote two
years ago. They are NumPy arrays referencing shared memory, and IPC
using multiprocessing.Queue is possible by monkey patching how ndarrays
import numpy as np
import sharedmem as sm
shared_array = sm.zeros(n)
I.e. the only difference from ndarrays is that pickle.dumps and
multiprocessing.Queue do not make a copy of the buffer, and that
allocated memory is shared between professes (e.g. created with os.fork,
subprocess or multiprocessing.)
A named memory map of the paging file is used on Windows. Unix System V
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).
- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not
I need help with testing, particularly on Linux / Apple and with the
most recent NumPy.
I'm an idiot with build tools, hence no setup.py. Invoke Cython and then
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.
Den 11.04.2011 01:20, skrev Sturla Molden:
> I'm an idiot with build tools, hence no setup.py. Invoke Cython and
> then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
> sharedmemory_sysv.pyx and ntqueryobject.c for Windows.
Eh, that is sharedmemory_win.pyx and ntqueryobject.c for Windows :-)
On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux
<[hidden email]> wrote:
> Hey Sturla,
> It's really great that you are still working on that. I'll test the code
> under Linux.
> The scipy community has moved to github. If I create a repository under
> github and put the code on it, would you use it? If I find time, I'll add
> a setup.py.
just wanted to say that I find this module, and shared memory in
general very interesting, because I work with very large image data
I have a non python question:
for Java there seems to exist a module/package/class called nio