Shared memory ndarrays (update)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Shared memory ndarrays (update)

Sturla Molden-2
Here is an update for the shared memory arrays that Gaël and I wrote two
years ago. They are NumPy arrays referencing shared memory, and IPC
using multiprocessing.Queue is possible by monkey patching how ndarrays
are pickled.

Usage:

     import numpy as np
     import sharedmem as sm

     shared_array = sm.zeros(n)

I.e. the only difference from ndarrays is that pickle.dumps and
multiprocessing.Queue do not make a copy of the buffer, and that
allocated memory is shared between professes (e.g. created with os.fork,
subprocess or multiprocessing.)

A named memory map of the paging file is used on Windows. Unix System V
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).

Changes:

- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not
sufficient serialization).

I need help with testing, particularly on Linux / Apple and with the
most recent NumPy.

I'm an idiot with build tools, hence no setup.py. Invoke Cython and then
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.


Regards,
Sturla

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user

sharedmem.zip (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] Shared memory ndarrays (update)

Sturla Molden-2
Den 11.04.2011 01:20, skrev Sturla Molden:
>
> I'm an idiot with build tools, hence no setup.py. Invoke Cython and
> then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
> sharedmemory_sysv.pyx and ntqueryobject.c for Windows.

Eh, that is sharedmemory_win.pyx and ntqueryobject.c for Windows :-)

Sturla
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] Shared memory ndarrays (update)

Gael Varoquaux
In reply to this post by Sturla Molden-2
Hey Sturla,

It's really great that you are still working on that. I'll test the code
under Linux.

The scipy community has moved to github. If I create a repository under
github and put the code on it, would you use it? If I find time, I'll add
a setup.py.

Gaël

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] Shared memory ndarrays (update)

Sebastian Haase-3
On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux
<[hidden email]> wrote:

> Hey Sturla,
>
> It's really great that you are still working on that. I'll test the code
> under Linux.
>
> The scipy community has moved to github. If I create a repository under
> github and put the code on it, would you use it? If I find time, I'll add
> a setup.py.
>
> Gaël
>
Hi,

just wanted to say that I find this module, and shared memory in
general very interesting, because I work with very large image data
sets.

I have a non python question:
for Java there seems to exist a module/package/class called nio

http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.html
   public abstract class MappedByteBuffer    extends ByteBuffer
   A direct byte buffer whose content is a memory-mapped region of a file.

Could this be used to share memory between   Java applications  and  numpy ?

Just hoping that someone here knows Java much better than me.
- Sebastian Haase
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user