Share memory between python an C++

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Share memory between python an C++

Wolfgang Mader-5
Hello list,

I have a couple of functions in the form of shared C++ libraries, and want to use them from within python. Some of them involve big chunks of data which could be represented easily using numpy data types. Therefore, I am searching for a way to call the C++ function, pass a reference or pointer as argument, pointing to memory I have allocated in python, such that I can use the result of the function w/o copying. It should be possible to hide technicalities from a python user. I would apprechiate any hint.

Best, Wolfgang
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Share memory between python an C++

J. David Lee
Hi Wolfgang,

I'm fairly sure that if you pass a numpy array as an object into a C
module, you can access the data pointer directly. You should check that
the C_CONTIGUOUS flag is set for the array and make sure the type is
correct before you pass the data on, but as far as I know, that's all
you have to do. You will probably want to look at the PyArray_DATA and
PyArray_BYTES macros in the numpy API.

David

On 10/26/2012 07:54 PM, FDM wrote:
> Hello list,
>
> I have a couple of functions in the form of shared C++ libraries, and want to use them from within python. Some of them involve big chunks of data which could be represented easily using numpy data types. Therefore, I am searching for a way to call the C++ function, pass a reference or pointer as argument, pointing to memory I have allocated in python, such that I can use the result of the function w/o copying. It should be possible to hide technicalities from a python user. I would apprechiate any hint.
>
> Best, Wolfgang
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Share memory between python an C++

Gael Varoquaux
In reply to this post by Wolfgang Mader-5
Hi,

On Sat, Oct 27, 2012 at 02:54:06AM +0200, FDM wrote:
> I have a couple of functions in the form of shared C++ libraries, and
> want to use them from within python. Some of them involve big chunks of
> data which could be represented easily using numpy data types.
> Therefore, I am searching for a way to call the C++ function, pass a
> reference or pointer as argument, pointing to memory I have allocated
> in python, such that I can use the result of the function w/o copying.

You should rely on the numpy support in Cython, and use Cython to call
the C++ function.
See for instance the following Cython file that we use in the
sickit-learn to call the Murmurhash library:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/murmurhash.pyx
This file is somewhat lacking an example of passing an array as a pointer
to C code. This can be done by passing the '.data' attribute of the
array, that is converted by Cython to a pointer. The following file has
examples of this:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/liblinear.pyx

Be sure to read the Cython docs. They are excellent :)

Also, I have written an small example showing how to do something similar
in Cython, which is to use memory allocated in C++ without copy, and
with clean garbage collection. This is much harder, I find, and I try to
avoid it, but it comes in handy sometimes:
http://gael-varoquaux.info/blog/?p=157

Hope this helps,

Gaƫl

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Share memory between python an C++

Wolfgang Mader-5
In reply to this post by Wolfgang Mader-5
On Saturday 27 October 2012 02:54:06 FDM wrote:

> Hello list,
>
> I have a couple of functions in the form of shared C++ libraries, and want
> to use them from within python. Some of them involve big chunks of data
> which could be represented easily using numpy data types. Therefore, I am
> searching for a way to call the C++ function, pass a reference or pointer
> as argument, pointing to memory I have allocated in python, such that I can
> use the result of the function w/o copying. It should be possible to hide
> technicalities from a python user. I would apprechiate any hint.
>
> Best, Wolfgang
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user

Thank you for your hints!
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Share memory between python an C++

David Warde-Farley-3
In reply to this post by Gael Varoquaux
On Sat, Oct 27, 2012 at 4:29 AM, Gael Varoquaux
<[hidden email]> wrote:

> This file is somewhat lacking an example of passing an array as a pointer
> to C code. This can be done by passing the '.data' attribute of the
> array, that is converted by Cython to a pointer. The following file has
> examples of this:
> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/liblinear.pyx

Just to add to what Gael said, IIRC the .data attribute on ndarrays
has a somewhat uncertain future in Cython, as memory views obviate the
need for it (is that right?). Anyway, something to keep in mind. You
can always use the PyArray_DATA macro, I think.

David
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Share memory between python an C++

Kurt Smith
On Mon, Oct 29, 2012 at 3:01 PM, David Warde-Farley
<[hidden email]> wrote:

> On Sat, Oct 27, 2012 at 4:29 AM, Gael Varoquaux
> <[hidden email]> wrote:
>
>> This file is somewhat lacking an example of passing an array as a pointer
>> to C code. This can be done by passing the '.data' attribute of the
>> array, that is converted by Cython to a pointer. The following file has
>> examples of this:
>> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/liblinear.pyx
>
> Just to add to what Gael said, IIRC the .data attribute on ndarrays
> has a somewhat uncertain future in Cython, as memory views obviate the
> need for it (is that right?). Anyway, something to keep in mind. You
> can always use the PyArray_DATA macro, I think.

Or, you can always grab the address of the 0-th element of the array, which is
more portable and does not depend on the NumPy C-API.  So for a 2-dimensional
numpy array, you would do:

def func(np.ndarray[double, ndim=2] arr):
    other_c_func(&arr[0,0], arr.size)


>
> David
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user