ANN: SfePy-00.50.00

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN: SfePy-00.50.00

Robert Cimrman
I am pleased announce the release of SfePy 00.50.00.

SfePy is a finite element analysis software in Python, based primarily
on Numpy and SciPy.

Mailing lists, issue tracking, mercurial repository: http://sfepy.org
Home page: http://sfepy.kme.zcu.cz

People who contributed to this release: Ondrej Certik, Ryan Krauss,
Vladimir Lukes.

Major improvements:
- finite strain elasticity: neo-Hookean, Mooney-Rivlin materials in the
total Lagrangian (TL) formulation
- solving problems in complex numbers
- generalized equations to allow linear combination of terms
- run-time type of state term arguments
- refactoring to follow Python coding style guidelines
- new terms

For more information on this release, see
http://sfepy.googlecode.com/svn/web/releases/005000_RELEASE_NOTES.txt

Best regards,
Robert Cimrman
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Arrays and strange memory usage ...

christophe grimault
Hi,

I have a application that is very demanding in memory ressources. So I
started to to look closer at python + numpy/scipy as far as memory is
concerned.

I can't explain the following :

I start my python, + import scipy. A 'top' in the console shows that :

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python

Now after typing :

z = scipy.arange(1000000)

I get :
14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python

So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data
type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at
least).

It gets even worse with complex float. I tried :
z = arange(1000000) + 1j*arange(1000000)

Expecting 8 Mb, since z.dtype gives "complex64", the "top" shows an
increase by 31 Mb.

This is very annoying. Can someone explain this ? Is there a way to
create numpy arrays with the same (approximately ! I know the array
class adds some overhead...) memory footprint as in C/C++ ?

Thanks in advance




_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

David Cournapeau
On Wed, Sep 3, 2008 at 2:11 AM, christophe grimault
<[hidden email]> wrote:
> Hi,
>
> I have a application that is very demanding in memory ressources. So I
> started to to look closer at python + numpy/scipy as far as memory is
> concerned.

I you are really tight on memory, you will have problems with python
and most programming language which do not let you control memory in a
fine grained manner. Now, it depends on what you mean by memory
demanding: if you have barely enough memory for holding your data, it
will extremely difficult to do it in python, and difficult to do in
any language, including C and other manually managed languages.

>
> I can't explain the following :
>
> I start my python, + import scipy. A 'top' in the console shows that :
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
> 14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python
>
> Now after typing :
>
> z = scipy.arange(1000000)
>
> I get :
> 14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python
>
> So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data
> type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at
> least).

a = scipy.arange(1e6)
a.itemsize * a.size

Give me 8e6 bytes. arange is float64 by default, and I get a similar
memory increase (~ 8Mb).

>
> It gets even worse with complex float. I tried :
> z = arange(1000000) + 1j*arange(1000000)
>
> Expecting 8 Mb,

Again, this is strange, it should default to float128. Which version
of numpy/scipy are you using ?

I do not get unexpected results on my machine; results may vary
because memory allocator in python tends to overcommit to avoid
reallocating all the time, but IIRC, data are allocated with malloc
and not the python allocator in numpy. More importantly though, that's
not really representative of a typical numpy program, and would depend
on what you are doing anyway.

> This is very annoying. Can someone explain this ? Is there a way to
> create numpy arrays with the same (approximately ! I know the array
> class adds some overhead...) memory footprint as in C/C++ ?

Arrays themselves have a similar footprint as C/C++ (for big arrays,
where data >> array stucture overhead). But you will quickly find that
depending on what you are doing (linear algebra, for example), you
will need copies. Note that the same problem exists in C/C++, that's
very difficult to avoid (you need things like expression template and
co).

cheers,

David

cheers,

David
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

Robert Kern-2
On Tue, Sep 2, 2008 at 18:19, David Cournapeau <[hidden email]> wrote:

> On Wed, Sep 3, 2008 at 2:11 AM, christophe grimault
> <[hidden email]> wrote:
>> Hi,
>>
>> I have a application that is very demanding in memory ressources. So I
>> started to to look closer at python + numpy/scipy as far as memory is
>> concerned.
>
> I you are really tight on memory, you will have problems with python
> and most programming language which do not let you control memory in a
> fine grained manner. Now, it depends on what you mean by memory
> demanding: if you have barely enough memory for holding your data, it
> will extremely difficult to do it in python, and difficult to do in
> any language, including C and other manually managed languages.
>
>>
>> I can't explain the following :
>>
>> I start my python, + import scipy. A 'top' in the console shows that :
>>
>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
>> 14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python
>>
>> Now after typing :
>>
>> z = scipy.arange(1000000)
>>
>> I get :
>> 14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python
>>
>> So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data
>> type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at
>> least).
>
> a = scipy.arange(1e6)
> a.itemsize * a.size
>
> Give me 8e6 bytes. arange is float64 by default, and I get a similar
> memory increase (~ 8Mb).

No, the default is int (int32 on 32-bit systems, int64 on most 64-bit
systems) if you give it integer arguments and float64 if you give it
float arguments.

>> It gets even worse with complex float. I tried :
>> z = arange(1000000) + 1j*arange(1000000)
>>
>> Expecting 8 Mb,
>
> Again, this is strange, it should default to float128. Which version
> of numpy/scipy are you using ?

You mean complex128.

One thing to be aware of is that there are temporaries involved.
1j*arange(1000000) will allocate almost 16 MB of memory just by itself
and then allocate another 16 MB for the result of the addition. The
memory may not get returned to the OS when an object gets deallocated
although it will be reused by Python.

FWIW, here is what I get with SVN numpy on OS X:

>>> import numpy
45564 Python       0.0%  0:00.49   1    16    127 5172K  1292K  7960K    28M

>>> a = numpy.arange(1000000)
45564 Python       0.0%  0:00.50   1    16    128 9092K  1292K    12M    32M

>>> z = numpy.arange(1000000) + 1j * numpy.arange(1000000)
45564 Python       0.0%  0:00.60   1    16    129   24M  1292K    27M    47M

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

cdavid
Robert Kern wrote:
>
> No, the default is int (int32 on 32-bit systems, int64 on most 64-bit
> systems) if you give it integer arguments and float64 if you give it
> float arguments.
>  

Ah, my bad, I should have thought about the difference between 1e6 and
1000000.

>  
>>> It gets even worse with complex float. I tried :
>>> z = arange(1000000) + 1j*arange(1000000)
>>>
>>> Expecting 8 Mb,
>>>      
>> Again, this is strange, it should default to float128. Which version
>> of numpy/scipy are you using ?
>>    
>
> You mean complex128.
>  

Yes; I just wanted to point out that 1j*arange(1000000) is expected to
take ~16Mb, not 8.

> One thing to be aware of is that there are temporaries involved.
> 1j*arange(1000000) will allocate almost 16 MB of memory just by itself
> and then allocate another 16 MB for the result of the addition. The
> memory may not get returned to the OS when an object gets deallocated
> although it will be reused by Python.
>  

I think on linux, for those sizes, the memory is given back to the OS
right away because it is above the mmap threshold, and free gives the
memory right away in those cases. Since I see the exact same behavior as
you on top (b = np.arange(1e6) + 1.j np.arange(1e6) only adding 16 Mb),
maybe the Mac OS X malloc does something similar.

cheers,

David
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

Francesc Alted
In reply to this post by christophe grimault
A Tuesday 02 September 2008, christophe grimault escrigué:

> Hi,
>
> I have a application that is very demanding in memory ressources. So
> I started to to look closer at python + numpy/scipy as far as memory
> is concerned.
>
> I can't explain the following :
>
> I start my python, + import scipy. A 'top' in the console shows that
> :
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
> 14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python
>
> Now after typing :
>
> z = scipy.arange(1000000)
>
> I get :
> 14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python
>
> So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the
> data type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++
> at least).

You should look at the "RES" column instead of "VIRT" one.  "RES" column
shows the *real* memory that you are consuming.  So, in this case, you
have consumed 11MB - 8044KB ~ 3 MB.  However, you are undergoing here
the effects of number representation truncation.  Your consumed memory
should be rather: 8044KB + 3906KB = 11950KB, but as it is converted to
MB (scale changes happens automatically in 'top' when the figures need
more than 4 digits to be represented), 11959KB is truncated and 950KB
are gone, so this is why the final figure you are seeing is 11MB.  This
can be a bit misleading at first sight, but be sure that your machine
(and NumPy) is doing fine and works as expected.

Cheers,

--
Francesc Alted
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

christophe grimault
OK, I understand now !
Thanks very for the explanation.

Chris

On Wed, 2008-09-03 at 10:49 +0200, Francesc Alted wrote:

> A Tuesday 02 September 2008, christophe grimault escrigué:
> > Hi,
> >
> > I have a application that is very demanding in memory ressources. So
> > I started to to look closer at python + numpy/scipy as far as memory
> > is concerned.
> >
> > I can't explain the following :
> >
> > I start my python, + import scipy. A 'top' in the console shows that
> > :
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
> > 14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python
> >
> > Now after typing :
> >
> > z = scipy.arange(1000000)
> >
> > I get :
> > 14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python
> >
> > So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the
> > data type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++
> > at least).
>
> You should look at the "RES" column instead of "VIRT" one.  "RES" column
> shows the *real* memory that you are consuming.  So, in this case, you
> have consumed 11MB - 8044KB ~ 3 MB.  However, you are undergoing here
> the effects of number representation truncation.  Your consumed memory
> should be rather: 8044KB + 3906KB = 11950KB, but as it is converted to
> MB (scale changes happens automatically in 'top' when the figures need
> more than 4 digits to be represented), 11959KB is truncated and 950KB
> are gone, so this is why the final figure you are seeing is 11MB.  This
> can be a bit misleading at first sight, but be sure that your machine
> (and NumPy) is doing fine and works as expected.
>
> Cheers,
>


_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

Francesc Alted-2
A Wednesday 03 September 2008, christophe grimault escrigué:
> OK, I understand now !
> Thanks very for the explanation.

You are welcome.  For the sort of situations that you are facing of, I
normally query directly the Linux kernel so as to get a finer
perception of the resources wasted.  Here it goes the function that I
use:

def show_stats(explain, tref):
    "Show the used memory"
    # Build the command to obtain memory info (only for Linux 2.6.x)
    cmd = "cat /proc/%s/status" % os.getpid()
    sout = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE).stdout
    for line in sout:
        if line.startswith("VmSize:"):
            vmsize = int(line.split()[1])
        elif line.startswith("VmRSS:"):
            vmrss = int(line.split()[1])
        elif line.startswith("VmData:"):
            vmdata = int(line.split()[1])
        elif line.startswith("VmStk:"):
            vmstk = int(line.split()[1])
        elif line.startswith("VmExe:"):
            vmexe = int(line.split()[1])
        elif line.startswith("VmLib:"):
            vmlib = int(line.split()[1])
    sout.close()
    print "Memory usage: ******* %s *******" % explain
    print "VmSize: %7s kB\tVmRSS: %7s kB" % (vmsize, vmrss)
    print "VmData: %7s kB\tVmStk: %7s kB" % (vmdata, vmstk)
    print "VmExe:  %7s kB\tVmLib: %7s kB" % (vmexe, vmlib)
    tnow = time()
    print "WallClock time:", round(tnow - tref, 3)
    return tnow

And here it is an example of use:

# declare this at the beginning of your module
profile = True
[clip]
if profile: tref = time()
if profile: show_stats("Entering initial_append", tref)
[your statements here...]
if profile: show_stats("Before creating idx", tref)
[more statements...]
if profile: show_stats("After creating idx", tref)

I hope you get the idea.

--
Francesc Alted
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Arrays and strange memory usage ...

Uwe Schmitt-6
In reply to this post by christophe grimault


On 2 Sep., 19:11, christophe grimault
<[hidden email]> wrote:

> Hi,
>
> I have a application that is very demanding in memory ressources. So I
> started to to look closer at python + numpy/scipy as far as memory is
> concerned.
>
> I can't explain the following :
>
> I start my python, + import scipy. A 'top' in the console shows that :
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME COMMAND
> 14791 grimault  20   0 21624 8044 3200 S    0  0.4   0:00.43 python
>
> Now after typing :
>
> z = scipy.arange(1000000)
>
> I get :
> 14791 grimault  20   0 25532  11m 3204 S    0  0.6   0:00.44 python
>
> So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data
> type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at
> least).

I do not see the 7MB. Virtual memory increased by 3,9 MB and RES
(which is the number you are looking for), differs by 3+X MB,
I do not know how RES is rounded. But this is not a contraction.

Greetings, Uwe
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user