I am pleased announce the release of SfePy 00.50.00.
SfePy is a finite element analysis software in Python, based primarily on Numpy and SciPy. Mailing lists, issue tracking, mercurial repository: http://sfepy.org Home page: http://sfepy.kme.zcu.cz People who contributed to this release: Ondrej Certik, Ryan Krauss, Vladimir Lukes. Major improvements: - finite strain elasticity: neo-Hookean, Mooney-Rivlin materials in the total Lagrangian (TL) formulation - solving problems in complex numbers - generalized equations to allow linear combination of terms - run-time type of state term arguments - refactoring to follow Python coding style guidelines - new terms For more information on this release, see http://sfepy.googlecode.com/svn/web/releases/005000_RELEASE_NOTES.txt Best regards, Robert Cimrman _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
Hi,
I have a application that is very demanding in memory ressources. So I started to to look closer at python + numpy/scipy as far as memory is concerned. I can't explain the following : I start my python, + import scipy. A 'top' in the console shows that : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python Now after typing : z = scipy.arange(1000000) I get : 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at least). It gets even worse with complex float. I tried : z = arange(1000000) + 1j*arange(1000000) Expecting 8 Mb, since z.dtype gives "complex64", the "top" shows an increase by 31 Mb. This is very annoying. Can someone explain this ? Is there a way to create numpy arrays with the same (approximately ! I know the array class adds some overhead...) memory footprint as in C/C++ ? Thanks in advance _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
On Wed, Sep 3, 2008 at 2:11 AM, christophe grimault
<[hidden email]> wrote: > Hi, > > I have a application that is very demanding in memory ressources. So I > started to to look closer at python + numpy/scipy as far as memory is > concerned. I you are really tight on memory, you will have problems with python and most programming language which do not let you control memory in a fine grained manner. Now, it depends on what you mean by memory demanding: if you have barely enough memory for holding your data, it will extremely difficult to do it in python, and difficult to do in any language, including C and other manually managed languages. > > I can't explain the following : > > I start my python, + import scipy. A 'top' in the console shows that : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND > 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python > > Now after typing : > > z = scipy.arange(1000000) > > I get : > 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python > > So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data > type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at > least). a = scipy.arange(1e6) a.itemsize * a.size Give me 8e6 bytes. arange is float64 by default, and I get a similar memory increase (~ 8Mb). > > It gets even worse with complex float. I tried : > z = arange(1000000) + 1j*arange(1000000) > > Expecting 8 Mb, Again, this is strange, it should default to float128. Which version of numpy/scipy are you using ? I do not get unexpected results on my machine; results may vary because memory allocator in python tends to overcommit to avoid reallocating all the time, but IIRC, data are allocated with malloc and not the python allocator in numpy. More importantly though, that's not really representative of a typical numpy program, and would depend on what you are doing anyway. > This is very annoying. Can someone explain this ? Is there a way to > create numpy arrays with the same (approximately ! I know the array > class adds some overhead...) memory footprint as in C/C++ ? Arrays themselves have a similar footprint as C/C++ (for big arrays, where data >> array stucture overhead). But you will quickly find that depending on what you are doing (linear algebra, for example), you will need copies. Note that the same problem exists in C/C++, that's very difficult to avoid (you need things like expression template and co). cheers, David cheers, David _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
On Tue, Sep 2, 2008 at 18:19, David Cournapeau <[hidden email]> wrote:
> On Wed, Sep 3, 2008 at 2:11 AM, christophe grimault > <[hidden email]> wrote: >> Hi, >> >> I have a application that is very demanding in memory ressources. So I >> started to to look closer at python + numpy/scipy as far as memory is >> concerned. > > I you are really tight on memory, you will have problems with python > and most programming language which do not let you control memory in a > fine grained manner. Now, it depends on what you mean by memory > demanding: if you have barely enough memory for holding your data, it > will extremely difficult to do it in python, and difficult to do in > any language, including C and other manually managed languages. > >> >> I can't explain the following : >> >> I start my python, + import scipy. A 'top' in the console shows that : >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND >> 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python >> >> Now after typing : >> >> z = scipy.arange(1000000) >> >> I get : >> 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python >> >> So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data >> type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at >> least). > > a = scipy.arange(1e6) > a.itemsize * a.size > > Give me 8e6 bytes. arange is float64 by default, and I get a similar > memory increase (~ 8Mb). No, the default is int (int32 on 32-bit systems, int64 on most 64-bit systems) if you give it integer arguments and float64 if you give it float arguments. >> It gets even worse with complex float. I tried : >> z = arange(1000000) + 1j*arange(1000000) >> >> Expecting 8 Mb, > > Again, this is strange, it should default to float128. Which version > of numpy/scipy are you using ? You mean complex128. One thing to be aware of is that there are temporaries involved. 1j*arange(1000000) will allocate almost 16 MB of memory just by itself and then allocate another 16 MB for the result of the addition. The memory may not get returned to the OS when an object gets deallocated although it will be reused by Python. FWIW, here is what I get with SVN numpy on OS X: >>> import numpy 45564 Python 0.0% 0:00.49 1 16 127 5172K 1292K 7960K 28M >>> a = numpy.arange(1000000) 45564 Python 0.0% 0:00.50 1 16 128 9092K 1292K 12M 32M >>> z = numpy.arange(1000000) + 1j * numpy.arange(1000000) 45564 Python 0.0% 0:00.60 1 16 129 24M 1292K 27M 47M -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
Robert Kern wrote:
> > No, the default is int (int32 on 32-bit systems, int64 on most 64-bit > systems) if you give it integer arguments and float64 if you give it > float arguments. > Ah, my bad, I should have thought about the difference between 1e6 and 1000000. > >>> It gets even worse with complex float. I tried : >>> z = arange(1000000) + 1j*arange(1000000) >>> >>> Expecting 8 Mb, >>> >> Again, this is strange, it should default to float128. Which version >> of numpy/scipy are you using ? >> > > You mean complex128. > Yes; I just wanted to point out that 1j*arange(1000000) is expected to take ~16Mb, not 8. > One thing to be aware of is that there are temporaries involved. > 1j*arange(1000000) will allocate almost 16 MB of memory just by itself > and then allocate another 16 MB for the result of the addition. The > memory may not get returned to the OS when an object gets deallocated > although it will be reused by Python. > I think on linux, for those sizes, the memory is given back to the OS right away because it is above the mmap threshold, and free gives the memory right away in those cases. Since I see the exact same behavior as you on top (b = np.arange(1e6) + 1.j np.arange(1e6) only adding 16 Mb), maybe the Mac OS X malloc does something similar. cheers, David _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by christophe grimault
A Tuesday 02 September 2008, christophe grimault escrigué:
> Hi, > > I have a application that is very demanding in memory ressources. So > I started to to look closer at python + numpy/scipy as far as memory > is concerned. > > I can't explain the following : > > I start my python, + import scipy. A 'top' in the console shows that > : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND > 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python > > Now after typing : > > z = scipy.arange(1000000) > > I get : > 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python > > So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the > data type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ > at least). You should look at the "RES" column instead of "VIRT" one. "RES" column shows the *real* memory that you are consuming. So, in this case, you have consumed 11MB - 8044KB ~ 3 MB. However, you are undergoing here the effects of number representation truncation. Your consumed memory should be rather: 8044KB + 3906KB = 11950KB, but as it is converted to MB (scale changes happens automatically in 'top' when the figures need more than 4 digits to be represented), 11959KB is truncated and 950KB are gone, so this is why the final figure you are seeing is 11MB. This can be a bit misleading at first sight, but be sure that your machine (and NumPy) is doing fine and works as expected. Cheers, -- Francesc Alted _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
OK, I understand now !
Thanks very for the explanation. Chris On Wed, 2008-09-03 at 10:49 +0200, Francesc Alted wrote: > A Tuesday 02 September 2008, christophe grimault escrigué: > > Hi, > > > > I have a application that is very demanding in memory ressources. So > > I started to to look closer at python + numpy/scipy as far as memory > > is concerned. > > > > I can't explain the following : > > > > I start my python, + import scipy. A 'top' in the console shows that > > : > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND > > 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python > > > > Now after typing : > > > > z = scipy.arange(1000000) > > > > I get : > > 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python > > > > So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the > > data type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ > > at least). > > You should look at the "RES" column instead of "VIRT" one. "RES" column > shows the *real* memory that you are consuming. So, in this case, you > have consumed 11MB - 8044KB ~ 3 MB. However, you are undergoing here > the effects of number representation truncation. Your consumed memory > should be rather: 8044KB + 3906KB = 11950KB, but as it is converted to > MB (scale changes happens automatically in 'top' when the figures need > more than 4 digits to be represented), 11959KB is truncated and 950KB > are gone, so this is why the final figure you are seeing is 11MB. This > can be a bit misleading at first sight, but be sure that your machine > (and NumPy) is doing fine and works as expected. > > Cheers, > _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
A Wednesday 03 September 2008, christophe grimault escrigué:
> OK, I understand now ! > Thanks very for the explanation. You are welcome. For the sort of situations that you are facing of, I normally query directly the Linux kernel so as to get a finer perception of the resources wasted. Here it goes the function that I use: def show_stats(explain, tref): "Show the used memory" # Build the command to obtain memory info (only for Linux 2.6.x) cmd = "cat /proc/%s/status" % os.getpid() sout = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout for line in sout: if line.startswith("VmSize:"): vmsize = int(line.split()[1]) elif line.startswith("VmRSS:"): vmrss = int(line.split()[1]) elif line.startswith("VmData:"): vmdata = int(line.split()[1]) elif line.startswith("VmStk:"): vmstk = int(line.split()[1]) elif line.startswith("VmExe:"): vmexe = int(line.split()[1]) elif line.startswith("VmLib:"): vmlib = int(line.split()[1]) sout.close() print "Memory usage: ******* %s *******" % explain print "VmSize: %7s kB\tVmRSS: %7s kB" % (vmsize, vmrss) print "VmData: %7s kB\tVmStk: %7s kB" % (vmdata, vmstk) print "VmExe: %7s kB\tVmLib: %7s kB" % (vmexe, vmlib) tnow = time() print "WallClock time:", round(tnow - tref, 3) return tnow And here it is an example of use: # declare this at the beginning of your module profile = True [clip] if profile: tref = time() if profile: show_stats("Entering initial_append", tref) [your statements here...] if profile: show_stats("Before creating idx", tref) [more statements...] if profile: show_stats("After creating idx", tref) I hope you get the idea. -- Francesc Alted _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by christophe grimault
On 2 Sep., 19:11, christophe grimault <[hidden email]> wrote: > Hi, > > I have a application that is very demanding in memory ressources. So I > started to to look closer at python + numpy/scipy as far as memory is > concerned. > > I can't explain the following : > > I start my python, + import scipy. A 'top' in the console shows that : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND > 14791 grimault 20 0 21624 8044 3200 S 0 0.4 0:00.43 python > > Now after typing : > > z = scipy.arange(1000000) > > I get : > 14791 grimault 20 0 25532 11m 3204 S 0 0.6 0:00.44 python > > So the memory increased by ~ 7 Mb. I was expecting 4 Mb since the data > type is int32, giving 4*1000000 = 4 Mb of memory chunk (in C/C++ at > least). I do not see the 7MB. Virtual memory increased by 3,9 MB and RES (which is the number you are looking for), differs by 3+X MB, I do not know how RES is rounded. But this is not a contraction. Greetings, Uwe _______________________________________________ SciPy-user mailing list [hidden email] http://projects.scipy.org/mailman/listinfo/scipy-user |
Free forum by Nabble | Edit this page |