ipython memory leak

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ipython memory leak

keflavich
Hi, I'm running a script that loads a lot of fits images into the
global namespace; I need at least most of them to be present for
debugging purposes.  However, every time I re-run the script, it
consumes exactly the same amount of memory: nothing is freed even
though all of the variables are overwritten.  I've tried playing with
the gc module and %clear out and manually deleting the variables, but
the memory is never freed.  After 2-3 runs, ipython crashes
complaining of a memory issues.  Plotting with matplotlib may be part
of the problem, but the memory leak (?) still occurs if I remove the
plotting commands.  Any tips on cleaning things up?

The memory error:
python2.5(751) malloc: *** mmap(size=102875136) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Thanks,
Adam
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Michael Abshoff-3
Keflavich wrote:

Hi,

> Hi, I'm running a script that loads a lot of fits images into the
> global namespace; I need at least most of them to be present for
> debugging purposes.  However, every time I re-run the script, it
> consumes exactly the same amount of memory: nothing is freed even
> though all of the variables are overwritten.  I've tried playing with
> the gc module and %clear out and manually deleting the variables, but
> the memory is never freed.  After 2-3 runs, ipython crashes
> complaining of a memory issues.  Plotting with matplotlib may be part
> of the problem, but the memory leak (?) still occurs if I remove the
> plotting commands.  Any tips on cleaning things up?

I have seen similar things with Sage, i.e. if you allocate a matrix and
do not delete it via del before exit it does not get deallocated despite
its reference count being one. One the other hand if you you reassign
that matrix to a different one the now no longer referenced matrix
object is being deallocated. We are using a rather old ipython release,
so it might have been fixed upstream. It might also be a bug in our code
base, but this sounded eerily familiar.

> The memory error:
> python2.5(751) malloc: *** mmap(size=102875136) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> Thanks,
> Adam

Should this be discussed here or what is the general policy to discuss
ipyhton related issues?

Cheers,

Michael

> _______________________________________________
> SciPy-user mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/scipy-user
>

_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Gael Varoquaux
On Thu, Apr 10, 2008 at 07:53:16PM +0200, Michael.Abshoff wrote:
> Should this be discussed here or what is the general policy to discuss
> ipyhton related issues?

They should be discussed on the ipython-dev
(http://projects.scipy.org/mailman/listinfo/ipython-dev ) or
ipython-users (http://projects.scipy.org/mailman/listinfo/ipython-user )
mailing lists. They will recieve more attention, especially from some of
the ipython developpers who do not use scientific tools.

Cheers,

Gaël
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

keflavich
OK, thanks, I'll forward my message there.  I wasn't sure which
package was causing the issue, but if you both think it's ipython I'll
go to those groups.

Adam

On Apr 10, 1:33 pm, Gael Varoquaux <[hidden email]>
wrote:

> On Thu, Apr 10, 2008 at 07:53:16PM +0200, Michael.Abshoff wrote:
> > Should this be discussed here or what is the general policy to discuss
> > ipyhton related issues?
>
> They should be discussed on the ipython-dev
> (http://projects.scipy.org/mailman/listinfo/ipython-dev) or
> ipython-users (http://projects.scipy.org/mailman/listinfo/ipython-user)
> mailing lists. They will recieve more attention, especially from some of
> the ipython developpers who do not use scientific tools.
>
> Cheers,
>
> Gaël
> _______________________________________________
> SciPy-user mailing list
> [hidden email]://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Stéfan van der Walt
On 10/04/2008, Keflavich <[hidden email]> wrote:
> OK, thanks, I'll forward my message there.  I wasn't sure which
>  package was causing the issue, but if you both think it's ipython I'll
>  go to those groups.

I have a suspicion they'll tell you that IPython can't leak memory
(other than what Python leaks by itself), since it is written in pure
Python.

Stéfan
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Gael Varoquaux
On Fri, Apr 11, 2008 at 12:03:48AM +0200, Stéfan van der Walt wrote:
> On 10/04/2008, Keflavich <[hidden email]> wrote:
> > OK, thanks, I'll forward my message there.  I wasn't sure which
> >  package was causing the issue, but if you both think it's ipython I'll
> >  go to those groups.

> I have a suspicion they'll tell you that IPython can't leak memory
> (other than what Python leaks by itself), since it is written in pure
> Python.

Sure it can: references that do not get destroyed.

Actually iptyhon does leak memory by default: the result of each command
you enter are stored in the _%i variable, where %i is the number of the
prompt. And if you return huge results without storing them in a
variable that will get garbage collected, you leak memory quite badly.
That's a feature, not a bug (:-P), but it is easy to work around, simply
delete those variables.

Cheers,

Gaël
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Brian Granger-2
In reply to this post by Stéfan van der Walt
>  I have a suspicion they'll tell you that IPython can't leak memory
>  (other than what Python leaks by itself), since it is written in pure
>  Python.

Yep, try, to run the code with regular python (not ipython) and I bet
you will still see the problem :)

>  Stéfan
>
>
> _______________________________________________
>  SciPy-user mailing list
>  [hidden email]
>  http://projects.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Robert Kern-2
In reply to this post by Gael Varoquaux
On Thu, Apr 10, 2008 at 5:06 PM, Gael Varoquaux
<[hidden email]> wrote:

>  Sure it can: references that do not get destroyed.
>
>  Actually iptyhon does leak memory by default: the result of each command
>  you enter are stored in the _%i variable, where %i is the number of the
>  prompt.

That's not the usual definition of a memory leak. A program leaks
memory when it has allocated memory, but it has lost track of it and
cannot reach it anymore. Memory leaks are, by definition, bugs. If
it's a feature, it's not a memory leak.

  http://en.wikipedia.org/wiki/Memory_leak

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

Robert Kern-2
In reply to this post by keflavich
On Thu, Apr 10, 2008 at 12:45 PM, Keflavich <[hidden email]> wrote:

> Hi, I'm running a script that loads a lot of fits images into the
>  global namespace; I need at least most of them to be present for
>  debugging purposes.  However, every time I re-run the script, it
>  consumes exactly the same amount of memory: nothing is freed even
>  though all of the variables are overwritten.  I've tried playing with
>  the gc module and %clear out and manually deleting the variables, but
>  the memory is never freed.  After 2-3 runs, ipython crashes
>  complaining of a memory issues.  Plotting with matplotlib may be part
>  of the problem, but the memory leak (?) still occurs if I remove the
>  plotting commands.  Any tips on cleaning things up?
>
>  The memory error:
>  python2.5(751) malloc: *** mmap(size=102875136) failed (error code=12)
>  *** error: can't allocate region
>  *** set a breakpoint in malloc_error_break to debug

Can you put a loop at the top of your code so that it runs 3-4 times
just as a script without ipython? If you can run your script using gdb
with a breakpoint set in malloc_error_break, you could give us a
backtrace that might help us pin down the problem.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipython memory leak

keflavich
I'm afraid I don't know what gdb is; is that related to pdb?

Anyway, it turns out running it in a loop doesn't cause any problems:
it won't crash if the code is looped over, only if it is called in
ipython.  It also proved difficult to run in pure python, probably
because I picked the wrong version of python / wrong startup file
(between scipy, scisoft, and macOS I had to strictly define where to
search for packages in ipython).

More tidbits:
  - I don't get a crash unless I'm using matplotlib, however I still
do use all of my computer's memory and drive it to a standstill
  - from that, I think the crash happens when pylab/matplotlib tries
to write to memory and gets only virtual memory, but that's a guess

So my best guess is that the %run magic is keeping everything in
memory for some reason?  That's as far as I can speculate.

Thanks for the help,
Adam

On Apr 10, 4:20 pm, "Robert Kern" <[hidden email]> wrote:

> On Thu, Apr 10, 2008 at 12:45 PM,Keflavich<[hidden email]> wrote:
> > Hi, I'm running a script that loads a lot of fits images into the
> >  global namespace; I need at least most of them to be present for
> >  debugging purposes.  However, every time I re-run the script, it
> >  consumes exactly the same amount of memory: nothing is freed even
> >  though all of the variables are overwritten.  I've tried playing with
> >  the gc module and %clear out and manually deleting the variables, but
> >  the memory is never freed.  After 2-3 runs, ipython crashes
> >  complaining of a memory issues.  Plotting with matplotlib may be part
> >  of the problem, but the memory leak (?) still occurs if I remove the
> >  plotting commands.  Any tips on cleaning things up?
>
> >  The memory error:
> >  python2.5(751) malloc: *** mmap(size=102875136) failed (error code=12)
> >  *** error: can't allocate region
> >  *** set a breakpoint in malloc_error_break to debug
>
> Can you put a loop at the top of your code so that it runs 3-4 times
> just as a script without ipython? If you can run your script using gdb
> with a breakpoint set in malloc_error_break, you could give us a
> backtrace that might help us pin down the problem.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-user mailing list
> [hidden email]://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________
SciPy-user mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/scipy-user
Loading...