[SciPy-User] deterministic python code which is unstable

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[SciPy-User] deterministic python code which is unstable

Frédéric Parrenin-2
Dear all,


I have a simple python code which should be deterministic (no call to random functions), but which gives two different results when I run it several times.
Always the same two different results.

I am wondering if this is a python bug and where I should report it.

Best regards,

Frédéric



_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Robert Kern-2
On Thu, Jun 5, 2014 at 11:46 AM, Frédéric Parrenin
<[hidden email]> wrote:
> Dear all,
>
>
> I have a simple python code which should be deterministic (no call to random
> functions), but which gives two different results when I run it several
> times.
> Always the same two different results.
>
> I am wondering if this is a python bug and where I should report it.

Without any details about what your code does and what libraries it
may be calling, it's very hard to say. Some algorithms use
pseudorandomization to break ties or avoid worst-case behavior on
structured data. What functions from scipy are you calling?

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

josef.pktd



On Thu, Jun 5, 2014 at 6:55 AM, Robert Kern <[hidden email]> wrote:
On Thu, Jun 5, 2014 at 11:46 AM, Frédéric Parrenin
<[hidden email]> wrote:
> Dear all,
>
>
> I have a simple python code which should be deterministic (no call to random
> functions), but which gives two different results when I run it several
> times.
> Always the same two different results.
>
> I am wondering if this is a python bug and where I should report it.

Without any details about what your code does and what libraries it
may be calling, it's very hard to say. Some algorithms use
pseudorandomization to break ties or avoid worst-case behavior on
structured data. What functions from scipy are you calling?

some linalg functions do this (at least on Windows) when there is no
unique solution

a recent case

Josef
 

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Alan G Isaac-2
In reply to this post by Frédéric Parrenin-2
On 6/5/2014 6:46 AM, Frédéric Parrenin wrote:
> I have a simple python code which should be deterministic (no call to random functions), but which gives two different results when I run it several
> times.
> Always the same two different results.
>
> I am wondering if this is a python bug and where I should report it.


This can even happen if you iterate over sets or dicts,
if your code depends on the order in which items are fetched.

Alan Isaac


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Frédéric Parrenin-2
Thanks for the answers.

The code uses the following modules/functions:

import math as m
import numpy as np
import matplotlib.pyplot as mpl
from scipy.interpolate import interp1d
from scipy.optimize import leastsq
from matplotlib.backends.backend_pdf import PdfPages
from scipy.special import erf

Mainly, the code solves a leastsq problem.

For those who are interested, I put the code here:
https://drive.google.com/file/d/0BzX8dPORePBsWEpSb0I5S05ZQlk/edit?usp=sharing
I run it on a standard debian 7 system (only matplotlib has been updated).

Best regards,

Frédéric





2014-06-05 15:18 GMT+02:00 Alan G Isaac <[hidden email]>:
On 6/5/2014 6:46 AM, Frédéric Parrenin wrote:
> I have a simple python code which should be deterministic (no call to random functions), but which gives two different results when I run it several
> times.
> Always the same two different results.
>
> I am wondering if this is a python bug and where I should report it.


This can even happen if you iterate over sets or dicts,
if your code depends on the order in which items are fetched.

Alan Isaac


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Robert Kern-2
On Thu, Jun 5, 2014 at 2:57 PM, Frédéric Parrenin
<[hidden email]> wrote:

> Thanks for the answers.
>
> The code uses the following modules/functions:
>
> import math as m
> import numpy as np
> import matplotlib.pyplot as mpl
> from scipy.interpolate import interp1d
> from scipy.optimize import leastsq
> from matplotlib.backends.backend_pdf import PdfPages
> from scipy.special import erf
>
> Mainly, the code solves a leastsq problem.

scipy.optimize.leastsq() and numpy.linalg.solve() are likely culprits.

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Frédéric Parrenin-2
Thanks for the hint.

I would suggest to output a warning in scipy.optimize.leastsq() and numpy.linalg.solve() when such circumstances happen.

It is disturbing to have an unstable model and to not know where it comes from.

Best regards,

Frédéric




2014-06-05 16:10 GMT+02:00 Robert Kern <[hidden email]>:
On Thu, Jun 5, 2014 at 2:57 PM, Frédéric Parrenin
<[hidden email]> wrote:
> Thanks for the answers.
>
> The code uses the following modules/functions:
>
> import math as m
> import numpy as np
> import matplotlib.pyplot as mpl
> from scipy.interpolate import interp1d
> from scipy.optimize import leastsq
> from matplotlib.backends.backend_pdf import PdfPages
> from scipy.special import erf
>
> Mainly, the code solves a leastsq problem.

scipy.optimize.leastsq() and numpy.linalg.solve() are likely culprits.

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Robert Kern-2
On Thu, Jun 5, 2014 at 4:14 PM, Frédéric Parrenin
<[hidden email]> wrote:
> Thanks for the hint.
>
> I would suggest to output a warning in scipy.optimize.leastsq() and
> numpy.linalg.solve() when such circumstances happen.

We don't know when that happens. It's not our code that is doing it
but the underlying linear algebra libraries, which are a black box to
us as far as this is concerned.

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Frédéric Parrenin-2
Is it not possible to bug report those linear algebra libraries?

Frédéric




2014-06-05 17:33 GMT+02:00 Robert Kern <[hidden email]>:
On Thu, Jun 5, 2014 at 4:14 PM, Frédéric Parrenin
<[hidden email]> wrote:
> Thanks for the hint.
>
> I would suggest to output a warning in scipy.optimize.leastsq() and
> numpy.linalg.solve() when such circumstances happen.

We don't know when that happens. It's not our code that is doing it
but the underlying linear algebra libraries, which are a black box to
us as far as this is concerned.

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Andreas Hilboll
On 05.06.2014 17:49, Frédéric Parrenin wrote:
> Is it not possible to bug report those linear algebra libraries?

If you could pinpoint the problem to scipy.optimize.leastsq, you could
look into the 'ier' and 'mesg' return values to find a reason for
possible problems.

Cheers,
  Andreas.


>
>
> 2014-06-05 17:33 GMT+02:00 Robert Kern <[hidden email]
> <mailto:[hidden email]>>:
>
>     On Thu, Jun 5, 2014 at 4:14 PM, Frédéric Parrenin
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     > Thanks for the hint.
>     >
>     > I would suggest to output a warning in scipy.optimize.leastsq() and
>     > numpy.linalg.solve() when such circumstances happen.
>
>     We don't know when that happens. It's not our code that is doing it
>     but the underlying linear algebra libraries, which are a black box to
>     us as far as this is concerned.
>
>     --
>     Robert Kern
>     _______________________________________________
>     SciPy-User mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


--
-- Andreas.
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Robert Kern-2
In reply to this post by Frédéric Parrenin-2
On Thu, Jun 5, 2014 at 4:49 PM, Frédéric Parrenin
<[hidden email]> wrote:
> Is it not possible to bug report those linear algebra libraries?

We build against BLAS and LAPACK, which are API specifications that
are implemented by many different vendors. One can build numpy and
scipy against any of those implementations. You can try to track down
the implementation of the BLAS and LAPACK library that you are
currently using and try to fix it, if it's an open source one.

These problems usually happen when there is some degeneracy or
ill-conditioning in your data. Your time might be better spent
identifying the source of that degeneracy. It might be causing you
more problems than indeterminacy from run to run (i.e. *both* of your
answers might be wrong). A bit of deterministic preconditioning on
your side might eliminate the problem.

--
Robert Kern
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Sturla Molden-3
Robert Kern <[hidden email]> wrote:

> We build against BLAS and LAPACK, which are API specifications that
> are implemented by many different vendors. One can build numpy and
> scipy against any of those implementations. You can try to track down
> the implementation of the BLAS and LAPACK library that you are
> currently using and try to fix it, if it's an open source one.

scipy.optimize.leastsq uses a QR solver from MINPACK.



Sturla

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Pauli Virtanen-3
In reply to this post by Frédéric Parrenin-2
Frédéric Parrenin <parrenin <at> ujf-grenoble.fr> writes:
> Is it not possible to bug report those linear algebra libraries?

The behavior is not a bug, and not really fixable by the linear algebra
libraries either. Modern compilers produce code that can lead to
nondeterministic behavior under circumstances (think SSE and memory
allocation alignment).

--
Pauli Virtanen

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: deterministic python code which is unstable

Pauli Virtanen-3
Pauli Virtanen <pav <at> iki.fi> writes:
> Frédéric Parrenin <parrenin <at> ujf-grenoble.fr> writes:
> > Is it not possible to bug report those linear algebra libraries?
>
> The behavior is not a bug, and not really fixable by the linear algebra
> libraries either. Modern compilers produce code that can lead to
> nondeterministic behavior under circumstances (think SSE and memory
> allocation alignment).

http://software.intel.com/sites/default/files/article/164389/fp-consistency-122712_1.pdf

i.e., the alternatives are

(i) faster execution
(ii) reproducibility of rounding error

Compilers default usually to (i).

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user