Weird randomization issue with multiprocessing module

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird randomization issue with multiprocessing module

zoof
Hi,

I have a very strange problem that maybe someone might have some ideas
as to what is going on.  I'm making a series of random draws across
multiple processor threads on a Debian Wheezy, dual core AMD Phenom box
with scipy 0.10.1 and numpy 1.6.2 (once I debug my code, I will run it
on a more powerful workstation).  Since each series of draws is in a
separate thread, I would have presumed that the draws are completely
independent of one another.  But strangely, for a total of 1000 threads
(managed by multiprocessing.Pool), early in the series of threads, I see
precisely the same numbers in some of the variables produced by two
adjacent threads.  The probability of this happening should be
vanishingly small but strangely happens twice within the first 25
threads and by cursory examination not at all in the remaining 975.  The
numbers are being drawn using scipy.stats.norm and random.random.

Any ideas?

Thanks in advance,
Ted To
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Weird randomization issue with multiprocessing module

Thomas Kluyver-2
On 30 November 2012 17:25, Ted To <[hidden email]> wrote:
The
numbers are being drawn using scipy.stats.norm and random.random.

I think the default seed for the random number generator uses the system time, so maybe if your threads/processes spin up too quickly, some of them could get the same seed. But it says it should also be using /dev/urandom in the seed, which should negate that.

Thomas

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Weird randomization issue with multiprocessing module

josef.pktd
On Fri, Nov 30, 2012 at 12:43 PM, Thomas Kluyver <[hidden email]> wrote:

> On 30 November 2012 17:25, Ted To <[hidden email]> wrote:
>>
>> The
>> numbers are being drawn using scipy.stats.norm and random.random.
>
>
> I think the default seed for the random number generator uses the system
> time, so maybe if your threads/processes spin up too quickly, some of them
> could get the same seed. But it says it should also be using /dev/urandom in
> the seed, which should negate that.

for some discussion
http://mail.scipy.org/pipermail/numpy-discussion/2008-December/039179.html

Josef

>
> Thomas
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Weird randomization issue with multiprocessing module

zoof
On 11/30/2012 12:57 PM, [hidden email] wrote:

> On Fri, Nov 30, 2012 at 12:43 PM, Thomas Kluyver <[hidden email]> wrote:
>> On 30 November 2012 17:25, Ted To <[hidden email]> wrote:
>>>
>>> The
>>> numbers are being drawn using scipy.stats.norm and random.random.
>>
>>
>> I think the default seed for the random number generator uses the system
>> time, so maybe if your threads/processes spin up too quickly, some of them
>> could get the same seed. But it says it should also be using /dev/urandom in
>> the seed, which should negate that.
>
> for some discussion
> http://mail.scipy.org/pipermail/numpy-discussion/2008-December/039179.html

Many thanks -- adding a seed() at the beginning of the routine did the
trick!
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user