10 messages
Open this post in threaded view
|
Report Content as Inappropriate

 Hi all I use scipy.stats.mannwhitneyu extensively because my data is not at all normal. I have run into a few "gotchas" with this function and I wanted to discuss possible workarounds with the list. 1) When this function returns a significant result, it is non-trivial to determine the direction of the effect! The Mann-Whitney test is NOT a test on difference of medians or means, so you cannot determine the direction from these statistics. Wikipedia has a good example of why it is not a test for difference of median. http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Illustration_of_object_of_testI've reprinted it here. The data are the finishing order of hares and tortoises. Obviously this is contrived but it indicates the problem. First the setup: results_l = 'H H H H H H H H H T T T T T T T T T T H H H H H H H H H H T T T T T T T T T'.split(' ') h = [i for i in range(len(results_l)) if results_l[i] == 'H'] t = [i for i in range(len(results_l)) if results_l[i] == 'T'] And the results: In [12]: scipy.stats.mannwhitneyu(h, t) Out[12]: (100.0, 0.0097565768849708391) In [13]: np.median(h), np.median(t) Out[13]: (19.0, 18.0) Hares are significantly faster than tortoises, but we cannot determine this from the output of mannwhitneyu. This could be fixed by either returning u1 and u2 from the guts of the function, or testing them in the function and returning the comparison. My current workaround is testing the means which is absolutely wrong in theory but usually correct in practice. 2) The documentation states that the sample sizes must be at least 20. I think this is because the normal approximation for U is not valid for smaller sample sizes. Is there a table of critical values for U in scipy.stats that is appropriate for small sample sizes or should the user implement his or her own? 3) This is picky but is there a reason that it returns a one-tailed p-value, while other tests (eg ttest_*) default to two-tailed? Thanks for any thoughts, tips, or corrections and please don't take these comments as criticisms ... if I didn't enjoy using scipy.stats so much I wouldn't bother bringing this up! Chris _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user
Open this post in threaded view
|
Report Content as Inappropriate

 On Thu, Feb 14, 2013 at 7:06 PM, Chris Rodgers <[hidden email]> wrote: > Hi all > > I use scipy.stats.mannwhitneyu extensively because my data is not at > all normal. I have run into a few "gotchas" with this function and I > wanted to discuss possible workarounds with the list. Can you open a ticket ? http://projects.scipy.org/scipy/reportI partially agree, but any changes won't be backwards compatible, and I don't have time to think about this enough. > > 1) When this function returns a significant result, it is non-trivial > to determine the direction of the effect! The Mann-Whitney test is NOT > a test on difference of medians or means, so you cannot determine the > direction from these statistics. Wikipedia has a good example of why > it is not a test for difference of median. > http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Illustration_of_object_of_test> > I've reprinted it here. The data are the finishing order of hares and > tortoises. Obviously this is contrived but it indicates the problem. > First the setup: > results_l = 'H H H H H H H H H T T T T T T T T T T H H H H H H H H H H > T T T T T T T T T'.split(' ') > h = [i for i in range(len(results_l)) if results_l[i] == 'H'] > t = [i for i in range(len(results_l)) if results_l[i] == 'T'] > > And the results: > In [12]: scipy.stats.mannwhitneyu(h, t) > Out[12]: (100.0, 0.0097565768849708391) > > In [13]: np.median(h), np.median(t) > Out[13]: (19.0, 18.0) > > Hares are significantly faster than tortoises, but we cannot determine > this from the output of mannwhitneyu. This could be fixed by either > returning u1 and u2 from the guts of the function, or testing them in > the function and returning the comparison. My current workaround is > testing the means which is absolutely wrong in theory but usually > correct in practice. In some cases I'm reluctant to return the direction when we use a two-sided test. In this case we don't have a one sided tests. In analogy to ttests, I think we could return the individual u1, u2 > > 2) The documentation states that the sample sizes must be at least 20. > I think this is because the normal approximation for U is not valid > for smaller sample sizes. Is there a table of critical values for U in > scipy.stats that is appropriate for small sample sizes or should the > user implement his or her own? not available in scipy. I never looked at this. pull requests for this are welcome if it works. It would be backwards compatible. > > 3) This is picky but is there a reason that it returns a one-tailed > p-value, while other tests (eg ttest_*) default to two-tailed? legacy wart, that I don't like,  but it wasn't offending me enough to change it. > > > Thanks for any thoughts, tips, or corrections and please don't take > these comments as criticisms ... if I didn't enjoy using scipy.stats > so much I wouldn't bother bringing this up! Thanks for the feedback. In large parts review of the functions relies on comments by users (and future contributors). The main problem is how to make changes without breaking current usage, since many of those functions are widely used. Josef > > Chris > _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user_______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user
Open this post in threaded view
|
Report Content as Inappropriate

 On Fri, Feb 15, 2013 at 11:16 AM,  <[hidden email]> wrote: > On Thu, Feb 14, 2013 at 7:06 PM, Chris Rodgers <[hidden email]> wrote: >> Hi all >> >> I use scipy.stats.mannwhitneyu extensively because my data is not at >> all normal. I have run into a few "gotchas" with this function and I >> wanted to discuss possible workarounds with the list. > > Can you open a ticket ? http://projects.scipy.org/scipy/report> > I partially agree, but any changes won't be backwards compatible, and > I don't have time to think about this enough. > >> >> 1) When this function returns a significant result, it is non-trivial >> to determine the direction of the effect! The Mann-Whitney test is NOT >> a test on difference of medians or means, so you cannot determine the >> direction from these statistics. Wikipedia has a good example of why >> it is not a test for difference of median. >> http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Illustration_of_object_of_test>> >> I've reprinted it here. The data are the finishing order of hares and >> tortoises. Obviously this is contrived but it indicates the problem. >> First the setup: >> results_l = 'H H H H H H H H H T T T T T T T T T T H H H H H H H H H H >> T T T T T T T T T'.split(' ') >> h = [i for i in range(len(results_l)) if results_l[i] == 'H'] >> t = [i for i in range(len(results_l)) if results_l[i] == 'T'] >> >> And the results: >> In [12]: scipy.stats.mannwhitneyu(h, t) >> Out[12]: (100.0, 0.0097565768849708391) >> >> In [13]: np.median(h), np.median(t) >> Out[13]: (19.0, 18.0) >> >> Hares are significantly faster than tortoises, but we cannot determine >> this from the output of mannwhitneyu. This could be fixed by either >> returning u1 and u2 from the guts of the function, or testing them in >> the function and returning the comparison. My current workaround is >> testing the means which is absolutely wrong in theory but usually >> correct in practice. > > In some cases I'm reluctant to return the direction when we use a > two-sided test. In this case we don't have a one sided tests. > In analogy to ttests, I think we could return the individual u1, u2 to expand a bit: For the Kolmogorov Smirnov test, we refused to return an indication of the direction. The alternative is two-sided and the distribution of the test statististic and the test statistic are different in the one-sided test. So we shouldn't draw any one-sided conclusions from the two-sided test. In the t_test and mannwhitenyu the test statistic is normally distributed (in large samples), so we can infer the one-sided test from the two-sided statistic and p-value. If there are tables for the small sample case, we would need to check if we get consistent interpretation between one- and two-sided tests. Josef > >> >> 2) The documentation states that the sample sizes must be at least 20. >> I think this is because the normal approximation for U is not valid >> for smaller sample sizes. Is there a table of critical values for U in >> scipy.stats that is appropriate for small sample sizes or should the >> user implement his or her own? > > not available in scipy. I never looked at this. > pull requests for this are welcome if it works. It would be backwards > compatible. > >> >> 3) This is picky but is there a reason that it returns a one-tailed >> p-value, while other tests (eg ttest_*) default to two-tailed? > > legacy wart, that I don't like,  but it wasn't offending me enough to change it. > >> >> >> Thanks for any thoughts, tips, or corrections and please don't take >> these comments as criticisms ... if I didn't enjoy using scipy.stats >> so much I wouldn't bother bringing this up! > > Thanks for the feedback. > In large parts review of the functions relies on comments by users > (and future contributors). > > The main problem is how to make changes without breaking current > usage, since many of those functions are widely used. > > Josef > > >> >> Chris >> _______________________________________________ >> SciPy-User mailing list >> [hidden email] >> http://mail.scipy.org/mailman/listinfo/scipy-user_______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user
Open this post in threaded view
|
Report Content as Inappropriate

 On Fri, Feb 15, 2013 at 11:35 AM,  <[hidden email]> wrote: > On Fri, Feb 15, 2013 at 11:16 AM,  <[hidden email]> wrote: >> On Thu, Feb 14, 2013 at 7:06 PM, Chris Rodgers <[hidden email]> wrote: >>> Hi all >>> >>> I use scipy.stats.mannwhitneyu extensively because my data is not at >>> all normal. I have run into a few "gotchas" with this function and I >>> wanted to discuss possible workarounds with the list. >> >> Can you open a ticket ? http://projects.scipy.org/scipy/report>> >> I partially agree, but any changes won't be backwards compatible, and >> I don't have time to think about this enough. >> >>> >>> 1) When this function returns a significant result, it is non-trivial >>> to determine the direction of the effect! The Mann-Whitney test is NOT >>> a test on difference of medians or means, so you cannot determine the >>> direction from these statistics. Wikipedia has a good example of why >>> it is not a test for difference of median. >>> http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Illustration_of_object_of_test>>> >>> I've reprinted it here. The data are the finishing order of hares and >>> tortoises. Obviously this is contrived but it indicates the problem. >>> First the setup: >>> results_l = 'H H H H H H H H H T T T T T T T T T T H H H H H H H H H H >>> T T T T T T T T T'.split(' ') >>> h = [i for i in range(len(results_l)) if results_l[i] == 'H'] >>> t = [i for i in range(len(results_l)) if results_l[i] == 'T'] >>> >>> And the results: >>> In [12]: scipy.stats.mannwhitneyu(h, t) >>> Out[12]: (100.0, 0.0097565768849708391) >>> >>> In [13]: np.median(h), np.median(t) >>> Out[13]: (19.0, 18.0) >>> >>> Hares are significantly faster than tortoises, but we cannot determine >>> this from the output of mannwhitneyu. This could be fixed by either >>> returning u1 and u2 from the guts of the function, or testing them in >>> the function and returning the comparison. My current workaround is >>> testing the means which is absolutely wrong in theory but usually >>> correct in practice. >> >> In some cases I'm reluctant to return the direction when we use a >> two-sided test. In this case we don't have a one sided tests. >> In analogy to ttests, I think we could return the individual u1, u2 > > to expand a bit: > For the Kolmogorov Smirnov test, we refused to return an indication of > the direction. The alternative is two-sided and the distribution of > the test statististic and the test statistic are different in the > one-sided test. > So we shouldn't draw any one-sided conclusions from the two-sided test. > > In the t_test and mannwhitenyu the test statistic is normally > distributed (in large samples), so we can infer the one-sided test > from the two-sided statistic and p-value. > > If there are tables for the small sample case, we would need to check > if we get consistent interpretation between one- and two-sided tests. > > Josef > >> >>> >>> 2) The documentation states that the sample sizes must be at least 20. >>> I think this is because the normal approximation for U is not valid >>> for smaller sample sizes. Is there a table of critical values for U in >>> scipy.stats that is appropriate for small sample sizes or should the >>> user implement his or her own? >> >> not available in scipy. I never looked at this. >> pull requests for this are welcome if it works. It would be backwards >> compatible. since I just looked at a table collection for some other test, they also have Mann-Whitney U statistic http://faculty.washington.edu/heagerty/Books/Biostatistics/TABLES/Wilcoxon/but I didn't check if it matches the test statistic in scipy.stats Josef >> >>> >>> 3) This is picky but is there a reason that it returns a one-tailed >>> p-value, while other tests (eg ttest_*) default to two-tailed? >> >> legacy wart, that I don't like,  but it wasn't offending me enough to change it. >> >>> >>> >>> Thanks for any thoughts, tips, or corrections and please don't take >>> these comments as criticisms ... if I didn't enjoy using scipy.stats >>> so much I wouldn't bother bringing this up! >> >> Thanks for the feedback. >> In large parts review of the functions relies on comments by users >> (and future contributors). >> >> The main problem is how to make changes without breaking current >> usage, since many of those functions are widely used. >> >> Josef >> >> >>> >>> Chris >>> _______________________________________________ >>> SciPy-User mailing list >>> [hidden email] >>> http://mail.scipy.org/mailman/listinfo/scipy-user_______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user
Open this post in threaded view
|
Report Content as Inappropriate

 Thanks Josef. Your points make sense to me. While we're on the subject, maybe I should ask whether this function is even appropriate for my data. My data are Poisson-like integer counts, and I want to know if the rate is significantly higher in dataset1 or dataset2. I'm reluctant to use poissfit because there is a scientific reason to believe that my data might deviate significantly from Poisson, although I haven't checked this statistically. Mann-whitney U seemed like a safe alternative because it doesn't make distributional assumptions and it deals with ties, which is especially important for me because half the counts or more can be zero. Does that seem like a good choice, as long as I have >20 samples and the large-sample approximation is appropriate? Comments welcome. Thanks Chris On Fri, Feb 15, 2013 at 8:58 AM,  <[hidden email]> wrote: > On Fri, Feb 15, 2013 at 11:35 AM,  <[hidden email]> wrote: >> On Fri, Feb 15, 2013 at 11:16 AM,  <[hidden email]> wrote: >>> On Thu, Feb 14, 2013 at 7:06 PM, Chris Rodgers <[hidden email]> wrote: >>>> Hi all >>>> >>>> I use scipy.stats.mannwhitneyu extensively because my data is not at >>>> all normal. I have run into a few "gotchas" with this function and I >>>> wanted to discuss possible workarounds with the list. >>> >>> Can you open a ticket ? http://projects.scipy.org/scipy/report>>> >>> I partially agree, but any changes won't be backwards compatible, and >>> I don't have time to think about this enough. >>> >>>> >>>> 1) When this function returns a significant result, it is non-trivial >>>> to determine the direction of the effect! The Mann-Whitney test is NOT >>>> a test on difference of medians or means, so you cannot determine the >>>> direction from these statistics. Wikipedia has a good example of why >>>> it is not a test for difference of median. >>>> http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Illustration_of_object_of_test>>>> >>>> I've reprinted it here. The data are the finishing order of hares and >>>> tortoises. Obviously this is contrived but it indicates the problem. >>>> First the setup: >>>> results_l = 'H H H H H H H H H T T T T T T T T T T H H H H H H H H H H >>>> T T T T T T T T T'.split(' ') >>>> h = [i for i in range(len(results_l)) if results_l[i] == 'H'] >>>> t = [i for i in range(len(results_l)) if results_l[i] == 'T'] >>>> >>>> And the results: >>>> In [12]: scipy.stats.mannwhitneyu(h, t) >>>> Out[12]: (100.0, 0.0097565768849708391) >>>> >>>> In [13]: np.median(h), np.median(t) >>>> Out[13]: (19.0, 18.0) >>>> >>>> Hares are significantly faster than tortoises, but we cannot determine >>>> this from the output of mannwhitneyu. This could be fixed by either >>>> returning u1 and u2 from the guts of the function, or testing them in >>>> the function and returning the comparison. My current workaround is >>>> testing the means which is absolutely wrong in theory but usually >>>> correct in practice. >>> >>> In some cases I'm reluctant to return the direction when we use a >>> two-sided test. In this case we don't have a one sided tests. >>> In analogy to ttests, I think we could return the individual u1, u2 >> >> to expand a bit: >> For the Kolmogorov Smirnov test, we refused to return an indication of >> the direction. The alternative is two-sided and the distribution of >> the test statististic and the test statistic are different in the >> one-sided test. >> So we shouldn't draw any one-sided conclusions from the two-sided test. >> >> In the t_test and mannwhitenyu the test statistic is normally >> distributed (in large samples), so we can infer the one-sided test >> from the two-sided statistic and p-value. >> >> If there are tables for the small sample case, we would need to check >> if we get consistent interpretation between one- and two-sided tests. >> >> Josef >> >>> >>>> >>>> 2) The documentation states that the sample sizes must be at least 20. >>>> I think this is because the normal approximation for U is not valid >>>> for smaller sample sizes. Is there a table of critical values for U in >>>> scipy.stats that is appropriate for small sample sizes or should the >>>> user implement his or her own? >>> >>> not available in scipy. I never looked at this. >>> pull requests for this are welcome if it works. It would be backwards >>> compatible. > > since I just looked at a table collection for some other test, they > also have Mann-Whitney U statistic > http://faculty.washington.edu/heagerty/Books/Biostatistics/TABLES/Wilcoxon/> but I didn't check if it matches the test statistic in scipy.stats > > Josef > >>> >>>> >>>> 3) This is picky but is there a reason that it returns a one-tailed >>>> p-value, while other tests (eg ttest_*) default to two-tailed? >>> >>> legacy wart, that I don't like,  but it wasn't offending me enough to change it. >>> >>>> >>>> >>>> Thanks for any thoughts, tips, or corrections and please don't take >>>> these comments as criticisms ... if I didn't enjoy using scipy.stats >>>> so much I wouldn't bother bringing this up! >>> >>> Thanks for the feedback. >>> In large parts review of the functions relies on comments by users >>> (and future contributors). >>> >>> The main problem is how to make changes without breaking current >>> usage, since many of those functions are widely used. >>> >>> Josef >>> >>> >>>> >>>> Chris >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> [hidden email] >>>> http://mail.scipy.org/mailman/listinfo/scipy-user> _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user_______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user
Open this post in threaded view
|
Report Content as Inappropriate

Open this post in threaded view
|
Report Content as Inappropriate

Open this post in threaded view
|
Report Content as Inappropriate

Open this post in threaded view
|
Report Content as Inappropriate