Dear SciPy folks,
Those of you whose duties include teaching basic stats might be interested in these interactive tutorial files, designed to illustrate basic concepts. Running the code opens up an interactive figure window. When you click on a figure to add new points, the statistical tests shown in the figure change accordingly. http://www.dartmouth.edu/~raj/intro-stats.html The code has lots of comments in it, which attempt to explain the concepts as explicitly as possible. No prior knowledge of Python or statistics is assumed. These programs require the SciPy and Matplotlib modules. The tutorials are: - interactive_mean_std_normal_distribution.py - interactive_one_sample_t_test.py - interactive_two_sample_t_test.py - interactive_correlation_plot.py The same webpage also contains Matlab versions of the scripts. Please feel more than free to use any of the code for teaching, if you find it useful. Yours, Rajeev Raizada Research Assistant Professor Neukom Institute for Computational Science Dartmouth College HB 6255 Hanover NH 03755 Tel: 603 646 0175 E.mail: [hidden email] WWW: http://www.dartmouth.edu/~raj _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 6:54 AM, Rajeev Raizada
<[hidden email]> wrote: > > Those of you whose duties include teaching basic stats > might be interested in these interactive tutorial files, > designed to illustrate basic concepts. Wonderful codes! Many thanks for sharing these, they are a fantastic illustration... <shameless plug> For those using the ipython qt console from git's master, you can load Rajeevs' codes straight in via: %loadpy http://www.dartmouth.edu/~raj/Python/interactive_mean_std_normal_distribution.py %loadpy http://www.dartmouth.edu/~raj/Python/interactive_two_sample_t_test.py %loadpy http://www.dartmouth.edu/~raj/Python/interactive_correlation_plot.py </plug> Cheers, f _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 10:37 AM, Fernando Perez <[hidden email]> wrote:
> On Mon, Feb 14, 2011 at 6:54 AM, Rajeev Raizada > <[hidden email]> wrote: >> >> Those of you whose duties include teaching basic stats >> might be interested in these interactive tutorial files, >> designed to illustrate basic concepts. > > Wonderful codes! Many thanks for sharing these, they are a fantastic > illustration... > > <shameless plug> > For those using the ipython qt console from git's master, you can load > Rajeevs' codes straight in via: > > %loadpy http://www.dartmouth.edu/~raj/Python/interactive_mean_std_normal_distribution.py > > %loadpy http://www.dartmouth.edu/~raj/Python/interactive_two_sample_t_test.py > > %loadpy http://www.dartmouth.edu/~raj/Python/interactive_correlation_plot.py > > </plug> And additionally thanks for the link to http://en.wikipedia.org/wiki/Anscombe%27s_quartet I didn't know about those. (and, to follow in Fernando's footsteps) <plug without links> use higher order moments use general measures of dependence use non-linearity tests use a robust estimator and use partial correlation in the multivariate case </plug> Josef > > Cheers, > > f > _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Rajeev Raizada
Hi Rajeev,
Thanks for sharing this stuff. Sadly I'm having a trouble running these codes on my mac that uses Python 2.7.1 |EPD 7.0-1 (32-bit)|. It raises the following warning: Warning: divide by zero encountered in double_scalars and nothing happens as I click around on the canvas. Is there something that I should possibly do to fix this? Thanks so much, Bhanukiran FOSSEE IIT-B, India. On Mon, Feb 14, 2011 at 8:24 PM, Rajeev Raizada <[hidden email]> wrote: > Dear SciPy folks, > > Those of you whose duties include teaching basic stats > might be interested in these interactive tutorial files, > designed to illustrate basic concepts. > Running the code opens up an interactive figure window. > When you click on a figure to add new points, > the statistical tests shown in the figure change accordingly. > > http://www.dartmouth.edu/~raj/intro-stats.html > > The code has lots of comments in it, which attempt to explain the > concepts as explicitly as possible. No prior knowledge of Python or > statistics is assumed. These programs require the SciPy and > Matplotlib modules. > > The tutorials are: > - interactive_mean_std_normal_distribution.py > - interactive_one_sample_t_test.py > - interactive_two_sample_t_test.py > - interactive_correlation_plot.py > > The same webpage also contains Matlab versions of the scripts. > Please feel more than free to use any of the code for teaching, > if you find it useful. > > Yours, > > Rajeev Raizada > Research Assistant Professor > Neukom Institute for Computational Science > Dartmouth College > HB 6255 > Hanover NH 03755 > Tel: 603 646 0175 > E.mail: [hidden email] > WWW: http://www.dartmouth.edu/~raj > _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user > SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 6:04 PM, bhanukiran perabathini
<[hidden email]> wrote: > and nothing happens as I click around on the canvas. This seems to be a bug in some of the matplotlib backends. If you grab the window and resize it, everything should appear (let's hope). Cheers Stéfan _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
2011/2/14 Stéfan van der Walt <[hidden email]>:
> On Mon, Feb 14, 2011 at 6:04 PM, bhanukiran perabathini > <[hidden email]> wrote: >> and nothing happens as I click around on the canvas. > > This seems to be a bug in some of the matplotlib backends. If you > grab the window and resize it, everything should appear (let's hope). > Same for me on linux with a tarball from svn mpl 1.0 rev 8222 and 'WXAgg' or 'Qt4Agg' backend (only 2 I tried). It works as expected if I run ipython with the -pylab switch though. Skipper _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by josef.pktd
On Mon, Feb 14, 2011 at 11:03 AM, <[hidden email]> wrote:
> On Mon, Feb 14, 2011 at 10:37 AM, Fernando Perez <[hidden email]> wrote: >> On Mon, Feb 14, 2011 at 6:54 AM, Rajeev Raizada >> <[hidden email]> wrote: >>> >>> Those of you whose duties include teaching basic stats >>> might be interested in these interactive tutorial files, >>> designed to illustrate basic concepts. >> >> Wonderful codes! Many thanks for sharing these, they are a fantastic >> illustration... >> >> <shameless plug> >> For those using the ipython qt console from git's master, you can load >> Rajeevs' codes straight in via: >> >> %loadpy http://www.dartmouth.edu/~raj/Python/interactive_mean_std_normal_distribution.py >> >> %loadpy http://www.dartmouth.edu/~raj/Python/interactive_two_sample_t_test.py >> >> %loadpy http://www.dartmouth.edu/~raj/Python/interactive_correlation_plot.py >> >> </plug> > > And additionally thanks for the link to > http://en.wikipedia.org/wiki/Anscombe%27s_quartet > > I didn't know about those. > > (and, to follow in Fernando's footsteps) > > <plug without links> > > use higher order moments > use general measures of dependence > use non-linearity tests > use a robust estimator > and use partial correlation in the multivariate case <plug continued> since it works fine on my computer, and it's fun I added and changed the following lines to plot_the_correlation() in interactive _correlation_plot.py import numpy as np from scikits.statsmodels.api import RLM, add_constant resrlmbeta = RLM(y_coords, add_constant(x_coords)).fit().params fitted_robust = np.dot(add_constant(axis_range*scipy.array([-1,1])), resrlmbeta).ravel() #### Plot the best-fit line in red handle_of_regression_line_plot = pylab.plot(axis_range*scipy.array([-1,1]), np.column_stack((y_intercept + slope*axis_range*scipy.array([-1,1]), fitted_robust)),'r-') I now have two lines, one OLS the other one robust. Setting first some points roughly in a straight line and then adding a few outliers shows the difference between standard OLS and robust estimators. It's a very quick hack: RLM doesn't have a predict method (a bug I just found), although fittedvalues exist. I don't know how to add a second plot() with fittedvalues to the same handle without running to the matplotlib manual, nor how to make separate colors for the lines if they are in the same plot(). This could be very nice to illustrate the different weighting options in RLM, the default doesn't look very robust if there are many outliers. > </plug> Thanks, Josef > > Josef > >> >> Cheers, >> >> f >> _______________________________________________ >> SciPy-User mailing list >> [hidden email] >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 9:45 AM, <[hidden email]> wrote:
> since it works fine on my computer, and it's fun > > I added and changed the following lines to plot_the_correlation() in > interactive _correlation_plot.py Until we have our own nice file exchange, this is a good example of how at least popping these little scripts on https://gist.github.com/ would be good, so your improvements can be added on top of Rajeev's original code cleanly. I don't want to put them up so the first version credit goes to him, though... Cheers, f _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 1:33 PM, Fernando Perez <[hidden email]> wrote:
> On Mon, Feb 14, 2011 at 9:45 AM, <[hidden email]> wrote: >> since it works fine on my computer, and it's fun >> >> I added and changed the following lines to plot_the_correlation() in >> interactive _correlation_plot.py > > Until we have our own nice file exchange, this is a good example of > how at least popping these little scripts on https://gist.github.com/ > would be good, so your improvements can be added on top of Rajeev's > original code cleanly. Except it would add 100% time in administrative overhead for me, and I would still download them and make local changes (as with Alexandre's permutation test), instead of firing up git. (And, when I'm allowed to, they or a variation of them will eventually end up in statsmodels.) :) Josef (Eventually I will learn how to use git and github, but not today.) > > I don't want to put them up so the first version credit goes to him, though... > > Cheers, > > f > _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user > SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 13:29, <[hidden email]> wrote:
> On Mon, Feb 14, 2011 at 1:33 PM, Fernando Perez <[hidden email]> wrote: >> On Mon, Feb 14, 2011 at 9:45 AM, <[hidden email]> wrote: >>> since it works fine on my computer, and it's fun >>> >>> I added and changed the following lines to plot_the_correlation() in >>> interactive _correlation_plot.py >> >> Until we have our own nice file exchange, this is a good example of >> how at least popping these little scripts on https://gist.github.com/ >> would be good, so your improvements can be added on top of Rajeev's >> original code cleanly. > > Except it would add 100% time in administrative overhead for me, and I > would still download them and make local changes (as with Alexandre's > permutation test), instead of firing up git. The gist functionality is entirely web-based. You don't even need a Github account. Just click the link and see. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by josef.pktd
On Mon, Feb 14, 2011 at 11:29 AM, <[hidden email]> wrote:
> Except it would add 100% time in administrative overhead for me, and I > would still download them and make local changes (as with Alexandre's > permutation test), instead of firing up git. (And, when I'm allowed > to, they or a variation of them will eventually end up in > statsmodels.) Not trying to push you in any direction, just wanted to clarify that you can use gist, fork and contribute to a snippet without knowing that git exists. I just grabbed some random snippet: https://gist.github.com/826461 clicked the 'fork' button and then had my own fork, changed the code in the web browser by clicking on the 'Edit' button, and clicked the 'Save Gist' button, and now my fork shows two new revisions: https://gist.github.com/826469 Again, this isn't trying to be pushy with you, just showing how this system lets anyone fork, evolve and contribute to any trivial snippet without even knowing what git is for or having it installed on their computer. All they need is a web browser. Cheers, f _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Robert Kern-2
Hi guys,
Many thanks indeed for the positive feedback about the stats scripts! Sorry about the problems that people have been encountering when using some Matplotlib backends. I'm not sure what to do about that, as I think it may need to be fixed within the Matplotlib backend-code itself. Good to hear that ipython -pylab fixes it, or at least provides a workaround. I had been testing on the Mac OS X backend. The "divide by zero" problem mentioned by Bhanukiran should, I believe, just be a warning printed in the Python window, and should not actually stop the script from running. I had encountered that warning in the correlation-plot script, as calculating a correlation with just two points in it produces a NaN p-value and spits out a warning. So, that script includes a line to disable to warning, although I am not sure if my including that was a good idea. I guess it's possible that the other scripts might produce similar warnings too, although I didn't encounter it myself. Perhaps, e.g. in the one-sample t-test, if there are just two points and those two points have identical y-values? Re Github: I would be more than happy to upload the scripts to there. Please let me know where it would be best for me to put them. Josef's addition of robust stats is very interesting, although these tutorials are meant to be extremely introductory, and the robust-stats add-on may be a bit advanced, from a purely tutorial point of view. However, no harm at all in having both versions available. Thanks again, Raj _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Fernando Perez
On Mon, Feb 14, 2011 at 3:24 PM, Fernando Perez <[hidden email]> wrote:
> On Mon, Feb 14, 2011 at 11:29 AM, <[hidden email]> wrote: >> Except it would add 100% time in administrative overhead for me, and I >> would still download them and make local changes (as with Alexandre's >> permutation test), instead of firing up git. (And, when I'm allowed >> to, they or a variation of them will eventually end up in >> statsmodels.) > > Not trying to push you in any direction, just wanted to clarify that > you can use gist, fork and contribute to a snippet without knowing > that git exists. I just grabbed some random snippet: > > https://gist.github.com/826461 > > clicked the 'fork' button and then had my own fork, changed the code > in the web browser by clicking on the 'Edit' button, and clicked the > 'Save Gist' button, and now my fork shows two new revisions: > > https://gist.github.com/826469 > > Again, this isn't trying to be pushy with you, just showing how this > system lets anyone fork, evolve and contribute to any trivial snippet > without even knowing what git is for or having it installed on their > computer. All they need is a web browser. Thanks for the explanation, I got my first gist clone, and I take back the 100 percent overhead. Is there a way to see the changesets, the diff between different revisions or forks? I didn't find a button. Maybe a nice complement to the mailinglist, which for me still has the advantage of instant search. Josef > > Cheers, > > f > _______________________________________________ > SciPy-User mailing list > [hidden email] > http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Mon, Feb 14, 2011 at 6:27 PM, <[hidden email]> wrote:
> Thanks for the explanation, I got my first gist clone, and I take back > the 100 percent overhead. Glad to hear that. > Is there a way to see the changesets, the diff between different > revisions or forks? I didn't find a button. Unfortunately it's a requested but not yet available feature: http://support.github.com/discussions/feature-requests/542-show-diffs-between-gist-revisions To actually see the diffs, as of now you'll need to get a real git clone and use local tools, I'm afraid. I hope they add a nice UI to compare revisions in a gist's history sometime soon... Cheers, f _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Raj-67
Hi Raj,
On Mon, Feb 14, 2011 at 1:28 PM, Raj <[hidden email]> wrote: > > Re Github: I would be more than happy to upload the scripts to there. > Please let me know where it would be best for me to put them. It's basically up to you: a 'gist' is meant as a lightweight container for one or more (but hopefully a few, say three or four max) files with a simple revision history. The system lets you keep the files, make incremental improvements, and there can be some discussion right there. Here's a good example of a gist with a few files, some history and some discussion: https://gist.github.com/566233 Alternatively, you can create a full-blown git repository. This requires a little more setup (it's not that hard though), but you get a proper named url, a bug tracker, pull requests, a wiki, etc. This is what numpy, ipython, nipy*, etc. use, and in a few days also scipy and matplotlib. I mentioned the gist situation because your snippets seemed like a good use case for such a lightweight, low-overhead tool. But honestly, if you'd like to keep developing them, you might want to consider making a little 'pystats-tutorials' proper git repository so the project can grow more. I can imagine something like this growing over time into a very useful collection of tools for interactive stats demos, and at that point you'll benefit from the full infrastructure of a real github repo. It's still 100% free, and the time you invest in learning how to use the tools will pay tenfold for you on many fronts. And if you get stuck with git, head over to J. Haxby's lab and talk to Yarick or Michael H., they are more or less world-class gurus on that stuff... A number of us really got into these tools after watching those two in awe. Best, f _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
Hi Fernando,
Thanks for the gist info. I'll try it out soon! By the way, I made a YouTube video showing the interactive demos in action: http://www.youtube.com/watch?&v=nCv_MhaeFo8 I'd love to hear any comments or suggestions. Raj _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Fernando Perez
Hi Fernando,
Thanks for the info about setting up a gist. Very helpful! I'll try it soon. By the way, I made a YouTube video showing the interactive demos in action: http://www.youtube.com/watch?v=nCv_MhaeFo8 I'd be very interested to hear any comments or suggestions. I also put a note into the scripts and on the webpage suggesting that Linux users run them in an iPython started with "ipython -pylab", in order to work around the problems with some Matplotlib backends. Raj P.S. I'll convey your enthusiastic comments to Yarik and Michael about their git virtuosity! Our offices are very near to each other and we use the same coffee machine, so I see them all the time. _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by josef.pktd
On Mon, Feb 14, 2011 at 10:03 AM, <[hidden email]> wrote:
> And additionally thanks for the link to > http://en.wikipedia.org/wiki/Anscombe%27s_quartet See also http://matplotlib.sourceforge.net/examples/pylab_examples/anscombe.html Very nice examples... _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
In reply to this post by Rajeev Raizada
> 2011/2/14 Stéfan van der Walt <[hidden email]>: > > On Mon, Feb 14, 2011 at 6:04 PM, bhanukiran perabathini > > <[hidden email]> wrote: > >> and nothing happens as I click around on the canvas. > > > > This seems to be a bug in some of the matplotlib backends. If you > > grab the window and resize it, everything should appear (let's hope). > > > Same for me on linux with a tarball from svn mpl 1.0 rev 8222 and > 'WXAgg' or 'Qt4Agg' backend (only 2 I tried). It works as expected if > I run ipython with the -pylab switch though. > Skipper For those for which nothing seems to happen when you click on the canvas, Adding pylab.ion() right after the comment ####################################################################### # This is the main part of the program, which calls the above functions ####################################################################### seems to fix the problem, at least on Ubuntu 9.10, matplotlib 0.99.0, GtkAgg backend. Thank you, Rajeev Raizada, for the great code! _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
On Feb 17, 12:01 pm, Nathaniel Peterson
<[hidden email]> wrote: > For those for which nothing seems to happen when you click on the canvas, > Adding > pylab.ion() > right after the comment > ####################################################################### > # This is the main part of the program, which calls the above functions > ####################################################################### > seems to fix the problem, at least on Ubuntu 9.10, matplotlib 0.99.0, > GtkAgg backend. Thanks, Nathaniel! I confess that I don't really understand the ion() command. The Matplotlib documentation says that this turns on interactive mode, but what exactly does that entail? I thought that interactive mode referred only to when you are running a Python session and typing in new commands, with each new plotting command adding to the plot straight away. But I guess it must also have some effect on GUI behaviour during the running of a script, too. I would be very grateful to hear people's thoughts on whether they might be any potential downsides to inserting this ion() command. Apparently it can sometimes make graphics much more sluggish (?). Matplotlib GUIs appear to run slightly differently on Linux depending upon which backend is being used, and I don't know enough to know whether this ion() command would help to make the scripts' behaviour either more consistent or less consistent across various backends. Also, I don't know enough to know whether the ion() command ends up having the same effect as running the scripts from an "ipython -pylab" session. Perhaps there end up being some subtle differences (?) > Thank you, Rajeev Raizada, for the great code! Thank you for the very positive feedback! Much appreciated! Sorry about the Linux/backend complications. That stuff is way beyond my understanding! :-) Raj _______________________________________________ SciPy-User mailing list [hidden email] http://mail.scipy.org/mailman/listinfo/scipy-user |
Free forum by Nabble | Edit this page |