Pylab - standard packages

classic Classic list List threaded Threaded
150 messages Options
1234 ... 8
Reply | Threaded
Open this post in threaded view
|

Pylab - standard packages

Thomas Kluyver-2
Hi again,

It now looks like we're going to use Pylab as the name for the 'scipy
stack'. Now I want to turn to the question of what it should include.
The idea is that Python distributions can call themselves Pylab
compliant if they provide at least a defined set of packages. Also, I
hope that it will become a metapackage in Linux distributions, so that
users can 'apt-get install pylab' or similar

As a minimum, I assume we should require Python, Numpy, Scipy and
Matplotlib. Does anyone disagree?

I also think we should specify minimum versions. The standard itself
will be versioned, so we can raise these over time. For Python, I
intend the requirement to be 2.x >= 2.6 or 3.x >= 3.2. What are
sensible minimum versions of Numpy, Scipy and Matplotlib?

Should the standard include an interface? IPython, a more traditional
IDE, or both? On the one hand, specifying a standard interface means
users can share experience better, and exchange richer files, like
IPython notebooks or IDE project structures. Matlab, for instance,
wins praise for including a powerful IDE by default. On the other
hand, we've got several interesting UI efforts still taking shape -
IPython notebooks, Spyder, IEP - and declaring one standard would make
the alternatives less visible. I'm honestly torn on this - I can see
good arguments for and against.

Other scientific packages we might consider include pandas (which
provides functionality similar to core parts of R), Sympy, Cython,
various scikits projects, h5py, and doubtless many others I haven't
thought of. We could also specify general purpose Python packages such
as requests, or a GUI toolkit.

On the NumFOCUS list, Chris Kees raised the idea that there could be
two or more levels of packages, e.g. 'core' and 'recommended'. I don't
think we should add that kind of complexity in the first version, but
keep in mind that we could differentiate it later.

Finally, I mean the standard to specify that the distribution must
offer a way of installing arbitrary extra Python packages into it, so
the standard shouldn't try to include everything you might need for
scientific computing. The aim is to offer a key set of tools so you
can get started without having to add things. "Get started with what?"
is the key question we have to answer. In my field, for example,
statistical tests are fundamental, while symbolic maths is hardly
used.

All your opinions are welcome,

Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Matthew Brett
Hi,

On Tue, Sep 18, 2012 at 11:26 PM, Thomas Kluyver <[hidden email]> wrote:

> Hi again,
>
> It now looks like we're going to use Pylab as the name for the 'scipy
> stack'. Now I want to turn to the question of what it should include.
> The idea is that Python distributions can call themselves Pylab
> compliant if they provide at least a defined set of packages. Also, I
> hope that it will become a metapackage in Linux distributions, so that
> users can 'apt-get install pylab' or similar
>
> As a minimum, I assume we should require Python, Numpy, Scipy and
> Matplotlib. Does anyone disagree?
>
> I also think we should specify minimum versions. The standard itself
> will be versioned, so we can raise these over time. For Python, I
> intend the requirement to be 2.x >= 2.6 or 3.x >= 3.2. What are
> sensible minimum versions of Numpy, Scipy and Matplotlib

I think I don't understand any more what is proposed.  You mean pylab
would be a sort of seal of conformity to the "pylab" standard?  So you
know you have pylab iff you have Python > .... etc?  Then
distributions like EPD and Python XY would be pylab-certified in some
sense?

> Should the standard include an interface? IPython, a more traditional
> IDE, or both? On the one hand, specifying a standard interface means
> users can share experience better, and exchange richer files, like
> IPython notebooks or IDE project structures. Matlab, for instance,
> wins praise for including a powerful IDE by default. On the other
> hand, we've got several interesting UI efforts still taking shape -
> IPython notebooks, Spyder, IEP - and declaring one standard would make
> the alternatives less visible. I'm honestly torn on this - I can see
> good arguments for and against.

I can only say that I invariably install all of numpy, scipy,
matplotlib and ipython.  Is there enough agreement on the virtues of
the more IDE-like GUIs to chose one?  Is there a good reason not to
include Ipython?

I have a hunch that Ipython + notebook may well become standard soon,
requiring some more dependencies such as zeromq, and providing that
would be a significant benefit.

> Other scientific packages we might consider include pandas (which
> provides functionality similar to core parts of R), Sympy, Cython,
> various scikits projects, h5py, and doubtless many others I haven't
> thought of. We could also specify general purpose Python packages such
> as requests, or a GUI toolkit.

Cython seems to me a strong contender there - I would guess a high
proportion of numerical python developers who also use scipy will have
had some need for Cython.  Is there any data you know of to address
that?

I would personally love to see an hdf5 library included - we really
need a good fast standard storage protocol  that we can rely on being
available on the user's system.

Best,

Matthew
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

David Baddeley
In reply to this post by Thomas Kluyver-2
I guess the current main 'Pylab compatible' distributions are EPD and Python(x,y). It might make sense to base the standard around their core components. I'd definitely include Cython in the core package list, and would consider specifying that a c- compiler should be bundled on platforms which don't provide one (ie windows), and that Python (and numpy) headers be provided so that the distribution is build-ready (especially as both easy_install and pip install from source rather than downloading compiled packages).  I'd suggest making any specification of IDE packages relatively loose and say that they should be there, but leave the choice of package to the implementer, eg: 

"A Pylab distribution should include:
- A GUI framework which is compatible with Matplotlib (e.g. wx or Qt - I think it makes sense to push people towards these 2 frameworks rather than, e.g. GTK or Tk)
- An editor which supports python syntax highlighting and ideally treats tab as 4 spaces (a minimalist install might fall back on IDLE)
- An interactive shell  (e.g. IPython)"


From: Thomas Kluyver <[hidden email]>
To: SciPy Users List <[hidden email]>
Sent: Wednesday, 19 September 2012 10:26 AM
Subject: [SciPy-User] Pylab - standard packages

Hi again,

It now looks like we're going to use Pylab as the name for the 'scipy
stack'. Now I want to turn to the question of what it should include.
The idea is that Python distributions can call themselves Pylab
compliant if they provide at least a defined set of packages. Also, I
hope that it will become a metapackage in Linux distributions, so that
users can 'apt-get install pylab' or similar

As a minimum, I assume we should require Python, Numpy, Scipy and
Matplotlib. Does anyone disagree?

I also think we should specify minimum versions. The standard itself
will be versioned, so we can raise these over time. For Python, I
intend the requirement to be 2.x >= 2.6 or 3.x >= 3.2. What are
sensible minimum versions of Numpy, Scipy and Matplotlib?

Should the standard include an interface? IPython, a more traditional
IDE, or both? On the one hand, specifying a standard interface means
users can share experience better, and exchange richer files, like
IPython notebooks or IDE project structures. Matlab, for instance,
wins praise for including a powerful IDE by default. On the other
hand, we've got several interesting UI efforts still taking shape -
IPython notebooks, Spyder, IEP - and declaring one standard would make
the alternatives less visible. I'm honestly torn on this - I can see
good arguments for and against.

Other scientific packages we might consider include pandas (which
provides functionality similar to core parts of R), Sympy, Cython,
various scikits projects, h5py, and doubtless many others I haven't
thought of. We could also specify general purpose Python packages such
as requests, or a GUI toolkit.

On the NumFOCUS list, Chris Kees raised the idea that there could be
two or more levels of packages, e.g. 'core' and 'recommended'. I don't
think we should add that kind of complexity in the first version, but
keep in mind that we could differentiate it later.

Finally, I mean the standard to specify that the distribution must
offer a way of installing arbitrary extra Python packages into it, so
the standard shouldn't try to include everything you might need for
scientific computing. The aim is to offer a key set of tools so you
can get started without having to add things. "Get started with what?"
is the key question we have to answer. In my field, for example,
statistical tests are fundamental, while symbolic maths is hardly
used.

All your opinions are welcome,

Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Thomas Kluyver-2
In reply to this post by Matthew Brett
On 18 September 2012 23:44, Matthew Brett <[hidden email]> wrote:
> I think I don't understand any more what is proposed.  You mean pylab
> would be a sort of seal of conformity to the "pylab" standard?  So you
> know you have pylab iff you have Python > .... etc?  Then
> distributions like EPD and Python XY would be pylab-certified in some
> sense?

I see my proposal as having three parts: the name, the website (which
newcomers find when they search for the name), and the standard, which
distributions like EPD and Python(x,y) will, I hope, conform to. These
distributions are then ways to install pylab - or a dedicated user can
install the pieces individually, and still end up with pylab. I'm not
trying to make a canonical scientific Python distribution - that's a
much bigger challenge.

> Is there a good reason not to include Ipython?

Perhaps other interfaces work better for some use cases, or some
individuals? Python(x,y) favours Spyder, for instance, and ships an
old version of IPython to maintain compatibility. If we included
IPython in the standard, Python(x,y) would probably not meet the
minimum specified version, and would not be Pylab compliant for the
time being.

With my IPython hat on, I hasten to add that we're working with Spyder
developers to improve things, so that hopefully a future version of
Python(x,y) can include an up-to-date version of IPython.

Thanks,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Matthew Brett
Hi,

On Wed, Sep 19, 2012 at 12:03 AM, Thomas Kluyver <[hidden email]> wrote:

> On 18 September 2012 23:44, Matthew Brett <[hidden email]> wrote:
>> I think I don't understand any more what is proposed.  You mean pylab
>> would be a sort of seal of conformity to the "pylab" standard?  So you
>> know you have pylab iff you have Python > .... etc?  Then
>> distributions like EPD and Python XY would be pylab-certified in some
>> sense?
>
> I see my proposal as having three parts: the name, the website (which
> newcomers find when they search for the name), and the standard, which
> distributions like EPD and Python(x,y) will, I hope, conform to. These
> distributions are then ways to install pylab - or a dedicated user can
> install the pieces individually, and still end up with pylab. I'm not
> trying to make a canonical scientific Python distribution - that's a
> much bigger challenge.

OK - but the website and the name point us to the standard, and thence
to some installers for that standard?   You are not proposing any new
installers, but that the standard basically says something like:

""""You have Pylab if you have python >= 2.6, scipy > 0.9.0, ...

Your options for installing these are:
Windows : - python x, y or EPD or individual download and install
OSX : EPD or individual download and install
Linux : apt-get / yum / whatever install python-pylab
"""

So, if we are adding packages to this collection, we are more or less
lobbying python (x, y) or EPD for those changes?

Best,

Matthew
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Thomas Kluyver-2
On 19 September 2012 00:16, Matthew Brett <[hidden email]> wrote:
> OK - but the website and the name point us to the standard, and thence
> to some installers for that standard?   You are not proposing any new
> installers, but that the standard basically says something like:

Yes, that's the idea.

> So, if we are adding packages to this collection, we are more or less
> lobbying python (x, y) or EPD for those changes?

I expect that the packages we're likely to specify are already
included in those distributions. We might end up lobbying for
additions to EPD Free, but I think that its description as 'Scientific
Python essentials' fits with what we're trying to achieve, so
hopefully Enthought are open to dialogue. This page shows what EPD
Free currently includes (packages with tick marks):
http://www.enthought.com/products/epdlibraries.php

David:
> consider specifying that a c- compiler should be bundled on platforms which don't provide one (ie windows),

Good point. Can someone more Windows-savvy suggest how practical this
is? I assume the VS compiler can't be redistributed, so is mingw
sufficiently lightweight to expect all distributions to include it?
Many packages with compiled components provide executable installers
for Windows.

Also, I don't think Macs actually provide a C compiler - when I had to
test stuff on a Mac, I had to install Xcode before I could do
anything. Will distributions need to include a compiler on the Mac as
well, or would the wording of the definition exclude that?

So I'm leaning towards not requiring a compiler, but I could yet be persuaded.

Thanks,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Travis Oliphant-6

On Sep 18, 2012, at 6:51 PM, Thomas Kluyver wrote:

On 19 September 2012 00:16, Matthew Brett <[hidden email]> wrote:
OK - but the website and the name point us to the standard, and thence
to some installers for that standard?   You are not proposing any new
installers, but that the standard basically says something like:

Yes, that's the idea.

So, if we are adding packages to this collection, we are more or less
lobbying python (x, y) or EPD for those changes?

I expect that the packages we're likely to specify are already
included in those distributions. We might end up lobbying for
additions to EPD Free, but I think that its description as 'Scientific
Python essentials' fits with what we're trying to achieve, so
hopefully Enthought are open to dialogue. This page shows what EPD
Free currently includes (packages with tick marks):
http://www.enthought.com/products/epdlibraries.php

Don't forget about the new-comer to the freely-available binary distributions, Anaconda CE: which is a reference implementation as well and includes all the packages we are discussing:   http://www.continuum.io/downloads.html

But, in general, the point of this conversation is not to lobby anyone to change their tools.   It is to define what the community considers to be a reference implementation (however it is obtained) with version numbers of specific packages.


Good point. Can someone more Windows-savvy suggest how practical this
is? I assume the VS compiler can't be redistributed, so is mingw
sufficiently lightweight to expect all distributions to include it?
Many packages with compiled components provide executable installers
for Windows.

It's easy to do.   The next version of Anaconda CE is going to contain a C-compiler for Windows, for example.   You can't really include Cython in the standard without a C-compiler.   This, to me, makes the case for pylab-full (i.e. you want to have some definition that includes Cython, and you need a compiler to put Cython in there). 


Also, I don't think Macs actually provide a C compiler - when I had to
test stuff on a Mac, I had to install Xcode before I could do
anything. Will distributions need to include a compiler on the Mac as
well, or would the wording of the definition exclude that?

The best thing to do is just encourage people to install XCode, I think.  

-Travis


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Thomas Kluyver-2
On 19 September 2012 01:03, Travis Oliphant <[hidden email]> wrote:
> The next version of Anaconda CE is going to contain a C-compiler for
> Windows, for example.

Thanks for this info. Do you have a list of all the packages in
Anaconda CE, for comparison with the lists for EPD [1] and Python(x,y)
[2]?

[1] http://www.enthought.com/products/epdlibraries.php
[2] http://code.google.com/p/pythonxy/wiki/StandardPlugins

> The best thing to do is just encourage people to install XCode, I think.

So would the standard specifically treat Windows differently with
respect to bundling a C compiler, or would the wording somehow
differentiate platforms that don't provide a C compiler, but you've
probably already installed one?

Best wishes,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Jason Grout-5
In reply to this post by David Baddeley
On 9/18/12 6:02 PM, David Baddeley wrote:
> I guess the current main 'Pylab compatible' distributions are EPD and
> Python(x,y).

Sage probably also qualifies as a "scientific python distribution" (as
well as a lot more...).

The package list is here: http://sagemath.org/packages/standard/ (and I
hasten to add that we're just about finished with the review needed for
us to upgrade to IPython 0.13...)

Thanks,

Jason

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Travis Oliphant-6
In reply to this post by Thomas Kluyver-2

On Sep 18, 2012, at 7:15 PM, Thomas Kluyver wrote:

On 19 September 2012 01:03, Travis Oliphant <[hidden email]> wrote:
The next version of Anaconda CE is going to contain a C-compiler for
Windows, for example.

Thanks for this info. Do you have a list of all the packages in
Anaconda CE, for comparison with the lists for EPD [1] and Python(x,y)
[2]?

[1] http://www.enthought.com/products/epdlibraries.php
[2] http://code.google.com/p/pythonxy/wiki/StandardPlugins


At the bottom of the page there is a link to this pop-up window.    

anaconda launcher
bitarray 0.8.0
bitey
cython 0.16
dateutil 1.5
disco 0.4.2 (Linux only)
erlang (Linux only)
flask 0.9
gevent 0.13.7
gevent-websocket 0.3.6
gevent_zeromq 0.2.5
greenlet0.4.0
h5py 2.0.1
hdf5 1.8.9
PIL 1.1.7
ipython 0.13
jinja2 2.6
libevert 2.0.20 llvm 3.1
llvmpy 0.8.2.dev
matplotlib 1.1.1
mpi4py 1.3
mpich2 1.4.1p1
networkx 1.7
nose 1.1.2
numba 0.1.dev
numexpr 2.0.1
numpy 1.7.rc1
opencv 2.4.2
openssl 1.0.1c
pandas 0.8.1
pip 1.1
pixman 0.26.2
py2cairo 1.10.0
pycurl 7.19.0
pygments 1.5
pysal 1.4.0
pysam 0.6
pytables 2.4.0
python 2.7.3
pytz 2012d
pyyaml 3.10
pyzmq 2.2.0
redis 2.4.15 (Linux only)
redis py-2.4.13
requests 0.13.5
scikit-learn 0.11
scikits-image 0.6.1
scipy 0.11.0rc2
sqlalchemy 0.7.8
sqlite 3.7.13
statsmodels 0.4.3
sympy 0.7.1
theano 0.5.0
tornado 2.3
werkzeug 0.8.3
But, this is not complete because the Windows version does include spyder and pyside.  




The best thing to do is just encourage people to install XCode, I think.

So would the standard specifically treat Windows differently with
respect to bundling a C compiler, or would the wording somehow
differentiate platforms that don't provide a C compiler, but you've
probably already installed one?


I think the full standard would assume Cython, but then implementations would get to pick how they supported Cython (i.e. requiring Xcode install on Mac, vs bundling with Windows). 

-Travis





Best wishes,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

josef.pktd


On Tue, Sep 18, 2012 at 8:24 PM, Travis Oliphant <[hidden email]> wrote:

On Sep 18, 2012, at 7:15 PM, Thomas Kluyver wrote:

On 19 September 2012 01:03, Travis Oliphant <[hidden email]> wrote:
The next version of Anaconda CE is going to contain a C-compiler for
Windows, for example.

What c compiler do you get for 64-bit?

I don't know what the audience is for pylab, but if I want to run 64 bit python on Windows then I might not need mingw (if there are still problems with 64 bit version) and the only alternative is Microsoft compilers.

my 64 bit python 3.2 is all Gohlke binaries, without any compiler available yet
one of my 32 bit pythons is python-xy with bundled c compiler

So, c compiler looks optional to me.

Josef
 

Thanks for this info. Do you have a list of all the packages in
Anaconda CE, for comparison with the lists for EPD [1] and Python(x,y)
[2]?

[1] http://www.enthought.com/products/epdlibraries.php
[2] http://code.google.com/p/pythonxy/wiki/StandardPlugins


At the bottom of the page there is a link to this pop-up window.    

anaconda launcher
bitarray 0.8.0
bitey
cython 0.16
dateutil 1.5
disco 0.4.2 (Linux only)
erlang (Linux only)
flask 0.9
gevent 0.13.7
gevent-websocket 0.3.6
gevent_zeromq 0.2.5
greenlet0.4.0
h5py 2.0.1
hdf5 1.8.9
PIL 1.1.7
ipython 0.13
jinja2 2.6
libevert 2.0.20 llvm 3.1
llvmpy 0.8.2.dev
matplotlib 1.1.1
mpi4py 1.3
mpich2 1.4.1p1
networkx 1.7
nose 1.1.2
numba 0.1.dev
numexpr 2.0.1
numpy 1.7.rc1
opencv 2.4.2
openssl 1.0.1c
pandas 0.8.1
pip 1.1
pixman 0.26.2
py2cairo 1.10.0
pycurl 7.19.0
pygments 1.5
pysal 1.4.0
pysam 0.6
pytables 2.4.0
python 2.7.3
pytz 2012d
pyyaml 3.10
pyzmq 2.2.0
redis 2.4.15 (Linux only)
redis py-2.4.13
requests 0.13.5
scikit-learn 0.11
scikits-image 0.6.1
scipy 0.11.0rc2
sqlalchemy 0.7.8
sqlite 3.7.13
statsmodels 0.4.3
sympy 0.7.1
theano 0.5.0
tornado 2.3
werkzeug 0.8.3
But, this is not complete because the Windows version does include spyder and pyside.  




The best thing to do is just encourage people to install XCode, I think.

So would the standard specifically treat Windows differently with
respect to bundling a C compiler, or would the wording somehow
differentiate platforms that don't provide a C compiler, but you've
probably already installed one?


I think the full standard would assume Cython, but then implementations would get to pick how they supported Cython (i.e. requiring Xcode install on Mac, vs bundling with Windows). 

-Travis





Best wishes,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Christoph Gohlke
In reply to this post by Thomas Kluyver-2
On 9/18/2012 3:26 PM, Thomas Kluyver wrote:

> Hi again,
>
> It now looks like we're going to use Pylab as the name for the 'scipy
> stack'. Now I want to turn to the question of what it should include.
> The idea is that Python distributions can call themselves Pylab
> compliant if they provide at least a defined set of packages. Also, I
> hope that it will become a metapackage in Linux distributions, so that
> users can 'apt-get install pylab' or similar
>
> As a minimum, I assume we should require Python, Numpy, Scipy and
> Matplotlib. Does anyone disagree?
>
> I also think we should specify minimum versions. The standard itself
> will be versioned, so we can raise these over time. For Python, I
> intend the requirement to be 2.x >= 2.6 or 3.x >= 3.2. What are
> sensible minimum versions of Numpy, Scipy and Matplotlib?
>
> Should the standard include an interface? IPython, a more traditional
> IDE, or both? On the one hand, specifying a standard interface means
> users can share experience better, and exchange richer files, like
> IPython notebooks or IDE project structures. Matlab, for instance,
> wins praise for including a powerful IDE by default. On the other
> hand, we've got several interesting UI efforts still taking shape -
> IPython notebooks, Spyder, IEP - and declaring one standard would make
> the alternatives less visible. I'm honestly torn on this - I can see
> good arguments for and against.
>
> Other scientific packages we might consider include pandas (which
> provides functionality similar to core parts of R), Sympy, Cython,
> various scikits projects, h5py, and doubtless many others I haven't
> thought of. We could also specify general purpose Python packages such
> as requests, or a GUI toolkit.
>
> On the NumFOCUS list, Chris Kees raised the idea that there could be
> two or more levels of packages, e.g. 'core' and 'recommended'. I don't
> think we should add that kind of complexity in the first version, but
> keep in mind that we could differentiate it later.
>
> Finally, I mean the standard to specify that the distribution must
> offer a way of installing arbitrary extra Python packages into it, so
> the standard shouldn't try to include everything you might need for
> scientific computing. The aim is to offer a key set of tools so you
> can get started without having to add things. "Get started with what?"
> is the key question we have to answer. In my field, for example,
> statistical tests are fundamental, while symbolic maths is hardly
> used.
>
> All your opinions are welcome,
>
> Thomas

Hello,

the recent poll "Scientific Python packages: Popularity check" by Pierre
Raybaut [1] might be of interest.

In addition to EPD, Pythonxy, AnacondaCE, and Sage, also consider
WinPython [2] and ActivePython [3] as potential "Pylab compatible"
distributions.

[1] http://www.doodle.com/rzssq2dbnus4a34r
[2] http://code.google.com/p/winpython/wiki/PackageIndex
[3]
http://www.activestate.com/activepython/python-financial-scientific-modules

Christoph
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Almar Klein-2
In reply to this post by josef.pktd


On 19 September 2012 02:50, <[hidden email]> wrote:


On Tue, Sep 18, 2012 at 8:24 PM, Travis Oliphant <[hidden email]> wrote:

On Sep 18, 2012, at 7:15 PM, Thomas Kluyver wrote:

On 19 September 2012 01:03, Travis Oliphant <[hidden email]> wrote:
The next version of Anaconda CE is going to contain a C-compiler for
Windows, for example.

What c compiler do you get for 64-bit?

There is the mingw-w64 project: http://mingw-w64.sourceforge.net/. I played with that a while ago. I got it working (when I had the right build), at least for compiling some of my Cython code. Both 32bit and 64bit mingw compilers can be relatively easily shipped with a distribution. They're 120Mb and 240-ish Mb (on disk), respectively.


The best thing to do is just encourage people to install XCode, I think. 

I believe there are now easier ways to install a C compiler on Mac, but I cannot find the link right now.


implementations would get to pick how they supported Cython (i.e. requiring Xcode install on Mac, vs bundling with Windows).

I'm personally also in favor of including Cython. I agree that a distribution does not need to come with a C compiler, as long as it comes with proper instructions on how to get one. Idea: maybe on Windows we can together make a C-compiler-installer-for-pylab, that Windows distributions can share.

regards,
  Almar


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Almar Klein-2
In reply to this post by Thomas Kluyver-2
I also think we should specify minimum versions. 

I think Fernando proposed in the NumFocus thread to specify *exact* version, so that you can exactly reproduce scientific results. I liked that idea. Having said that, I think its probably best to use minimum version first and work on exact versions once we've got the other parts of pylab straightened out...
 

Should the standard include an interface? IPython, a more traditional
IDE, or both? On the one hand, specifying a standard interface means
users can share experience better, and exchange richer files, like
IPython notebooks or IDE project structures. Matlab, for instance,
wins praise for including a powerful IDE by default. On the other
hand, we've got several interesting UI efforts still taking shape -
IPython notebooks, Spyder, IEP - and declaring one standard would make
the alternatives less visible. I'm honestly torn on this - I can see
good arguments for and against.

I think the goal should be to share functionality. Experience is much more subjective. By specifying a single interface you're making it harder for users to use other/new interfaces. A lot of interesting stuff is still going on in these areas, and I don't think there is one interface that is as widely used as numpy is for numerics. You have a good point with sharing of richer files, but maybe we should try keeping it easy to share code between interfaces. In any case, I don't think it justifies selecting one interface in the standard.

The nicety of shipping an IDE with the framework (like Matlab) is something that should be left to the distributions IMO. Python(x,y) does that, and we're working on something similar based on IEP.

One thing that I've liked from the start about this pylab idea, is that it makes distributions "siblings" under one parent (i.e. the pylab standard). It has a unifying effect, and I'm afraid that we lose that if we select one interface.

 
On the NumFOCUS list, Chris Kees raised the idea that there could be
two or more levels of packages, e.g. 'core' and 'recommended'. I don't
think we should add that kind of complexity in the first version, but
keep in mind that we could differentiate it later.

I liked the idea (can't remember where I've read/heard it) that different groups can maintain their own page with packages that are relevant to their field. As someone working in medical imaging I would suggest skimage and pydicom should be in the standard, but as a biologists you may not even know what pydicom is :)  So it's kind of two levels, but the second level is partitioned in different topics. 

regards,
  Almar

--
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: [hidden email]


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Thomas Kluyver-2
In reply to this post by Christoph Gohlke
On 19 September 2012 07:44, Christoph Gohlke <[hidden email]> wrote:
> the recent poll "Scientific Python packages: Popularity check" by Pierre
> Raybaut [1] might be of interest.

Thanks Christoph, that is interesting. After numpy, scipy & matplotlib
(each around 270 votes), other popular modules include PyQt (162), PIL
(118), SymPy (112) and ETS (102). N.B. Cython and IPython weren't
included in the poll.

Almar:
> I liked the idea (can't remember where I've read/heard it) that different groups can maintain their own page with
> packages that are relevant to their field. As someone working in medical imaging I would suggest skimage and
> pydicom should be in the standard, but as a biologists you may not even know what pydicom is :)  So it's kind of
> two levels, but the second level is partitioned in different topics.

I quite like this idea as well, although it's not without its own
troubles. The different fields aren't neatly delineated, so it will be
rather subjective who should find what package useful. And it's
probably too much complexity for distributions to provide multiple
bundled pylab profiles, so users would still be installing those extra
packages themselves.

My issue with the simpler multiple-levels (core vs recommended/full)
idea is that it's not clear who the target audience is. Given that
this is a user-facing name, I think each level needs a clear "get this
version if you ..." story. If we decide that Cython is important and
it should be in the 'full' level, who only needs the 'core' set? The
answer can't include 'if you need to interface with C libraries',
because newcomers won't know what they need in that kind of detail.

Best wishes,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Almar Klein-2


On 19 September 2012 11:29, Thomas Kluyver <[hidden email]> wrote:
On 19 September 2012 07:44, Christoph Gohlke <[hidden email]> wrote:
> the recent poll "Scientific Python packages: Popularity check" by Pierre
> Raybaut [1] might be of interest.

Thanks Christoph, that is interesting. After numpy, scipy & matplotlib
(each around 270 votes), other popular modules include PyQt (162), PIL
(118), SymPy (112) and ETS (102). N.B. Cython and IPython weren't
included in the poll.

Almar:
> I liked the idea (can't remember where I've read/heard it) that different groups can maintain their own page with
> packages that are relevant to their field. As someone working in medical imaging I would suggest skimage and
> pydicom should be in the standard, but as a biologists you may not even know what pydicom is :)  So it's kind of
> two levels, but the second level is partitioned in different topics.

I quite like this idea as well, although it's not without its own
troubles. The different fields aren't neatly delineated, so it will be
rather subjective who should find what package useful. And it's
probably too much complexity for distributions to provide multiple
bundled pylab profiles, so users would still be installing those extra
packages themselves.

I agree. One thing that might happen is that some fields get their own distribution. Or maybe we will finally get packaging right and it won't be a problem... 

One argument in favor of grouping advanced packages in topics is that is will scale better. As more and more fields start using Python, and more and more functionality becomes available, a full-version of pylab would become huge, even though most users only need a specific set of packages. 

My issue with the simpler multiple-levels (core vs recommended/full)
idea is that it's not clear who the target audience is. Given that
this is a user-facing name, I think each level needs a clear "get this
version if you ..." story. If we decide that Cython is important and
it should be in the 'full' level, who only needs the 'core' set? The
answer can't include 'if you need to interface with C libraries',
because newcomers won't know what they need in that kind of detail.

Good point. Plus it adds confusion. On the other hand, a user can still upgrade later if he finds out he needs to. 

  Almar


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Joe Harrington
In reply to this post by Thomas Kluyver-2
At first the idea of a "Pylab" standard appealed to me (it still does,
but with caveats, which see below).  Like everyone, my lab has a
standard Python stack on all our machines, and it's a pretty long list.
It would be nice to cut it in half, and let someone else worry about the
integration issues.  So I tried to make my list of what to include that
wasn't specific to my discipline.

...And I discovered that this idea is a wickedly slippery slope.  While
I have my own reasons for choosing certain packages over others, the
community has organized to encourage lots of topical subpackages, from
broad-use packages like matplotlib to specialist things like astropy.
Even within an area, such as plotting, there are options, and
competition is critical to get great packages.

BUT if your package is in PylabTM and your competition isn't, you have a
*big* advantage.  You'll get installed by every Linux distro, and the
computational Python distros will want to be in compliance as well, so
you'll show up there.  Regardless of intent, if the community makes this
a standard and a distro does not follow it, it will look strange:
tutorials written to the Pylab standard will not work with that distro,
commonly shared code will not run in it without add-ons, etc.  So,
distro maintainers will want to be in compliance if they can be.  By
choosing among packages, Pylab will stifle competition, regardless of
our pure intent.

I still want there to be a Pylab, for selfish reasons cited above.
I can see a couple ways out, gotten to by asking, WHOSE NEEDS ARE WE
TRYING TO SERVE?

1. Make the set only include packages for which there is essentially no
competition.  numpy+scipy+ipython+binary file formats?  I think there is
consensus that Matplotlib is standard, but are we really ready to
formalize the "next-best" status of all other plotting packages?  I
think we actually might be, but I don't think that's true for any other
package with serious competiton.  So
numpy+matplotlib+scipy+ipython+binary file formats.  I am edgy about
ipython, but let's face it, almost everyone uses it.  The fact that it
contains some new material for which the competition hasn't been decided
(i.e., notebook and related features) is a complicating factor.  We will
want to jam other packages in here, like cython and sympy.  If we're
preserving competition, it will have serious holes.  It will be unwieldy
for beginners (too big) and will require lots of add-ons for experienced
people and production use.  DOES IT SERVE ANYONE?  I don't think anyone
will like it.

2. Include *everything*, and don't make choices among competing
packages.  DOES IT SERVE ANYONE?  Maybe big labs with lots of disk
space.

3. Define several levels of Pylab:

Level 1: Stuff needed for beginner tutorials, Numerics 101 classes, etc.
I think just numpy+matplotlib+scipy+ipython+binary file formats.
Declare that beginner tutorials stick to this set to be Pylab-approved
tutorials and get a link from the Pylab-level-1 page.  The tutorials
would also have to comply with coding standards like "import numpy as
np", etc.  We may also declare that compliant tutorials will not use
certain features of Ipython.  I see things like common scientific binary
file formats as very important, as anyone switching from another
language will have data.  DOES IT SERVE ANYONE?  Yes, beginners (and
their teachers).

Level 2: The above plus things like cython, symbolic math, serious
financial functions, unit conversions, serious stats, C, C++, FORTRAN,
etc., but not selecting any package where there is still competition.
Yes, this will leave holes, but people operating at this level are
capable of making choices, and distros are welcome to fill in.  DOES IT
SERVE ANYONE?  It's a good, smallish base for technical users to build
upon.

Level 3: The above plus ALL stable, general-purpose packages that meet a
certain standard, including different competitive variants for the same
task.  DOES IT SERVE ANYONE?  Big installations with one specialty.

Level 4: The above plus specialist packages like astropy.  DOES IT SERVE
ANYONE?  Big installations with many specialties.

I like option 3.  In all my thinking on this problem, I could not find a
minimal set that was useful to advanced users and didn't stifle
competition.  So, instead we have a minimal set that isn't geared toward
advanced users, and that encourages consistency across tutorials.  Then
we have levels of increasing inclusiveness, size, and utility to
advanced users, none of which stifles competition.

Most importantly, the "in" list for any level certifies that all
included packages are stable, actively maintained, well documented, and
do not fight with the rest of the packages.

I think that requiring functionality rather than specific packages is
not very useful.  What utility is there in saying you have an IDE, a
syntax-highlighting editor, etc. without specifying which one?  I can
see saying "a C compiler", though.  Perhaps saying "one or more of ..."
and listing specific IDEs/editors/etc. would be ok.  That at least
provides the service of certifying that those listed actually work well
with the stack.

Finally, I think that there should be installers.  Certifying the work
of others is nice, but the real benefit will be in actually having
something to install.  But if there are two entities, one that certifies
and one that makes an installer for each defined Pylab level, I'm fine
with that.  I'd just hate to see a defined Pylab level for which no pure
installer (i.e., no installer for just those packages) is available.

Thanks, and sorry for the length.

--jh--
Prof. Joseph Harrington
Planetary Sciences Group
Department of Physics
PS 441
4000 Central Florida Blvd.
University of Central Florida
Orlando, FL 32816-2385
USA
... On sabbatical at: ...
Max-Planck-Institut für Astronomie
Königstuhl 17
D-69117 Heidelberg
Germany
[hidden email]
planets.ucf.edu
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

Thomas Kluyver-2
Hi Joe,

On 19 September 2012 11:21, Joe Harrington <[hidden email]> wrote:

> BUT if your package is in PylabTM and your competition isn't, you have a
> *big* advantage.  You'll get installed by every Linux distro, and the
> computational Python distros will want to be in compliance as well, so
> you'll show up there.  Regardless of intent, if the community makes this
> a standard and a distro does not follow it, it will look strange:
> tutorials written to the Pylab standard will not work with that distro,
> commonly shared code will not run in it without add-ons, etc.  So,
> distro maintainers will want to be in compliance if they can be.  By
> choosing among packages, Pylab will stifle competition, regardless of
> our pure intent.

Inevitably, yes, we reduce competition to some extent. But I'm not
sure this is such a bad thing: we want there to be 'one obvious way to
do things'. Having a lot of alternatives can be confusing for users,
and it can fragment developer effort, so none of the alternatives are
as good as they could be.

Of course, we don't want to suppress competition too much -
alternatives can try valuable new ideas and add features that would be
difficult in the standard packages. The standard wouldn't stop people
installing alternatives, or distributions from including those
alternatives as well. That we specify matplotlib as standard doesn't
stop anyone using Chaco, Mayavi or Visvis. And the standard will be
versioned, so if in a couple of years a new package has become
indispensable, we can add it to the standard.

> Make the set only include packages for which there is essentially no
> competition.  numpy+scipy+ipython+binary file formats?

That's tricky to define. Depending on who you ask, for example,
IPython has several competing projects. And if a new competitor to
Numpy were to emerge, would we avoid taking sides, even though our
whole stack depends on Numpy? We even limit the choice for Python
itself, by requiring packages that use the C-API.

We have to take sides in some debates - not as a partisan move to
bolster our favourite projects, but so that users get a coherent stack
of useful tools. Useful tools have competition, and we can't say every
alternative is important.

As an aside, you refer to 'binary file formats' - is there a
particular project for working with those that I should be aware of?

> 3. Define several levels of Pylab:

And now we're up to 4 levels, I see. That's too complex. I feel
strongly that if we're going for multiple levels, it should be no more
than 2, at least for now. And even for the more inclusive one, I don't
want to build a kitchen-sink specification that includes a big list of
packages. That's for distributions to define, not the standard. As I
mentioned before, I think we need simple descriptions of 'why you want
version X'.

> Level 1: Stuff needed for beginner tutorials, Numerics 101 classes, etc.

What 'beginners' need isn't necessarily the base of the stack, though.
In fact, more often than not, beginners use high-level packages, and
you learn about the stuff lower down the stack as you move on. And the
needs of 'beginners' might be very different in different fields. In
biology, we begin with statistical tests, not array manipulation.

Also, hard drives are large and University internet connections are
fast - why shouldn't a beginner just download the more inclusive
'recommended' level, and leave some parts unused?

I'm focussing on the user here. I don't want to define multiple levels
just to push the tricky decisions about inclusion downstream to
distributions and users. If the standard is to have multiple flavours,
we need a clear story about how it benefits users.

> Finally, I think that there should be installers.

Sorry, this is not part of my plan, at least for now. Maintaining a
distribution for several different OSs is a much bigger task, and it's
not one that I want to take on. I'm optimistic that existing
distributions will meet that need.

Also, I intend the name Pylab to mean a particular set of packages,
however you've installed them. If we make a Pylab distribution, it
gives the impression that EPD, for example, is not Pylab. Then we'd be
back in the current situation, but with one more Python distribution.

Thanks,
Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

ralfgommers
In reply to this post by Thomas Kluyver-2


On Wed, Sep 19, 2012 at 12:26 AM, Thomas Kluyver <[hidden email]> wrote:
Hi again,

It now looks like we're going to use Pylab as the name for the 'scipy
stack'. Now I want to turn to the question of what it should include.
The idea is that Python distributions can call themselves Pylab
compliant if they provide at least a defined set of packages. Also, I
hope that it will become a metapackage in Linux distributions, so that
users can 'apt-get install pylab' or similar

As a minimum, I assume we should require Python, Numpy, Scipy and
Matplotlib. Does anyone disagree?

I also think we should specify minimum versions. The standard itself
will be versioned, so we can raise these over time. For Python, I
intend the requirement to be 2.x >= 2.6 or 3.x >= 3.2. What are
sensible minimum versions of Numpy, Scipy and Matplotlib?

I'd propose numpy >= 1.5.1, scipy >= 0.10.0, matplotlib >= 1.1. EPD, Python(x,y) and AnacondaCE all fulfill those requirements. Sage likely does too, but they don't list versions on their website.

Ralf
 

Should the standard include an interface? IPython, a more traditional
IDE, or both? On the one hand, specifying a standard interface means
users can share experience better, and exchange richer files, like
IPython notebooks or IDE project structures. Matlab, for instance,
wins praise for including a powerful IDE by default. On the other
hand, we've got several interesting UI efforts still taking shape -
IPython notebooks, Spyder, IEP - and declaring one standard would make
the alternatives less visible. I'm honestly torn on this - I can see
good arguments for and against.

Other scientific packages we might consider include pandas (which
provides functionality similar to core parts of R), Sympy, Cython,
various scikits projects, h5py, and doubtless many others I haven't
thought of. We could also specify general purpose Python packages such
as requests, or a GUI toolkit.

On the NumFOCUS list, Chris Kees raised the idea that there could be
two or more levels of packages, e.g. 'core' and 'recommended'. I don't
think we should add that kind of complexity in the first version, but
keep in mind that we could differentiate it later.

Finally, I mean the standard to specify that the distribution must
offer a way of installing arbitrary extra Python packages into it, so
the standard shouldn't try to include everything you might need for
scientific computing. The aim is to offer a key set of tools so you
can get started without having to add things. "Get started with what?"
is the key question we have to answer. In my field, for example,
statistical tests are fundamental, while symbolic maths is hardly
used.

All your opinions are welcome,

Thomas
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Pylab - standard packages

ralfgommers
In reply to this post by Thomas Kluyver-2


On Wed, Sep 19, 2012 at 1:03 AM, Thomas Kluyver <[hidden email]> wrote:
On 18 September 2012 23:44, Matthew Brett <[hidden email]> wrote:

> Is there a good reason not to include Ipython?

Perhaps other interfaces work better for some use cases, or some
individuals? Python(x,y) favours Spyder, for instance, and ships an
old version of IPython to maintain compatibility. If we included
IPython in the standard, Python(x,y) would probably not meet the
minimum specified version, and would not be Pylab compliant for the
time being.

The part of IPython that's pretty much universal is the command line interface, so I don't see a problem with setting the minimum version to 0.10.

Ralf


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
1234 ... 8