Co-ordinating Python astronomy libraries?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Co-ordinating Python astronomy libraries?

James Turner-4
Dear Python users in astronomy,

At SciPy 2009, I arranged an astronomy BoF where we discussed the
fact that there are now a number of astronomy libraries for Python
floating around and maybe it would be good to collect more code into
a single place. People seemed receptive to this idea and weren't sure
why it hasn't already happened, given that there has been an Astrolib
page at SciPy for some years now, with an associated SVN repository:

   http://scipy.org/AstroLib

After the meeting last August, I was supposed to contact the mailing
list and some library authors I had talked to previously, to discuss
this further. My apologies for taking 10 months to do that! I did
draft an email the day after the BoF, but then we ran into a hurdle
with setting up new committers to the AstroLib repository (which has
taken a lot longer than expected to resolve), so it seemed a bad
time to suggest that new people start using it.

To discuss these issues further, we'd like to encourage everyone to
sign up for the AstroPy mailing list if you are not already on it.
The traffic is just a few messages per month.

   http://lists.astropy.scipy.org/mailman/listinfo/astropy

We (the 2009 BoF group) would also like to hear on the list about
why people have decided to host their own astronomy library (eg. not
being aware of the one at SciPy). Are you interested in contributing
to Astrolib? Do you have any other comments or concerns about
co-ordinating tools? Our motivation is to make libraries easy to
find and install, allow sharing code easily, help rationalize
available functionality and fill in what's missing. A standard
astronomy library with a single set of documentation should be more
coherent and easier to maintain. The idea is not to limit authors'
flexibility of take ownership of their code -- the sub-packages
can still be maintained by different people.

If you're at SciPy this week, Perry Greenfield and I would be happy
to talk to you. If you would like to add your existing library to
Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at
STScI for access (contact details at http://scipy.org/AstroLib).
Note that the repository is being moved to a new server this week,
after which the URLs will be updated at scipy.org.

Thanks!

James Turner (Gemini).

Bcc: various library authors

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

Erin Sheldon
Dear James and AstroPy -

Thanks for your note, and prompting!

My collegues and I have been writing data analysis related tools
for a some time now.  We are all astronomers, and in addition to general
analysis tools we have a growing library of astro utilities.  I'd to
make others aware of these, both because they may be useful and because
more eyes will find more bugs.  We would also welcome collaborators.
So far we have hosted our own site because the tools are often so
general rather than astronomy specific, but if there is interest
we could migrate or mirror some of these to the astrolib svn archive.

As the tools mature we have been putting them at this google sites
repository:

  http://code.google.com/p/esutil/

Example astronomy codes currently are WCS utilities (wcsutil),
cosmology calculations (cosmology), coordinate transformations (coords),
and heirarchical triangular mesh sky search tools (htm).

Of more general interest may be the numpy_util, stat, random, ostools,
io, and integrate sub-packages. In addition to new things, there are a
lot of routines derived from IDL, the Goddard IDL astronomy libraries
and the SDSSIDL and IDLUTILS packages.  In particular, the structure
routines in those IDL packages have correspondence to recarray
routines in our packages.

For those writing C/C++ extensions, the include/NumpyVector.h template
class designed to simplify working with 1-d numpy arrays.   There is
also a NumpyVoidVector for arrays whose type is determined at runtime.

The recfile package is incorporated into esutil
(http://code.google.com/p/recfile/) and is used for efficient io of rec
files (recfile and sfile sub-packages).


I would say that a primary focus of the Astro tools is on using
numerical python arrays, especially recarrays.  For example, the
coordinate transformation and WCS codes take arrays and return arrays.
For example:

  l,b = coords.eq2gal(ra,dec)

  wcs=wcsutil.WCS(fits_header)
  ra,dec = wcs.image2sky(x,y)

where everything here can be an array.

This is opposed to the other libraries out that work with coordinates as
objects.  There are clearly tradeoffs;  we generally read data from FITS
tables or databases as numerical python arrays and so it is more natural
to work with data that way.  I would say this is complimentary to the
other approach.

In addition to above sub-packages, a few more are very close to
ready:

  * pgnumpy: a numerical python interface to postgres
  * numpydb:a numerical python interface to berkeley db.
  * columns: A simple, efficient column-oriented, pythonic database
    with indexing provided by numpydb.
  * sdsspy: Tools for working with SDSS data.
  * mangle: tools for working with mangle masks.


I hope people find these useful,

Erin Scott Sheldon

Cosmology Group
Brookhaven National Laboratory

on behalf of Brian Gerke and Amy Kimball



On Wed, Jun 30, 2010 at 5:48 PM, James Turner <[hidden email]> wrote:

> Dear Python users in astronomy,
>
> At SciPy 2009, I arranged an astronomy BoF where we discussed the
> fact that there are now a number of astronomy libraries for Python
> floating around and maybe it would be good to collect more code into
> a single place. People seemed receptive to this idea and weren't sure
> why it hasn't already happened, given that there has been an Astrolib
> page at SciPy for some years now, with an associated SVN repository:
>
>   http://scipy.org/AstroLib
>
> After the meeting last August, I was supposed to contact the mailing
> list and some library authors I had talked to previously, to discuss
> this further. My apologies for taking 10 months to do that! I did
> draft an email the day after the BoF, but then we ran into a hurdle
> with setting up new committers to the AstroLib repository (which has
> taken a lot longer than expected to resolve), so it seemed a bad
> time to suggest that new people start using it.
>
> To discuss these issues further, we'd like to encourage everyone to
> sign up for the AstroPy mailing list if you are not already on it.
> The traffic is just a few messages per month.
>
>   http://lists.astropy.scipy.org/mailman/listinfo/astropy
>
> We (the 2009 BoF group) would also like to hear on the list about
> why people have decided to host their own astronomy library (eg. not
> being aware of the one at SciPy). Are you interested in contributing
> to Astrolib? Do you have any other comments or concerns about
> co-ordinating tools? Our motivation is to make libraries easy to
> find and install, allow sharing code easily, help rationalize
> available functionality and fill in what's missing. A standard
> astronomy library with a single set of documentation should be more
> coherent and easier to maintain. The idea is not to limit authors'
> flexibility of take ownership of their code -- the sub-packages
> can still be maintained by different people.
>
> If you're at SciPy this week, Perry Greenfield and I would be happy
> to talk to you. If you would like to add your existing library to
> Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at
> STScI for access (contact details at http://scipy.org/AstroLib).
> Note that the repository is being moved to a new server this week,
> after which the URLs will be updated at scipy.org.
>
> Thanks!
>
> James Turner (Gemini).
>
> Bcc: various library authors
>
> _______________________________________________
> AstroPy mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/astropy
>
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Co-ordinating Python astronomy libraries?

Mubdi Rahman
In reply to this post by James Turner-4
Hi James and AstroPy colleagues,

Thanks for the note - coordinating the astronomical python community has been a long time coming, and I'm glad that someone's taken the initiative to start this line of communication.

Last year, a number of us located here in Toronto started pyAstroLib, with the goal of converting the NASA IDL Astronomy Library into Python. We roadmapped the course of doing an initial conversion of the entire library, with the exception of scripts that already have python equivalents (i.e. those provided by pyFits, et cetera). In the case of existing functionality, the documentation of pyAstroLib would point users in the direction of the tools that do the equivalent.

We further roadmapped a plan to go beyond the structure of the IDL AstroLib (we were calling the direct python conversion of these libraries the "legacy" library), and developing a more structured, streamlined and less redundant general purpose astronomy library, so as to make entry into the world of astronomical python as painless as possible. Beyond this stage, we had a number of dream-goals, including a lightweight python-based replacement to DS9 with a more intuitive UI (think Adobe Photoshop), which could be used as a standalone package or embedded within other python scripts.  

We had a number of design requirements for the library:
- The library was to be fully open-source (we chose the LGPL as our model).
- The library was to require as few external libraries as necessary (we've limited ourselves to NumPy, SciPy, PyFits, and MatPlotLib).
- The library was intended to be seamlessly cross-platform. Personally, I'm a Linux/Windows user, where as some of the others in our collaboration are pure Mac or pure Linux users.

We realized that at some point, the project would merge with a number of others out there, and our goal was to have a bit of a code base before we started expanding and merging.  Needless to say, we were ambitious.

That was a year ago.

In that time, we had started the conversion process - a large number of the general IDL astronomical functions have been converted (as one can pull from our git repo on sourceforge), complete with docstrings, but without extensive testing. But as is always the case, each of us individually became more concerned about the functionality that we needed for our research. For instance, much of my reliance on IDL has been due to the coordinate conversion functions - so I had just ported that part of wcslib into python. (The other half of my reliance on IDL has come from the pixel to wcs coordinate transformations and the image manipulation and alignment scripts, which I'm currently working on).

We've also been receiving a lot of contributed code for a variety of tools and functions that other users have needed. Up till now, we haven't had a place to put this code. Now considering the path that this project has taken, we're moving in a different direction, which is to round up the contributed code, even if it doesn't actually fit in the IDL AstroLib framework, give it a common API, and just get people using it as is for now, and slowly categorize and refine the code in subsequent releases.

That's basically where we are with pyAstroLib. I hope this clears up some of the confusion and helps coordinating effort.

Mubdi Rahman

on behalf of the pyAstroLib crew.


----- Original Message ----
> From: James Turner <[hidden email]>
> To: AstroPy <[hidden email]>; SciPy Users List <[hidden email]>
> Sent: Wed, June 30, 2010 5:48:23 PM
> Subject: Co-ordinating Python astronomy libraries?
>
> Dear Python users in astronomy,

At SciPy 2009, I arranged an astronomy
> BoF where we discussed the
fact that there are now a number of astronomy
> libraries for Python
floating around and maybe it would be good to collect
> more code into
a single place. People seemed receptive to this idea and
> weren't sure
why it hasn't already happened, given that there has been an
> Astrolib
page at SciPy for some years now, with an associated SVN
> repository:

 
> >http://scipy.org/AstroLib

After the meeting last August, I was
> supposed to contact the mailing
list and some library authors I had talked to
> previously, to discuss
this further. My apologies for taking 10 months to do
> that! I did
draft an email the day after the BoF, but then we ran into a
> hurdle
with setting up new committers to the AstroLib repository (which
> has
taken a lot longer than expected to resolve), so it seemed a bad
time
> to suggest that new people start using it.

To discuss these issues
> further, we'd like to encourage everyone to
sign up for the AstroPy mailing
> list if you are not already on it.
The traffic is just a few messages per
> month.

 
> href="http://lists.astropy.scipy.org/mailman/listinfo/astropy" target=_blank
> >http://lists.astropy.scipy.org/mailman/listinfo/astropy

We (the 2009
> BoF group) would also like to hear on the list about
why people have decided
> to host their own astronomy library (eg. not
being aware of the one at
> SciPy). Are you interested in contributing
to Astrolib? Do you have any other
> comments or concerns about
co-ordinating tools? Our motivation is to make
> libraries easy to
find and install, allow sharing code easily, help
> rationalize
available functionality and fill in what's missing. A
> standard
astronomy library with a single set of documentation should be
> more
coherent and easier to maintain. The idea is not to limit
> authors'
flexibility of take ownership of their code -- the
> sub-packages
can still be maintained by different people.

If you're at
> SciPy this week, Perry Greenfield and I would be happy
to talk to you. If you
> would like to add your existing library to
Astrolib, please contact Perry
> Greenfield or Mark Sienkiewicz at
STScI for access (contact details at
> href="http://scipy.org/AstroLib" target=_blank
> >http://scipy.org/AstroLib).
Note that the repository is being moved to a
> new server this week,
after which the URLs will be updated at
> scipy.org.

Thanks!

James Turner (Gemini).

Bcc: various
> library authors
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Co-ordinating Python astronomy libraries?

Erik Tollerud-2
In reply to this post by James Turner-4
Dear AstroPy list and James,

Thanks for your efforts to do this coordination - clearly there's
quite a bit out there doing similar work.

For the last couple years I (and a few others) have been working on
Astropysics (http://packages.python.org/Astropysics/ and source code
at https://launchpad.net/astropysics).  This initially spun out of my
personal need for some of the utilities these other projects discussed
(e.g. IDL astron-like functions), but I've been turning it in a
slightly different direction.  Astropysics now is being written as a
more "pythonic" way of doing the same tasks instead of starting from
cloning IDL procedural techniques.  I'm trying to leverage all the
existing tools that have been written in python for other domains
(e.g. Enthought Traits to easily make good GUIs, sphinx to make useful
documentation, distribute to make packaging easier).  Further, where
useful, I use object-oriented techniques instead of the procedural
approach people familiar with IDL and IRAF are used to.  The idea is
to start doing things on "Spectrum" and "CCDImage" objects instead of
passing arrays into functions and so on, and each astronomer can then
sub-class these classes to do whatever they want.  The aim here is to
eliminate the tendancy for people to take public codes and change them
internally, leading to some very painful efforts in disentangling
versions.

So far, Astropysics includes the following modules:
* constants – physical constants and cosmological calculations
* coords – coordinate classes and coordinate conversions
* models – model and data-fitting classes and tools
* objcat – flexible, dynamically updated object catalog
* ccd – image processing tools
* spec – spectrum and SED classes and tools
* phot – photometry/flux measurement classes and tools
* obstools – miscellaneous tools for observation
* io – data input/output classes and tools
* plotting – astronomy-oriented matplotlib and mayavi plotting tools
* pipeline – classes for data reduction and analysis pipelines
* utils – utility classes and functions

And these GUI tools, which require Enthought Traits (and Enthought
chaco for plots in the GUI):
* Spylot – Spectrum Plotter
* Fitgui – Interactive Curve Fitting
* Spectarget – MOS Targeting

I'm making a concerted effort to document everything in a consistent
manner using sphinx (http://sphinx.pocoo.org) - the resulting
documents (http://packages.python.org/Astropysics/) end up being much
more useful.

I also try to bind in python wrappers around external tools like
sextractor, Mike Blanton's kcorrect, MOS mask design tools, and
galfit (planned) ... this happens mostly as I need them for my own
science.  But this cuts down on the time re-implementing standard
programs that aren't necessarily worth re-working.

There are two main things left before I consider it "releasable" in
terms of the baseline functionality: First, it needs WCS support to be
integrated in the coords framework, and second, the objcat package
should have a web server module that lets you show/edit the catalog
over the internet instead of within python.  Another major goal I'd
like to do when I or someone else gets a chance is something like ATV
(http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI
framework that the other gui tools are written in.

I hope this explains the direction I have in mind for astropysics.  As
far as I can tell, I have a slightly different philosophy - I'm trying
to set up something of a framework of my design, rather than a
function library.  This is why I have not been working on adding it to
astropy, because astropy seems much more like the traditional
library... That and I'm not a fan of the Trac project management
system.  At any rate, I think there's definitely room for all in the
community.  And if anyone likes what they see, feel free to drop me a
line if you want to contribute.

--
Erik Tollerud
Graduate Student
Center For Cosmology
University of California, Irvine
http://ps.uci.edu/~etolleru



On Wed, Jun 30, 2010 at 2:48 PM, James Turner <[hidden email]> wrote:

> Dear Python users in astronomy,
>
> At SciPy 2009, I arranged an astronomy BoF where we discussed the
> fact that there are now a number of astronomy libraries for Python
> floating around and maybe it would be good to collect more code into
> a single place. People seemed receptive to this idea and weren't sure
> why it hasn't already happened, given that there has been an Astrolib
> page at SciPy for some years now, with an associated SVN repository:
>
>   http://scipy.org/AstroLib
>
> After the meeting last August, I was supposed to contact the mailing
> list and some library authors I had talked to previously, to discuss
> this further. My apologies for taking 10 months to do that! I did
> draft an email the day after the BoF, but then we ran into a hurdle
> with setting up new committers to the AstroLib repository (which has
> taken a lot longer than expected to resolve), so it seemed a bad
> time to suggest that new people start using it.
>
> To discuss these issues further, we'd like to encourage everyone to
> sign up for the AstroPy mailing list if you are not already on it.
> The traffic is just a few messages per month.
>
>   http://lists.astropy.scipy.org/mailman/listinfo/astropy
>
> We (the 2009 BoF group) would also like to hear on the list about
> why people have decided to host their own astronomy library (eg. not
> being aware of the one at SciPy). Are you interested in contributing
> to Astrolib? Do you have any other comments or concerns about
> co-ordinating tools? Our motivation is to make libraries easy to
> find and install, allow sharing code easily, help rationalize
> available functionality and fill in what's missing. A standard
> astronomy library with a single set of documentation should be more
> coherent and easier to maintain. The idea is not to limit authors'
> flexibility of take ownership of their code -- the sub-packages
> can still be maintained by different people.
>
> If you're at SciPy this week, Perry Greenfield and I would be happy
> to talk to you. If you would like to add your existing library to
> Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at
> STScI for access (contact details at http://scipy.org/AstroLib).
> Note that the repository is being moved to a new server this week,
> after which the URLs will be updated at scipy.org.
>
> Thanks!
>
> James Turner (Gemini).
>
> Bcc: various library authors
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

jct72
Hi Erik, this is completely independent from the STS software, that
shares some modules' name with you :
https://www.stsci.edu/trac/ssb/astrolib  ?
Johann

On 07/04/2010 01:14 AM, Erik Tollerud wrote:

> Dear AstroPy list and James,
>
> Thanks for your efforts to do this coordination - clearly there's
> quite a bit out there doing similar work.
>
> For the last couple years I (and a few others) have been working on
> Astropysics (http://packages.python.org/Astropysics/ and source code
> at https://launchpad.net/astropysics).  This initially spun out of my
> personal need for some of the utilities these other projects discussed
> (e.g. IDL astron-like functions), but I've been turning it in a
> slightly different direction.  Astropysics now is being written as a
> more "pythonic" way of doing the same tasks instead of starting from
> cloning IDL procedural techniques.  I'm trying to leverage all the
> existing tools that have been written in python for other domains
> (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful
> documentation, distribute to make packaging easier).  Further, where
> useful, I use object-oriented techniques instead of the procedural
> approach people familiar with IDL and IRAF are used to.  The idea is
> to start doing things on "Spectrum" and "CCDImage" objects instead of
> passing arrays into functions and so on, and each astronomer can then
> sub-class these classes to do whatever they want.  The aim here is to
> eliminate the tendancy for people to take public codes and change them
> internally, leading to some very painful efforts in disentangling
> versions.
>
> So far, Astropysics includes the following modules:
> * constants – physical constants and cosmological calculations
> * coords – coordinate classes and coordinate conversions
> * models – model and data-fitting classes and tools
> * objcat – flexible, dynamically updated object catalog
> * ccd – image processing tools
> * spec – spectrum and SED classes and tools
> * phot – photometry/flux measurement classes and tools
> * obstools – miscellaneous tools for observation
> * io – data input/output classes and tools
> * plotting – astronomy-oriented matplotlib and mayavi plotting tools
> * pipeline – classes for data reduction and analysis pipelines
> * utils – utility classes and functions
>
> And these GUI tools, which require Enthought Traits (and Enthought
> chaco for plots in the GUI):
> * Spylot – Spectrum Plotter
> * Fitgui – Interactive Curve Fitting
> * Spectarget – MOS Targeting
>
> I'm making a concerted effort to document everything in a consistent
> manner using sphinx (http://sphinx.pocoo.org) - the resulting
> documents (http://packages.python.org/Astropysics/) end up being much
> more useful.
>
> I also try to bind in python wrappers around external tools like
> sextractor, Mike Blanton's kcorrect, MOS mask design tools, and
> galfit (planned) ... this happens mostly as I need them for my own
> science.  But this cuts down on the time re-implementing standard
> programs that aren't necessarily worth re-working.
>
> There are two main things left before I consider it "releasable" in
> terms of the baseline functionality: First, it needs WCS support to be
> integrated in the coords framework, and second, the objcat package
> should have a web server module that lets you show/edit the catalog
> over the internet instead of within python.  Another major goal I'd
> like to do when I or someone else gets a chance is something like ATV
> (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI
> framework that the other gui tools are written in.
>
> I hope this explains the direction I have in mind for astropysics.  As
> far as I can tell, I have a slightly different philosophy - I'm trying
> to set up something of a framework of my design, rather than a
> function library.  This is why I have not been working on adding it to
> astropy, because astropy seems much more like the traditional
> library... That and I'm not a fan of the Trac project management
> system.  At any rate, I think there's definitely room for all in the
> community.  And if anyone likes what they see, feel free to drop me a
> line if you want to contribute.
>
>    
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

Thomas Robitaille
In reply to this post by James Turner-4
Hi all,

Unlike several people who have replied so far, I have not been involved in general purpose astronomy libraries, but rather a few small packages, including:

- APLpy (FITS image plotting built on matplotlib) at http://aplpy.sourceforge.net (co-developed with Eli Bressert)
- ATpy (Seamless multi-format table handler) at http://atpy.sourceforge.net (also co-developed with Eli Bressert)
- IDLSave (To read IDL save files into python) at http://idlsave.sourceforge.net
- python-montage (Montage wrapper) at http://python-montage.sourceforgenet

The main reason for keeping these separate rather than group them into a single package was that each of these packages accomplishes a well-defined task, and we were not prepared to develop all the other tools needed in a general-purpose library. However, I think that from a user point of view, it would be nice to have something more unified.

The main point I want to make is that we need to distinguish between merging all development into a single repository, and bundling packages. There are cases where merging the development of packages does not make sense. For example, in the case of IDLSave, the module was originally developed for the astronomy community, but in fact can be used by any scientist that uses IDL. By developing it as part of a general astronomy libraries makes it less likely that non-astronomers will find it and use it. Another example is Tom Aldcroft's asciitable module (http://cxc.harvard.edu/contrib/asciitable/). This was developed to read in ASCII tables, but was not actually designed in an astronomy specific ways, since ASCII tables are of course not limited to astronomy. In the case of such a package, developing it as part of a general astronomy library would be detrimental, but bundling it as part of a general package might be desirable.

So here is what I personally see as the ideal (hierarchical) setup:

1. A core library of essential routines, which handle for example FITS I/O, VO tools, WCS, coordinate transformations, etc. These could be held on a common development server. This would be a kind of 'numpy' or 'scipy' for astronomy, e.g. 'astropy' or 'astrocore'. Documentation for these would be merged and unified.

2. An astronomy 'bundle' which would include these essential routines, as well as extra astronomy packages (e.g. asciitable, IDLSave, ATpy, etc.). This bundle could be installable on top of the system python, or the enthought python distribution, e.g. 'astrolab'. Documentation for the extra packages would be left to the developers, but could be required to be of a consistent style (e.g. Sphinx) for inclusion in the bundle.

3. An 'all-in-one' package that would also include a full Python distribution, with numpy, scipy, matplotlib, etc - essentially an alternative to EPD for astronomy, e.g. 'APD (Astronomy Python Distribution)'. It could even come with a custom ipython prompt with all packges pre-loaded. All dependencies would be included.

This would then cater to the different levels of python users in Astronomy. One quick note is that if we follow this or a similar model of bundling some packages without merging development repositories, is that developers of small packages will need to be more careful what license they release their code under, to ensure they can be bundled and re-released (but this should be trivial).

Cheers,

Thomas Robitaille

On Jun 30, 2010, at 5:48 PM, James Turner wrote:

Dear Python users in astronomy,

At SciPy 2009, I arranged an astronomy BoF where we discussed the
fact that there are now a number of astronomy libraries for Python
floating around and maybe it would be good to collect more code into
a single place. People seemed receptive to this idea and weren't sure
why it hasn't already happened, given that there has been an Astrolib
page at SciPy for some years now, with an associated SVN repository:

  http://scipy.org/AstroLib

After the meeting last August, I was supposed to contact the mailing
list and some library authors I had talked to previously, to discuss
this further. My apologies for taking 10 months to do that! I did
draft an email the day after the BoF, but then we ran into a hurdle
with setting up new committers to the AstroLib repository (which has
taken a lot longer than expected to resolve), so it seemed a bad
time to suggest that new people start using it.

To discuss these issues further, we'd like to encourage everyone to
sign up for the AstroPy mailing list if you are not already on it.
The traffic is just a few messages per month.

  http://lists.astropy.scipy.org/mailman/listinfo/astropy

We (the 2009 BoF group) would also like to hear on the list about
why people have decided to host their own astronomy library (eg. not
being aware of the one at SciPy). Are you interested in contributing
to Astrolib? Do you have any other comments or concerns about
co-ordinating tools? Our motivation is to make libraries easy to
find and install, allow sharing code easily, help rationalize
available functionality and fill in what's missing. A standard
astronomy library with a single set of documentation should be more
coherent and easier to maintain. The idea is not to limit authors'
flexibility of take ownership of their code -- the sub-packages
can still be maintained by different people.

If you're at SciPy this week, Perry Greenfield and I would be happy
to talk to you. If you would like to add your existing library to
Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at
STScI for access (contact details at http://scipy.org/AstroLib).
Note that the repository is being moved to a new server this week,
after which the URLs will be updated at scipy.org.

Thanks!

James Turner (Gemini).

Bcc: various library authors

_______________________________________________
AstroPy mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/astropy


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

Perry Greenfield

On Jul 6, 2010, at 2:36 PM, Thomas Robitaille wrote:

> Hi all,
>
> Unlike several people who have replied so far, I have not been  
> involved in general purpose astronomy libraries, but rather a few  
> small packages, including:
>
> - APLpy (FITS image plotting built on matplotlib) at http://aplpy.sourceforge.net 
>  (co-developed with Eli Bressert)
> - ATpy (Seamless multi-format table handler) at http://atpy.sourceforge.net 
>  (also co-developed with Eli Bressert)
> - IDLSave (To read IDL save files into python) at http://idlsave.sourceforge.net
> - python-montage (Montage wrapper) at http://python-montage.sourceforgenet
>
> The main reason for keeping these separate rather than group them  
> into a single package was that each of these packages accomplishes a  
> well-defined task, and we were not prepared to develop all the other  
> tools needed in a general-purpose library. However, I think that  
> from a user point of view, it would be nice to have something more  
> unified.
>
> The main point I want to make is that we need to distinguish between  
> merging all development into a single repository, and bundling  
> packages. There are cases where merging the development of packages  
> does not make sense. For example, in the case of IDLSave, the module  
> was originally developed for the astronomy community, but in fact  
> can be used by any scientist that uses IDL. By developing it as part  
> of a general astronomy libraries makes it less likely that non-
> astronomers will find it and use it. Another example is Tom  
> Aldcroft's asciitable module (http://cxc.harvard.edu/contrib/asciitable/ 
> ). This was developed to read in ASCII tables, but was not actually  
> designed in an astronomy specific ways, since ASCII tables are of  
> course not limited to astronomy. In the case of such a package,  
> developing it as part of a general astronomy library would be  
> detrimental, but bundling it as part of a general package might be  
> desirable.
>
In principle yes, some of these things are generic. But having  
libraries in a common repository doesn't preclude distributing them  
separately from astronomy.

I also tend to thing that premature fragmentation is as bad a problem  
as being way too monolithic. I'd tend to wait until it was clear we  
had too much stuff in one repository before worrying about how to  
split things up. Sometimes it never becomes an issue. If some items  
grow non-astronomical contributors, they can go to scipy (or something  
similar).

There are some advantages to a common repository:

1) We become more aware of areas of commonality and that foster  
greater sense of making things work better together.
2) Eventually it would help make for more consistent documentation and  
style, and ways of integrating the documentation.
3) Making the process of integrating common or redundant functionality  
easier later (I'll say a little about that in a separate email)

And there are downsides of course:

1) Not everyone wants to use the same version control software (svn,  
git, hg...) or wiki, or issue tracker.
2) Coordination with other developers and projects is more work than  
doing things on your own.

[...]

> 3. An 'all-in-one' package that would also include a full Python  
> distribution, with numpy, scipy, matplotlib, etc - essentially an  
> alternative to EPD for astronomy, e.g. 'APD (Astronomy Python  
> Distribution)'. It could even come with a custom ipython prompt with  
> all packges pre-loaded. All dependencies would be included.
>
In this area STScI and Gemini are attempting to do something like this  
now as a joint project. We are just starting on this effort, but we  
hope to  make a fairly easy to install distribution that includes most  
core tools astronomers would need (or something that would be easy to  
install optional items on). But we don't want to talk too much about  
this until we have something to have people try (I  think Gemini would  
like to have something basic by the end of the year). We intend to  
include IRAF and the common external IRAF packages as well.

> This would then cater to the different levels of python users in  
> Astronomy. One quick note is that if we follow this or a similar  
> model of bundling some packages without merging development  
> repositories, is that developers of small packages will need to be  
> more careful what license they release their code under, to ensure  
> they can be bundled and re-released (but this should be trivial).
>
Yes, licensing is one of the important issues to deal with...

Perry

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

Thomas Robitaille
In reply to this post by James Turner-4
Hi all,

After reading all the replies, I have the following suggestion to make.

The model scipy follows is to have a 'core' scipy package, and 'scikit' packages which share a common namespace, and which are meant as addons to the scipy package, but have not yet (or might never) make it to the main scipy package:

http://www.scipy.org/scipy/scikits/

"Scipy Toolkits are independent and seperately installable projects hosted under a common namespace. Packages that are distributed in this way are here (instead of in monolithic scipy) for at least one of three general reasons. Each of these reasons use the same high-level namespace (scikits)."

I think we can use this model, and that the following approach can be used here, namely:

- start with a very basic 'astropy' package, with e.g. support for FITS/WCS
- agree to coordinate astronomy packages with a common namespace (e.g. 'astrokit', so for example, APLpy would become astrokit.aplpy). This can help us manage the namespace (as suggested in Joe Harrington's email)
- as astrokit modules mature, they can (if the authors are willing) be merged into the main 'astropy' package, once they have met a number of conditions, including e.g. unit tests, sphinx documentation, limited dependencies (e.g. numpy/scipy/matplotlib, and any package in the 'astropy' package), and compatible license.

The advantage of this model is that this encourages the growth from the bottom up of a core astronomy package, which is manageable, as well as the independent development of other packages. It also means that the core package will be quite stable, because it will only accrete 'astrokit' modules as they become stable and mature. At the same time, it encourages developers to make their own innovative astrokit, but without commitment from the maintainers of the core package to accrete it in future.

In passing, this also leaves the possibility for those who want to develop meta-pacakges of astrokit modules. Also, it will make it easier for those who want to build fully fledged astronomy python distributions.

There have been many ideas passed around in the mailing list, but I think that some are maybe too ambitious. I do think that the idea suggested above is much more manageable. The concrete steps would be:

- setup a central repository for the core packages, as well as astrokit pacakges (although we should leave the option for people to develop astrokit packages outside this repository too, and rely on trust and communication to avoid namespace clashes)
- start a core package with e.g. FITS and WCS support (e.g. pyfits + pywcs)
- set up a list of 'registered' astrokit names to avoid conflict.
- set up a list of recommendations and requirements for astrokit pacakges
- encourage developers of existing packages to use this namespace and follow the guidelines. This would be very little work in some cases, which is why I think it could work.

Sharing a namespace is (I think) the first step to showing that we are willing to work on something coherent, and users would only see two top-level namespaces - astropy and astrokit, which would be a great improvement over the current situation where it is not immediately obvious which packages are astronomy related.

Comments/suggestions/criticism welcome!

Cheers,

Tom

On Jun 30, 2010, at 5:48 PM, James Turner wrote:

> Dear Python users in astronomy,
>
> At SciPy 2009, I arranged an astronomy BoF where we discussed the
> fact that there are now a number of astronomy libraries for Python
> floating around and maybe it would be good to collect more code into
> a single place. People seemed receptive to this idea and weren't sure
> why it hasn't already happened, given that there has been an Astrolib
> page at SciPy for some years now, with an associated SVN repository:
>
>   http://scipy.org/AstroLib
>
> After the meeting last August, I was supposed to contact the mailing
> list and some library authors I had talked to previously, to discuss
> this further. My apologies for taking 10 months to do that! I did
> draft an email the day after the BoF, but then we ran into a hurdle
> with setting up new committers to the AstroLib repository (which has
> taken a lot longer than expected to resolve), so it seemed a bad
> time to suggest that new people start using it.
>
> To discuss these issues further, we'd like to encourage everyone to
> sign up for the AstroPy mailing list if you are not already on it.
> The traffic is just a few messages per month.
>
>   http://lists.astropy.scipy.org/mailman/listinfo/astropy
>
> We (the 2009 BoF group) would also like to hear on the list about
> why people have decided to host their own astronomy library (eg. not
> being aware of the one at SciPy). Are you interested in contributing
> to Astrolib? Do you have any other comments or concerns about
> co-ordinating tools? Our motivation is to make libraries easy to
> find and install, allow sharing code easily, help rationalize
> available functionality and fill in what's missing. A standard
> astronomy library with a single set of documentation should be more
> coherent and easier to maintain. The idea is not to limit authors'
> flexibility of take ownership of their code -- the sub-packages
> can still be maintained by different people.
>
> If you're at SciPy this week, Perry Greenfield and I would be happy
> to talk to you. If you would like to add your existing library to
> Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at
> STScI for access (contact details at http://scipy.org/AstroLib).
> Note that the repository is being moved to a new server this week,
> after which the URLs will be updated at scipy.org.
>
> Thanks!
>
> James Turner (Gemini).
>
> Bcc: various library authors
>
> _______________________________________________
> AstroPy mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/astropy

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

James Turner-4
Hi Thomas,

I think this seems a good idea, even if it only solves part of the
problem. Astrokits (or just scikits?) could be a good place for code
to go whilst it's maturing, or for other reasons, with a low barrier
to entry. We could encourage authors to follow the same standards/
guidelines as the core library, without enforcing them.

A couple of limitations spring to mind. First, a proliferation of
astrokits with overlapping functionality could perpetuate duplication
and incoherence. As Perry said, there's no harm in doing something a
couple of ways to see which is best, but if there are 5 different
wrappers for WCSLIB, that's just confusing and makes it hard to focus
on quality. WCS is not the best example, since you proposed putting
that in the core, but in general this seems like something to keep in
mind and it isn't solved just by associating different libraries.

My second immediate concern would be distribution -- installing lots
of astrokits could be a bigger problem for end users than a single
library. Of course that could be solved by including most of the
astrokits in our Gemini/STScI Python distribution once it's available,
but these are early days and I don't think we're ready to promise the
whole astronomy/science community a solution to its installation
problems. Maybe others (eg. scisoft) would also pick them up though.

Nevertheless, I think astrokits can only be better than lots of
totally uncoordinated libraries and they could be a good path to
accepting code into the core and/or encouraging contributions that
might not happen otherwise. I think I'd back this proposal, unless
others have objections I haven't considered yet -- but I'm not
speaking for Perry's group, which ultimately manages Astrolib.

Cheers,

James.


> Hi all,
>
> After reading all the replies, I have the following suggestion to make.
>
> The model scipy follows is to have a 'core' scipy package, and 'scikit' packages which share a common namespace, and which are meant as addons to the scipy package, but have not yet (or might never) make it to the main scipy package:
>
> http://www.scipy.org/scipy/scikits/
>
> "Scipy Toolkits are independent and seperately installable projects hosted under a common namespace. Packages that are distributed in this way are here (instead of in monolithic scipy) for at least one of three general reasons. Each of these reasons use the same high-level namespace (scikits)."
>
> I think we can use this model, and that the following approach can be used here, namely:
>
> - start with a very basic 'astropy' package, with e.g. support for FITS/WCS
> - agree to coordinate astronomy packages with a common namespace (e.g. 'astrokit', so for example, APLpy would become astrokit.aplpy). This can help us manage the namespace (as suggested in Joe Harrington's email)
> - as astrokit modules mature, they can (if the authors are willing) be merged into the main 'astropy' package, once they have met a number of conditions, including e.g. unit tests, sphinx documentation, limited dependencies (e.g. numpy/scipy/matplotlib, and any package in the 'astropy' package), and compatible license.
>
> The advantage of this model is that this encourages the growth from the bottom up of a core astronomy package, which is manageable, as well as the independent development of other packages. It also means that the core package will be quite stable, because it will only accrete 'astrokit' modules as they become stable and mature. At the same time, it encourages developers to make their own innovative astrokit, but without commitment from the maintainers of the core package to accrete it in future.
>
> In passing, this also leaves the possibility for those who want to develop meta-pacakges of astrokit modules. Also, it will make it easier for those who want to build fully fledged astronomy python distributions.
>
> There have been many ideas passed around in the mailing list, but I think that some are maybe too ambitious. I do think that the idea suggested above is much more manageable. The concrete steps would be:
>
> - setup a central repository for the core packages, as well as astrokit pacakges (although we should leave the option for people to develop astrokit packages outside this repository too, and rely on trust and communication to avoid namespace clashes)
> - start a core package with e.g. FITS and WCS support (e.g. pyfits + pywcs)
> - set up a list of 'registered' astrokit names to avoid conflict.
> - set up a list of recommendations and requirements for astrokit pacakges
> - encourage developers of existing packages to use this namespace and follow the guidelines. This would be very little work in some cases, which is why I think it could work.
>
> Sharing a namespace is (I think) the first step to showing that we are willing to work on something coherent, and users would only see two top-level namespaces - astropy and astrokit, which would be a great improvement over the current situation where it is not immediately obvious which packages are astronomy related.
>
> Comments/suggestions/criticism welcome!
>
> Cheers,
>
> Tom
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [AstroPy] Co-ordinating Python astronomy libraries?

Erik Tollerud-2
I think the "astrokits" idea is a good one - (definitely not melding
into scikits, though - that's too big of an umbrella community for me
to be comfortable with).  I'm somewhat concerned about resulting long
import statements("from astrokits.apackage.amodule.submodule import
SomeReallyLongClassOrFunctionName"), but I guess that might be the
necessary price to pay.

I still think, though, that using that same sort of infrastructure,
while relaxing the namespace requirement, is a better solution - that
is, you can have an organizational structure like Thomas suggests
without requiring that the packages start with "astrokits" - just
strongly recommend the guidelines of consistent documentation
(ideally, I think, using sphinx) and consistent use of the standard
setup.py schemes (most important here is consistency in compiling any
Fortran or C codes... or better yet have everyone use Cython for any
interface layers), and you get the same advantages.  As James rightly
pointed out, if you go the astrokits route, you still don't really
solve the installation problem because everything is installed
separately - all that's really important is a central listing of the
"sanctioned" astrokits.

Another thing that would probably be a very good thing to do if
possible would be to allow for a "proprietary time" - that is, allow
people to develop something only they have access to until they
publish the first paper or whatever using/describing that tool, and
then afterwards it goes public as an astrokit for everyone to see.

So in my mind it seems reasonable to set up a shared repository/web
site with the option (but not requirement) of using an "astrokits"
namespace, with the understanding that when possible, the core
"astropy" modules would be used to avoid duplicating effort (and the
code must follow the aforementioned rather loose standards).



On Wed, Jul 7, 2010 at 6:01 PM, Kelle Cruz <[hidden email]> wrote:
> Well, Eli and Tom have already done the listing on AstroPython:
> http://www.astropython.org/resources
> or
> http://www.astropython.org/resources?sort=tag

I hadn't realized how complete this listing is - it's a great start,
but the one thing that's missing for what I had in mind is an access
API or rigorous entry standard.  That is, it has to be possible to
write scripts that work like pip to grab download links and use them
to automatically build and install packages.  This isn't at all hard
to do, it just requires either a secondary database with an API, or a
slightly more standardized entry format with source download links
(which don't have to be hosted on the same page), and some
standardized description of the non-python requirements.  With that,
it'd be pretty easy to write something like an "astropip" that could
be used to automate installation/upgrade of any of the listed
packages.

An "astropip" has the advantage of allowing us to just roll our own
sorts of metapackages and get around the setuptools/distribute
shortcomings.  The more I think about it, that might be the way to go.


> - I think a AAS Splinter is a great idea...looks like the deadline isn't
> until Dec 1 for Seattle, but we should get on it since they are assigned on
> a first-come, first-served basis based on room availability.  It might make
> more sense to do Boston in May 2011 because people won't be as busy with
> other meeting things and it should be nice there in spring

I would think Seattle might be better because a lot more people will
be there... and anyway, a dreary Seattle winter day is much more
likely to keep people in a room talking about python libraries, isn't
it?  (Also, for full disclosure, I'm on the west coast, so I
personally would prefer it given that I'm not sure I can make it to
Boston...)



On Thu, Jul 8, 2010 at 11:03 AM, Perry Greenfield <[hidden email]> wrote:
> Well, more the former, but also to enable something along the latter (though
> not necessarily part of a single install). When things are here and there,
> it is more difficult to package those together in an automatic way. Not
> impossible, just more work and more ways for things to go wrong.

You are right about this, but as I've described above, I think with
better consistency in how all the packages are installed, it'll be far
easier to deal with these problems.  And in my mind, the gains are
much greater because it give access to a much larger brain-trust of
people who want a fair amount of freedom (at least initially) in how
they're doing their project.



> But trac is fairly portable since it is widely used. Migrating the info from
> some of the others can be a  more difficult problem (lock-in issues). As for
> svn, git, hg, bazaar issues, nothing is going to satisfy everyone, I agree

The Trac thing is a personal preference - I admit it might well be the
best all-around compromise, although I still think allowing the option
of external hosting might be a good idea, as long as they have
standardized entries in whatever listing is used.  I want to
reiterate, though, that I think it's important to *not* use svn over
the other options I mentioned - distributed version control systems
(e.g. bzr, hg, or git) are far more suited to the kind of development
I think most astronomers typically are used to, as they can make their
own local copy, do their own thing without any access to the central
repository, and later merge it back in.



> That works to a certain level. But before long, there are n flavors of
> representing this and that, and combining 10 packages like this to use in
> your own  code can get to be a real chore. You'll spend a lot of time moving
> from one convention to another (and not catching some bugs). I don't think
> making everyone conform is a good solution, but eventually standardizing on
> core libraries is a very good thing in the long run. I think there is a
> reasonable middle ground on this kind of thing.

You're absolutely right about the problems of fragmentation and the
virtues of later standardization. But right now, given that those core
libraries don't exist, the freedom to experiment is also very
important.  Of course, I'm biased in that I'm working for a rather
more pythonic/object-oriented flavor than the traditional IDL-like
libraries so I'm leveraged a bit towards the experiment side... but I
think the point is still valid, so as you say, a general middle ground
is important.


--
Erik Tollerud
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user