Central File Exchange for Scipy

classic Classic list List threaded Threaded
48 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Central File Exchange for Scipy

Jason Grout-5

I've been funded over the summer by an NSF grant to build a library of
Sage "interacts" (basically small snippets of Sage/Python code).
Fernando Perez strongly encouraged me at a recent Sage Days to adopt a
version controlled snippet model (ala Gist).  The other night I threw
together a very rough proof-of-concept, just-barely-working initial
start of something along these lines (I hope I put enough disclaimers in
there!)  My code is here:

https://github.com/jasongrout/snippets

It requires Flask and Mercurial to be installed. Yes, the irony of using
a Mercurial backend in a project on github is not lost on me.  Though I
prefer git, the python API to use and create mercurial repositories was
too nice to turn down in this prototyping stage.  I tried to encapsulate
the VCS commands in a wrapper class so that the backend could be easily
switched to git if need be.

This reminded me of the Central File Exchange thread from last November,
and in particular, several people saying that they were working on a
version-controlled snippet database [1].  William or Andrew, have you
posted your work anywhere?  I think we have very similar goals.

I won't be able to work on this heavily for about a month (until after
the semester), but I will be hitting it pretty hard during the summer.
The hope is to have a good production version by July.

If anyone else is working on a related project, please let me know, as
we can probably collaborate.  If anyone wants to fork the github repo
above and work on it, feel free!

To prevent license discussions from eating up too much energy/time, I
have decided that the site that I set up will have all snippets be
(modified) BSD licensed.  That may change, of course, before it's
actually implemented, but it's not up for debate now.  To encourage
collaboration you folks, I'm willing to put a BSD license on the
codebase as well if possible, though the Mercurial docs page seems to
indicate that I'm forced to use GPLv2 (+?) if I use their API, which I
am at this point [2].

Thanks,

Jason

[1] http://mail.scipy.org/pipermail/scipy-user/2010-November/027690.html

[2] http://mercurial.selenic.com/wiki/MercurialApi

--
Jason Grout

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Joshua Holbrook
On Mon, Apr 18, 2011 at 7:40 PM, Jason Grout
<[hidden email]> wrote:

>
> I've been funded over the summer by an NSF grant to build a library of
> Sage "interacts" (basically small snippets of Sage/Python code).
> Fernando Perez strongly encouraged me at a recent Sage Days to adopt a
> version controlled snippet model (ala Gist).  The other night I threw
> together a very rough proof-of-concept, just-barely-working initial
> start of something along these lines (I hope I put enough disclaimers in
> there!)  My code is here:
>
> https://github.com/jasongrout/snippets

This sounds awesome! *watched*

> It requires Flask and Mercurial to be installed. Yes, the irony of using
> a Mercurial backend in a project on github is not lost on me.  Though I
> prefer git, the python API to use and create mercurial repositories was
> too nice to turn down in this prototyping stage.  I tried to encapsulate
> the VCS commands in a wrapper class so that the backend could be easily
> switched to git if need be.

I don't think there's anything wrong with using mercurial here. If you
want to support git pulls (though a lot of pythonistas prefer hg it
seems the scientific community's leaning towards git), you could
always make a system to convert repos to git on-the-fly, maybe using
something like http://offbytwo.com/git-hg/ . In fact, I think I would
do something like that as long as it wasn't too slow.

Or, y'know, rewrite the VCS commands. :) Either way.

> This reminded me of the Central File Exchange thread from last November,
> and in particular, several people saying that they were working on a
> version-controlled snippet database [1].  William or Andrew, have you
> posted your work anywhere?  I think we have very similar goals.

Yeah, it sounds like you're doing something really similar, though
with a Sage focus. How well do these Sage interacts things work with
more vanilla python? I for one don't really use Sage, so if it was
ridiculously different I don't know how much utility the SagExchange
would be for me.

> To prevent license discussions from eating up too much energy/time, I
> have decided that the site that I set up will have all snippets be
> (modified) BSD licensed.

Word. I approve. I hope good things come of this. I'm mostly working
with javascript these days, but if I have time I might take a look or
two.

--Josh
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Jason Grout-5
On 4/18/11 11:54 PM, Joshua Holbrook wrote:

>
> Yeah, it sounds like you're doing something really similar, though
> with a Sage focus. How well do these Sage interacts things work with
> more vanilla python? I for one don't really use Sage, so if it was
> ridiculously different I don't know how much utility the SagExchange
> would be for me.

The site would be simply a snippet site, geared towards python.
Additionally, for each snippet, it would be able to send the snippet to
a server to execute the snippet and display the results, like the
htmlnotebook project in the IPython trunk.  Except we'd probably also
use another project we're currently working on, a "single-cell Sage
Notebook server": https://github.com/jasongrout/simple-python-db-compute 
(see http://wiki.sagemath.org/DrakeSageGroup for rough notes of our
progress and todo list).  The "interacts" part just is some client-side
javascript that makes sliders, textboxes, etc.

In a sense, the Sage interact site would be like a combination of Gist,
Wolfram Demonstrations, Pastebin, Central File Exchange, etc.  For your
purposes, you could ignore the "interact" part and just think of it as a
version-controlled snippet site that also lets you execute the snippets
on a remote server.



>
>> To prevent license discussions from eating up too much energy/time, I
>> have decided that the site that I set up will have all snippets be
>> (modified) BSD licensed.
>
> Word. I approve. I hope good things come of this. I'm mostly working
> with javascript these days, but if I have time I might take a look or
> two.

We could definitely use some Javascript expertise, especially when we
start working on integrating the single-cell server mentioned above with
the snippet site.

Thanks,

Jason
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Gael Varoquaux
In reply to this post by Jason Grout-5
On Mon, Apr 18, 2011 at 10:40:01PM -0500, Jason Grout wrote:
> I've been funded over the summer by an NSF grant to build a library of
> Sage "interacts" (basically small snippets of Sage/Python code).
> Fernando Perez strongly encouraged me at a recent Sage Days to adopt a
> version controlled snippet model (ala Gist).  The other night I threw
> together a very rough proof-of-concept, just-barely-working initial
> start of something along these lines (I hope I put enough disclaimers in
> there!)  My code is here:

> https://github.com/jasongrout/snippets

This is great. Congratulations for doing this. And congratulations for
doing it in such a way that it appeals to the scipy community in addition
to the sage community!

Gael
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Joshua Holbrook
In reply to this post by Jason Grout-5
On Mon, Apr 18, 2011 at 9:14 PM, Jason Grout
<[hidden email]> wrote:

> The site would be simply a snippet site, geared towards python.
> Additionally, for each snippet, it would be able to send the snippet to
> a server to execute the snippet and display the results, like the
> htmlnotebook project in the IPython trunk.  Except we'd probably also
> use another project we're currently working on, a "single-cell Sage
> Notebook server": https://github.com/jasongrout/simple-python-db-compute
> (see http://wiki.sagemath.org/DrakeSageGroup for rough notes of our
> progress and todo list).  The "interacts" part just is some client-side
> javascript that makes sliders, textboxes, etc.

This sounds pretty sweet. What do you mean by "single-cell?" Looking
forward to seeing how it goes! Either way, I'm watching the compute
server project now too.

>
> We could definitely use some Javascript expertise, especially when we
> start working on integrating the single-cell server mentioned above with
> the snippet site.
>

I'd hardly call myself a javascript expert XD but feel free to hit me
up if/when you're ready. You might find it easiest to get ahold of me
in the #scipy irc channel (irc://irc.freenode.net/#scipy) as
"jesusabdullah."

Good luck!

--Josh
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Pauli Virtanen-3
In reply to this post by Jason Grout-5
Mon, 18 Apr 2011 22:40:01 -0500, Jason Grout wrote:
[clip]
> I won't be able to work on this heavily for about a month (until after
> the semester), but I will be hitting it pretty hard during the summer.
> The hope is to have a good production version by July.
>
> If anyone else is working on a related project, please let me know, as
> we can probably collaborate.  If anyone wants to fork the github repo
> above and work on it, feel free!

I have something here:

    https://github.com/pv/scipyshare

Version controlled, yes, but on the file system, and not only for
snippets but also for slightly larger projects + reference links to
projects on PyPi and elsewhere.

There's no great UI yet, but the basic functionality is there. The
community part functionality is not fully there yet, though.

        Pauli

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Pauli Virtanen-3
Tue, 19 Apr 2011 08:48:17 +0000, Pauli Virtanen wrote:
[clip]
> There's no great UI yet, but the basic functionality is there.

Should have said that there's essentially *no* real UI yet, it's
basically just dumping data and forms to HTML; but real UI should be easy
to add once the rest of the things work.

        Pauli

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

william ratcliff
Andrew Wilson and I were also playing with this--one thing I was looking into before getting waylaid by work was using S3 as a backing store for git/hg.


William

On Tue, Apr 19, 2011 at 4:51 AM, Pauli Virtanen <[hidden email]> wrote:
Tue, 19 Apr 2011 08:48:17 +0000, Pauli Virtanen wrote:
[clip]
> There's no great UI yet, but the basic functionality is there.

Should have said that there's essentially *no* real UI yet, it's
basically just dumping data and forms to HTML; but real UI should be easy
to add once the rest of the things work.

       Pauli

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Alan G Isaac-2
In reply to this post by Jason Grout-5
On 4/18/2011 11:40 PM, Jason Grout wrote:
> though the Mercurial docs page seems to
> indicate that I'm forced to use GPLv2 (+?) if I use their API


That's only if you use their *internal* API,
which is discouraged.

fwiw,
Alan Isaac

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

How to create Interface classes at runtime?

Stephen Waterbury-2
In reply to this post by william ratcliff
I've been reading the traits modules has_traits.py and protocols.py
and trying to figure out what I'd use to create Interface classes
at runtime, but my brain is beginning to melt so thought I'd ask for
some advice.  I have code that creates zope.interface-style Interface
classes at runtime by instantiating InterfaceClass, but PyProtocols,
which is used by traits, has a significantly different set of apis
and the code is somewhat more difficult to read than zope.interface
code, so it's not obvious how to create traits-style Interface classes
at runtime ... any hints would be appreciated!

Thanks,
Steve
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: How to create Interface classes at runtime?

Robert Kern-2
On Tue, Apr 19, 2011 at 09:55, Stephen Waterbury
<[hidden email]> wrote:
> I've been reading the traits modules has_traits.py and protocols.py
> and trying to figure out what I'd use to create Interface classes
> at runtime, but my brain is beginning to melt so thought I'd ask for
> some advice.

I can answer your question over on enthought-dev:

  https://mail.enthought.com/mailman/listinfo/enthought-dev

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Jason Grout-5
In reply to this post by Alan G Isaac-2
On 4/19/11 8:35 AM, Alan G Isaac wrote:
> On 4/18/2011 11:40 PM, Jason Grout wrote:
>> though the Mercurial docs page seems to
>> indicate that I'm forced to use GPLv2 (+?) if I use their API
>
>
> That's only if you use their *internal* API,
> which is discouraged.


Good point.  I started just using the external API (i.e., the commands
module), but that required that I always work through the repository on
disk (locking it and then writing files to disk, then committing from
the disk contents), parse the output of log, etc.  It's *much* more
convenient to be able to commit things from memory, be able to directly
traverse the commit tree, etc.  So yes, it's a matter of convenience
now, but I also suspect that it is a matter of speed when in production.

Jason
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Jason Grout-5
In reply to this post by Pauli Virtanen-3
On 4/19/11 3:48 AM, Pauli Virtanen wrote:

> Mon, 18 Apr 2011 22:40:01 -0500, Jason Grout wrote:
> [clip]
>> I won't be able to work on this heavily for about a month (until after
>> the semester), but I will be hitting it pretty hard during the summer.
>> The hope is to have a good production version by July.
>>
>> If anyone else is working on a related project, please let me know, as
>> we can probably collaborate.  If anyone wants to fork the github repo
>> above and work on it, feel free!
>
> I have something here:
>
>      https://github.com/pv/scipyshare
>
> Version controlled, yes, but on the file system, and not only for
> snippets but also for slightly larger projects + reference links to
> projects on PyPi and elsewhere.

Nice!  I've never done anything with Django, so I added instructions to
the README for people like me and submitted a pull request.

I'll be looking at this more closely over the next month.  We certainly
have the same types of goals, but I don't have experience with Django,
so that's a bit of a hurdle.

Jason
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

josef.pktd
In reply to this post by Jason Grout-5
On Tue, Apr 19, 2011 at 12:27 PM, Jason Grout
<[hidden email]> wrote:

> On 4/19/11 8:35 AM, Alan G Isaac wrote:
>> On 4/18/2011 11:40 PM, Jason Grout wrote:
>>> though the Mercurial docs page seems to
>>> indicate that I'm forced to use GPLv2 (+?) if I use their API
>>
>>
>> That's only if you use their *internal* API,
>> which is discouraged.
>
>
> Good point.  I started just using the external API (i.e., the commands
> module), but that required that I always work through the repository on
> disk (locking it and then writing files to disk, then committing from
> the disk contents), parse the output of log, etc.  It's *much* more
> convenient to be able to commit things from memory, be able to directly
> traverse the commit tree, etc.  So yes, it's a matter of convenience
> now, but I also suspect that it is a matter of speed when in production.

there is a GSoC proposal on this issue

http://mercurial.selenic.com/wiki/SummerOfCode/2011/IdanKamara

So maybe this will improve in future.

Josef

>
> Jason
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Jason Grout-5
In reply to this post by Jason Grout-5
On 4/19/11 12:48 PM, Jason Grout wrote:
> but I don't have experience with Django,
> so that's a bit of a hurdle.

Do you have any resources that you recommend for learning Django other
than the standard Django documentation [1]?  I see the Django Book [2],
for example, and it appears pretty well written, but is for Django 1.0,
so I don't know how much still applies.

Also, it would be very helpful in browsing the codebase if there was a
short description of each app in the DESIGN.rst and a short blurb about
how it fit in with the rest of the website.  I was a bit loss on parts
of the app.  I didn't figure out how an actual revision of a file was
stored or retrieved, for example.  Is it stored using a DVCS, or is each
version stored separately?

Thanks,

Jason

[1] http://docs.djangoproject.com/en/1.3/

[2] http://www.djangobook.com/
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Pauli Virtanen-3
In reply to this post by Jason Grout-5
On Tue, 19 Apr 2011 12:48:10 -0500, Jason Grout wrote:
[clip]
> Nice!  I've never done anything with Django, so I added instructions to
> the README for people like me and submitted a pull request.

Thanks, applied.

> I'll be looking at this more closely over the next month.  We certainly
> have the same types of goals, but I don't have experience with Django,
> so that's a bit of a hurdle.

The basic urls -> view (<-> model) -> template CRUD flow is pretty
straightforward in Django, and I suppose that's mostly all that is needed
for this particular project.

The project layout maybe looks a bit framework-y, but it's basically just
a Python package with all the urls.py hooking it into the rest of Django.

        Pauli

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Pauli Virtanen-3
In reply to this post by Jason Grout-5
On Tue, 19 Apr 2011 13:25:05 -0500, Jason Grout wrote:
> On 4/19/11 12:48 PM, Jason Grout wrote:
>> but I don't have experience with Django, so that's a bit of a hurdle.
>
> Do you have any resources that you recommend for learning Django other
> than the standard Django documentation [1]?  I see the Django Book [2],
> for example, and it appears pretty well written, but is for Django 1.0,
> so I don't know how much still applies.

Django is pretty slow-moving, and most of things in the book probably
still apply.

Personally, I'd start like this:

- Go through the Django tutorial:

  http://docs.djangoproject.com/en/1.3/intro/tutorial01/

  (The admin interface is in my opinion not so important,
  makes things simpler to not think too deeply about it.)

- Browse through interesting sections of the main docs;
  e.g. database query, template syntax docs:

  http://docs.djangoproject.com/en/1.3/topics/db/queries/

  http://docs.djangoproject.com/en/1.3/topics/templates/

> Also, it would be very helpful in browsing the codebase if there was a
> short description of each app in the DESIGN.rst and a short blurb about
> how it fit in with the rest of the website.  I was a bit loss on parts
> of the app.

Things were a bit in motion, so yes, it's not terribly well documented at
the moment.

Anyway, the basic subpackages of `scipyshare` are at the moment like so:

    catalog     -- main application/snippet catalog data
    community   -- user-assigned tags, ratings, etc.
    filestorage -- managing storing sets of files on the FS
    front       -- just some dummy front page
    importing   -- dealing with importing from PyPi
    user        -- user login/logout, profile, etc.

The aim would be to keep these parts knowing only little about each
other.

The deploy/ directory basically only contains the configuration files.

> I didn't figure out how an actual revision of a file was
> stored or retrieved, for example.  Is it stored using a DVCS, or is each
> version stored separately?

Stored separately on the filesystem.

For each revision, you get a catalog.models.Revision object in a
catalog.models.Entry. The set of files associated with each Revision is
managed by filestorage.models.FileSet.

Since a DVCS would not be exposed to the outside of the application, I
decided against using one internally. A better storage system can be
swapped in later on, if such a thing turns out to be necessary (which I
doubt a bit -- for example the MoinMoin wiki works fine with a filesystem
based wiki page storage).

        Pauli

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Bruce Southey
On 04/19/2011 02:00 PM, Pauli Virtanen wrote:

> On Tue, 19 Apr 2011 13:25:05 -0500, Jason Grout wrote:
>> On 4/19/11 12:48 PM, Jason Grout wrote:
>>> but I don't have experience with Django, so that's a bit of a hurdle.
>> Do you have any resources that you recommend for learning Django other
>> than the standard Django documentation [1]?  I see the Django Book [2],
>> for example, and it appears pretty well written, but is for Django 1.0,
>> so I don't know how much still applies.
> Django is pretty slow-moving, and most of things in the book probably
> still apply.
>
> Personally, I'd start like this:
>
> - Go through the Django tutorial:
>
>    http://docs.djangoproject.com/en/1.3/intro/tutorial01/
>
>    (The admin interface is in my opinion not so important,
>    makes things simpler to not think too deeply about it.)
>
> - Browse through interesting sections of the main docs;
>    e.g. database query, template syntax docs:
>
>    http://docs.djangoproject.com/en/1.3/topics/db/queries/
>
>    http://docs.djangoproject.com/en/1.3/topics/templates/
>
I agree especially about the tutorial being a very good place to start
and go back to.

You just have to address the Django changes when the code from 0.9x
gives errors in your installed version. While a little stressing, I did
not find these that hard to fix.

Version 2 of the book (that I have not read) is at:
http://www.djangobook.com/en/2.0/


Bruce

PS Your site, so you choose the license :-)
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

Joshua Holbrook
> Django django django

FWIW, I'm a fan of the little frameworks. ;) Flask looks nice!

--Josh
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Central File Exchange for Scipy

william ratcliff
Flask is nice, but why reinvent the wheel?

On Tue, Apr 19, 2011 at 3:14 PM, Joshua Holbrook <[hidden email]> wrote:
> Django django django

FWIW, I'm a fan of the little frameworks. ;) Flask looks nice!

--Josh
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
123