[SciPy-User] Opportunity to contribute: 'pip upgrade'

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[SciPy-User] Opportunity to contribute: 'pip upgrade'

Nathaniel Smith
Hi all,

Just wanted to bring this to the attention of a wider audience --

Right now, if you're using pip to manage packages in a virtualenv,
then upgrading to a new version of some package is *super annoying*,
because you run

pip install -U <whatever package>

and then it starts upgrading not just the package you requested, but
all the packages that it depends on. And if one of those packages is
numpy or scipy or whatever, then suddenly your laptop is overheating
and what should have been a 10 second process takes an hour and might
fail and arrrghh frustrating. Then this also has nasty knock-on
effects at the ecosystem level, because it motivates lots of numerical
python packages to simply lie about what their dependencies are, on
the theory that 'pip install -U' can't mess up a dependency that it
can't see. Which resolves the immediate problem, but is obviously not
a good long-term solution.

From some recent discussions on distutils-sig, one outcome was that
the pip maintainers are actually totally happy to fix this if anyone
wants to write a patch -- basically the idea would be to add a 'pip
upgrade' command that works the way you'd expect (only upgrades the
named package, plus whatever other packages are minimally required in
order to satisfy its new dependencies -- basically the equivalent of
'pip install foo=<latest version>'), and deprecate 'install -U':

    https://github.com/pypa/pip/issues/59#issuecomment-147149208

So if anyone is feeling inspired to make a highly-visible fix to one
of the most-widely-used Python programs, and earn the undying
gratitude of lots of maintainers and users, then writing a patch to
implement 'pip upgrade' and submitting it upstream would be a great
opportunity for that ;-)

-n

--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Jouni K. Seppänen
Nathaniel Smith <[hidden email]> writes:

> basically the idea would be to add a 'pip upgrade' command that works
> the way you'd expect (only upgrades the named package, plus whatever
> other packages are minimally required in order to satisfy its new
> dependencies -- basically the equivalent of 'pip install foo=<latest
> version>')

I submitted a patch that attempts to do this:
https://github.com/pypa/pip/pull/3194

--
Jouni K. Seppänen
http://www.iki.fi/jks

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Nathaniel Smith
On Sun, Oct 18, 2015 at 11:27 AM, Jouni K. Seppänen <[hidden email]> wrote:

> Nathaniel Smith <[hidden email]> writes:
>
>> basically the idea would be to add a 'pip upgrade' command that works
>> the way you'd expect (only upgrades the named package, plus whatever
>> other packages are minimally required in order to satisfy its new
>> dependencies -- basically the equivalent of 'pip install foo=<latest
>> version>')
>
> I submitted a patch that attempts to do this:
> https://github.com/pypa/pip/pull/3194

Awesome!

-n

--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Neal Becker
Nathaniel Smith wrote:

> On Sun, Oct 18, 2015 at 11:27 AM, Jouni K. Seppänen <[hidden email]> wrote:
>> Nathaniel Smith <[hidden email]> writes:
>>
>>> basically the idea would be to add a 'pip upgrade' command that works
>>> the way you'd expect (only upgrades the named package, plus whatever
>>> other packages are minimally required in order to satisfy its new
>>> dependencies -- basically the equivalent of 'pip install foo=<latest
>>> version>')
>>
>> I submitted a patch that attempts to do this:
>> https://github.com/pypa/pip/pull/3194
>
> Awesome!
>
> -n
>

Perhaps related,
I am running linux/fedora.  My vendor (Fedora) provides may python packages,
but they aren't always the latest.  I find the most convenient approach is
to use:
pip install foo --user

But in many cases, this won't work correctly.  I often find that although I
have the latest numpy in my local --user, pip is confused about the older
one install into the standard sys location.  It attempts to build another
copy of numpy (for example), and eventually fails on trying to remove the
system copy (pip running un-privileged).

This has been a big annoyance for a long time.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Daπid
On 19 October 2015 at 13:58, Neal Becker <[hidden email]> wrote:

Perhaps related,
I am running linux/fedora.  My vendor (Fedora) provides may python packages,
but they aren't always the latest.  I find the most convenient approach is
to use:
pip install foo --user

I install everything into a virtualenv and ignore system's Python. This way I have much better control over what is going on (if I install inkscape, it drags along a possibly outdated Numpy version, for example), and risk free updates. And if anything ever goes wrong, it is much easier to nuke (remove a folder) and start it all over.

/David.


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Nathaniel Smith
In reply to this post by Neal Becker

On Oct 19, 2015 4:58 AM, "Neal Becker" <[hidden email]> wrote:
>
> Nathaniel Smith wrote:
>
> > On Sun, Oct 18, 2015 at 11:27 AM, Jouni K. Seppänen <[hidden email]> wrote:
> >> Nathaniel Smith <[hidden email]> writes:
> >>
> >>> basically the idea would be to add a 'pip upgrade' command that works
> >>> the way you'd expect (only upgrades the named package, plus whatever
> >>> other packages are minimally required in order to satisfy its new
> >>> dependencies -- basically the equivalent of 'pip install foo=<latest
> >>> version>')
> >>
> >> I submitted a patch that attempts to do this:
> >> https://github.com/pypa/pip/pull/3194
> >
> > Awesome!
> >
> > -n
> >
>
> Perhaps related,
> I am running linux/fedora.  My vendor (Fedora) provides may python packages,
> but they aren't always the latest.  I find the most convenient approach is
> to use:
> pip install foo --user
>
> But in many cases, this won't work correctly.

Not sure what to say without more details about what these "many cases" are. IIUC this should work and if it doesn't then it's a bug (though there are adjacent things that aren't expected to work, e.g. 'setup.py install --user').

-n


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Neal Becker
Nathaniel Smith wrote:

> On Oct 19, 2015 4:58 AM, "Neal Becker" <[hidden email]> wrote:
>>
>> Nathaniel Smith wrote:
>>
>> > On Sun, Oct 18, 2015 at 11:27 AM, Jouni K. Seppänen <[hidden email]> wrote:
>> >> Nathaniel Smith <[hidden email]> writes:
>> >>
>> >>> basically the idea would be to add a 'pip upgrade' command that works
>> >>> the way you'd expect (only upgrades the named package, plus whatever
>> >>> other packages are minimally required in order to satisfy its new
>> >>> dependencies -- basically the equivalent of 'pip install foo=<latest
>> >>> version>')
>> >>
>> >> I submitted a patch that attempts to do this:
>> >> https://github.com/pypa/pip/pull/3194
>> >
>> > Awesome!
>> >
>> > -n
>> >
>>
>> Perhaps related,
>> I am running linux/fedora.  My vendor (Fedora) provides may python
> packages,
>> but they aren't always the latest.  I find the most convenient approach
>> is to use:
>> pip install foo --user
>>
>> But in many cases, this won't work correctly.
>
> Not sure what to say without more details about what these "many cases"
> are. IIUC this should work and if it doesn't then it's a bug (though there
> are adjacent things that aren't expected to work, e.g. 'setup.py install
> --user').
>

Well I had just installed numpy-1.10.1 and pandas-0.17.0, and here's what
happens:

$ pip install --user --up pandas
Requirement already up-to-date: pandas in
/home/nbecker/.local/lib/python2.7/site-packages
Requirement already up-to-date: python-dateutil in
/home/nbecker/.local/lib/python2.7/site-packages (from pandas)
Requirement already up-to-date: pytz>=2011k in
/home/nbecker/.local/lib/python2.7/site-packages (from pandas)
Collecting numpy>=1.7.0 (from pandas)
  Using cached numpy-1.10.1.tar.gz
Collecting six>=1.5 (from python-dateutil->pandas)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: numpy, six
  Found existing installation: numpy 1.8.2
    DEPRECATION: Uninstalling a distutils installed project (numpy) has been
deprecated and will be removed in a future version. This is due to the fact
that uninstalling a distutils project will only partially uninstall the
project.
    Uninstalling numpy-1.8.2:
      Successfully uninstalled numpy-1.8.2
  Running setup.py install for numpy
  C-c C-c  Rolling back uninstall of numpy

As you see, it seems to be installing numpy-1.10.1 again, even though it's
already installed

  /home/nbecker/.local/lib/python2.7/site-packages:
  total used in directory 4264 available 781585920
  drwx------. 1 nbecker nbecker   15272 Oct 19 14:15 .
  drwxrwxr-x. 1 nbecker nbecker     190 Oct 15 10:44 pandas-0.17.0-
py2.7.egg-info
  drwxrwxr-x. 1 nbecker nbecker     506 Oct 15 10:44 pandas
  -rw-rw-r--. 1 nbecker nbecker     869 Oct 15 10:41 easy-install.pth
  -rw-rw-r--. 1 nbecker nbecker    2060 Oct 15 10:37 numpy-1.10.1-py2.7.egg-
info
  drwxrwxr-x. 1 nbecker nbecker     572 Oct 15 10:37 numpy
  drwxrwxr-x. 1 nbecker nbecker     136 Oct 15 10:32 pytz-2015.6.dist-info
  drwxrwxr-x. 1 nbecker nbecker     272 Oct 15 10:32 pytz


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Opportunity to contribute: 'pip upgrade'

Nathaniel Smith

On Oct 19, 2015 11:18 AM, "Neal Becker" <[hidden email]> wrote:
>
> Nathaniel Smith wrote:
>
> > On Oct 19, 2015 4:58 AM, "Neal Becker" <[hidden email]> wrote:
> >>
> >> Nathaniel Smith wrote:
> >>
> >> > On Sun, Oct 18, 2015 at 11:27 AM, Jouni K. Seppänen <[hidden email]> wrote:
> >> >> Nathaniel Smith <[hidden email]> writes:
> >> >>
> >> >>> basically the idea would be to add a 'pip upgrade' command that works
> >> >>> the way you'd expect (only upgrades the named package, plus whatever
> >> >>> other packages are minimally required in order to satisfy its new
> >> >>> dependencies -- basically the equivalent of 'pip install foo=<latest
> >> >>> version>')
> >> >>
> >> >> I submitted a patch that attempts to do this:
> >> >> https://github.com/pypa/pip/pull/3194
> >> >
> >> > Awesome!
> >> >
> >> > -n
> >> >
> >>
> >> Perhaps related,
> >> I am running linux/fedora.  My vendor (Fedora) provides may python
> > packages,
> >> but they aren't always the latest.  I find the most convenient approach
> >> is to use:
> >> pip install foo --user
> >>
> >> But in many cases, this won't work correctly.
> >
> > Not sure what to say without more details about what these "many cases"
> > are. IIUC this should work and if it doesn't then it's a bug (though there
> > are adjacent things that aren't expected to work, e.g. 'setup.py install
> > --user').
> >
>
> Well I had just installed numpy-1.10.1 and pandas-0.17.0, and here's what
> happens:
>
> $ pip install --user --up pandas
> Requirement already up-to-date: pandas in
> /home/nbecker/.local/lib/python2.7/site-packages
> Requirement already up-to-date: python-dateutil in
> /home/nbecker/.local/lib/python2.7/site-packages (from pandas)
> Requirement already up-to-date: pytz>=2011k in
> /home/nbecker/.local/lib/python2.7/site-packages (from pandas)
> Collecting numpy>=1.7.0 (from pandas)
>   Using cached numpy-1.10.1.tar.gz
> Collecting six>=1.5 (from python-dateutil->pandas)
>   Using cached six-1.10.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six
>   Found existing installation: numpy 1.8.2
>     DEPRECATION: Uninstalling a distutils installed project (numpy) has been
> deprecated and will be removed in a future version. This is due to the fact
> that uninstalling a distutils project will only partially uninstall the
> project.
>     Uninstalling numpy-1.8.2:
>       Successfully uninstalled numpy-1.8.2
>   Running setup.py install for numpy
>   C-c C-c  Rolling back uninstall of numpy
>
> As you see, it seems to be installing numpy-1.10.1 again, even though it's
> already installed
>
>   /home/nbecker/.local/lib/python2.7/site-packages:
>   total used in directory 4264 available 781585920
>   drwx------. 1 nbecker nbecker   15272 Oct 19 14:15 .
>   drwxrwxr-x. 1 nbecker nbecker     190 Oct 15 10:44 pandas-0.17.0-
> py2.7.egg-info
>   drwxrwxr-x. 1 nbecker nbecker     506 Oct 15 10:44 pandas
>   -rw-rw-r--. 1 nbecker nbecker     869 Oct 15 10:41 easy-install.pth
>   -rw-rw-r--. 1 nbecker nbecker    2060 Oct 15 10:37 numpy-1.10.1-py2.7.egg-
> info
>   drwxrwxr-x. 1 nbecker nbecker     572 Oct 15 10:37 numpy
>   drwxrwxr-x. 1 nbecker nbecker     136 Oct 15 10:32 pytz-2015.6.dist-info
>   drwxrwxr-x. 1 nbecker nbecker     272 Oct 15 10:32 pytz

I'm struck that pip seems to have found a copy of numpy 1.8.2 somewhere, yet we can tell that it's neither in your regular user install directory (or it would be listed), nor is it a fedora-installed version (because if it were then the uninstall would have errored out for lack of write permissions). But in a default setup these are the only two places an install can go. What's going on with your sys.path? Where is this 1.8.2?

-n


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user