Scipy-User

[SciPy-User] Travis, wheels, and scikit-image

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
Fabien
Reply | Threaded
Open this post in threaded view
♦
♦
|

[SciPy-User] Travis, wheels, and scikit-image

Fabien
Folks,

I have a project which requires several packages from the scipy track. I
configured travis to use cached wheels in order to reduce build time.
Everything worked fine until I had to add scikit-image to my
dependencies. Here are the relevant parts of my travis config:

language: python
sudo: false
cache:
   directories:
     - ~/.cache/pip
env:
   global:
     - PIP_WHEEL_DIR=$HOME/.cache/pip/wheels
     - PIP_FIND_LINKS=file://$HOME/.cache/pip/wheels
     - NUMPY=numpy
[...]
before_install:
   - pip install -U pip
   - pip install wheel
install:
   - "pip wheel -r requirements.txt"
   - "pip install -r requirements.txt"
   - "pip install coveralls"
   - "pip install nose"
   - "pip install -e ."
script:
   - nosetests

And of requirements.txt:

numpy
scipy
scikit-image


The problem is that scikit-image wants to import scipy from setup.py,
and I get this error from my travis log:

9.50s$ pip wheel -r requirements.txt
Collecting numpy (from -r requirements.txt (line 2))
   File was already downloaded
/home/travis/.cache/pip/wheels/numpy-1.10.0.post2-cp27-none-linux_x86_64.whl
Collecting scipy (from -r requirements.txt (line 3))
   File was already downloaded
/home/travis/.cache/pip/wheels/scipy-0.16.0-cp27-none-linux_x86_64.whl
Collecting scikit-image (from -r requirements.txt (line 8))
   Downloading scikit-image-0.11.3.tar.gz (18.6MB)
     100% |████████████████████████████████| 18.6MB 22kB/s
     Complete output from command python setup.py egg_info:
     Traceback (most recent call last):
       File "<string>", line 20, in <module>
       File "/tmp/pip-build-WRt8fn/scikit-image/setup.py", line 77, in
<module>
         import scipy
     ImportError: No module named scipy


Is my way to configure travis a bad one? Should I handle my wheels
differently? I am also thinking about using conda in travis as explained
here (http://conda.pydata.org/docs/travis.html) but I wanted to avoid it
as long as not necessary: it seems that it doesn't use the travis cache?

Thanks for your advices,

Fabien

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
ralfgommers
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Travis, wheels, and scikit-image

ralfgommers


On Sat, Oct 10, 2015 at 2:54 PM, Fabien <[hidden email]> wrote:
Folks,

I have a project which requires several packages from the scipy track. I configured travis to use cached wheels in order to reduce build time. Everything worked fine until I had to add scikit-image to my dependencies. Here are the relevant parts of my travis config:

requirements.txt is fragile. I suggest to use a pre-install shell script. Example: https://github.com/scikit-image/scikit-image/blob/master/tools/travis_before_install.sh

Ralf



language: python
sudo: false
cache:
  directories:
    - ~/.cache/pip
env:
  global:
    - PIP_WHEEL_DIR=$HOME/.cache/pip/wheels
    - PIP_FIND_LINKS=file://$HOME/.cache/pip/wheels
    - NUMPY=numpy
[...]
before_install:
  - pip install -U pip
  - pip install wheel
install:
  - "pip wheel -r requirements.txt"
  - "pip install -r requirements.txt"
  - "pip install coveralls"
  - "pip install nose"
  - "pip install -e ."
script:
  - nosetests

And of requirements.txt:

numpy
scipy
scikit-image


The problem is that scikit-image wants to import scipy from setup.py, and I get this error from my travis log:

9.50s$ pip wheel -r requirements.txt
Collecting numpy (from -r requirements.txt (line 2))
  File was already downloaded /home/travis/.cache/pip/wheels/numpy-1.10.0.post2-cp27-none-linux_x86_64.whl
Collecting scipy (from -r requirements.txt (line 3))
  File was already downloaded /home/travis/.cache/pip/wheels/scipy-0.16.0-cp27-none-linux_x86_64.whl
Collecting scikit-image (from -r requirements.txt (line 8))
  Downloading scikit-image-0.11.3.tar.gz (18.6MB)
    100% |████████████████████████████████| 18.6MB 22kB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-build-WRt8fn/scikit-image/setup.py", line 77, in <module>
        import scipy
    ImportError: No module named scipy


Is my way to configure travis a bad one? Should I handle my wheels differently? I am also thinking about using conda in travis as explained here (http://conda.pydata.org/docs/travis.html) but I wanted to avoid it as long as not necessary: it seems that it doesn't use the travis cache?

Thanks for your advices,

Fabien

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Fabien
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Travis, wheels, and scikit-image

Fabien
On 10/10/2015 03:00 PM, Ralf Gommers wrote:

> On Sat, Oct 10, 2015 at 2:54 PM, Fabien <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Folks,
>
>     I have a project which requires several packages from the scipy
>     track. I configured travis to use cached wheels in order to reduce
>     build time. Everything worked fine until I had to add scikit-image
>     to my dependencies. Here are the relevant parts of my travis config:
>
>
> requirements.txt is fragile. I suggest to use a pre-install shell
> script. Example:
> https://github.com/scikit-image/scikit-image/blob/master/tools/travis_before_install.sh
> called from:
> https://github.com/scikit-image/scikit-image/blob/master/.travis.yml#L48
>
> Ralf


Thanks Ralf. A bit oversized for my needs but I thinks that it will make
it.

Anyone has experience with conda on Travis? The solution provided here
((http://conda.pydata.org/docs/travis.html)) really seems inefficient
since all packages have to be downloaded at each build...

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Daπid
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Travis, wheels, and scikit-image

Daπid

On 13 October 2015 at 17:14, Fabien <[hidden email]> wrote:
Thanks Ralf. A bit oversized for my needs but I thinks that it will make it.


A simplest option, specify them manually. Doable if you don't have many packages, and you can use it to pre-install the ones that may give you trouble:

before_install:
  - uname -a
  - free -m
  - df -h
  - ulimit -a
  - pip install -U pip wheel setuptools
  - pip install numpy cython nose six matplotlib
  - pip install -v -U scipy
  - python -V
Note that I added the -v flag for scipy because it takes around 10 min to install on Travis, and Travis kills jobs that don't produce output after that time. This is only relevant once per Scipy release, the rest of the times it will use the cached wheel. The full script is here:

https://raw.githubusercontent.com/Dapid/GPy/a02beebf27a95733e7a49a86f1a6642b2cb160fe/.travis.yml

Anyway, downloading and installing miniconda takes less than a minute, so inefficient is relative:

https://travis-ci.org/SheffieldML/GPy/jobs/85069851


/David.





_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
josef.pktd
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Travis, wheels, and scikit-image

josef.pktd
In reply to this post by Fabien


On Tue, Oct 13, 2015 at 11:14 AM, Fabien <[hidden email]> wrote:
On 10/10/2015 03:00 PM, Ralf Gommers wrote:
On Sat, Oct 10, 2015 at 2:54 PM, Fabien <[hidden email]
 <mailto:[hidden email]>> wrote:

    Folks,

    I have a project which requires several packages from the scipy
    track. I configured travis to use cached wheels in order to reduce
    build time. Everything worked fine until I had to add scikit-image
    to my dependencies. Here are the relevant parts of my travis config:


requirements.txt is fragile. I suggest to use a pre-install shell
script. Example:
https://github.com/scikit-image/scikit-image/blob/master/tools/travis_before_install.sh
called from:
https://github.com/scikit-image/scikit-image/blob/master/.travis.yml#L48

Ralf


Thanks Ralf. A bit oversized for my needs but I thinks that it will make it.

Anyone has experience with conda on Travis? The solution provided here ((http://conda.pydata.org/docs/travis.html)) really seems inefficient since all packages have to be downloaded at each build...


statsmodels has been using conda on Travis for a while now. I don't remember seeing any problems related to downloading, so I never paid attention to whether we download on each testrun or not.

(Also, we still haven't switched to the new Travis infrastructure. )

In the Travis log, conda reports that it's downloading around 200 MB for that test run.

Josef
 

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Matthew Brett
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Travis, wheels, and scikit-image

Matthew Brett
In reply to this post by Daπid
Hi,

On Tue, Oct 13, 2015 at 8:39 AM, Daπid <[hidden email]> wrote:

>
> On 13 October 2015 at 17:14, Fabien <[hidden email]> wrote:
>>
>> Thanks Ralf. A bit oversized for my needs but I thinks that it will make
>> it.
>
>
>
> A simplest option, specify them manually. Doable if you don't have many
> packages, and you can use it to pre-install the ones that may give you
> trouble:
>
> before_install:
>   - uname -a
>   - free -m
>   - df -h
>   - ulimit -a
>   - pip install -U pip wheel setuptools
>   - pip install numpy cython nose six matplotlib
>   - pip install -v -U scipy
>   - python -V
>
> Note that I added the -v flag for scipy because it takes around 10 min to
> install on Travis, and Travis kills jobs that don't produce output after
> that time. This is only relevant once per Scipy release, the rest of the
> times it will use the cached wheel. The full script is here:
>
> https://raw.githubusercontent.com/Dapid/GPy/a02beebf27a95733e7a49a86f1a6642b2cb160fe/.travis.yml

There's a good example of using pre-built travis wheels here :
https://github.com/matplotlib/matplotlib/blob/master/tools/travis_tools.sh

Cheers,

Matthew
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/scipy-user
Free forum by Nabble Edit this page