[SciPy-User] setuptools messing with sdists using numpy.distutils and Fortran libraries

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[SciPy-User] setuptools messing with sdists using numpy.distutils and Fortran libraries

Juan Luis Cano
I'm trying to build a Python package using some Fortran libraries, and
started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop". However, despite being stated that
one should just import setuptools before numpy.distutils imports:

http://mail.scipy.org/pipermail/numpy-discussion/2013-September/067784.html

I found that "setup.py sdist" works differently whether setuptools has
been imported or not - i.e. not the same files get excluded or included.

In particular, I had problems with the .pyf files of a Fortran library I
created, included using config.add_extension. Without setuptools it
works, and the .pyf files get included, but with setuptools they are
missing. This results in failed installations later on.

In case you want the specific example, here is the diff between the
working and non-working setup.py:

https://github.com/Pybonacci/poliastro/compare/0.1.x...master#diff-29

The use of setuptools is not crucial for this project, but I'm
interested in knowing what's going on here.

Thanks in advance!

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: setuptools messing with sdists using numpy.distutils and Fortran libraries

ralfgommers



On Mon, Dec 23, 2013 at 7:46 PM, Juan Luis Cano <[hidden email]> wrote:
I'm trying to build a Python package using some Fortran libraries, and
started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop". However, despite being stated that
one should just import setuptools before numpy.distutils imports:

http://mail.scipy.org/pipermail/numpy-discussion/2013-September/067784.html

I found that "setup.py sdist" works differently whether setuptools has
been imported or not - i.e. not the same files get excluded or included.

In particular, I had problems with the .pyf files of a Fortran library I
created, included using config.add_extension. Without setuptools it
works, and the .pyf files get included, but with setuptools they are
missing. This results in failed installations later on.

You could special case "develop" in this manner to not have setuptools mess up other commands: https://github.com/scipy/scipy/blob/master/setup.py#L205
 
In case you want the specific example, here is the diff between the
working and non-working setup.py:

https://github.com/Pybonacci/poliastro/compare/0.1.x...master#diff-29

The use of setuptools is not crucial for this project, but I'm
interested in knowing what's going on here.
 
Both setuptools and numpy.distutils monkeypatch the behavior of distutils commands, so there's very little logic to what happens. Different commands typically break in different ways. You can try to debug it to understand but if you value your sanity, you should try to avoid that:)

Cheers,
Ralf


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: setuptools messing with sdists using numpy.distutils and Fortran libraries

Juan Luis Cano
On 12/23/2013 09:39 PM, Ralf Gommers wrote:



On Mon, Dec 23, 2013 at 7:46 PM, Juan Luis Cano <[hidden email]> wrote:
I'm trying to build a Python package using some Fortran libraries, and
started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop". However, despite being stated that
one should just import setuptools before numpy.distutils imports:

http://mail.scipy.org/pipermail/numpy-discussion/2013-September/067784.html

I found that "setup.py sdist" works differently whether setuptools has
been imported or not - i.e. not the same files get excluded or included.

In particular, I had problems with the .pyf files of a Fortran library I
created, included using config.add_extension. Without setuptools it
works, and the .pyf files get included, but with setuptools they are
missing. This results in failed installations later on.

You could special case "develop" in this manner to not have setuptools mess up other commands: https://github.com/scipy/scipy/blob/master/setup.py#L205
 
In case you want the specific example, here is the diff between the
working and non-working setup.py:

https://github.com/Pybonacci/poliastro/compare/0.1.x...master#diff-29

The use of setuptools is not crucial for this project, but I'm
interested in knowing what's going on here.
 
Both setuptools and numpy.distutils monkeypatch the behavior of distutils commands, so there's very little logic to what happens. Different commands typically break in different ways. You can try to debug it to understand but if you value your sanity, you should try to avoid that:)

Yeah, at least for today I value it :) Thank you very much!

Cheers

Juan Luis

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: setuptools messing with sdists using numpy.distutils and Fortran libraries

Thomas Kluyver-2
In reply to this post by Juan Luis Cano
On 23 December 2013 18:46, Juan Luis Cano <[hidden email]> wrote:
and started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop".

For IPython, we actually went down the rabbit hole and added a 'symlink' command which is like 'develop', but doesn't use setuptools. We'd found that using 'develop' with Python 2 and 3 at the same time created conflicts in the entry points, so we rewrote it to use simple launcher scripts. The scripts are installed normally, only the package is symlinked into site-packages. If you install after symlinking, though, it tries to install into the source tree, so we now have an 'unsymlink' command as well.

The implementation is not supposed to be generic, but if anyone else wants to take a look, the code is here:
https://github.com/ipython/ipython/blob/master/setupbase.py#L385

Thomas

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: setuptools messing with sdists using numpy.distutils and Fortran libraries

josef.pktd
In reply to this post by Juan Luis Cano



On Mon, Dec 23, 2013 at 1:46 PM, Juan Luis Cano <[hidden email]> wrote:
I'm trying to build a Python package using some Fortran libraries, and
started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop". However, despite being stated that
one should just import setuptools before numpy.distutils imports:

http://mail.scipy.org/pipermail/numpy-discussion/2013-September/067784.html

I found that "setup.py sdist" works differently whether setuptools has
been imported or not - i.e. not the same files get excluded or included.

In particular, I had problems with the .pyf files of a Fortran library I
created, included using config.add_extension. Without setuptools it
works, and the .pyf files get included, but with setuptools they are
missing. This results in failed installations later on.

I don't have much idea about including fortran.

But did you try to include *.pyf in MANIFEST.in? 
That's in my experience often a source of missing or extra files being included in the sdist or installed files

Josef

 

In case you want the specific example, here is the diff between the
working and non-working setup.py:

https://github.com/Pybonacci/poliastro/compare/0.1.x...master#diff-29

The use of setuptools is not crucial for this project, but I'm
interested in knowing what's going on here.

Thanks in advance!

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: setuptools messing with sdists using numpy.distutils and Fortran libraries

Juan Luis Cano
On 12/25/2013 06:21 PM, [hidden email] wrote:



On Mon, Dec 23, 2013 at 1:46 PM, Juan Luis Cano <[hidden email]> wrote:
I'm trying to build a Python package using some Fortran libraries, and
started using also setuptools at a certain point because I wanted to
take advantage of "setup.py develop". However, despite being stated that
one should just import setuptools before numpy.distutils imports:

http://mail.scipy.org/pipermail/numpy-discussion/2013-September/067784.html

I found that "setup.py sdist" works differently whether setuptools has
been imported or not - i.e. not the same files get excluded or included.

In particular, I had problems with the .pyf files of a Fortran library I
created, included using config.add_extension. Without setuptools it
works, and the .pyf files get included, but with setuptools they are
missing. This results in failed installations later on.

I don't have much idea about including fortran.

But did you try to include *.pyf in MANIFEST.in? 
That's in my experience often a source of missing or extra files being included in the sdist or installed files

Josef


I didn't, because I found it weird to specify these files in two different places (setup.py and MANIFEST.in). I just checked and it solves the issue. Good to know!

Juan Luis

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user