[SciPy-User] Install Scipy with Anaconda's MKL libraries

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[SciPy-User] Install Scipy with Anaconda's MKL libraries

David Hagen
Anaconda now comes with free MKL libraries. I am trying to install the development branch of Scipy into an Anaconda environment, which means running the full Scipy install, not just the wheel. Scipy needs BLAS/LAPACK in order to compile, so the built in MKL seems to make sense. However, it seems that Scipy can't find the MKL libraries because it gives the "no lapack/blas resources found" error during setup.py when running a pip install. Is there an environment variable I need to set in order to tell Scipy's setup where my LAPACK library lives?

Here is exactly what I am running:

conda create --name scipy_master python=3.6 numpy cython
activate scipy_master
pip install https://github.com/scipy/scipy/tarball/master

I asked this on Stack Overflow last week but didn't get an answer.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov
Welcome to the world of pain with building scientific packages from source on Windows!

You need Fortran and C/C++ compilers on Windows to build scipy from source (and also enough disk space):


The mingwpy project mentioned in the wiki does not support Python 3.5+ due to UCRT.

mkl binaries linked with numpy and scipy and provided in Anaconda are not enough for building scipy.

If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.
But I had no luck with this, looks like this still requires dealing with include paths for blas and lapack.



On Sat, Apr 29, 2017 at 8:56 PM, David Hagen <[hidden email]> wrote:
Anaconda now comes with free MKL libraries. I am trying to install the development branch of Scipy into an Anaconda environment, which means running the full Scipy install, not just the wheel. Scipy needs BLAS/LAPACK in order to compile, so the built in MKL seems to make sense. However, it seems that Scipy can't find the MKL libraries because it gives the "no lapack/blas resources found" error during setup.py when running a pip install. Is there an environment variable I need to set in order to tell Scipy's setup where my LAPACK library lives?

Here is exactly what I am running:

conda create --name scipy_master python=3.6 numpy cython
activate scipy_master
pip install https://github.com/scipy/scipy/tarball/master

I asked this on Stack Overflow last week but didn't get an answer.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

laughingrice
In reply to this post by David Hagen
I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files. I just looked under the mac couldn’t find any of the header files.
That would make sense if Intel has not changed the licensing for MKL in the last few years.
This means that you cannot link new code to the MKL shipped with Anaconda but would need to get your own development license for MKL.


On Apr 29, 2017, at 21:56, David Hagen <[hidden email]> wrote:

Anaconda now comes with free MKL libraries. I am trying to install the development branch of Scipy into an Anaconda environment, which means running the full Scipy install, not just the wheel. Scipy needs BLAS/LAPACK in order to compile, so the built in MKL seems to make sense. However, it seems that Scipy can't find the MKL libraries because it gives the "no lapack/blas resources found" error during setup.py when running a pip install. Is there an environment variable I need to set in order to tell Scipy's setup where my LAPACK library lives?

Here is exactly what I am running:

conda create --name scipy_master python=3.6 numpy cython
activate scipy_master
pip install https://github.com/scipy/scipy/tarball/master

I asked this on Stack Overflow last week but didn't get an answer.
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
In reply to this post by David Hagen
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Matthieu Brucher-2
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

dhirschfeld
This post has NOT been accepted by the mailing list yet.
AFAIK it is impossible to compile Scipy with MinGW-w64 so you're on a hiding to nothing going down that route.

The only way to compile Scipy on Windows is with the Intel Fortran compiler and you can choose from either MSVC or Intel for the C compiler.

There was a project to try and get MinGW working properly on Windows but that appears to have stalled:
https://github.com/mingwpy/mingwpy.github.io


HTH,
Dave
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

William Heymann
In reply to this post by David Hagen
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
Is there a recipe for this combination? I installed MKL from that link and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure there's some environment variables that need to be set, but I don't know what they are.

On Tue, May 2, 2017 at 4:00 AM, William Heymann <[hidden email]> wrote:
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov

Did you follow these instructions?

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl


On Tue, May 2, 2017, 10:00 PM David Hagen <[hidden email]> wrote:
Is there a recipe for this combination? I installed MKL from that link and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure there's some environment variables that need to be set, but I don't know what they are.

On Tue, May 2, 2017 at 4:00 AM, William Heymann <[hidden email]> wrote:
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
So those instructions get me farther than I have gotten before.

Is this actually necessary?:

Note: You will need to rebuild Python from source files. This is due to the downloaded Python binary on Windows might be incompatible with the Visual Studio version you used. Otherwise you will encounter runtime crash when run numpy or script tests.

This is what I have done so far:

1) Download Scipy zip file from Github mater branch
2) Unzip file
3) Add a site.cfg file with these contents:
[mkl]
library_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\lib\intel64_win;C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\compiler\lib\intel64_win
include_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\include
mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
lapack_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md

4) cd into that directory
5) run: python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem install

It gets really far into the build but this is the error that results:

building extension "scipy.spatial.qhull" sources
creating build\src.win-amd64-3.6\scipy\spatial
Could not locate executable icc
Could not locate executable ecc
Traceback (most recent call last):
  File "setup.py", line 417, in <module>
    setup_package()
  File "setup.py", line 413, in setup_package
    setup(**metadata)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\core.py
", line 166, in setup
    return old_setup(**new_attr)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\core.py", line 148, in setu
p
    dist.run_commands()
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 955, in run_
commands
    self.run_command(cmd)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_clib.py", line 74, in run
    self.run_command('build_src')
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\cmd.py", line 313, in run_c
ommand
    self.distribution.run_command(command)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 148, in run
    self.build_sources()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 165, in build_sources
    self.build_extension_sources(ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 324, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 377, in generate_sources
    source = func(extension, build_dir)
  File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config
    if config_cmd.check_func('open_memstream', decl=True, call=True):
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 312, in check_func
    self._check_compiler()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 51, in _check_compiler
    self.compiler.initialize()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\intelcc
ompiler.py", line 86, in initialize
    MSVCCompiler.initialize(self, plat_name)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 53, in initialize
    os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 32, in _merge
    if new in old:
TypeError: argument of type 'NoneType' is not iterable

I notice that this is calling into Numpy. Do I have modify Numpy in order to compile Scipy? I skipped the Numpy specific steps in the instructions because I don't need the development version of Numpy.

On Tue, May 2, 2017 at 11:15 PM, Denis Akhiyarov <[hidden email]> wrote:

Did you follow these instructions?

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl


On Tue, May 2, 2017, 10:00 PM David Hagen <[hidden email]> wrote:
Is there a recipe for this combination? I installed MKL from that link and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure there's some environment variables that need to be set, but I don't know what they are.

On Tue, May 2, 2017 at 4:00 AM, William Heymann <[hidden email]> wrote:
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov
Ok, I followed instructions on this page:


Installed parallel_studio_xe_2016_update4 with mkl, ifort, and icc.

1. I compiled numpy (python setup.py install) from source with this site.cfg file:

[mkl]
include_dirs = C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\include
library_dirs = C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\lib\intel64;C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\compiler\lib\intel64
mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll
lapack_libs = mkl_lapack95_lp64


2. I replaced site.cfg file in ..\Lib\site-packages\numpy\distutils with the one above, since this is used by scipy in step 3.

3. I compiled scipy (python setup.py install) from source.

Compilation went without errors, both packages import without errors in python.


On Wed, May 3, 2017 at 8:26 PM, David Hagen <[hidden email]> wrote:
So those instructions get me farther than I have gotten before.

Is this actually necessary?:

Note: You will need to rebuild Python from source files. This is due to the downloaded Python binary on Windows might be incompatible with the Visual Studio version you used. Otherwise you will encounter runtime crash when run numpy or script tests.

This is what I have done so far:

1) Download Scipy zip file from Github mater branch
2) Unzip file
3) Add a site.cfg file with these contents:
[mkl]
library_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\lib\intel64_win;C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\compiler\lib\intel64_win
include_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\include
mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
lapack_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md

4) cd into that directory
5) run: python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem install

It gets really far into the build but this is the error that results:

building extension "scipy.spatial.qhull" sources
creating build\src.win-amd64-3.6\scipy\spatial
Could not locate executable icc
Could not locate executable ecc
Traceback (most recent call last):
  File "setup.py", line 417, in <module>
    setup_package()
  File "setup.py", line 413, in setup_package
    setup(**metadata)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\core.py
", line 166, in setup
    return old_setup(**new_attr)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\core.py", line 148, in setu
p
    dist.run_commands()
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 955, in run_
commands
    self.run_command(cmd)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_clib.py", line 74, in run
    self.run_command('build_src')
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\cmd.py", line 313, in run_c
ommand
    self.distribution.run_command(command)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 148, in run
    self.build_sources()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 165, in build_sources
    self.build_extension_sources(ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 324, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 377, in generate_sources
    source = func(extension, build_dir)
  File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config
    if config_cmd.check_func('open_memstream', decl=True, call=True):
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 312, in check_func
    self._check_compiler()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 51, in _check_compiler
    self.compiler.initialize()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\intelcc
ompiler.py", line 86, in initialize
    MSVCCompiler.initialize(self, plat_name)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 53, in initialize
    os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 32, in _merge
    if new in old:
TypeError: argument of type 'NoneType' is not iterable

I notice that this is calling into Numpy. Do I have modify Numpy in order to compile Scipy? I skipped the Numpy specific steps in the instructions because I don't need the development version of Numpy.

On Tue, May 2, 2017 at 11:15 PM, Denis Akhiyarov <[hidden email]> wrote:

Did you follow these instructions?

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl


On Tue, May 2, 2017, 10:00 PM David Hagen <[hidden email]> wrote:
Is there a recipe for this combination? I installed MKL from that link and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure there's some environment variables that need to be set, but I don't know what they are.

On Tue, May 2, 2017 at 4:00 AM, William Heymann <[hidden email]> wrote:
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
Am I correct that there is no free version of ifort and icc? Will this break after the 30-trial ends?

On Thu, May 4, 2017 at 2:58 AM, Denis Akhiyarov <[hidden email]> wrote:
Ok, I followed instructions on this page:


Installed parallel_studio_xe_2016_update4 with mkl, ifort, and icc.

1. I compiled numpy (python setup.py install) from source with this site.cfg file:

[mkl]
include_dirs = C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\include
library_dirs = C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\lib\intel64;C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\compiler\lib\intel64
mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll
lapack_libs = mkl_lapack95_lp64


2. I replaced site.cfg file in ..\Lib\site-packages\numpy\distutils with the one above, since this is used by scipy in step 3.

3. I compiled scipy (python setup.py install) from source.

Compilation went without errors, both packages import without errors in python.


On Wed, May 3, 2017 at 8:26 PM, David Hagen <[hidden email]> wrote:
So those instructions get me farther than I have gotten before.

Is this actually necessary?:

Note: You will need to rebuild Python from source files. This is due to the downloaded Python binary on Windows might be incompatible with the Visual Studio version you used. Otherwise you will encounter runtime crash when run numpy or script tests.

This is what I have done so far:

1) Download Scipy zip file from Github mater branch
2) Unzip file
3) Add a site.cfg file with these contents:
[mkl]
library_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\lib\intel64_win;C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\compiler\lib\intel64_win
include_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\include
mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
lapack_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md

4) cd into that directory
5) run: python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem install

It gets really far into the build but this is the error that results:

building extension "scipy.spatial.qhull" sources
creating build\src.win-amd64-3.6\scipy\spatial
Could not locate executable icc
Could not locate executable ecc
Traceback (most recent call last):
  File "setup.py", line 417, in <module>
    setup_package()
  File "setup.py", line 413, in setup_package
    setup(**metadata)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\core.py
", line 166, in setup
    return old_setup(**new_attr)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\core.py", line 148, in setu
p
    dist.run_commands()
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 955, in run_
commands
    self.run_command(cmd)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_clib.py", line 74, in run
    self.run_command('build_src')
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\cmd.py", line 313, in run_c
ommand
    self.distribution.run_command(command)
  File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974, in run_
command
    cmd_obj.run()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 148, in run
    self.build_sources()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 165, in build_sources
    self.build_extension_sources(ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 324, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\build_src.py", line 377, in generate_sources
    source = func(extension, build_dir)
  File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config
    if config_cmd.check_func('open_memstream', decl=True, call=True):
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 312, in check_func
    self._check_compiler()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
\config.py", line 51, in _check_compiler
    self.compiler.initialize()
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\intelcc
ompiler.py", line 86, in initialize
    MSVCCompiler.initialize(self, plat_name)
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 53, in initialize
    os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
  File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
mpiler.py", line 32, in _merge
    if new in old:
TypeError: argument of type 'NoneType' is not iterable

I notice that this is calling into Numpy. Do I have modify Numpy in order to compile Scipy? I skipped the Numpy specific steps in the instructions because I don't need the development version of Numpy.

On Tue, May 2, 2017 at 11:15 PM, Denis Akhiyarov <[hidden email]> wrote:

Did you follow these instructions?

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl


On Tue, May 2, 2017, 10:00 PM David Hagen <[hidden email]> wrote:
Is there a recipe for this combination? I installed MKL from that link and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure there's some environment variables that need to be set, but I don't know what they are.

On Tue, May 2, 2017 at 4:00 AM, William Heymann <[hidden email]> wrote:
Intel has made MKL, TBB, and a few other things completely free to use, even in a commercial project. Visual Studio is also free unless you are a very large company.


I have been using that for other projects without any problems and compiling with Visual Studio has been very easy.

On Tue, May 2, 2017 at 1:40 AM, David Hagen <[hidden email]> wrote:
I'll try to stick with MinGW-w64 for now, but I don't even get to the compilation phase. If I install lapack and blas from conda-forge, it still says that lapack/blas are not found, but you indicated that I need to set some paths. Are there instructions for this? I have no idea what environment variables to set in order to tell Scipy to use these packages.

On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <[hidden email]> wrote:

I still suggest Intel+MSVC compilers, since you can use trial version or request license for open-source projects from Intel. This is what Anaconda team is using. Also this is what Christoph Gohlke wheels are based on:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy

If you end up with m2w64, here is lapack for conda, you may still have to modify paths:

https://anaconda.org/conda-forge/lapack

And blas:

https://anaconda.org/search?q=Blas


On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <[hidden email]> wrote:
Why do you want to pay Intel? You can install the MKL and develop with it, no sweat.

2017-04-30 22:41 GMT+01:00 David Hagen <[hidden email]>:
> Welcome to the world of pain with building scientific packages from source on Windows!

I am beginning to feel it.

> You need Fortran and C/C++ compilers on Windows to build scipy from source

I have MinGW-w64 installed, which seems to be the recommended method.

> I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.

I understand now. It looks like MKL is not the way to go unless I want to pay Intel.

> If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.

My only goal is to install and use Scipy master somewhere where it won't break my stable installation. I thought Anaconda would be a good place to start because once I activate an Anaconda environment, I should be able to treat like a normal Python installation and follow the normal install-from-source instructions. I went ahead and installed that m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I should change my question to: how do I install Scipy on Windows from source? Though when I search for this specifically on the web, the answer seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by Scipy. Do you know of a conda/pip package that provides ATLAS for building Scipy or another more suitable BLAS/LAPACK?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user




--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Stephen P. Molnar
On 05/04/2017 08:16 AM, David Hagen wrote:

> Am I correct that there is no free version of ifort and icc? Will this
> break after the 30-trial ends?
>
> On Thu, May 4, 2017 at 2:58 AM, Denis Akhiyarov
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ok, I followed instructions on this page:
>
>     https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows
>     <https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows>
>
>     Installed parallel_studio_xe_2016_update4 with mkl, ifort, and icc.
>
>     1. I compiled numpy (python setup.py install) from source with this
>     site.cfg file:
>
>     [mkl]
>     include_dirs = C:\Program Files
>     (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\include
>     library_dirs = C:\Program Files
>     (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\lib\intel64;C:\Program
>     Files
>     (x86)\IntelSWTools\parallel_studio_xe_2016.4.062\compilers_and_libraries_2016\windows\compiler\lib\intel64
>     mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll
>     lapack_libs = mkl_lapack95_lp64
>
>
>     2. I replaced site.cfg file in ..\Lib\site-packages\numpy\distutils
>     with the one above, since this is used by scipy in step 3.
>
>     3. I compiled scipy (python setup.py install) from source.
>
>     Compilation went without errors, both packages import without errors
>     in python.
>
>
>     On Wed, May 3, 2017 at 8:26 PM, David Hagen <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         So those instructions get me farther than I have gotten before.
>
>         Is this actually necessary?:
>
>         Note: You will need to rebuild Python from source files. This is
>         due to the downloaded Python binary on Windows might
>         be incompatible with the Visual Studio version you used.
>         Otherwise you will encounter runtime crash when run numpy or
>         script tests.
>
>         This is what I have done so far:
>
>         1) Download Scipy zip file from Github mater branch
>         2) Unzip file
>         3) Add a site.cfg file with these contents:
>         [mkl]
>         library_dirs = C:\Program Files
>         (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\lib\intel64_win;C:\Program
>         Files
>         (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\compiler\lib\intel64_win
>         include_dirs = C:\Program Files
>         (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\windows\mkl\include
>         mkl_libs =
>         mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
>         lapack_libs =
>         mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
>
>         4) cd into that directory
>         5) run: python setup.py config --compiler=intelemw
>         --fcompiler=intelvem build_clib --compiler=intelemw
>         --fcompiler=intelvem build_ext --compiler=intelemw
>         --fcompiler=intelvem install
>
>         It gets really far into the build but this is the error that
>         results:
>
>         building extension "scipy.spatial.qhull" sources
>         creating build\src.win-amd64-3.6\scipy\spatial
>         Could not locate executable icc
>         Could not locate executable ecc
>         Traceback (most recent call last):
>            File "setup.py", line 417, in <module>
>              setup_package()
>            File "setup.py", line 413, in setup_package
>              setup(**metadata)
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\core.py
>         ", line 166, in setup
>              return old_setup(**new_attr)
>            File "C:\Anaconda3\envs\scipy_master\lib\distutils\core.py",
>         line 148, in setu
>         p
>              dist.run_commands()
>            File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py",
>         line 955, in run_
>         commands
>              self.run_command(cmd)
>            File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py",
>         line 974, in run_
>         command
>              cmd_obj.run()
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \build_clib.py", line 74, in run
>              self.run_command('build_src')
>            File "C:\Anaconda3\envs\scipy_master\lib\distutils\cmd.py",
>         line 313, in run_c
>         ommand
>              self.distribution.run_command(command)
>            File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py",
>         line 974, in run_
>         command
>              cmd_obj.run()
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \build_src.py", line 148, in run
>              self.build_sources()
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \build_src.py", line 165, in build_sources
>              self.build_extension_sources(ext)
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \build_src.py", line 324, in build_extension_sources
>              sources = self.generate_sources(sources, ext)
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \build_src.py", line 377, in generate_sources
>              source = func(extension, build_dir)
>            File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config
>              if config_cmd.check_func('open_memstream', decl=True,
>         call=True):
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \config.py", line 312, in check_func
>              self._check_compiler()
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\command
>         \config.py", line 51, in _check_compiler
>              self.compiler.initialize()
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\intelcc
>         ompiler.py", line 86, in initialize
>              MSVCCompiler.initialize(self, plat_name)
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
>         mpiler.py", line 53, in initialize
>              os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
>            File
>         "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\distutils\msvc9co
>         mpiler.py", line 32, in _merge
>              if new in old:
>         TypeError: argument of type 'NoneType' is not iterable
>
>         I notice that this is calling into Numpy. Do I have modify Numpy
>         in order to compile Scipy? I skipped the Numpy specific steps in
>         the instructions because I don't need the development version of
>         Numpy.
>
>         On Tue, May 2, 2017 at 11:15 PM, Denis Akhiyarov
>         <[hidden email] <mailto:[hidden email]>>
>         wrote:
>
>             Did you follow these instructions?
>
>             https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
>             <https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>
>
>
>             On Tue, May 2, 2017, 10:00 PM David Hagen <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>                 Is there a recipe for this combination? I installed MKL
>                 from that link and the latest Visual Studio. Scipy did
>                 not find MKL on its own. I'm sure there's some
>                 environment variables that need to be set, but I don't
>                 know what they are.
>
>                 On Tue, May 2, 2017 at 4:00 AM, William Heymann
>                 <[hidden email] <mailto:[hidden email]>> wrote:
>
>                     Intel has made MKL, TBB, and a few other things
>                     completely free to use, even in a commercial
>                     project. Visual Studio is also free unless you are a
>                     very large company.
>
>                     https://software.intel.com/en-us/articles/free-mkl
>                     <https://software.intel.com/en-us/articles/free-mkl>
>
>                     I have been using that for other projects without
>                     any problems and compiling with Visual Studio has
>                     been very easy.
>
>                     On Tue, May 2, 2017 at 1:40 AM, David Hagen
>                     <[hidden email] <mailto:[hidden email]>> wrote:
>
>                         I'll try to stick with MinGW-w64 for now, but I
>                         don't even get to the compilation phase. If I
>                         install lapack and blas from conda-forge, it
>                         still says that lapack/blas are not found, but
>                         you indicated that I need to set some paths. Are
>                         there instructions for this? I have no idea what
>                         environment variables to set in order to tell
>                         Scipy to use these packages.
>
>                         On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov
>                         <[hidden email]
>                         <mailto:[hidden email]>> wrote:
>
>                             I still suggest Intel+MSVC compilers, since
>                             you can use trial version or request license
>                             for open-source projects from Intel. This is
>                             what Anaconda team is using. Also this is
>                             what Christoph Gohlke wheels are based on:
>
>                             http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
>                             <http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy>
>
>                             If you end up with m2w64, here is lapack for
>                             conda, you may still have to modify paths:
>
>                             https://anaconda.org/conda-forge/lapack
>                             <https://anaconda.org/conda-forge/lapack>
>
>                             And blas:
>
>                             https://anaconda.org/search?q=Blas
>                             <https://anaconda.org/search?q=Blas>
>
>
>                             On Sun, Apr 30, 2017, 5:22 PM Matthieu
>                             Brucher <[hidden email]
>                             <mailto:[hidden email]>> wrote:
>
>                                 Why do you want to pay Intel? You can
>                                 install the MKL and develop with it, no
>                                 sweat.
>
>                                 2017-04-30 22:41 GMT+01:00 David Hagen
>                                 <[hidden email]
>                                 <mailto:[hidden email]>>:
>
>                                     > Welcome to the world of pain with building scientific packages from source on Windows!
>
>                                     I am beginning to feel it.
>
>                                     > You need Fortran and C/C++ compilers on Windows to build scipy from source
>
>                                     I have MinGW-w64 installed, which
>                                     seems to be the recommended method.
>
>                                     > I’m pretty sure that anaconda does not come with the development files for MKL, only the runtime files.
>
>                                     I understand now. It looks like MKL
>                                     is not the way to go unless I want
>                                     to pay Intel.
>
>                                     > If you don't need mkl and lapack/blas is good enough, then m2w64-toolchain from conda should have all necessary dependencies for building scipy.
>
>                                     My only goal is to install and use
>                                     Scipy master somewhere where it
>                                     won't break my stable installation.
>                                     I thought Anaconda would be a good
>                                     place to start because once I
>                                     activate an Anaconda environment, I
>                                     should be able to treat like a
>                                     normal Python installation and
>                                     follow the normal
>                                     install-from-source instructions. I
>                                     went ahead and installed that
>                                     m2w64-toolchain package, but it
>                                     still doesn't find any BLAS/LAPACK.
>                                     Maybe I should change my question
>                                     to: how do I install Scipy on
>                                     Windows from source? Though when I
>                                     search for this specifically on the
>                                     web, the answer seems to be
>                                     "Don't.". It seems that MinGW-w64
>                                     and ATLAS are recommended by Scipy.
>                                     Do you know of a conda/pip package
>                                     that provides ATLAS for building
>                                     Scipy or another more suitable
>                                     BLAS/LAPACK?
>
>                                     _______________________________________________
>                                     SciPy-User mailing list
>                                     [hidden email]
>                                     <mailto:[hidden email]>
>                                     https://mail.python.org/mailman/listinfo/scipy-user
>                                     <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>
>                                 --
>                                 Information System Engineer, Ph.D.
>                                 Blog: http://blog.audio-tk.com/
>                                 LinkedIn:
>                                 http://www.linkedin.com/in/matthieubrucher
>                                 <http://www.linkedin.com/in/matthieubrucher>
>                                 _______________________________________________
>                                 SciPy-User mailing list
>                                 [hidden email]
>                                 <mailto:[hidden email]>
>                                 https://mail.python.org/mailman/listinfo/scipy-user
>                                 <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>                             _______________________________________________
>                             SciPy-User mailing list
>                             [hidden email]
>                             <mailto:[hidden email]>
>                             https://mail.python.org/mailman/listinfo/scipy-user
>                             <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>                         _______________________________________________
>                         SciPy-User mailing list
>                         [hidden email] <mailto:[hidden email]>
>                         https://mail.python.org/mailman/listinfo/scipy-user
>                         <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>                     _______________________________________________
>                     SciPy-User mailing list
>                     [hidden email] <mailto:[hidden email]>
>                     https://mail.python.org/mailman/listinfo/scipy-user
>                     <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>                 _______________________________________________
>                 SciPy-User mailing list
>                 [hidden email] <mailto:[hidden email]>
>                 https://mail.python.org/mailman/listinfo/scipy-user
>                 <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>             _______________________________________________
>             SciPy-User mailing list
>             [hidden email] <mailto:[hidden email]>
>             https://mail.python.org/mailman/listinfo/scipy-user
>             <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>         _______________________________________________
>         SciPy-User mailing list
>         [hidden email] <mailto:[hidden email]>
>         https://mail.python.org/mailman/listinfo/scipy-user
>         <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>     _______________________________________________
>     SciPy-User mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/scipy-user
>     <https://mail.python.org/mailman/listinfo/scipy-user>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/scipy-us
No, you are not correct about ifort.  It's wstill available.  ayou just
won't receive support.

--
Stephen P. Molnar, Ph.D. Life is a fuzzy set
www.molecular-modeling.net Stochastic and multivariate
(614)312-7528 (c)
Skype: smolnar1
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
> No, you are not correct about ifort.  It's wstill available.  ayou just won't receive support.

I must be unable to Google correctly. I can find free MKL for everyone. I can find free ifort and icc for students. I can find free ifort and icc for Linux for open source developers. But I can't find free ifort and icc for Windows for open source developers. This is what I am looking at: https://software.intel.com/en-us/articles/free-ipsxe-tools-and-libraries

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov
You can sign-up for beta program which includes Windows ifort compiler. 


"The beta program officially ends June 22nd, 2017. The beta license provided will expire October 12, 2017."

On Thu, May 4, 2017 at 9:54 AM, David Hagen <[hidden email]> wrote:
> No, you are not correct about ifort.  It's wstill available.  ayou just won't receive support.

I must be unable to Google correctly. I can find free MKL for everyone. I can find free ifort and icc for students. I can find free ifort and icc for Linux for open source developers. But I can't find free ifort and icc for Windows for open source developers. This is what I am looking at: https://software.intel.com/en-us/articles/free-ipsxe-tools-and-libraries

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Stephen P. Molnar
In reply to this post by David Hagen
On 05/04/2017 10:54 AM, David Hagen wrote:

>  > No, you are not correct about ifort.  It's wstill available.  ayou
> just won't receive support.
>
> I must be unable to Google correctly. I can find free MKL for everyone.
> I can find free ifort and icc for students. I can find free ifort and
> icc for Linux for open source developers. But I can't find free ifort
> and icc for Windows for open source developers. This is what I am
> looking at:
> https://software.intel.com/en-us/articles/free-ipsxe-tools-and-libraries
>
>
> _______________________________________________
> SciPy-User mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/scipy-user
>

Check out:

https://software.intel.com/en-us/free-software

--
Stephen P. Molnar, Ph.D. Life is a fuzzy set
www.molecular-modeling.net Stochastic and multivariate
(614)312-7528 (c)
Skype: smolnar1
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I am following Stephen's instructions with the Beta distribution of Intel Parallel Studio. The compilation of Numpy fails with this error:

LINK : fatal error LNK1158: cannot run 'rc.exe'

It also has this message during setup:

customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl

and this:

Found executable C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe

Do I need to configure Numpy to use the Parallel Studio compilers rather than whatever else I may have installed?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I noticed that I had two versions of Visual Studio installed: Microsoft Visual C++ 14.0 and Microsoft Visual Studio 2017. I thought that this might be confusing the installer, so I uninstalled the older Microsoft Visual C++ 14.0. However, the Numpy installation now gives a new error:

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++

This seems to make it clear that uninstalling was a mistake, but the Numpy build instructions say to specifically get the latest version of Visual Studio.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
12
Loading...