[SciPy-User] Install Scipy with Anaconda's MKL libraries

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I reinstalled Microsoft Visual C++ 14.0. Then following various advice on the web, I copied rc.exe and rcdll.exe from 

C:\Program Files (x86)\Windows Kits\8.1\bin\x64

into 

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64

It seems that MSVC++ 14.0 and that weird file copying is necessary for compiling Numpy. I can now install Numpy from source using python setup.py install. I then copied the site.cfg into every numpy/distutils directory I could find.

However, compiling Scipy gives a LNK2001 linking error with some _gfortran symbols:

mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_st_write
mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_st_write_done
mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_stop_numeric_f08
mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_transfer_character_write
mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_transfer_integer_write
mach.lib(d1mach.o) : error LNK2001: unresolved external symbol _gfortran_stop_string

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Goldsmith
In reply to this post by David Hagen
Hi, David.  You (re-?)learned a frustrating lesson: when testing the hypothesis that an installation is "in the way," before/instead of uninstalling, rename/move.

Now that i've moralized, with evidence of a "bug in the doc", my belief is that you will be asked to file a ticket (doc bugs are considered just as legitimate bugs as code bugs).  The only exception to this i can see is if someone else chimes in with a reasonable explanation as to why that's a "feature," not a bug, but in this instance, given what you say, that doesn't seem likely.

DLG 

On Sat, May 6, 2017 at 7:50 AM David Hagen <[hidden email]> wrote:
I noticed that I had two versions of Visual Studio installed: Microsoft Visual C++ 14.0 and Microsoft Visual Studio 2017. I thought that this might be confusing the installer, so I uninstalled the older Microsoft Visual C++ 14.0. However, the Numpy installation now gives a new error:

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++

This seems to make it clear that uninstalling was a mistake, but the Numpy build instructions say to specifically get the latest version of Visual Studio.
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov
1. The documentation definitely needs improvements.

The difference here is that CPython requirements for MSVC compilers from Visual Studio are not the same as ifort requirements for Visual Studio. 
For example MSVC compiler for Python 2.7 (aka.ms/vcpython27) is likely not enough to install ifort.
So 2.7 - MSVC 2008, 3.4 - MSVC 2010, 3.5-3.6 MSVC 2015/2017: https://wiki.python.org/moin/WindowsCompilers


2. VS 2017 is supported by setuptools 34.4.0+

3. Anyway I followed Intel article about scipy and ifort using VS 2015 with C++ compiler installed. In python 3.5, 64-bit both scipy and numpy compile without any issues even with beta 2018 ifort compiler. Don't forget to use the command prompt provided by Intel, Step 3 - Configuration:










On Sat, May 6, 2017 at 10:25 AM, David Goldsmith <[hidden email]> wrote:
Hi, David.  You (re-?)learned a frustrating lesson: when testing the hypothesis that an installation is "in the way," before/instead of uninstalling, rename/move.

Now that i've moralized, with evidence of a "bug in the doc", my belief is that you will be asked to file a ticket (doc bugs are considered just as legitimate bugs as code bugs).  The only exception to this i can see is if someone else chimes in with a reasonable explanation as to why that's a "feature," not a bug, but in this instance, given what you say, that doesn't seem likely.

DLG 

On Sat, May 6, 2017 at 7:50 AM David Hagen <[hidden email]> wrote:
I noticed that I had two versions of Visual Studio installed: Microsoft Visual C++ 14.0 and Microsoft Visual Studio 2017. I thought that this might be confusing the installer, so I uninstalled the older Microsoft Visual C++ 14.0. However, the Numpy installation now gives a new error:

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++

This seems to make it clear that uninstalling was a mistake, but the Numpy build instructions say to specifically get the latest version of Visual Studio.
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
> The difference here is that CPython requirements for MSVC compilers from Visual Studio are not the same as ifort requirements for Visual Studio.

I currently have Python 3.6.1. I have both Visual Studio C++ 14.0 (I think this is the same thing as VS2015) and Visual Studio 2017 installed. I have Parallel Studio XE 2018 Beta installed. I am not sure if that is all compatible because 2018 Beta isn't on there. Should I uninstall VS 2017?

> VS 2017 is supported by setuptools 34.4.0+

Should I upgrade from 27.2.0, which for whatever reason is the version that conda installs with Python 3.6.1?

> Anyway I followed Intel article about scipy and ifort using VS 2015 with C++ compiler installed.

I am able to run plain python setup.py install on the latest version of Numpy and it installs fine. (Doing the same with Scipy errors out with missing symbols described previously.) However, running the install command given for Intel64 in the link it fails. I followed everything except for recompile Python. It gets to here:

running config
running build_clib
running build_src
build_src
building py_modules sources
building library "npymath" sources
Could not locate executable icc
Could not locate executable ecc

And then throws a NoneType is not iterable exception:

...
File "c:\numpy-1.12.1\numpy\distutils\msvc9compiler.py", line 53, in initialize
  os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
File "c:\numpy-1.12.1\numpy\distutils\msvc9compiler.py", line 32, in _merge
  if new in old:
TypeError: argument of type 'NoneType' is not iterable

This happen both with and without modifying the compile options. It looks like it can't find icc, but without a real error, it is difficult to tell if that was a notify, warning, or error.

It installs with just python setup.py install. But if I do that is that the reason I get symbol errors trying to install Scipy? Is it just managing to muddle through with one of my other compilers but not be linkable by Scipy? I have Intel C++ Compiler installed as part of the Parallel Studio install.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I uninstalled all versions of MSVS and installed Visual Studio 2015 update 3. This did not fix anything, but at least I ruled out conflicts there.

To get past the NoneType error from my previous message I had to fix a bug in Numpy; namely, in the file

numpy\distutils\msvc9compiler.py

the _merge function fails if old is None. I merely swap the two `if` statements and it got past. I will submit a pull request.

Then the installer couldn't find icl.exe. I also had to add this to my PATH environment variable:

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018.0.065\windows\bin\intel64

Then I got this error

LINK : fatal error LNK1104: cannot open file 'libiomp5md.lib'

which I tracked down to MPI. The instructions here had told me to add several compiler options in numpy/distutils/intelccompiler.py, one of which was openmp. Once I reverted the compiler options to their original state, this error went away.

Then I got this error

LINK : fatal error LNK1104: cannot open file 'libmmd.lib'

I tracked this down to the compiler libraries not be on the library path so I added this to the LIB environment variable:

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018.0.065\windows\compiler\lib\intel64

Finally Numpy installed and imports.

But while it was nice to learn how to install Numpy from source, the main goal was to install Scipy from source. I thought that installing Numpy from source would fix my earlier problems with unresolved external _gfortran symbols. Alas, I still get this error while installing Scipy:

building 'scipy.integrate._quadpack' extension
compiling C sources
C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018.0.065\windows\bin\intel64\icl.exe /c /nologo /O3 /MD /W3 /Qstd=c99 -DSCIPY_MKL_H -DHAVE_CBLAS -
Iscipy\_lib\src -I"C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2018.0.019\compilers_and_libraries_2018\windows\mkl\include" -IC:\Anaconda3\envs\numpy
_source\lib\site-packages\numpy-1.12.1-py3.6-win-amd64.egg\numpy\core\include -IC:\Anaconda3\envs\numpy_source\include -IC:\Anaconda3\envs\numpy_source\include
/Tcscipy\integrate\_quadpackmodule.c /Fobuild\temp.win-amd64-3.6\Release\scipy\integrate\_quadpackmodule.obj
could not find library 'quadpack' in directories ['C:\\Program Files (x86)\\IntelSWTools\\parallel_studio_xe_2018.0.019\\compilers_and_libraries_2018\\windows\\
mkl\\lib\\intel64']
...
   Creating library build\temp.win-amd64-3.6\Release\scipy\integrate\_quadpack.c
p36-win_amd64.lib and object build\temp.win-amd64-3.6\Release\scipy\integrate\_quadpack.cp36-win_amd64.exp
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_st_write referenced in function d1mach_
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_st_write_done referenced in function d1mach_
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_stop_numeric_f08 referenced in function d1mach_
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_transfer_character_write referenced in function d1mach_
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_transfer_integer_write referenced in function d1mach_
mach.lib(d1mach.o) : error LNK2019: unresolved external symbol _gfortran_stop_string referenced in function d1mach_
build\lib.win-amd64-3.6\scipy\integrate\_quadpack.cp36-win_amd64.pyd : fatal error LNK1120: 6 unresolved externals

What does this even mean?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
Ok, I solved everything. Here's the solution to the unresolved external symbols error: DELETE THE BUILD FOLDER. If Scipy starts compiling with one compiler, it will used the cached binaries when linking later even if it is now using a different compiler. Deleting the build folder forces it to start over.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Goldsmith
Ah yes, the old "start the build fresh each time" problem--been there, done that.  Even if something says i shouldn't have to start over, i no longer trust that: always start with a fresh build.

DLG

On Sat, May 13, 2017 at 6:01 PM David Hagen <[hidden email]> wrote:
Ok, I solved everything. Here's the solution to the unresolved external symbols error: DELETE THE BUILD FOLDER. If Scipy starts compiling with one compiler, it will used the cached binaries when linking later even if it is now using a different compiler. Deleting the build folder forces it to start over.
_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
I spoke too soon. I "successfully" compiled Numpy and Scipy from master and both can be imported. However, some Scipy items don't import. It doesn't really matter what exactly I import. If I run scipy.test() everything dies. Can someone interpret this error message?

>>> from scipy.special import roots_legendre
Traceback (most recent call last):
  File "__init__.pxd", line 987, in numpy.import_array (scipy\special\_ufuncs_cx
x.cxx:3729)
RuntimeError: module compiled against API version 0xb but this version of numpy
is 0xa

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\numpy_scipy\lib\site-packages\scipy-1.0.0.dev0+unknown
-py3.6-win-amd64.egg\scipy\special\__init__.py", line 640, in <module>
    from ._ufuncs import *
  File "scipy\special\_ufuncs.pyx", line 1, in init scipy.special._ufuncs (scipy
\special\_ufuncs.c:39403)
  File "scipy\special\_ufuncs_extra_code_common.pxi", line 34, in init scipy.spe
cial._ufuncs_cxx (scipy\special\_ufuncs_cxx.cxx:4376)
  File "__init__.pxd", line 989, in numpy.import_array (scipy\special\_ufuncs_cx
x.cxx:3772)
ImportError: numpy.core.multiarray failed to import

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

Denis Akhiyarov
It looks like you have multiple numpy versions installed. Did you uninstall numpy, scipy with both pip and conda before building and installing numpy and scipy from source? I presume that you also built numpy from clean source without cache?

Since you are using conda environments, make sure that you activate it before the build process and before launching python interpreter.

On Sun, May 14, 2017 at 4:37 PM, David Hagen <[hidden email]> wrote:
I spoke too soon. I "successfully" compiled Numpy and Scipy from master and both can be imported. However, some Scipy items don't import. It doesn't really matter what exactly I import. If I run scipy.test() everything dies. Can someone interpret this error message?

>>> from scipy.special import roots_legendre
Traceback (most recent call last):
  File "__init__.pxd", line 987, in numpy.import_array (scipy\special\_ufuncs_cx
x.cxx:3729)
RuntimeError: module compiled against API version 0xb but this version of numpy
is 0xa

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\numpy_scipy\lib\site-packages\scipy-1.0.0.dev0+unknown
-py3.6-win-amd64.egg\scipy\special\__init__.py", line 640, in <module>
    from ._ufuncs import *
  File "scipy\special\_ufuncs.pyx", line 1, in init scipy.special._ufuncs (scipy
\special\_ufuncs.c:39403)
  File "scipy\special\_ufuncs_extra_code_common.pxi", line 34, in init scipy.spe
cial._ufuncs_cxx (scipy\special\_ufuncs_cxx.cxx:4376)
  File "__init__.pxd", line 989, in numpy.import_array (scipy\special\_ufuncs_cx
x.cxx:3772)
ImportError: numpy.core.multiarray failed to import

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user



_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
It looks like you have multiple numpy versions installed. Did you uninstall numpy, scipy with both pip and conda before building and installing numpy and scipy from source? I presume that you also built numpy from clean source without cache?

I have numpy installed in a separate conda environment because I don't want to lose my production environment while playing with Numpy and Scipy master. I got this error after creating a fresh conda environment, activating it, conda installing python and cython in it, and then running setup.py on freshly downloaded numpy and scipy master folders. How can I check if my other Numpy copies are visible in the search path for modules? Is there a command I can run to diagnose the version compatibility of the numpy and scipy modules that are imported and where they are located?

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Install Scipy with Anaconda's MKL libraries

David Hagen
It looks like you have multiple numpy versions installed. Did you uninstall numpy, scipy with both pip and conda before building and installing numpy and scipy from source? I presume that you also built numpy from clean source without cache?

I can confirm it was another version of Numpy installed alongside. Anaconda had pulled it in when I installed matplotlib. Once I removed it and installed matplotlib from pip, then everything worked fine.

_______________________________________________
SciPy-User mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/scipy-user
12
Loading...