Compile fortran module with f2py and Python 3.6 on Windows 10

windows-10 anaconda mingw python-3.6 f2py
dsholes picture dsholes · Feb 16, 2018 · Viewed 10k times · Source

I'm trying (and failing) to compile a fortran module (specifically igrf12.f from the BGS) using f2py and Python 3.6 on Windows 10. Python was installed using Anaconda 4.4.10.

My setup:

  • Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)] on win32
  • Windows 10 Enterprise (version 1703)
  • NumPy 1.14.0

I followed f2py directions from the SciPy documentation and a very helpful guide from Dr.Michael Hirsch. Dr.Hirsch has created the pyigrf12 module, but installation through pip failed for me, which is what initially sparked my interest in f2py.

I will outline a few of the methods I am using. Regardless of the method, I always begin by creating a *.pyf signature file using f2py igrf12.f -m pyigrf12 -h igrf12.pyf and adding intent(in/out) attributes appropriately.

  • Method 1: Use C:\MinGW and --compiler=mingw32
  • Method 2: Use C:\MinGW and --compiler=msvc
  • Method 3: use anaconda version of mingw and --compiler=mingw32
  • Method 4: use anaconda version of mingw and --compiler=msvc

Method 1:

For background, I have MinGW at C:\MinGW and I have added C:\MinGW\bin to my user Path. Unfortunately, I did not install this version of MinGW (I inherited this computer from a colleague), so I do not know where it was sourced. The gcc --version and gfortran --version is 5.3.0.

I run f2py -c igrf12.pyf igrf12.f --compiler=mingw32. This fails with this error message:

Building import library (arch=AMD64):
"C:\Users\Sholes\AppData\Local\Continuum\anaconda3\libs\libpython36.a" 
(from C:\Users\Sholes\AppData\Local\Continuum\anaconda3\python36.dll)
    objdump.exe: C:\Users\Sholes\AppData\Local\Continuum\anaconda3\python36.dll: File format not recognized

Traceback (most recent call last):
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\Scripts\\f2py.py", line 28, in <module>
    main()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\f2py\f2py2e.py", line 648, in main
    run_compile()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\f2py\f2py2e.py", line 633, in run_compile
    setup(ext_modules=[ext])
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\core.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\command\build.py", line 47, in run
    old_build.run(self)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\command\build_ext.py", line 117, in run
    force=self.force)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\ccompiler.py", line 733, in new_compiler
    compiler = klass(None, dry_run, force)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 104, in __init__
    build_import_library()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 416, in build_import_library
    return _build_import_library_amd64()
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 472, in _build_import_library_amd64
    generate_def(dll_file, def_file)
  File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 302, in generate_def
    raise ValueError("Symbol table not found")
ValueError: Symbol table not found

The issue seems to involve building libpython36.a from python36.dll.

After a quick google search, a suggestion on a github forum for pywafo was to use the msvc compiler instead of mingw32, leading to Method 2.

Method 2: For background, the files related to my msvc are being pulled from C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.25017\bin\HostX86\x64\. Not sure if that's helpful.

I run f2py -c igrf12.pyf igrf12.f --compiler=msvc. This produces two files:

  • pyigrf12.cp36-win_amd64.pyd in my current working directory and,
  • libigrf12.BMWM6WD5Y3O3UTOEQITBXCIICXVMBEZS.gfortran-win_amd64.dll in .\UNKNOWN\.libs\

When I try to import pyigrf12, first I receive ImportError: DLL load failed: The specified module could not be found. Using Dependency Walker, I receive a lot of errors:

f2py Dependency Walker Output

but the most obvious one to address is related to the libigrf12.BMWM6WD5Y3O3UTOEQITBXCIICXVMBEZS.gfortran-win_amd64.dll. I move this libigrf12 dll file to my current working directory, alongside pyigrf12.cp36-win_amd64.pyd.

Now when I try import pyigrf12, I receive ImportError: DLL load failed: %1 is not a valid Win32 application.. Some stackoverflow posts seem to indicate that this is a problem with a conflict between a 32-bit dll and 64-bit Python. Can anyone offer insight into this? After more searching, I decided to try to use the anaconda version of mingw and libpython.

Method 3:

I run conda install mingw libpython, and it installs mingw 4.7-1 and libpython 2.1-py36_0. I run f2py -c igrf12.pyf igrf12.f --compiler=mingw32, and it fails with the following error message:

Building msvcr library: 
"C:\Users\Sholes\AppData\Local\Continuum\anaconda3\libs\libvcruntime140.a" 
(from C:\Users\Sholes\AppData\Local\Continuum\anaconda3\vcruntime140.dll)
    objdump.exe: C:\Users\Sholes\AppData\Local\Continuum\anaconda3\vcruntime140.dll: File format not recognized
    Traceback (most recent call last):
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\Scripts\\f2py.py", line 28, in <module>
        main()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\f2py\f2py2e.py", line 648, in main
        run_compile()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\f2py\f2py2e.py", line 633, in run_compile
        setup(ext_modules=[ext])
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\core.py", line 169, in setup
        return old_setup(**new_attr)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\command\build.py", line 47, in run
        old_build.run(self)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\command\build_ext.py", line 117, in run
        force=self.force)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\ccompiler.py", line 733, in new_compiler
        compiler = klass(None, dry_run, force)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 107, in __init__
        msvcr_success = build_msvcr_library()
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 399, in build_msvcr_library
        generate_def(dll_file, def_file)
      File "C:\Users\Sholes\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 302, in generate_def
        raise ValueError("Symbol table not found")
    ValueError: Symbol table not found

Now the issue seems to relate to building libvcruntime140.a from vcruntime140.dll. Once again the dll file format is not recognized by objdump.exe.

Method 4:

My final attempt was to run f2py -c igrf12.pyf igrf12.f --compiler=msvc with the anaconda version of mingw installed. For this case, gfortran failed with this error:

   C:\Users\Sholes\AppData\Local\Continuum\anaconda3\Scripts\gfortran.bat -Wall -g -Wall -g -shared 
..\..\..\AppData\Local\Temp\tmpugo__0q9\Release\igrf12.o -Lc:\users\sholes\appdata\local\continuum\anaconda3\mingw\lib\gcc\x86_64-w64-mingw32\4.7.0 -LC:\Users\Sholes\AppData\Local\Continuum\anaconda3\libs -LC:\Users\Sholes\AppData\Local\Continuum\anaconda3\PCbuild\amd64 -o C:\Users\Sholes\AppData\Local\Temp\tmpugo__0q9\Release\extra-dll\libigrf12.75XJA5DX6DTO7YIZ7X6ZHJYTRDCCYQYR.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,C:\Users\Sholes\AppData\Local\Temp\tmpugo__0q9\Release\libigrf12.75XJA5DX6DTO7YIZ7X6ZHJYTRDCCYQYR.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64

gfortran.exe: error: unrecognized command line option '-mlong-double-64'

At this point, I just want to know if it is possible to create fortran extensions with my set up and f2py. I don't have any background compiling C or fortran extensions on Windows, and based on all of the questions related to issues with Python 3.6 scipy and numpy installation on Windows, it seems like this is a common problem without a simple solution.

Any feedback or insight would be greatly appreciated.

Answer

dsholes picture dsholes · Feb 20, 2018

Finally got this working.

Short version:

Make sure you use 64-bit compilers (triple check the build for mingw-w64) for 64-bit Python. Not as obvious as it sounds to an f2py newb on Windows.

Long version:

I uninstalled my existing copy of MinGW (I suspect it was a 32 bit version) and instead downloaded a specific 64-bit build of mingw-w64 7.2.0 from sourceforge, specifically x86_64-7.2.0-release-posix-seh-rt_v5-rev1.7z. This stackoverflow question was useful.

I unzipped and copied the "mingw64" folder into my C: drive (C:\mingw64). I added C:\mingw64\bin to my user Path.

I uninstalled the anaconda version of MinGW with conda uninstall mingw. Note, this is only necessary if you've previously installed MinGW using conda.

Upon running f2py -c igrf12.pyf igrf12.f --compiler=mingw32 (in same directory as igrf12.pyf, see Scipy Documentation for how to generate *.pyf file), pyigrf12.cp36-win_amd64.pyd is created without any errors. I can finally import pyigrf12 successfully and access the underlying Fortran subroutines (e.g. igrf12syn).

Note, I can also run f2py -c igrf12.pyf igrf12.f --compiler=msvc successfully, but then I have to manually copy and paste the libigrf12....gfortran-win_amd64.dll (generated in .\UNKNOWN\.libs\) into the same directory as pyigrf12.cp36-win_amd64.pyd to avoid ImportError: DLL load failed: The specified module could not be found. mentioned in Method 2 of my question.

Just to re-iterate: Make sure C:\mingw64\bin is added to your path!

By the way, f2py was painless for me on macOS Sierra and Ubuntu. If the above still doesn't work for you, I recommend trying on Linux, macOS or Windows Subsystem for Linux.

Related questions

Conda command is not recognized on Windows 10

I installed Anaconda 4.4.0 (Python 3.6 version) on Windows 10 by following the instructions here: https://www.continuum.io/downloads. However, when I open the Command prompt window and try to write conda list I get the 'conda' command is not recognized... error. …

Read More
Anaconda Installed but Cannot Launch Navigator

Anaconda (listed as "Python 3.6.0 (Anaconda 4.3.1 64 bit)" ) is in my programs and features list, but there is seeming no Anaconda Navigator desktop app, as their seems to be no icon on my desktop and I am unable to search for it …

Read More
Python is in conda env, but has not been activated in Windows virtual env

I created a Windows (10) Python virtual environment (env3.7.3). When I open a cmd window activated in the virtual env, I get the following warning message when starting Python in the virtual env: (env3.7.3) C:\Users\redex\OneDrive\Documents\Education\Machine …

Read More