Pyarrow does not install with python 3.7 (anaconda 5.3.0, windows x64 version)

techvslife picture techvslife · Oct 11, 2018 · Viewed 9.2k times · Source

I installed the 64-bit windows version of python 3.7 by installing anaconda 5.3.0. Then I tried installing pyarrow ("conda install pyarrow").

Anaconda ended up DOWNGRADING python to 3.6. And things did not work so well thereafter (weird error messages). I ended up having to uninstall Anaconda and re-installing it (had to uninstall since it does not do repair or re-install if the program folder is not empty).

That leaves me python 3.7 but no pyarrow. Is there another package that will give me parquet support with python and pandas? Or is there a way to get pyarrow to work with python 3.7?

UPDATE: As per Omri374's suggestion, I tried

pip install pyarrow

That doesn't solve my separate anaconda rollback to python 3.6 problem (i.e., when doing "conda install pyarrow"), but it does install pyarrow. Note that it gives the following output though--trying to update pip produced a rollback to python 3.6, so I don't recommend it:

twisted 18.7.0 requires PyHamcrest>=1.9.0, which is not installed.
Installing collected packages: pyarrow
Successfully installed pyarrow-0.11.0  
You are using pip version 10.0.1, however version 18.1 is available. 
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

This is the pip version that comes with 64-bit win anaconda 5.3.0 (python 3.7):

>conda list pip 
#Name Version Build Channel  
pip 10.0.1 py37_0

p.s. Since pyarrow seems to need the conda-forge channel, this is my channel list ("conda config --show channels"): channels:

  - conda-forge
  - anaconda-fusion
  - defaults  

Answer

techvslife picture techvslife · Oct 26, 2018

The latest pyarrow package version solves the problem. I can now install pyarrow using anaconda under python 3.7:

(base) C:\>conda install pyarrow
Solving environment: done

## Package Plan ##

  environment location: C:\Users\xxx
  added / updated specs:
    - pyarrow


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pyarrow-0.11.0             |   py37h33f27b4_0         1.5 MB
    libboost-1.67.0            |       hd9e427e_4        31.8 MB
    conda-4.5.11               |        py37_1000         654 KB  conda-forge
    gflags-2.2.1               |           vc14_0         283 KB  conda-forge
    lz4-c-1.8.3                |    he025d50_1000         254 KB  conda-forge
    xz-5.2.4                   |    h2fa13f4_1001         805 KB  conda-forge
    thrift-cpp-0.11.0          |           vc14_2         2.2 MB  conda-forge
    arrow-cpp-0.11.0           |   py37h8e05e8c_0         7.5 MB
    glog-0.3.5                 |       h6538335_1         130 KB
    zstd-1.3.3                 |           vc14_1         1.2 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        46.2 MB

The following NEW packages will be INSTALLED:

    arrow-cpp:  0.11.0-py37h8e05e8c_0
    gflags:     2.2.1-vc14_0          conda-forge [vc14]
    glog:       0.3.5-h6538335_1
    libboost:   1.67.0-hd9e427e_4
    lz4-c:      1.8.3-he025d50_1000   conda-forge
    pyarrow:    0.11.0-py37h33f27b4_0
    thrift-cpp: 0.11.0-vc14_2         conda-forge [vc14]
    xz:         5.2.4-h2fa13f4_1001   conda-forge
    zstd:       1.3.3-vc14_1          conda-forge [vc14]

The following packages will be UPDATED:

    conda:      4.5.11-py37_0                     --> 4.5.11-py37_1000 conda-forge

FURTHER UPDATE: I discovered that there is a problem affecting the conda-forge channel and the new version of python/anaconda. (Or there may be two problems.)

What seems to solve the problem for now is to have only "defaults" in the channels list, not conda-forge. ("conda config --show channels") (But that may be only because the "defaults" channel does not have some problem update package.)

The symptoms are these:

  1. Attempts to update anaconda trigger a rollback of python to 3.6:
The following packages will be DOWNGRADED: 

ipykernel:      4.10.0-py37_0             --> 4.9.0-py36_1
pyarrow:        0.11.1-py37h33f27b4_0     --> 0.8.0-py36_vc14_0       conda-forge [vc14]
python:         3.7.0-hea74fb7_0          --> 3.6.6-hea74fb7_0
  1. Attempts to update give an (uncorrectable) error on the "automat" package:
ERROR conda.core.link:_execute(502):
An error occurred while installing package 'conda-forge::automat-0.7.0-py_1'.
CondaError: Cannot link a source that does not exist.
C:\Users\xxx\anaconda3\Scripts\conda.exe 
Running conda clean --packages may resolve your problem. Attempting to roll back.

Rolling back transaction: Rolling back transaction: done 
CondaError: Cannot link a source that does not exist.
C:\Users\xxx\anaconda3\Scripts\conda.exe 
Running conda clean --packages may resolve your problem.