PyCurl installed but not found

python curl libcurl pycurl
majackson picture majackson · Sep 12, 2011 · Viewed 15k times · Source

I've been trying to install pycurl in a virtualenv with easy_install, and it appears to install correctly:

(xxx) $ easy_install pycurl
Searching for pycurl
Reading http://pypi.python.org/simple/pycurl/
Reading http://pycurl.sourceforge.net/
Reading http://pycurl.sourceforge.net/download/
Best match: pycurl 7.19.0
Downloading http://pycurl.sourceforge.net/download/pycurl-7.19.0.tar.gz
Processing pycurl-7.19.0.tar.gz
Running pycurl-7.19.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-B2W9dY/pycurl-7.19.0/egg-dist-tmp-RmIsVr
Using curl-config (libcurl 7.19.7)
src/pycurl.c:85:4: warning: #warning "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests"
src/pycurl.c: In function ‘do_multi_info_read’:
src/pycurl.c:2843: warning: call to ‘_curl_easy_getinfo_err_string’ declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info
src/pycurl.c: In function ‘multi_socket_callback’:
src/pycurl.c:2355: warning: call to ‘_curl_easy_getinfo_err_string’ declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info
In function ‘util_curl_unsetopt’,
    inlined from ‘do_curl_unsetopt’ at src/pycurl.c:1551:
src/pycurl.c:1476: warning: call to ‘_curl_easy_setopt_err_CURLSH’ declared with attribute warning: curl_easy_setopt expects a CURLSH* argument for this option
gcc (GCC) 4.4.4 20100726 (Red Hat 4.4.4-13)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

zip_safe flag not set; analyzing archive contents...
Adding pycurl 7.19.0 to easy-install.pth file

Installed /home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/pycurl-7.19.0-py2.7-linux-x86_64.egg
Processing dependencies for pycurl
Finished processing dependencies for pycurl

However, when trying to use, I get the following error:

(xxx) $ python
Python 2.7.1 (r271:86832, Sep 12 2011, 13:51:42) 
[GCC 4.4.4 20100726 (Red Hat 4.4.4-13)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pycurl

Despite the fact that I can clearly see a pycurl-7.19.0-py2.7-linux-x86_64.egg in my ~/.virtualenvs/xxx/lib/python2.7/site-packages directory.

Any suggestions on how to solve this problem would be much appreciated!

Distribution is Amazon Linux (EC2, 64 bit), Python is 2.7, installed from source. curl, libcurl and libcurl-dev are all supplied by the Amazon yum repos and are at version 7.19.7-26.20.

I can't install pycurl package provided by the Amazon repos because the distro-supplied version of python is 2.6, and my project runs on 2.7.

It is not a path issue, the contents of sys.path is as follows:

>>> import sys; sys.path
['', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/pygeoip-0.2.1-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/daryl_lib-1.8.4-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/xxx-1.8.4-py2.7.egg', '/home/ec2-user/dealutils-1.5.0-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/six-1.0.0-py2.7.egg', '/home/ec2-user/fm2c-1.3.0-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/affiliatewindow-1.3.0-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/icodes-1.3.0-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/meld3-0.6.7-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/tornado-1.2-py2.7.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/pycurl-7.19.0-py2.7-linux-x86_64.egg', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages', '/root', '/home/ec2-user/.virtualenvs/xxx/lib/python27.zip', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/plat-linux2', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/lib-tk', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/lib-old', '/home/ec2-user/.virtualenvs/xxx/lib/python2.7/lib-dynload', '/opt/python2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/opt/python2.7/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg', '/opt/python2.7/lib/python2.7', '/opt/python2.7/lib/python2.7/plat-linux2', '/opt/python2.7/lib/python2.7/lib-tk', '/opt/python2.7/lib/python2.7/site-packages']

Answer

majackson picture majackson · Sep 13, 2011

I have managed (with some help) so solve the problem. The problem is that after installation, the pycurl.so is not copied to the site-packages for this virtualenv.

When installing via pip -v install pycurl, the following output is given:

Downloading/unpacking pycurl
  Using version 7.19.0 (newest of versions: 7.19.0, 7.19.0, 7.18.2, 7.18.1, 7.16.4, 7.16.2.1, 7.16.2, 7.16.1, 7.15.5.1)
  Downloading pycurl-7.19.0.tar.gz (71Kb): 71Kb downloaded
  Running setup.py egg_info for package pycurl
        Using curl-config (libcurl 7.19.7)
    running egg_info
    creating pip-egg-info/pycurl.egg-info
    writing pip-egg-info/pycurl.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pycurl.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pycurl.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/pycurl.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/pycurl.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/pycurl.egg-info/SOURCES.txt'
Installing collected packages: pycurl
  Running setup.py install for pycurl
    Using curl-config (libcurl 7.19.7)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/curl
    copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
    running build_ext
    building 'pycurl' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/src
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_CURL_SSL=1 -I/opt/python2.7/include/python2.7 -c src/pycurl.c -o build/temp.linux-x86_64-2.7/src/pycurl.o
    src/pycurl.c:85:4: warning: #warning "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests"
    src/pycurl.c: In function ‘do_multi_info_read’:
    src/pycurl.c:2843: warning: call to ‘_curl_easy_getinfo_err_string’ declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info
    src/pycurl.c: In function ‘multi_socket_callback’:
    src/pycurl.c:2355: warning: call to ‘_curl_easy_getinfo_err_string’ declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info
    In function ‘util_curl_unsetopt’,
        inlined from ‘do_curl_unsetopt’ at src/pycurl.c:1551:
    src/pycurl.c:1476: warning: call to ‘_curl_easy_setopt_err_CURLSH’ declared with attribute warning: curl_easy_setopt expects a CURLSH* argument for this option
    gcc -pthread -shared build/temp.linux-x86_64-2.7/src/pycurl.o -L/opt/python2.7/lib -lcurl -lpython2.7 -o build/lib.linux-x86_64-2.7/pycurl.so unknown option: --static-libs Usage: curl-config [OPTION] Available values for OPTION include: --ca ca bundle install path --cc compiler --cflags pre-processor and compiler flags --checkfor [version] check for (lib)curl of the specified version --features newline separated list of enabled features --help display this help and exit --libs library linking information --prefix curl install prefix --protocols newline separated list of enabled protocols --version output version information --vernum output the version information as a number (hexadecimal)
    gcc (GCC) 4.4.4 20100726 (Red Hat 4.4.4-13)
    Copyright (C) 2010 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    running install_lib
    creating /home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/curl
    copying build/lib.linux-x86_64-2.7/curl/__init__.py -> /home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/curl
    byte-compiling /home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/curl/__init__.py to __init__.pyc
    running install_data
    copying ChangeLog -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying COPYING -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying COPYING2 -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying INSTALL -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying README -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying TODO -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl
    copying doc/curlshareobject.html -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/html
    copying doc/pycurl.html -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/html
    copying doc/curlobject.html -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/html
    copying doc/curlmultiobject.html -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/html
    copying doc/callbacks.html -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/html
    copying examples/retriever.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/file_upload.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/linksys.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/xmlrpc_curl.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/retriever-multi.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/sfquery.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying examples/basicfirst.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/examples
    copying tests/test.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_ftp.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_share.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi5.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi_timer.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_post.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_gtk.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_post3.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_debug.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi4.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_cb.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi_vs_thread.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_stringio.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_xmlrpc.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi_socket_select.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi6.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi_socket.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/util.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi3.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_internals.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_post2.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_memleak.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_multi2.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_socketopen.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    copying tests/test_getinfo.py -> /home/ec2-user/.virtualenvs/xxx/share/doc/pycurl/tests
    running install_egg_info
    running egg_info
    creating pycurl.egg-info
    writing pycurl.egg-info/PKG-INFO
    writing top-level names to pycurl.egg-info/top_level.txt
    writing dependency_links to pycurl.egg-info/dependency_links.txt
    writing manifest file 'pycurl.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pycurl.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pycurl.egg-info/SOURCES.txt'
    Copying pycurl.egg-info to /home/ec2-user/.virtualenvs/xxx/lib/python2.7/site-packages/pycurl-7.19.0-py2.7.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-lEWzoB-record/install-record.txt'
Successfully installed pycurl
Cleaning up...
  Removing temporary dir /home/ec2-user/.virtualenvs/xxx/build...

The key error here is: gcc -pthread -shared build/temp.linux-x86_64-2.7/src/pycurl.o -L/opt/python2.7/lib -lcurl -lpython2.7 -o build/lib.linux-x86_64-2.7/pycurl.so unknown option: --static-libs Usage: curl-config [OPTION] Available values for OPTION include: --ca ca bundle install path --cc compiler --cflags pre-processor and compiler flags --checkfor [version] check for (lib)curl of the specified version --features newline separated list of enabled features --help display this help and exit --libs library linking information --prefix curl install prefix --protocols newline separated list of enabled protocols --version output version information --vernum output the version information as a number (hexadecimal)

Which shows that curl-config is being called with the argument --static-libs, which it fails to recognise and instead prints it's 'usage' output. Most importantly, it is returning a non-error-code output which makes the compiler assume the pycurl.so compilation was successful.

This is described in more detail here

The solution to this problem was to modify the pycurl's setup.py and change line 101 to read --libs instead of --static-libs. This causes pycurl.so to be built and copied to the site-packages directory successfully.

Related questions

Trouble with pycurl.POSTFIELDS

I'm familiar with CURL in PHP but am using it for the first time in Python with pycurl. I keep getting the error: Exception Type: error Exception Value: (2, '') I have no idea what this could mean. Here is my …

Read More
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am getting error Expecting value: line 1 column 1 (char 0) when trying to decode JSON. The URL I use for the API call works fine in the browser, but gives this error when done through a curl request. The following is …

Read More
how to use python to execute a curl command

I want to execute a curl command in python. Usually, I just need enter the command in terminal and press return key. However, I don't know how it works in python. The command shows below: curl -d @request.json --header "…

Read More