Getting HTML with Pycurl

python pycurl
Sinthet picture Sinthet · Jul 2, 2011 · Viewed 21.6k times · Source

I've been trying to retrieve a page of HTML using pycurl, so I can then parse it for relevant information using str.split and some for loops. I know Pycurl retrieves the HTML, since it prints it to the terminal, however, if I try to do something like

html = str(c.perform())

The variable will just hold a string which says "None".

How can I use pycurl to get the html, or redirect whatever it sends to the console so it can be used as a string as described above?

Thanks a lot to anyone who has any suggestions!

Answer

Corey Goldberg picture Corey Goldberg · Jul 2, 2011

this will send a request and store/print the response body:

from StringIO import StringIO    
import pycurl

url = 'http://www.google.com/'

storage = StringIO()
c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.WRITEFUNCTION, storage.write)
c.perform()
c.close()
content = storage.getvalue()
print content

if you want to store the response headers, use:

c.setopt(c.HEADERFUNCTION, storage.write)

Related questions

Why can't Python find shared objects that are in directories in sys.path?

I'm trying to import pycurl: $ python -c "import pycurl" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: libcurl.so.4: cannot open shared object file: No such file or directory Now, libcurl.so.4 is …

Read More
Execute curl command within a Python script

I am trying to execute a curl command within a python script. If I do it in the terminal, it looks like this: curl -X POST -d '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://…

Read More
Making an API call in Python with an API that requires a bearer token

Looking for some help with integrating a JSON API call into a Python program. I am looking to integrate the following API into a Python .py program to allow it to be called and the response to be printed. The …

Read More