Fetch a Wikipedia article with Python

python urllib2 user-agent wikipedia http-status-code-403
dkp picture dkp · Sep 23, 2008 · Viewed 26.4k times · Source

I try to fetch a Wikipedia article with Python's urllib:

f = urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes")           
s = f.read()
f.close()

However instead of the html page I get the following response: Error - Wikimedia Foundation:

Request: GET http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes, from 192.35.17.11 via knsq1.knams.wikimedia.org (squid/2.6.STABLE21) to ()
Error: ERR_ACCESS_DENIED, errno [No Error] at Tue, 23 Sep 2008 09:09:08 GMT

Wikipedia seems to block request which are not from a standard browser.

Anybody know how to work around this?

Answer

Florian Bösch picture Florian Bösch · Sep 23, 2008

You need to use the urllib2 that superseedes urllib in the python std library in order to change the user agent.

Straight from the examples

import urllib2
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
infile = opener.open('http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes')
page = infile.read()

Related questions

Changing user agent on urllib2.urlopen

How can I download a webpage with a user agent other than the default one on urllib2.urlopen?

Read More
Import error: No module name urllib2

Here's my code: import urllib2.request response = urllib2.urlopen("http://www.google.com") html = response.read() print(html) Any help?

Read More
How to download image using requests

I'm trying to download and save an image from the web using python's requests module. Here is the (working) code I used: img = urllib2.urlopen(settings.STATICMAP_URL.format(**data)) with open(path, 'w') as f: f.write(img.read()) …

Read More