Extract the main article text from a Wikipedia page using Python

python parsing extract wikipedia
Paolo picture Paolo · Apr 28, 2014 · Viewed 7.2k times · Source

I've been searching for hours on how to extract the main text of a Wikipedia article, without all the links and references. I've tried wikitools, mwlib, BeautifulSoup and more. But I haven't really managed to.

Is there any easy and fast way for me to take the clear text (the actual article), and put it in a Python variable?

SOLUTION: Omid Raha solved it :)

Answer

Omid Raha picture Omid Raha · Apr 28, 2014

You can use this package, that is a python wrapper for Wikipedia API,

Here is a quick start.

First install it:

pip install wikipedia

Example:

import wikipedia
p = wikipedia.page("Python programming language")
print(p.url)
print(p.title)
content = p.content # Content of page.

Output:

http://en.wikipedia.org/wiki/Python_(programming_language)
Python (programming language)

Related questions

How to extract top-level domain name (TLD) from URL

how would you extract the domain name from a URL, excluding any subdomains? My initial simplistic attempt was: '.'.join(urlparse.urlparse(url).netloc.split('.')[-2:]) This works for http://www.foo.com, but not http://www.…

Read More
How do I parse a string to a float or int?

In Python, how can I parse a numeric string like "545.2222" to its corresponding float value, 545.2222? Or parse the string "31" to an integer, 31? I just want to know how to parse a float str to a float, and (separately) an int …

Read More
Why can't Python parse this JSON data?

I have this JSON in a file: { "maps": [ { "id": "blabla", "iscategorical": "0" }, { "id": "blabla", "iscategorical": "0" } ], "masks": [ "id": "valore" ], "om_points": "value", "parameters": [ "id": "valore" ] } I wrote this script to print all of the JSON data: import json from pprint import pprint …

Read More