How to crawl entire Wikipedia?

java web-crawler wikipedia websphinx
Mr CooL picture Mr CooL · Feb 22, 2010 · Viewed 18k times · Source

I've tried WebSphinx application.

I realize if I put wikipedia.org as the starting URL, it will not crawl further.

Hence, how to actually crawl the entire Wikipedia? Can anyone gimme some guidelines? Do I need to specifically go and find those URLs and put multiple starting URLs?

Anyone has suggestions of good website with the tutorial on usng WebSphinx's API?

Answer

Andrew picture Andrew · Feb 22, 2010

If your goal is to crawl all of Wikipedia, you might want to look at the available database dumps. See http://download.wikimedia.org/.

Related questions

How to get a web page's source code from Java

I just want to retrieve any web page's source code from Java. I found lots of solutions so far, but I couldn't find any code that works for all the links below: http://www.cumhuriyet.com.tr?hn=298710 http://www.…

Read More
Java Web Crawler Libraries

I wanted to make a Java based web crawler for an experiment. I heard that making a Web Crawler in Java was the way to go if this is your first time. However, I have two important questions. How will …

Read More
Syntax error, insert "... VariableDeclaratorId" to complete FormalParameterList

I am facing some issues with this code: import edu.uci.ics.crawler4j.crawler.CrawlConfig; import edu.uci.ics.crawler4j.crawler.CrawlController; import edu.uci.ics.crawler4j.fetcher.PageFetcher; import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig; …

Read More