Free Large datasets to experiment with Hadoop

resources hadoop opendata
Sundar picture Sundar · Apr 20, 2010 · Viewed 41k times · Source

Do you know any large dataset to experiment with Hadoop which is free/low cost? Any pointers/links related are appreciated.

Preference:

  • At least one GB of data.

  • Production log data of webserver.

Few of them which I found so far:

  1. Wikipedia dump

  2. http://wiki.freebase.com/wiki/Data_dumps

  3. http://aws.amazon.com/publicdatasets/

Also can we run our own crawler to gather data from sites e.g. Wikipedia? Any pointers on how to do this is appreciated as well.

Answer

Binary Nerd picture Binary Nerd · Apr 23, 2010

Few points about your question regarding crawling and wikipedia.

You have linked to the wikipedia data dumps and you can use the Cloud9 project from UMD to work with this data in Hadoop.

They have a page on this: Working with Wikipedia

Another datasource to add to the list is:

  • ClueWeb09 - 1 billion webpages collected between Jan and Feb 09. 5TB Compressed.

Using a crawler to generate data should be posted in a separate question to one about Hadoop/MapReduce I would say.

Related questions

How to change queue of currently running hadoop job?

My hadoop job was running over 10 hours but since I put it in wrong queue, the containers are kept getting killed by the scheduler. How do I change the queue of currently running hadoop job without restarting it? Thank you

Read More
Android: How do I get string from resources using its name?

I would like to have 2 languages for the UI and separate string values for them in my resource file res\values\strings.xml: <string name="tab_Books_en">Books</string> <string name="tab_Quotes_en"&…

Read More
Change app language programmatically in Android

Is it possible to change the language of an app programmatically while still using Android resources? If not, is it possible to request a resource in an specific language? I would like to let the user change the language of …

Read More