Obtaining reddit data

web-scraping reddit
Budhapest picture Budhapest · Jan 14, 2013 · Viewed 24k times · Source

I am interested in obtaining data from different reddit subreddits. Does anyone know if there is a reddit/other api similar like twitter does to crawl all the pages?

Answer

bboe picture bboe · Jan 14, 2013

Yes, reddit has an API that can be used for a variety of purposes such as data collection, automatic commenting bots, or even to assist in subreddit moderation.

There are a few places to discover information on reddit's API:

  • github reddit wiki -- provides the overview and rules for using reddit's API (follow the rules)
  • automatically generated API docs -- provides information on the requests needed to access most of the API endpoints
  • /r/redditdev -- the reddit community dedicated to answering questions both about reddit's source code and about reddit's API

If there is a particular programming language you are already familiar with, you should check out the existing set of API wrappers for various languages. Despite my bias (I am the package maintainer) I am quite certain PRAW, for python, has support for the largest number of reddit API features.

Related questions

HTTP Error 504: Gateway Time-out when trying to read a reddit comments post

I am encountering an error when trying to get a comments' http from reddit. This has happened to various URLs (not all of them with special characters) and this is one of them. In one hour time frame, there may …

Read More
How can I get the Google cache age of any URL or web page?

In my project I need the Google cache age to be added as important information. I tried to search sources for the Google cache age, that is, the number of days since Google last re-indexed the page listed. Where can …

Read More
How to find elements by class

I'm having trouble parsing HTML elements with "class" attribute using Beautifulsoup. The code looks like this soup = BeautifulSoup(sdata) mydivs = soup.findAll('div') for div in mydivs: if (div["class"] == "stylelistrow"): print div I get an error on the same …

Read More