Getting historical data from Twitter

api twitter polling
Marco picture Marco · Nov 2, 2009 · Viewed 21.8k times · Source

For a research project I would like to get the last 3 months worth of Twitter messages. Technical challenges aside, is this possible? by using some sort of slow polling mechanism to keep the rate limiter at bay?

The Twitter API states "Clients may request up to 3,200 statuses via the page and count parameters for timeline REST API" Are these per hour? Per day? or...ever?

Any suggestions? Would it even be theoretically possible? Did some one do something similar before?

Thanks! Marco

Answer

Ted Pennings picture Ted Pennings · Nov 14, 2009

Twitter notoriously does not make "available" tweets older than three weeks. In some cases you can only get one week. You're better off storing tweets for the next three months. Many rightly doubt if they're even persisted by Twitter.

Are you looking for just any tweets? If so, check out the Streaming API's status/sample method. The streaming API uses persistent HTTP sockets that can be a pain to program, but it's quite graceful when you get it working. I'd recommend setting up a little script to dump tweets from status/sample into a DB. You should have a TON of data after just a few days.

Related questions

Twitter API returns error 215, Bad Authentication Data

I am trying to call following Twitter's API to get a list of followers for a user. http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=username And I am getting this error message in response. { code = 215; …

Read More
Find ALL tweets from a user (not just the first 3,200)

With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find …

Read More
How to make an API call using meteor

Ok here is the twitter API, http://search.twitter.com/search.atom?q=perkytweets Can any one give me any hint about how to go about calling this API or link using Meteor Update:: Here is the code that i …

Read More