Filter Twitter feeds only by language

python twitter nlp tweepy twitter-streaming-api
Sudo picture Sudo · Nov 12, 2014 · Viewed 27.4k times · Source

I am using Tweepy API for extracting Twitter feeds. I want to extract all Twitter feeds of a specific language only. The language filter works only if track filter is provided. The following code returns 406 error:

l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(languages=["en"])

How can I extract all the tweets from certain language using Tweepy?

Answer

Luigi picture Luigi · Nov 14, 2014

You can't (without special access). Streaming all the tweets (unfiltered) requires a connection to the firehose, which is granted only in specific use cases by Twitter. Honestly, the firehose isn't really necessary--proper use of track can get you more tweets than you know what to do with.

Try using something like this:

stream.filter(languages=["en"], track=["a", "the", "i", "you", "u"]) # etc

Filtering by words like that will get you many, many tweets. If you want real data for the most-used words, check out this article from Time: The 500 Most Frequently Used Words on Twitter. You can use up to 400 keywords, but that will likely approach the 1% limit of tweets at a given time interval. If your track parameter matches 60% of all tweets at a given time, you will still only get 1% (which is a LOT of tweets).

Related questions

How to extract hours and minutes from a datetime.datetime object?

I am required to extract the time of the day from the datetime.datetime object returned by the created_at attribute. But I do not understand how to do that. This is my code for getting the datetime.datetime object. …

Read More
Pretty-Print JSON Data to a File using Python

A project for class involves parsing Twitter JSON data. I'm getting the data and setting it to the file without much trouble, but it's all in one line. This is fine for the data manipulation I'm trying to do, but …

Read More
python: [Errno 10054] An existing connection was forcibly closed by the remote host

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after some times of running (around 1), when the Twitter's rate limit …

Read More