Google Trends Quota Limit

Question

I'm trying to pull data from Google trends and got a "You have reached your daily limit" error after only 2 tries.

Is there any way to go around this? I know Google API projects have special quota limits but Google Trends doesn't have an API. I also read that we may need to pass it a cookie file so that it seems like I'm logged in. Has anyone faced this issue before?

Answer 1

I'm struggling with the same issue! From your question I can't figure out what stage have you achieved... But here is the solution that I've found:

1. You should emulate browser with cookies. I think the best way to do it is to use Mechanize library.
2. At first your program should "login" using GET request to "https://accounts.google.com/Login?hl=en"
3. Immediately after that you can access some other personal resources, but not google trends!
4. After some significant time you can successfully get google trends data as CSV.
5. I still have not discovered the exact time period, but it is more than 10 minutes and less than several hours :). That is why saving your cookies for latter use is a good idea!

Few more tips:

• If you are developing using python / ruby under Windows do not forget to set up CA ROOT certificates package for OpenSSL library. Otherwise HTTPS connection will fail and you won't login! See Getting the `certificate verify failed (OpenSSL::SSL::SSLError)` erro with Mechanize object

• I recommend you to save cookies to external file at program shutdown. And restoring them at startup.

• Do not forget to allow redirects, because Google is using redirects all the time.

Ruby code example:

require 'mechanize'
require 'logger'
begin
  agent = Mechanize.new { |a|
    a.user_agent = 'Opera/9.80 (Windows NT 5.1) Presto/2.12.388 Version/12.16'

    cert_store = OpenSSL::X509::Store.new
    cert_store.add_file 'cacert.pem'
    a.cert_store = cert_store

    a.log = Logger.new('mech.log')

    if File.file?('mech.cookies')
      cookies = Mechanize::CookieJar.new
      cookies.load('mech.cookies')
      a.cookie_jar = cookies
    end

    a.open_timeout = 5
    a.read_timeout = 6
    a.keep_alive   = true
    a.redirect_ok  = true
  }

  LOGIN_URL = "https://accounts.google.com/Login?hl=en&continue=http://www.google.com/trends/"
  login_page = agent.get(LOGIN_URL)
  login_form = login_page.forms.first
  login_form.Email = *
  login_form.Passwd = *
  login_response_page = agent.submit(login_form)

  page = agent.get(url)

  # DO SOME TRENDS REQUESTS AFTER SIGNIFICANT PERIOD OF TIME

ensure
  if agent
    agent.cookie_jar.save('mech.cookies')
  end
end