"daily limit exceeded" when using Google custom search API

python google-custom-search google-api-python-client
yvetterowe picture yvetterowe · Apr 7, 2014 · Viewed 10.7k times · Source

I wanna crawl 200 results for about 2000 queries but it gives me a "daily limit exceeded" error.

I want to confirm how many results can we crawl per day. Is there any solution that can solve this problem? Or the only way is to crawl a small part of queries each day...?

My code to crawl google is as follow:

def crawl(query_list):
    http = httplib2.Http()

    # Construct the service object for the interacting with the CustomSearch API.
    service = discovery.build('customsearch', 'v1',  developerKey='my api key', http=http)

    res_list = []
    for query in query_list:
        json_res = service.cse().list(q = query, cx = 'my search engine id', num = 200,).execute()
        res_list.append(json_res)

Thank you!

Answer

Amaury Medeiros picture Amaury Medeiros · Apr 7, 2014

According to this post:

The first 100 queries per day are free. Any more, then you have to pay $5 per 1000 queries, for up to 10,000 queries per day, just enable billing to do so. Each query returns a maximum of 10 results, so you can retrieve 1000 URL’s from your search per day for free.

Related questions

Programmatically searching google in Python using custom search

I have a snippet of code using the pygoogle python module that allows me to programmatically search for some term in google succintly: g = pygoogle(search_term) g.pages = 1 results = g.get_urls()[0:10] I just found out that this has …

Read More
download images with google custom search api

I have used google image api in python to download 20 first image result with the following code: import os import sys import time from urllib import FancyURLopener import urllib2 import simplejson searchTerm = "Cat" # Replace spaces ' ' in search term …

Read More
ModuleNotFoundError: No module named 'google.appengine'

I want to do a google search in python3 on windows. The google instructions say they support python3 and to type "gcloud topic init" for details - but that fails saying no interpreter for python2.7. Do I have to install …

Read More