Youtube Reports API HttpError 429 FreeQuotaGroupCLIENT_PROJECT-100
I'm pulling youtube analytics by youtube bulk reports. https://developers.google.com/youtube/reporting/v1/reports/
Everything works fine accept when we have many users, we encounter
<HttpError 429 when requesting https://youtubereporting.googleapis.com/v1/media/CHANNEL/****/jobs/****/reports/***?alt=media returned "Insufficient tokens for quota group and limit 'FreeQuotaGroupCLIENT_PROJECT-100s' of service 'youtubereporting.googleapis.com', using the limit by ID '****'.">
I know there is a limit number of API calls per 100 seconds.
Is there anyway to increase this limit, because I saw it's FreeQuotaGroupCLIENT_PROJECT-100s, so there might be Paid Quota or something else.
If not, what's the best way to handle fallback? We can't use sleep method because there are many parallel process, they won't wait for another.
Thank you.
Answer
The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting"). Check this related SO post which states that:
Receiving a status 429 is not an error, it is the other server "kindly" asking you to please stop spamming requests. Obviously, your rate of requests has been too high and the server is not willing to accept this.
You should not seek to "dodge" this, or even try to circumvent server security settings by trying to spoof your IP, you should simply respect the server's answer by not sending too many requests.
If everything is set up properly, you will also have received a "Retry-after" header along with the 429 response. This header specifies the number of seconds you should wait before making another call. The proper way to deal with this "problem" is to read this header and to sleep your process for that many seconds.
The discovery response does not change frequently; cache the discovery response locally or retry using exponential back-off. You need to slow down the rate at which you are sending the requests.