Python HTTP Error 429 with urllib2

Question

I am using the following code to resolve redirects to return a links final url

def resolve_redirects(url):
    return urllib2.urlopen(url).geturl()

Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        return urllib2.urlopen(url).geturl()

Also, what would happen if there is an exception in the except block?

Answer 1

It would be better to make sure the HTTP code is actually 429 before re-trying.

That can be done like this:

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError, e:
        if e.code == 429:
             time.sleep(5);
             return resolve_redirects(url)
        raise

This will also allow arbitrary numbers of retries (which may or may not be desired).

https://docs.python.org/2/howto/urllib2.html#httperror