Try/except when using Python requests module

Question

Doing some API testing and trying to create a function that given an inputted URL it will return the json response, however if a HTTP error is the response an error message will be returned.

I was using urllib2 before, but now trying to use requests instead. However it looks like my except block is never executed, regardless of the error.

testURL = 'http://httpbin.org/status/404'


def return_json(URL):
    try:
        response = requests.get(URL)
        json_obj = response.json()
        return json_obj
    except requests.exceptions.HTTPError as e:
        return "Error: " + str(e)

The result I get from running the above...

<Response [404]>

Answer 1

If you want the response to raise an exception for a non-200 status code use response.raise_for_status(). Your code would then look like:

testURL = 'http://httpbin.org/status/404'


def return_json(URL):
    response = requests.get(testURL)

    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as e:
        # Whoops it wasn't a 200
        return "Error: " + str(e)

    # Must have been a 200 status code
    json_obj = response.json()
    return json_obj

You can tell that this is clearly simpler than the other solutions here and doesn't require you to check the status code manually. You would also just catch an HTTPError since that is what raise_for_status will raise. Catching RequestsException is a poor idea. That will catch things like ConnectionErrors or TimeoutErrors, etc. None of those mean the same thing as what you're trying to catch.