Canonicalize / normalize a URL?

url python-3.x normalization
XZS picture XZS · May 14, 2012 · Viewed 10.5k times · Source

I am searching for a library function to normalize a URL in Python, that is to remove "./" or "../" parts in the path, or add a default port or escape special characters and so on. The result should be a string that is unique for two URLs pointing to the same web page. For example http://google.com and http://google.com:80/a/../ shall return the same result.

I would prefer Python 3 and already looked through the urllib module. It offers functions to split URLs but nothing to canonicalize them. Java has the URI.normalize() function that does a similar thing (though it does not consider the default port 80 equal to no given port), but is there something like this is python?

Answer

Thomas K picture Thomas K · May 14, 2012

How about this:

In [1]: from urllib.parse import urljoin

In [2]: urljoin('http://example.com/a/b/c/../', '.')
Out[2]: 'http://example.com/a/b/'

Inspired by answers to this question. It doesn't normalize ports, but it should be simple to whip up a function that does.

Related questions

opening a url with urllib in python 3

i'm trying to open the URL of this API from the sunlight foundation and return the data from the page in json. this is the code Ive produced, minus the parenthesis around myapikey. import urllib.request.urlopen import json urllib.…

Read More
download image from url using python urllib but receiving HTTP Error 403: Forbidden

I want to download image file from a url using python module "urllib.request", which works for some website (e.g. mangastream.com), but does not work for another (mangadoom.co) receiving error "HTTP Error 403: Forbidden". What could be the …

Read More
How can I get query string values in JavaScript?

Is there a plugin-less way of retrieving query string values via jQuery (or without)? If so, how? If not, is there a plugin which can do so?

Read More