Python urllib over TOR?

Question

Sample code:

#!/usr/bin/python
import socks
import socket
import urllib2

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 9050, True)
socket.socket = socks.socksocket

print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()

TOR is running a SOCKS proxy on port 9050 (its default). The request goes through TOR, surfacing at an IP address other than my own. However, TOR console gives the warning:

"Feb 28 22:44:26.233 [warn] Your application (using socks4 to port 80) is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information. Consider using Socks4A (e.g. via privoxy or socat) instead. For more information, please see https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS."

i.e. DNS lookups aren't going through the proxy. But that's what the 4th parameter to setdefaultproxy is supposed to do, right?

From http://socksipy.sourceforge.net/readme.txt:

setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])

rdns - This is a boolean flag than modifies the behavior regarding DNS resolving. If it is set to True, DNS resolving will be preformed remotely, on the server.

Same effect with both PROXY_TYPE_SOCKS4 and PROXY_TYPE_SOCKS5 selected.

It can't be a local DNS cache (if urllib2 even supports that) because it happens when I change the URL to a domain that this computer has never visited before.

Answer 1

The problem is that httplib.HTTPConnection uses the socket module's create_connection helper function which does the DNS request via the usual getaddrinfo method before connecting the socket.

The solution is to make your own create_connection function and monkey-patch it into the socket module before importing urllib2, just like we do with the socket class.

import socks
import socket
def create_connection(address, timeout=None, source_address=None):
    sock = socks.socksocket()
    sock.connect(address)
    return sock

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)

# patch the socket module
socket.socket = socks.socksocket
socket.create_connection = create_connection

import urllib2

# Now you can go ahead and scrape those shady darknet .onion sites