Python shortest unique id from strings

python md5 hashlib
jack picture jack · Jun 19, 2012 · Viewed 19.9k times · Source

I have more than 100 million unique strings (VARCHAR(100) UNIQUE in MySQL database). Now I use the code below to create unique hash from them (VARCHAR(32) UNIQUE) in order to reduct index size of the InnoDB table (a unique index on varchar(100) is roughly 3 times larger than on varchar(32) field).

id = hashlib.md5(str).hexdigest()

Is there any other method to create shorter ids from those strings and make reasonable uniqueness guarantees?

Answer

simplylizz picture simplylizz · Jun 19, 2012

You can save it as integer:

id_ = int(hashlib.md5(your_str).hexdigest(), 16)

Or as binary string:

id_ = hashlib.md5(your_str).digest()

Related questions

Generating an MD5 checksum of a file

Is there any simple way of generating (and checking) MD5 checksums of a list of files in Python? (I have a small program I'm working on, and I'd like to confirm the checksums of the files).

Read More
Get MD5 hash of big files in Python

I have used hashlib (which replaces md5 in Python 2.6/3.0) and it worked fine if I opened a file and put its content in hashlib.md5() function. The problem is with very big files that their sizes could exceed RAM size. …

Read More
Hashing a file in Python

I want python to read to the EOF so I can get an appropriate hash, whether it is sha1 or md5. Please help. Here is what I have so far: import hashlib inputFile = raw_input("Enter the name of the …

Read More