How to efficiently use MySQLDB SScursor?

Sylvain picture Sylvain · Nov 27, 2009 · Viewed 17.6k times · Source

I have to deal with a large result set (could be hundreds thousands of rows, sometimes more).
They unfortunately need to be retrieved all at once (on start up).

I'm trying to do that by using as less memory as possible.
By looking on SO I've found that using SSCursor might be what I'm looking for, but I still don't really know how to exactly use them.

Is doing a fetchall() from a base cursor or a SScursor the same (in term of memory usage)?
Can I 'stream' from the sscursor my rows one by one (or a few by a few) and if yes,
what is the best way to do so?

Answer

unutbu picture unutbu · Nov 27, 2009

I am in agreement with Otto Allmendinger's answer, but to make explicit Denis Otkidach's comment, here is how you can iterate over the results without using Otto's fetch() function:

import MySQLdb.cursors
connection=MySQLdb.connect(
    host="thehost",user="theuser",
    passwd="thepassword",db="thedb",
    cursorclass = MySQLdb.cursors.SSCursor)
cursor=connection.cursor()
cursor.execute(query)
for row in cursor:
    print(row)