Efficient way to bulk insert with get_or_create() in Django (SQL, Python, Django)

python django bulkinsert
kemar picture kemar · Feb 12, 2010 · Viewed 23.6k times · Source

Is there a more efficient way for doing this?

for item in item_list:
    e, new = Entry.objects.get_or_create(
        field1 = item.field1,
        field2 = item.field2,
    )

Answer

Glenn Maynard picture Glenn Maynard · Feb 12, 2010

You can't do decent bulk insertions with get_or_create (or even create), and there's no API for doing this easily.

If your table is simple enough that creating rows with raw SQL isn't too much of a pain, it's not too hard; something like:

INSERT INTO site_entry (field1, field2)
(
         SELECT i.field1, i.field2
         FROM (VALUES %s) AS i(field1, field2)
         LEFT JOIN site_entry as existing
                 ON (existing.field1 = i.field1 AND existing.field2 = i.field2)
         WHERE existing.id IS NULL
)

where %s is a string like ("field1, field2"), ("field3, field4"), ("field5, field6") that you'll have to create and escape properly yourself.

Related questions

django bulk create ignore duplicates

I'm trying to bulk insert into a MySQL db for a very large dataset and would love to use django's bulk_create while ignoring duplicate errors. Sample model: class MyModel(models.Model): my_id=models.IntegerField(primary_key=True) start_…

Read More
'pip' is not recognized as an internal or external command

I'm running into a weird error when trying to install Django on my computer. This is the sequence that I typed into my command line: C:\Python34> python get-pip.py Requirement already up-to-date: pip in c:\python34\lib\site-packages …

Read More
No module named MySQLdb

I am using Python version 2.5.4 and install MySQL version 5.0 and Django. Django is working fine with Python, but not MySQL. I am using it in Windows Vista.

Read More