I have a typical Relational Database model laid out in Django where a typical model contains some ForeignKeys
, some ManyToManyFields
, and some fields that extend Django's DateTimeField
.
I want to save data that I am receiving in JSON format (not flat) from an external api. I wan't it such that data gets saved to respective tables (not the whole json string to one field). What is the most clean and simple approach to do this? Is there a library available to make this task simpler?
Here's an example to clarify my question,
Models-
class NinjaData(models.Model):
id = models.IntegerField(primary_key=True, unique=True)
name = models.CharField(max_length=60)
birthdatetime = MyDateTimeField(null=True)
deathdatetime = MyDatetimeField(null=True)
skills = models.ManyToManyField(Skills, null=True)
weapons = models.ManyToManyField(Weapons, null=True)
master = models.ForeignKey(Master, null=True)
class Skills(models.Model):
id = models.IntegerField(primary_key=True, unique=True)
name = models.CharField(max_length=60)
difficulty = models.IntegerField(null=True)
class Weapons(models.Model):
id = models.IntegerField(primary_key=True, unique=True)
name = models.CharField(max_length=60)
weight = models.FloatField(null=True)
class Master(models.Model):
id = models.IntegerField(primary_key=True, unique=True)
name = models.CharField(max_length=60)
is_awesome = models.NullBooleanField()
now, I typically have to save json string data that I obtain from an external api (secret ninja api) into this model, json looks like this
JSON-
{
"id":"1234",
"name":"Hitori",
"birthdatetime":"11/05/1999 20:30:00",
"skills":[
{
"id":"3456",
"name":"stealth",
"difficulty":"2"
},
{
"id":"678",
"name":"karate",
"difficulty":"1"
}
],
"weapons":[
{
"id":"878",
"name":"shuriken",
"weight":"0.2"
},
{
"id":"574",
"name":"katana",
"weight":"0.5"
}
],
"master":{
"id":"4",
"name":"Schi fu",
"is_awesome":"true"
}
}
now logic for handling a typical ManyToManyField is fairly simple,
logic code -
data = json.loads(ninja_json)
ninja = NinjaData.objects.create(id=data['id'], name=data['name'])
if 'weapons' in data:
weapons = data['weapons']
for weapon in weapons:
w = Weapons.objects.get_or_create(**weapon) # create a new weapon in Weapon table
ninja.weapons.add(w)
if 'skills' in data:
...
(skipping rest of the code for brevity)
There are many approaches that i could use,
view
function that does all the work of converting json to model instances__init__
method save()
method create
, get_or_create
, filter
etc. ManyToManyField
and put it there,I would like to know if there is a single most obvious way to save data in this json form to database without coding the above logic multiple times, what would be the most elegant approach that you would suggest?
Thanks all for reading the long post,
In my opinion the cleanest place for the code you need is as a new Manager method (eg from_json_string) on a custom manager for the NinjaData model.
I don't think you should override the standard create, get_or_create etc methods since you're doing something a bit different from what they normally do and it's good to keep them working normally.
Update: I realised I'd probably want this for myself at some point so I have coded up and lightly tested a generic function. Since it recursively goes through and affects other models I'm no longer certain it belongs as a Manager method and should probably be a stand-alone helper function.
def create_or_update_and_get(model_class, data):
get_or_create_kwargs = {
model_class._meta.pk.name: data.pop(model_class._meta.pk.name)
}
try:
# get
instance = model_class.objects.get(**get_or_create_kwargs)
except model_class.DoesNotExist:
# create
instance = model_class(**get_or_create_kwargs)
# update (or finish creating)
for key,value in data.items():
field = model_class._meta.get_field(key)
if not field:
continue
if isinstance(field, models.ManyToManyField):
# can't add m2m until parent is saved
continue
elif isinstance(field, models.ForeignKey) and hasattr(value, 'items'):
rel_instance = create_or_update_and_get(field.rel.to, value)
setattr(instance, key, rel_instance)
else:
setattr(instance, key, value)
instance.save()
# now add the m2m relations
for field in model_class._meta.many_to_many:
if field.name in data and hasattr(data[field.name], 'append'):
for obj in data[field.name]:
rel_instance = create_or_update_and_get(field.rel.to, obj)
getattr(instance, field.name).add(rel_instance)
return instance
# for example:
from django.utils.simplejson import simplejson as json
data = json.loads(ninja_json)
ninja = create_or_update_and_get(NinjaData, data)