How to define avro schema for complex json document?

json serialization mapreduce avro
user2727704 picture user2727704 · Jan 27, 2015 · Viewed 22.2k times · Source

I have a JSON document that I would like to convert to Avro and need a schema to be specified for that purpose. Here is the JSON document for which I would like to define the avro schema:

{
 "uid": 29153333,
 "somefield": "somevalue",
 "options": [
   {
     "item1_lvl2": "a",
     "item2_lvl2": [
       {
         "item1_lvl3": "x1",
         "item2_lvl3": "y1"
       },
       {
         "item1_lvl3": "x2",
         "item2_lvl3": "y2"
       }
     ]
   }
 ]
}

I'm able to define the schema for the non-complex types but not for the complex "options" field:

{
  "namespace" : "my.com.ns",
  "type" :  "record",
  "fields" : [
     {"name": "uid", "type": "int"},
     {"name": "somefield", "type": "string"}
     {"name": "options", "type": .....}
  ]
}

Thanks for the help!

Answer

Marcel Krcah picture Marcel Krcah · Jan 27, 2015

You need to use Avro complex types, specifically arrays and records. And then nest these together:

{
  "namespace" : "my.com.ns",
  "name": "myrecord",
  "type" :  "record",
  "fields" : [
     {"name": "uid", "type": "int"},
     {"name": "somefield", "type": "string"},
     {"name": "options", "type": {
        "type": "array",
        "items": {
            "type": "record",
            "name": "lvl2_record",
            "fields": [
                {"name": "item1_lvl2", "type": "string"},
                {"name": "item2_lvl2", "type": {
                    "type": "array",
                    "items": {
                        "type": "record",
                        "name": "lvl3_record",
                        "fields": [
                            {"name": "item1_lvl3", "type": "string"},
                            {"name": "item2_lvl3", "type": "string"}
                        ]
                    }
                }}
            ]
        }
     }}
  ]
}

Also, to improve readiblity, you can split the schema into multiple files.

Related questions

How do I turn a C# object into a JSON string in .NET?

I have classes like these: class MyDate { int year, month, day; } class Lad { string firstName; string lastName; MyDate dateOfBirth; } And I would like to turn a Lad object into a JSON string like this: { "firstName":"Markoff", "lastName":"Chaney", "dateOfBirth": { "year":"1901", "…

Read More
Convert form data to JavaScript object with jQuery

How do I convert all elements of my form to a JavaScript object? I'd like to have some way of automatically building a JavaScript object from my form, without having to loop over each element. I do not want a …

Read More
Serializing to JSON in jQuery

I need to serialize an object to JSON. I'm using jQuery. Is there a "standard" way to do this? My specific situation: I have an array defined as shown below: var countries = new Array(); countries[0] = 'ga'; countries[1] = 'cd'; ... and I …

Read More