How to convert a boto3 Dynamo DB item to a regular dictionary in Python?

Question

In Python, when an item is retrieved from Dynamo DB using boto3, a schema like the following is obtained.

{
  "ACTIVE": {
    "BOOL": true
  },
  "CRC": {
    "N": "-1600155180"
  },
  "ID": {
    "S": "bewfv43843b"
  },
  "params": {
    "M": {
      "customer": {
        "S": "TEST"
      },
      "index": {
        "N": "1"
      }
    }
  },
  "THIS_STATUS": {
    "N": "10"
  },
  "TYPE": {
    "N": "22"
  }
}

Also when inserting or scanning, dictionaries have to be converted in this fashion. I haven't been able to find a wrapper that takes care of such conversion. Since apparently boto3 does not support this, are there better alternatives than implementing code for it?

Answer 1

In order to understand how to solve this, it's important to recognize that boto3 has two basic modes of operation: one that uses the low-level Client API, and one that uses higher level abstractions like Table. The data structure shown in the question is an example of what is consumed/produced by the low-level API, which is also used by the AWS CLI and the dynamodb web services.

To answer your question - if you can work exclusively with the high-level abstractions like Table when using boto3 then things will be quite a bit easier for you, as the comments suggest. Then you can sidestep the whole problem - python types are marshaled to and from the low-level data format for you.

However, there are some times when it's not possible to use those high-level constructs exclusively. I specifically ran into this problem when dealing with DynamoDB streams attached to Lambdas. The inputs to the lambda are always in the low-level format, and that format is harder to work with IMO.

After some digging I found that boto3 itself has some nifty features tucked away for doing conversions. These features are used implicitly in all of the internal conversions mentioned previously. To use them directly, import the TypeDeserializer/TypeSerializer classes and combine them with dict comprehensions like so:

import boto3

low_level_data = {
  "ACTIVE": {
    "BOOL": True
  },
  "CRC": {
    "N": "-1600155180"
  },
  "ID": {
    "S": "bewfv43843b"
  },
  "params": {
    "M": {
      "customer": {
        "S": "TEST"
      },
      "index": {
        "N": "1"
      }
    }
  },
  "THIS_STATUS": {
    "N": "10"
  },
  "TYPE": {
    "N": "22"
  }
}

# Lazy-eval the dynamodb attribute (boto3 is dynamic!)
boto3.resource('dynamodb')

# To go from low-level format to python
deserializer = boto3.dynamodb.types.TypeDeserializer()
python_data = {k: deserializer.deserialize(v) for k,v in low_level_data.items()}

# To go from python to low-level format
serializer = boto3.dynamodb.types.TypeSerializer()
low_level_copy = {k: serializer.serialize(v) for k,v in python_data.items()}

assert low_level_data == low_level_copy