Performant Entity Serialization: BSON vs MessagePack (vs JSON)

Question

Recently I've found MessagePack, an alternative binary serialization format to Google's Protocol Buffers and JSON which also outperforms both.

Also there's the BSON serialization format that is used by MongoDB for storing data.

Can somebody elaborate the differences and the dis-/advantages of BSON vs MessagePack?

---

Just to complete the list of performant binary serialization formats: There are also Gobs which are going to be the successor of Google's Protocol Buffers. However in contrast to all the other mentioned formats those are not language-agnostic and rely on Go's built-in reflection there are also Gobs libraries for at least on other language than Go.

Answer 1

// Please note that I'm author of MessagePack. This answer may be biased.

Format design

1. Compatibility with JSON

   In spite of its name, BSON's compatibility with JSON is not so good compared with MessagePack.

   BSON has special types like "ObjectId", "Min key", "UUID" or "MD5" (I think these types are required by MongoDB). These types are not compatible with JSON. That means some type information can be lost when you convert objects from BSON to JSON, but of course only when these special types are in the BSON source. It can be a disadvantage to use both JSON and BSON in single service.

   MessagePack is designed to be transparently converted from/to JSON.

2. MessagePack is smaller than BSON

   MessagePack's format is less verbose than BSON. As the result, MessagePack can serialize objects smaller than BSON.

   For example, a simple map {"a":1, "b":2} is serialized in 7 bytes with MessagePack, while BSON uses 19 bytes.

3. BSON supports in-place updating

   With BSON, you can modify part of stored object without re-serializing the whole of the object. Let's suppose a map {"a":1, "b":2} is stored in a file and you want to update the value of "a" from 1 to 2000.

   With MessagePack, 1 uses only 1 byte but 2000 uses 3 bytes. So "b" must be moved backward by 2 bytes, while "b" is not modified.

   With BSON, both 1 and 2000 use 5 bytes. Because of this verbosity, you don't have to move "b".

4. MessagePack has RPC

   MessagePack, Protocol Buffers, Thrift and Avro support RPC. But BSON doesn't.

These differences imply that MessagePack is originally designed for network communication while BSON is designed for storages.

Implementation and API design

1. MessagePack has type-checking APIs (Java, C++ and D)

   MessagePack supports static-typing.

   Dynamic-typing used with JSON or BSON are useful for dynamic languages like Ruby, Python or JavaScript. But troublesome for static languages. You must write boring type-checking codes.

   MessagePack provides type-checking API. It converts dynamically-typed objects into statically-typed objects. Here is a simple example (C++):

    #include <msgpack.hpp>

    class myclass {
    private:
        std::string str;
        std::vector<int> vec;
    public:
        // This macro enables this class to be serialized/deserialized
        MSGPACK_DEFINE(str, vec);
    };

    int main(void) {
        // serialize
        myclass m1 = ...;

        msgpack::sbuffer buffer;
        msgpack::pack(&buffer, m1);

        // deserialize
        msgpack::unpacked result;
        msgpack::unpack(&result, buffer.data(), buffer.size());

        // you get dynamically-typed object
        msgpack::object obj = result.get();

        // convert it to statically-typed object
        myclass m2 = obj.as<myclass>();
    }

2. MessagePack has IDL

   It's related to the type-checking API, MessagePack supports IDL. (specification is available from: http://wiki.msgpack.org/display/MSGPACK/Design+of+IDL)

   Protocol Buffers and Thrift require IDL (don't support dynamic-typing) and provide more mature IDL implementation.

3. MessagePack has streaming API (Ruby, Python, Java, C++, ...)

   MessagePack supports streaming deserializers. This feature is useful for network communication. Here is an example (Ruby):

    require 'msgpack'

    # write objects to stdout
    $stdout.write [1,2,3].to_msgpack
    $stdout.write [1,2,3].to_msgpack

    # read objects from stdin using streaming deserializer
    unpacker = MessagePack::Unpacker.new($stdin)
    # use iterator
    unpacker.each {|obj|
      p obj
    }