Benchmarks

Note

Benchmarks are hard.

Repeatedly calling the same function in a tight loop will lead to the instruction cache staying hot and branches being highly predictable. That’s not representative of real world access patterns. It’s also hard to write a nonbiased benchmark. I wrote msgspec, naturally whatever benchmark I publish it’s going to perform well in.

Even so, people like to see benchmarks. I’ve tried to be as nonbiased as I can be, and the results hopefully indicate a few tradeoffs you make when you choose different serialization formats. I encourage you to write your own benchmarks before making these decisions.

Benchmark - Encoding/Decoding

Here we show a simple benchmark serializing some structured data. The data we’re serializing has the following schema (defined here using msgspec.Struct types):

import datetime
import msgspec

class File(msgspec.Struct, tag="file"):
    name: str
    created_by: str
    created_at: datetime.datetime
    updated_at: datetime.datetime
    nbytes: int

class Directory(msgspec.Struct, tag="directory"):
    name: str
    created_by: str
    created_at: datetime.datetime
    updated_at: datetime.datetime
    contents: list[File | Directory]

The libraries we’re benchmarking are the following:

• ujson (link)

• orjson (link)

• msgpack (link)

• msgspec.msgpack

• msgspec.json

Each benchmark creates a message containing one or more File/Directory instances, then then serializes and deserializes it in a loop.

The full benchmark source can be found here.

1 Object

Some workflows involve sending around very small messages. Here the overhead per function call dominates (parsing of options, allocating temporary buffers, etc…).

From the chart above, you can see that msgspec.msgpack and msgspec.json performed quite well, encoding/decoding faster than all other options, even those implementing the same serialization protocol. This is partly due to the use of Struct types here - since all keys are statically known, the msgspec decoders can apply a few optimizations not available to other Python libraries that rely on dict types instead.

That said, all of these methods serialize/deserialize pretty quickly relative to other python operations, so unless you’re counting every microsecond your choice here probably doesn’t matter that much.

1000 Objects

Here we serialize a tree of 1000 File/Directory objects. There’s a lot more data here, so the per-call overhead will no longer dominate, and we’re now measuring the efficiency of the encoding/decoding.

Benchmark - Schema Validation

The above benchmarks aren’t 100% fair to msgspec, as it also performs schema validation on deserialization, checking that the message matches the specified schema. None of the other options benchmarked support this natively. Instead, many users perform validation post deserialization using additional tools like pydantic.

Here we benchmark the following validation libraries, measuring JSON encoding and decoding time.

• msgspec

• pydantic

• cattrs

• mashumaro

The full benchmark source can be found here.

This plot shows the performance benefit of performing type validation during message decoding (as done by msgspec) rather than as a secondary step with a third-party library like pydantic. In this benchmark msgspec is ~10x faster than mashumaro, ~12x faster than cattrs, and ~80x faster than pydantic.

Validating after decoding is slower for two reasons:

• It requires traversing over the entire output structure a second time (which can be slow due to pointer chasing)

• It may require converting some python objects to their desired output types (e.g. converting a decoded dict to a pydantic model), resulting in allocating many temporary python objects.

In contrast, libraries like msgspec that validate during decoding have none of these issues. Only a single pass over the decoded data is taken, and the specified output types are created correctly the first time, avoiding the need for additional unnecessary allocations.

Benchmark - Memory Usage

Here we benchmark loading a medium-sized JSON file (~65 MiB) containing information on all the noarch packages in conda-forge. We compare the following libraries:

• msgspec with msgspec.Struct schemas pre-defined

• msgspec

• json

• ujson

• orjson

• simdjson

For each library, we measure both the peak increase in memory usage (RSS) and the time to JSON decode the file.

The full benchmark source can be found here.

Results (smaller is better):

	memory (MiB)	vs.	time (ms)	vs.
msgspec structs	83.6	1.0x	170.6	1.0x
msgspec	145.3	1.7x	383.1	2.2x
json	213.5	2.6x	526.4	3.1x
ujson	230.6	2.8x	666.8	3.9x
orjson	263.9	3.2x	410.0	2.4x
simdjson	403.7	4.8x	615.1	3.6x

• msgspec decoding into Struct types uses the least amount of memory, and is also the fastest to decode. This makes sense; Struct types are cheaper to allocate and more memory efficient than dict types, and for large messages these differences can really add up.

• msgspec decoding without a schema is the second best option for both memory usage and speed. When decoding without a schema, msgspec makes the assumption that the underlying message probably still has some structure; short dict keys are temporarily cached to be reused later on, rather than reallocated every time. This means that instead of allocating 10,000 copies of the string "name", only a single copy is allocated and reused. For large messages this can lead to significant memory savings. json and orjson also use similar optimizations, but not as effectively.

• orjson and simdjson use 3-5x more memory than msgspec in this benchmark. In addition to the reasons above, both of these decoders require copying the original message into a temporary buffer. In this case, the extra copy adds an extra 65 MiB of overhead!

Benchmark - Structs

Here we benchmark common msgspec.Struct operations, comparing their performance against other similar libraries. The cases compared are:

• msgspec

• Standard Python classes

• dataclasses

• attrs

• pydantic

For each library, the following operations are benchmarked:

• Time to define a new class. Many libraries that abstract away class boilerplate add overhead when defining classes, slowing import times for libraries that make use of these classes.

• Time to create an instance of that class.

• Time to compare two instances for equality (==/!=).

• Time to compare two instances for order (</>/<=/>=)

The full benchmark source can be found here.

Results (smaller is better):

	import (μs)	create (μs)	equality (μs)	order (μs)
msgspec	9.92	0.09	0.02	0.03
standard classes	6.86	0.45	0.13	0.29
dataclasses	489.07	0.47	0.27	0.30
attrs	428.38	0.42	0.29	2.15
pydantic	371.52	4.84	10.56	N/A

• Standard Python classes are the fastest to import (any library can only add overhead here). Still, msgspec isn’t that much slower, especially compared to other options.

• Structs are optimized to be cheap to create, and that shows for the creation benchmark. They’re roughly 5x faster than standard classes/attrs/dataclasses, and 50x faster than pydantic.

• For equality comparison, msgspec Structs are roughly 6x to 500x faster than the alternatives.

• For order comparison, msgspec Structs are roughly 10x to 70x faster than the alternatives.

Benchmark - Garbage Collection

msgspec.Struct instances implement several optimizations for reducing garbage collection (GC) pressure and decreasing memory usage. Here we benchmark structs (with and without gc=False) against standard Python classes (with and without __slots__).

For each option we create a large dictionary containing many simple instances of the benchmarked type, then measure:

• The amount of time it takes to do a full garbage collection (gc) pass

• The total amount of memory used by this data structure

The full benchmark source can be found here.

Results (smaller is better):

	GC time (ms)	Memory Used (MiB)
standard class	80.46	211.66
standard class with __slots__	80.06	120.11
msgspec struct	13.96	120.11
msgspec struct with gc=False	1.07	104.85

• Standard Python classes are the most memory hungry (since all data is stored in an instance dict). They also result in the largest GC pause, as the GC has to traverse the entire outer dict, each class instance, and each instance dict. All that pointer chasing has a cost.

• Standard classes with __slots__ are less memory hungry, but still results in an equivalent GC pauses.

• msgspec.Struct instances have the same memory layout as a class with __slots__ (and thus have the same memory usage), but due to deferred GC tracking a full GC pass completes in a fraction of the time.

• msgspec.Struct instances with gc=False have the lowest memory usage (lack of GC reduces memory by 16 bytes per instance). They also have the lowest GC pause (75x faster than standard classes!) since the entire composing dict can be skipped during GC traversal.

Benchmark - Library Size

Here we compare the on-disk size of a few Python libraries.

The full benchmark source can be found here.

Results (smaller is better)

	version	size (MiB)	vs. msgspec
msgspec	0.12.0	0.34	1.00x
orjson	3.8.5	0.56	1.64x
msgpack	1.0.4	0.99	2.91x
pydantic	1.10.4	8.71	25.67x

The functionality available in msgspec is comparable to that of orjson, msgpack, and pydantic combined. However, the total installed binary size of msgspec is a fraction of that of any of these libraries.