Benchmarks#
Note
Benchmarks are hard.
Repeatedly calling the same function in a tight loop will lead to the instruction cache staying hot and branches being highly predictable. That’s not representative of real world access patterns. It’s also hard to write a nonbiased benchmark. I wrote msgspec, naturally whatever benchmark I publish it’s going to perform well in.
Even so, people like to see benchmarks. I’ve tried to be as nonbiased as I can be, and the results hopefully indicate a few tradeoffs you make when you choose different serialization formats. I encourage you to write your own benchmarks before making these decisions.
Benchmark - Encoding/Decoding#
Here we show a simple benchmark serializing some structured data. The data
we’re serializing has the following schema (defined here using msgspec.Struct
types):
import datetime
import msgspec
class File(msgspec.Struct, tag="file"):
name: str
created_by: str
created_at: datetime.datetime
updated_at: datetime.datetime
nbytes: int
class Directory(msgspec.Struct, tag="directory"):
name: str
created_by: str
created_at: datetime.datetime
updated_at: datetime.datetime
contents: list[File | Directory]
The libraries we’re benchmarking are the following:
Each benchmark creates a message containing one or more File
/Directory
instances, then then serializes and deserializes it in a loop.
The full benchmark source can be found here.
1 Object#
Some workflows involve sending around very small messages. Here the overhead per function call dominates (parsing of options, allocating temporary buffers, etc…).
From the chart above, you can see that msgspec.msgpack
and msgspec.json
performed quite well, encoding/decoding faster than all other options, even
those implementing the same serialization protocol. This is partly due to the
use of Struct
types here - since all keys are statically known, the msgspec
decoders can apply a few optimizations not available to other Python libraries
that rely on dict
types instead.
That said, all of these methods serialize/deserialize pretty quickly relative to other python operations, so unless you’re counting every microsecond your choice here probably doesn’t matter that much.
1000 Objects#
Here we serialize a tree of 1000 File
/Directory
objects. There’s a lot
more data here, so the per-call overhead will no longer dominate, and we’re now
measuring the efficiency of the encoding/decoding.
Benchmark - Schema Validation#
The above benchmarks aren’t 100% fair to msgspec
, as it also performs
schema validation on deserialization, checking that the message matches the
specified schema. None of the other options benchmarked support this natively.
Instead, many users perform validation post deserialization using additional
tools like pydantic.
Here we benchmark the following validation libraries, measuring JSON encoding and decoding time.
The full benchmark source can be found here.
This plot shows the performance benefit of performing type validation during
message decoding (as done by msgspec
) rather than as a secondary step with
a third-party library like pydantic. In this benchmark msgspec
is ~10x
faster than mashumaro
, ~12x faster than cattrs
, and ~80x faster than
pydantic
.
Validating after decoding is slower for two reasons:
It requires traversing over the entire output structure a second time (which can be slow due to pointer chasing)
It may require converting some python objects to their desired output types (e.g. converting a decoded
dict
to a pydantic model), resulting in allocating many temporary python objects.
In contrast, libraries like msgspec
that validate during decoding have none
of these issues. Only a single pass over the decoded data is taken, and the
specified output types are created correctly the first time, avoiding the need
for additional unnecessary allocations.
Benchmark - Memory Usage#
Here we benchmark loading a medium-sized JSON file (~65 MiB)
containing information on all the noarch
packages in conda-forge. We
compare the following libraries:
For each library, we measure both the peak increase in memory usage (RSS) and the time to JSON decode the file.
The full benchmark source can be found here.
Results (smaller is better):
memory (MiB) |
vs. |
time (ms) |
vs. |
|
---|---|---|---|---|
msgspec structs |
83.6 |
1.0x |
170.6 |
1.0x |
msgspec |
145.3 |
1.7x |
383.1 |
2.2x |
json |
213.5 |
2.6x |
526.4 |
3.1x |
ujson |
230.6 |
2.8x |
666.8 |
3.9x |
orjson |
263.9 |
3.2x |
410.0 |
2.4x |
simdjson |
403.7 |
4.8x |
615.1 |
3.6x |
msgspec
decoding into Struct types uses the least amount of memory, and is also the fastest to decode. This makes sense;Struct
types are cheaper to allocate and more memory efficient thandict
types, and for large messages these differences can really add up.msgspec
decoding without a schema is the second best option for both memory usage and speed. When decoding without a schema,msgspec
makes the assumption that the underlying message probably still has some structure; short dict keys are temporarily cached to be reused later on, rather than reallocated every time. This means that instead of allocating 10,000 copies of the string"name"
, only a single copy is allocated and reused. For large messages this can lead to significant memory savings.json
andorjson
also use similar optimizations, but not as effectively.orjson
andsimdjson
use 3-5x more memory thanmsgspec
in this benchmark. In addition to the reasons above, both of these decoders require copying the original message into a temporary buffer. In this case, the extra copy adds an extra 65 MiB of overhead!
Benchmark - Structs#
Here we benchmark common msgspec.Struct
operations, comparing their
performance against other similar libraries. The cases compared are:
msgspec
Standard Python classes
For each library, the following operations are benchmarked:
Time to define a new class. Many libraries that abstract away class boilerplate add overhead when defining classes, slowing import times for libraries that make use of these classes.
Time to create an instance of that class.
Time to compare two instances for equality (
==
/!=
).Time to compare two instances for order (
<
/>
/<=
/>=
)
The full benchmark source can be found here.
Results (smaller is better):
import (μs) |
create (μs) |
equality (μs) |
order (μs) |
|
---|---|---|---|---|
msgspec |
9.92 |
0.09 |
0.02 |
0.03 |
standard classes |
6.86 |
0.45 |
0.13 |
0.29 |
dataclasses |
489.07 |
0.47 |
0.27 |
0.30 |
attrs |
428.38 |
0.42 |
0.29 |
2.15 |
pydantic |
371.52 |
4.84 |
10.56 |
N/A |
Standard Python classes are the fastest to import (any library can only add overhead here). Still,
msgspec
isn’t that much slower, especially compared to other options.Structs are optimized to be cheap to create, and that shows for the creation benchmark. They’re roughly 5x faster than standard classes/
attrs
/dataclasses
, and 50x faster thanpydantic
.For equality comparison, msgspec Structs are roughly 6x to 500x faster than the alternatives.
For order comparison, msgspec Structs are roughly 10x to 70x faster than the alternatives.
Benchmark - Garbage Collection#
msgspec.Struct
instances implement several optimizations for reducing garbage
collection (GC) pressure and decreasing memory usage. Here we benchmark structs
(with and without gc=False) against standard Python
classes (with and without __slots__).
For each option we create a large dictionary containing many simple instances of the benchmarked type, then measure:
The amount of time it takes to do a full garbage collection (gc) pass
The total amount of memory used by this data structure
The full benchmark source can be found here.
Results (smaller is better):
GC time (ms) |
Memory Used (MiB) |
|
---|---|---|
standard class |
80.46 |
211.66 |
standard class with __slots__ |
80.06 |
120.11 |
msgspec struct |
13.96 |
120.11 |
msgspec struct with gc=False |
1.07 |
104.85 |
Standard Python classes are the most memory hungry (since all data is stored in an instance dict). They also result in the largest GC pause, as the GC has to traverse the entire outer dict, each class instance, and each instance dict. All that pointer chasing has a cost.
Standard classes with
__slots__
are less memory hungry, but still results in an equivalent GC pauses.msgspec.Struct
instances have the same memory layout as a class with__slots__
(and thus have the same memory usage), but due to deferred GC tracking a full GC pass completes in a fraction of the time.msgspec.Struct
instances withgc=False
have the lowest memory usage (lack of GC reduces memory by 16 bytes per instance). They also have the lowest GC pause (75x faster than standard classes!) since the entire composing dict can be skipped during GC traversal.
Benchmark - Library Size#
Here we compare the on-disk size of a few Python libraries.
The full benchmark source can be found here.
Results (smaller is better)
version |
size (MiB) |
vs. msgspec |
|
---|---|---|---|
msgspec |
0.12.0 |
0.34 |
1.00x |
orjson |
3.8.5 |
0.56 |
1.64x |
msgpack |
1.0.4 |
0.99 |
2.91x |
pydantic |
1.10.4 |
8.71 |
25.67x |
The functionality available in msgspec
is comparable to that of orjson,
msgpack, and pydantic combined. However, the total installed binary size of
msgspec
is a fraction of that of any of these libraries.