Benchmarks are hard.
Repeatedly calling the same function in a tight loop will lead to the instruction cache staying hot and branches being highly predictable. That’s not representative of real world access patterns. It’s also hard to write a nonbiased benchmark. I wrote quickle, naturally whatever benchmark I publish it’s going to perform well in.
Even so, people like to see benchmarks. I’ve tried to be as nonbiased as I can be, and the results hopefully indicate a few tradeoffs you make when you choose different serialization formats. I encourage you to write your own benchmarks before making these decisions.
Here we show a simple benchmark serializing some structured data. The data
we’re serializing has the following schema (defined here using
import quickle from typing import List, Optional class Address(quickle.Struct): street: str state: str zip: int class Person(quickle.Struct): first: str last: str age: int addresses: Optional[List[Address]] = None telephone: Optional[str] = None email: Optional[str] = None
The libraries we’re benchmarking are the following:
msgpack- msgpack with dict message types
orjson- orjson with dict message types
pyrobuf- pyrobuf with protobuf message types
pickle- pickle with dict message types
quickle- quickle with dict message types
Each benchmark creates one or more instances of a
Person message, and
serializes it/deserializes it in a loop. The full benchmark code can be found
Benchmark - 1 Object¶
Some workflows involve sending around very small messages. Here the overhead
per function call dominates (parsing of options, allocating temporary buffers,
etc…). Libraries like
msgpack, where internal structures
are allocated once and can be reused will generally perform better here than
pickle, where each call needs to allocate some temporary
You can use the radio buttons on the bottom to sort by total roundtrip time, dumps (serialization) time, loads (deserialization) time, or serialized message size.
From the chart above, you can see that
quickle structs is the fastest
method for both serialization and deserialization. It also results in the
second smallest message size (behind
pyrobuf). This makes sense, struct
types don’t need to serialize the fields in each message (things like
last, …), only the values, so there’s less data to send
around. Since python is dynamic, each object serialized requires a few pointer
chases, so serializing fewer objects results in faster and smaller messages.
I’m actually surprised at how much overhead
pyrobuf has (the actual
protobuf encoding should be pretty efficient), I suspect there’s some
optimizations that could still be done there.
That said, all of these methods serialize/deserialize pretty quickly relative to other python operations, so unless you’re counting every microsecond your choice here probably doesn’t matter that much.
Benchmark - 1000 Objects¶
Here we serialize a list of 1000
Person objects. There’s a lot more data
here, so the per-call overhead will no longer dominate, and we’re now measuring
the efficiency of the encoding/decoding.
As with before
quickle structs and
quickle both perform well here.
What’s interesting is that
orjson have now moved to the
back for deserialization time.
The reason for this is memoization. Since each message here is structured
(all dicts have the same keys),
orjson are serializing the
same strings multiple times. In contrast,
support memoization - identical objects in a message will only be serialized
once, and then referenced later on. This results in smaller messages and faster
deserialization times. For messages without repeat objects, memoization is an
added cost you don’t need. But as soon as you get more than a handful of
repeat objects, the performance win becomes important.
pickle tuples, and
pyrobuf don’t require
memoization to be efficient here, as the repeated field names aren’t serialized
as part of the message.
Benchmark - 10,000 Objects¶
Here we run the same benchmark as before, but 10,000
Like the 1000 object benchmark, the cost of serializing/deserializing repeated
strings dominate for the