Every byte on disk, every field on the wire, every column in a table is an encoding decision. You made it once. You will live with it forever.
Most engineers meet encoding as a series of separate choices. UTF-8 in one course. JSON in another. Protobuf in a system design interview. Database schemas in a fourth. They are not separate choices. They are the same decision, repeated at every layer of the stack.
This article traces T10 — Encoding from its root form to its production form. The pattern is identical at every level. The constraint it imposes is identical. The failure mode is identical.
The root form
Information can be represented in multiple ways. The choice of representation determines what operations are efficient, what invariants hold, and what can cross a system boundary.
That is the root. Every subsequent encoding decision is a specialisation of it.
The reason T10 recurs is direct. Every serialisation is an encoding decision. Every API contract is an encoding decision. Every data schema is an encoding decision. Every compression algorithm is an encoding decision. The choice determines how the data can be processed, validated, and evolved.
The tradeoff is always the same. A denser encoding costs more to produce and read. A more permissive encoding costs more to validate. A schema that anticipates change costs more to design. You pay somewhere.
Level 1 — Binary representation
The first encoding decision is the smallest. A number becomes a sequence of bits. An integer takes 32 bits or 64. A float takes a sign bit, an exponent, and a mantissa.
The tradeoff appears immediately. A 32-bit integer is compact but overflows at roughly two billion. A 64-bit integer holds larger values but doubles the storage. You cannot have both.
The failure mode also appears immediately. Data written with one binary layout cannot be read by code expecting another. Endianness disagreements corrupt values silently. Precision limits in floats produce results that look right until they do not.
Level 2 — Character encoding
Text is the next layer. A byte sequence must be interpreted as characters. ASCII maps 128 characters to seven bits. UTF-8 maps every Unicode character to a variable-length sequence of bytes.
The tradeoff is capacity against uniformity. ASCII is fixed-width and cheap to index. UTF-8 handles every script on earth but requires walking the byte sequence to count characters.
The failure mode is familiar to anyone who has read a corrupted file. Bytes written as UTF-8 and read as ASCII produce mojibake. The bytes are unchanged. The interpretation is wrong. The encoding is the contract, and the contract was broken on one side of the boundary.
Level 3 — Data serialisation
The same idea moves up a layer. Now a structured object — a record with fields — must become a byte sequence for storage or transmission.
JSON encodes each field as a name-value pair in text. Protobuf encodes each field with a numeric tag and a compact binary value. Avro encodes the values without field names, using a separately stored schema to interpret them.
The tradeoff is explicit at this level. JSON is human-readable and self-describing but larger and slower to parse. Protobuf is compact and fast but requires the schema to interpret the bytes. Avro pushes the schema out of the message entirely, cutting message size at the cost of coordinating schemas between producer and consumer.
Each format encodes a bet about how the data will change.
Level 4 — Schema evolution
This is where encoding stops being a static choice and becomes a lifecycle problem.
Once a system is in production, the schema must change. New fields appear. Old fields lose their meaning. Types shift. The encoding must accommodate this without breaking every reader and every writer at the same time.
Protobuf answers with field numbering. Every field carries a stable numeric tag. Adding a new field means picking an unused number. Old readers skip tags they do not recognise. New writers include the new tag. The old and new versions coexist. The cost is a rule you can never break: you never reuse a field number. The number is part of the contract, forever.
Avro answers differently. Every message is written against a schema, and the schema is stored in a registry. Readers fetch the writer's schema from the registry. They compare it to their own and resolve the differences at read time. This allows fuller evolution — renames, defaults, type changes — because the resolution is explicit. The cost is a coordination service that must be available whenever data is read.
Two encodings. Two bets. Protobuf bets that field numbers are cheaper to preserve than a schema registry is to operate. Avro bets the opposite.
Level 5 — API and database schemas
The same pattern climbs one more level. An API endpoint has a schema — the shape of its request and response. A database table has a schema — the columns and their types. Both are encoding decisions. Both must evolve.
An API that adds a required field breaks every client that does not yet send it. An API that removes a field breaks every client that still reads it. The safe evolutions are additive: new optional fields, new endpoints, new response fields that old clients ignore.
A database column change follows the same logic. Adding a nullable column is safe. Adding a required column requires a default or a migration. Renaming a column breaks every query written against the old name. Dropping a column breaks every consumer that still reads it.
The tradeoff and the failure mode are the same as at the serialisation level. The scale is larger. The blast radius is wider.
Level 6 — Vector embeddings
The most recent layer is the strangest. A vector embedding encodes semantic meaning as a point in a high-dimensional space. "King" and "queen" land near each other. "King" and "bicycle" do not.
This is still an encoding decision. The model chosen to produce the embedding is the schema. The dimensionality is the layout. Two embeddings from different models are not comparable — the coordinate system is different, even if the vector lengths match.
The same failure mode appears. Vectors written by one model and read by code expecting another produce nonsense that looks like signal. The bytes parse. The distances compute. The results are wrong.
The law hiding under all of this
Hyrum's Law names the constraint that binds every encoding decision above. Once a system is in use, all observable behaviours become part of the implicit contract. This includes the encoding of the response.
You can change the content. You cannot change the structure without breaking callers who depended on a detail you thought was an implementation choice.
This is why encoding decisions compound. The field you added carelessly is now load-bearing. The optional field a client started requiring is now required in practice. The character encoding you never documented is now assumed. Every observable byte is a promise you did not know you made.
What this means for your system right now
Every encoding you own is a bet on how the data will change. The bet was placed when the format was chosen. It is being paid every time the data crosses a boundary.
What is your schema evolution story? If you use JSON, the story is discipline — additive changes only, no field renames, tolerant readers. If you use Protobuf, the story is field numbering — a permanent ledger of numbers you have used. If you use Avro, the story is the registry — a service that must be available for reads to work.
What is your Hyrum surface? Every observable property of your responses is now part of the contract. Field order in JSON. Trailing whitespace. Numeric precision. The exact wording of error messages. Something you did not intend to expose is being depended on right now.
What happens when the schema is wrong? A message written under one schema and read under another may parse successfully and mean something different. Silent corruption is the worst failure mode at this level. Loud failures you can debug. Quiet ones you discover months later.
The signal that tells you this applies to your system
You have a producer and a consumer written by different teams. The message format was agreed once and never revisited. New fields have been added without a review. Old fields have quietly stopped being populated. No one is certain which fields are still required.
That system is one careless change away from a Hyrum failure. The encoding is doing more work than anyone remembers agreeing to.
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in the Reference Book, Chapter 3 (Twelve Threads, T10). Free chapter available at computingseries.com/books/ref.