A team deploys a schema migration on a Tuesday morning. Everything looks fine. Four hours later, a downstream service starts writing corrupted records. Nobody made an alarm ring. Nobody rolled anything back. The producer changed the contract. The consumer kept running.
This is FM8 — Schema/Contract Violation. It is one of the few failure modes that does not announce itself. It waits.
The incident
Consider the shape of the failure. One service publishes to a Kafka topic. Six others consume from it. The producer team ships an Avro schema change — a field renamed, or an enum value removed, or a semantic that used to mean one thing now meaning another. The change passes code review. It passes tests. It ships.
The consumers keep deserialising. Some field they used to read is now empty. Some enum value they used to switch on now falls through the default branch. Some numeric field that used to be dollars is now cents. No exception fires. The consumer writes the misinterpreted value to its own database.
Four hours later, the reconciliation job at the end of the day flags a discrepancy. The corruption is already downstream of the point of ingestion. Every consumer of the affected data has now written derived state based on wrong inputs.
This is why FM8 is dangerous. The contract violation itself is invisible. What you see is the second-order failure — FM9, Silent Data Corruption — arriving hours or days later.
What FM8 actually is
FM8 is one side of a system boundary changing its interface, data format, or semantic contract without coordinating with the other side. The receiving side breaks.
The boundary can be anything. An API removes a field from its response. A Kafka topic changes its Avro schema. A database table renames a column. A shared library changes the meaning of a function argument. In every case, the producer changed the contract without ensuring all consumers were updated first.
The failure mode does not require malice. It does not require carelessness. It requires only the belief that a change is safe because "nothing else uses that field." That belief is almost always wrong.
Hyrum's Law states the reason. Every observable behaviour of your interface, given enough consumers, is depended on by somebody. If your API returns fields in a certain order, someone parses that order. If your enum values are numbered from zero, someone hard-coded the numbering. If your timestamps arrive with millisecond precision, someone assumes millisecond precision. Treat every observable as a contract, because someone is treating it as one.
The tradeoff behind the failure
FM8 is where AT3 — Simplicity vs Flexibility does its damage.
The simple change is the one that removes the field, renames the column, or reshapes the response. The producer's code becomes cleaner. The producer's tests still pass. The producer's diff is small.
The flexible change is the one that preserves every previous observable. Add the new field alongside the old. Keep the deprecated enum value. Emit both the old shape and the new shape until every consumer has migrated. The producer's code becomes bigger. The migration lives across many deployments. The diff is not small.
Every team feels the pull toward simplicity. The producer team sees the field as theirs. The consumer teams see the field as a contract. Both are right about their own view. Only one view survives contact with production.
When you choose AT3 — Simplicity, you are choosing the producer's convenience over the consumer's stability. That is a legitimate choice when there are no consumers. It stops being legitimate the moment there is one.
How FM8 forms in practice
Three patterns produce most FM8 incidents.
Pattern one: the invisible consumer. The producer team knows about the three consumers they built for. They do not know about the fourth consumer that another team added last quarter. The producer's change is safe against the three known consumers. The fourth breaks silently.
Pattern two: the semantic drift. The field name stays the same. The type stays the same. The meaning changes. A status field used to include pending for jobs waiting on a queue; now pending only means waiting on manual approval, and queued jobs get a new state. Every consumer that switched on status == "pending" is now wrong. No schema check catches this. No test catches this. The type system cannot see meaning.
Pattern three: the removed edge case. The producer removes an enum value that "nobody uses anymore." One consumer's default branch treats an unknown enum as a benign no-op. That consumer's downstream state is now wrong for the removed case, and no error propagates. This is the direct bridge from FM8 into FM9.
Where FM8 leads
FM8 rarely stays alone. It is a producer of other failure modes.
The most common compound is FM8 → FM9. The contract violation causes fields to be silently misinterpreted rather than causing an explicit error. Incorrect data propagates. Nothing alerts. The corruption spreads through every derived system that reads from the affected consumer. When you finally discover it, the corruption is now upstream of dashboards, ML training data, billing rollups, and audit logs.
The reference material identifies this as Pattern 3 — the slow corruption: FM8 (schema change silently misinterpreted) → FM9 (incorrect data propagates) → FM4 (systems disagree on state) → discovered weeks later during reconciliation.
The reason this pattern is dangerous is the delay. FM4, Data Consistency Failure, is visible. Systems disagree — one side says A, another says B, and the disagreement itself is a signal. FM9 is invisible. Systems agree on the wrong value. There is nothing to page on. The failure is discovered by a human noticing a number that does not make sense, weeks after the event.
Detection: what actually catches FM8
Two mechanisms reliably catch FM8 before production.
Consumer-driven contract testing. Every consumer publishes a specification of what it expects from the producer. The producer's CI pipeline runs those specifications against every proposed change. If the change breaks any consumer's spec, the pipeline fails. The producer cannot ship a change that a known consumer would reject.
Schema registries. For Kafka, a schema registry enforces compatibility rules on every schema version. Backward-incompatible changes are rejected at registration time, not at deserialisation time in production. Similar registries exist for API schemas — OpenAPI diff tools, GraphQL schema linting, Protobuf compatibility checkers. The rule is the same: incompatible changes fail at the boundary before they ever reach a running consumer.
Neither of these mechanisms is exotic. Both exist in mature form. The reason FM8 keeps happening is that teams treat schema changes as internal changes. They are not. Every schema change is a change to a public interface, whether or not the interface has that word in its name.
Prevention: the discipline
The prevention rule is simple to state.
Backward-compatible changes only. Add fields, never remove them. Add enum values, never remove them. Never change field semantics.
When a real removal is needed, do it in three deployments, not one. Deployment one adds the new field alongside the old. Deployment two migrates every consumer to the new field, verified by the schema registry. Deployment three removes the old field, after all consumers have shipped and been observed reading only the new one. Each of the three deployments is backward-compatible on its own. Only the sequence removes the field.
This is slower than the one-deployment version. It is also the only version that does not depend on knowing every consumer in advance — which you do not.
The signal that tells you this applies to your system
Your team ships a schema change and pages nobody. Your producer's CI does not run consumer contract tests. Your Kafka topics have no registered schema, or the registered schema is not enforced. Your API changes are reviewed for producer correctness, not for consumer compatibility.
If any of these describe your pipeline, FM8 is already sitting in your backlog waiting to trigger. The question is not whether you will ship a breaking change. The question is how long the resulting corruption will run before somebody notices a number that does not add up.
Draw your service map. For every arrow into your service, ask who owns the other end. For every arrow out of your service, ask who owns the other end. Now ask: how would either side know if the other changed the contract? If the answer is "we would find out in production," you have named your next incident before it happens.
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in the Reference Book, Chapter 7 (Failure Modes). Free chapter available at computingseries.com/books/ref.