Your dashboard looks fine. Your data warehouse has been corrupted since Tuesday. Nobody knows yet.
This is FM9 — Silent Data Corruption. It is the failure mode that does not page. It does not throw. It does not spike a graph. It writes wrong values, the wrong values flow downstream, and the system agrees with itself the entire way.
FM4 — Data Consistency Failure is visible. Two systems disagree. Someone eventually notices. FM9 is invisible. Every system agrees on the wrong value.
The incident
A pipeline ingests events at the edge. It writes them to a queue. A worker consumes the queue, transforms each event, and writes the result to a warehouse. Dashboards read from the warehouse. Nothing has changed in the code for weeks.
On Tuesday, a producer at the edge added a field to its event payload. The receiving contract still accepted the old shape. No parser errors fired. No schema registry rejected the change. The pipeline kept running.
The new field shifted the byte layout of one downstream deserialiser. A 64-bit user ID got read into a 32-bit column. Values above two billion silently truncated. The database accepted every write. The warehouse absorbed every row.
By Saturday, the analytics team noticed that a specific customer cohort had vanished from a weekly report. The cohort had not vanished. Their IDs had collided with lower IDs, rewriting rows in place. Four days of enterprise usage data was gone.
This is FM8 — Schema / Contract Violation feeding FM9. The interaction is named in the source material as Pattern 3: the slow corruption. Schema change is silently misinterpreted. Incorrect data propagates. Systems disagree on state. The problem surfaces weeks later during reconciliation.
Why it forms
The source lists the specific mechanisms. Each one shows up in real incidents.
Missing validation at write time. The pipeline trusted the producer. It parsed the payload into a struct and wrote whatever came out. Nothing checked that the resulting values made sense for the target column.
Missing checksums at rest or in transit. The bytes on disk matched the bytes the writer sent. Both were wrong. Integrity checks confirm that the message was not corrupted in flight. They do not confirm that the message was correct when sent.
Non-idempotent operations retried. A retry after a partial write can double-apply a mutation. If the retry logic assumes the first attempt failed, and the first attempt actually succeeded, the second attempt corrupts the row.
Type coercions that silently truncate. A 64-bit ID stored in a 32-bit field. A decimal cast to an integer. A timestamp stored in a smaller precision. Every language and every database offers ways to do this without raising an error.
Double-processing under at-least-once delivery. Every modern queue delivers at-least-once by default. Without deduplication at the consumer, a redelivered event applies its side effect twice. The system runs. The counts are wrong.
Each of these looks harmless in code review. Each becomes an FM9 the moment the input distribution shifts.
Why nothing paged
This is the part that matters. Every other failure mode in the F3 catalogue eventually surfaces to a metric.
FM1 shows up as a health check failure. FM2 shows up as a rising error rate across dependants. FM3 shows up as memory or connection graphs climbing to a ceiling. FM5 shows up as P99 latency drift. FM7 shows up as a QPS spike.
FM9 does not produce any of these. The system throughput is normal. The error rate is normal. The latency is normal. The disks are filling at their usual pace. Every operational signal reports a healthy pipeline. The only signal is the data itself, and nobody was watching the data.
This is the tradeoff being made. AT9 — Correctness vs Performance. Every validation check adds cost at the boundary. Every checksum adds cost per record. Every reconciliation job spends compute comparing a source of truth to a derived copy. Teams under throughput pressure remove these checks to make numbers go up. The checks that remain protect the parts of the pipeline that already had incidents.
Choosing performance is not wrong. Choosing it without naming what you gave up is what produces the four-day gap.
How to detect it
The source names four detection mechanisms. They are not optional monitoring. They are the only way FM9 becomes visible before it hits a report.
End-to-end reconciliation. Compare the derived system to the source of truth on a schedule. Row counts. Aggregate sums. Hash of a sample. Any discrepancy is an alert. Reconciliation is the only mechanism that catches corruption that the pipeline itself validated as clean.
Checksums on stored data. Not the storage-layer checksum that catches disk rot. An application-level checksum written with the record. If the checksum recomputes to a different value on read, the record was tampered with or written wrong.
Anomaly detection on data distributions. The daily count of active users has a shape. When the shape breaks — a cohort disappears, a value distribution shifts, a null rate jumps — the alert fires on the data, not the infrastructure. Most teams alert on infrastructure. FM9 lives in the gap.
Regular data audits. Compare expected values to actual values. This is not automated monitoring. It is a scheduled job that a human reviews. It is the last line of defence for corruption that flows through every automated check.
How to prevent it
Four preventions map directly to the four ways FM9 forms.
Idempotency keys for all write operations. Every mutation carries a client-generated key. The receiver stores keys and rejects duplicates. At-least-once delivery becomes exactly-once at the application layer. This closes the retry double-apply path.
Input validation at every system boundary. Not only at the user-facing edge. Every service that receives data from another service validates it against a schema. Range checks. Type checks. Referential checks where cheap. The producer contract is not enough — the consumer verifies.
Checksums on stored data. The consumer writes a checksum with the record. The reader recomputes and compares. A silent bit flip, a partial write, a truncated field — each produces a mismatch.
Reconciliation jobs. A scheduled comparison between derived state and source of truth. Run it hourly for critical data. Alert on any drift. This is the only mechanism that catches an FM9 that already happened.
The interaction that makes it worse
FM9 rarely arrives alone. The source names two dangerous pairs.
FM8 + FM9 is the pattern in the incident above. A schema or contract changed on one side of a boundary. The other side kept parsing. The parse did not fail — it silently misinterpreted. Wrong values flowed downstream and looked right. This is why schema registries and consumer-driven contract tests exist. They convert an FM9 into an FM8 that fails loudly at deploy time.
FM4 + FM9 is subtler. A consistency failure produced two versions of a value. If some system alerted on the disagreement, it would surface as FM4. If nothing alerted, the wrong value propagated as FM9. The presence or absence of a reconciliation check decides which failure mode you experience.
The signal that tells you this applies to your system
You have a data pipeline with more than two hops. Somewhere along the way, a service transforms records into a shape that a downstream service consumes. You have no reconciliation job comparing the final store to the source. You have no schema registry gating producer changes. Your dashboards read from a derived store, and the derived store has never had a discrepancy alert fire.
The pipeline has not corrupted anything visible yet. That is not the same as being correct. That is the same as not having looked.
The question worth sitting with: if a field silently truncated at midnight tonight, how would you find out, and how many days of data would you lose before you did?
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in the Reference Book, Chapter 7 (Failure Modes). Free chapter available at computingseries.com/books/ref.