A user clicks "place order." The request fans out to payment, inventory, notification, and a database. Each of those calls its own dependencies. The request takes 4.2 seconds. The user abandons the page after 3. An engineer opens the logs of six services and starts correlating timestamps by hand.
Two hours later, she finds it. The notification service was calling an email provider with a 3-second timeout. The other five services were innocent. With distributed tracing, she would have found it in five minutes.
This is the component that makes that difference.
The concept: trace, span, context
A trace is the complete execution of one request across every service it touched. A span is one unit of work inside that trace. Every span carries a start time, duration, service name, operation name, status, and tags.
Each span has one parent and zero or more children. The tree of spans is the trace. A single trace ID links every span belonging to the same request. A span ID identifies each unit uniquely.
The tree structure is not stored explicitly. It is reconstructed from parent span IDs. The root span has no parent. Every other span points at the one that called it.
This is Thread T3 — Data Structures applied to request execution. Book 2's graph traversal walked static graphs of nodes and edges. A distributed trace is the same tree, built at runtime from a real request's fan-out. Depth-first walks compute critical paths. Breadth-first walks group spans by level. The algorithms transfer directly.
Thread T11 — Observability places tracing as the third pillar next to metrics and logs. Metrics tell you the system is slow. Logs tell you what one service did. Only traces show you the call graph.
Context propagation: how the trace stays whole
Without a mechanism to carry the trace ID across service boundaries, each service starts its own isolated trace. There is no way to join them later. Context propagation is that mechanism.
The W3C TraceContext standard defines the traceparent HTTP header. The format packs a version, the trace ID, the parent span ID, and a sampled flag into one string. Service A puts the header on every outgoing call. Service B reads the header on every incoming request.
When Service A calls Service B, it creates a child span, injects the header, and makes the request. Service B parses the header, creates its own span with the received span ID as its parent, and processes the request. The two spans now live in the same trace, linked by trace ID and parent pointer.
Miss one boundary and the trace breaks. The downstream service starts a new orphaned trace. In the original trace, the call becomes an opaque span with a duration but no internal detail.
Sampling: head-based vs tail-based
Storing every span from every request costs too much. Sampling decides which traces survive.
Head-based sampling makes the decision at the start of the request. The decision goes into the traceparent flags. Every downstream service respects it. It is fast, cheap, and stateless. It has one weakness: rare errors may never be sampled. If the error rate is 0.1% and the sampling rate is 1%, only 0.001% of error traces are captured.
Tail-based sampling makes the decision after the request completes. All spans go to a buffer. When the trace is complete, an agent decides to keep or discard it based on the full outcome. Errors are always kept. Slow traces are always kept. Normal traces are sampled at a lower rate.
Tail-based sampling captures the events that matter most. The cost is that every span must be collected before the decision. At 50,000 requests per second with 10 spans per request, that is 500,000 spans per second flowing into the agent. The agent must buffer every in-flight trace.
This is AT2 — Latency vs Throughput. Head-based sampling optimises throughput at the cost of missing rare traces. Tail-based sampling optimises debugging correctness at the cost of far higher collection overhead.
The deeper tradeoff behind the whole system is AT9 — Correctness vs Performance. A trace tells you the correct answer about where a request spent its time. Every span you record costs CPU, memory, and network. You are choosing how much production overhead to accept in exchange for how much diagnostic truth.
Critical path: the only span worth optimising first
The critical path is the longest sequence of dependent spans in a trace. It determines the total duration. Optimising a span that is not on the critical path saves nothing.
Consider a trace where order-service calls three children in parallel — payment (200ms), inventory (50ms), postgres (30ms) — then calls notification (3100ms) sequentially. The parallel branch takes max(200, 50, 30) = 200ms. The sequential notification adds 3100ms. The critical path runs through notification. Cutting inventory from 50ms to 10ms changes nothing.
The algorithm is a tree walk. For each span, sum children's durations if they run sequentially, take the maximum if they run in parallel. Recurse. The heaviest path from root to leaf is the critical path. Optimisation targeting anything else has expected impact of zero.
This is where trace data pays for itself. Without the tree, an engineer guesses. With the tree, she picks the one span that governs the total and ignores the rest.
Where it fails
FM11 — Observability Blindness. The dominant failure is missing context propagation. A service processes an incoming request correctly. It then calls a downstream service using an HTTP client that does not forward headers. The downstream service starts a new, orphaned trace. In the original trace, the downstream work becomes a single opaque span. The engineer sees the latency but not the cause. Automatic instrumentation libraries fix this for common HTTP clients without code changes. Custom or legacy clients require manual work.
FM5 — Latency Amplification, invisible edition. A request calls a third-party API you cannot instrument. The call takes 2 seconds. In the trace, it appears as a single 2-second leaf span with no internal structure. The critical path analysis names it as the bottleneck. The trace tells you nothing about why. External spans must be tagged with enough context — URL, operation, error codes — to diagnose without internal visibility.
Async transports break traces silently. HTTP forwards traceparent naturally. Kafka, RabbitMQ, SQS, and cron jobs do not. Each requires explicit context injection into the message payload. Miss one stage and the pipeline splits into fragments. The debugging value of tracing is proportional to the fraction of the call graph that is instrumented. A 90% instrumented system leaves 10% of its graph invisible — and that 10% tends to be where the hard bugs live.
Sampling bias distorts performance analysis. Head-based sampling at 1% captures a representative sample of typical requests. It underrepresents any specific customer, endpoint, or region with systematically worse performance. Aggregate percentiles computed from the sample can miss systemic issues affecting one segment. Stratified sampling — sample all traffic to specific endpoints — supplements uniform sampling for targeted investigation.
The signal that tells you this applies to your system
A user-facing request touches four or more services. Latency spikes are diagnosed by reading logs from each service and correlating timestamps by hand. Investigations that should take minutes take hours. Nobody can point at the one service on the critical path with confidence.
A related signal: your trace UI shows a span labelled "external service call" or "database query" with a multi-second duration and no children. You have a bottleneck with no diagnostic visibility. Either instrument the call, or tag the span with enough metadata to diagnose without it.
A third signal: the same request traced twice produces two different-looking trees because context is dropped at some boundary. The graph you are debugging is not the graph the system actually executed.
The harder question
Every trace you keep costs storage. Every span you drop is a debugging story you can never recover. You cannot store everything. You cannot afford to lose the traces you will need six weeks from now for the incident that has not happened yet.
How do you design a retention policy that keeps the traces you will want to look at without knowing in advance which ones those are?
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in Book 3: Scaling Systems, Chapter 37 (Distributed Tracing). Free chapter available at computingseries.com/books/book3.