2. Trace Error Propagation Across Microservices

Why Error Attribution Needs the Call Graph, Not Just the Error List

In a microservice incident, the service that logs the most errors is rarely the service that caused them. A database connection pool exhaustion in payment-db-proxy surfaces as HTTP 500s in checkout-api, retry storms in order-service, and timeout warnings in the API gateway. If you rank services by error count, you blame the loudest victim. The correct attribution requires reconstructing the actual request topology from distributed traces and then walking each failing trace upstream to find the earliest service whose own span turned anomalous — before any of its children failed.

This case builds a trace-based analyzer that does exactly that: it ingests spans in OpenTelemetry-style records, rebuilds per-trace call graphs from parentSpanId links, marks spans as anomalous using explicit error status plus latency deviation, and then attributes each failing trace to the deepest upstream anomaly that is not merely a consequence of a failing child.

Input Model and the Attribution Rule

Each span record carries the fields needed to rebuild structure and judge health. The analyzer assumes one trace per user request and a single root span per trace.

The attribution rule is deliberately simple and defensible: a span is anomalous if status == "ERROR" or its duration exceeds a per-service latency threshold. A failing trace is attributed to the earliest anomalous span — the anomalous span with no anomalous ancestor. Downstream anomalous spans are recorded as victims and excluded from the root-cause candidate set. This is what separates the originator from the propagation chain.

Reconstructing Call Graphs and Detecting Propagation

The implementation below is self-contained and runnable. It builds an adjacency map per trace, computes anomaly flags, finds earliest anomalous spans, and reports the propagation path from originator to victims.

from collections import defaultdict

# --- Example trace data (illustrative, not measured) ---
spans = [
    {"traceId": "t1", "spanId": "s1", "parentSpanId": "",   "service": "gateway",       "status": "OK",    "durationMs": 820},
    {"traceId": "t1", "spanId": "s2", "parentSpanId": "s1", "service": "checkout-api",  "status": "ERROR", "durationMs": 790},
    {"traceId": "t1", "spanId": "s3", "parentSpanId": "s2", "service": "order-service", "status": "ERROR", "durationMs": 760},
    {"traceId": "t1", "spanId": "s4", "parentSpanId": "s3", "service": "payment-db-proxy","status": "ERROR","durationMs": 740},
    {"traceId": "t1", "spanId": "s5", "parentSpanId": "s4", "service": "postgres",      "status": "ERROR", "durationMs": 720},
    {"traceId": "t2", "spanId": "u1", "parentSpanId": "",   "service": "gateway",       "status": "OK",    "durationMs": 120},
    {"traceId": "t2", "spanId": "u2", "parentSpanId": "u1", "service": "checkout-api",  "status": "OK",    "durationMs": 95},
]

# Per-service latency thresholds (ms) from a healthy baseline window
latency_threshold = {"gateway": 500, "checkout-api": 300,
                     "order-service": 300, "payment-db-proxy": 200, "postgres": 150}

def is_anomalous(span):
    return span["status"] == "ERROR" or span["durationMs"] > latency_threshold.get(span["service"], 10**9)

def build_graph(trace_spans):
    by_id = {s["spanId"]: s for s in trace_spans}
    children = defaultdict(list)
    for s in trace_spans:
        if s["parentSpanId"]:
            children[s["parentSpanId"]].append(s["spanId"])
    return by_id, children

def attribute(trace_spans):
    by_id, children = build_graph(trace_spans)
    anomalous = {sid for sid, s in by_id.items() if is_anomalous(s)}
    # Earliest anomalous = anomalous span with no anomalous ancestor
    origins = []
    for sid in anomalous:
        p = by_id[sid]["parentSpanId"]
        has_anomalous_ancestor = False
        while p:
            if p in anomalous:
                has_anomalous_ancestor = True
                break
            p = by_id[p]["parentSpanId"]
        if not has_anomalous_ancestor:
            origins.append(sid)
    return origins, anomalous, by_id, children

def propagation_path(origin_id, by_id, children):
    path, stack = [], [origin_id]
    while stack:
        sid = stack.pop()
        path.append(by_id[sid]["service"])
        stack.extend(children[sid])
    return path

traces = defaultdict(list)
for s in spans:
    traces[s["traceId"]].append(s)

for tid, tspans in traces.items():
    origins, anomalous, by_id, children = attribute(tspans)
    if not origins:
        print(f"{tid}: healthy, no attribution")
        continue
    for o in origins:
        victims = [by_id[a]["service"] for a in anomalous if a != o]
        print(f"{tid}: origin={by_id[o]['service']}  "
              f"path={' -> '.join(propagation_path(o, by_id, children))}  "
              f"victims={victims}")

Running this against the example data prints:

t1: origin=payment-db-proxy  path=payment-db-proxy -> postgres  victims=['checkout-api', 'order-service', 'postgres']
t2: healthy, no attribution

Note the key result: checkout-api and order-service both carry ERROR status and would top an error-count ranking, yet the analyzer attributes the incident to payment-db-proxy because it is the earliest anomalous span — its parent order-service is anomalous only because its child failed. The postgres span is a downstream victim of the proxy's failure, not the originator.

Reading the Result and Guarding Against False Attribution

The output gives you two things at once: the attributed origin service and the propagation path. The path is what you hand to an on-call engineer — it shows the failure entered at payment-db-proxy and fanned out to postgres and up the call chain. Two practical guards keep the attribution honest:

• Missing spans break the chain. If a service does not propagate traceId/parentSpanId, the graph fragments and the "earliest anomalous" span may be a victim whose real parent is invisible. Check for orphan spans (non-empty parentSpanId with no matching parent) and report them rather than silently attributing.
• Fan-out inflates victims. A single origin with many children produces many "victims"; that is expected. Do not treat victim count as severity — use it to size blast radius, not to rank root causes.

Verification is straightforward: for a known injected fault (e.g., force payment-db-proxy to return errors), the analyzer should attribute to that service and list its ancestors as victims. If it instead attributes to checkout-api, the parent links or thresholds are wrong — inspect the reconstructed graph for that traceId before trusting the output.