2. Stack Trace Symbolication and Grouping

Why raw addresses are not signatures

A crash report that arrives from a client typically contains a stack trace made of module-relative offsets or absolute virtual addresses, for example CrashRpt!0x1a4f or 0x7ff6c3d21a4f. Those numbers are meaningless for grouping: ASLR relocates modules on every launch, compiler and linker versions shift code layout, and different builds produce different offsets for the same logical fault. Before any clustering can happen, each frame must be resolved to a symbol — a function name, and ideally a source file and line — using the matching debug symbol files (PDB on Windows, DWARF/.debug on Linux/macOS, dSYM on Apple platforms). Only after symbolication can you build a stable, comparable signature.

The second half of the task is equally important: grouping must be deterministic and must not silently merge unrelated crashes. A signature derived from unresolved frames (module+0x1a4f) is not a real cluster — it is a symbolication failure that happens to look like one. This case builds a small pipeline that resolves frames, constructs normalized signatures, groups reports, and explicitly separates "unresolved" buckets from genuine crash clusters.

Signature construction rules

A normalized stack signature is built from the resolved frames after applying a fixed set of transformations. The rules below are the contract the implementation must honor; they are what makes grouping reproducible across machines and runs.

• Frame selection: take the top N frames (default 8) starting from the faulting frame. Deep recursion and framework boilerplate add noise, so cap depth.
• Noise filtering: drop frames matching a denylist of runtime/allocator/exception-dispatch symbols (e.g. RaiseException, malloc, std::terminate, __chkstk) that appear in almost every crash and carry no discriminating signal.
• Normalization: strip template arguments, parameter lists, and namespace/ABI decorations down to a canonical form; lowercase; collapse whitespace. Keep the module name only when it disambiguates (same function name in two modules).
• Hashing: join normalized frames with | and hash (SHA-1 truncated to 16 hex chars) to produce a compact signature_id. Store the readable frame list alongside for debugging.
• Resolution status: every frame carries a status — resolved, module_only (module known, symbol missing), or unknown. A signature is trusted only if the faulting frame and at least the first three frames are resolved.

Pipeline architecture

The diagram below shows the data flow from raw report to grouped cluster, including the branch that diverts untrusted signatures into a separate symbolication-failure bucket instead of the crash clusters.

Reference implementation

The following Python module is self-contained and runnable. It defines a small symbol store keyed by (module, build_id), resolves frames, builds signatures, and groups reports. The example data is synthetic and included so the script runs end-to-end without external files.

import hashlib, re, json
from collections import defaultdict

# --- Symbol store: (module, build_id) -> {offset: symbol} ---
# In production this is populated from PDB/DWARF parsing or a symbol server.
SYMBOLS = {
    ("CrashRpt", "b1a2"): {
        0x1a4f: "CrashRpt!Renderer::drawFrame(int)",
        0x1b02: "CrashRpt!Renderer::flush()",
        0x1c10: "CrashRpt!main",
    },
    ("CrashRpt", "c3d4"): {  # different build, shifted offsets
        0x1a80: "CrashRpt!Renderer::drawFrame(int)",
        0x1b33: "CrashRpt!Renderer::flush()",
        0x1c41: "CrashRpt!main",
    },
}

NOISE = re.compile(r"(RaiseException|malloc|free|std::terminate|__chkstk|abort)", re.I)
TEMPLATE = re.compile(r"<[^>]*>")
PARAMS = re.compile(r"\([^)]*\)")

def normalize_symbol(sym):
    s = TEMPLATE.sub("", sym)
    s = PARAMS.sub("", s)
    s = s.lower().strip()
    return re.sub(r"\s+", "", s)

def resolve_frame(module, build_id, offset):
    table = SYMBOLS.get((module, build_id))
    if table is None:
        return {"module": module, "symbol": None, "status": "unknown"}
    sym = table.get(offset)
    if sym is None:
        return {"module": module, "symbol": None, "status": "module_only"}
    return {"module": module, "symbol": sym, "status": "resolved"}

def build_signature(frames, top_n=8):
    kept = [f for f in frames if f["status"] == "resolved"
            and not NOISE.search(f["symbol"])][:top_n]
    if not kept:
        return None, kept
    joined = "|".join(normalize_symbol(f["symbol"]) for f in kept)
    sig = hashlib.sha1(joined.encode()).hexdigest()[:16]
    return sig, kept

def is_trusted(frames):
    resolved = [f for f in frames if f["status"] == "resolved"]
    return len(resolved) >= 4 and frames[0]["status"] == "resolved"

def process(report):
    frames = [resolve_frame(m, b, o) for (m, b, o) in report["frames"]]
    sig, kept = build_signature(frames)
    trusted = is_trusted(frames)
    return {
        "report_id": report["id"],
        "signature_id": sig if trusted else None,
        "trusted": trusted,
        "frames": kept,
        "bucket": "cluster" if trusted else "symbolication_failure",
    }

# --- Synthetic example reports ---
REPORTS = [
    {"id": "r1", "frames": [("CrashRpt","b1a2",0x1a4f),("CrashRpt","b1a2",0x1b02),("CrashRpt","b1a2",0x1c10)]},
    {"id": "r2", "frames": [("CrashRpt","c3d4",0x1a80),("CrashRpt","c3d4",0x1b33),("CrashRpt","c3d4",0x1c41)]},
    {"id": "r3", "frames": [("CrashRpt","dead",0x1a4f),("CrashRpt","dead",0x1b02)]},
]

results = [process(r) for r in REPORTS]
clusters = defaultdict(list)
for r in results:
    key = r["signature_id"] or "UNRESOLVED"
    clusters[key].append(r["report_id"])

print(json.dumps({"results": results, "clusters": dict(clusters)}, indent=2))

Running this produces two reports (r1, r2) that resolve to the same signature despite coming from different builds with different offsets — that is the whole point of symbolication. Report r3 references an unknown build id, so its frames are unknown, it fails the trust check, and it lands in the symbolication_failure bucket rather than forming a bogus cluster.

Distinguishing failures from real clusters

The trust rule is the guardrail. A signature is only admitted to the cluster table when the faulting frame resolves and at least three additional frames resolve. Everything else is routed to a failure bucket with a reason code so operators can fix the symbol pipeline instead of chasing phantom crashes.

Verification checklist

• Two reports of the same bug from different builds (different offsets) collapse to one signature_id.
• A report with an unknown build id never appears in the cluster table; it appears in symbolication_failure with a reason.
• Noise frames (RaiseException, allocator, CRT stubs) are absent from the stored frame list.
• Re-running the pipeline on the same input yields identical signatures — determinism check.
• Signature length is bounded (top-8 frames) so deep recursion does not fragment clusters.

With symbolication and trust-gated grouping in place, downstream frequency and trend analysis operates on real clusters rather than offset noise, and symbol-pipeline regressions surface as a distinct, monitorable failure class.