4. Fuse Metrics and Logs for Disk Saturation Diagnosis

Why a Single Signal Cannot Separate Disk Failure Modes

Write-latency incidents on a disk-backed service almost always surface as the same symptom: application threads block on write(), request latency climbs, and the p99 spikes. But three very different root causes produce that identical symptom: the filesystem has run out of free blocks or inodes (capacity exhaustion), the device is being hammered by competing I/O so queue depth and wait time explode (I/O contention), or the physical medium is failing and retries/remaps inflate service time (hardware degradation). Metrics alone blur these together — a saturated %util looks the same whether the disk is full, busy, or dying. Logs alone are too sparse and only fire after damage is done. The fusion analyzer in this case combines both so the diagnosis names the actual mechanism.

Signal Selection and the Discriminating Features

The design principle is to pick, from each source, the features that diverge across the three hypotheses rather than the features that merely confirm "the disk is slow." The table below is the core of the analyzer's decision logic and should be treated as the specification the code implements.

The key insight is the load-independence test: contention-driven latency collapses when the competing workload stops, whereas hardware-degradation latency persists at idle because the device itself is slow. The analyzer therefore correlates the latency window against a load proxy (write IOPS) and against the presence of medium-error log lines.

Building the Fusion Analyzer

The analyzer ingests two aligned streams over the incident window: a metrics series (one sample per 10 s) and a parsed log-event list. It computes per-hypothesis evidence scores, then applies a small set of hard gates so that a decisive log signature (e.g. ENOSPC) overrides soft metric scoring. The implementation below is self-contained and runnable with the Python standard library only.

import re
from dataclasses import dataclass, field

@dataclass
class Sample:
    t: int                 # seconds since window start
    free_blocks_pct: float
    free_inodes_pct: float
    io_wait_ms: float
    queue_depth: float
    write_iops: float
    await_ms: float
    realloc_delta: int

@dataclass
class LogEvent:
    t: int
    text: str

ENOSPC_RE   = re.compile(r"ENOSPC|EDQUOT|No space left on device", re.I)
MEDERR_RE   = re.compile(r"medium error|I/O error|remap|bad sector|UNC", re.I)

def score_hypotheses(samples, events, baseline_iops):
    n = len(samples)
    avg = lambda f: sum(f(s) for s in samples) / n

    free_b = avg(lambda s: s.free_blocks_pct)
    free_i = avg(lambda s: s.free_inodes_pct)
    wait   = avg(lambda s: s.io_wait_ms)
    qd     = avg(lambda s: s.queue_depth)
    iops   = avg(lambda s: s.write_iops)
    await_ = avg(lambda s: s.await_ms)
    realloc = sum(s.realloc_delta for s in samples)

    enospc = sum(1 for e in events if ENOSPC_RE.search(e.text))
    mederr = sum(1 for e in events if MEDERR_RE.search(e.text))

    # Load-independence: does await stay high when IOPS is low?
    low_load = [s for s in samples if s.write_iops < 0.5 * baseline_iops]
    await_low_load = (sum(s.await_ms for s in low_load) / len(low_load)
                      if low_load else 0.0)

    scores = {"capacity": 0.0, "contention": 0.0, "hardware": 0.0}

    # Capacity: free space/inodes near zero, ENOSPC present
    if free_b < 5:  scores["capacity"] += 2.0
    if free_i < 5:  scores["capacity"] += 1.5
    if enospc > 0:  scores["capacity"] += 3.0

    # Contention: high queue depth, elevated IOPS, latency tracks load
    if qd > 8:                       scores["contention"] += 2.0
    if iops > 1.5 * baseline_iops:   scores["contention"] += 2.0
    if await_low_load < 0.5 * await_: scores["contention"] += 1.5

    # Hardware: medium errors, reallocations, load-independent latency
    if mederr > 0:                   scores["hardware"] += 3.0
    if realloc > 0:                  scores["hardware"] += 2.0
    if await_low_load >= 0.8 * await_ and await_ > 20:
        scores["hardware"] += 2.0

    # Hard gates: decisive log signatures override soft scoring
    if enospc > 0 and mederr == 0:
        return "capacity", scores
    if mederr > 0 and enospc == 0:
        return "hardware", scores
    return max(scores, key=scores.get), scores

Feed it a window like the following (values are illustrative teaching data, not a real capture). Here free space is healthy, queue depth is high, IOPS is roughly double baseline, and latency collapses at low load — the contention signature:

baseline_iops = 400
samples = [
    Sample(0,  62.0, 88.0, 18.0, 11.0, 820, 24.0, 0),
    Sample(10, 61.5, 88.0, 22.0, 13.0, 910, 27.0, 0),
    Sample(20, 61.0, 87.5, 9.0,  3.0,  180, 6.0,  0),  # low-load sample
]
events = [LogEvent(5, "app: write retry, slow path")]
cause, scores = score_hypotheses(samples, events, baseline_iops)
print(cause, scores)
# contention {'capacity': 0.0, 'contention': 5.5, 'hardware': 0.0}

Swap in a window where free_blocks_pct is 1.2 and a No space left on device line appears, and the hard gate returns capacity regardless of the metric scores. Add a medium error kernel line plus a positive realloc_delta, and it returns hardware. The three branches are exercised by the same function, which is what makes the fusion meaningful rather than three separate detectors.

Reading the Evidence Together

The diagram below shows how the two signal families converge on a single diagnosis and where the hard gates sit relative to the soft scoring.

Validation and Failure Symptoms

Validate the analyzer against labeled incident windows where the true cause is known from post-incident review. The pass condition is that the returned cause matches the label for every window, and that the score margin between the winning and runner-up hypothesis is positive. Two failure modes are worth watching for:

• Contention misread as hardware. If the low-load sample set is empty (the window never captured an idle moment), await_low_load defaults to 0 and the load-independence test cannot fire. Ensure the window includes at least one low-IOPS sample, or widen the window.
• Capacity masked by contention. A full filesystem under heavy write load shows both ENOSPC and high queue depth. The hard gate correctly returns capacity, but only if the ENOSPC line is inside the aligned window — verify timestamp alignment before trusting the result.

When the analyzer returns a low-margin result with no hard gate triggered, treat it as inconclusive and escalate to the causal-graph ranking stage rather than forcing a single label.