2. Requirements engineering for analytics platforms

Analytics platforms fail in predictable ways, and most of those failures trace back to requirements that were never written down. A dashboard that loads in 40 seconds, a nightly job that silently drops 3% of events, a schema change that breaks six downstream models — none of these are engineering accidents. They are the consequence of treating requirements as a conversation rather than an artifact. This section covers the four requirement classes that determine whether an analytics platform is usable: use case discovery, service-level agreements (SLAs), data contracts, and non-functional requirements.

Use Case Discovery: From Stakeholder Intent to Queryable Requirements

Use case discovery is not a meeting where you ask people what reports they want. It is a structured elicitation process that produces testable statements about who consumes what data, at what grain, with what freshness, and to answer what decision. The most reliable technique is to work backwards from the decision, not forwards from the data source.

Start with a decision inventory. For each business or operational decision, capture five fields: the decision owner, the decision cadence (real-time, hourly, daily, weekly), the metric or entity being decided on, the acceptable error or latency, and the action taken if the data is unavailable. A fraud-blocking decision has a cadence of seconds and near-zero tolerance for missing data. A quarterly capacity-planning decision tolerates a two-day lag and a 5% sampling error. These two use cases will drive completely different pipeline architectures, and conflating them is the single most common source of over-engineering and under-delivery.

Translate each decision into a use case record with an explicit grain statement. "Daily active users" is not a requirement; "count of distinct user_id with at least one session_start event, partitioned by UTC calendar day, refreshed by 06:00 UTC" is. The grain statement forces you to name the entity, the event, the time semantics, and the refresh boundary — all of which become testable acceptance criteria later.

Service-Level Agreements for Data

An SLA for an analytics platform is a contract about observable properties of a data product. It differs from a software SLA in that it must specify both the delivery guarantee and the correctness guarantee. Four dimensions are typically sufficient:

Each dimension needs a measurement method, not just a number. Freshness is measured as now() - max(event_time) sampled every minute. Completeness requires a source-side counter to compare against — without it, you cannot distinguish "no events occurred" from "events were lost." Accuracy requires a reconciliation job that compares the platform's aggregate against an independent system of record. An SLA without a measurement method is a wish.

Tier your SLAs. Not every dataset deserves 99.9% availability. A three-tier model — critical (revenue, fraud, safety), standard (operational reporting), and exploratory (ad-hoc analysis) — lets you allocate engineering effort where it changes decisions. Publish the tier alongside the dataset in the catalog so consumers self-select correctly.

Data Contracts

A data contract is the interface specification between a data producer and its consumers. It is versioned, machine-readable, and enforced at the boundary. The contract covers schema, semantics, and change policy.

# contracts/orders_v2.yaml
contract: orders
version: 2.1.0
owner: payments-platform-team
schema:
  order_id:      {type: string,  nullable: false, unique: true}
  customer_id:   {type: string,  nullable: false}
  order_total:   {type: decimal, nullable: false, precision: 12, scale: 2}
  currency:      {type: string,  nullable: false, enum: [USD, EUR, GBP, JPY]}
  created_at:    {type: timestamp, nullable: false, timezone: UTC}
  status:        {type: string,  nullable: false, enum: [pending, paid, shipped, cancelled]}
semantics:
  order_total: "Gross amount before tax and discounts, in minor units of `currency`."
  created_at:  "Order creation time in the producer's system, not the ingestion time."
sla:
  freshness:    "PT15M"
  completeness: "0.995"
change_policy:
  breaking:     [remove_field, rename_field, narrow_type, remove_enum_value]
  compatible:   [add_optional_field, widen_type, add_enum_value]
  notice_period: "P30D for breaking changes"

Three properties make a contract enforceable rather than aspirational. First, it is validated at the producer boundary — a CI check runs schema validation against a sample of produced records before deployment. Second, it is validated at the consumer boundary — the ingestion job rejects records that violate the contract and routes them to a dead-letter queue rather than silently coercing them. Third, breaking changes require a deprecation window and a migration path, typically a dual-write period where both v1 and v2 are published.

Non-Functional Requirements

Non-functional requirements (NFRs) describe how the platform behaves under load, failure, and change. For analytics platforms, five categories matter most:

• Scalability: the growth rate the platform must absorb without re-architecture. State it as a multiple, e.g., "3× current event volume within 12 months at constant per-record cost."
• Latency: distinguish ingestion latency (event time to landing), processing latency (landing to queryable), and query latency (p50/p95/p99 for the defined query classes).
• Cost efficiency: cost per million records processed and cost per terabyte scanned. These are the metrics that keep a platform sustainable.
• Recoverability: RPO (recovery point objective) and RTO (recovery time objective) per tier. A critical dataset might need RPO ≤ 5 min and RTO ≤ 30 min; an exploratory dataset might tolerate RPO of 24 hours.
• Observability: the platform must expose freshness, completeness, and schema-drift metrics as first-class signals, not as afterthoughts.

NFRs are where requirements engineering most often degrades into vague adjectives. "The platform should be scalable and reliable" is not a requirement. "The ingestion tier must sustain 50,000 events/sec with p99 end-to-end latency under 90 seconds, and degrade to at-most-once delivery rather than blocking when the sink is unavailable" is. The second version can be tested, and therefore can be met.

Putting It Together: A Worked Example

Consider a payments team that needs a "daily settlement reconciliation" dataset. The decision is whether to release funds to merchants. The cadence is daily, before 07:00 local time. The grain is one row per merchant_id per settlement date. The error tolerance is zero — a wrong settlement total is a financial incident.

From this use case, the SLA tier is critical: freshness ≤ 06:00, completeness ≥ 99.99%, availability 99.95%, accuracy ≤ 0.01% deviation against the ledger. The data contract specifies merchant_id, settlement_date, gross_amount, fee_amount, net_amount, and currency, with semantics that pin net_amount = gross_amount - fee_amount as an invariant. The NFRs require RPO ≤ 5 minutes and RTO ≤ 30 minutes, because a missed settlement window has direct financial impact.

Now the requirements are testable. A CI job validates the contract on producer output. An ingestion check rejects records violating the invariant. A reconciliation job compares net_amount sums against the ledger and alerts on deviation above 0.01%. A freshness monitor pages the on-call engineer if the dataset is not queryable by 06:00. Each of these is a direct consequence of a requirement that was written down, owned, and measured.

The discipline is not in producing more documents. It is in refusing to accept a requirement that cannot be tested, and in refusing to build a pipeline whose behavior is not governed by an explicit contract. Use case discovery tells you what to build, SLAs tell you how well it must perform, data contracts tell you what shape it must take, and NFRs tell you how it must behave under stress. Together they form the specification that makes an analytics platform an engineered system rather than an accumulation of scripts.