3. Consumer rating ecosystems

A legal compliance gap analysis compares what the law requires against what the market actually experiences. Consumer ratings are one of the few observable signals of that lived experience, but they are not a neutral mirror. Before any rating is used as evidence of a compliance gap, the analyst must understand where the rating came from, what scale it uses, how individual scores were combined into an aggregate, and how much of the underlying population the aggregate actually represents. This section builds that understanding systematically, then shows how to judge whether a given rating source is reliable enough to support a gap finding.

Where ratings come from: four source families

Consumer rating data arrives through four structurally different channels, and each carries a distinct bias profile.

• Platform-native reviews. Marketplaces and service platforms (app stores, e-commerce sites, booking platforms) collect ratings at the point of transaction. Volume is high and tied to verified purchases, but the platform controls solicitation timing and often suppresses or reorders reviews. Ratings skew toward extreme experiences because satisfied users rarely return to rate.
• Independent review sites. Third-party aggregators that do not sell the product. They reduce platform conflict of interest but depend on self-selected reviewers, so the sample is rarely representative of the customer base.
• Regulator and ombudsman complaint data. Formal complaints filed with public authorities. These are the most legally relevant source because they map to statutory categories, but they capture only the small fraction of grievances that escalate to formal channels.
• Survey and panel research. Structured instruments with controlled sampling. The most methodologically defensible source, but expensive, infrequent, and often commissioned by the very firms being assessed.

The practical consequence is that no single source is authoritative. A gap analysis that relies on platform reviews alone will overstate problems that generate dramatic complaints and understate quiet, systemic failures such as unclear contract terms or slow refunds.

Scales are not interchangeable

Ratings arrive on incompatible scales: 1–5 stars, 0–10 net promoter scores, letter grades, percentage satisfaction, and binary recommend/do-not-recommend. Two problems follow. First, the midpoint means different things. On a 1–5 star scale, 3 is neutral; on a 0–10 NPS scale, 7 is passive and only 9–10 count as promoters. Second, distributions are not normal. Star ratings cluster at the top (a "5-star ceiling") and are bounded below, so a drop from 4.6 to 4.3 may represent a far larger shift in sentiment than the arithmetic difference suggests.

Before comparing any two sources, normalize them to a common reference. A defensible approach is to convert each to a percentile rank within its own source population, or to a proportion of respondents above a defined threshold. Record the transformation explicitly, because a gap finding that depends on an undocumented rescaling is not reproducible and will not survive scrutiny.

Aggregation methods and what they hide

An aggregate rating is a compression, and every compression loses information. The most common methods are the simple mean, the volume-weighted mean, the Bayesian or "true" average (which shrinks low-volume items toward a prior), and recency-weighted averages. Each behaves differently when sample sizes are small or when sentiment is shifting.

Consider a worked example. A service has 40 ratings: 30 at 5 stars, 6 at 4 stars, 3 at 2 stars, and 1 at 1 star. The simple mean is (150 + 24 + 6 + 1) / 40 = 181 / 40 = 4.53. A Bayesian average with a prior mean of 4.0 and prior weight of 10 yields (181 + 40) / (40 + 10) = 221 / 50 = 4.42. The gap between methods is 0.11 stars — small here, but it grows sharply for low-volume items, and it can flip the ranking of two services. The analyst should always ask which method the source uses and whether the displayed score is the raw mean or a shrunk estimate.

Aggregation also hides distribution shape. Two services can both show 4.2 stars while one has a tight cluster around 4 and the other is bimodal, with many 5s and many 1s. The bimodal service has a polarized experience that a mean cannot reveal, and polarization is often exactly what a compliance gap looks like: a subset of customers systematically harmed while the majority are satisfied.

Reliability gates for gap analysis

Not every rating is fit to support a compliance conclusion. Three gates filter the usable from the merely suggestive.

Gate 1 — sample size and segment coverage. An aggregate of 12 reviews cannot support a claim about a customer population, and an aggregate of 4,000 that contains only 9 reviews from the affected segment (say, customers who requested a refund) tells you nothing about that segment. The relevant n is the n within the segment where the alleged gap lives, not the headline total.

Gate 2 — distribution shape. Inspect the histogram, not just the mean. A bimodal distribution with a heavy 1-star tail is a signal that a specific failure mode is concentrated in a subgroup. Cross-tabulate the low scores against complaint categories to see whether they cluster around a legally relevant issue such as misrepresentation, billing, or cancellation.

Gate 3 — recency and stability. Ratings decay. A 2019 average cannot describe a 2024 practice, and a sharp recent shift may reflect a change in solicitation policy rather than a change in service quality. Compare rolling 12-month windows and flag any source whose methodology changed during the period.

Applying the framework: a short worked check

Suppose a subscription service shows a platform average of 4.4 stars from 2,100 reviews, while an ombudsman dataset records 180 complaints in the same year, 62% of them about automatic renewal. The headline rating looks healthy, but the complaint category maps directly to a statutory disclosure obligation. Running the gates: the platform sample is large (Gate 1 passes), but the distribution is bimodal with a 1-star cluster of 190 reviews (Gate 2 flags), and 140 of those 190 mention renewal (Gate 2 confirms concentration). The ombudsman data is recent and category-coded (Gate 3 passes). The defensible finding is not "customers are unhappy" but "a concentrated subgroup reports renewal-related harm consistent with a disclosure gap" — a claim the rating data supports once it is decomposed rather than averaged.

Recording provenance

Every rating used in the analysis should carry a provenance record: source name and type, collection period, scale, aggregation method, sample size overall and per segment, and any known methodology changes. Without this record, a gap finding cannot be reproduced or defended, and a reviewer cannot tell whether a difference between two sources reflects a real gap or merely a difference in how each source counts and averages. Treat the provenance record as part of the finding, not as an appendix.