3. Query-by-Committee Disagreement on a Small Ensemble

Why committee disagreement is a different signal

Margin and entropy sampling both read uncertainty from a single model's output distribution. Query-by-committee (QBC) instead asks a structural question: do independently trained hypotheses disagree about this input? If several classifiers that all fit the labeled data reasonably well produce different predictions on an unlabeled point, that point lies in a region where the version space is still wide — exactly where one more label buys the most information. The acquisition score is the disagreement among committee members, and the sample with the highest disagreement is queried next.

This case builds a three-member committee on a synthetic two-dimensional binary problem, computes disagreement with two complementary measures (vote entropy and KL divergence to the mean), and runs a fixed-budget active learning loop against random sampling. The deliverable is a working script plus a comparison of decision boundaries before and after the queried points are added.

Committee design: diversity is the whole point

A committee of identical models is useless — they will agree everywhere and the disagreement signal collapses to zero. Diversity must be engineered deliberately. Three cheap, standard sources of diversity are used here:

• Algorithm diversity: a LogisticRegression, a GaussianNB, and a DecisionTreeClassifier with max_depth=4. Different inductive biases mean different error patterns.
• Data diversity: each member is trained on a bootstrap resample of the current labeled pool (bagging), so members see slightly different training sets.
• Feature/parameter diversity (optional): varying regularization strength or random seeds for the tree. Kept minimal here to keep the loop fast and the comparison clean.

All members must expose predict_proba so that soft disagreement can be computed. The tree is wrapped so its probability output is well-behaved on the small labeled pool.

Disagreement measures

Two scores are computed for every unlabeled sample x across M committee members:

• Vote entropy: each member casts a hard vote; the fraction voting for class c is p_vote(c); score is -Σ p_vote(c) log p_vote(c). Maximum when votes split evenly (e.g., 2–1 on a 3-member committee gives the highest possible value).
• KL divergence to the mean: average the members' predicted probabilities into p̄, then score is (1/M) Σ KL(p_m ‖ p̄). This uses confidence, not just hard votes, and is more sensitive when members are confidently wrong in different directions.

Vote entropy is coarse on a 3-member committee (only a few discrete values), so KL divergence is the primary acquisition function; vote entropy is reported as a sanity check and for tie-breaking.

Reference implementation

The script below is self-contained: it generates the data, builds the committee, runs both acquisition strategies under the same budget, and prints the accuracy trajectory. It uses only numpy and scikit-learn.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(0)

# 1. Data: two interleaving moons, 600 points, 2D features
X, y = make_moons(n_samples=600, noise=0.30, random_state=0)
X_pool, y_pool = X[:500], y[:500]          # unlabeled pool (labels hidden)
X_test, y_test = X[500:], y[500:]          # held-out evaluation set

# 2. Committee factory: 3 diverse members, each bagged on the labeled set
def build_committee(X_lab, y_lab, seed):
    r = np.random.default_rng(seed)
    idx = r.integers(0, len(X_lab), size=len(X_lab))
    Xb, yb = X_lab[idx], y_lab[idx]
    members = [
        LogisticRegression(max_iter=500, C=1.0).fit(Xb, yb),
        GaussianNB().fit(Xb, yb),
        DecisionTreeClassifier(max_depth=4, random_state=seed).fit(Xb, yb),
    ]
    return members

def committee_proba(members, Xq):
    return np.stack([m.predict_proba(Xq) for m in members], axis=0)  # (M, N, 2)

# 3. Disagreement scores
def vote_entropy(P):
    votes = P.argmax(axis=2)                       # (M, N)
    N = votes.shape[1]
    scores = np.zeros(N)
    for i in range(N):
        counts = np.bincount(votes[:, i], minlength=2) / votes.shape[0]
        counts = counts[counts > 0]
        scores[i] = -np.sum(counts * np.log(counts))
    return scores

def kl_to_mean(P, eps=1e-9):
    P = np.clip(P, eps, 1 - eps)
    pbar = P.mean(axis=0)                          # (N, 2)
    kl = np.sum(P * (np.log(P) - np.log(pbar[None, :, :])), axis=2)  # (M, N)
    return kl.mean(axis=0)

# 4. Active learning loop
def run(strategy, budget=40, seed_init=8):
    r = np.random.default_rng(42)
    init = r.choice(len(X_pool), size=seed_init, replace=False)
    labeled = list(init)
    unlabeled = np.setdiff1d(np.arange(len(X_pool)), init)
    accs = []
    for step in range(budget):
        members = build_committee(X_pool[labeled], y_pool[labeled], seed=step)
        if strategy == "random":
            pick = r.choice(unlabeled)
        else:
            P = committee_proba(members, X_pool[unlabeled])
            scores = kl_to_mean(P) if strategy == "kl" else vote_entropy(P)
            pick = unlabeled[np.argmax(scores)]
        labeled.append(pick)
        unlabeled = unlabeled[unlabeled != pick]
        # evaluate on held-out set with a fresh committee
        eval_members = build_committee(X_pool[labeled], y_pool[labeled], seed=999)
        pred = np.mean([m.predict(X_test) for m in eval_members], axis=0) >= 0.5
        accs.append(accuracy_score(y_test, pred.astype(int)))
    return np.array(accs)

for strat in ["random", "kl"]:
    a = run(strat)
    print(f"{strat:6s}  final_acc={a[-1]:.3f}  acc@20={a[19]:.3f}  acc@40={a[-1]:.3f}")

Expected behavior on this seed: the KL-based run reaches roughly 0.85–0.90 held-out accuracy by 40 queries, while random sampling typically lags by 5–10 points at the same budget. The exact numbers depend on the seed and the noise level; treat them as a sanity band, not a benchmark. If the KL run does not beat random, the usual causes are (a) the committee is not diverse enough (all members agree everywhere), (b) the initial labeled set is too small for the tree to fit, or (c) the pool is dominated by easy points far from the boundary.

Reading the disagreement signal

The figure below shows how the committee's disagreement concentrates along the true decision boundary and how the queried points pull the boundary into shape. The left panel is the initial committee's averaged probability surface with the seed labels; the right panel is the same surface after 40 KL-selected queries. The queried points (orange) cluster where the members split, which is precisely where the boundary is under-determined.

What to verify and how to troubleshoot

Three checks confirm the loop is doing what QBC promises:

1. Disagreement is non-degenerate. Print kl_to_mean(P).max() and .mean() each round. If the max is near zero, the committee has collapsed to agreement — increase diversity (different seeds, different max_depth, or add a fourth member).
2. Queried points sit near the boundary. Overlay the queried indices on a scatter plot; they should cluster where the two classes interleave, not in the interior of either class.
3. Label efficiency improves. Compare the accuracy trajectory of kl vs random at the same budget. A consistent gap of several points across seeds is the expected outcome; a single lucky seed is not evidence.

Reusable acquisition checklist

With the loop running and the three checks passing, the committee's disagreement has been turned into a concrete labeling policy: query where the members argue, and the decision boundary tightens with far fewer labels than random sampling requires.