2. Multi-objective optimization fundamentals

Multi-objective optimization (MOO) replaces the single "best" answer with a set of trade-off solutions. Before any Bayesian machinery can be trusted, you need a precise vocabulary for comparing candidates, describing the achievable trade-off boundary, and scoring how good an approximation of that boundary is. This section builds that vocabulary from the ground up, using a running engineering example so that every definition maps to something you can compute.

Formulating the problem and fixing conventions

A multi-objective problem is written as minimizing a vector-valued objective over a decision space:

minimize  f(x) = [f1(x), f2(x), ..., fm(x)]
subject to x ∈ X ⊆ R^d

Throughout this course we adopt the minimization convention: smaller is better for every objective. If an objective is naturally maximized (throughput, yield, accuracy), negate it or convert it before optimization, and record that transformation. Mixing conventions is the single most common source of inverted Pareto results. For the running example, consider a heat-sink design with two decision variables — fin pitch p (mm) and fin height h (mm) — and two objectives to minimize: thermal resistance R(p,h) in K/W and pressure drop ΔP(p,h) in Pa. Both objectives are expensive to evaluate, which is exactly why a Bayesian surrogate will later be used.

Pareto dominance

Solution a dominates solution b (written a ≺ b) if and only if two conditions hold simultaneously:

• a is no worse than b in every objective: f_i(a) ≤ f_i(b) for all i = 1..m;
• a is strictly better in at least one objective: f_j(a) < f_j(b) for some j.

Dominance is a partial order, not a total one. Two solutions can be mutually non-dominating: each is better in one objective and worse in another. In the heat-sink example, design A = (R = 0.42 K/W, ΔP = 18 Pa) and design B = (R = 0.35 K/W, ΔP = 31 Pa) are non-dominating — A is cheaper in pressure, B is cooler. Neither can be discarded on objective grounds alone. A design C = (R = 0.45 K/W, ΔP = 25 Pa) is dominated by A, because A is better in both objectives, so C is strictly inferior and should be removed from consideration.

The Pareto front

The set of all non-dominated solutions in the feasible objective space is the Pareto front (formally the Pareto-optimal set in decision space maps to the Pareto front in objective space). For two objectives it is a curve; for three it is a surface; beyond three it is a hypersurface that is hard to visualize but still well-defined. The front separates "achievable and non-improvable" from "achievable but improvable."

In practice you never obtain the true front — you obtain an approximation set produced by an optimizer or a surrogate-driven acquisition loop. The quality of that approximation is what the metrics below measure. A useful mental model: the true front is the ideal trade-off boundary; your approximation is a finite sample of points that should be (a) close to the boundary and (b) spread along it.

Hypervolume

The hypervolume indicator (HV, also called the S-metric) is the m-dimensional volume of objective space dominated by an approximation set and bounded by a chosen reference point r. Under minimization, the region is {y : y ≤ r and y is dominated by some point in the set}. Larger HV is better. It is the only unary indicator that is strictly Pareto-compliant: if set A dominates set B pointwise, then HV(A) > HV(B) for any valid reference point.

For two objectives, HV is computed by sorting the approximation points by increasing f1 and summing rectangles. Using the five approximation points in the figure (illustrative values, not measured):

points (f1, f2): A(0.30, 12), B(0.36, 17), C(0.44, 24), D(0.52, 30), E(0.62, 38)
reference r = (0.70, 45)

sort by f1 ascending, then accumulate:
  A: (0.36-0.30) * (45-12) = 0.06 * 33 = 1.98
  B: (0.44-0.36) * (45-17) = 0.08 * 28 = 2.24
  C: (0.52-0.44) * (45-24) = 0.08 * 21 = 1.68
  D: (0.62-0.52) * (45-30) = 0.10 * 15 = 1.50
  E: (0.70-0.62) * (45-38) = 0.08 *  7 = 0.56
HV ≈ 1.98 + 2.24 + 1.68 + 1.50 + 0.56 = 7.96

The reference point must be worse than every point in every objective; otherwise the volume is undefined or misleading. A common practice is to set r slightly beyond the worst observed value per objective, and to keep it fixed across all runs being compared. Changing r between runs invalidates HV comparisons.

Performance metrics for approximation sets

HV alone does not tell you whether a set is close to the true front or merely spread out. A practical evaluation combines convergence, spread, and cardinality. The table below summarizes the indicators most often used in Bayesian MOO papers and engineering reports.

IGD is defined as IGD = (1/|P*|) Σ_{p∈P*} min_{a∈A} d(p, a), where P* is a sample of the true front and A the approximation. Because it averages over the true front, a set that misses an entire region of the front receives a large penalty — this is the coverage property that GD lacks. In surrogate-driven Bayesian optimization, the "true front" is usually replaced by a high-fidelity reference set computed once from a dense sampling or a long run, and that reference must be documented.

Putting it to work: evaluating a surrogate-driven run

A reproducible evaluation loop for the heat-sink example looks like this. First, define fixed normalization bounds and a fixed reference point before any run begins, and store them alongside results. Second, run the optimizer (or Bayesian acquisition loop) for a fixed evaluation budget, collecting the final non-dominated set. Third, compute HV, IGD against the stored reference front, and spacing. Fourth, compare against a baseline — for instance, Latin hypercube sampling of the same budget — rather than reporting absolute numbers in isolation.

# illustrative evaluation skeleton (pseudo-code)
ref_front = load("heatsink_reference_front.csv")   # fixed, documented
r         = (0.70, 45.0)                            # fixed reference point
bounds    = {"R": (0.30, 0.70), "dP": (10.0, 45.0)}

for run in runs:
    A = run.non_dominated_set()
    A_norm = normalize(A, bounds)
    hv  = hypervolume(A_norm, normalize(r, bounds))
    igd = inverted_generational_distance(A_norm, ref_front)
    sp  = spacing(A_norm)
    record(run.id, hv, igd, sp, len(A))

Typical failure symptoms and their causes: HV that improves while IGD stagnates usually means the set is spreading but not converging — check surrogate uncertainty and acquisition settings. High spacing with low HV often indicates clustering near one objective; increase diversity pressure or the reference-point strategy. IGD that is suspiciously low while HV is poor usually signals a mismatched or leaked reference front — verify that the reference was generated independently of the evaluated run.

With these definitions fixed — minimization convention, dominance, front, HV with a documented reference point, and a paired metric protocol — the next layer of the course can safely introduce surrogate models and acquisition functions, because every candidate they propose can now be scored against a consistent, reproducible standard.