3. Model-free value estimation in control loops

In the previous chapter we assumed the agent had access to a model of the environment — transition probabilities and reward functions it could query at will. In most engineering control loops that assumption fails. A servo axis, a thermal plant, or a chemical reactor rarely exposes an analytic model accurate enough for planning, and even when a nominal model exists, friction, backlash, and unmodeled dynamics corrupt it. Model-free value estimation replaces the model with sampled experience: the controller acts, observes the resulting reward and next state, and updates its estimate of long-run performance directly from those samples. This chapter develops the three canonical estimators — Monte Carlo returns, temporal-difference (TD) learning, and eligibility traces — and shows how each maps onto an online control loop.

Setting Up the Estimation Problem

We work with a Markov decision process defined by states s, actions a, a reward signal r, and a discount factor γ ∈ [0,1). The quantity we want is the state-value function under a fixed policy π:

V^π(s) = E_π[ G_t | S_t = s ]
G_t = r_{t+1} + γ r_{t+2} + γ² r_{t+3} + ...

Here G_t is the return — the discounted sum of future rewards from time step t onward. In a control loop, r is typically a negative tracking error, a penalty for control effort, or a shaped reward combining both. The discount factor γ controls the effective horizon: γ = 0.95 at a 100 Hz control rate gives a horizon of roughly 20 steps, or 0.2 s of lookahead; γ = 0.99 extends that to about 1 s. Choosing γ is a control-design decision, not a hyperparameter afterthought.

Monte Carlo Returns

The Monte Carlo (MC) estimator is the most direct: run the policy until an episode terminates, compute the actual return G_t at every visited state, and average those returns. The update for a single sample is:

V(S_t) ← V(S_t) + α [ G_t − V(S_t) ]

where α is the learning rate. MC is unbiased — it estimates exactly the quantity defined above — but it has two properties that hurt in control loops. First, it requires episodic tasks: you must reach a terminal state before any update happens. A position controller that runs continuously has no natural episode end, so you would have to impose artificial episodes (e.g., 500 ms windows). Second, the variance of G_t grows with horizon length because every reward along the trajectory is a random variable. In a noisy plant, MC updates are slow to converge.

MC is still useful when episodes are short and cheap, or when you need an unbiased reference to validate a TD implementation.

Temporal-Difference Learning

TD learning bootstraps: instead of waiting for the full return, it uses the current estimate of the next state as a stand-in for the remaining rewards. The one-step TD(0) update is:

δ_t = r_{t+1} + γ V(S_{t+1}) − V(S_t)
V(S_t) ← V(S_t) + α δ_t

The quantity δ_t is the TD error. It is the single most important signal in this chapter: it measures the surprise between the reward actually received plus the discounted value of where we landed, and the value we previously predicted for where we were. When δ_t is consistently positive, the policy is doing better than expected; when negative, worse. In an online control loop, δ_t can be monitored as a convergence diagnostic — it should decay toward zero as V approaches V^π.

TD(0) updates after every step, so it works on non-episodic tasks and adapts quickly to changing dynamics. Its bias comes from bootstrapping: if V(S_{t+1}) is wrong, the update propagates that error. In practice TD(0) converges faster than MC on most control problems because its variance is much lower, even though it introduces bias.

Eligibility Traces and TD(λ)

TD(0) only credits the state visited one step ago. If a reward arrives 30 steps after the action that caused it, TD(0) needs many sweeps to propagate that credit backward. Eligibility traces solve this by maintaining a short-term memory of which states were recently visited, and applying the current TD error to all of them, weighted by recency.

For each state s, the trace e_t(s) is updated as:

e_t(s) = γλ e_{t−1}(s) + 1  if s = S_t
e_t(s) = γλ e_{t−1}(s)      otherwise

and the value update becomes:

V(s) ← V(s) + α δ_t e_t(s)   for all s

The parameter λ ∈ [0,1] interpolates between TD(0) (λ = 0, only the current state updated) and Monte Carlo (λ = 1, all states in the episode updated equally). Intermediate values like λ = 0.9 give most of the low-variance benefit of TD while propagating credit much further back in time. This is the TD(λ) algorithm.

In a control loop, traces are typically implemented as a vector e the same size as the weight vector w, updated every step:

e ← γλ e + φ(S_t)
w ← w + α δ_t e

This is the backward view of TD(λ), and it is what you implement in practice — it costs one vector addition and one scalar-vector multiply per step, which is negligible at typical control rates.

Choosing an Estimator for a Control Loop

The table below summarizes the trade-offs as they appear in an online control setting. The right choice depends on episode structure, noise level, and how far back in time credit must propagate.

Worked Example: Online Value Estimation for a DC Motor Position Loop

Consider a DC motor position controller running at 100 Hz. The state is s = [θ_err, θ̇_err], the action is a voltage command, and the reward is r = −(θ_err² + 0.01 u²). We want to estimate V^π(s) online using TD(λ) with a linear approximator.

Setup. Use 5×5 normalized radial basis features over the error range [−0.5, 0.5] rad and error-rate range [−5, 5] rad/s, giving φ(s) ∈ R²⁵. Initialize w = 0, e = 0, α = 0.05, γ = 0.97, λ = 0.9.

Per-step update (executed every 10 ms):

# observe transition (s, r, s')
δ = r + γ * dot(w, φ(s')) - dot(w, φ(s))
e = γ * λ * e + φ(s)
w = w + α * δ * e
s = s'

What to observe. Log |δ| averaged over a 1 s window. With a well-tuned α, this should decay by roughly an order of magnitude within the first 30–60 s of operation and then fluctuate around a small residual set by process noise. If |δ| oscillates or grows, α is too large for the feature scale — halve it. If |δ| decays but V predictions remain implausible (e.g., positive values in high-error regions where rewards are strongly negative), the features are too coarse; add resolution near the operating point.

Typical failure symptom. A common mistake is forgetting to reset e at episode boundaries when using episodic tasks. The trace then carries stale credit across episodes, producing systematically biased values. If you see V drifting upward without bound on a task where returns are bounded, check trace reset logic first.

From Value Estimation to Control

Value estimation alone does not produce a controller. The estimates V(s) feed into a policy improvement step: for each state, compare the estimated value of candidate actions and shift the policy toward the better ones. In the next chapter we extend the TD framework to action-value functions Q(s,a), which makes this comparison direct and leads to Q-learning and SARSA. The eligibility-trace machinery developed here carries over unchanged — only the object being estimated changes from V to Q.

The key engineering takeaway is that model-free value estimation turns a control loop into a self-assessing system. The TD error δ_t is not just a learning signal; it is a real-time measure of how well the current policy matches the reward structure. Monitoring it gives you a window into whether the controller is behaving as designed, without needing a plant model.