4. Q-learning and SARSA for setpoint regulation

Setpoint regulation is the canonical control problem: drive a plant output to a constant reference and hold it there despite disturbances and model uncertainty. In this chapter we treat that task as a finite Markov decision process (MDP) and solve it with two tabular temporal-difference (TD) methods — Q-learning and SARSA. Both learn an action-value function Q(s, a) from experience, but they differ in one crucial respect: Q-learning is off-policy and bootstraps from the greedy action, while SARSA is on-policy and bootstraps from the action the agent actually takes. That single difference changes what each algorithm converges to when exploration is active, and it changes how each behaves on a real plant.

Formulating setpoint regulation as an MDP

For a plant with output y and reference r, define the error e = r − y. A practical tabular formulation discretizes the state as the pair (e, Δe), where Δe = e_k − e_{k−1} is the error increment over one control interval. Discretizing both dimensions gives a compact grid — for example, error in [−1.0, 1.0] with 21 bins and error increment in [−0.2, 0.2] with 11 bins yields 231 states. Actions are discrete control increments, e.g. u ∈ {−2, −1, 0, +1, +2} in actuator units. The reward should encode both regulation accuracy and effort:

r_k = −( w_e · e_k² + w_u · u_k² ) − penalty_if_|e_k| > e_max
w_e = 1.0, w_u = 0.01, e_max = 0.5

The quadratic error term drives the output toward the setpoint; the small effort term discourages chattering; the penalty terminates episodes that leave the safe band. Episodes end after a fixed horizon (e.g. 200 steps) or on violation of e_max.

Tabular Q-learning and SARSA updates

Both algorithms maintain a table Q[s][a] initialized to zero. After observing transition (s, a, r, s′), Q-learning applies

Q[s][a] += α · ( r + γ · max_a' Q[s'][a'] − Q[s][a] )

whereas SARSA applies

Q[s][a] += α · ( r + γ · Q[s'][a'] − Q[s][a] )

with a′ the next action actually selected by the behavior policy. With α = 0.1 and γ = 0.95, both updates are cheap and numerically stable. The distinction matters because Q-learning's target uses the best possible next action regardless of what the agent does, so it learns the optimal greedy policy even while exploring. SARSA's target includes the exploratory action, so it learns the value of the policy it is currently following — including its exploration noise.

Exploration schedules

Tabular TD methods require sufficient state-action coverage. An ε-greedy behavior policy selects a random action with probability ε and the greedy action otherwise. A fixed ε never stops exploring; a decaying schedule balances coverage against exploitation. A robust choice is exponential decay with a floor:

epsilon = max(eps_min, eps_0 * exp(-episode / tau))
eps_0 = 1.0, eps_min = 0.05, tau = 150 episodes

Early episodes explore broadly; later episodes exploit the learned policy while retaining a 5% random-action floor to keep the table fresh under disturbances. An alternative is linear decay over a fixed budget, which is easier to reason about when the number of episodes is known in advance.

Convergence on benchmark plants

For a finite MDP with bounded rewards, both algorithms converge to their respective fixed points provided every state-action pair is visited infinitely often and the step size satisfies the Robbins–Monro conditions (e.g. α_k = 1/k). In practice, constant α is used and convergence is judged empirically. On a discretized first-order plant y_{k+1} = 0.9·y_k + 0.1·u_k with a unit step reference, a well-tuned tabular agent typically reaches a mean absolute error below 0.05 within a few hundred episodes. Q-learning tends to converge to a slightly more aggressive policy; SARSA, because it accounts for exploration, often produces smoother control with fewer overshoots.

Practical implementation notes

Three implementation details dominate real performance. First, state discretization: too coarse and the agent cannot resolve small errors; too fine and the table becomes sparse and learning slows. A useful heuristic is to size bins so that one bin spans roughly the noise level of the measurement. Second, reward scaling: keep rewards in a range near [−1, 0] so that α and γ remain interpretable; large rewards destabilize the TD error. Third, action persistence: applying a new action every step can cause chattering. Many implementations hold the action for N control intervals (e.g. N = 5), which effectively reduces the action space and smooths the response.

Choosing between the two

Use Q-learning when the plant is safe to explore and you want the best possible greedy policy. Use SARSA when exploratory actions carry real cost — for example, a thermal plant where a random large actuator command could trip a safety limit. In both cases, verify convergence by monitoring the moving average of episode return and the final steady-state error, and confirm that the learned policy holds the setpoint within the required tolerance across several reference steps and disturbance injections.