3. Feature engineering for time series

Machine learning models such as gradient boosting, random forests, and linear regressors have no intrinsic notion of time. They treat each row as an independent observation. The entire burden of encoding temporal structure therefore falls on feature engineering. A well-constructed feature matrix can turn a generic regressor into a competitive forecaster; a poorly constructed one produces leakage, overfitting, or flat, useless predictions. This chapter covers the four workhorse families of time-series features — lag features, rolling statistics, calendar effects, and Fourier terms — and shows how to build them correctly.

Lag Features: Encoding Autocorrelation

A lag feature is simply the target value shifted back by k time steps. If y[t] is the value at time t, then lag_k[t] = y[t-k]. For a daily sales series, lag_1 is yesterday's sales, lag_7 is the same weekday last week, and lag_365 captures annual seasonality. The model learns which lags matter by fitting coefficients or split thresholds against them.

The critical rule is that lag features must be computed using only information available at prediction time. When forecasting y[t], you may use y[t-1], y[t-2], and so on, but never y[t] itself or any future value. A common mistake is to shift the entire series once and then split into train/test — this leaks future values into the training set. The correct approach is to build the full feature matrix first, then split by time index, ensuring the test period's features only reference past data.

Choosing which lags to include depends on the autocorrelation function (ACF) and partial autocorrelation function (PACF). For a series with weekly seasonality, lags 1, 7, 14, and 28 are natural candidates. Including too many lags inflates dimensionality and invites overfitting; including too few misses structure. A practical starting point is to include lags at the dominant seasonal periods plus a few short-term lags (1–3).

Rolling Statistics: Smoothing and Trend Capture

Rolling statistics summarize a window of past values. Common choices are the rolling mean, rolling standard deviation, rolling minimum, and rolling maximum. For a window of size w ending at t-1, the rolling mean is:

roll_mean_w[t] = mean(y[t-w], ..., y[t-1])

These features give the model a sense of local level and volatility. A rolling mean over 7 days captures the current weekly baseline; a rolling standard deviation captures regime changes in variance. Rolling features are especially valuable when the series has non-stationary level shifts that lags alone cannot represent.

Two implementation details matter. First, the window must end at t-1, not t, to avoid leakage. Second, the first w-1 rows will contain NaN values; these must be dropped or imputed before training. In pandas, this is expressed as:

df['roll_mean_7'] = df['y'].shift(1).rolling(window=7).mean()
df['roll_std_7']  = df['y'].shift(1).rolling(window=7).std()

The shift(1) before rolling is the safeguard against leakage. Without it, the rolling mean at time t would include y[t], which is exactly what you are trying to predict.

Calendar Effects: Human Time vs. Clock Time

Calendar features encode the position of a timestamp within human cycles: day of week, month, quarter, week of year, is-weekend, is-holiday, days-to-next-holiday, and so on. These are deterministic and known in advance, which makes them safe to use for future forecasting — unlike lags, which require past target values.

For a retail demand series, day-of-week and holiday flags often explain more variance than any lag. For energy load, hour-of-day and is-holiday are dominant. The engineering task is to extract these from the timestamp column and encode them appropriately. Cyclical features such as day-of-week should be one-hot encoded or represented as sine/cosine pairs, not left as raw integers, because the model would otherwise treat Friday (5) as "more" than Monday (1) in a linear sense.

A subtle trap is the "calendar drift" problem: if your training data covers 2021–2023 and you forecast 2024, holidays shift. A fixed is_christmas flag is fine, but a feature like "day of year" will misalign if the forecast horizon crosses a leap year. Prefer relative features (days until Christmas) over absolute ones when the horizon is long.

Fourier Terms: Compact Seasonality

When seasonality is smooth and periodic — daily temperature, hourly electricity load, annual rainfall — Fourier terms provide a compact representation. For a period P and order K, the features are:

sin_k[t] = sin(2 * pi * k * t / P)
cos_k[t] = cos(2 * pi * k * t / P)   for k = 1..K

Each order captures a harmonic of the fundamental frequency. Order 1 captures the primary cycle, order 2 the first overtone, and so on. With K = 3 and P = 24 (hourly data), you get six features that describe a smooth daily curve. This is far more parsimonious than 23 hourly dummies and generalizes better to unseen hours.

The choice of K controls smoothness. Too low and the model underfits the seasonal shape; too high and it starts fitting noise. A practical heuristic is to start with K = 2 or 3 and increase until validation error stops improving. Fourier terms are also additive: you can combine a daily period (P=24) with a weekly period (P=168) by concatenating their sine/cosine pairs.

Putting It Together: A Feature Matrix

The table below shows a small slice of a feature matrix for a daily sales series with weekly seasonality. Each row is one training example; the target is y[t].

Notice that lag_1 for 2024-03-02 equals the target of 2024-03-01, and roll_mean_7 is computed over the seven days ending the day before. This alignment is what prevents leakage.

Common Failure Modes and How to Detect Them

• Leakage via shift order: Rolling features computed without shift(1) produce suspiciously low training error but poor test performance. Check by verifying that no feature at time t correlates perfectly with the target.
• NaN propagation: The first w-1 rows of rolling features are NaN. If not dropped, some libraries silently treat them as zero, distorting the model. Always inspect df.isna().sum() after feature construction.
• Overly long lags: Including lag_365 on two years of data leaves only one usable training row. Match lag depth to available history.
• Fourier order too high: Validation error decreases then increases as K grows. Plot the curve and pick the elbow.

Visual Overview of the Feature Pipeline

The diagram below shows how a raw timestamped series is transformed into four feature families and then assembled into a model-ready matrix. The key constraint — all features at time t use only data up to t-1 — is enforced at the lag and rolling stages.

The pipeline is deliberately modular: each feature family can be added, removed, or tuned independently, and the leakage guard applies uniformly across all of them. In practice, you will iterate — start with lags and calendar features, measure validation error, then add rolling statistics and Fourier terms where they help. The next chapter moves from feature construction to model selection and validation strategy for time series.