3. Vision-based occupant detection

Vision-based occupant detection is the perception layer that feeds the Occupant Classification System (OCS) with spatial and semantic information about who and what occupies each seat. Unlike weight-based sensors, a camera can distinguish an adult from a rear-facing child seat, detect a hand resting on a seat cushion, or recognize a pet. However, these capabilities depend entirely on three engineering decisions made before any neural network is trained: where the camera is mounted, how the cabin is illuminated in the near-infrared (NIR) band, and how raw frames are preprocessed before inference. This section treats those three decisions as a coupled design problem.

Camera Placement and Field of View

In-cabin monitoring cameras are typically mounted in one of four locations: the steering column / instrument cluster, the rearview mirror (overhead front-facing), the headliner above the B-pillar, or the front dome light. Each position trades off occlusion, perspective distortion, and coverage.

A single overhead camera at the rearview mirror is the most common choice for front-row OCS. Mounted at roughly 600–700 mm above the seat cushion and pitched down 25°–35°, it captures the seat pan, backrest, and headrest of both front seats. The critical geometric constraint is that the camera must see the seat cushion surface, not just the occupant's torso, because classification depends on pressure-relevant features such as whether a child seat base is present. A field of view (FOV) of 90°–120° horizontal is typical; wider FOV reduces resolution per pixel, which directly hurts small-object detection such as a forgotten phone or a child's hand.

For rear-row monitoring, a headliner-mounted camera above the second row pitched forward at 20°–30° works well, but the B-pillar creates a blind zone. A practical rule: place the camera so the optical axis intersects the seat cushion at a point roughly 150–200 mm forward of the backrest, ensuring the cushion-to-backrest junction is visible. The table below summarizes typical mounting parameters for a front-row OCS camera.

IR Illumination Design

Cabin lighting is uncontrolled: direct sunlight, deep shadow, and nighttime darkness all occur. Visible-light cameras fail under these conditions, so in-cabin OCS uses NIR illumination, typically at 850 nm or 940 nm, paired with a camera whose IR-cut filter is removed or switchable.

The choice between 850 nm and 940 nm is a trade-off. At 850 nm, silicon sensors retain high quantum efficiency (roughly 2–3× that of 940 nm), so less LED power is needed. However, 850 nm emits a faint red glow that can be visible to passengers, which is undesirable in a premium vehicle. At 940 nm, the emission is invisible to the human eye, but the sensor response drops significantly, requiring higher drive current or longer exposure. Many production systems use 940 nm for passenger comfort and compensate with larger LED arrays or binning.

Illumination uniformity matters more than raw power. A single high-power LED creates a hot spot that saturates the center of the frame while leaving seat edges dark. A typical design uses 4–8 LEDs arranged in a ring or bar around the camera lens, diffused by a molded optic. The goal is to keep luminance variation across the seat area within about ±20%. A useful verification step is to capture a flat white target placed on the seat and inspect the histogram: the standard deviation of pixel intensity across the seat region should be low, and no region should clip at 255.

Image Preprocessing Pipeline

Raw NIR frames are not fed directly to the classifier. A preprocessing chain improves robustness and reduces compute. The typical sequence is: region-of-interest (ROI) extraction, lens distortion correction, exposure normalization, and resizing.

ROI extraction crops the frame to the seat area, discarding the dashboard, windows, and other irrelevant regions. This reduces the input tensor size and prevents the network from learning spurious correlations with background objects. For a 1920×1080 frame, the front-left seat ROI might be a 640×640 crop.

Lens distortion correction is essential for wide-FOV cameras. A 120° FOV lens introduces barrel distortion that bends straight seat edges. The correction uses the standard pinhole model with radial and tangential coefficients obtained from calibration. OpenCV's cv2.undistort() with a precomputed camera matrix and distortion vector is the common implementation.

Exposure normalization compensates for the large dynamic range between a sunlit cabin and a dark one. A simple approach is per-frame histogram equalization, but this can amplify noise in dark frames. A more stable method is to compute the median intensity of the seat ROI and apply a linear gain so the median maps to a target value (e.g., 128). This preserves relative contrast while preventing under- or over-exposure.

# Example preprocessing for a single NIR frame
import cv2
import numpy as np

def preprocess(frame, camera_matrix, dist_coeffs, roi):
    # 1. Undistort
    undistorted = cv2.undistort(frame, camera_matrix, dist_coeffs)
    # 2. Crop ROI (x, y, w, h)
    x, y, w, h = roi
    crop = undistorted[y:y+h, x:x+w]
    # 3. Exposure normalization to median 128
    gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
    median = np.median(gray)
    if median > 0:
        gain = 128.0 / median
        crop = np.clip(crop.astype(np.float32) * gain, 0, 255).astype(np.uint8)
    # 4. Resize to network input
    resized = cv2.resize(crop, (256, 256))
    return resized

After preprocessing, the frame is ready for the classifier. The figure below shows the complete signal chain from scene to classifier input, including the coupling between illumination and preprocessing.

Failure Modes and Verification

Three failure modes dominate field complaints. First, sunlight saturation: direct sunlight through a window can exceed the NIR camera's dynamic range, clipping the occupant's face and torso. The symptom is a white blob in the frame and a classifier confidence drop. Mitigation includes automatic exposure control with a short exposure time and, in severe cases, a mechanical IR-cut filter that switches in during daylight. Second, LED aging: NIR LED output degrades over time, reducing contrast. A periodic self-test that compares the median intensity of a known target against a baseline can flag this. Third, lens contamination: dust or fingerprints on the camera lens scatter NIR light, creating a hazy image. The verification is straightforward: capture a frame of a uniform target and check that the modulation transfer function (MTF) at the seat's spatial frequency remains above the design threshold.

To validate the full chain, place a calibrated target on the seat, capture frames under three conditions (dark, ambient, direct sun), and confirm that the preprocessed output maintains a stable histogram with no clipping and that the classifier returns the correct occupancy class. If any condition fails, isolate the stage: bypass preprocessing and inspect the raw frame, then re-enable stages one at a time. This staged approach turns a vague "camera doesn't work" report into a specific, fixable defect.