2. Environment setup and reproducibility

A TensorFlow project that runs correctly on one machine but fails on another is almost always a symptom of an uncontrolled environment. Before writing a single layer of a model, you should be able to answer three questions precisely: which Python interpreter and which package versions are in use, whether TensorFlow is actually executing on the GPU, and whether two runs of the same script produce identical numbers. This chapter establishes those three guarantees.

Isolating the Runtime with Virtual Environments

System Python is shared by the operating system and every other tool on the machine. Installing TensorFlow into it invites version conflicts that are painful to unwind. A virtual environment gives each project its own interpreter and its own site-packages directory, so upgrading NumPy for one project cannot break another.

Two tools dominate current practice. venv ships with Python and is sufficient for pure-Python dependency graphs. Conda is preferable when you need CUDA and cuDNN libraries managed alongside Python packages, because it can install the native runtime libraries that TensorFlow's GPU build links against.

# Option A: venv (Python 3.10+)
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip

# Option B: conda
conda create -n tf-gpu python=3.10 -y
conda activate tf-gpu

Once the environment is active, install TensorFlow and pin the versions that matter. TensorFlow 2.15 and later bundle CUDA support through the tensorflow[and-cuda] extra on Linux, which removes the need to install CUDA separately. On Windows, native GPU support was discontinued after TensorFlow 2.10, so Windows users should either pin to 2.10 or run training inside WSL2.

pip install "tensorflow[and-cuda]==2.16.1"
pip install numpy==1.26.4 pandas==2.2.2

Record the resolved environment so it can be rebuilt exactly. pip freeze > requirements.txt captures every transitive dependency; conda env export > environment.yml does the same for Conda, including native libraries. Commit this file alongside the code. A dependency file without pinned versions is a reproducibility bug waiting to surface.

Verifying GPU Configuration

Installing a GPU build of TensorFlow does not guarantee that the GPU is used. The runtime silently falls back to CPU when a driver is missing, when the CUDA version does not match, or when another process has claimed all device memory. Always verify explicitly.

import tensorflow as tf

print("TF version:", tf.__version__)
print("Built with CUDA:", tf.test.is_built_with_cuda())
print("GPUs visible:", tf.config.list_physical_devices("GPU"))

# Confirm a real kernel executes on the device
with tf.device("/GPU:0"):
    a = tf.random.normal((4096, 4096))
    b = tf.matmul(a, a)
print("Result device:", b.device)

If list_physical_devices("GPU") returns an empty list, check the driver first with nvidia-smi. A common failure is a driver that is too old for the CUDA runtime TensorFlow expects; the error surfaces as Could not load dynamic library 'libcudart.so.12' in the log. Another frequent symptom is CUDA_ERROR_OUT_OF_MEMORY when a previous process has not released the device. Enabling memory growth prevents TensorFlow from reserving the entire GPU at startup and makes multi-process experimentation practical:

gpus = tf.config.list_physical_devices("GPU")
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

For multi-GPU workstations, tf.distribute.MirroredStrategy() replicates the model across visible devices and aggregates gradients. Restrict visibility with the CUDA_VISIBLE_DEVICES environment variable when you want a training job to use only a subset of cards.

Deterministic Training

Neural network training is stochastic by design: weight initialization, dropout masks, data shuffling, and certain GPU kernels all introduce randomness. Reproducibility means controlling every source of that randomness so that a fixed seed yields a fixed result.

TensorFlow exposes a single entry point, tf.keras.utils.set_random_seed(), which seeds Python's random, NumPy, and TensorFlow's global generator together. Setting only one of them is a common mistake that leaves residual nondeterminism.

import os, random
import numpy as np
import tensorflow as tf

SEED = 42
os.environ["PYTHONHASHSEED"] = str(SEED)
random.seed(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)
tf.keras.utils.set_random_seed(SEED)

# Force deterministic GPU kernels
tf.config.experimental.enable_op_determinism()

enable_op_determinism() is the decisive switch. Without it, operations such as tf.reduce_sum on a GPU may accumulate floating-point values in a nondeterministic order, producing differences in the last few bits that amplify over thousands of steps. The trade-off is performance: deterministic kernels are often slower, and some operations have no deterministic GPU implementation and will raise an error rather than silently diverge.

Two further sources deserve attention. First, dataset pipelines: pass shuffle(..., seed=SEED) and set tf.data.AUTOTUNE carefully, since parallel prefetching can reorder elements. Second, data augmentation layers such as RandomFlip and RandomRotation accept a seed argument; supply it. Finally, set the environment variable TF_DETERMINISTIC_OPS=1 before importing TensorFlow if you need determinism enforced at the process level.

Putting It Together

The diagram below shows how the three concerns interlock. The virtual environment defines the software boundary; the GPU configuration defines the hardware boundary; the determinism controls define the numerical boundary. A reproducible experiment requires all three to be pinned and recorded.

Verification Checklist

Before committing a training script, run this sequence. It takes under a minute and catches the majority of environment defects.

When the final loss differs between two runs on the same machine, the cause is almost always one of three things: a missing enable_op_determinism() call, a dataset shuffle without a seed, or a data augmentation layer relying on the global generator. Isolate them by disabling augmentation first, then fixing the shuffle seed, then enabling op determinism. If the mismatch persists after all three, the GPU driver or library version has changed between runs — re-pin the environment and rebuild it from requirements.txt.