2. Python environment and tooling

Applied machine learning work fails in surprisingly mundane ways. A model that trains perfectly on your laptop may crash on a colleague's machine because of a NumPy version mismatch, or a notebook that produced a clean result last month may now yield different numbers because an unpinned dependency silently upgraded. This chapter treats the Python environment as a first-class engineering artifact: something you design, version, and reproduce deliberately rather than something that "just happens" to work on your machine.

Why Isolation Is Non-Negotiable

Every Python installation has a site-packages directory where third-party libraries live. If you install scikit-learn, pandas, and torch globally, all projects share one dependency graph. The moment two projects require incompatible versions of the same library — say, one needs pandas==1.5.* and another needs pandas==2.2.* — you are forced into a conflict that no amount of pip install --upgrade can resolve cleanly. Virtual environments solve this by giving each project its own isolated site-packages and its own interpreter reference.

Two tools dominate this space today. venv ships with the standard library and is sufficient for most workflows. conda (or its faster sibling mamba) manages both Python packages and non-Python binary dependencies such as CUDA libraries, which matters when you install GPU-enabled PyTorch or XGBoost. A practical rule: use venv when your dependencies are pure Python or have reliable wheels; use conda when you need system-level libraries, specific CUDA toolkits, or R integration.

Creating and Activating an Environment

The following sequence creates a project-local environment, activates it, and installs a minimal ML stack. The commands assume a Unix-like shell; on Windows PowerShell, activation uses .\.venv\Scripts\Activate.ps1 instead.

# Create an isolated environment inside the project directory
python -m venv .venv

# Activate it (Linux / macOS)
source .venv/bin/activate

# Confirm the interpreter is the isolated one
which python
# -> /path/to/project/.venv/bin/python

# Install core libraries
python -m pip install --upgrade pip
python -m pip install numpy pandas scikit-learn matplotlib jupyterlab

After activation, python and pip resolve to the environment's copies. Deactivating with deactivate returns you to the system interpreter. A common mistake is installing packages before activating — they land in the global site-packages and remain invisible to the project. Always verify with which python or python -c "import sys; print(sys.prefix)" before installing.

Package Management and Locking

Installing packages is easy; reproducing the exact set months later is the hard part. pip freeze captures every installed package with its exact version, but it also captures transitive dependencies you never explicitly chose, producing a long, brittle list. A cleaner approach separates direct dependencies from the full lock:

# requirements.in — direct dependencies only
numpy==1.26.4
pandas==2.2.1
scikit-learn==1.4.1.post1
matplotlib==3.8.3
jupyterlab==4.1.4

You then compile this into a fully pinned lock file using pip-compile from pip-tools, or use uv pip compile for a much faster resolver. The lock file records every transitive package with a hash, so pip install -r requirements.txt --require-hashes guarantees byte-identical installs. For conda users, conda env export --no-builds > environment.yml produces a portable specification, though cross-platform reproducibility often requires conda-lock.

Jupyter: Interactive Exploration with Discipline

JupyterLab is the default interactive environment for data exploration and model prototyping. Its strength — executing cells out of order and inspecting intermediate state — is also its reproducibility hazard. A notebook that runs top-to-bottom on a fresh kernel is reproducible; one that depends on a variable defined three cells earlier and then deleted is not.

Adopt three habits. First, restart the kernel and run all cells before committing or sharing a notebook; if it fails, the notebook is not a valid artifact. Second, keep heavy computation in .py modules and import them into the notebook, so the logic is testable and version-controllable. Third, record the environment inside the notebook itself:

import sys, platform
import numpy, pandas, sklearn

print(sys.version)
print(platform.platform())
for m in (numpy, pandas, sklearn):
    print(m.__name__, m.__version__)

For a stronger guarantee, install the environment as a Jupyter kernel so the notebook is bound to a specific interpreter:

python -m ipykernel install --user \
  --name ml-project --display-name "Python (ml-project)"

Selecting "Python (ml-project)" from the kernel menu ensures the notebook always uses the project's packages, regardless of which environment is active in the terminal.

IDEs and Editor Configuration

For production code, notebooks are a poor fit. VS Code and PyCharm both provide debugging, refactoring, type checking, and integrated testing. The critical configuration step is pointing the IDE at the project's virtual environment, not the system interpreter. In VS Code, Ctrl+Shift+P → Python: Select Interpreter and choose .venv/bin/python. In PyCharm, set the project interpreter under Settings → Project → Python Interpreter.

Add a .gitignore that excludes .venv/, __pycache__/, .ipynb_checkpoints/, and large data files. Commit the lock file, the source modules, and a short README describing how to recreate the environment. Never commit the environment directory itself — it contains absolute paths and platform-specific binaries.

Reproducibility Beyond Packages

Package versions are necessary but not sufficient. Reproducibility also requires controlling random seeds, data versions, and hardware nondeterminism. Set seeds explicitly at the start of every experiment:

import random, numpy as np, torch

SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed_all(SEED)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Even with seeds set, GPU operations may be nondeterministic due to parallel reduction order. Document this limitation rather than pretending it away. For datasets, store a checksum alongside the file and verify it on load; a silently truncated CSV is a common source of irreproducible results.

A Minimal Reproducible Project Layout

The following structure keeps environment, code, and data concerns separated. It is small enough to adopt immediately and complete enough to scale.

To verify the whole chain, simulate a fresh clone in a temporary directory: create a new venv, install from the lock file, and run the training entry point. If it succeeds without manual intervention, the environment is reproducible. If it fails, the error message tells you exactly which assumption was implicit — a missing system library, an unpinned dependency, or a hardcoded absolute path.

Treating the environment as code — pinned, locked, versioned, and verified — turns "it works on my machine" into "it works on any machine that follows the README." That property is what separates a prototype from an engineering artifact, and it is the foundation every subsequent chapter builds on.