4. Exploratory data analysis and visualization

Exploratory Data Analysis (EDA) is the disciplined practice of interrogating a dataset before modeling it. The goal is not to produce polished charts for a report, but to answer a small set of concrete questions: What is the shape and granularity of the data? Which columns are complete, which are sparse, and which are suspicious? What distributions, relationships, and anomalies exist? In an applied machine learning system, EDA directly determines feature engineering choices, imputation strategy, model family selection, and the validation split. Skipping it is the most common cause of silent failures later in the pipeline.

Establishing the Data Contract First

Before plotting anything, load the data and verify its structural assumptions. With pandas, the first pass is cheap and highly informative:

import pandas as pd

df = pd.read_csv("transactions.csv", parse_dates=["event_time"])
print(df.shape)                 # (rows, columns)
print(df.dtypes)                # inferred types per column
print(df.isna().mean().sort_values(ascending=False).head(10))
print(df.duplicated(subset=["user_id", "event_time"]).sum())
print(df["amount"].describe(percentiles=[.01, .25, .5, .75, .99]))

These five calls answer the questions that matter most: row count and column count, whether timestamps parsed correctly, which columns have missing values and at what rate, whether duplicate keys exist, and the central tendency plus tail behavior of a numeric column. A column that should be numeric but shows object dtype usually indicates embedded currency symbols, thousands separators, or sentinel strings such as "N/A" or "-". Fixing dtype at this stage prevents downstream aggregation errors.

Univariate Distributions and the Shape of Skew

For each numeric feature, compare a histogram against a box plot. The histogram reveals modality (unimodal, bimodal, long-tailed), while the box plot exposes outliers and quartile spread. A strongly right-skewed variable such as transaction amount often benefits from a log transform before it is fed to a linear model, because the transform compresses the tail and stabilizes variance. For categorical features, use value_counts(normalize=True) and watch for high-cardinality columns and rare categories. A category appearing in fewer than roughly 1% of rows will often be invisible to a train/test split and can cause target leakage if it encodes an identifier.

Bivariate and Multivariate Relationships

Correlation matrices are useful but easily misread. Pearson correlation captures only linear monotonic relationships and is sensitive to outliers; Spearman rank correlation is more robust for skewed data. Always pair a correlation heatmap with at least one scatter plot of the strongest pair, because two variables can share a strong nonlinear relationship that Pearson reports as near zero. For categorical targets, group-by aggregation is more informative than correlation: compute the mean and count of the target per category, then sort by count to separate genuine signal from small-sample noise.

The following diagram summarizes how the EDA outputs feed directly into downstream pipeline decisions. Each analytical artifact maps to a specific engineering action, which is the practical reason EDA is not optional.

Choosing the Right Visualization Tool

pandas plotting is the fastest path from a DataFrame to a rough chart and is ideal for the first exploratory pass. matplotlib gives full control over axes, ticks, and layout, which matters when a figure must be reproducible in a paper or dashboard. seaborn adds statistical aggregation on top of matplotlib: sns.histplot, sns.boxplot, sns.heatmap, and sns.pairplot handle grouping and confidence intervals with far less code. plotly produces interactive figures where hovering reveals exact values and zooming isolates clusters; this is valuable when exploring high-dimensional or geographic data, but interactive HTML output is heavier and should not be embedded in every notebook cell.

A practical rule: use seaborn for the analysis notebook, matplotlib for the final static figure, and plotly when a stakeholder needs to explore the data themselves. Do not mix all three in one figure.

Profiling Large Datasets Without Loading Everything

When a CSV exceeds available memory, the standard read_csv call fails or triggers swapping. Three techniques keep EDA feasible:

• Column selection and dtype downcasting. Read only needed columns with usecols, and convert float64 to float32 or int64 to int32 where precision allows. This alone can cut memory by 50%.
• Chunked iteration. pd.read_csv(..., chunksize=500_000) yields DataFrames one block at a time. Accumulate running statistics (count, sum, sum of squares, min, max) rather than concatenating chunks.
• Sampling for visual EDA. Draw a random sample of 50,000–100,000 rows for distribution and scatter plots. The sample is sufficient to reveal shape and outliers; reserve full-dataset scans for exact aggregates.

import numpy as np

stats = {"n": 0, "sum": 0.0, "sumsq": 0.0, "min": np.inf, "max": -np.inf}
for chunk in pd.read_csv("transactions.csv", usecols=["amount"], chunksize=500_000):
    a = chunk["amount"].to_numpy(dtype="float64")
    stats["n"] += a.size
    stats["sum"] += a.sum()
    stats["sumsq"] += (a ** 2).sum()
    stats["min"] = min(stats["min"], a.min())
    stats["max"] = max(stats["max"], a.max())

mean = stats["sum"] / stats["n"]
var = stats["sumsq"] / stats["n"] - mean ** 2
print(f"n={stats['n']:,} mean={mean:.3f} std={var**0.5:.3f}")

This streaming pattern computes exact mean and standard deviation in one pass with constant memory. For automated profiling, ydata-profiling (formerly pandas-profiling) generates an HTML report covering dtypes, missingness, distributions, correlations, and duplicate rows. On large files, pass a sample or use its minimal mode; the full report on tens of millions of rows is slow and produces an unwieldy document.

From EDA to a Reproducible Checklist

The output of EDA should be a short, written record that the modeling stage can act on. A useful format is a table with one row per column, capturing the decision that was made and why. This prevents the analysis from being repeated and makes the pipeline auditable.

Two verification steps close the loop. First, confirm that the cleaned DataFrame has the expected dtypes and no unexpected nulls introduced by transformations. Second, re-run the distribution plots on the transformed features and confirm that skew has decreased and that no new outliers were created by the transform. If a log transform produces -inf from zero or negative inputs, the clipping step was missed. If a category encoding produces a column with a single unique value, the rare-category grouping threshold was too aggressive. These symptoms are easy to catch at EDA time and expensive to debug after training.

EDA is complete when every column has a documented decision, the target distribution is understood, the split strategy respects time and group structure, and the transformed features have been visually re-verified. At that point the dataset is ready for feature engineering and model training, and the EDA record becomes the first section of the experiment log.