MACHINE LEARNING

How to Never Get Fooled by “Beginner’s Luck”: A Guide to k-fold cross validation in ml

July 13, 2026 · 31 min read

The Problem 

Imagine you are the head talent scout for the Mumbai Indians, looking for India’s next superstar batsman. You watch a young domestic player smash a brilliant 100 runs in a single match on a flat, batsman-friendly pitch in Bengaluru, and you immediately sign him for ₹10 Crores.

But during the actual IPL season, he struggles miserably on the spinning pitches of Chennai and the slow, bouncy tracks of Delhi.

You made the ultimate scouting mistake: you judged his entire capability based on just one lucky day in one specific stadium. You didn’t test him across different conditions to see if he is truly a versatile player.

What is k-fold cross validation? (And Why Should You Care?)

The Analogy: The 5-Stadium Fitness Test

Instead of buying that batsman based on one match, what if you tested him using a smarter strategy?

You make him play 5 matches across 5 completely different stadiums in India (Mumbai, Chennai, Delhi, Kolkata, and Bengaluru).

By the end of this 5-round cycle, every single stadium has served as the “test exam” exactly once. You calculate his average score across all 5 rounds. If his average is high, you know he is a world-class player who can perform anywhere, under any condition.

The Formal Concept

In Machine Learning, k-fold cross validation in ml is a data-splitting technique used to evaluate how well a model will perform on unseen, real-world data.

Instead of testing our model just once, we divide our entire dataset into k equal parts (called “folds”). We train the model on k-1 parts, test it on the remaining 1 part, and repeat this process k times. Every single data point gets a turn to be in the test set exactly once.

Why the Simpler Method Fails

Before k-fold, data scientists used a simpler method called the Train-Test Split (or the Holdout Method). They would split their data once: 80% for training and 20% for testing.

This fails because of luck and bias. If your random 20% test split happens to contain only the easiest, most predictable data points, your model will score a 99% accuracy. You will celebrate, thinking your model is a genius. But when you deploy it in the real world, it will crash because it never practiced on the hard, messy data points that were hidden in the 80% training set.

A single train-test split is highly sensitive to how the coin flips. k-fold cross validation in ml removes this luck factor entirely.

k-fold cross validation

Where is k-fold cross validation in ml Used in Real Life?

Example 1: CBSE Board Exam Paper Setting (Education)

Example 2: Bollywood Movie Success Prediction (Entertainment)

Example 3: Indian Railways Seat Availability & Price Prediction

The Math Behind k-fold cross validation in ml 

To find out how well our model performs across all k rounds, we calculate the average error. Here is the mathematical formula for k-Fold Cross-Validation:

CV(k) = (1/k) · Σᵢ MSEᵢ

Let us break down this equation symbol by symbol:

To understand MSEᵢ, let’s look at its internal formula:

MSEᵢ = (1/nᵢ) · Σⱼ (yⱼ – ŷⱼ)²

What does “Minimizing” this mean?

In human terms, minimizing CV(k) means we want our model’s average mistake across all different test rounds to be as close to zero as possible. We are looking for a model that is not just a “one-hit wonder” on a single lucky split, but a consistent performer across all types of unseen data.

What do the Weights and Biases represent here?

During each of the k rounds, the model adjusts its internal knobs:

Because the model is trained k times on different combinations of data, it generates k slightly different sets of weights (w in code). By evaluating the model’s performance across all folds, we can choose the best settings to ensure our final weights are perfectly balanced.

k-fold-math

Why Simple Models Fail ?

Let’s see exactly what happens when we use a single, unlucky Train-Test split versus a robust 5-Fold Cross-Validation. Run this code to generate the animation!

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

# ==========================================
# 1. DATA GENERATION (Punjab Wheat Yield)
# ==========================================
# X: Normalized Rainfall (-1 to 1 represents dry to heavy monsoon)
# Y: Wheat Yield in Tonnes per Acre
np.random.seed(42)
total_farmers = 50
X = np.random.uniform(-1.0, 1.0, total_farmers)
# True relationship is a cubic curve (non-linear crop response to rain)
Y = 1.5 * (X**3) – 0.8 * (X**2) – 0.5 * X + 0.6 + np.random.normal(0, 0.12, total_farmers)

# — LEFT SIDE: Biased Holdout Split (The Wrong Way) —
# We split the data once. But our test set only contains heavy rainfall days (X >= 0.4)
# This simulates getting “unlucky” with an unrepresentative train-test split!
train_mask_left = X < 0.4
test_mask_left = ~train_mask_left

X_train_left = X[train_mask_left]
Y_train_left = Y[train_mask_left]
X_test_left = X[test_mask_left]
Y_test_left = Y[test_mask_left]

# — RIGHT SIDE: 5-Fold Cross-Validation (The Correct Way) —
# We shuffle the data and split it into 5 equal folds
indices = np.arange(total_farmers)
np.random.shuffle(indices)
X_shuffled = X[indices]
Y_shuffled = Y[indices]

folds_X = np.array_split(X_shuffled, 5)
folds_Y = np.array_split(Y_shuffled, 5)

# ==========================================
# 2. MODEL CONFIGURATIONS & HELPER FUNCTIONS
# ==========================================
deg_left = 5 # Overparameterized model (prone to overfitting)
deg_right = 3 # Robust model matching the true complexity

def get_poly_features(x, degree):
“””Generates polynomial feature matrix manually.”””
features = [x**d for d in range(degree + 1)]
return np.column_stack(features)

# Precompute polynomial features
Phi_train_left = get_poly_features(X_train_left, deg_left)
Phi_test_left = get_poly_features(X_test_left, deg_left)

# Initialize model weights
w_left = np.zeros(deg_left + 1)
w_right = np.zeros(deg_right + 1)

# Learning rates for Gradient Descent
lr_left = 0.05
lr_right = 0.1

# Lists to store loss history for live plotting
loss_history_train_left = []
loss_history_test_left = []
loss_history_train_right = []
loss_history_val_right = []

# Smooth X values for drawing the fitted curves
X_smooth = np.linspace(-1.1, 1.1, 200)
Phi_smooth_left = get_poly_features(X_smooth, deg_left)
Phi_smooth_right = get_poly_features(X_smooth, deg_right)

# ==========================================
# 3. ANIMATION SETUP
# ==========================================
fig, axs = plt.subplots(2, 2, figsize=(14, 10))
fig.suptitle(“Holdout Split vs. 5-Fold Cross-Validation”, fontsize=16, fontweight=’bold’)

def update(frame):
global w_left, w_right

# Determine current fold for the 5-Fold CV (each fold runs for 20 frames)
fold_idx = frame // 20

# Reset weights of the CV model at the start of each fold to show training from scratch
if frame % 20 == 0:
w_right = np.zeros(deg_right + 1)

# — UPDATE LEFT MODEL (Holdout Split) —
# Perform 5 steps of Gradient Descent per frame for smooth convergence
for _ in range(5):
pred_left = Phi_train_left @ w_left
error_left = pred_left – Y_train_left
grad_left = (Phi_train_left.T @ error_left) / len(Y_train_left)
w_left -= lr_left * grad_left

# Calculate current losses
loss_train_left = np.mean((Phi_train_left @ w_left – Y_train_left)**2)
loss_test_left = np.mean((Phi_test_left @ w_left – Y_test_left)**2)
loss_history_train_left.append(loss_train_left)
loss_history_test_left.append(loss_test_left)

# — UPDATE RIGHT MODEL (5-Fold CV) —
# Extract training and validation sets for the current fold
X_val_right = folds_X[fold_idx]
Y_val_right = folds_Y[fold_idx]
X_train_right = np.concatenate([folds_X[j] for j in range(5) if j != fold_idx])
Y_train_right = np.concatenate([folds_Y[j] for j in range(5) if j != fold_idx])

Phi_train_right = get_poly_features(X_train_right, deg_right)
Phi_val_right = get_poly_features(X_val_right, deg_right)

for _ in range(5):
pred_right = Phi_train_right @ w_right
error_right = pred_right – Y_train_right
grad_right = (Phi_train_right.T @ error_right) / len(Y_train_right)
w_right -= lr_right * grad_right

loss_train_right = np.mean((Phi_train_right @ w_right – Y_train_right)**2)
loss_val_right = np.mean((Phi_val_right @ w_right – Y_val_right)**2)
loss_history_train_right.append(loss_train_right)
loss_history_val_right.append(loss_val_right)

# — PLOTTING —
# Clear axes to redraw
for ax in axs.flat:
ax.cla()

# 1. Top Left: Left Model Fit
axs[0, 0].scatter(X_train_left, Y_train_left, color=’blue’, label=’Train Data (Dry/Normal)’, alpha=0.7)
axs[0, 0].scatter(X_test_left, Y_test_left, color=’red’, marker=’X’, s=80, label=’Test Data (Heavy Rain)’, alpha=0.8)
Y_smooth_left = Phi_smooth_left @ w_left
axs[0, 0].plot(X_smooth, Y_smooth_left, color=’darkblue’, linewidth=2.5, label=’Degree 5 Fit’)
axs[0, 0].set_ylim(-1.5, 2.5)
axs[0, 0].set_title(f”Holdout Split (Unrepresentative)\nTest Loss: {loss_test_left:.4f}”, fontsize=12)
axs[0, 0].set_xlabel(“Rainfall (Normalized)”)
axs[0, 0].set_ylabel(“Wheat Yield (Tonnes/Acre)”)
axs[0, 0].legend(loc=’upper left’)
axs[0, 0].grid(True, linestyle=’–‘, alpha=0.5)

# 2. Top Right: Right Model Fit
axs[0, 1].scatter(X_train_right, Y_train_right, color=’green’, label=’Train Folds’, alpha=0.7)
axs[0, 1].scatter(X_val_right, Y_val_right, color=’orange’, marker=’D’, s=60, label=f’Validation Fold {fold_idx+1}’, alpha=0.9)
Y_smooth_right = Phi_smooth_right @ w_right
axs[0, 1].plot(X_smooth, Y_smooth_right, color=’darkgreen’, linewidth=2.5, label=’Degree 3 Fit’)
axs[0, 1].set_ylim(-1.5, 2.5)
axs[0, 1].set_title(f”5-Fold CV — Fold {fold_idx+1}/5\nVal Loss: {loss_val_right:.4f}”, fontsize=12)
axs[0, 1].set_xlabel(“Rainfall (Normalized)”)
axs[0, 1].set_ylabel(“Wheat Yield (Tonnes/Acre)”)
axs[0, 1].legend(loc=’upper left’)
axs[0, 1].grid(True, linestyle=’–‘, alpha=0.5)

# 3. Bottom Left: Left Loss History
axs[1, 0].plot(loss_history_train_left, color=’blue’, label=’Train Loss’)
axs[1, 0].plot(loss_history_test_left, color=’red’, label=’Test Loss’)
axs[1, 0].set_yscale(‘log’)
axs[1, 0].set_title(“Holdout Loss History (Log Scale)”, fontsize=12)
axs[1, 0].set_xlabel(“Frames”)
axs[1, 0].set_ylabel(“Mean Squared Error”)
axs[1, 0].legend()
axs[1, 0].grid(True, linestyle=’–‘, alpha=0.5)

# 4. Bottom Right: Right Loss History
axs[1, 1].plot(loss_history_train_right, color=’green’, label=’Train Loss’)
axs[1, 1].plot(loss_history_val_right, color=’orange’, label=’Val Loss’)
axs[1, 1].set_yscale(‘log’)
axs[1, 1].set_title(“5-Fold CV Loss History (Log Scale)”, fontsize=12)
axs[1, 1].set_xlabel(“Frames”)
axs[1, 1].set_ylabel(“Mean Squared Error”)
axs[1, 1].legend()
axs[1, 1].grid(True, linestyle=’–‘, alpha=0.5)

# Dynamic Storytelling Frame Titles
if frame < 10:
fig.suptitle(f”Frame {frame} — Both starting blind”, fontsize=14, fontweight=’bold’)
elif frame < 40:
fig.suptitle(f”Frame {frame} — Holdout overfitting; CV rotating through Folds 1 & 2″, fontsize=14, fontweight=’bold’)
elif frame < 70:
fig.suptitle(f”Frame {frame} — Holdout failing on unseen test data; CV mastering Folds 3 & 4″, fontsize=14, fontweight=’bold’)
else:
fig.suptitle(f”Frame {frame} — Holdout failed completely! CV generalized beautifully.”, fontsize=14, fontweight=’bold’)

plt.tight_layout()

# Run the animation for 100 frames (20 frames per fold * 5 folds)
ani = FuncAnimation(fig, update, frames=100, interval=100, repeat=False)
ani.save(‘comparison_k-fold_cross_validation_in_ml.gif’, writer=’pillow’, fps=10)
plt.close()

# ==========================================
# 4. POST-TRAINING DIAGNOSTIC PLOTS
# ==========================================
# Generate final predictions on the entire dataset to compare performance
Phi_all_left = get_poly_features(X, deg_left)
Phi_all_right = get_poly_features(X, deg_right)

pred_all_left = Phi_all_left @ w_left
pred_all_right = Phi_all_right @ w_right

residuals_left = Y – pred_all_left
residuals_right = Y – pred_all_right

fig_diag, axs_diag = plt.subplots(1, 2, figsize=(14, 6))

# Subplot 1: Actual vs Predicted
axs_diag[0].scatter(Y, pred_all_left, color=’red’, alpha=0.6, label=’Holdout (Deg 5)’)
axs_diag[0].scatter(Y, pred_all_right, color=’green’, alpha=0.6, label=’5-Fold CV (Deg 3)’)
axs_diag[0].plot([Y.min(), Y.max()], [Y.min(), Y.max()], ‘k–‘, lw=2, label=’Perfect Prediction’)
axs_diag[0].set_title(“Actual vs. Predicted Wheat Yield”, fontsize=12, fontweight=’bold’)
axs_diag[0].set_xlabel(“Actual Yield (Tonnes/Acre)”)
axs_diag[0].set_ylabel(“Predicted Yield (Tonnes/Acre)”)
axs_diag[0].legend()
axs_diag[0].grid(True, linestyle=’–‘, alpha=0.5)

# Subplot 2: Residuals Plot
axs_diag[1].scatter(pred_all_left, residuals_left, color=’red’, alpha=0.6, label=’Holdout Residuals’)
axs_diag[1].scatter(pred_all_right, residuals_right, color=’green’, alpha=0.6, label=’5-Fold CV Residuals’)
axs_diag[1].axhline(y=0, color=’black’, linestyle=’–‘, lw=2)
axs_diag[1].set_title(“Residuals Analysis (Errors)”, fontsize=12, fontweight=’bold’)
axs_diag[1].set_xlabel(“Predicted Yield (Tonnes/Acre)”)
axs_diag[1].set_ylabel(“Residuals (Actual – Predicted)”)
axs_diag[1].legend()
axs_diag[1].grid(True, linestyle=’–‘, alpha=0.5)

plt.tight_layout()
plt.savefig(‘comparison_k-fold_cross_validation_in_ml.png’)
plt.close()

View post on imgur.com

Comparison Animation

What You Will See in This GIF:

How k-fold cross validation in ml Actually Learns ?

Now let’s zoom in on the 5-Fold Cross-Validation process itself. How does the model adjust its weights fold by fold?

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

# =====================================================================
# 1. DATA GENERATION (Punjab Wheat Yield Dataset)
# =====================================================================
# X: Normalized Rainfall (-1.0 represents drought, 1.0 represents heavy monsoon)
# Y: Wheat Yield in Tonnes per Acre
np.random.seed(42)
total_farms = 50
X = np.random.uniform(-1.0, 1.0, total_farms)
# True relationship is a cubic curve (crop yield response to rainfall)
Y = 1.5 * (X**3) – 0.8 * (X**2) – 0.5 * X + 0.6 + np.random.normal(0, 0.1, total_farms)

# Shuffle data to ensure random distribution across folds
shuffled_indices = np.arange(total_farms)
np.random.shuffle(shuffled_indices)
X_shuffled = X[shuffled_indices]
Y_shuffled = Y[shuffled_indices]

# Split into 5 equal folds (k=5)
k_folds = 5
folds_X = np.array_split(X_shuffled, k_folds)
folds_Y = np.array_split(Y_shuffled, k_folds)

# =====================================================================
# 2. MODEL CONFIGURATION & HELPER FUNCTIONS
# =====================================================================
degree = 3 # Cubic polynomial model matching the true complexity
learning_rate = 0.15
epochs_per_fold = 40
total_frames = k_folds * epochs_per_fold

def get_polynomial_features(x, deg):
“””Manually generates polynomial features: [1, x, x^2, x^3]”””
features = [x**i for i in range(deg + 1)]
return np.column_stack(features)

# Generate smooth X values for plotting the regression curve
X_smooth = np.linspace(-1.1, 1.1, 200)
Phi_smooth = get_polynomial_features(X_smooth, degree)

# Initialize tracking lists for the live loss plot
history_frames = []
history_train_loss = []
history_val_loss = []

# Store the final weights of each fold to compute the average model later
all_fold_weights = []

# =====================================================================
# 3. ANIMATION SETUP
# =====================================================================
fig, (ax_fit, ax_loss) = plt.subplots(1, 2, figsize=(15, 6))

# Global weight variable that updates frame-by-frame
w = np.zeros(degree + 1)

def update(frame):
global w

# Determine which fold is currently active as the validation set
fold_idx = frame // epochs_per_fold
epoch = frame % epochs_per_fold

# Extract training and validation sets for the current fold
X_val = folds_X[fold_idx]
Y_val = folds_Y[fold_idx]

X_train = np.concatenate([folds_X[i] for i in range(k_folds) if i != fold_idx])
Y_train = np.concatenate([folds_Y[i] for i in range(k_folds) if i != fold_idx])

# Generate polynomial features
Phi_train = get_polynomial_features(X_train, degree)
Phi_val = get_polynomial_features(X_val, degree)

# At the start of a new fold, reset weights to show learning from scratch
if epoch == 0:
w = np.zeros(degree + 1)

# — MANUAL GRADIENT DESCENT STEP —
predictions_train = Phi_train @ w
errors_train = predictions_train – Y_train
gradient = (Phi_train.T @ errors_train) / len(Y_train)
w -= learning_rate * gradient

# Calculate Mean Squared Error (MSE)
train_loss = np.mean((Phi_train @ w – Y_train)**2)
val_loss = np.mean((Phi_val @ w – Y_val)**2)

# Save weights at the end of each fold’s training cycle
if epoch == epochs_per_fold – 1:
all_fold_weights.append(w.copy())

# Store loss history
history_frames.append(frame)
history_train_loss.append(train_loss)
history_val_loss.append(val_loss)

# — PLOT 1: LIVE MODEL FITTING (LEFT) —
ax_fit.cla()
ax_fit.scatter(X_train, Y_train, color=’forestgreen’, alpha=0.7, label=’Training Folds (40 Farms)’)
ax_fit.scatter(X_val, Y_val, color=’darkorange’, marker=’D’, s=80, label=f’Validation Fold {fold_idx+1} (10 Farms)’)

# Plot current fitted curve
Y_smooth_pred = Phi_smooth @ w
ax_fit.plot(X_smooth, Y_smooth_pred, color=’navy’, linewidth=3, label=’Current Model Fit’)

ax_fit.set_xlim(-1.2, 1.2)
ax_fit.set_ylim(-1.5, 2.5)
ax_fit.set_xlabel(“Rainfall (Normalized)”, fontsize=11)
ax_fit.set_ylabel(“Wheat Yield (Tonnes/Acre)”, fontsize=11)
ax_fit.legend(loc=’upper left’)
ax_fit.grid(True, linestyle=’–‘, alpha=0.5)

# — PLOT 2: LIVE LOSS CURVE (RIGHT) —
ax_loss.cla()
ax_loss.plot(history_frames, history_train_loss, color=’forestgreen’, linewidth=2, label=’Train Loss’)
ax_loss.plot(history_frames, history_val_loss, color=’darkorange’, linewidth=2, label=’Validation Loss’)

# Draw vertical lines to mark fold transitions
for f in range(1, fold_idx + 1):
ax_loss.axvline(x=f * epochs_per_fold, color=’red’, linestyle=’:’, alpha=0.7)
ax_loss.text(f * epochs_per_fold – 18, 1.5, f”Fold {f}\nDone”, color=’red’, fontsize=9, fontweight=’bold’)

ax_loss.set_xlim(0, total_frames)
ax_loss.set_ylim(0, 2.0)
ax_loss.set_xlabel(“Total Training Frames”, fontsize=11)
ax_loss.set_ylabel(“Mean Squared Error (Loss)”, fontsize=11)
ax_loss.legend(loc=’upper right’)
ax_loss.grid(True, linestyle=’–‘, alpha=0.5)

# Annotate key moments
if epoch < 5:
ax_loss.annotate(‘Loss drops fast!’, xy=(frame, train_loss), xytext=(frame + 10, train_loss + 0.4),
arrowprops=dict(facecolor=’black’, shrink=0.05, width=1, headwidth=6))

# Dynamic Storytelling Titles
if epoch == 0:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Resetting weights! I know nothing.”, fontsize=14, fontweight=’bold’)
elif epoch < 15:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Learning fast! Loss = {train_loss:.4f}”, fontsize=14, fontweight=’bold’)
elif epoch < 30:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Getting warmer! Loss = {train_loss:.4f}”, fontsize=14, fontweight=’bold’)
else:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Converged! Validation Loss = {val_loss:.4f}”, fontsize=14, fontweight=’bold’)

# Run and save the animation
ani = FuncAnimation(fig, update, frames=total_frames, interval=80, repeat=False)
ani.save(‘training_k-fold_cross_validation_in_ml.gif’, writer=’pillow’, fps=10)
plt.close()

# =====================================================================
# 4. POST-TRAINING DIAGNOSTIC PLOTS (Actual vs Predicted & Residuals)
# =====================================================================
# Compute the final model weights by averaging the weights from all 5 folds
final_avg_weights = np.mean(all_fold_weights, axis=0)

# Generate predictions on the entire dataset using the averaged model
Phi_all = get_polynomial_features(X, degree)
predictions_all = Phi_all @ final_avg_weights
residuals = Y – predictions_all

fig_diag, (ax_diag1, ax_diag2) = plt.subplots(1, 2, figsize=(14, 6))

# Subplot 1: Actual vs Predicted Plot
ax_diag1.scatter(Y, predictions_all, color=’teal’, edgecolor=’k’, alpha=0.7, s=60, label=’Farms’)
ax_diag1.plot([Y.min(), Y.max()], [Y.min(), Y.max()], ‘r–‘, lw=2, label=’Perfect Prediction Line’)
ax_diag1.set_title(“Actual vs. Predicted Wheat Yield (Averaged Model)”, fontsize=12, fontweight=’bold’)
ax_diag1.set_xlabel(“Actual Yield (Tonnes/Acre)”, fontsize=11)
ax_diag1.set_ylabel(“Predicted Yield (Tonnes/Acre)”, fontsize=11)
ax_diag1.legend()
ax_diag1.grid(True, linestyle=’–‘, alpha=0.5)

# Subplot 2: Residuals Plot
ax_diag2.scatter(predictions_all, residuals, color=’crimson’, edgecolor=’k’, alpha=0.7, s=60, label=’Residuals’)
ax_diag2.axhline(y=0, color=’black’, linestyle=’–‘, lw=2)
ax_diag2.set_title(“Residuals Analysis (Prediction Errors)”, fontsize=12, fontweight=’bold’)
ax_diag2.set_xlabel(“Predicted Yield (Tonnes/Acre)”, fontsize=11)
ax_diag2.set_ylabel(“Residual (Actual – Predicted)”, fontsize=11)
ax_diag2.legend()
ax_diag2.grid(True, linestyle=’–‘, alpha=0.5)

plt.tight_layout()
plt.savefig(‘training_k-fold_cross_validation_in_ml.png’)
plt.close()

View post on imgur.com

Training Animation

What You Will See in This GIF:

This GIF shows the heart of k-fold cross validation in ml.

What Happens When We Push Too Far?

What happens when models of different complexities are evaluated using k-fold cross validation in ml? Let’s compare an underfit model, a robust model, and an overfit model side-by-side.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

# =====================================================================
# 1. DATA GENERATION (Punjab Wheat Yield Dataset)
# =====================================================================
np.random.seed(101)
total_farms = 45
X = np.random.uniform(-1.0, 1.0, total_farms)
# True relationship is a cubic curve with added noise
Y = 1.8 * (X**3) – 1.0 * (X**2) – 0.6 * X + 0.8 + np.random.normal(0, 0.12, total_farms)

# Shuffle and split into 5 folds
shuffled_indices = np.arange(total_farms)
np.random.shuffle(shuffled_indices)
X_shuffled = X[shuffled_indices]
Y_shuffled = Y[shuffled_indices]

k_folds = 5
folds_X = np.array_split(X_shuffled, k_folds)
folds_Y = np.array_split(Y_shuffled, k_folds)

# =====================================================================
# 2. MODEL CONFIGURATIONS & STABLE FEATURE SCALING
# =====================================================================
degrees = [1, 3, 15] # Underfit, Good Fit, Overfit
learning_rates = [0.1, 0.1, 0.02] # Lower learning rate for Degree 15 to ensure stability
epochs_per_fold = 30
total_frames = k_folds * epochs_per_fold

def get_scaled_poly_features(x, degree, mean=None, std=None):
“””Generates scaled polynomial features manually to prevent gradient explosion.”””
features = []
for d in range(1, degree + 1):
features.append(x**d)
features = np.column_stack(features)

if mean is None or std is None:
mean = np.mean(features, axis=0)
std = np.std(features, axis=0)
std[std == 0] = 1.0 # Prevent division by zero

features_scaled = (features – mean) / std
bias = np.ones((len(x), 1))
return np.hstack([bias, features_scaled]), mean, std

# Precompute smooth plotting features for each degree
X_smooth = np.linspace(-1.1, 1.1, 200)

# Initialize weights and loss histories for all three models
weights = {deg: np.zeros(deg + 1) for deg in degrees}
loss_histories_train = {deg: [] for deg in degrees}
loss_histories_val = {deg: [] for deg in degrees}

# =====================================================================
# 3. ANIMATION SETUP
# =====================================================================
# 2 Rows, 3 Columns: Row 1 = Model Fits, Row 2 = Live Loss Curves
fig, axs = plt.subplots(2, 3, figsize=(18, 10))

def update(frame):
global weights

fold_idx = frame // epochs_per_fold
epoch = frame % epochs_per_fold

# Extract training and validation sets for the current fold
X_val = folds_X[fold_idx]
Y_val = folds_Y[fold_idx]
X_train = np.concatenate([folds_X[i] for i in range(k_folds) if i != fold_idx])
Y_train = np.concatenate([folds_Y[i] for i in range(k_folds) if i != fold_idx])

# Reset weights to zero at the start of each fold to show training from scratch
if epoch == 0:
weights = {deg: np.zeros(deg + 1) for deg in degrees}

# Clear all axes to redraw
for ax in axs.flat:
ax.cla()

# Loop through each model complexity (Degree 1, 3, 15)
for col_idx, deg in enumerate(degrees):
lr = learning_rates[col_idx]

# Generate scaled polynomial features
Phi_train, mean_tr, std_tr = get_scaled_poly_features(X_train, deg)
Phi_val, _, _ = get_scaled_poly_features(X_val, deg, mean_tr, std_tr)
Phi_smooth, _, _ = get_scaled_poly_features(X_smooth, deg, mean_tr, std_tr)

# Perform 3 steps of Gradient Descent per frame for smooth visual convergence
for _ in range(3):
pred_train = Phi_train @ weights[deg]
error = pred_train – Y_train
gradient = (Phi_train.T @ error) / len(Y_train)
weights[deg] -= lr * gradient

# Calculate current losses
train_loss = np.mean((Phi_train @ weights[deg] – Y_train)**2)
val_loss = np.mean((Phi_val @ weights[deg] – Y_val)**2)

# Append to histories
loss_histories_train[deg].append(train_loss)
loss_histories_val[deg].append(val_loss)

# — ROW 1: PLOT MODEL FIT —
ax_fit = axs[0, col_idx]
ax_fit.scatter(X_train, Y_train, color=’forestgreen’, alpha=0.6, label=’Train Folds’)
ax_fit.scatter(X_val, Y_val, color=’darkorange’, marker=’D’, s=70, label=’Val Fold’)

Y_smooth_pred = Phi_smooth @ weights[deg]
ax_fit.plot(X_smooth, Y_smooth_pred, color=’navy’, linewidth=2.5, label=f’Degree {deg} Fit’)
ax_fit.set_ylim(-1.0, 2.5)
ax_fit.set_xlabel(“Rainfall (Normalized)”)
ax_fit.set_ylabel(“Wheat Yield”)
ax_fit.legend(loc=’upper left’)
ax_fit.grid(True, linestyle=’–‘, alpha=0.5)

# — ROW 2: PLOT LIVE LOSS CURVES —
ax_loss = axs[1, col_idx]
ax_loss.plot(loss_histories_train[deg], color=’forestgreen’, label=’Train Loss’)
ax_loss.plot(loss_histories_val[deg], color=’darkorange’, label=’Val Loss’)
ax_loss.set_yscale(‘log’)
ax_loss.set_xlabel(“Total Frames”)
ax_loss.set_ylabel(“MSE Loss (Log Scale)”)
ax_loss.legend(loc=’upper right’)
ax_loss.grid(True, linestyle=’–‘, alpha=0.5)

# Draw vertical lines to mark fold transitions
for f in range(1, fold_idx + 1):
ax_loss.axvline(x=f * epochs_per_fold, color=’red’, linestyle=’:’, alpha=0.5)

# Set individual panel titles
if deg == 1:
ax_fit.set_title(f”Degree 1: Underfit\nVal Loss: {val_loss:.4f}”, fontsize=12, fontweight=’bold’)
ax_loss.set_title(“Underfit Loss History”, fontsize=11)
elif deg == 3:
ax_fit.set_title(f”Degree 3: Robust Fit\nVal Loss: {val_loss:.4f}”, fontsize=12, fontweight=’bold’)
ax_loss.set_title(“Robust Loss History”, fontsize=11)
else:
ax_fit.set_title(f”Degree 15: Overfit\nVal Loss: {val_loss:.4f}”, fontsize=12, fontweight=’bold’)
ax_loss.set_title(“Overfit Loss History”, fontsize=11)

# Dynamic Storytelling Frame Titles
if epoch == 0:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Resetting weights! Models starting blind.”, fontsize=14, fontweight=’bold’)
elif epoch < 10:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Learning fast! Degree 15 is starting to wiggle.”, fontsize=14, fontweight=’bold’)
elif epoch < 20:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Degree 3 is stable. Degree 15 is chasing noise!”, fontsize=14, fontweight=’bold’)
else:
fig.suptitle(f”Fold {fold_idx+1} — Epoch {epoch}: Converged! Notice the massive validation loss on Degree 15.”, fontsize=14, fontweight=’bold’)

plt.tight_layout()

# Run and save the animation
ani = FuncAnimation(fig, update, frames=total_frames, interval=100, repeat=False)
ani.save(‘edgecase_k-fold_cross_validation_in_ml.gif’, writer=’pillow’, fps=8)
plt.close()

View post on imgur.com

Edge Case Animation

What You Will See in This GIF:

Line-by-Line Code Walkthrough

Here is a clean breakdown of the most important logic across our scripts:

Common Setup & Data Generation

The Core of k-fold cross validation in ml

Model Training & Math

Scikit-Learn Production Pipeline

Quick Recap ✅

Test Yourself 🧠

Q1: Why is a simple 80/20 train-test split risky in Machine Learning?

Answer: B

Q2: In 5-fold cross validation in ml, how many times does each individual data point act as the “test” data?

Answer: A

Q3: If your model has near-zero training loss but massive validation loss, what is happening?

Answer: C

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
// STILL BROWSING?
Build along, don't just read.
Get labs & articles matched to what you're into — free, takes 30 seconds.
Start building free
0
Would love your thoughts, please comment.x
()
x