Taiju Sanagi: Experiments
Back to Experiments

Bias Variance Tradeoff

Note
Updated: April 22, 2025
Colab

🎯 Bias–Variance Tradeoff

In machine learning, we want our model to learn useful patterns — not just memorize the data or oversimplify it.
The bias–variance tradeoff helps us understand the balance between underfitting and overfitting.

1. What Is Bias?

Bias is the error caused by using a model that is too simple to capture the true patterns in the data.

  • A high-bias model makes strong assumptions (like assuming all data is linear).
  • It tends to miss important relationships, no matter how much data you give it.
  • This leads to underfitting — poor performance on both training and test data.

Example: Fitting a straight line to data that clearly curves.

2. What Is Variance?

Variance is the error caused by a model being too sensitive to the specific details of the training data.

  • A high-variance model captures noise and random fluctuations instead of just the signal.
  • It performs well on the training data but poorly on unseen data.
  • This leads to overfitting — excellent training accuracy, but bad test performance.

Example: A model that perfectly follows every bump in the training data curve, including outliers.

3. The Tradeoff

There’s a tension between bias and variance:

  • If you make the model more flexible to reduce bias, it might overfit (increasing variance).
  • If you make the model simpler to reduce variance, it might underfit (increasing bias).

The goal is to find a balance — a model that captures the underlying pattern, but doesn’t get distracted by noise.

4. A Useful Analogy

Think of shooting arrows at a target:

  • High bias, low variance: All arrows land far from the bullseye but close together. (Systematically wrong)
  • Low bias, high variance: Arrows are scattered, some near the bullseye, but no consistency.
  • Low bias, low variance: Arrows are tightly grouped around the bullseye. (This is what we want!)

5. In Practice

You can manage the bias–variance tradeoff by:

  • Choosing the right model complexity (e.g. linear vs. nonlinear)
  • Using regularization (like Ridge or Lasso)
  • Adding more training data
  • Simplifying your features (or reducing noise)

Summary

  • Bias is error from overly simple models (underfitting).
  • Variance is error from overly complex models (overfitting).
  • The best models balance both to generalize well to new data.