Mirza Ishraq
← All writing

By · · 9 min read

Overfitting vs Underfitting: How to Tell Which One You Have

Overfitting vs underfitting comes down to one comparison: error on the data the model trained on, against error on data it has never seen.

An overfit model scores great on its training data and badly on new data. It memorized the noise.

An underfit model scores badly on both. It was too simple to catch the trend in the first place.

Some of what’s below comes from my grad work in Big Data Analytics at SDSU, which was before the AI boom, when you built and tested your own models. The rest comes from notes I made in a machine learning course on Udemy.

The experiment is my own. Every number below is real output from NumPy 2.2.6, including the ugly ones.

What is overfitting vs underfitting?

Data has two parts: the pattern you care about and noise. Noise is the random, irrelevant variation that doesn’t mean anything.

Measurement error is noise. So is a student who had a bad morning.

Overfitting is fitting the noise. The model bends itself to hit every training point, including the ones that were just bad luck.

Training error comes out low. Error on new data comes out high.

In bias and variance terms, that’s low bias and high variance. Variance here means how much the fitted model would change if you resampled the training data.

Underfitting is the opposite. The model is too simple to follow the trend, so it misses even the training points.

Both errors come out high. That’s high bias and low variance.

A straight line through a curve is the classic case.

The trap is that training error alone can’t tell you which one you have. A model that memorizes every training point scores zero on the training data, and that number tells you nothing about how it will behave in the real world.

What does overfitting look like in Python?

Let’s fake some data so we know the right answer. The true pattern rises and then levels off.

I sample 15 training points, add noise with a standard deviation of 0.3, then draw 200 fresh points as a test set.

That noise sets a floor. Even a perfect model can’t beat the noise variance on average.

That’s the spread of the noise itself, 0.3 squared, or 0.09. So a test error near 0.09 is about as good as it gets, and a lucky test set can land a little under it.

I fit polynomials of different degrees to the same 15 points using NumPy’s Polynomial.fit.

Degree 1 is a straight line. Degree 14 has 15 coefficients for 15 points, so it can hit every one.

import numpy as np
from numpy.polynomial import Polynomial

rng = np.random.default_rng(7)

def true_curve(x):
    return 3 * (1 - np.exp(-5 * x))

x_train = np.linspace(0, 1, 15)
y_train = true_curve(x_train) + rng.normal(0, 0.3, 15)
x_test = rng.uniform(0, 1, 200)
y_test = true_curve(x_test) + rng.normal(0, 0.3, 200)

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

print("degree  train MSE  test MSE")
for degree in (1, 4, 14):
    model = Polynomial.fit(x_train, y_train, degree)
    print(f"{degree:>6}  {mse(y_train, model(x_train)):>9.3f}  {mse(y_test, model(x_test)):>8.3f}")

Here’s what it printed:

degree  train MSE  test MSE
     1      0.222     0.255
     4      0.029     0.086
    14      0.000     7.005

Read it row by row:

  • Degree 1 is underfitting. Both errors are high, and they’re close to each other. The line can’t follow the curve.
  • Degree 4 is the good fit. Test error is 0.086, right at the noise floor.
  • Degree 14 is overfitting. Training error is exactly zero, and test error is 7.005, about 80 times worse than degree 4.

I also swept every degree from 1 to 14. Training error only ever went down, from 0.222 to 0.000.

Test error fell until degree 4, crept up after that, and blew up past degree 11: 0.132 at degree 11, 0.626 at 12, and 7.005 at 14.

This is one random draw. I re-ran the setup with eight different seeds. The size of the blow-up changed a lot, but degree 14 always scored 0.000 on training and at least 2.0 on test.

How do you spot overfitting in a real model?

You can’t plot a model with fifty features and eyeball the curve. So you compare numbers instead. Split your data, then look at both errors together:

  • Both high, and close together: underfitting.
  • Training low, test much higher: overfitting.
  • Both low, and close together: a good fit.

That’s why supervised learning uses a train and test split, and often a third validation split. If you’ve followed my KNN classifier with scikit-learn, that’s the same reason the test set stays untouched until the end.

For neural networks and other models trained step by step, there’s a second view: error over time. One epoch is one full pass of the training data through the model.

Plot training error and validation error after each epoch and you get two curves. If neural networks are new to you, my TensorFlow tensors guide starts from the basics.

A healthy model shows both falling and levelling off. An overfitting model shows the training curve still falling while the validation curve turns upward.

When should you stop training?

At the point where validation error turns up. Everything after that is memorization.

To see it, I trained the same kind of model, a degree-14 polynomial, with gradient descent instead of solving for the best fit in one step.

The code writes it in a Chebyshev basis, which spans the same polynomials but keeps the numbers well behaved. Gradient descent nudges the weights a little after every epoch, which is how neural networks learn.

This time I used three sets. The training set fits the weights.

The validation set picks the stopping epoch. The test set only grades the final result, so it never influences a choice.

import numpy as np
from numpy.polynomial import chebyshev as C

rng = np.random.default_rng(7)

def true_curve(x):
    return 3 * (1 - np.exp(-5 * x))

x_train = np.linspace(0, 1, 15)
y_train = true_curve(x_train) + rng.normal(0, 0.3, 15)
x_val = rng.uniform(0, 1, 200)
y_val = true_curve(x_val) + rng.normal(0, 0.3, 200)
x_test = rng.uniform(0, 1, 200)
y_test = true_curve(x_test) + rng.normal(0, 0.3, 200)

DEGREE = 14
def features(x):
    return C.chebvander(2 * x - 1, DEGREE)

A_train, A_val, A_test = features(x_train), features(x_val), features(x_test)

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

w = np.zeros(DEGREE + 1)
learning_rate = 0.1
best_val, best_epoch, best_w = np.inf, 0, w.copy()

for epoch in range(1, 200_001):
    gradient = 2 * A_train.T @ (A_train @ w - y_train) / len(y_train)
    w -= learning_rate * gradient
    val_error = mse(y_val, A_val @ w)
    if val_error < best_val:
        best_val, best_epoch, best_w = val_error, epoch, w.copy()
    if epoch in (10, 1_000, 10_000, 50_000, 200_000):
        print(f"epoch {epoch:>7}  train {mse(y_train, A_train @ w):.3f}  validation {val_error:.3f}")

print(f"best validation error {best_val:.3f} at epoch {best_epoch}")
print(f"test error at that epoch {mse(y_test, A_test @ best_w):.3f}")
print(f"test error if we kept going {mse(y_test, A_test @ w):.3f}")

The output:

epoch      10  train 0.088  validation 0.442
epoch    1000  train 0.001  validation 0.317
epoch   10000  train 0.001  validation 0.138
epoch   50000  train 0.000  validation 0.368
epoch  200000  train 0.000  validation 2.646
best validation error 0.138 at epoch 10919
test error at that epoch 0.127
test error if we kept going 1.183

Three things stand out.

First, training error was already tiny by epoch 1,000 and stayed near zero. If I had only watched that column, I’d have said everything looked great.

Second, validation error kept improving long after training error stopped moving.

It bottomed out at epoch 10,919, then climbed to 2.646 by epoch 200,000 while training error sat at 0.000. The stopping point is where validation turns upward, not where training flattens.

Third, look at the test error. Stopped at the best epoch, the same model scores 0.127 on fresh data. Left running, it scores 1.183.

This trick is called early stopping. It’s a small fix with a big effect.

It also beat the one-step fit from earlier, which scored 7.005 on test. Same degree-14 family, very different result. Both scores come from 200 fresh points on the same curve.

How do you fix overfitting and underfitting?

The two need opposite fixes.

To fix underfitting, give the model more room:

  • Use a more flexible model, such as a higher polynomial degree or a deeper network.
  • Add features that carry real signal.
  • Train for longer, since an underfit model may simply not have converged yet.
  • Reduce any regularization you added.

To fix overfitting, take room away, or add evidence:

  • Get more training data. It’s the most reliable fix, and often the hardest.
  • Use a simpler model. Degree 4 beat degree 14 here.
  • Stop early, as above.
  • Add regularization, which penalizes large weights. For neural networks, dropout does something similar.
  • Use cross-validation, so one lucky split can’t fool you.

Most models have a dial for this. The number of neighbors k in KNN is one such dial.

With k = 1 each point is its own neighbor, so the model memorizes the training set. A large k smooths so much that it starts to underfit.

The simplest model of all is a straight line, and it’s a good place to start. I covered how it’s fitted in linear regression in Python. The scikit-learn docs also have a worked underfitting vs overfitting example if you want a second take.

Where does overfitting matter outside a toy problem?

Two of my SDSU projects sit right on this problem. Both are described in my grad projects repo.

The first was my individual final project for LING 583, a statistical text analysis class.

I used the 2020 US Election Tweets dataset from Kaggle to predict which candidate a tweet was about, from the text alone. I trained several different classifiers, then brought in the hashtags to see how they related to each candidate.

Comparing classifiers is a model-selection problem, and it’s the reason this whole post exists.

The only fair way to pick between them is how each one does on tweets it never trained on. A classifier with the best training score has told you nothing yet.

My notebook ended with a summary of the preprocessing, the models, the predictions, the limitations, and the results.

The limitations part is a habit worth keeping. A model that only reports its wins hasn’t really been tested.

The second was my part of the BDA 600 capstone. I built an LSTM for time series prediction on crypto data, and used the VADER sentiment analyzer on Twitter posts.

The project’s stated goal was a tool for cryptocurrency investors, and the team published the result as a StoryMap called Forecast and Predict Crypto Trend.

Time series overfit in an expensive way. A model can fit the past almost perfectly, and the past is exactly the data it has already seen.

The test that counts is forward in time. That’s why you split by date, not at random: train on the earlier dates and test on the later ones. A random split lets the model peek at the future.

What should you remember about overfitting vs underfitting?

  • Training error alone is deceptive. Always measure on data the model hasn’t seen.
  • Both errors high means underfit. Training low and test high means overfit.
  • More complexity never raises training error. It can raise test error.
  • Pick your stopping point, or your model, using a validation set. Grade once on the test set.
  • The noise floor is real. Chasing zero error means chasing noise.

Have you ever trained a model that looked perfect until it met new data, and what gave it away?