PythiloLesson 7 of 18 · progress kept in this browser

Training, test and overfitting

A model is only useful on data it has not seen. So you hold some data back: train on one part, measure on the other.

import torch

x = torch.arange(10.0)
seed = torch.Generator().manual_seed(0)
order = torch.randperm(
    len(x), generator=seed)
cut = int(len(x) * 0.8)
train_x = x[order[:cut]]
test_x = x[order[cut:]]
print(train_x, test_x)

When the training loss keeps falling but the test loss starts rising, the model is overfitting: it is learning the noise of its training points. A big model on few points does this easily. Weight decay, more data or a smaller model pull it back.

Your task

Write split(x, y, fraction, seed) returning (train_x, train_y, test_x, test_y): shuffle with torch.randperm and a generator seeded with seed, and put the first int(len(x) * fraction) shuffled rows in training. Write gap(train_loss, test_loss) returning how much worse the test loss is, as test_loss - train_loss, and is_overfitting(history) which, for a list of (train_loss, test_loss) pairs, is true when the last test loss is above the lowest test loss seen by more than 10%.

This lesson uses torch, which Python in this browser does not include. Read it here, and run and check it in the Pythilo app on iPhone or iPad.