Training, test and overfitting
A model is only useful on data it has not seen. So you hold some data back: train on one part, measure on the other.
import torch
x = torch.arange(10.0)
seed = torch.Generator().manual_seed(0)
order = torch.randperm(
len(x), generator=seed)
cut = int(len(x) * 0.8)
train_x = x[order[:cut]]
test_x = x[order[cut:]]
print(train_x, test_x)When the training loss keeps falling but the test loss starts rising, the model is overfitting: it is learning the noise of its training points. A big model on few points does this easily. Weight decay, more data or a smaller model pull it back.
Your task
Write split(x, y, fraction, seed) returning (train_x, train_y, test_x, test_y): shuffle with torch.randperm and a generator seeded with seed, and put the first int(len(x) * fraction) shuffled rows in training. Write gap(train_loss, test_loss) returning how much worse the test loss is, as test_loss - train_loss, and is_overfitting(history) which, for a list of (train_loss, test_loss) pairs, is true when the last test loss is above the lowest test loss seen by more than 10%.
The explanation opens here once every check passes.