How good is a language model? The standard number is perplexity: the exponential of the average cross-entropy on text it has not seen.

It reads as "how many tokens the model is choosing between". A model that guesses uniformly from a 50-token vocabulary has perplexity 50. A perfect one has perplexity 1.

import math
import torch
import torch.nn.functional as F

logits = torch.zeros(3, 50)  # no idea
targets = torch.tensor([4, 7, 1])
loss = F.cross_entropy(logits, targets)
print(math.exp(loss.item()))  # 50.0

Lower is better, but only between models with the same tokenizer: splitting text differently changes what one step means.

Write perplexity(logits, targets) returning exp of the mean cross-entropy as a float, for logits (n, vocab) and target ids (n,). Write accuracy(logits, targets) returning the share of positions where the highest-scoring token is the target, as a float.

The explanation opens here once every check passes.