WRITE · TRACE · UNDERSTAND
Byte-pair Merge Step
A BPE training step counts adjacent token pairs, chooses the most frequent, then merges non-overlapping occurrences left to right.
Your experiment
merge_pair(tokens) returns (new_tokens, selected_pair) after one BPE merge. Ties choose the lexicographically smallest pair; joining concatenates token strings. With fewer than two tokens, return (a copy, None).
Python runs in your browser. Your draft stays on this device.
What to look for
A BPE training step counts adjacent token pairs, chooses the most frequent, then merges non-overlapping occurrences left to right. This is a token-string teaching model, not a production tokenizer.
Watch your code work
Ready to experimentWrite a solution or load the worked answer, then choose Trace & check.
Step through arrays, variables and training curves produced by your own Python.