WRITE · TRACE · UNDERSTAND
Causal Attention Mask
An autoregressive position may read its prefix, including itself, but never later tokens.
Your experiment
causal_weights(scores) takes a square nonempty score matrix and returns row softmax with columns j > i set to probability zero. Use stable exponentials over the unmasked prefix.
Python runs in your browser. Your draft stays on this device.
What to look for
An autoregressive position may read its prefix, including itself, but never later tokens. Mask before softmax and normalize each allowed row.
Watch your code work
Ready to experimentWrite a solution or load the worked answer, then choose Trace & check.
Step through arrays, variables and training curves produced by your own Python.