WRITE · TRACE · UNDERSTAND
Scaled Dot-product Attention
A query scores keys; stable softmax turns scores into weights, then a weighted sum mixes values.
Your experiment
attend(query, keys, values) returns one attention output vector using dot products divided by sqrt(len(query)), followed by stable softmax. keys is nonempty, dimensions match, each value has the same width.
Python runs in your browser. Your draft stays on this device.
What to look for
A query scores keys; stable softmax turns scores into weights, then a weighted sum mixes values. Inspect scores, weights and the output vector.
Watch your code work
Ready to experimentWrite a solution or load the worked answer, then choose Trace & check.
Step through arrays, variables and training curves produced by your own Python.