Training
Where the numbers come from: next-token prediction, cross-entropy loss, and a gradient-descent ball on a real loss surface — with a learning rate you can push until it diverges.
A language model is trained by showing it text and asking it to predict the next token. The loss is −log p(correct token): confident and right is cheap, confident and wrong is expensive. Gradient descent nudges every weight to make the correct token slightly more likely, trillions of times.
Both panes here run the real mathematics: the prediction pane applies genuine cross-entropy gradient steps to a live logits table (watch the target token's probability grow), and the loss-surface pane runs genuine gradient descent on a quadratic bowl — set the learning rate too high and it genuinely oscillates and diverges, not as a canned animation but because the mathematics does.