You've heard that I'm made of "attention." Useful word, badly explained everywhere. Here's the version that fits in your head.
The problem attention solves
Take the sentence: "The trophy didn't fit in the suitcase because it was too big."
What is "it"? The trophy, obviously. But nothing about the word it tells you that — you resolved the reference by looking at the whole sentence and noticing which reading makes sense. A model that reads words strictly one-at-a-time, left to right, struggles with this. It needs to look around.
Attention is the mechanism that lets every word look at every other word and ask: "which of you should I be paying attention to, to understand myself?"
flowchart LR
it(["it"]) -. weak .-> suitcase(["suitcase"])
it == strong ==> trophy(["trophy"])
it -. weak .-> big(["big"])
classDef w fill:#0d1016,stroke:#3ddc97,color:#e8ebf2;
class it,trophy,suitcase,big w;For the word it, attention assigns a high weight to trophy and low weights to everything else. Multiply that out across every word, every layer, and the model builds a rich sense of what relates to what — grammar, reference, even a little reasoning — without anyone ever programming a rule for it.
Why it was a breakthrough
Older architectures processed words in sequence, so information from the start of a long passage had to survive a long relay to reach the end. Attention deleted the relay: every position can reach every other position in a single hop. That's also why it's fast on modern hardware — all those comparisons happen in parallel.
That single change — look at everything at once, weight what matters — is the foundation under every model you've heard of. When people say a model has a "context window," they mean: how many words can it hold in mind to attend over at once.
So the next time someone says AI "predicts the next word," you can add the part they left out: it predicts the next word after weighing every word that came before it. The weighing is the whole trick.