The Google Paper That Changed Everything in AI

Not this Transformers!

Have you ever played Chinese Whispers?

20 people stand in a line. The first person whispers a message. By the time it reaches the end? Completely mangled. Not because anyone was careless, just because information degrades through a chain.

That's exactly how RNNs worked with language.

In Post 2, we talked about how RNNs finally gave machines sequential memory, each word updating a hidden state that gets passed to the next word. Real progress. But the hidden state was like a Post-it note. Write too much on it, and you start writing over the old stuff.

By the time the model reached word 50 of a long sentence, word 1 was basically gone. Chinese whispers, but for context.

Here's the test case that broke RNNs completely:

"The animal didn't cross the street because it was too tired."

What does "it" refer to? You knew instantly. The animal. Not the street. Because streets don't get tired.

An RNN had to get to "tired" after passing the context of "animal" through every single word in between. By then, the connection was too diluted. The model would guess wrong on sentences like this, or hedge awkwardly, because the relevant context was too far back.

This is the exact problem a team at Google set out to solve in 2017.

Their paper's title: "Attention Is All You Need."

Group chat vs the whisper chain

Let me give you the Transformer's core idea before we touch a single formula.

RNN world: it's a chain of whispers. Priya tells Arjun the weekend plan, Arjun tells Rahul, Rahul tells you. By the time it reaches you, "meet at 7 at the new café near the station" has become "maybe something at 8? somewhere near stuff."

Transformer world: it's a group chat. Everyone sees everyone's messages at once. When someone says "that place", every person in the chat knows exactly which place they mean, because they all read the same thread simultaneously.

That's the Transformer. Every word looks at every other word at the same time. Directly. In parallel. No chain. No forgetting.

The mechanism that makes this possible is called self-attention.

Self-Attention: What Does This Word Mean Here?

Here's the thing about language: the same word means different things depending on context.

• "I went to the bank to deposit money."

• "I sat on the bank of the river."

Same word. Completely different meanings. An N-gram or Bag of Words model would treat both "bank"s identically. A Transformer uses self-attention to figure out, based on the surrounding words, which "bank" this actually is.

Self-attention asks, for every word in the sentence: how much should I pay attention to every other word to understand what I mean?

"The animal didn't cross the street because it was too tired."

When the model processes "it", self-attention lets it look across the entire sentence simultaneously and compute: how relevant is "animal" to understanding "it"? Very. How relevant is "street"? Not much. How relevant is "tired"? A lot. It's the clue that disambiguates.

The result: "it" gets a strong connection to "animal" and the ambiguity is resolved.

Q, K, V: The Three Questions

Now let's get into how this actually works. I'll show you the mechanics, but I'll keep it grounded in intuition the whole way.

Self-attention uses three vectors for every word: Query (Q), Key (K), and Value (V). These come from multiplying the word's embedding by three different learned weight matrices.

Here's the mental model I want you to use:

Think of a library. You walk in with a query, "I want something about space travel." Every book in the library has a key, a summary tag on the spine. You compare your query against every key to find how relevant each book is. Then you pull the values, the actual content, from the most relevant books, weighted by how well they matched.

That's exactly what happens in self-attention:

• Q (Query): "What am I looking for?" The current word asking: what context do I need?

• K (Key): "What do I contain?" Every word advertising its content

• V (Value): "What do I actually give you?" The information transferred if you're relevant

The attention score between word i and word j is:

$$\text{score}(i, j) = Q_i \cdot K_j^T$$

This is a dot product. It measures how aligned the query and key vectors are. High alignment = high score = "these two words are relevant to each other."

Then we run all the scores through softmax to turn them into probabilities that sum to 1:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Let's unpack each piece:

• QKᵀ: the dot product of all queries against all keys. One score per pair of words.

• ÷√dₖ: we divide by the square root of the dimension of the key vectors. Why? Dot products get very large when vectors are long, pushing softmax into regions where gradients nearly vanish. This keeps the numbers stable.

• softmax(...): converts raw scores into attention weights. If the score between "it" and "animal" is 8.2 and between "it" and "street" is 1.1, softmax turns these into something like 0.89 and 0.03.

• × V: multiply each value vector by its attention weight and sum them up. High-weight words contribute more to the final representation. Low-weight words contribute almost nothing.

The output for "it" is now a weighted blend of information from every word in the sentence, dominated by "animal" and "tired," almost ignoring "street" and "the."

A Worked Example

If you want to see the actual numbers, here they are. If not, skip straight to Multi-Head Attention.

Let me make this concrete. Take a tiny 4-word sentence: "The cat sat."

After embedding, each word becomes a vector. After multiplying by the learned weight matrices, we get Q, K, V for each word.

For "cat" as the query, we compute dot products against every key.

Say the weight matrices produced these Q and K vectors (2D for simplicity — real models use 64+ dimensions):

• Q_cat = [1.0, 0.5]

• K_The = [0.6, 0.4]

• K_cat = [1.8, 1.2]

• K_sat = [1.2, 0.6]

The dot product for cat → The: (1.0 × 0.6) + (0.5 × 0.4) = 0.6 + 0.2 = 0.8

Do this for every pair:

Divide by √dₖ (say dₖ = 4, so √4 = 2):

Apply softmax:

• e^0.4 ≈ 1.49, e^1.6 ≈ 4.95, e^1.05 ≈ 2.86 → sum ≈ 9.30

• Weights: The = 0.16, cat = 0.53, sat = 0.31

The output vector for "cat" = 0.16 × V_The + 0.53 × V_cat + 0.31 × V_sat

"Cat" is paying most attention to itself (makes sense, its own meaning is most relevant to itself), some attention to "sat" (the action it's performing), and less to "The."

This contextualised vector for "cat" now carries information not just about the word "cat" in isolation, but about "cat" in the context of this specific sentence.

Multi-Head Attention: Multiple Lenses at Once

One self-attention pass captures one type of relationship. But language has multiple kinds of relationships happening simultaneously.

Think about how you process a sentence:

• Part of your brain tracks grammatical roles (subject, verb, object)

• Another part tracks semantic similarity (is this word related in meaning?)

• Another tracks coreference (what does "it" refer to?)

• Another tracks syntactic dependencies (what verb does this noun belong to?)

Multi-head attention does exactly this. Instead of running self-attention once, you run it h times in parallel, each with different weight matrices (Wᵢ^Q, Wᵢ^K, Wᵢ^V):

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$

where each head is:

$$\text{head}_i = \text{Attention}(QW_i^Q,\ KW_i^K,\ VW_i^V)$$

Each head projects Q, K, V into a lower-dimensional space (if the model dimension is 512 and you have 8 heads, each head works in 64 dimensions). They all run simultaneously. Then all h outputs are concatenated and projected back to the original dimension via W^O.

The result: different heads naturally specialise. In a trained model, some heads have been shown to track syntactic structure, others track coreference, others track positional proximity, not because anyone programmed this, but because the model learned these are useful things to track.

Go back to the Transformer Explainer and look at the attention heads. Click between them. Notice how different heads light up different relationships for the same word. That's specialisation emerging from training.

But Wait, Where's the Word Order?

Here's something that should be bothering you.

Self-attention lets every word attend to every other word simultaneously. That means it treats the sentence as an unordered set of tokens. "The cat sat" and "Sat cat the" would produce identical attention computations, because every word still gets to attend to every other word.

But order matters. "The dog bit the man" ≠ "The man bit the dog."

RNNs had this for free. They processed words sequentially, so order was baked in. The Transformer gave up sequence processing for parallelism. It needs to put order back in manually.

That's what positional encoding does. Before the input even reaches the attention mechanism, we add a position vector to each word's embedding. The original paper used sine and cosine functions at different frequencies:

PE(pos,2i)=sin⁡(pos100002i/d) PE(pos,2i+1)=cos⁡(pos100002i/d)

Where pos is the position (0, 1, 2...) and i is the dimension index. These functions create a unique "fingerprint" for each position, the model can learn to read these fingerprints and know where in the sequence each word sits.

Modern models (like the ones powering ChatGPT and Claude) use learned positional embeddings or more advanced schemes like RoPE (Rotary Position Embedding) and ALiBi. RoPE is a different mechanism entirely: position is encoded into the attention computation rather than added to embeddings. But the goal is the same: tell the model where each token sits.

The Encoder and Decoder

The original Transformer had two halves: an encoder and a decoder. Understanding why clarifies a lot about how modern LLMs work.

The Encoder reads the input and builds a rich contextual representation of it. For each layer:

1. Multi-head self-attention (every word attends to every other word in the input)

2. Feed-forward network (processes each position independently)

3. Add & Norm (residual connection + layer normalisation)

The encoder output is a set of contextualised vectors, one per input token, that capture the meaning of each word in the context of the whole input.

The Decoder generates the output, one token at a time. Each decoder layer has three sub-layers:

1. Masked self-attention: the decoder can only attend to positions it has already generated (masking prevents it from "seeing the future" during training)

2. Cross-attention: the decoder attends to the encoder's output, connecting the generated text back to the input

3. Feed-forward network

The mask in masked self-attention is critical. During training, the model sees the full target sequence. The mask forces it to only use the words up to position t when predicting position t+1, otherwise it's just copying the answer.

Translation was the original task: encode the English sentence, decode it into French token by token, with each generated token attending back to the full encoded English. The encoder built meaning; the decoder generated output informed by that meaning.

The Full Picture

Let's zoom out. Here's what happens when you type a message to an LLM:

1. Tokenisation: your text is split into tokens (roughly word-pieces)

2. Embedding: each token becomes a vector

3. Positional encoding: position information added to each vector

4. N × Transformer blocks: each block runs multi-head self-attention + feed-forward, building richer and richer representations

5. Output projection: the final layer maps to vocabulary size, one score per token

6. Softmax + sampling: convert scores to probabilities, sample the next token

Repeat from step 3 for each new token until the model generates an end-of-sequence token.

Every word you see in a response came from this loop. The temperature slider in the Transformer Explainer controls how deterministic the sampling is. Low temperature = always pick the highest probability token. High temperature = more randomness.

Want to see self-attention working on a real model? Source: transformer-explainer.poloclub.github.io

What's next?

We now have the brain. The architecture. The mechanism that went from "the animal didn't cross the street because it was too tired" to "the answer is: the animal."

But here's the thing: a freshly initialised Transformer is just random weights. It doesn't know anything.

How do you take this architecture and turn it into something that can write code, summarise documents, and hold a conversation in 50 languages?

That's pre-training, fine-tuning, and RLHF.

The next post is about how LLMs are actually born, and why the training process is as interesting as the architecture.

This is Post 4 of Build the Brain*: the series going from zero to building production agentic AI systems.*

Next up: How LLMs Are Born: Pre-training, Fine-tuning, RLHF → Post 5

If this clicked, share it. And go play with the Transformer Explainer, seeing attention weights in a live model makes everything click faster than any diagram can.