A visual, step-by-step guide to the most cited AI paper of all time.
Understand Transformers from the ground up with practical intuition,
key formulas, architecture details, and training insights.
The Transformer didn't appear out of nowhere. Here's the evolution of sequence modeling.
1990s
Recurrent Neural Networks (RNN)
Process sequences word by word. Forget long-range dependencies. No parallelization.
1997
Long Short-Term Memory (LSTM)
Gates control memory. Better long-range memory. Still sequential. Still slow to train.
2014
Bahdanau Attention + RNN
Attention mechanism added on top of RNNs. First time attention showed its power for translation.
2017
🚀 Transformer — No RNN at all!
Vaswani et al. remove RNNs entirely. Pure attention. Parallel. Faster. Better.
2018–now
BERT, GPT, ChatGPT, Claude...
All modern LLMs are built on the Transformer. This paper started it all.
Key Insight
The fundamental breakthrough was realizing that attention alone — without any recurrence or convolution — is sufficient for sequence-to-sequence tasks.
This enabled massive parallelization during training, which meant you could train on much larger datasets in much less time.
TRAINING TIME COMPARISON
ConvS2S (best before)~9.6×10¹⁸ FLOPs
Transformer (base)3.3×10¹⁸ FLOPs
Transformer uses ~3x less compute than the best prior model
Chapter 01
The Problem with RNNs
Before Transformers, the best sequence models were RNNs and LSTMs. But they had a fundamental flaw.
😰
RNN / LSTM Problems
Sequential processing — can't parallelize
Long sentences cause "context bottleneck"
Early words are compressed into a single vector
Gradient vanishes over long sequences
Slow to train on large datasets
🚀
Transformer Solutions
All tokens processed in parallel
Every token can directly "see" every other token
No information bottleneck — direct connections
Constant path length between any two positions
Train faster, scale to larger datasets
// RNN Context Bottleneck — The sentence "I grew up in France, so I speak fluent..."
I
→
h₁
→
grew
→
h₂
→
up
→
h₃
→
in
···
France
···
so
···
I
···
speak
···
fluent
→
SINGLE VECTOR ⚠️ "France" almost forgotten
→
???
The word "France" (early in the sentence) has almost no influence on predicting "French" at the end. Transformers fix this.
Chapter 02
Step 1 — Tokenization
Before any neural network can process text, it must be converted into numbers. Tokenization is how we break text into pieces called tokens.
// Interactive Tokenizer — type anything below0 tokens
VOCAB SIZE 50,257
METHOD BPE
MODEL GPT-2
How BPE Works — Step by Step
1
Start with characters. Every word is split into individual characters + end-of-word marker.
old</w>+finest</w>
2
Count all adjacent pairs. Find the most frequent character pair across the whole corpus.
es → "es" appears 13x (finest×9 + lowest×4)
3
Merge the most frequent pair. Replace all occurrences with a new token.
finest</w>
4
Repeat until vocabulary size reached. GPT-2 performs 50,000 merges to build its 50,257-token vocabulary.
Each dimension uses a different frequency sine/cosine wave. Low dimensions oscillate fast (capture local order); high dimensions oscillate slowly (capture global position).
Dimension:dim 0
Chapter 04
The Core — Self-Attention
This is the heart of the Transformer. Every token looks at every other token and decides how much to "attend" to each one.
Query, Key, Value — The Library Analogy
Q
Query
"What am I looking for?" — You walking into the library with a request. Computed as: Input × W_Q
K
Key
"What do I have?" — Each book's label on the shelf advertising its contents. Computed as: Input × W_K
V
Value
"What do I actually give you?" — The actual content of the book once you check it out. Computed as: Input × W_V
The Scaled Dot-Product Attention Formula
Attention(Q, K, V) = softmax( Q · Kᵀ / √d_k ) · V
Q · Kᵀ
Attention Scores
Dot product between every Query and every Key. High score = high relevance.
÷ √d_k
Scaling
Prevents dot products from getting too large (which causes tiny gradients). d_k = 64 in the base model.
softmax
Normalization
Converts raw scores into probabilities (0 to 1, summing to 1). These are the attention weights.
· V
Weighted Sum
Multiply weights by Values. High-attention tokens contribute more to the output context vector.
// Interactive Self-Attention — click a word to see its attention weightsSentence: "The cat sat on the mat"
SELECT QUERY TOKEN:
ATTENTION WEIGHTS:
ATTENTION HEATMAP:
Step-by-Step Attention Computation
1
Compute Q, K, V matrices
Q = X · W_Q shape: (seq_len × d_k)
K = X · W_K shape: (seq_len × d_k)
V = X · W_V shape: (seq_len × d_v) // In base model: d_k = d_v = 64, d_model = 512
2
Compute raw attention scores
scores = Q · Kᵀ shape: (seq_len × seq_len) // Every token attends to every other token — O(n²) complexity
3
Scale + Softmax
scaled = scores / √d_k // √64 = 8
weights = softmax(scaled) // sums to 1 per row
4
Weighted sum of Values
output = weights · V shape: (seq_len × d_v) // Each token now has a context-aware representation
Chapter 05
Multi-Head Attention
Instead of one attention function, we run h=8 attention heads in parallel. Each head learns to focus on different aspects of language.
// 8 Attention Heads — each learns different patternsd_k = d_model / h = 512/8 = 64 per head
Why multiple heads? A single head averages all attention — it misses nuance. Multiple heads can specialize: one for syntax, one for semantics, one for coreference resolution.
The paper's ablation study showed 8 heads optimal. Too few — can't capture all relationships. Too many — dimensions per head become too small.
ABLATION: NUMBER OF HEADS (Table 3)
Chapter 06
Full Architecture
The complete Transformer: an Encoder stack and a Decoder stack, connected through cross-attention.
ENCODER — N=6 identical layers
🔍
Multi-Head Self-Attention
All positions attend to all positions
d_model=512
➕
Add & Norm
Residual connection + LayerNorm
⚡
Feed-Forward Network
FFN(x) = max(0, xW₁+b₁)W₂+b₂
d_ff=2048
➕
Add & Norm
Second residual + LayerNorm
DECODER — N=6 identical layers
🔒
Masked Multi-Head Self-Attention
Can't see future tokens (causal mask)
masked
➕
Add & Norm
Residual + LayerNorm
🔗
Cross-Attention
Q from decoder, K/V from encoder output
cross
⚡
Feed-Forward Network
Same as encoder FFN
d_ff=2048
Key Hyperparameters — Base vs Big Model
Parameter
Symbol
Base Model
Big Model
Meaning
Layers
N
6
6
Stack depth
Model dimension
d_model
512
1024
Embedding size
FFN inner dim
d_ff
2048
4096
4× d_model
Attention heads
h
8
16
Parallel heads
Key/Value dim
d_k, d_v
64
64
d_model / h
Dropout
P_drop
0.1
0.3
Regularization
Parameters
—
65M
213M
Total trainable
Chapter 07
Training — Details
The optimizer, learning rate schedule, and regularization that made the Transformer work.
Optimizer
Adam (β₁=0.9, β₂=0.98, ε=10⁻⁹)
Standard Adam with unusual β₂ (very close to 1) to maintain longer history of second moment. The small ε prevents division by zero.
Instruction tuning: post-training adapts base model to follow human prompts and formats.
Sampling at inference: temperature, top-k, and top-p convert logits into controlled generation behavior.
Emergent behavior: scaling data, parameters, and compute often unlocks capabilities nonlinearly.
Inference Pipeline (Token by Token)
Step A
Prompt tokens enter the model and produce logits for the next token.
Step B
Decoding strategy selects one token, appends it, and repeats.
Step C
KV cache stores past attention states to avoid recomputing old tokens each step.
Practical implication: latency and memory become first-class constraints at long context lengths.
Why Quality Improves with Scale (Intuition)
More Data Expands language coverage and reduces blind spots in rare patterns.
More Parameters Increases representational capacity for nuanced concepts and relations.
More Compute Lets optimization run longer and deeper, improving final perplexity and utility.
Better Post-Training Alignment and instruction tuning convert raw capability into practical behavior.
Chapter 09
LLM Lens — Missing Intuition Added
Added from modern Transformer explainers: next-token probabilities, scaling law intuition, and how pretraining transfers into downstream tasks.
// Next Token Probability Lab
TEMP1.0
Autoregressive generation (one token at a time)
Why "Large" Matters
As parameter count grows, models often cross capability thresholds. Some tasks stay near-zero, then jump sharply once scale is sufficient.
Pretraining → Fine-Tuning → Transfer
1) Pretraining learns general language statistics. 2) Fine-tuning adapts to a task. 3) Transfer learning reuses the same base model across many tasks.
Context Vectors, Not Static Tokens
A token embedding is static. Self-attention transforms it into a context vector that changes with sentence context.
Mask Before Softmax
Causal masking should set future positions to $-\infty$ before softmax, preventing probability leakage into future tokens.
Dropout in Attention
Attention dropout regularizes by randomly dropping valid attention links during training while preserving causal constraints.
Causal Masking Lab (Before vs After Softmax)
This visual shows why masking must happen before softmax. If future tokens participate in softmax first, they steal probability mass even if you zero them later.
// Decoder Masking Simulator
QUERY POS4
Normalization Lab: Simple vs Softmax
Based on the blog's examples, this shows why softmax creates sharper, valid probability distributions and handles negative scores safely.
// Attention Score Normalization
TOP SCORE6
Simple normalize x/sum(x)
Softmax exp(x)/sum(exp(x))
Chapter 10
Practical Limits and Modern Fixes
The original Transformer is brilliant, but not perfect. Here are the major trade-offs and what modern architectures do to improve them.
Quadratic Attention Cost
Self-attention compares every token with every token, so memory and compute scale as O(n²). Long contexts become expensive.
Context Length Ceiling
Longer context windows require larger KV caches and higher latency. This affects both training cost and inference throughput.
Data + Alignment Needs
Raw scale alone is not enough. Data quality, instruction tuning, and alignment methods strongly affect usefulness and safety.
The Transformer outperformed every previous model while using a fraction of the compute.
Transformer (big) — EN→DE
28.4
BLEU Score
Previous best (GNMT Ensemble)
26.3
BLEU Score — EN→DE
Transformer (big) — EN→FR
41.8
BLEU Score
Previous best (ConvS2S Ensemble)
41.3
BLEU Score — EN→FR
BLEU Score Comparison — EN→DE (All Models)
Self-Attention vs Recurrent vs Convolutional (Table 1)
Layer Type
Complexity/Layer
Sequential Ops
Max Path Length
Winner?
Self-Attention
O(n² · d)
O(1)
O(1)
✓ Best
Recurrent
O(n · d²)
O(n)
O(n)
✗
Convolutional
O(k · n · d²)
O(1)
O(log_k n)
~
Self-Attn (restricted)
O(r · n · d)
O(1)
O(n/r)
~
n = sequence length, d = dimension, k = kernel size, r = restriction neighborhood. Self-attention wins on path length (O(1) means direct connection!) but has quadratic complexity in sequence length.
Deep Dive
The Quadratic Bill
Every token attending to every other token is what made the Transformer parallel — and what made long context expensive. This is the cost side of the paper's Table 1, in numbers you can move yourself.
// Sequence Length vs Attention Cost — pick a context windowcells = n²
sequence length (tokens)
ATTENTION MATRIX PER LENGTH — CELLS (n²) AND fp16 MEMORY, LOG-SCALED BARS:
Counts one attention matrix — one head, one layer, fp16 — before softmax copies and backward storage. Causal decoding fills only the lower triangle; the quadratic clock is unchanged.
The Cost of One Attention Matrix
cells = n² · memory ≈ n² × 2 bytes
n²
Pairwise grid
Every query must score every key before softmax can normalize even one row. Table 1 prices a self-attention layer at O(n²·d) — the n² is this grid.
× 2 bytes
fp16 cell
One score stored at half precision. Softmax and the backward pass want extra copies, so this is the floor of the bill, not the total.
× h × N
Stacked everywhere
The base model fills this grid once per head (8) in each of its 6 encoder + 6 decoder layers — 96 grids every forward pass.
÷ √d_k
Scaling ≠ savings
The scale factor keeps gradients healthy; it changes nothing about the cell count. Only fewer tokens — or fewer computed pairs — shrinks the bill.
💸
What the Square Charges
Doubling the window quadruples the cells — 512 → 32K is ×4,096
Scores must exist before softmax can normalize a single row
Training also stashes them for the backward pass
⚡
What the Square Buys
Any two positions one hop apart — O(1) maximum path length
All positions scored in parallel, not n sequential steps
The paper's escape hatch — restrict to r neighbors — costs O(r·n·d) but degrades path length to O(n/r)
FlashAttention (2022)
Keeps the exact O(n²) math but never materializes the matrix — tiles and online softmax streamed through on-chip SRAM. Memory falls toward O(n); it is now the default kernel in every major stack.
Sparse Attention (2020)
Longformer and BigBird attend in local windows plus a few global positions — cost linear in n for a fixed window. The paper's "restricted" row in Table 1 sketched this trade years early.
Linear Attention (2020)
Kernel feature maps (linear transformers, Performer) reorder the arithmetic so cost grows ≈O(n), approximating softmax attention — trading exactness for length.
The KV Cache Surtax
At inference, step t attends to all t cached keys — generating n tokens costs ~n²/2 attention work. MQA/GQA shrink that cache; they do not repeal the square.
Context-Window Economics
512 — BERT (2018): sentences and paragraphs fit; the quadratic bill was invisible, so nobody felt it.
1K–2K — GPT-2 / GPT-3 (2019–2020): windows doubled and the bill quadrupled — still manageable with naive kernels.
8K–128K — GPT-4 class (2023): affordable because FlashAttention-class kernels stopped materializing the matrix, with sparse windows on top.
1M+ — Gemini 1.5 class (2024): millions of tokens from a mixture of tricks — sparsity, chunked recurrence, position interpolation — not one silver bullet.
Why the Ceiling Moved by Tricks
Memory: fused kernels compute exact attention without ever storing the n² scores.
FLOPs: sparsity and linear-attention approximations change the exponent itself.
Positions: RoPE-style interpolation stretches models trained short to run long cheaply.
Verdict: the window grew from 512 to millions by refusing to pay the square — not by paying it.
Test Yourself
Quick Quiz
Check your understanding of the key concepts from the paper.
Reference
All Key Formulas
A complete reference of every important formula from the paper.
Scaled Dot-Product Attention
Attention(Q,K,V) = softmax(QKᵀ/√d_k)V
Core attention mechanism. Scale by √d_k to prevent gradient saturation. Apply softmax to get weights, multiply by values.