GPT From Scratch — Building a Decoder-Only Transformer

Based on Andrej Karpathy's lecture: "Let's build GPT from scratch, in code, spelled out." Video: https://www.youtube.com/watch?v=kCc8FmEb1nY GitHub: https://github.com/karpathy/nanoGPT

Overview

This article covers how to build a decoder-only Transformer (the architecture behind GPT) from scratch using PyTorch. The implementation is trained on the Tiny Shakespeare dataset (~1MB of all Shakespeare works) to produce character-level text generation.

Key insight: GPT is fundamentally a language model — it models the sequence of tokens and completes them. Under the hood, it's the Transformer architecture from the 2017 paper "Attention Is All You Need".

---

1. Language Modeling Fundamentals

A language model learns the probability distribution of the next token given the previous context:

P(token_n | token_1, token_2, ..., token_{n-1})

ChatGPT does this at the token level (subword units using BPE/tiktoken, ~50K vocabulary), but in this tutorial we work at the character level (65-character vocabulary) for simplicity.

Key difference in tokenization approaches:

Approach	Vocabulary	Sequence Length	Example ("Hi there")
Character-level	65	Very long	[46, 47, ...] — one int per char
BPE/subword (GPT-2)	50,257	Short	[3 ints]
SentencePiece (Google)	Varies	Medium	Subword units

---

2. Data Preparation

Dataset: Tiny Shakespeare

• Single file (input.txt, ~1MB)
• All of Shakespeare concatenated
• ~1M characters
• ~300K tokens when subword-tokenized

Train/Validation Split

The first 90% is used for training, the last 10% is held out as validation to detect overfitting.

Tokenization (Character-Level)

chars = sorted(list(set(text)))
vocab_size = len(chars)  # 65

# Encoder: character → integer
stoi = {ch: i for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]

# Decoder: integer → character
itos = {i: ch for i, ch in enumerate(chars)}
decode = lambda l: ''.join([itos[i] for i in l])

Data Loading Strategy

Rather than feeding the entire sequence at once (computationally prohibitive), random chunks (blocks) are sampled during training:

• Block size (context length): how many previous characters the model sees
• Batch size: how many chunks are processed in parallel

Key trick: A chunk of block_size + 1 characters actually contains block_size individual training examples — one for each position in the context. For example, with block_size=8 and input [18, 47, 56, 57, 58, 1, 15, 47, 58]:

Context → Target
[18] → 47
[18, 47] → 56
[18, 47, 56] → 57
...up to...
[18, 47, 56, 57, 58, 1, 15, 47] → 58

This is not just for efficiency — it also trains the model to handle varying context lengths, which is essential during sampling (generation can start with as little as one character).

---

3. The Bigram Model (Baseline)

The simplest possible language model — it only looks at the current character to predict the next one:

class BigramLanguageModel(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()
        self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)

    def forward(self, idx, targets=None):
        logits = self.token_embedding_table(idx)  # (B, T, C)
        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)
        return logits, loss

    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            logits, loss = self(idx)
            logits = logits[:, -1, :]  # take last time step
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

Result: ~2.5 validation loss — essentially random guessing with character frequency knowledge.

---

4. The Transformer Architecture

Built piece by piece, the final model has these components:

4.1 Token & Position Embeddings

The input tokens are mapped to a continuous embedding space, and position information is added via learned position embeddings:

self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
self.position_embedding_table = nn.Embedding(block_size, n_embd)

4.2 Self-Attention (The Core Innovation)

What it does: Each token "looks at" all previous tokens in the context and aggregates information from them.

The math:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

Where: - Q (Query): "What am I looking for?" - K (Key): "What do I contain?" - V (Value): "What information do I carry?"

Scaled dot-product attention (/√d_k): prevents the dot products from growing too large, which would push softmax into regions with extremely small gradients.

Causal masking: A triangular mask ensures tokens can only attend to previous positions (the autoregressive property), implemented as:

tril = torch.tril(torch.ones(T, T))
wei = wei.masked_fill(tril == 0, float('-inf'))

4.3 Single Head Attention

class Head(nn.Module):
    def __init__(self, head_size):
        super().__init__()
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)
        q = self.query(x)
        wei = q @ k.transpose(-2, -1) * C**-0.5
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = F.softmax(wei, dim=-1)
        v = self.value(x)
        out = wei @ v
        return out

4.4 Multi-Head Attention

Multiple attention heads run in parallel, each learning different relationship patterns. Their outputs are concatenated and projected back:

class MultiHeadAttention(nn.Module):
    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
        self.proj = nn.Linear(n_embd, n_embd)

    def forward(self, x):
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.proj(out)
        return out

Standard configuration: each head operates in 64-dimensional space. With n_embd=384 and 6 heads, each head is 384/6 = 64 dimensions.

4.5 Feed-Forward Network (FFN)

After communication via attention, each token independently processes information through an MLP. The inner layer is 4× wider than the embedding dimension (a key detail from the original paper):

class FeedForward(nn.Module):
    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),
        )

    def forward(self, x):
        return self.net(x)

4.6 The Transformer Block

Attention (communication) and FFN (computation) are combined with residual connections and layer normalization:

class Block(nn.Module):
    def __init__(self, n_embd, n_head):
        super().__init__()
        head_size = n_embd // n_head
        self.sa = MultiHeadAttention(n_head, head_size)
        self.ffwd = FeedForward(n_embd)
        self.ln1 = nn.LayerNorm(n_embd)
        self.ln2 = nn.LayerNorm(n_embd)

    def forward(self, x):
        x = x + self.sa(self.ln1(x))   # Pre-norm formulation
        x = x + self.ffwd(self.ln2(x))
        return x

Residual connections: The "fork" pattern — apply a transformation, then add the result back to the original input. This allows gradients to flow directly through deep networks.

LayerNorm vs BatchNorm: - BatchNorm: Normalizes across the batch dimension (columns) - LayerNorm: Normalizes across the feature dimension (rows) - LayerNorm is preferred in Transformers because it doesn't depend on batch size and behaves identically at train/test time

Pre-norm formulation: A modern refinement — LayerNorm is applied before the sublayer (attention/FFN), not after. Slight departure from the original paper but now standard practice.

4.7 Full GPT Model

class GPTLanguageModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)])
        self.ln_f = nn.LayerNorm(n_embd)
        self.lm_head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding_table(idx)
        pos_emb = self.position_embedding_table(torch.arange(T, device=device))
        x = tok_emb + pos_emb
        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.lm_head(x)
        # ... loss calculation ...
        return logits, loss

    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -block_size:]  # crop to block size
            logits, loss = self(idx_cond)
            logits = logits[:, -1, :]
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

During generation, the context is cropped to block_size — only the last block_size tokens are used as context since the model was never trained on longer sequences.

---

5. Training

Loss Function: Cross-Entropy

loss = F.cross_entropy(logits, targets)

The model outputs logits (raw scores) over the vocabulary for each position. Cross-entropy loss compares these against the true next token.

Hyperparameters (Final Model)

Parameter	Small (demo)	Scaled
Batch size	4	64
Block size	8	256
Embedding dim (n_embd)	32	384
Number of heads	4	6
Number of layers	1	6
Dropout	0.0	0.2
Learning rate	1e-3	1e-3
Parameters	~10M	~10M

Dropout: Randomly disables 20% of neurons every forward/backward pass. Acts as a regularization technique — effectively trains an ensemble of subnetworks.

Training Loop

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
for iter in range(max_iters):
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

Results

Model	Validation Loss	Quality
Bigram (no attention)	~2.50	Random character matching
Single-head attention (demo)	~2.08	Starts forming word-like patterns
Multi-head + LayerNorm	~2.06	Slight improvement
Scaled model (6 layers, 256 block)	~1.48	Recognizable Shakespeare-like text

At validation loss 1.48, the generated text looks structurally like Shakespeare (character names, line format) but is semantically nonsensical.

---

6. Connection to ChatGPT

Scale Comparison

	This tutorial	GPT-3
Parameters	10M	175B
Training tokens	~300K	300B
Ratio	1×	~1,000,000×

Architecture is structurally identical — same embeddings, attention, FFN, residual connections, layer norms.

Two Training Stages

Stage 1: Pre-training (what this tutorial covers) - Train on raw internet text - Model learns to be a document completer, not an assistant - Unconditioned text generation

Stage 2: Fine-tuning / Alignment 1. Supervised fine-tuning: Train on Q&A pairs (human-written) 2. Reward model training: Human raters rank model outputs → train a reward model to predict preferences 3. Reinforcement Learning (RLHF/PPO): Fine-tune the policy to maximize reward model scores

This alignment stage transforms the raw language model from a "document completer" into a "helpful assistant."

---

7. Code Architecture (nanoGPT)

The full nanoGPT repo consists of two main files:

• model.py (~300 lines): The Transformer model itself (near-identical structure to the tutorial)
• Batches all heads into 4D tensors (B, T, H, C) for efficiency
• Uses GELU activation instead of ReLU (to match OpenAI's GPT-2 checkpoints)

• Separates parameters into weight-decay and non-weight-decay groups

• train.py (~300 lines): Training boilerplate

• Checkpoint saving/loading
• Learning rate scheduling
• Distributed training across multiple GPUs
• Model compilation

---

8. Key Takeaways

1. Attention is the core mechanism — it allows tokens to communicate with each other, while the FFN processes information per-token
2. Autoregressive decoding — each token is generated one at a time, conditioned on all previous tokens
3. The architecture has barely changed since the 2017 "Attention Is All You Need" paper — GPT models use a simplified decoder-only version (no encoder, no cross-attention)
4. Scale is everything — the same architecture at 10M parameters (this tutorial) and 175B parameters (GPT-3) differs only in scale, not design
5. Pre-training ≠ alignment — a raw language model is a document completer, not an assistant. The assistant behavior comes from the separate fine-tuning stage