Building a Language Model From Scratch, in Eleven Stages

I spent two days building a language model from scratch with MLX. Not wiring up a library — writing the attention, the tokenizer, the training loop, the sampling, line by line — and then pushing it forward one stage at a time: a character-level bigram, a hand-written transformer, my own BPE implementation, pretraining a 29.9M model on 390 million tokens, then KV cache, quantisation, reinforcement learning, curiosity, and finally swapping in a real open-source model and hooking it up to tools and web search.

Eleven stages. The valuable part isn't the working thing at the end. It's the pile of numbers measured along the way — several of which came out the exact opposite of my intuition.

A 25× comparison

Same task: two-digit times three-digit multiplication, answered directly, no scratch paper.

First, train it with reinforcement learning. LoRA + GRPO, 120 rounds, 21 minutes:

37.3%  →  39.7%        +2.3 points

Then a different approach: train nothing at all, hand it a calculator, and let it decide whether to call it.

39.8%  →  99.2%        +59.4 points
76 of 128 questions flipped to correct, 0 broken

A 25× difference — and the second one didn't change a single byte of the weights.

That comparison laid out the through-line of the whole project.

The through-line

Reinforcement learning amplifies behaviour that is already there. It cannot create an ability that isn't. An ability that isn't there gets bolted on from outside.

I didn't read that anywhere. Four experiments measured it out, from two directions.

The first direction is failure. I tried to teach the model to write stories containing a dragon. The baseline was 0.4%; hundreds of rounds later it hadn't moved at all, and the system's own curriculum algorithm concluded the task showed "no learning progress" and dropped it. That isn't the algorithm failing — that's the algorithm being right. There simply isn't enough "dragon" in the weights to amplify.

Mental multiplication is the same thing in another costume. It asks the model to complete a multi-step calculation inside a single forward pass, and its 37% accuracy is probably scraped together from pattern-matching on partial products. RL can re-weight existing behaviour; it cannot invent a new algorithm. Hence 2.3 points.

The second direction is success. Hand it a calculator: +59.4 instantly. More interesting is the last stage, where I used RL to teach the model when it should look something up.

It had been searching for 62% of the facts it already knew — the capital of France, the chemical symbol for gold, things it can answer off the top of its head. After training:

Question type	Tool-call rate	Accuracy
Factual (it already knew the answer)	62% → 0%	94% → 100%
Arithmetic (genuinely needs the calculator)	100% → 100%	100% → 100%

Optimal on both at once, with no accuracy given up. And I never labelled a single question with "should this be searched". The reward was one sentence: one point for a correct answer, minus 0.2 for every tool call. When to use the tool is something it worked out on its own.

Why did this one work? Because "when to look something up" is something the model already did right some of the time — it genuinely skipped the search on 38% of the factual questions. That's a policy, not a missing ability.

Put the four side by side and the shape appears:

Experiment	Target	Is the ability in the weights?	Result
Stories containing a dragon	0.4% baseline	No	0 — the curriculum dropped it
Mental 2-digit × 3-digit multiplication	A missing algorithm	No	+2.3 points
Hand it a calculator	(no training at all)	—	+59.4 points, instantly
When to look something up	Already right sometimes	Yes	62% → 0%, accuracy intact

So deciding whether to train or to bolt on comes down to one question: does it already get this right some of the time? If yes, train. If no, bolt it on.

Three times I nearly read the result backwards

This part is worth more than the conclusion.

Once. In that +2.3 RL run, I evaluated every 20 rounds and got:

43.8%  42.7%  39.6%  41.7%  49.0%  43.8%      baseline 42.2%

A flat line, plus one spurious 49.0%. What I wrote down at the time was: "it learned nothing."

Then I changed how I measured: the same 512 questions, the same weights, toggling only the adapter.

adapter off : 37.30%      19 wrong → right
adapter on  : 39.65%       7 right → wrong

The effect was real. It had been invisible because every evaluation drew a fresh set of questions, and the difficulty spread between question sets (±5%) completely drowned out the +2.3%.

To compare two models, you have to make them sit the same exam.

That lesson showed up three times in this project. The first was comparing perplexity between two models, where you have to pin the evaluation batch first. The third was comparing with-tools against without. Skip it, and the conclusion inverts every time.

Twice. In the "when to look things up" run, the first 20 rounds barely moved (62% → 56%). I said at the time: if it hasn't moved by round 30, kill it and retune.

Between rounds 20 and 40 it collapsed to 0%, and then stayed perfectly stable for the next 30.

RL is often plateau-then-jump: signal accumulates early, and once the policy crosses some threshold it converges fast to a new stable point. Declaring "it can't learn this" while you're on the flat part is not safe.

Three times. The training log said 72% arithmetic accuracy while the formal evaluation at the same moment said 100%. Training sampling deliberately runs at high temperature — without it, repeated samples of the same question come back identical, there's no variance within the group, and the gradient is exactly zero. The price is worse answers during training.

The number you see during training is performance under sampling noise, not a score.

Three crashes

More interesting than the successes. None of them was a bug; each was a mechanism behaving exactly as specified.

Reward hacking. I deliberately wrote a leaky reward: one point if the story is under 120 tokens, with not a word about the story still having to be a story. After 200 rounds the success rate was 100% and the average length had fallen from 169 to 30.

The exploit it found was better than the one I'd predicted — it didn't write short stories, it skipped the story entirely and wrote the ending:

prompt      : Once upon a time
continuation: , Timmy was very tired. He went home and slept well. The end.

"The last two sentences" both satisfy the length rule and sit in a text distribution it knows well, so that's where it converged.

A reward function encodes what you said, not what you wanted.

Noisy TV. I wanted to give the model curiosity, and my first instinct was to reward surprise: get it to write things it finds unexpected itself. 120 rounds later the intrinsic reward had climbed from 2.3 to 6.2 — from the reward function's point of view, an unqualified success.

What it actually wrote was:

d cubs rush enthusiasmHarry reots m streng than Jenny was

Gibberish. Noise is permanently the least predictable thing there is, so an agent chasing surprise will sit and stare at television static forever. This failure has a name in the literature. Surprise is not novelty — curiosity has to distinguish "I don't know this yet" from "this is unknowable in principle."

Incidentally, distinct-2, the diversity metric, hit a record high of 0.967 at the time. It had been saturated with gibberish. Any single metric can be gamed; what exposed this one was perplexity plus three other tasks all disagreeing at once.

A KL term pointing the wrong way. To stop the model drifting during RL, you add a constraint pulling it back toward the original. The most intuitive formulation is correct in expectation but wrong as a quantity you differentiate — it is signed and unbounded below, so gradient descent will simply drive probabilities toward zero forever.

A unit test caught it outright: with the reference model 2 nats more confident than the current policy, one update step widened the gap to 3.06. It walked the other way.

An accidental by-product

Quantisation — squeezing 16-bit weights down to 4 — gave no speedup at all on the small model. My explanation at the time was that reading weights is only 11% of a single step, and the rest is fixed overhead.

Then I redid the same thing on a 1.7B model:

Precision	Weight size	Generation speed	Time per token
bf16	3.8 GB	61.8 tok/s	16.2 ms
8-bit	1.7 GB	103.1 tok/s	9.7 ms
4-bit	934 MB	167.5 tok/s	6.0 ms

Fit those three points to time per token = fixed overhead + weight size / memory bandwidth, and the bandwidth solves to 281 GB/s.

The M4 Pro's rated memory bandwidth is 273 GB/s. Three percent off.

I measured this machine's memory bandwidth out of three generation speeds.

The same reasoning explains why KV cache is only 3.2× faster on the small model when theory says it should be more than ten, and why batched generation at 64 ways is only 8× faster. Where the bottleneck is decides everything.

The ceiling

What I set out to build was a system that learns by itself, goes and finds knowledge online, and accumulates experience.

The first half held up: knowledge and ability can be supplied at inference time; they don't have to be packed into the weights. A calculator, a search tool, a page fetcher — the same loop with different functions hanging off it. Ask the 29.9M model what the capital of France is and it answers "a sign of friendship and friendship"; put a 1.7B model behind a search tool and it can tell me the name of a model released after its training cut-off.

But the ceiling is fixed by two things, and both were confirmed by experiment.

First, the base model's ability. Self-training amplifies nothing that isn't there — the dragon it could never learn is the evidence.

Second, what the verifier can check. A model can never exceed what can be checked. If my reward function can only verify whether there's a cat in the story, then the model will evolve to 94% along the "has a cat" axis and stop there.

The self-improvement loop is real: generate → verify → keep the good ones → train → get stronger. But self-training with no verifier anchoring it collapses, and the two crashes above are two shapes of exactly that disease.

Nobody knows how to build AGI. But a system that keeps getting better at tasks you can verify — I've now run every single component of one by hand.

That's probably the biggest thing these two days gave me: a lot of conclusions only become yours once you've measured them yourself. You think you understood it when you read it — right up until you stare at a flat line, write down "it learned nothing", change how you measure, and find the effect was there the whole time.