How to Build a Large Language Model From Scratch

Everyone has heard that ChatGPT “predicts the next word.” But what does that actually mean? What is happening inside the model when it does that? And how does a system that starts knowing absolutely nothing end up generating coherent text?

The clearest way to answer those questions is not to describe a finished model. It is to build one, from the very first line of code, watching each piece come into existence and understanding why it has to be the way it is.

This is that build. It uses a small character-level language model trained on Grimm’s Fairy Tales: modest in scale, simple enough to understand completely, and built on the same principles that underpin every large language model in production today. GPT, Claude, Gemini, Qwen- all of them are this, scaled up.

The code is available in full on GitHub. Everything below explains what it does and why.

The Task: What the Model Has to Learn

Before writing a single line of code, you need to decide what the model is going to do. The answer is simple and the same for every language model ever built: given everything that came before, predict what comes next.

Take a line of text: “beyond the forest.” Cover the second character. The first character, B, is the input. The hidden character, E, is the answer. Move the boundary one step forward: now BE is the input and Y is the answer. Keep moving: BEY predicts O, BEYO predicts N, and so on. Do this for every character in every line of every text in the dataset, and you have thousands of examples of the same small task.

This sounds too simple to produce coherent paragraphs. The key is that the model never generates a whole paragraph at once. It generates one character, appends it to the input, generates the next character based on the updated input, and repeats. One small, consistent prediction chained over and over becomes a full passage.

That is the entire job. Everything else in this article is how the model learns to do it well.

Step One: Turning Text Into Numbers

Machine learning models cannot work with letters. They work with numbers. The first task is building a reversible translation between the two.

The translation unit is called a token. A token could be a whole word, part of a word, a byte, or a single character. For this build, one character equals one token. That keeps the translation visible and forces the model to learn words from their component pieces rather than receiving them pre-assembled.

To build the translation, you first collect every distinct character that appears in the training text: every letter (upper and lower case separately), every space, every punctuation mark. Sort that list,t and you have the vocabulary. Each character gets an ID number based on its position in the list.

Two dictionaries complete the setup. One maps characters to their IDs (encoding). One maps IDs back to characters (decoding). If encoding followed immediately by decoding changes anything about the original text, the translation has destroyed information before training has even started. Verify that the round trip is lossless.

Apply the encoder to the entire dataset, et and you have a long stream of integers stored in a PyTorch tensor. A tensor is a container for numbers with a defined shape. At this point, the numbers mean nothing to the model. You have built the entrance from language into mathematics. Nothing more.

Step Two: How the Model Starts Guessing

With the data as numbers, you need a structure that can take an input ID and produce a score for every possible next character.

The simplest structure that achieves this is a table with one row for every possible input character and one column for every possible output character. This build’s vocabulary has 70 characters. Hence, the table is 70 by 70. If capital B appears at position 7 in the vocabulary, row 7 of the table holds the model’s current preferences for what character should follow B.

The entries start as random numbers. The model has no preferences yet. It might rank a semicolon above E even though the obvious answer in “beyond” is E. That is not a bug. The structure can express a preference. It just has not been given any reason to prefer the right one.

These entries are the first parameters of the model. Each parameter is a single adjustable number, and training moves those numbers toward values that make better predictions.

When the model receives an input ID, it selects the corresponding row. The resulting numbers are called logits: raw scores that can be negative or greater than one, with no constraint on their range. The largest logit indicates the character the model currently favours next.

Step Three: Measuring How Wrong It Is

Before the model can improve, it needs to know how badly it is doing. That requires measuring the gap between what it predicted and what was actually correct.

The logits are converted to probabilities using a mathematical function called softmax. Softmax preserves the ranking of the logits but converts them into positive numbers that sum to exactly one, giving you a probability distribution across every possible next character.

If the model assigns the letter E a 2% probability when E is the correct answer, that is a bad prediction. To turn “bad” into a number the computer can work with, you take the negative logarithm of the probability assigned to the correct answer. When the model assigns high probability to the right answer, the logarithm is small, and the negative makes the result small. When it assigns near-zero probability to the right answer, the logarithm becomes a large negative number, and the negative makes the loss large.

This calculation, applied across all predictions and averaged, is called cross-entropy loss. Cross-entropy is important because it compares the correct answer against the full set of alternatives. The model should not just push E upward in isolation. It must make E more plausible relative to every other character it could have predicted. That is the constraint that makes the training signal meaningful.

Step Four: Learning From the Error

The loss tells you that the model was wrong. It does not tell you which parameters to change or by how much. That is what the backward pass does.

Imagine increasing one parameter by a tiny amount and checking whether the loss increases or decreases. If it decreases, moving that parameter in the same direction helps. If it increases, move it the other way. The rate at which the loss changes with respect to one parameter is that parameter’s gradient.

Calculating this experiment separately for every parameter would be impossibly slow in a model with hundreds of thousands of them. PyTorch solves this by recording every mathematical operation during the forward pass and then applying the chain rule through that recorded path to calculate all gradients simultaneously. This is called automatic differentiation, and it is the reason PyTorch exists.

Once every parameter has a gradient, the update is simple: subtract a small fraction of the gradient from each parameter. The learning rate controls the size of that fraction. Move too fast, and the model overshoots good values. Move too slowly and training takes forever.

Four steps, repeated thousands of times, constitute the entire training process:

1. Pass the input through the model and produce scores.
2. Compare the scores against the correct answers and calculate the loss.
3. Run the backward pass to calculate gradients.
4. Update every parameter using those gradients.

Run this cycle once and at least one parameter changes. Run it across millions of examples,s and the table gradually records which characters tend to follow one another throughout the text.

Step Five: The Bigram Model and Its Failure

The structure built so far, one row per input character, one column per possible output, is called a bigram language model. It looks only at the most recent character when making each prediction.

After brief training, it produces something. Common characters, spaces, and punctuation appear in roughly realistic proportions. It might produce fragments that look vaguely like words. But it fails in a specific and revealing way: it cannot use context.

The words “was” and “saw” contain the same characters in a different order. A bigram model treats the final character in both as the same input and predicts the same next word. It cannot tell you apart.

Every tool built so far- the tokenizer, the loss function, the batching logic, and the training loop- remains correct and will be reused. What must change is the part that produces the logits. The model needs to consider not just the final character but everything that came before it.

Step Six: Embeddings and Position

The first change is how the model represents each character internally.

A vocabulary ID like 7 for capital B is just an address. The number 7 has no relationship to 8 or 6. The model cannot reason about characters using arbitrary index numbers. What it needs is a rich internal representation: a list of adjustable numbers that can encode meaningful relationships.

An embedding table solves this. It has one row for every character in the vocabulary and one column for every feature in the model’s internal representation. For this build, each character gets a vector of 128 adjustable numbers. Looking up a character in the embedding table replaces its ID with those 128 values. The values start as random numbers, and their meaning emerges entirely from training: the model adjusts them because specific embeddings lead to lower loss.

Token embeddings tell the model which character occupies a position. But order matters. The characters in“was” and “saw” produce the same token embeddings in the same arrangement, but a model that scrambled them would produce identical predictions for both words at every position.

Positional embeddings address this. A second table has one row for every position in the context window, positions 0 through 127. Instead of indexing it by character ID, you index it by position number and add the resulting vector to the token embedding at the same location. The same character now begins from a different combined representation depending on where it appears in the sequence. The model can tell that position 3 is different from position 70.

After embeddings, every position in the input has a 128-dimensional vector that encodes both what character it is and where it sits in the sequence. But no position yet knows anything about any other position. Each vector contains only its own information.

Step Seven: Attention

This mechanism makes modern language models powerful.

A simple way to share information across positions is to average. Take the vector at the current position and all earlier positions and compute their mean. At the first character, the average contains only the first vector. At the fourth character, it contains the mix of the first four. At the last character, it contains the entire sequence blended.

This works in two important ways. Information travels across positions. And the triangular structure ensures no position ever receives information from the future, which is essential: the model cannot cheat by seeing what comes next when it is learning to predict what comes next.

It fails in one important way. Every earlier position contributes equally to the average. The model cannot make one character matter more than another. As the context grows, useful information gets diluted by everything that came before it, relevant or not.

Attention replaces the fixed average with a learned, weighted mixture. The weight for each earlier position isn’t set by hand. It is computed from the representations themselves.

Each position generates three things from its embedding vector using three separate learned transformations:

A query: what this position is looking for.

A key: what this position can offer to others.

A value: the information this position can contribute if selected.

To compute the attention weights for one position, you compare its query against the keys of every permitted earlier position using a dot product. A dot product measures how aligned two vectors are: a high value means a strong match, and a low value means a weak match. The resulting scores are divided by the square root of the vector dimension to keep their scale stable as the representations get wider.

Those raw scores are converted to weights using softmax, after the scores for future positions are set to negative infinity so they receive weights of zero. The model then computes a weighted sum of the value vectors, where each value is weighted by how strongly its key matched the current query.

The result: each position gets a new vector that is a learned combination of information from the positions it was allowed to look at. The combination differs for every position and every sequence because it depends on the current representations, which training adjusts.

This is self-attention. “Self” because the queries, keys, and values all came from the same sequence. “Causal” because future positions are masked out.

Step Eight: Multi-Head Attention and Feed-Forward Processing

One attention head computes one kind of relevance. Multiple kinds of relevance may be useful simultaneously: one head might track which earlier character had the same grammatical role, another might track the nearest punctuation boundary, another might track repeated words. No one programs these roles. They emerge from what reduces the loss.

Multi-head attention runs several independent attention heads in parallel. Each head in this build works with 32 features (128 divided by 4 heads). Their outputs are concatenated back to 128 features and passed through one final learned projection that allows information collected by separate heads to mix.

Attention moves information across positions. Within each position, though, the weighted average is linear: it cannot represent certain kinds of relationships. A feed-forward network adds non-linear processing to each position independently. It expands the 128-feature vector to 512, applies a ReLU activation (which sets negative values to zero), and projects back to 128. The non-linear activation is essential. Without it, multiple linear layers would mathematically collapse into one, providing no additional representational capacity.

Step Nine: Residual Connections and Layer Normalisation

As the network grows deeper, two practical problems appear.

When each new component replaces its input, it can lose useful information. Residual connections solve this by adding each component’s output to its input rather than replacing it. Attention computes an update. It adds the update to the input representation. The result preserves what was already there while incorporating new information.

Repeated transformations can push vectors to unstable scales, making training unreliable. Layer normalisation addresses this by normalising features within each position, then applying a learned scale and shift. This keeps the values in a predictable range throughout the entire network.

One block, consisting of multi-head attention with a residual connection, layer normalisation, and a feed-forward network with another residual connection and another layer normalisation, can now gather contextual information, process it non-linearly, and preserve what each position already knew. The model stacks four such blocks.

Step Ten: The Final Projection and the Complete Forward Pass

After four blocks of attention and processing, each position has a rich 128-dimensional vector that every permitted earlier position has influenced. The model’s task is to predict the next character, so it must convert that 128-dimensional representation back into a score for every character in the vocabulary.

A final linear layer maps the 128 internal features to 70 logits, one per character. Those logits go into cross-entropy, which compares them against the correct next characters and produces the training loss. The complete model, in one forward pass, does this:

The input IDs become token embeddings and positional embeddings that are added together. The combined vectors pass through four transformer blocks, each refining every position’s representation using context from earlier positions. A final linear projection converts the resulting vectors into vocabulary-sized logit distributions. Cross-entropy measures the loss, and the backward pass distributes gradients to every parameter.

This model, built to these specifications with a character vocabulary of 70 and a context window of 128, contains approximately 830,000 parameters.

Step Eleven: Generation

Training uses known targets. Generation does not. The model must produce its own inputs.

Start with a prompt. Encode it. Pass it through the model and take the logits at the final position. Convert those logits to probabilities with softmax. Sample one character ID from that distribution. Append the sampled character to the sequence. Repeat.

Each newly sampled character becomes part of the context for the next prediction. A plausible first choice leads into a stable second. A less likely choice pulls probabilities in a different direction. The text’s entire trajectory depends on each decision, compounding at every step.

Temperature controls how conservatively the model samples. Dividing the logits by a small number (temperature below 1) sharpens the distribution, making the highest-probability characters increasingly dominant. Dividing by a larger number (temperature above 1) flattens the distribution, giving lower-probability characters more opportunity. Temperature does not add information the model never learned. It changes how cautiously the model picks from what it knows.

If the generated sequence exceeds the context window of 128 characters, only the most recent 128 are fed into the model for each prediction. Older context is dropped.

What This Reveals About Every Language Model

This small character-level model trained on fairy tales is not just a learning exercise. It is an X-ray of every large language model ever built.

GPT-4, Claude, Gemini, and Qwen all do exactly what this model does, scaled. They tokenize text. They embed tokens. They apply positional encoding. They run layers of multi-head causal self-attention with residual connections and layer normalization. They project to vocabulary logits. They train on next-token prediction using cross-entropy loss with gradient descent.

The differences are quantitative, not qualitative. A production language model has billions of parameters, not 830,000. It trains on trillions of tokens, not a single book of fairy tales. It uses a subword tokenizer instead of character-by-character. It applies sophisticated optimization techniques and distributed training across thousands of GPUs.

The architecture is the same.

What makes this worth understanding is not that it lets you build GPT-4 on your laptop. You cannot. It makes the behaviour of language models comprehensible. When a model generates text one token at a time, you now know why. When it seems to “remember” something from earlier in the conversation, you know that earlier context is literally part of its input at each step, shaping every prediction through the attention mechanism. When it generates something plausible but wrong, you know it is doing exactly what it was trained to do: produce the most statistically likely continuation, not the true one.

The complete code and runnable notebook are available on GitHub. You can train it on any text and watch the same progression: random characters first, then realistic spacing, then recognizable letter combinations, then fragments of words, then something that begins to look, imperfectly but unmistakably, like language.