From Text to Tokens to Answers: What Actually Happens Inside an LLM?

When you type a question into ChatGPT, Claude or Copilot, an answer appears a second later. It feels like the AI read your message, thought about it, and wrote a reply.

That's not quite what happens. What actually happens is stranger, simpler, and much more interesting. In this post we'll open up the "black box" and follow one sentence through a real AI model, step by step. You don't need any maths or programming background.

Here's the sentence we'll follow:

The engineer restarted the server because it

It's unfinished on purpose. We'll give it to the AI and watch exactly how it decides what comes next.

The Whole Journey in Five Steps

And then the most surprising part: it starts all over again for the next word. An AI doesn't write a whole answer at once. It writes one small piece at a time, and every piece is a new trip through these five steps.

Let's walk through them.

Step 1: Tokens, Cutting Text into Pieces

A computer can't read words the way you do. So the first thing the AI does is chop your text into small pieces called tokens.

A token is often a whole word, but not always. Sometimes it's part of a word, and sometimes it's just a few letters. Here's how GPT-2 cut up our sentence:

That's 8 tokens. Notice that "restarted" became two pieces: restart + ed.

Why? Each AI has a fixed "dictionary" of pieces it knows. GPT-2's dictionary has about 50,000 pieces. It has "restart" but not "restarted", so it builds the word from two pieces it does know, a bit like building a word out of Lego bricks.

Newer AIs have bigger dictionaries. The one used by GPT-4o has about 200,000 pieces, and it keeps "restarted" as one piece, so the same sentence is only 7 tokens.

Here are a few more examples of how words get split:

Step 2: Numbers, Giving Each Piece a Meaning

Next, each token is swapped for a number from the AI's dictionary. "server" is number 4382. "it" is number 340. These numbers are just positions in a list, like page numbers in a phone book. They don't mean anything yet.

Then comes the clever part. Each token is turned into a long list of numbers, called an embedding. In GPT-2, every token gets a list of 768 numbers. The start of the list for "server" looks like this:

0.09, 0.03, 0.11, -0.07, -0.09, -0.22, ... and 762 more

Nobody typed those numbers in by hand. The AI learned them during training, by reading huge amounts of text. And here's the remarkable result: words with similar meanings end up with similar lists of numbers.

Think of it like a giant map. Every word has a location on the map, and words that mean similar things are placed near each other. We measured how close some words are on GPT-2's map, on a scale where 1 means "pointing the same way" and 0 means "unrelated":

Nobody told the AI that "restart" and "reboot" are related. It worked that out by itself, just from seeing how the words are used. This same idea of "meaning as location" powers AI search tools, such as assistants that answer questions from your own documents.

The AI also adds a little extra information to each token about where it sits in the sentence: first word, second word, and so on. Without that, "the engineer restarted the server" and "the server restarted the engineer" would look identical to it.

Step 3: Attention, Words Looking Back for Clues

Read our sentence again: "The engineer restarted the server because it..."

You instantly know that "it" means the server, not the engineer. How does an AI work that out? With a mechanism called attention. It's the big idea behind modern AI, and it was introduced in a famous 2017 research paper called Attention Is All You Need.

Here's a simple way to picture it. Imagine every word in the sentence is a person in a meeting. When it's "it"'s turn, it looks around the room and asks: "Which of you can help me understand what I mean?" Some words are very helpful, some aren't. "it" then listens mostly to the helpful ones and updates its own understanding.

Two important rules:

• Words can only look backwards. "it" can look at "server" and "engineer", but it can never peek at words that come later. That's what lets the AI write from left to right.
• There are many "listeners" at once. GPT-2 has 144 separate attention units (12 in each of its 12 layers). Each one learns to look for a different kind of clue: grammar, who-did-what, which noun a word refers to, and so on.

What the AI actually focused on

We looked inside the model to see where "it" was paying attention. One of its attention units, deep in the middle of the model, did exactly what you'd hope:

64% of its focus went to "server", and just 2% to "engineer". The AI worked out that the thing that might need restarting is the server, not the person. Nobody programmed that rule in. The model learned it by reading.

Step 4: Layers, Meaning Builds Up Step by Step

Attention is just one part of the process. Each layer of the model has two parts:

1. Attention, where words gather clues from each other (Step 3).
2. A thinking step, where each word's numbers are processed on their own. This is where a lot of the model's general knowledge is stored.

GPT-2 stacks 12 of these layers on top of each other. The biggest modern models stack around 100.

A good way to picture this is a notepad. Each word carries its own notepad. At every layer, the AI reads the notepad and adds a few more notes. At the start, the notepad for "it" just says "the word it". By the top layer, it says something more like: "refers to the server, comes after 'because', a reason is about to be given, probably about the server's condition."

Watching the guess form

We can actually stop the AI after each layer and ask, "If you had to guess the next word right now, what would it be?" Here's what GPT-2 said:

At the start, the guesses are rubbish. By layer 3, the AI has the grammar: after "because it", a word like "was" usually comes next. In the middle layers, "failed" starts to climb, which makes sense in a sentence about restarting a server, and it even takes the lead for a moment. By the end, the AI settles on "was". You're literally watching understanding build up, layer by layer.

Step 5: The Next Word, Scoring Every Option

After the last layer, the AI does one final thing. It takes the notepad for the last word ("it") and compares it against every single piece in its dictionary, all 50,000 of them, giving each one a score.

Those scores are turned into percentages. Here are GPT-2's top ten choices for the word after "The engineer restarted the server because it":

There's one clear favourite, followed by lots of reasonable alternatives. Even the top ten together only add up to about 70%. The rest is spread across thousands of other possible words.

This is the single most important thing to understand about AI chatbots:

Choosing the Word: Safe or Creative

The AI gives a list of chances. Something still has to pick one. How that pick is made changes the AI's personality a lot.

Always pick the favourite. If the AI always picks the top word, it's predictable. GPT-2 then finishes our sentence like this:

The engineer restarted the server because it was running out of memory.

Add some randomness. Most chat tools let the AI sometimes pick a less likely word, which makes answers more varied and natural. A setting called temperature controls how adventurous it is:

We ran our sentence three times with a little randomness switched on. Same AI, same sentence, three different endings:

• "...because it was not running the latest version of OpenSSL."
• "...because it was running an outdated version of Linux."
• "...because it was going to crash."

That's why you can ask ChatGPT the same question twice and get two different answers. It's not changing its mind. It's rolling slightly different dice each time.

Then It Does It All Again

Once the AI picks "was", it adds that word to the sentence and runs the whole process again, now on 9 tokens, to choose the next word ("running"). Then again on 10 tokens to choose "out". Then "of", then "memory", then the full stop.

That's why AI answers appear word by word on your screen. You're watching the loop happen live. A long answer is hundreds of trips through the five steps.

To save time, AI systems remember the work they've already done for earlier words, so each new word only needs a small amount of fresh work. But the first word of an answer always takes a moment, because the AI has to read your entire message first. That's why a long question can mean a short pause before the reply starts.

What About ChatGPT, Claude and Copilot?

When you use a chat app, your message isn't sent on its own. The app quietly adds:

• Hidden instructions from the company that built it (for example, "You are a helpful assistant").
• The earlier messages in your conversation.
• Any documents or web results it looked up for you.

All of that is glued together into one long list of tokens and pushed through the same five steps. That explains a few things you may have noticed:

• Long conversations eventually "forget" the start. Every model has a limit on how many tokens it can handle at once.
• AI can be tricked by hidden text inside a document or web page, because instructions and content arrive as the same kind of tokens. This is called prompt injection.
• AI assistants that take actions (book things, run tools) are just this same loop running again and again, as explained in what happens inside an AI agent.

Small Model vs Big Model

That last row matters. GPT-2 can only continue text. Modern chatbots get extra training on conversations and feedback, which teaches them to follow instructions, answer helpfully and refuse harmful requests. But under all that polish, the engine is the same: read the tokens, predict the next one, repeat.

The Short Version

An AI doesn't see words, doesn't look up facts, and doesn't write an answer in one go. It turns your text into numbers, lets those numbers share clues with each other, scores every possible next word, picks one, and does it all again. Everything you see in a chatbot is built on that simple loop.

Want a closer, picture-led look at the design behind Steps 3 and 4? Read Transformers Explained Visually. New to AI? Start with What is Artificial Intelligence?. Curious how real AI products are built around this engine? Read the new AI application stack.

For the Curious: Try It Yourself

This part is optional and for readers who know a little Python. These are the exact snippets used to produce the results above.

See how text is cut into tokens (pip install tiktoken):

import tiktoken
 
text = "The engineer restarted the server because it"
for name in ["gpt2", "o200k_base"]:
    enc = tiktoken.get_encoding(name)
    ids = enc.encode(text)
    print(name, ids)
    print([enc.decode([i]) for i in ids])

gpt2 [464, 11949, 15765, 276, 262, 4382, 780, 340]
['The', ' engineer', ' restart', 'ed', ' the', ' server', ' because', ' it']
o200k_base [976, 36894, 170882, 290, 6017, 2236, 480]
['The', ' engineer', ' restarted', ' the', ' server', ' because', ' it']

See the next-word chances and the word-by-word loop (pip install torch transformers):

import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
 
tok = GPT2TokenizerFast.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2").eval()
 
ids = tok("The engineer restarted the server because it", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits[0, -1]   # scores for the NEXT token only
probs = logits.softmax(-1)
for p, i in zip(*probs.topk(5)):
    print(f"{tok.decode(int(i))!r:10} {p.item():.1%}")
 
# The generation loop: predict, pick the top token, append, repeat
for _ in range(6):
    with torch.no_grad():
        next_id = model(ids).logits[0, -1].argmax()
    ids = torch.cat([ids, next_id.view(1, 1)], dim=1)
print(tok.decode(ids[0]))

' was'     42.9%
' had'     10.5%
' wasn'    2.9%
' didn'    2.9%
' failed'  2.1%
The engineer restarted the server because it was running out of memory.