Transformers Explained Visually: The Architecture Behind Modern AI

ChatGPT, Claude, Gemini, Copilot, Google Translate, the AI that writes captions for videos, even the AI that predicts the shape of proteins. They look like completely different tools, but almost all of them are built on the same invention: the transformer.

It's even in the name. ChatGPT stands for Chat Generative Pre-trained Transformer.

This post explains what a transformer is, using pictures first and words second. There's no maths you need to follow. Where a formula helps, it sits in an optional box you can skip.

The Problem Transformers Solved

To understand why transformers were such a big deal, you need to know how AI read text before them.

Until 2017, the best language AIs were recurrent neural networks (RNNs). They read a sentence the way you'd read through a keyhole: one word at a time, from left to right, carrying a small "memory" from each word to the next.

That approach had two big problems:

1. It was slow to train. Word 6 can't be processed until word 5 is done, which can't start until word 4 is done. Computers with thousands of parallel processors (GPUs) mostly sat waiting.
2. It forgot things. Information from early words faded as it was passed along the chain, a bit like a game of telephone. In a long paragraph, the start was often lost by the end.

In 2017, a team of eight researchers at Google published a paper with a bold title: Attention Is All You Need. Their idea: throw away the chain completely. Let every word look directly at every other word, all at the same time. They called the design the transformer.

The Big Idea in One Sentence

A transformer lets every word in a sentence look at every other word, decide which ones matter, and update its own meaning based on them, all in parallel.

That "decide which ones matter" part is called attention. Everything else in a transformer exists to support it.

Step 1: Words Become Numbers (Embeddings)

Computers only work with numbers, so the first step is to turn each word (or piece of a word, called a token) into a list of numbers called an embedding.

You can picture an embedding as a location on a giant map of meaning. Words with similar meanings are placed close together: "cat" near "dog", "server" near "computer", "restart" near "reboot". Nobody draws this map by hand. The AI learns it by reading huge amounts of text. The previous post measured some of those distances in a real model.

In the original 2017 transformer, each word's list had 512 numbers. In BERT it's 768. In today's largest models it's many thousands.

Adding position: who came first?

Here's a catch. Because a transformer looks at all words at once rather than in a chain, it has no natural sense of order. Without help, "dog bites man" and "man bites dog" would look identical.

The fix is to add a second signal to every word: a kind of seat number that tells the model where the word sits in the sentence. This is called positional encoding. The original transformer created those seat numbers from smooth wave patterns (that's what the wavy lines on the cover image represent). Many modern models use a newer method that "rotates" the numbers slightly depending on position, but the purpose is the same.

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

Step 2: Attention, the Heart of the Transformer

Read this sentence:

The animal didn't cross the street because it was too tired.

What does "it" mean? The animal, obviously. Now change one word:

The animal didn't cross the street because it was too wide.

Now "it" means the street. You worked that out instantly by looking at the other words for clues. That is exactly what attention does.

We ran both sentences through BERT and looked at where the word "it" was focusing. Here's one of its attention units (called a head) in action:

With "tired", "it" puts 28% of its focus on "animal" and only 7% on "street". With "wide", that flips completely: 1% on "animal" and 59% on "street". Nobody programmed a rule about tired animals or wide streets. The model learned it from reading.

Step 3: Queries, Keys and Values

So how does a word decide where to focus? This is where three terms come in that you'll see in every explanation of transformers: query, key and value.

The easiest way to understand them is a library:

• Query: what you're looking for. You walk in and ask, "I need something about animals."
• Key: the label on each book's spine. You compare your question with every label.
• Value: what's actually inside the book. Once you know which books match best, you read mostly from those.

In a transformer, every word plays all three roles at once. From its own list of numbers, each word creates:

• a query: "what kind of information am I looking for?"
• a key: "here's what I can offer to others"
• a value: "here's the information I'll share if you pick me"

Then, for each word:

1. Ask. Take its query.
2. Match. Compare that query with the key of every word. A good match gives a high score.
3. Weigh. Turn the scores into percentages that add up to 100%.
4. Blend. Mix together the values of all the words, using those percentages. That blend becomes the word's new, context-aware meaning.

After this step, "it" is no longer just the generic word "it". It's now closer to "it, meaning the animal". That's the whole trick.

Attention(Q, K, V) = softmax( Q · Kᵀ / √d ) · V

Step 4: Many Heads, Many Points of View

One round of attention can only focus on one kind of relationship at a time. So a transformer runs several attention heads side by side, each with its own query, key and value settings. This is called multi-head attention.

Think of a team reading the same document, each with a different highlighter: one marks grammar, one marks who-did-what, one marks which nouns go together. Afterwards, their notes are combined.

The original transformer used 8 heads per layer. BERT uses 12. Very large models use 64 to 128 or more. Here are four real heads from BERT reading a short sentence, each doing a visibly different job:

• Layer 4, head 6 mostly looks at the word before.
• Layer 3, head 10 almost always looks at the word after.
• Layer 2, head 4 links "cat" and "mat" across the sentence ("mat" puts 34% of its focus on "cat", and "cat" puts 41% on "mat"), even though four words sit between them.
• Layer 12, head 8 sends nearly everything to [SEP], a special marker at the end of the input. This looks useless, and in a sense it is: heads with nothing useful to do learn to "park" their attention on a marker like this.

Step 5: The Rest of the Building Block

Attention is the star, but a transformer layer (also called a block) has a few more parts. Here's the full design, the same one used in the GPT family of models behind most chatbots:

Inside each block:

1. Multi-head self-attention. Words share clues with each other (Steps 2 to 4).
2. Add & normalise. The result is added to what the word already knew, rather than replacing it, and the numbers are rescaled to stay in a stable range.
3. Feed-forward network. Each word now "thinks" on its own for a moment, processing what it has just gathered. In the original transformer this step briefly expands each word's 512 numbers to 2,048 and back. Much of a model's general knowledge is believed to live in these parts.
4. Add & normalise again.

Those "add" steps are shortcuts (the purple lines in the diagram). They mean each block only has to add a little extra understanding on top of what's already there, like adding notes to a notepad instead of rewriting it from scratch. Without them, very deep stacks of blocks would be almost impossible to train.

Then the block is repeated. The original transformer stacked 6 of them. BERT has 12. GPT-3 has 96. With each layer, the words' meanings get richer: early layers tend to pick up grammar and nearby words, later layers pick up meaning and relationships across the whole text.

At the very top, the model scores every possible next word and turns the scores into percentages, exactly as traced in the previous post.

Three Kinds of Transformer

The original 2017 transformer had two halves, because it was built to translate between languages:

• an encoder, which reads the whole input sentence (for example, in English) and builds an understanding of it;
• a decoder, which writes the output (for example, in German) one word at a time, looking back both at what it has written so far and at the encoder's understanding.

Later models kept one half, the other, or both:

That table explains something about our "tired" and "wide" example. We used BERT because it reads in both directions, so "it" can see the word "tired" that comes after it. A decoder-only chatbot model can't: when it reaches "it", the word "tired" hasn't happened yet. Instead, it sorts out the meaning a moment later, when "tired" arrives and looks back at "animal".

Why Transformers Changed Everything

Transformers didn't just do translation a bit better. They changed what was possible.

1. They train fast on modern hardware. Because every word is processed at the same time, transformers make full use of GPUs, which are built to do thousands of calculations in parallel. The original base model reached top translation quality after about 12 hours on 8 GPUs, a fraction of the time older approaches needed.

2. They connect distant words directly. "Cat" and "mat" are one step apart, no matter how much text sits between them. No more game of telephone.

3. They get better when you make them bigger. This turned out to be the most important property of all. More data, more layers and more computing power kept producing better models, year after year. That's how we went from 65 million settings in the 2017 base model, to 1.5 billion in GPT-2 (2019), to 175 billion in GPT-3 (2020), and on to today's chat assistants.

4. They work on more than words. Anything that can be cut into pieces can go through a transformer. Images are cut into small squares (Vision Transformers, 2020). Audio is cut into short slices (speech recognition tools such as OpenAI's Whisper). Google DeepMind's AlphaFold 2, which predicts the 3D shapes of proteins, relies heavily on attention too.

A Few Common Misunderstandings

The Original 2017 Transformer at a Glance

For readers curious about the numbers on the cover image:

The Short Version

Before 2017, AI read text through a keyhole, one word at a time. The transformer opened the door: every word can see every other word, all at once, and decide for itself what matters. Nearly every AI tool you use today grew out of that one idea.

Curious what happens around the model when you actually press Enter? Follow the journey of a prompt from browser to GPU. Want to go further? See how this engine is used in practice in the new AI application stack, or how embeddings power document search in RAG From Scratch.