How Does a Transformer Work in AI? Attention, Tokens, and the Cost of Remembering Everything

How does a transformer work in AI? It turns words into numbers, lets each word weigh the words around it, and predicts the next word from that weighing. Judge the design by one rule: every word can look at every other word, and that single choice sets both its power and its cost. A decoder looks backward only. An encoder reads both ways.

The name is shared with the electrical device that steps voltage up and down on power lines. The AI architecture comes from a 2017 research paper by a team at Google. The power device does not appear again.

The four-stage map: from typed words to the next word

A transformer reads a prompt and writes one word at a time. It never plans a whole sentence, then writes it out. It picks the most likely next word, adds it to the prompt, and picks again. That loop is the whole generation process, and every part of the architecture serves it.

1. Tokenize: split the text into pieces called tokens.
2. Embed: turn each token into a list of numbers, and add a second list that marks its position.
3. Attend and transform: run the vectors through blocks of self-attention and a feed-forward network.
4. Predict: turn the final vectors into probabilities over the vocabulary, and pick the next token.

The loop repeats. Each new token joins the input, and the machine runs the four stages again. A 200-word answer needs about 200 passes, one per token.

A transformer is one kind of neural network, built for sequences. Its blocks are the same kind of layered math, and its training is the same method: show examples, measure the error, adjust the weights.

ChatGPT, Claude, and Gemini all run on this architecture. The names on the box differ. The four stages do not.

Why the old sequence models ran out of road?

Before 2017, sequence models read one word at a time. A recurrent neural network carried a memory from word to word, updating it at each step. Long sentences strained that design in two ways.

First, the path was long. The last word of a 50-word sentence sat 50 steps from the first, and each step could blur the signal. Gradients faded as they traveled backward through those steps, a problem called the vanishing gradient. Early words lost their voice by the time the model reached the end.

Second, the steps could not run in parallel. Word 30 had to wait for word 29, and word 29 for word 28. Training on long text took a long time because the hardware sat idle between steps.

The paper that fixed both problems arrived in June 2017: Attention Is All You Need, written by eight researchers at Google. It dropped recurrence entirely and let every word look at every other word in a single step. The paper has been cited more than 250,000 times. Its architecture now sits inside the chatbots, search engines, and image tools people use every day.

Step one: words become tokens

A transformer never sees a word. It sees tokens, the pieces a tokenizer cuts the text into. Common words survive whole. Rare words split into parts, and strange strings split into many small pieces.

Why split words into pieces at all? A fixed vocabulary cannot hold every word in every language, so the tokenizer builds words from parts it knows. Splitting keeps the vocabulary small while letting the model read any text, including typos, names, and code.

A measurement shows the splits in practice. The tokenizer was tiktoken 0.14.0 with the GPT-4 encoding, the encoding GPT-4 and ChatGPT use. The sentence ‘The cat sat on the mat.’ became 7 tokens. The pronoun sentence ‘The animal didn’t cross the street because it was too tired.’ became 13 tokens.

Text	Tokens	What the split shows
The cat sat on the mat.	7	6 words, 1.17 tokens per word
The animal didn’t cross the street because it was too tired.	13	11 words, 1.18 tokens per word
unhappiness	3	un / h / appiness
transformer	2	transform / er
https://valleyai.net/ai/how-does-a-neural-network-work/	17	one web address, 17 tokens
2017 paper abstract, 55 words	65	1.18 tokens per word

Two patterns show in the numbers. Short common words cost one token each, so plain sentences run near one token per word. Unusual words and web addresses cost far more: a single URL took 17 tokens, and a line of code took 19.

The abstract of the 2017 paper ran 65 tokens in the GPT-4 encoding. The older GPT-2 encoding needed 72 for the same words. Token counts are not fixed properties of the text. They belong to the tokenizer.

Tiktoken 0.14.0 ran the measurement on 11 November 2026. A different model’s tokenizer splits the same text differently, but the pattern holds: common words cost one token, rare words cost several.

Step two: words become vectors

Each token becomes a list of numbers called an embedding. The list holds hundreds of numbers in a small model and thousands in a large one, and the numbers place the token in a space where words with related meanings sit near each other. The word for a king sits near the word for a queen, and far from the word for a bicycle.

The numbers start as learned values. Training moves them until words that appear in similar contexts land near each other. The model never receives a dictionary definition. It learns the position from use alone.

A token needs its position as well. Attention reads all tokens at once, so without help it cannot tell the first word from the fifth. Positional encoding adds a second list of numbers to each embedding, a pattern that shifts with the token’s place in the sentence. The 2017 model used sine and cosine waves for that pattern. Newer models use rotary embeddings, which rotate the vectors instead.

Size sets the scale. The list is short in a small model and long in a large one, and the length is fixed for a given model. Every token in the same model carries a list of the same length, so the arithmetic downstream always lines up.

Step three: attention, worked with numbers

Attention is the step that gives the architecture its name. Each token asks the other tokens a question, scores their answers, and blends their content by those scores. The token for ‘it’ in the pronoun sentence needs to know which noun it stands for, and attention is how it finds out.

Three vectors run the step. The query states what a token is looking for. The key states what a token offers. The value holds the token’s content. The score between two tokens is the dot product of the query of one and the key of the other. High scores mean a strong match. The formula reads: attention equals softmax(QK^T / sqrt(dk)) V.

The three vectors work like a library search. The query is the question written on a slip. The key is the label on each book. The value is the book itself. The librarian ranks the labels against the question and hands over the books that match best.

Why divide by the square root of the key width? Long vectors produce large dot products, and large scores make the softmax step push nearly all the weight onto one token. Dividing by the square root keeps the scores in a range where several tokens can still matter.

A worked calculation shows the arithmetic. The vectors are small and chosen for the example, not measured from a real model. The query for ‘it’ is [1, 0, 1, 0]. The key for ‘animal’ is [1, 0, 1, 0] and the key for ‘street’ is [0, 1, 0, 1]. The key width is 4, so the square root is 2.

Step	‘animal’	‘street’
Dot product with the query	2.0	0.0
Divided by the square root of 4	1.0	0.0
Weight after softmax	73.1%	26.9%
Weight without the division	88.1%	11.9%

The division matters. Without it, ‘animal’ takes 88.1% of the weight. With it, the split is 73.1% against 26.9%, and ‘street’ keeps a voice. The blended output is [1.462, 0.807], mostly the value of ‘animal’ with a share of ‘street’.

Real models run this step with longer vectors, and the weights come from training. In GPT-3, each head works with vectors 128 numbers long. The arithmetic is the same.

Measured attention in a real model is messier. A team that ran BERT on a laptop and read its real attention weights found only 22 of its 144 heads shifted the pronoun the right way, and the team said so in its own write-up. CloudEngineerLab published those numbers. Attention charts show a window into the model. They leave most of the explanation out of view.

Attention scores similarity. Similarity is not truth, which is how an AI hallucination starts: a confident answer built from a plausible match.

One head tracks one kind of relation. A head might link pronouns to nouns, while another tracks verbs to their subjects. The 2017 model ran several heads in parallel and joined their outputs, so the block could watch many relations at once.

The block, repeated

Attention is half a block. The other half is a feed-forward network, a small network that processes each token’s vector on its own. Attention mixes information between tokens. The feed-forward network reshapes what each token holds.

Two additions hold the block together. A residual connection adds the block’s input to its output, so the signal has a direct path through. Layer normalization rescales the numbers after each step, keeping them in a range the next layer can use. Without those two, a deep stack would lose the signal or blow it up.

The 2017 model stacked six blocks in the encoder and six in the decoder. The numbers below are the paper’s own.

	Base model	Big model
Model width	512	1,024
Layers, encoder and decoder	6 and 6	6 and 6
Attention heads	8	16
Feed-forward width	2,048	4,096
Parameters	65 million	213 million

Training took 3.5 days on eight P100 graphics cards. The base model scored 28.4 BLEU on English-to-German translation and 41.8 on English-to-French, beating the best published results of the time on the WMT 2014 test set.

Three families, three jobs

The same block builds three kinds of model. The difference is which direction attention is allowed to look. That one choice decides what the model is good for.

Family	How attention looks	Best for	Examples
Encoder-only	Both directions	Understanding text	BERT
Decoder-only	Earlier words only	Generating text	GPT
Encoder-decoder	Both, then one direction	Turning one text into another	Translation

An encoder reads the whole sentence at once, so it can use the words after a gap to fill the gap. BERT learned by guessing masked words, and that training made it strong at classifying text, answering questions, and ranking search results. Google added BERT to Search in 2019, which is why a search engine can now match what a query means. Keywords still count, but meaning leads.

A decoder reads one direction only. It predicts the next word from the words before it, and the causal mask blocks the words ahead. That limit is a feature for generation: the model can write one word, feed it back, and write the next. GPT models are decoder-only. A large language model behind a chatbot shares that shape.

An encoder-decoder pairs both. The encoder reads the source text in both directions, and the decoder writes the target one word at a time while attending to the encoder’s output. Translation was the 2017 paper’s own task, and the same shape now serves summarization.

What attention costs?

Attention compares every token with every other token. The count of pairs grows with the square of the length, and the square grows fast.

Context length	Attention pairs
1,024 tokens	about 1.0 million
8,192 tokens	about 67.1 million
32,768 tokens	about 1.1 billion
131,072 tokens	about 17.2 billion
1,048,576 tokens	about 1.1 trillion

Why does generating text take longer than training on it? Training reads the whole prompt at once, so all tokens move through the network in parallel. Generation writes one token at a time, and each new token needs a full pass through the network before the next one can start.

Generation stores memory as well. Each new token adds its keys and values to a cache, and the cache grows with the context. The size per token comes from the model shape: two copies, key and value, times the layers, times the key-value heads, times the head width, times two bytes.

Model	Cache per token	Cache at full context
Llama-3.2-1B	64 KiB	8 GiB at 128K tokens
Llama-3-70B	320 KiB	65.5 GB at 200K tokens
GPT-3, 175B parameters	4.5 MiB	9.2 GiB at 2,048 tokens

Two ceilings split the cost. Training pays the pair count once per batch, in parallel. Generation pays it again at every step, one token at a time, and carries the growing cache. A long answer costs more than a long prompt of the same length, because the answer pays per token while the prompt pays once.

A phone cannot hold a 65.5 GB cache, so small models on phones run on a dedicated NPU with a short context.

What changed since 2017?

The block kept its shape. The parts around it changed. Grouped-query attention shares one key-value pair across several query heads, cutting the cache. Mixture of experts routes each token to a few specialist networks. Each specialist handles one kind of input, and the rest stay quiet. FlashAttention reorders the math so the pairs never all sit in memory at once. Rotary embeddings rotate the vectors to mark position.

Scale changed most. Frontier models now hold hundreds of billions of parameters, train for months on thousands of graphics cards, and read contexts of a million tokens. The mechanism is the same. The numbers around it are not.

The same blocks now read images and sound. A vision transformer cuts a picture into 16 by 16 patches and treats each patch like a word, which is how image generation models read a picture. Whisper, the speech model, trained on 680,000 hours of audio and uses the same attention step to turn sound into text.

Training follows how machines learn from examples: show text, measure the error, adjust the weights. The architecture changed how the model reads. The training loop did not change.

Try the measurement yourself. Paste a sentence into OpenAI’s public tokenizer and read the split: count the tokens, find the word that split into three pieces, and compare the count with the same sentence in a different encoding. Then open the illustrated transformer walkthrough and watch one attention head link a pronoun to its noun. The numbers on the screen are the architecture at work.

Frequently asked questions