One 2017 paper replaced recurrence with attention
In June 2017 eight researchers, most of them at Google published a paper titled "Attention Is All You Need". It proposed a network for translating sentences that dropped the recurrent layers everyone used and kept only one operation, attention, stacked many times. The base model had 65 million parameters and beat the previous best English-to-German translation systems while training in a fraction of the time.
The idea was the title. Earlier language networks read a sentence one token at a time, carrying a running summary forward, so the twentieth word only knew the first through whatever survived nineteen steps of compression. A transformer reads the whole sequence at once. Every token asks every other token "how relevant are you to me", weights them, and builds its meaning from the answer. The word "bank" attends to "river" or to "account" and comes out different each time.
Nine years later the same design, scaled up, is what answers when you type into ChatGPT. GPT stands for generative pretrained transformer. Claude, Gemini, Llama, DeepSeek and Qwen are transformers too, with the label dropped because there is nothing else to compare against.
Attention is a weighted lookup, and every token does one per layer
Take the sentence "The trophy did not fit in the suitcase because it was too big." To decide what "it" means, a reader looks back at "trophy" and "suitcase" and picks the one that makes sense with "too big". Attention is that lookup, done with numbers.
Each token is turned into three vectors: a query (what am I looking for), a key (what do I offer) and a value (what I contribute if picked). The token "it" compares its query against every other token's key, gets a relevance score per token, turns the scores into weights that sum to one, and adds up the values by those weights. Its new representation is mostly "trophy" with a little of everything else. The model does this in parallel for every token, in several independent "heads" that learn to track different relationships, and repeats it in dozens of stacked layers.
Two consequences follow directly. Because every token is processed in the same step, the whole layer runs as one big matrix multiplication, which is exactly what a GPU is built for, and the paper's base model trained in twelve hours on eight GPUs. And because every token compares against every other, the cost of one layer grows with the square of the sequence length: twice the tokens, four times the attention work.
Why the square is the number that reaches your bill
That quadratic cost is why the context window is the specification providers compete on, and why using it isn't free. A model that accepts 1,000,000 tokens has to do attention over a million-by-million grid of relevance scores, in every head, in every layer. The engineering that makes that affordable (sparse patterns, sliding windows, cheaper attention variants, and caching earlier tokens' keys and values so a follow-up turn doesn't redo them) is why a 1,000,000-token window is priced per token rather than refused.
In the AI model cost index, 47 of the 100 priced rows carry a window of 1,000,000 tokens or more, up from 512 tokens in GPT-1 and 2,048 in GPT-3. Every one of them still bills per input token, because every token you send is a row in that grid. A 300,000-token document asked ten questions is 3 million input tokens, and what are tokens in AI prices that at $6 on Claude Sonnet 5. The architecture explains the invoice.
It also explains the failure mode. A transformer has no memory outside its window. What isn't in the sequence doesn't exist for it, which is why a long conversation forgets its start and why retrieval, the subject of what is RAG, exists to put the right tokens back in.
Encoder, decoder or both: the shape decides what the model is for
The 2017 model had two halves: an encoder that read the source sentence and a decoder that wrote the translation. Nearly every model since keeps only one half.
| Shape | What it does | Trained to | Examples |
|---|---|---|---|
| Encoder only | Reads text and outputs a representation of it | Fill in masked words | BERT (2018), the embedding models behind search |
| Decoder only | Writes text one token at a time | Predict the next token | GPT, Claude, Gemini, Llama, DeepSeek, Qwen |
| Encoder and decoder | Reads one sequence, writes another | Translate or transform | The original 2017 model, T5 |
The chat models you use are all decoder only. That choice, which GPT-1 made in 2018 and every chat model has copied since, is why a language model generates rather than looks up: the whole network is a next-token predictor, what is temperature in AI covers the dial that picks the token, and what is an LLM follows that thread to hallucination and the rest. Encoder-only models didn't disappear. They became the embedding models that turn text into vectors for search, which is the other job attention turned out to be good at.
Vision followed the same path. In 2020 the Vision Transformer paper cut an image into 16-by-16 pixel patches, treated each patch as a token, and matched the best convolutional networks. That is why the same model can now read your screenshot and your paragraph: both arrive as a sequence of tokens and go through the same attention layers. Image generation is the exception; most image models are diffusion models, a different design that what is generative AI separates out.
Nine years from one paper to every frontier model, dated
The timeline below runs from the 2017 paper to the models Whizi routes to today. Orange marks the releases that shipped open weights, which is how the design spread beyond the two labs that built it first.
Parameter counts tell the scale story in three numbers: 65 million in the 2017 base model, 175 billion in GPT-3 three years later, and 2.4 trillion in the largest Qwen model the cost index prices in 2026, of which about 95 billion are active on any one token. The architecture didn't change in kind across that range. It got wider, deeper and better fed. The bet this page makes: no frontier model released before 2028 will be anything but a transformer, and the AI model timeline is where that gets checked.
- Read "transformer" as "the design every current text model shares"; it does not tell you which model is better
- Expect cost and slowdown to grow faster than linearly with context length, and keep long uploads to the section you need
- Assume a model forgets anything outside its window; re-state the important facts or use a project to pin them
- Treat vision and text as the same pipeline: a screenshot is tokens too, so ask about it the way you would ask about a paragraph
- Check the release date before trusting what a model knows; the architecture has no clock
Frequently asked questions
What does transformer mean in AI?
A transformer is a neural network architecture from the 2017 paper Attention Is All You Need. It processes every token of a sequence in parallel using attention, an operation where each token weighs its relevance to every other, and it is the design inside GPT, Claude, Gemini, Llama, DeepSeek and Qwen. The paper does not say why it chose the name.
What does GPT stand for?
Generative pretrained transformer. Generative because it produces text, pretrained because it learns from a large corpus before being tuned for chat, and transformer because that is the network architecture. OpenAI introduced the name with GPT-1 in June 2018.
What is the attention mechanism?
The operation at the centre of a transformer. Each token computes a relevance score against every other token in the sequence, converts the scores to weights, and blends the other tokens' representations by those weights. That is how "it" learns to mean "trophy" in one sentence and "suitcase" in another. It runs in parallel across all tokens and in several heads per layer.
Why did transformers replace recurrent neural networks?
Recurrent networks read tokens one after another and had to pass information forward through every step, which limited both what they remembered and how fast they trained. Transformers read the whole sequence at once, so they keep direct access to every earlier token and train as parallel matrix operations that GPUs run efficiently. The 2017 paper matched the best translation systems in a fraction of the training time.
Are all AI models transformers?
Every frontier LLM is, and so is every text model Whizi offers; the exceptions are research lines such as Mamba and hybrids such as Jamba, none of which leads a benchmark. Image generators such as Stable Diffusion and Flux are diffusion models, a different design, although many use a transformer inside to read the prompt. Vision models that read images are transformers since the 2020 Vision Transformer paper, which treats image patches as tokens.
