Transformer (architecture)
The transformer is a neural network architecture, introduced in 2017, that processes sequences using attention instead of recurrence. It is the basis of nearly all current large language models and many image and audio models.
Also called: transformer model
By Divya Sharma · Checked Nov 15, 2026 · How we review
Before transformers, language models mostly used recurrent networks that read text one word at a time. The 2017 paper "Attention Is All You Need" proposed an architecture based solely on attention, dropping recurrence and convolutions. It was more parallel and much faster to train, which made today's large models practical.
How it works
Text is split into tokens and each token becomes a vector. A transformer stacks many identical layers. In each, the attention mechanism lets every token weigh every other token in the context window to decide what is relevant, then a feed-forward network processes the result. Because all tokens are handled at once in training, the work spreads well across GPUs.
Variants
The original design had an encoder and a decoder for translation. Most chat models, such as the GPT and Llama families, are decoder-only and generate one token at a time; they are autoregressive models. Encoder-only models such as BERT are common for embeddings.
Limits and the name
Standard attention cost grows with the square of sequence length, which is why long context is expensive. Alternatives such as state space models aim to fix that. This page is about the architecture, not the Hugging Face Transformers library, which is software for running such models.
Related terms
- Attention mechanism
- An attention mechanism lets a neural network decide, for each token it processes, how much weight to give every other token in the input. Self-attention is the core operation inside a transformer.
- Large language model
- A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
- Autoregressive model
- An autoregressive model generates a sequence one piece at a time, predicting each next token from all the tokens before it and feeding that output back in. Most large language models work this way.
- State space model
- A state space model (SSM) is a sequence model that carries a fixed-size hidden state forward as it reads each token, so its cost grows linearly with input length. Mamba is the best-known example.
- Token (LLM)
- A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.