Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Transformer (architecture)

The transformer is a neural network architecture, introduced in 2017, that processes sequences using attention instead of recurrence. It is the basis of nearly all current large language models and many image and audio models.

Also called: transformer model

By Divya Sharma · Checked Nov 15, 2026 · How we review

Before transformers, language models mostly used recurrent networks that read text one word at a time. The 2017 paper "Attention Is All You Need" proposed an architecture based solely on attention, dropping recurrence and convolutions. It was more parallel and much faster to train, which made today's large models practical.

How it works

Text is split into tokens and each token becomes a vector. A transformer stacks many identical layers. In each, the attention mechanism lets every token weigh every other token in the context window to decide what is relevant, then a feed-forward network processes the result. Because all tokens are handled at once in training, the work spreads well across GPUs.

Variants

The original design had an encoder and a decoder for translation. Most chat models, such as the GPT and Llama families, are decoder-only and generate one token at a time; they are autoregressive models. Encoder-only models such as BERT are common for embeddings.

Limits and the name

Standard attention cost grows with the square of sequence length, which is why long context is expensive. Alternatives such as state space models aim to fix that. This page is about the architecture, not the Hugging Face Transformers library, which is software for running such models.

Related terms

Attention mechanism
An attention mechanism lets a neural network decide, for each token it processes, how much weight to give every other token in the input. Self-attention is the core operation inside a transformer.
Large language model
A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
Autoregressive model
An autoregressive model generates a sequence one piece at a time, predicting each next token from all the tokens before it and feeding that output back in. Most large language models work this way.
State space model
A state space model (SSM) is a sequence model that carries a fixed-size hidden state forward as it reads each token, so its cost grows linearly with input length. Mamba is the best-known example.
Token (LLM)
A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.

Sources