Tensor parallelism
Tensor parallelism splits each layer of a neural network across several GPUs, so that every GPU holds part of each weight matrix and computes its share of every step. It lets a model too large for one GPU run across many.
Also called: multi-GPU, pipeline parallelism, model parallelism
By Divya Sharma · Checked Nov 15, 2026 · How we review
When a model's weights do not fit in one GPU's VRAM, you have to split it. There are two main ways, and serving tools often combine them.
Tensor parallelism
Each large weight matrix is cut into slices, one per GPU. For every token, all GPUs work on the same layer at the same time, each on its slice, and then exchange partial results before the next layer. The approach was popularised by NVIDIA's Megatron-LM, which described an intra-layer model parallel method needing only a few added communication steps. Because GPUs talk to each other at every layer, tensor parallelism needs a fast link such as NVLink, and is usually kept within one machine. In vLLM and SGLang it is a single setting, the tensor parallel size.
Pipeline parallelism
Pipeline parallelism splits the model by layers instead: GPU one runs the first block of layers, GPU two the next, and so on, passing activations along. It needs far less communication, so it works across machines, but a GPU can sit idle while it waits for the previous stage unless many requests are kept in flight. Megatron-LM describes the two as complementary.
Why it matters
Tensor parallelism is how large open-weight models are served on a node of several GPUs, and it can also lower latency, since each GPU does less work per token. The GPU calculator shows when a model needs more than one card.
Limits
Communication overhead grows with the number of GPUs, so speed does not scale linearly. On consumer cards without a fast interconnect, CPU offloading or a smaller quantized model may be the more practical choice. Mixture-of-experts models add a third option, expert parallelism.
Related terms
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Model serving
- Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
- GPU
- A GPU (graphics processing unit) is a processor with many small cores built to run many calculations in parallel. Its speed at matrix maths makes it the standard hardware for training and running AI models.
- CPU offloading
- CPU offloading runs a model that does not fit in GPU memory by keeping some of its layers in system RAM, or on disk, and using the GPU for the rest. The model runs, but more slowly.
- Mixture of experts
- Mixture of experts (MoE) is a model design in which each layer holds several expert sub-networks and a router sends each token to only a few of them, so only part of the model runs per token.