Late interaction (ColBERT)
Late interaction is a retrieval method, introduced by ColBERT, that encodes the query and each document separately into one vector per token, then scores a match by comparing those token vectors in a cheap final step.
Also called: ColBERT, multi-vector retrieval, late interaction
By Divya Sharma · Checked Nov 15, 2026 · How we review
There are two common ways to score relevance with a transformer. A bi-encoder squeezes each text into one vector, which is fast but loses detail. A cross-encoder reads query and document together, which is accurate but too slow for large collections. Late interaction sits between them.
How it works
ColBERT encodes the document once, offline, keeping a vector for every token. At query time it encodes the query the same way. For each query token it finds the most similar document token, then adds those best matches up into a score. Because document vectors are precomputed, the expensive model work at query time is only on the short query. The interaction between the two texts happens late, in this final comparison.
The ColBERT paper reports accuracy competitive with BERT-based rerankers while running about two orders of magnitude faster.
Multi-vector retrieval
Multi-vector retrieval is the broader name for storing several vectors per item. The same idea is now used for images of pages, where each image patch gets a vector, as in visual document retrievers such as Nemotron ColEmbed V2.
Limits
Storing a vector per token takes much more space than one vector per chunk, though compression helps. Fewer vector databases support multi-vector search natively. Many teams use late interaction as a reranking stage over a cheaper first search.
Related terms
- Cross-encoder
- A cross-encoder is a model that reads a query and a document together in one pass and outputs a relevance score. It is more accurate than comparing separate embeddings, but too slow to run over a whole collection.
- Reranking
- Reranking is a second search step that takes a short list of candidate documents and reorders them by relevance to the query, using a more accurate but slower model than the first search.
- Vector search
- Vector search finds items by comparing their embeddings to the embedding of a query and returning the closest ones, so results match on meaning rather than on shared keywords.
- Embedding
- An embedding is a list of numbers (a vector) that represents a piece of text, an image or other data, produced so that items with similar meaning end up with vectors close together.
- Vector database
- A vector database stores embeddings alongside their source records and finds the stored vectors closest to a query vector, usually with an approximate index so search stays fast across millions of items.
Sources
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (Khattab and Zaharia, 2020)
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (Khattab and Zaharia, 2020)
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (Khattab and Zaharia, 2020)