Semantic cache
A semantic cache stores past LLM responses and returns one when a new request means roughly the same thing as an earlier one, judged by embedding similarity rather than an exact text match. It saves cost and latency on repeated questions.
Also called: semantic caching
By Divya Sharma · Checked Nov 15, 2026 · How we review
A normal cache only helps when a request is byte-for-byte identical to an earlier one. People rarely ask the same question the same way twice, so exact-match caching misses most repeats in a chat or support app. A semantic cache matches on meaning instead.
How it works
Each incoming prompt is turned into an embedding. The cache runs a vector search over the embeddings of earlier prompts. If the closest one is similar enough, measured for example with cosine similarity against a threshold you choose, the stored answer is returned and the model is never called. Otherwise the request goes to the model and the new pair is stored. Open-source GPTCache follows this design, and many LLM gateways and vector stores such as Redis offer it as a feature.
Semantic cache versus prompt caching
Prompt caching is done by the model provider. It reuses the computed state for an identical prompt prefix, still runs the model, and still returns a fresh answer. A semantic cache sits in front of the model and skips the call entirely, returning an old answer.
Limits
The threshold is the hard part. Too loose, and "cancel my order" may return the answer stored for "track my order". Too strict, and few requests hit the cache. Cached answers can also go stale, leak one user's personalised answer to another, or ignore conversation context, so semantic caching suits stateless, shared questions (FAQs, documentation lookups) better than personal or multi-turn chats. Scope the cache per tenant where data is private, and set an expiry.
Related terms
- Prompt caching
- Prompt caching is an API feature that stores the processed form of a repeated prompt prefix, such as a long system prompt or document, so later requests that start with the same prefix are cheaper and faster.
- LLM gateway
- An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
- Embedding
- An embedding is a list of numbers (a vector) that represents a piece of text, an image or other data, produced so that items with similar meaning end up with vectors close together.
- Vector search
- Vector search finds items by comparing their embeddings to the embedding of a query and returning the closest ones, so results match on meaning rather than on shared keywords.
- Cosine similarity
- Cosine similarity measures how alike two vectors are by the angle between them. It ranges from -1 to 1, ignores vector length, and is the most common way to compare embeddings in search.