Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

GPTCache

Zilliz's semantic cache for LLM calls: answers similar questions from stored replies, but releases stopped in 2024.

Reviewed by ReviewRamble editors · Updated

View on GitHub Open source

Share this page:XLinkedInFacebookRedditWhatsAppEmail

Our rating

2.9/5

A semantic cache that taught the industry the idea, now mostly dormant, and one whose default matching would hand back the wrong answer more often than the right one on its own new benchmark.

What is GPTCache?

GPTCache sits in front of an LLM and returns a stored answer when a new request looks like one it has seen. It turns each request into an embedding, searches a vector store for near neighbours, and asks a similarity evaluator whether the closest match is close enough to reuse. Every part is swappable: embeddings from ONNX, OpenAI, Cohere, Hugging Face or SentenceTransformers; answers in SQLite, PostgreSQL, MySQL, DynamoDB and others; vectors in Milvus, FAISS, Chroma, Qdrant, pgvector and more; eviction by LRU, FIFO or LFU, in memory or through Redis or memcached.

It began in 2023 at Zilliz, the company behind the Milvus vector database, and was wired into LangChain and LlamaIndex early. It is MIT-licensed, written in Python, and can also run as a Docker server for other languages.

The README now says the project no longer adds support for new APIs or models and points users to generic get and set calls. In September 2026 a contributor merged adapters for the Anthropic Messages API and the current OpenAI client, plus an optional evaluator that sends each candidate reuse to TypeSafe's hosted API for a yes or no.

Is GPTCache worth it?

GPTCache is stagnating, and anyone adopting it in late 2026 should say so in the design doc. The last PyPI release, 0.1.44, shipped on 1 August 2024, and the one before that in November 2023. Commits fell from 34 in June 2023 to a handful a year, then a burst of 10 on 22 September 2026: one contributor's adapters and benchmark, merged by a maintainer. There are 77 open issues and 26 open pull requests. The README still says the project is "under heavy development" two lines above a note that it no longer supports new models.

The bigger problem is the one every semantic cache has: a question that looks similar is not always the same question. "What is our refund window in Germany?" and "...in France?" sit close together as embeddings, and a cache that matches them serves a confident, wrong, stale answer with no model call to catch it. GPTCache's README acknowledges false positives; the September 2026 benchmark puts a number on them. On 1,536 labelled pairs, the repository's default cross-encoder reranker at its 0.80 threshold reused 880 answers, and 508 of those reuses were wrong, a precision of 42.2%.

The same pull request shows that a stricter judge helps: the optional JevEvaluation, which calls TypeSafe's hosted API, reached 97.3% to 99.7% precision on the same set while keeping 72% to 78% recall. That is a third-party service seeing every query and cached answer, a new dependency and a per-call cost, which undercuts the reason to cache locally.

The catch: what remains is a well-designed but ageing toolkit whose default settings are unsafe for anything where the exact question matters, such as prices, policies, account data or dates.

Our take: an exact-match cache plus the model provider's own prompt caching captures most of the savings with none of the wrong-answer risk. If semantic matching is truly needed, gateways such as LiteLLM and Bifrost now offer it inside a project that is still shipping releases.

Where it earns its place

  • A support lead at a SaaS company with a public FAQ bot

    Stop paying for the same 200 questions asked thousands of times a day

    Start with exact-match caching (embeddings disabled) in SQLite, which carries no wrong-answer risk. Add semantic matching only after labelling a sample of near-duplicate questions and raising the similarity threshold until wrong reuses stop.

  • A developer writing tests for an LLM-backed feature

    Run the test suite without calling the paid API on every run

    Wrap the client with GPTCache's exact-match cache so repeated test prompts return stored responses. Test runs become fast and free after the first pass; clear the cache when prompts change.

  • A platform engineer evaluating semantic caching for an internal assistant

    Measure how often a semantic cache would answer the wrong question

    Replay the 1,536-case reuse benchmark in examples/benchmark, then score a week of the firm's own query pairs. The default reranker's 42.2% precision on the shipped set is the baseline to beat before anything goes live.

Before you commit

Setup
pip install gptcache and cache.init() for exact matching; semantic mode needs an embedding model (ONNX by default), a vector store and a similarity evaluator configured in code, and extra libraries install on first use.
Integrations
Adapters for the OpenAI and Anthropic Messages clients, LangChain, LlamaIndex, llama.cpp and image and audio APIs; a Docker server exposes it to other languages.
Data and privacy
Cached prompts and answers sit in whatever store you configure, unencrypted unless that store encrypts them. The optional JevEvaluation sends the query and candidate answer to api.typesafe.ai.
Cost at scale
No licence cost; you pay for the embedding model, the vector store and, if used, TypeSafe API calls on every cache candidate.
Limits
In-memory eviction counts entries, not bytes, which the README warns can cause out-of-memory errors; not all module combinations are compatible; 77 open issues and 26 open pull requests.

What’s good

  • Modular: embeddings, stores, vector databases and evaluators all swappable
  • Broad backend support, from SQLite and FAISS to Milvus and DynamoDB
  • Exact-match mode works with embeddings turned off
  • Published benchmark of reuse precision and recall

What’s not

  • No PyPI release since August 2024
  • Default reranker reused wrong answers 508 times out of 880 on its own benchmark
  • README says no new model or API support will be added
  • The accurate evaluator sends queries and answers to a third-party API

Best for

Teams with high-volume, low-stakes repeat questions, such as public FAQ bots, who will tune thresholds against their own labelled traffic and accept maintaining a fork.

GPTCache alternatives

GPTCache: common questions

Is GPTCache worth it?
A semantic cache that taught the industry the idea, now mostly dormant, and one whose default matching would hand back the wrong answer more often than the right one on its own new benchmark. We rate it 2.9 out of 5.
How much does GPTCache cost?
Free and open source under MIT. You run the embedding model, cache store and vector store; the optional TypeSafe evaluator needs its own API key.
What are the downsides of GPTCache?
No PyPI release since August 2024; default reranker reused wrong answers 508 times out of 880 on its own benchmark; rEADME says no new model or API support will be added; the accurate evaluator sends queries and answers to a third-party API.
Who is GPTCache best for?
Teams with high-volume, low-stakes repeat questions, such as public FAQ bots, who will tune thresholds against their own labelled traffic and accept maintaining a fork.
How much does GPTCache cost for a team or in production?
No licence cost; you pay for the embedding model, the vector store and, if used, TypeSafe API calls on every cache candidate.
Is GPTCache safe for company data?
Cached prompts and answers sit in whatever store you configure, unencrypted unless that store encrypts them. The optional JevEvaluation sends the query and candidate answer to api.typesafe.ai.
What are the best GPTCache alternatives?
The closest alternatives we have reviewed are LiteLLM, Bifrost and Portkey AI Gateway.

Answers checked Nov 20, 2026.