Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

LLMLingua

Microsoft Research's prompt compressor: a small local model deletes low-value tokens before a prompt reaches the paid one.

Reviewed by ReviewRamble editors · Updated

View on GitHub Open source

Share this page:XLinkedInFacebookRedditWhatsAppEmail

Our rating

3.7/5

A credible research method for cutting long RAG prompts, but the 20x figure is the paper's best case and the package has not shipped a release since April 2024.

What is LLMLingua?

LLMLingua shortens prompts before they are sent to a large model. A smaller language model scores each token and the library drops the ones it judges least informative, so the output is a terse, often ungrammatical prompt that the large model can still read. The package bundles three published methods: LLMLingua (EMNLP 2023), LongLLMLingua (ACL 2024), which ranks and reorders retrieved passages around the user's question for RAG, and LLMLingua-2 (ACL 2024 Findings), a token classifier distilled from GPT-4 labels that the authors say runs 3x to 6x faster than the original. A fourth model, SecurityLingua, applies the same idea to spotting jailbreak prompts.

Which model runs locally depends on the method. The default PromptCompressor loads NousResearch/Llama-2-7b-hf, a 6.7-billion-parameter model, onto a CUDA GPU; the README offers microsoft/phi-2 or a GPTQ-quantised Llama 2 that fits in under 8 GB of GPU memory. LLMLingua-2 swaps that for an encoder: microsoft/llmlingua-2-xlm-roberta-large-meetingbank (about 559 million parameters) or a multilingual BERT-base version (about 177 million), small enough for CPU.

It is MIT-licensed, installs with pip install llmlingua, and has integrations in LangChain, LlamaIndex and Microsoft's Prompt flow.

Is LLMLingua worth it?

LLMLingua is the best-known academic answer to a real bill: retrieved documents and long transcripts that cost more to send than the answer is worth. The "up to 20x compression with minimal performance loss" in the README is the authors' own result from the 2023 paper on reasoning and in-context-learning benchmarks, not an independent measurement, and the README's own example compresses a 2,365-token maths prompt to 211 tokens, an 11.2x ratio. LongLLMLingua's headline claim is narrower and more useful for RAG: up to 21.4% better answers while using a quarter of the tokens, again on the authors' benchmarks.

The compressed text is lossy and looks it. In the README example, "highlighter pens" becomes "ilters" and numbers merge together; the large model usually copes, but anything that depends on exact figures, names, code or legal wording should be kept out of compression. The library lets you mark sections with compress=False tags, or force tokens such as newlines and question marks to survive, and that care is what separates a useful deployment from a quietly worse one.

The hidden cost is the compressor itself. The default path loads a 7B Llama 2 on a GPU for every call, which can cost more in latency and hardware than it saves on a cheap API model. LLMLingua-2 with the 559-million-parameter XLM-RoBERTa model is the sensible starting point. Ratios are reported in OpenAI tokens, because the library counts with tiktoken's gpt-3.5-turbo encoding, so savings on Claude or Gemini will differ.

The catch: maintenance has slowed to a trickle. The last PyPI release, 0.2.2, dates from 9 April 2024; 2026 commits so far pin GitHub Actions, and 101 issues are open. It works, but you are adopting research code, not a supported product.

Our take: compression earns its keep where prompts are long, repetitive and tolerant of paraphrase, such as meeting transcripts and stacks of retrieved passages. For short agent prompts, prompt caching from the model provider is the cheaper first move.

Where it earns its place

  • An ML engineer running a RAG assistant over policy documents

    Cut the input tokens spent on retrieved passages without losing answer quality

    Add LongLLMLingua as a LlamaIndex node postprocessor with rate set near 0.5, and keep the user question and citations uncompressed. Measure answer accuracy on 100 real questions before and after; the paper's 4x figure is a ceiling, not a promise.

  • A product team summarising hour-long sales calls

    Fit full transcripts into a cheaper model's context window

    Run LLMLingua-2 with the multilingual BERT-base model on CPU over the transcript at rate 0.33, keeping speaker labels with force_tokens. The compressor adds a model call per transcript but no API spend.

  • A security engineer at a company with a public chatbot

    Flag jailbreak attempts before they reach the main model

    SecurityLingua compresses an incoming prompt down to its apparent intent, which a guard step can then check. It is a 2025 research model with its own readme, so treat it as one signal alongside a dedicated guardrail.

Before you commit

Setup
pip install llmlingua pulls in torch and transformers; the first PromptCompressor call downloads the chosen model from Hugging Face, several GB for the default Llama 2.
Integrations
LangChain retriever, LlamaIndex node postprocessor, Microsoft Prompt flow tool, and plain Python calls (compress_prompt, structured_compress_prompt).
Data and privacy
Runs entirely locally once models are downloaded; nothing is sent to Microsoft. The 20x and 21.4% figures come from the authors' papers.
Cost at scale
No licence cost. The real cost is compressor inference: a GPU for the 7B default, or CPU time for LLMLingua-2's 177M or 559M encoders.
Limits
Last release 0.2.2 in April 2024; 101 open issues; savings counted in OpenAI tokens; lossy output unsuitable for code, exact figures or contractual text.

What’s good

  • Three peer-reviewed methods in one pip package
  • LLMLingua-2 runs on a 177M or 559M encoder, CPU-friendly
  • Per-section control over what gets compressed
  • Ready integrations for LangChain and LlamaIndex

What’s not

  • Headline 20x ratio is the authors' best case
  • Default model is a 7B Llama 2 on GPU
  • Lossy output can mangle numbers and names
  • No PyPI release since April 2024

Best for

RAG and transcript-heavy pipelines where long, paraphrase-tolerant context dominates the token bill and someone can test answer quality before and after.

LLMLingua alternatives

LLMLingua: common questions

Is LLMLingua worth it?
A credible research method for cutting long RAG prompts, but the 20x figure is the paper's best case and the package has not shipped a release since April 2024. We rate it 3.7 out of 5.
How much does LLMLingua cost?
Free and open source under MIT. You pay for the GPU or CPU time of the local compressor model.
What are the downsides of LLMLingua?
Headline 20x ratio is the authors' best case; default model is a 7B Llama 2 on GPU; lossy output can mangle numbers and names; no PyPI release since April 2024.
Who is LLMLingua best for?
RAG and transcript-heavy pipelines where long, paraphrase-tolerant context dominates the token bill and someone can test answer quality before and after.
How much does LLMLingua cost for a team or in production?
No licence cost. The real cost is compressor inference: a GPU for the 7B default, or CPU time for LLMLingua-2's 177M or 559M encoders.
Is LLMLingua safe for company data?
Runs entirely locally once models are downloaded; nothing is sent to Microsoft. The 20x and 21.4% figures come from the authors' papers.
What are the best LLMLingua alternatives?
The closest alternatives we have reviewed are Headroom, LlamaIndex and Caveman.

Answers checked Nov 20, 2026.