What is LLMLingua?
LLMLingua shortens prompts before they are sent to a large model. A smaller language model scores each token and the library drops the ones it judges least informative, so the output is a terse, often ungrammatical prompt that the large model can still read. The package bundles three published methods: LLMLingua (EMNLP 2023), LongLLMLingua (ACL 2024), which ranks and reorders retrieved passages around the user's question for RAG, and LLMLingua-2 (ACL 2024 Findings), a token classifier distilled from GPT-4 labels that the authors say runs 3x to 6x faster than the original. A fourth model, SecurityLingua, applies the same idea to spotting jailbreak prompts.
Which model runs locally depends on the method. The default PromptCompressor loads NousResearch/Llama-2-7b-hf, a 6.7-billion-parameter model, onto a CUDA GPU; the README offers microsoft/phi-2 or a GPTQ-quantised Llama 2 that fits in under 8 GB of GPU memory. LLMLingua-2 swaps that for an encoder: microsoft/llmlingua-2-xlm-roberta-large-meetingbank (about 559 million parameters) or a multilingual BERT-base version (about 177 million), small enough for CPU.
It is MIT-licensed, installs with pip install llmlingua, and has integrations in LangChain, LlamaIndex and Microsoft's Prompt flow.
Is LLMLingua worth it?
LLMLingua is the best-known academic answer to a real bill: retrieved documents and long transcripts that cost more to send than the answer is worth. The "up to 20x compression with minimal performance loss" in the README is the authors' own result from the 2023 paper on reasoning and in-context-learning benchmarks, not an independent measurement, and the README's own example compresses a 2,365-token maths prompt to 211 tokens, an 11.2x ratio. LongLLMLingua's headline claim is narrower and more useful for RAG: up to 21.4% better answers while using a quarter of the tokens, again on the authors' benchmarks.
The compressed text is lossy and looks it. In the README example, "highlighter pens" becomes "ilters" and numbers merge together; the large model usually copes, but anything that depends on exact figures, names, code or legal wording should be kept out of compression. The library lets you mark sections with compress=False tags, or force tokens such as newlines and question marks to survive, and that care is what separates a useful deployment from a quietly worse one.
The hidden cost is the compressor itself. The default path loads a 7B Llama 2 on a GPU for every call, which can cost more in latency and hardware than it saves on a cheap API model. LLMLingua-2 with the 559-million-parameter XLM-RoBERTa model is the sensible starting point. Ratios are reported in OpenAI tokens, because the library counts with tiktoken's gpt-3.5-turbo encoding, so savings on Claude or Gemini will differ.
The catch: maintenance has slowed to a trickle. The last PyPI release, 0.2.2, dates from 9 April 2024; 2026 commits so far pin GitHub Actions, and 101 issues are open. It works, but you are adopting research code, not a supported product.
Our take: compression earns its keep where prompts are long, repetitive and tolerant of paraphrase, such as meeting transcripts and stacks of retrieved passages. For short agent prompts, prompt caching from the model provider is the cheaper first move.
Where it earns its place
An ML engineer running a RAG assistant over policy documents
Cut the input tokens spent on retrieved passages without losing answer quality
Add LongLLMLingua as a LlamaIndex node postprocessor with rate set near 0.5, and keep the user question and citations uncompressed. Measure answer accuracy on 100 real questions before and after; the paper's 4x figure is a ceiling, not a promise.
A product team summarising hour-long sales calls
Fit full transcripts into a cheaper model's context window
Run LLMLingua-2 with the multilingual BERT-base model on CPU over the transcript at rate 0.33, keeping speaker labels with force_tokens. The compressor adds a model call per transcript but no API spend.
A security engineer at a company with a public chatbot
Flag jailbreak attempts before they reach the main model
SecurityLingua compresses an incoming prompt down to its apparent intent, which a guard step can then check. It is a 2025 research model with its own readme, so treat it as one signal alongside a dedicated guardrail.
Before you commit
- Setup
- pip install llmlingua pulls in torch and transformers; the first PromptCompressor call downloads the chosen model from Hugging Face, several GB for the default Llama 2.
- Integrations
- LangChain retriever, LlamaIndex node postprocessor, Microsoft Prompt flow tool, and plain Python calls (compress_prompt, structured_compress_prompt).
- Data and privacy
- Runs entirely locally once models are downloaded; nothing is sent to Microsoft. The 20x and 21.4% figures come from the authors' papers.
- Cost at scale
- No licence cost. The real cost is compressor inference: a GPU for the 7B default, or CPU time for LLMLingua-2's 177M or 559M encoders.
- Limits
- Last release 0.2.2 in April 2024; 101 open issues; savings counted in OpenAI tokens; lossy output unsuitable for code, exact figures or contractual text.
What’s good
- Three peer-reviewed methods in one pip package
- LLMLingua-2 runs on a 177M or 559M encoder, CPU-friendly
- Per-section control over what gets compressed
- Ready integrations for LangChain and LlamaIndex
What’s not
- Headline 20x ratio is the authors' best case
- Default model is a 7B Llama 2 on GPU
- Lossy output can mangle numbers and names
- No PyPI release since April 2024
Best for
RAG and transcript-heavy pipelines where long, paraphrase-tolerant context dominates the token bill and someone can test answer quality before and after.
LLMLingua alternatives
LLMLingua: common questions
- Is LLMLingua worth it?
- A credible research method for cutting long RAG prompts, but the 20x figure is the paper's best case and the package has not shipped a release since April 2024. We rate it 3.7 out of 5.
- How much does LLMLingua cost?
- Free and open source under MIT. You pay for the GPU or CPU time of the local compressor model.
- What are the downsides of LLMLingua?
- Headline 20x ratio is the authors' best case; default model is a 7B Llama 2 on GPU; lossy output can mangle numbers and names; no PyPI release since April 2024.
- Who is LLMLingua best for?
- RAG and transcript-heavy pipelines where long, paraphrase-tolerant context dominates the token bill and someone can test answer quality before and after.
- How much does LLMLingua cost for a team or in production?
- No licence cost. The real cost is compressor inference: a GPU for the 7B default, or CPU time for LLMLingua-2's 177M or 559M encoders.
- Is LLMLingua safe for company data?
- Runs entirely locally once models are downloaded; nothing is sent to Microsoft. The 20x and 21.4% figures come from the authors' papers.
- What are the best LLMLingua alternatives?
- The closest alternatives we have reviewed are Headroom, LlamaIndex and Caveman.
Answers checked Nov 20, 2026.