What is DeepInfer?
DeepInfer is the inference framework DeepSeek uses to serve DeepSeek-V4.1-Flash on its own API, published on GitHub under Apache 2.0 on 20 October 2026. It was built mainly on vLLM, with ideas taken from SGLang and TensorRT-LLM, and adds DeepSeek's own kernels (FlashMLA, DeepGEMM, DeepEP, TileLang), wide expert parallelism, speculative decoding with confidence-scheduled verification, batch-invariant inference and structured output.
This first release is deliberately partial. It supports one model, DeepSeek-V4.1-Flash, runs offline inference only, and targets NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. The KV cache store, prefill/decode disaggregation and the load balancers that make up a full service are not included; DeepSeek says they are tied to its internal infrastructure and describes their design in a deployment Q&A instead.
There are no tagged releases yet. The repository has issues switched off and takes questions through GitHub Discussions and pull requests.
Is DeepInfer worth it?
DeepInfer is most valuable as documentation that happens to run. DeepSeek says the system it comes from serves 155 trillion tokens a day from a single cluster and earns US$38 per GPU-hour at off-peak prices, and the repository's deployment Q&A explains how: prefill on ordinary 8-GPU machines, decode spread across 64 to 128 GPUs with one expert group per GPU, and a dedicated SSD cluster holding KV cache for reuse. Few labs publish that much about their serving stack.
The headline numbers are DeepSeek's own. Its decode chart shows about 26,000 output tokens per second per GPU at 256 requests per rank, falling to roughly 100 tokens per second for each user, measured on 64 SM100 GPUs with a 131,072-token context and counting every draft token as accepted. At about 60 requests per rank, which the Q&A calls the production operating point, its table puts throughput nearer 14,500 tokens per second per rank. We found no independent reproduction; the repository ships the sweep scripts to try.
The gaps decide who can use it today. There is no HTTP server, only a Python offline API, so serving real traffic means building the KV store, disaggregation and load balancing yourself. Hardware is limited to Blackwell-class NVIDIA GPUs and Ascend 950, so Hopper fleets are out, and the Q&A puts the smallest efficient deployment at 64 GPUs for prefill plus 64 for decode. The only model is V4.1-Flash, whose checkpoint alone is about 765 GB.
For anyone serving V4.1-Flash, vLLM and SGLang remain the practical engines, and DeepInfer's kernels and settings are the reference to tune them against. Its batch-invariant mode, which keeps outputs identical regardless of what else is in the batch, is the feature most worth borrowing for evaluation and RL work. DeepSeek says code for future models will land here, which would turn it from a reference into a contender.
Where it earns its place
An inference team at a GPU cloud serving DeepSeek-V4.1-Flash on B200-class nodes
Close the throughput gap with DeepSeek's own API
Run DeepInfer's benchmark sweep next to the current vLLM or SGLang deployment and copy what wins: wide expert parallelism at EP64, redundant-expert balancing, the five-token draft setting. The software is free; a fair comparison needs at least 64 SM100 GPUs for the decode side.
A research group running RL post-training on DeepSeek-V4.1-Flash
Get rollouts whose outputs do not change with batch composition
Set DEEPINFER_BATCH_INVARIANT=1 and pass a fixed seed to get identical outputs for identical inputs, on NVIDIA only. Offline batch generation is exactly what this release supports, so no serving layer has to be built.
A Chinese cloud provider standardised on Huawei Ascend
Serve DeepSeek's model on domestic accelerators with first-party code
Build with DEEPINFER_ACCELERATOR=ascend on CANN 9.2 for Ascend 950 NPUs. Apache 2.0 allows commercial use, but the provider still has to supply its own KV cache, request routing and API front end.
Before you commit
- Setup
- Build from source on Linux with GCC 13 or newer, Python 3.12 and CUDA 13 (or CANN 9.2 for Ascend) via ./build.sh, then run examples/offline_inference.py, which downloads DeepSeek-V4.1-Flash-1017 from Hugging Face on first run. No pip package or Docker image is published.
- Integrations
- Python offline API with structured output (regex, JSON schema, EBNF). DeepSeek publishes deepseek-recipe, an MIT-licensed Rust library, to convert Chat Completions, Responses and Messages requests to tokens; there is no built-in server.
- Data and privacy
- Runs entirely on your hardware; the code tree contains no usage-statistics module, and the only network call in the quickstart is the Hugging Face model download. Apache 2.0 with no field-of-use limits. No security advisories published.
- Cost at scale
- The Q&A puts the smallest efficient unit at 64 GPUs of prefill plus 64 of decode, with about 400 Gbps of interconnect per 8 GPUs and roughly 14.4 TB of KV cache storage per GPU for 24-hour retention. There is no paid or hosted edition.
- Limits
- Offline only; one supported model; SM100 and Ascend 950 only; batch invariance and deterministic sampling are NVIDIA-only. No tagged releases, issues disabled, and more than 200 pull requests opened in its first six days.
What's good
- Production engine behind DeepSeek's own API, Apache 2.0
- Detailed deployment Q&A with real parallelism and storage settings
- Batch-invariant inference and deterministic sampling on NVIDIA
- Runs on Huawei Ascend 950 as well as NVIDIA
What's not
- Offline inference only; no HTTP or OpenAI-compatible server
- KV cache store, disaggregation and load balancing not released
- One model (DeepSeek-V4.1-Flash) and SM100 or Ascend 950 only
- Throughput figures are self-reported and unreplicated
Best for
Inference engineers running DeepSeek-V4.1-Flash at cluster scale who want DeepSeek's kernels and settings as a reference.
DeepInfer alternatives
DeepInfer: common questions
- Is DeepInfer worth it?
- A rare look inside a production LLM service, but for now it is a reference engine for one model on two chip families, not something to put behind an API. We rate it 3.6 out of 5.
- How much does DeepInfer cost?
- Free and open source under the Apache 2.0 licence. You pay for Blackwell GPUs or Ascend NPUs.
- What are the downsides of DeepInfer?
- Offline inference only; no HTTP or OpenAI-compatible server; kV cache store, disaggregation and load balancing not released; one model (DeepSeek-V4.1-Flash) and SM100 or Ascend 950 only; throughput figures are self-reported and unreplicated.
- Who is DeepInfer best for?
- Inference engineers running DeepSeek-V4.1-Flash at cluster scale who want DeepSeek's kernels and settings as a reference.
- How much does DeepInfer cost for a team or in production?
- The Q&A puts the smallest efficient unit at 64 GPUs of prefill plus 64 of decode, with about 400 Gbps of interconnect per 8 GPUs and roughly 14.4 TB of KV cache storage per GPU for 24-hour retention. There is no paid or hosted edition.
- Is DeepInfer safe for company data?
- Runs entirely on your hardware; the code tree contains no usage-statistics module, and the only network call in the quickstart is the Hugging Face model download. Apache 2.0 with no field-of-use limits. No security advisories published.
- What are the best DeepInfer alternatives?
- The closest alternatives we have reviewed are vLLM, SGLang and TensorRT-LLM.
Answers checked Oct 26, 2026.