What is BitDecoding-FP4?
BitDecoding-FP4 is the code release for the paper "BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design" (arXiv 2611.09825, posted 6 November 2026, by Dayou Du, Woon Yee Ng, Guangming Sheng, Nick Lau, Jianyi Cheng and Luo Mai). It shrinks the KV cache, the per-request memory that grows with context length, to the 4-bit NVFP4 or MXFP4 formats and runs attention on the GPU's FP4 Tensor Cores instead of converting back to 16-bit first.
The paper's argument is that plain FP4 rounding loses too much accuracy, while the accurate 4-bit methods use codebook lookups that slow decoding down. Its fixes are a two-part FP4 split of the query (which the authors say cuts query logit error from 60% to 1%), a scaling step so long-context attention probabilities do not round to zero, and smoothing plus Hadamard rotation for outlier keys. It plugs into vLLM as an out-of-tree CUSTOM attention backend.
This is research code, not a product. The repository was created on 3 November 2026, has no tagged releases, and ships benchmark and evaluation scripts that reproduce the paper's tables. It needs one NVIDIA B200 or B300 (SM100 or SM103), CUDA 13.0 or newer with driver 580+, and Python 3.12.
Is BitDecoding-FP4 worth it?
BitDecoding-FP4 is worth reading if long-context serving on Blackwell is your job, and worth running only if you have a B200 or B300 to spare. It targets the part of serving that grows with every token of context: the KV cache. Storing it in 4 bits instead of 16 cuts its memory to roughly a quarter before scale factors, which lets more requests share a GPU.
Every headline number comes from the authors. Their README reports throughput of 3.00x BF16 KV for Qwen3-4B-Thinking-2507 and 2.62x for Qwen3-30B-A3B-Thinking-2507 on a B300, at 64K-token prompts on one GPU and 128K on two, across 1 to 64 concurrent requests; the abstract gives the range as 2.4 to 3.0x. On accuracy, the average over seven benchmarks (LongBench-Pro, MRCR, RULER-NIAH, AIME 2024 and 2025, LiveCodeBench v6, GPQA-Diamond) stays within 0.5 points of BF16 KV with NVFP4, and within 1 point with MXFP4. Larger FP8-weight models, GLM-4.7 and Hy4-preview, appear in one table. We found no independent reproduction; the paper went up on 6 November 2026.
The catch: the hardware gate is narrow. Only SM100 and SM103 parts are supported, so Hopper, consumer RTX cards and AMD are out, and the e2e benchmark needs a second B-series GPU for 128K. The install pins torch 2.11.0 and builds a vendored vLLM checkout from a submodule, so it will not drop into an existing vLLM deployment as a pip upgrade. One kernel helper is under NVIDIA's CUTLASS DSL licence rather than MIT, which a legal team will want to read. Key smoothing has to be calibrated per model, and the repository has 52 stars and five forks, so there is no community yet to ask.
Our take: for an inference team, this is a paper to hand to whoever tunes your KV-cache settings, with the benchmark scripts as a weekend's check on your own model; production FP4 KV should wait until vLLM, SGLang or TensorRT-LLM carry these ideas upstream.
Where it earns its place
An inference engineer at a GPU cloud running Qwen3 on B300 nodes
Find out whether FP4 KV cache can fit more long-context users per GPU
Run bench_e2e_throughput.py against BF16, FP8 and NVFP4 at 1 to 64 concurrent requests with your own model, then run the accuracy suite before trusting the gain. The code is free; the cost is a B300 for a few days.
A university group working on KV-cache quantisation
Compare a new method against a strong FP4 baseline
Use the evaluation harness, which covers seven benchmarks at 128K context, and the kernel benchmark against TRT-LLM NVFP4, citing arXiv 2611.09825.
Before you commit
- Setup
- Clone with submodules and run tools/bootstrap_env.sh, which needs uv and installs torch 2.11.0 (CUDA 13.0 wheels), a vendored vLLM and the kernel package into _local/envs/bd_fp4.
- Integrations
- Registers itself as vLLM's CUSTOM attention backend through a vLLM plug-in entry point; built on NVIDIA CUTLASS DSL, quack-kernels and FlashAttention-derived helpers.
- Data and privacy
- Runs on your own GPU; nothing is hosted. Evaluation datasets and models download from their usual hubs.
- Cost at scale
- No hosted or paid edition. The cost is Blackwell GPU time: one B200 or B300 for the 64K benchmarks, two for 128K.
- Limits
- SM100/SM103 only; tested on Qwen3-4B, Qwen3-30B-A3B, GLM-4.7 and Hy4-preview; key smoothing needs per-model calibration; no releases, five forks and no open issues at the time of writing.
What’s good
- Paper and code released together, MIT for original code
- Scripts reproduce the paper's kernel, throughput and accuracy tables
- Covers dense and MoE models, with GQA and MLA attention
- Plugs into vLLM as a CUSTOM attention backend
What’s not
- B200 or B300 (SM100/SM103) only
- All speed and accuracy figures are the authors' own
- Pinned torch and a vendored vLLM build, no tagged release
- One helper under NVIDIA's CUTLASS DSL licence, not MIT
Best for
Inference researchers and serving engineers with Blackwell GPUs who want to test 4-bit KV cache on long-context workloads.
BitDecoding-FP4 alternatives
BitDecoding-FP4: common questions
- Is BitDecoding-FP4 worth it?
- A careful paper with runnable code behind it, but until someone outside the lab reproduces it on more than two Qwen models, treat the 3x throughput as a lead to test, not a setting to ship. We rate it 3.4 out of 5.
- How much does BitDecoding-FP4 cost?
- Free under the MIT licence for original code; some kernel helpers fall under BSD 3-Clause and NVIDIA's CUTLASS DSL licence. You need a B200 or B300 GPU.
- What are the downsides of BitDecoding-FP4?
- B200 or B300 (SM100/SM103) only; all speed and accuracy figures are the authors' own; pinned torch and a vendored vLLM build, no tagged release; one helper under NVIDIA's CUTLASS DSL licence, not MIT.
- Who is BitDecoding-FP4 best for?
- Inference researchers and serving engineers with Blackwell GPUs who want to test 4-bit KV cache on long-context workloads.
- How much does BitDecoding-FP4 cost for a team or in production?
- No hosted or paid edition. The cost is Blackwell GPU time: one B200 or B300 for the 64K benchmarks, two for 128K.
- Is BitDecoding-FP4 safe for company data?
- Runs on your own GPU; nothing is hosted. Evaluation datasets and models download from their usual hubs.
- What are the best BitDecoding-FP4 alternatives?
- The closest alternatives we have reviewed are vLLM, SGLang, TensorRT-LLM and TileKernels.
Answers checked Nov 18, 2026.