What it is
TensorRT-LLM is NVIDIA's open-source library for running LLMs, and now diffusion models for image and video, as fast as possible on NVIDIA GPUs. It offers a Python LLM API, an OpenAI-compatible server (trtllm-serve), quantisation down to FP8 and NVIDIA's FP4 formats, in-flight batching, speculative decoding, disaggregated serving and multi-GPU and multi-node parallelism.
Version 1.2 changed how it works: the TensorRT engine backend was removed, PyTorch is now the only execution backend, and Hugging Face checkpoints load directly with no engine-build step. It targets NVIDIA hardware only, from Ampere to Blackwell and GB200 NVL72, on Linux.
The latest stable release, v1.2.1, dates from April 2026; the 1.3 line has shipped only as release candidates, and v1.4.0rc1 was tagged on 22 October 2026.
Our review
TensorRT-LLM is the engine to benchmark if your fleet is NVIDIA and cost per token is the metric that matters. It has about 14,900 GitHub stars, daily commits, and NVIDIA's own tech blog shows the pattern: DeepSeek-V4 optimisations for Blackwell, KV cache compression and NVL72-scale video generation land here first.
The old complaint about setup is now mostly history. Before 1.2, serving meant converting checkpoints and building model-specific engines; release 1.2 deleted that backend, so trtllm-serve takes a Hugging Face model name directly. What remains is dependency weight: the pip install wants CUDA 13, a matching PyTorch build and OpenMPI, and the docs warn that pip can quietly swap in the wrong torch build, which is why most teams use NVIDIA's NGC container.
Security needs more care than the marketing suggests. NVIDIA's May 2026 bulletin fixed several unsafe-deserialisation and race-condition flaws in the RPC and MPI layers in v1.2 and v1.2.1, and NVD published a second batch on 14 July 2026, including CVE-2026-24233 (8.4), a bypass of the restricted unpickler used for model weights, and CVE-2026-24229, an unauthenticated FastAPI endpoint in the disaggregated orchestrator. We could not find which release fixes the July set, which matters because the newest stable tag predates them.
Release rhythm is the other trade-off. Stable tags are rare while release candidates arrive weekly, 1.2 carried six breaking changes, and the tracker holds about 1,800 open issues and pull requests. vLLM and SGLang also run on AMD and TPU hardware, so the case for TensorRT-LLM rests on measured throughput on your own model and GPUs, not on reputation.
Where it earns its place
An inference lead serving a DeepSeek-class model on HGX B200 nodes
Lower cost per million tokens on hardware already bought
Run trtllm-serve from the NGC container with FP4 or FP8 checkpoints and disaggregated prefill and decode. The software is free; the saving only exists if a benchmark on your own traffic beats vLLM or SGLang by enough to cover the extra engineering.
A platform team at an enterprise with an NVIDIA AI Enterprise agreement
Run open models with a vendor support contract
Consume TensorRT-LLM inside NVIDIA's NIM containers, covered by AI Enterprise at a list price of $4,500 per GPU per year, or $1 per GPU-hour on cloud marketplaces, so an 8-GPU node lists at $36,000 a year.
A researcher on a single RTX or Ada workstation
Serve an 8B model locally with an OpenAI-style API
pip install tensorrt_llm on Ubuntu with CUDA 13, then trtllm-serve with an FP8 checkpoint such as nvidia/Qwen3-8B-FP8. It works, but vLLM or llama.cpp are simpler on one consumer card.
Before you commit
- Setup
- Pull the NGC release container, or pip install tensorrt_llm on Linux with CUDA 13, a CUDA 13 PyTorch build and libopenmpi-dev; trtllm-serve <hf_model> starts an OpenAI-compatible server on port 8000.
- Integrations
- OpenAI-compatible HTTP API; Python LLM API; NVIDIA Dynamo and Triton for orchestration; NIM containers; Slurm and Kubernetes deployment guides.
- Data and privacy
- Runs on your hardware with no hosted service. 2026 CVEs include remote code execution through unsafe deserialisation in RPC, MPI, ZeroMQ and weight loading, plus an unauthenticated orchestrator API, so keep internal ports off shared networks and load weights only from trusted sources. Apache 2.0, with some bundled third-party components under their own licences.
- Cost at scale
- The library is free; the cost is NVIDIA GPUs sized for weights and KV cache. Paid support comes through NVIDIA AI Enterprise at $4,500 per GPU per year list for self-managed systems, or $1 per GPU-hour on cloud marketplaces.
- Limits
- NVIDIA GPUs only, with FP8 on Hopper and Ada and not Ampere; about 1,800 open issues and pull requests; breaking changes in minor releases; the latest stable tag predates the July 2026 CVEs.
What's good
- First support for new NVIDIA hardware and formats such as FP4
- No engine-build step since 1.2; loads Hugging Face checkpoints
- OpenAI-compatible trtllm-serve
- Disaggregated serving and NVL72-scale parallelism
What's not
- NVIDIA GPUs and Linux only
- Two rounds of CVEs in 2026, several in deserialisation code
- Stable releases rare; 1.3 never left release candidate
- Heavy CUDA, PyTorch and MPI dependency stack
Best for
Teams running large NVIDIA GPU fleets who want maximum throughput per GPU.
Alternatives
TensorRT-LLM: common questions
- Is TensorRT-LLM worth it?
- TensorRT-LLM is where new NVIDIA silicon gets its first serious optimisations, and the price is a long CVE list and a stable release cadence measured in seasons. We rate it 4.0 out of 5.
- How much does TensorRT-LLM cost?
- Free and open source under the Apache 2.0 license (some bundled components carry their own licenses).
- What are the downsides of TensorRT-LLM?
- NVIDIA GPUs and Linux only; two rounds of CVEs in 2026, several in deserialisation code; stable releases rare; 1.3 never left release candidate; heavy CUDA, PyTorch and MPI dependency stack.
- Who is TensorRT-LLM best for?
- Teams running large NVIDIA GPU fleets who want maximum throughput per GPU.
- How much does TensorRT-LLM cost for a team or in production?
- The library is free; the cost is NVIDIA GPUs sized for weights and KV cache. Paid support comes through NVIDIA AI Enterprise at $4,500 per GPU per year list for self-managed systems, or $1 per GPU-hour on cloud marketplaces.
- Is TensorRT-LLM safe for company data?
- Runs on your hardware with no hosted service. 2026 CVEs include remote code execution through unsafe deserialisation in RPC, MPI, ZeroMQ and weight loading, plus an unauthenticated orchestrator API, so keep internal ports off shared networks and load weights only from trusted sources. Apache 2.0, with some bundled third-party components under their own licences.
- What are the best TensorRT-LLM alternatives?
- The closest alternatives we have reviewed are vLLM and SGLang.
Answers checked Oct 23, 2026.
Make TensorRT-LLM? Show this rating on your site
