What is Kyojin?
Kyojin is an inference engine from Yamz Labs built on turboderp's ExLlamaV3. It serves a short list of large open models through an OpenAI-compatible API, with a separate serve script per model family: GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.6-Flash and Hy4-flash on AMD Strix Halo (gfx1151, 128 GB), Qwen3.8-27B and two smaller packs on an NVIDIA RTX 3090, and DeepSeek V4.1 Flash on a Mac Studio through the companion kyojin-mac project, which uses MLX.
Most of the work is AMD kernel tuning, speculative decoding (on by default, returning the same tokens as plain decoding) and long-context handling. The project publishes its own quantised EXL3 packs on Hugging Face; the GLM-5.3-Flash pack is 100.1 GB.
The repository is MIT-licensed, written in Python with HIP and CUDA kernels, and has about 245 GitHub stars. It was created in October 2026 and had shipped 28 releases by 20 November 2026, when v1.9.1 came out.
Is Kyojin worth it?
Kyojin exists for one awkward machine. AMD's Strix Halo mini PCs have 128 GB of memory the GPU can use, enough for a 100 GB model, but general runners leave much of their speed on the table. Kyojin's README reports 32 tokens per second of decode and 633 of prefill for GLM-5.3-Flash at 8K context on one such box, and 53 and 1,576 for Qwen3.8-Flash-Next. On an RTX 3090 it reports 75 and 1,621 for Qwen3.8-27B at 5K, and 62 and 3,001 for DeepSeek V4.1 Flash on an M5 Ultra Mac Studio. These are the maintainers' figures, but the benchmark method is documented and a `tools/bench.sh` script lets you post your own.
The documentation is unusually exact about cost. The install guide says the first GLM launch spends about 10 minutes tuning kernels with the port closed, that GLM at a 131,072-token context leaves about 10 GiB free on a 128 GB machine, and that turning the IOMMU off gained 6% of prefill in their tests at the price of disabling the NPU. The known-limits list says the 763B DeepSeek V4.1 Flash run on one 128 GB machine, which streams experts from an NVMe drive, decodes at only about 5 to 11 tokens per second and is a demonstration, not a daily tool.
The servers bind to 127.0.0.1 by default, and we found no API key or authentication option in the serve scripts, so a proxy belongs in front before anything is exposed to a network. The engine also carries an optional run-time refusal hook that projects one direction out of the model's activations to make it refuse less. It is off unless you point it at a spec; the presets ship in a separate folder of each Hugging Face pack.
The catch: this is not a general model runner. Each model family has its own serve script and flags, the Strix Halo path needs Linux, ROCm 7 nightly wheels for gfx1151 and a roughly 12 GB SDK tree, and the newest Hy4-flash path handles one request at a time with greedy decoding only. Fifteen releases in October and 13 more by 20 November mean the commands in a guide can go stale within a week.
Our take: with a Strix Halo box and a wish to run GLM-5.3-Flash behind a coding agent, Kyojin is worth the build afternoon; anyone with a Mac or a mixed fleet will find ds4 or llama.cpp less work for a similar result.
Where it earns its place
A developer with a 128 GB Strix Halo mini PC
Run a strong open model behind a coding agent instead of paying for API tokens
Install the ROCm wheels, run `./build.sh`, download the 100.1 GB GLM-5.3-Flash pack and start `tools/glm/serve.py`; point the agent's OpenAI base URL at port 8000. The README reports about 32 tokens per second of decode at 8K.
A hobbyist with one RTX 3090
Serve Qwen3.8-27B with a very long context for document work
Build for CUDA and serve turboderp's 4.0 bpw pack (15 GiB) with a 245,760-token context and 6-bit K and V cache, per the README's command. Free beyond the card and power.
A hardware reviewer comparing mini PCs
Get comparable LLM speed numbers across machines
Run `tools/bench.sh` against a running server and submit the Markdown block through the project's benchmark report template, which records the machine, version and context depth.
Before you commit
- Setup
- Strix Halo: create a Python 3.12 venv, install ROCm torch and `rocm-sdk-devel` from AMD's gfx1151 nightly index, run `./build.sh`, then the model's serve script; the first GLM launch tunes kernels for about 10 minutes. RTX and Mac have their own guides.
- Integrations
- OpenAI-compatible `/v1/chat/completions` and `/v1/models`; a llama.cpp-compatible Prometheus `/metrics` endpoint; Nix flake and container builds contributed by users.
- Data and privacy
- Runs entirely on your machine; servers bind to 127.0.0.1 by default. MIT licence for Kyojin's additions; ExLlamaV3 is MIT, and model weights keep their makers' licences.
- Cost at scale
- No paid tier. The cost is a 128 GB Strix Halo machine, an RTX card with sm_80 or newer, or a large Mac Studio, plus electricity; packs run from 15 GiB to 184.7 GB of disk.
- Limits
- Strix Halo needs Linux, gfx1151 with 128 GB and ROCm 7.0 or newer. Hy4-flash serves one request at a time, greedy only, 64K measured. The 763B DeepSeek run decodes at about 5 to 11 tokens per second.
What’s good
- Published, reproducible speed figures per machine, model and version
- OpenAI-compatible server on 127.0.0.1 with speculative decoding on by default
- Install guide states memory headroom, tuning time and system settings plainly
- Active project: 28 releases and 14 contributors in under two months
What’s not
- Strix Halo build needs Linux, ROCm 7 nightly wheels and a long compile
- One serve script per model family; not a general runner
- No authentication option found in the servers
- Release pace means guides and flags change weekly
Best for
Owners of a 128 GB AMD Strix Halo machine who want GLM-5.3-Flash or Qwen3.8-Flash-Next served locally at the best speed the hardware allows.
Kyojin alternatives
Kyojin: common questions
- Is Kyojin worth it?
- Kyojin is the most carefully measured way to get GLM-5.3-Flash running fast on a 128 GB Strix Halo box, provided you are comfortable building ROCm kernels and living on a release that changes every few days. We rate it 3.9 out of 5.
- How much does Kyojin cost?
- Free and open source (MIT). You supply the hardware; model weights carry their own base licences.
- What are the downsides of Kyojin?
- Strix Halo build needs Linux, ROCm 7 nightly wheels and a long compile; one serve script per model family; not a general runner; no authentication option found in the servers; release pace means guides and flags change weekly.
- Who is Kyojin best for?
- Owners of a 128 GB AMD Strix Halo machine who want GLM-5.3-Flash or Qwen3.8-Flash-Next served locally at the best speed the hardware allows.
- How much does Kyojin cost for a team or in production?
- No paid tier. The cost is a 128 GB Strix Halo machine, an RTX card with sm_80 or newer, or a large Mac Studio, plus electricity; packs run from 15 GiB to 184.7 GB of disk.
- Is Kyojin safe for company data?
- Runs entirely on your machine; servers bind to 127.0.0.1 by default. MIT licence for Kyojin's additions; ExLlamaV3 is MIT, and model weights keep their makers' licences.
- What are the best Kyojin alternatives?
- The closest alternatives we have reviewed are DwarfStar (ds4), llama.cpp, Strata and MLX.
Answers checked Nov 20, 2026.