Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Kyojin

An ExLlamaV3 fork tuned to serve 100 GB-class open models such as GLM-5.3-Flash on one AMD Strix Halo box, an RTX card or a Mac Studio.

Reviewed by ReviewRamble editors · Updated

View on GitHub Open source

Share this page:XLinkedInFacebookRedditWhatsAppEmail

Our rating

3.9/5

Kyojin is the most carefully measured way to get GLM-5.3-Flash running fast on a 128 GB Strix Halo box, provided you are comfortable building ROCm kernels and living on a release that changes every few days.

What is Kyojin?

Kyojin is an inference engine from Yamz Labs built on turboderp's ExLlamaV3. It serves a short list of large open models through an OpenAI-compatible API, with a separate serve script per model family: GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.6-Flash and Hy4-flash on AMD Strix Halo (gfx1151, 128 GB), Qwen3.8-27B and two smaller packs on an NVIDIA RTX 3090, and DeepSeek V4.1 Flash on a Mac Studio through the companion kyojin-mac project, which uses MLX.

Most of the work is AMD kernel tuning, speculative decoding (on by default, returning the same tokens as plain decoding) and long-context handling. The project publishes its own quantised EXL3 packs on Hugging Face; the GLM-5.3-Flash pack is 100.1 GB.

The repository is MIT-licensed, written in Python with HIP and CUDA kernels, and has about 245 GitHub stars. It was created in October 2026 and had shipped 28 releases by 20 November 2026, when v1.9.1 came out.

Is Kyojin worth it?

Kyojin exists for one awkward machine. AMD's Strix Halo mini PCs have 128 GB of memory the GPU can use, enough for a 100 GB model, but general runners leave much of their speed on the table. Kyojin's README reports 32 tokens per second of decode and 633 of prefill for GLM-5.3-Flash at 8K context on one such box, and 53 and 1,576 for Qwen3.8-Flash-Next. On an RTX 3090 it reports 75 and 1,621 for Qwen3.8-27B at 5K, and 62 and 3,001 for DeepSeek V4.1 Flash on an M5 Ultra Mac Studio. These are the maintainers' figures, but the benchmark method is documented and a `tools/bench.sh` script lets you post your own.

The documentation is unusually exact about cost. The install guide says the first GLM launch spends about 10 minutes tuning kernels with the port closed, that GLM at a 131,072-token context leaves about 10 GiB free on a 128 GB machine, and that turning the IOMMU off gained 6% of prefill in their tests at the price of disabling the NPU. The known-limits list says the 763B DeepSeek V4.1 Flash run on one 128 GB machine, which streams experts from an NVMe drive, decodes at only about 5 to 11 tokens per second and is a demonstration, not a daily tool.

The servers bind to 127.0.0.1 by default, and we found no API key or authentication option in the serve scripts, so a proxy belongs in front before anything is exposed to a network. The engine also carries an optional run-time refusal hook that projects one direction out of the model's activations to make it refuse less. It is off unless you point it at a spec; the presets ship in a separate folder of each Hugging Face pack.

The catch: this is not a general model runner. Each model family has its own serve script and flags, the Strix Halo path needs Linux, ROCm 7 nightly wheels for gfx1151 and a roughly 12 GB SDK tree, and the newest Hy4-flash path handles one request at a time with greedy decoding only. Fifteen releases in October and 13 more by 20 November mean the commands in a guide can go stale within a week.

Our take: with a Strix Halo box and a wish to run GLM-5.3-Flash behind a coding agent, Kyojin is worth the build afternoon; anyone with a Mac or a mixed fleet will find ds4 or llama.cpp less work for a similar result.

Where it earns its place

  • A developer with a 128 GB Strix Halo mini PC

    Run a strong open model behind a coding agent instead of paying for API tokens

    Install the ROCm wheels, run `./build.sh`, download the 100.1 GB GLM-5.3-Flash pack and start `tools/glm/serve.py`; point the agent's OpenAI base URL at port 8000. The README reports about 32 tokens per second of decode at 8K.

  • A hobbyist with one RTX 3090

    Serve Qwen3.8-27B with a very long context for document work

    Build for CUDA and serve turboderp's 4.0 bpw pack (15 GiB) with a 245,760-token context and 6-bit K and V cache, per the README's command. Free beyond the card and power.

  • A hardware reviewer comparing mini PCs

    Get comparable LLM speed numbers across machines

    Run `tools/bench.sh` against a running server and submit the Markdown block through the project's benchmark report template, which records the machine, version and context depth.

Before you commit

Setup
Strix Halo: create a Python 3.12 venv, install ROCm torch and `rocm-sdk-devel` from AMD's gfx1151 nightly index, run `./build.sh`, then the model's serve script; the first GLM launch tunes kernels for about 10 minutes. RTX and Mac have their own guides.
Integrations
OpenAI-compatible `/v1/chat/completions` and `/v1/models`; a llama.cpp-compatible Prometheus `/metrics` endpoint; Nix flake and container builds contributed by users.
Data and privacy
Runs entirely on your machine; servers bind to 127.0.0.1 by default. MIT licence for Kyojin's additions; ExLlamaV3 is MIT, and model weights keep their makers' licences.
Cost at scale
No paid tier. The cost is a 128 GB Strix Halo machine, an RTX card with sm_80 or newer, or a large Mac Studio, plus electricity; packs run from 15 GiB to 184.7 GB of disk.
Limits
Strix Halo needs Linux, gfx1151 with 128 GB and ROCm 7.0 or newer. Hy4-flash serves one request at a time, greedy only, 64K measured. The 763B DeepSeek run decodes at about 5 to 11 tokens per second.

What’s good

  • Published, reproducible speed figures per machine, model and version
  • OpenAI-compatible server on 127.0.0.1 with speculative decoding on by default
  • Install guide states memory headroom, tuning time and system settings plainly
  • Active project: 28 releases and 14 contributors in under two months

What’s not

  • Strix Halo build needs Linux, ROCm 7 nightly wheels and a long compile
  • One serve script per model family; not a general runner
  • No authentication option found in the servers
  • Release pace means guides and flags change weekly

Best for

Owners of a 128 GB AMD Strix Halo machine who want GLM-5.3-Flash or Qwen3.8-Flash-Next served locally at the best speed the hardware allows.

Kyojin alternatives

Kyojin: common questions

Is Kyojin worth it?
Kyojin is the most carefully measured way to get GLM-5.3-Flash running fast on a 128 GB Strix Halo box, provided you are comfortable building ROCm kernels and living on a release that changes every few days. We rate it 3.9 out of 5.
How much does Kyojin cost?
Free and open source (MIT). You supply the hardware; model weights carry their own base licences.
What are the downsides of Kyojin?
Strix Halo build needs Linux, ROCm 7 nightly wheels and a long compile; one serve script per model family; not a general runner; no authentication option found in the servers; release pace means guides and flags change weekly.
Who is Kyojin best for?
Owners of a 128 GB AMD Strix Halo machine who want GLM-5.3-Flash or Qwen3.8-Flash-Next served locally at the best speed the hardware allows.
How much does Kyojin cost for a team or in production?
No paid tier. The cost is a 128 GB Strix Halo machine, an RTX card with sm_80 or newer, or a large Mac Studio, plus electricity; packs run from 15 GiB to 184.7 GB of disk.
Is Kyojin safe for company data?
Runs entirely on your machine; servers bind to 127.0.0.1 by default. MIT licence for Kyojin's additions; ExLlamaV3 is MIT, and model weights keep their makers' licences.
What are the best Kyojin alternatives?
The closest alternatives we have reviewed are DwarfStar (ds4), llama.cpp, Strata and MLX.

Answers checked Nov 20, 2026.