Skip to content
ReviewRamble
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Also inEspañolDeutschFrançais

Mesh LLM

Pools GPUs and memory across machines into one OpenAI-compatible local API, splitting models too big for one box by layer.

Reviewed by Sheela · Updated

View on GitHub Open source

Our rating

3.9/5

A serious attempt at turning a drawer of mismatched GPUs into one endpoint, still rough at the edges and honest that it is experimental.

What is Mesh LLM?

Mesh LLM is a Rust project that joins several computers into one inference pool and exposes it as a single OpenAI-compatible endpoint at localhost:9337/v1, with a web console on port 3131. Each node runs Skippy, the project's own runtime built on a patched llama.cpp. If one machine can hold a model it serves it alone; otherwise requests route to a peer that has the model, or large dense models are split into contiguous layer ranges across nodes.

Nodes talk over QUIC through iroh, with traffic encrypted end to end between peers. Meshes are either private, joined by invite token, or published through Nostr discovery so strangers can join. Release bundles cover macOS Metal, Linux and Windows with CUDA, ROCm, Vulkan and CPU flavours, and launchers wire the endpoint into Goose, Claude Code, OpenCode and Pi.

The repo is Apache-2.0, created in February 2026, with about 3,709 GitHub stars and 480 forks. The latest stable release is v0.78.1 (5 October 2026), with preview builds as recent as 30 October.

Is Mesh LLM worth it?

Mesh LLM's core idea is sound and narrower than its slogan: fit the model on one machine when you can, route to a peer that has it when you cannot, and only split a model across machines by layer when nothing else works. That ordering matters, because layer splits pay a network hop on every token, and the project's own docs treat splitting as the fallback rather than the headline.

Where it differs from Exo, the obvious comparison, is hardware breadth. Exo is strongest on Apple silicon with MLX and Thunderbolt RDMA; Mesh LLM ships one backend-neutral host with CUDA, ROCm, Vulkan, Metal and CPU runtimes, so an Nvidia desktop, an AMD box and a Mac can sit in the same mesh. Its own comparison page concedes that it has no RDMA transport and no native image generation. The agent launchers and the experimental model: "mesh" mode, which fans one prompt out to every model and arbitrates the answers in code, are extras Exo does not have.

The public mesh deserves caution. mesh-llm serve --auto joins the best published mesh it finds, and while traffic is encrypted in transit, the peer that serves your request reads your prompt. The docs say plainly that peer reputation is not proof of identity or model honesty, and the default trust policy is off. For anything private, run an unpublished mesh with owner allowlisting.

The catch is maturity. There are 211 open issues and pull requests, releases land weekly, and the README's Windows section lists three bugs that blocked the native path as of v0.76.0-rc8, with a manual workaround. Teams on Linux with Nvidia cards will have the easiest time; anyone expecting Ollama-level polish on Windows should wait a few releases.

Where it earns its place

  • A developer with a 24 GB gaming PC and a 64 GB Mac

    Run a model neither machine can hold alone

    Start a private mesh on one machine, join the other with the invite token, and load a layer-split GGUF package with --split. Free; the cost is that every token crosses the LAN, so expect it to be slower than a single machine that fits the model.

  • A five-person startup with three idle workstations

    Give everyone's coding agent one shared local endpoint

    Run mesh-llm serve on each box in an unpublished mesh with owner allowlisting, then point Claude Code, Goose or OpenCode at localhost:9337/v1 through the built-in launchers. No API bill, but someone owns updates on a project that ships weekly.

  • A hobbyist with spare GPU time

    Contribute compute to a public pool and use other people's models

    mesh-llm serve --auto joins a published mesh and starts serving. Only send prompts you would show a stranger, because the serving peer sees them.

Before you commit

Setup
One-line install script, Homebrew on Apple silicon, or Ubuntu and Arch packages; then mesh-llm setup and mesh-llm serve. Native Windows needed a prerelease and manual runtime steps as of v0.76.0-rc8, per the README.
Integrations
OpenAI-compatible /v1 API (chat completions, completions, Responses); launchers for Goose, Claude Code, OpenCode and Pi; an MCP server on the console port; plugins such as Flash-MoE and DwarfStar.
Data and privacy
Peer traffic is QUIC-encrypted and relays cannot read it, but the node that serves a request sees the prompt. Private meshes use invite tokens; owner allowlisting is opt-in because the default trust policy is off. Apache-2.0.
Cost at scale
No paid tier. Cost is your own hardware and power; layer splits trade speed for capacity, so a mesh is cheaper than a bigger GPU but slower than one machine that fits the model.
Limits
211 open issues and pull requests; weekly releases change behaviour; no RDMA or tensor parallelism; the model: "mesh" fan-out mode is marked experimental; --local-model-only needs absolute, non-symlinked GGUF paths.

What’s good

  • One OpenAI-compatible endpoint over many machines
  • CUDA, ROCm, Vulkan, Metal and CPU in one mesh
  • Serves locally first and splits by layer only when needed
  • Private meshes with invite tokens and owner allowlists

What’s not

  • Public mesh peers can read the prompts they serve
  • Default trust policy is off
  • Native Windows install needed workarounds in recent releases
  • Fast release pace and 211 open issues and PRs

Best for

Homelab owners and small teams with several GPU machines who want one local endpoint and are comfortable running experimental distributed software.

Mesh LLM alternatives

llama.cpp

Run models locallyC++132k stars

4.8/5

C/C++ engine for running LLMs locally on CPUs, Apple silicon and most GPUs, home of the GGUF model format.

Ollama

Run models locallyGo184k stars

4.6/5

One-command tool for downloading and running open LLMs locally, with a REST API and optional paid cloud models.

vLLM

Run models locallyPython96k stars

4.7/5

High-throughput LLM serving engine for GPUs, with an OpenAI-compatible server and wide model support.

Mesh LLM: common questions

Is Mesh LLM worth it?
A serious attempt at turning a drawer of mismatched GPUs into one endpoint, still rough at the edges and honest that it is experimental. We rate it 3.9 out of 5.
How much does Mesh LLM cost?
Free and open source (Apache-2.0). No paid edition; you supply the hardware.
What are the downsides of Mesh LLM?
Public mesh peers can read the prompts they serve; default trust policy is off; native Windows install needed workarounds in recent releases; fast release pace and 211 open issues and PRs.
Who is Mesh LLM best for?
Homelab owners and small teams with several GPU machines who want one local endpoint and are comfortable running experimental distributed software.
How much does Mesh LLM cost for a team or in production?
No paid tier. Cost is your own hardware and power; layer splits trade speed for capacity, so a mesh is cheaper than a bigger GPU but slower than one machine that fits the model.
Is Mesh LLM safe for company data?
Peer traffic is QUIC-encrypted and relays cannot read it, but the node that serves a request sees the prompt. Private meshes use invite tokens; owner allowlisting is opt-in because the default trust policy is off. Apache-2.0.
What are the best Mesh LLM alternatives?
The closest alternatives we have reviewed are llama.cpp, Ollama and vLLM.

Answers checked Oct 31, 2026.