Mesh LLM: Decentralizing AI Compute via iroh

Published July 11, 2026, by Rae McKelvey

When most people imagine the infrastructure required to run a Large Language Model (LLM), they envision a massive, centralized data center. This traditional model relies on proprietary hardware racks of GPUs owned by a few giants, metered API access, and a monthly bill that scales linearly with your success.

You essentially send your data into a "black box," praying that the pricing, the model version, and the privacy terms don't shift overnight. You surrender control over your hardware, your data's destination, and the timing of model updates.

As your demand grows, you have only one lever to pull: pay more.

A New Paradigm: The Compute Mesh

Mesh LLM changes this by pooling the GPU and memory resources you already possess across any number of machines. It aggregates this distributed hardware and presents it as a single, unified, OpenAI-compatible API.

Instead of worrying about infrastructure, you let the mesh determine the most efficient execution path:

• Local: Run the model on your own machine's GPU.
• Peer-to-Peer: Route the request to another node in the mesh.
• Distributed: Split the model across multiple machines.

Whether you want to share compute privately with a small team or open it up to the global community to power chat interfaces and AI agents, the setup is seamless. Simply point your OpenAI client to http://localhost:9337/v1 and the mesh handles the "where" and "how" of the computation.

Technical Architecture & "Skippy"

The system utilizes a plugin-based architecture where plugins define their capabilities in a manifest. The runtime then manages these plugins, routing calls and exposing functionality via HTTP, inference, mesh events, and the Model Context Protocol (MCP).

The available catalog is extensive, featuring over 40 models. These range from lightweight versions ($\approx 0.5 \times 10^9$ parameters) that fit on a standard laptop to massive Mixture-of-Experts (MoE) models with $235 \times 10^9$ parameters.

Distributed Inference (Split Mode)

For models too large for a single device, Mesh LLM employs a split-mode system internally known as Skippy. This partitions the model by layer ranges into distinct stages.

For example, a model's layers $L$ are divided such that: $\text{Node}_1: [0, 15], \text{Node}_2: [16, 31], \dots, \text{Node}_n: [L_{start}, L_{end}]$

Activations flow sequentially through this pipeline, allowing a cluster of modest machines to execute a model that none could host individually.

Powered by iroh

Every participant in the mesh—whether a compute provider or a simple requester—operates an iroh endpoint. This endpoint serves as the node's unique identity (a public key) and its sole network interface.

iroh simplifies the complex networking required for P2P, handling:

1. NAT Traversal
2. Hole-punching
3. Relay Fallbacks (Mesh LLM maintains two regional relays to ensure connectivity when direct paths fail).

Network Protocol Specifications

The system utilizes three distinct ALPNs to categorize traffic:

Within the mesh-llm/1 connection, bidirectional QUIC streams are demultiplexed using a single leading byte:

By leveraging iroh, the act of streaming activations to a remote pipeline stage becomes as simple as talking to localhost, just with a different endpoint ID. Mesh LLM then layers a custom gossip protocol on top to manage trust, versioning, and admission.

Getting Started

The software is remarkably lightweight ($\approx 18\text{ MB}$). Users can choose between the public mesh or a private deployment.

Deployment Checklist:

• Install Mesh LLM binary.
• Join public mesh or configure private keys.
• Point OpenAI client to localhost:9337/v1.
• (Future) Integrate via ACP (Agent Communication Protocol).

# Example conceptual command to start a node
mesh-llm start --join public-mesh --gpu-enabled

The ultimate goal is a future with more peer-to-peer interaction, fewer closed-source servers, and zero vendor lock-in.

---

About Iroh

Iroh is an open-source networking library designed to make "dialing any device" a reality. It provides a clean abstraction over complex pipes, allowing developers to build custom protocols or use ready-made ones. It is currently powering hundreds of thousands of devices in production.

• Documentation: [Visit Docs]
• Source Code: [GitHub]
• Community: [Discord]