WebLLM: A High-Performance Inference Engine for the Browser

🌐 Overview

WebLLM is a cutting-edge inference engine designed to execute Large Language Models (LLMs) directly within a web browser. By utilizing hardware acceleration, it eliminates the need for server-side processing.

Everything is handled locally via WebGPU, ensuring that the computation happens on the user's machine. This architecture allows developers to implement an OpenAI-compatible API for various open-source models, supporting advanced features like:

• Real-time streaming
• JSON-mode for structured data
• Function-calling (currently Work-in-Progress)

WebLLM serves as a companion to MLC LLM, a project dedicated to the universal deployment of language models across diverse hardware environments.

By bringing GPU acceleration to the browser, WebLLM enables the creation of private, high-performance AI assistants.

System Architecture Flow

---

🚀 Key Features

WebLLM provides a robust suite of capabilities for modern web AI development:

• Local Browser Inference: Leverages WebGPU to run powerful models without relying on external servers.
• OpenAI API Parity: Easy integration using familiar OpenAI patterns, including seeding, logit-level control, and streaming.
• Optimized JSON Generation: Uses a high-performance WebAssembly (Wasm) implementation for structured output.
  • Try it out: Check the WebLLM JSON Playground on HuggingFace.
• Broad Model Compatibility: Native support for industry-leading models (see the table below).
• Flexible Model Integration: Users can deploy their own custom models provided they are in the MLC format.
• Developer-Friendly Setup: Modular design compatible with NPM, Yarn, or CDN for rapid prototyping.
• Asynchronous Execution: Supports Web Workers and Service Workers to prevent UI freezing by offloading heavy math to background threads.
• Browser Extension Ready: Includes blueprints for building Chrome extensions.

---

📚 Supported Models

WebLLM supports a curated subset of models available via MLC Models. You can find the full list in the prebuiltAppConfig.model_list.

Note: If a specific model is missing, you can request it via a GitHub issue or compile your own.

---

🛠️ Getting Started

Quick Start Checklist

• Install a WebGPU-compatible browser (e.g., Chrome).
• Install the @mlc-ai/web-llm package.
• Initialize the MLCEngine.
• Generate your first completion.

Installation

You can add WebLLM to your project using your preferred package manager:

# Using npm
npm install @mlc-ai/web-llm

# Using yarn
yarn add @mlc-ai/web-llm

# Using pnpm
pnpm install @mlc-ai/web-llm

Integration

1. Module Import

Depending on your needs, you can import the entire library or specific functions:

// Import the full library
import * as webllm from "@mlc-ai/web-llm";

// Import only the engine factory
import { CreateMLCEngine } from "@mlc-ai/web-llm";

2. CDN Usage

For environments like JSFiddle or CodePen, use the ESM build:

import * as webllm from "https://esm.run/@mlc-ai/web-llm";

// Or via dynamic import
const webllm = await import("https://esm.run/@mlc-ai/web-llm");

Using the Engine

The primary gateway for all operations is the MLCEngine. To get started, use the CreateMLCEngine() factory function to instantiate the engine and load your chosen model.

In terms of computational complexity, the inference process generally scales with the sequence length $n$, often represented as: $\text{Complexity} \approx O(n^2) \text{ for attention, } O(n) \text{ for generation}$

💡 Examples

To see WebLLM in action, explore these resources:

1. Basic Chatbot: A simple implementation for generating chat completions.
2. WebLLM Chat: A comprehensive, advanced project demonstrating full capabilities.
3. Examples Folder: Visit the /examples directory in the GitHub repository for more use cases.