Slotstream: Running Massive LLMs on Memory-Constrained Macs

Slotstream is a specialized utility designed to execute the Qwen3.8-Flash-Next model—a behemoth with $125 \times 10^9$ parameters (Mixture-of-Experts)—on Apple Silicon Macs that lack the physical RAM to hold the entire model. By streaming experts directly from the SSD, it enables the use of a 104 GB (4-bit) model on machines with significantly less memory.

The project is written entirely in Swift, meaning there is no Python required for the runtime. It provides a seamless interface by mimicking the Ollama and OpenAI chat APIs, ensuring compatibility with your favorite existing frontend tools.

🚀 Performance Benchmarks

On a high-end M5 Pro with 48 GB RAM, the system achieves:

• Warm Decode Speed: $\approx 12$ tokens/second.
• Cold Start (TTFT): $\approx 3$ seconds.
• Peak Memory Usage: 32 GB (though this is auto-configurable).
• Disk Footprint: 104 GB.

Hardware Compatibility Matrix

To run this, a 512 GB SSD is the absolute realistic minimum. The following table outlines expected performance based on RAM tiers:

[!IMPORTANT] Only the 48 GB tier was measured on physical hardware. Other values are derived from the performance curve; note that lower-tier Macs often have slower SSDs, which may further impact speed.

🛠️ Installation & Setup

You can get started by following these steps:

• Install the binary
• Verify disk space
• Download model weights

Quick Install

Run the following command to install the prebuilt binary to ~/.slotstream/bin and update your PATH:

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

• To Upgrade: Run the same command again.
• To Uninstall: rm -rf ~/.slotstream

Manual Build

If you prefer to build from source (requires Command Line Tools, no Xcode needed):

git clone https://github.com/carloslfu/slotstream
cd slotstream
make build

Security & Verification

Releases are CI-built with signed provenance. You can verify the asset using: gh attestation verify slotstream-arm64.tar.gz --repo carloslfu/slotstream

📦 Managing the 104 GB Weights

The model consists of 24 files totaling 103.8 GB. You can trigger the download via serve, run, or explicitly using:

slotstream pull

Key Download Details:

• Safety: The tool checks free disk space and requires confirmation before starting.
• Resumability: Interruptions are safe; pull resumes from the last byte.
• Integrity: All files are validated against sha256 hashes embedded in the binary.
• Speed: Hugging Face limits transfers to $\approx 36\text{--}57$ MB/s. If your connection exceeds 400 Mbps, the bottleneck is the HF server, not your ISP.

To verify an existing installation: pull --verify (takes $\approx 8$ seconds).

💻 Usage Guide

Direct Interaction

For a quick test without starting a server:

slotstream run --prompt "Why is the sky blue?"

API Server Mode

To integrate with other apps, start the server on port 11434:

slotstream serve

Example API Call:

curl localhost:11434/api/chat -d '{
  "model": "qwen3.8-flash-next:4bit",
  "messages": [{"role": "user", "content": "hello"}]
}'

Compatibility:

• Supported: Open WebUI, Ollama CLI, OpenAI SDKs, Streaming, CORS, and standard sampling.
• Unsupported: Tools, images, JSON-schema, and logprobs (these will return a 400 Bad Request).

⚙️ Technical Nuances

Context & Latency

• Limit: The combined prompt and completion are capped at 32,768 tokens (adjustable via --max-context).
• Prefill: Long prompts are the primary bottleneck. An 8,000-token prompt takes $\approx 1$ minute on a 48 GB Mac and $\approx 3$ minutes on a 16 GB Mac.
• Caching: Follow-up turns only prefill new content. For example, the 8th turn in a chat takes $\approx 6.0$s compared to the $\approx 25.8$s it would take without prefix caching.
• Reproducibility: Use --no-prefix-cache if you need bit-identical results, as cached state can occasionally cause slight token variations.

Speculative Decoding (MTP)

On machines with extra RAM, you can enable Multi-Token Prediction via --mtp.

• How it works: A draft head predicts the next token, which is then verified in a single batch.
• Accuracy: $\approx 86\%$ first-draft accuracy.
• Cost: Requires a fixed 1.6 GB of RAM.
• Requirement: Needs a one-time conversion of the head tensors using Tools/mtp_convert.py.

Memory Architecture

Slotstream automatically calculates the optimal memory allocation. You can view your machine's specific plan with: slotstream memory plan

Example Memory Plan Output:

Device: 52 GB RAM $\rightarrow$ Target: 33.0 GB total process $\rightarrow$ Cache: $\sim 152$ of 512 experts per layer.

You can manually override these settings using --memory-gb N or --max-ram-percent P.