Show HN: Swiftlet — Running 80B Qwen Models on Mac (4.3GB RAM) and 35B on iPhone

Swiftlet is a specialized runtime developed using Swift and Metal, designed specifically for the Qwen3-Next and Qwen3.5/3.6 MoE (Mixture-of-Experts) hybrid model architectures.

Instead of loading the entire model into RAM, Swiftlet keeps only the essential dense core resident in memory. It dynamically streams the routed MoE weights from the SSD as they are needed.

🚀 Performance Benchmarks

The following metrics were captured on an M5 Mac:

Model (4-bit)	Disk Space	Peak RAM	Decode Speed
Qwen3.6-35B-A3B	18 GB	2.6 GB	7 – 11 tok/s
Qwen3-Next-80B-A3B	42 GB	4.3 GB	4.5 – 5 tok/s

[!NOTE] Mobile Performance: The 35B model is currently functional on an iPhone 17, utilizing approximately 2.5 GB of RAM and generating text at roughly 1 tok/s.

---

Context & Vision

It is important to acknowledge that ANEMLL previously demonstrated a proof-of-concept in early 2026, streaming a massive 397B MoE on an iPhone 17 Pro. Swiftlet aims to evolve this concept into a practical, installable application for base-model iPhones via an open-source runtime.

Currently, development is focused on optimizing kernel speed. Since the decode loop is limited by dispatch rather than I/O, there is significant room for improvement.

A Realistic Expectation: Because only $\approx 3\text{B}$ parameters are active per token, these models exhibit the conversational fluidity of large models but the factual recall capabilities of smaller ones.

---

🛠️ Quick Start Guide (macOS)

Follow these steps to get Swiftlet running on your Mac:

• Clone and Build

  git clone https://github.com/leonickson1/Swiftlet.git
  cd Swiftlet
  swift build -c release
  

• Prepare the Model (Download from Hugging Face via the repacker)

  # For the 35B model
  .build/release/swiftlet-repack \
    --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \
    --output ~/models/qwen3.6-35b.qpack
  
  # For the 80B model (42GB disk, ~4.3GB RAM)
  .build/release/swiftlet-repack \
    --from-hf Leonickson/Qwen3-Next-80B-A3B-qpack \
    --output ~/models/qwen3-next-80b.qpack
  

• Run Inference

  # Interactive Chat
  .build/release/swiftlet chat ~/models/qwen3.6-35b.qpack "Who wrote One Hundred Years of Solitude?"
  
  # One-shot generation with GPU acceleration
  .build/release/swiftlet generate ~/models/qwen3.6-35b.qpack --gpu --chat --prompt "Explain expert streaming in one paragraph."
  

• Launch API Server (OpenAI-compatible, loopback only)

  .build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080
  

Note: swiftlet-repack also supports converting raw MLX checkpoints using --from-hf mlx-community/...

System Requirements:

• Hardware: Apple Silicon
• OS: macOS 14+ or iOS 17+
• Storage: Free SSD space (18 GB for 35B / 42 GB for 80B)

---

📱 Mobile Experience

You can experience the 35B model on iPhone via Priv AI on the App Store:

1. Open Settings.
2. Navigate to Experimental Models.
3. Download the model.

This implementation is fully on-device; it streams from storage without any external server. If the feature isn't visible yet, it may still be in App Store review.

For Developers: You can build the app from source. The UI is available at leonickson1/localLLM. Clone that repository and place this repo next to it as swiftlet, then run the Xcode project on your device.

---

⚙️ How It Works

The efficiency of Swiftlet stems from the fact that these models only activate a small fraction of their total weights per token.

The Routing Logic

For every token, the model routes to a subset of experts:

• 80B Model: 10 out of 512 experts.
• 35B Model: 8 out of 256 experts.

Memory Management

Swiftlet splits the model into two categories:

1. Resident Weights: Attention, DeltaNet projections, routers, shared experts, and embeddings are kept in RAM.
   • 35B: $\approx 1.3\text{ GB}$ (4-bit)
   • 80B: $\approx 2.5\text{ GB}$ (4-bit)
2. Streamed Weights: Routed experts are stored in a .qpack container.

Key Technical Innovations:

• .qpack Containers: Experts are stored as fixed-stride blobs. This allows the system to fetch an expert using a single pread call, avoiding mmap overhead and page-cache thrashing.
• Smart Caching: A bounded pool uses a combination of LFU (Least Frequently Used) and recency eviction. Because Apple's SSDs are so fast, the throughput remains stable even with hit rates between 43% and 70%.
• Metal Integration: Uses runtime-compiled shaders, removing the need for a Metal toolchain during the build process.
• Linear Attention: $\approx 75\%$ of layers utilize Gated DeltaNet, which uses a fixed-size recurrent state. This means there is no expanding KV cache, regardless of the context length.

---

🛠️ Integration Options

Swiftlet is designed as a library first, offering four primary ways to be utilized:

1. Swift Package: Integrate SwiftletCore into any iOS/macOS app. Use SwiftletSession for features like streaming deltas, conversation caching, and memory-pressure management.
2. CLI Tools: Use swiftlet chat and swiftlet generate for benchmarking and local testing, or swiftlet-repack for model conversion.
3. OpenAI Server: swiftlet-server provides a local endpoint compatible with the OpenAI chat-completions API.
4. Priv AI: A production-ready iOS app that embeds SwiftletCore as its engine.

---

✅ Correctness & Validation

To ensure reliability, every stage of the forward pass—including Gated DeltaNet recurrence, gated GQA attention, and sparse MoE routing—has been rigorously tested:

• Reference Checks: Validated against mlx-lm implementations using per-layer fixtures in both f32 and int4.
• Decoding: Incremental decoding is verified to match whole-sequence processing.
• Hardware: Metal kernels are cross-referenced against a scalar CPU reference to ensure mathematical parity.