Apple ML Research: Deploying Muse Spark Open via MLX on Mac Studio

Local and Distributed LLM Inference using MLX and RDMA

Apple Machine Learning Research is showcasing how AI developers and researchers can utilize MLX on Mac with Apple silicon to experiment with cutting-edge models within a secure, private local environment.

By utilizing an open-source array framework that maximizes the unified memory architecture of Apple silicon, users can implement new fine-tuning and inference methodologies directly on their own hardware. While previous updates focused on the M5 architecture's GPU Neural Accelerators, this update focuses on the deployment of Muse Spark Open—a frontier-class model boasting nearly a trillion parameters.

Scaling Performance

Using the latest MLX release, Muse Spark Open can be run on a cluster of four Mac Studios, achieving a generation speed of over 36 tokens per second (at a context size of 8,192 tokens), which exceeds the average human reading pace.

The scaling path for these models is as follows:

Apple tested three specific parallelization strategies:

• 2 M5 Pipeline: Pipeline parallelization across two Mac Studios.
• 2 M5 Tensor: Tensor parallelization across two Mac Studios.
• 4 M5 Tensor: Tensor parallelization across four Mac Studios.

---

Benchmarking Large Language Models (LLMs)

Using the mlx_lm package, Apple compared the performance of the previous generation M3 Ultra and the new M5 Ultra. The M5's primary advantage is the GPU Neural Accelerator, which provides hardware-level support for matrix multiplication—a critical component for ML workloads.

Model Specifications

The research evaluated three distinct frontier models:

Note: Muse Spark Open utilizes a Mixture of Experts (MoE) architecture. This means only a small fraction of its parameters are active during inference, significantly lowering the required compute.

Evaluation Methodology

The team focused on three primary metrics to determine efficiency:

• TTFT: Time to First Token (measured in seconds).
• Generation Throughput: Speed of output (tokens/second).
• Peak Power: Maximum energy draw (watts).

All tests were conducted using the mlx_lm.benchmark tool. For instance, to test Qwen 3.8 with a context of 8,192 tokens and a generation of 128 tokens, the following command was used:

mlx_lm.benchmark --model Qwen/Qwen3.8-27B -p 8192 -g 128 -b 1

Because many local LLM users deploy agentic coding applications that require massive context windows, performance was measured at both $8,192$ and $65,536$ tokens.

---

Single Mac Studio Performance Analysis

For models like Qwen 3.8 (27B) and Mistral Medium 3.5 (128B), a single Mac Studio with 512GB of RAM is sufficient. Even at the largest context size (65,536 tokens), these models only occupy roughly 68GB and 158GB of RAM, respectively.

1. Compute-Bound Performance (TTFT)

The Time to First Token (TTFT) is primarily limited by compute power. This is where the M5 Ultra's Neural Accelerators provide a massive leap. Compared to the M3 Ultra, the M5 Ultra is up to 4.6$\times$ faster in TTFT, making it the ideal desktop for high-context tasks.

2. Memory-Bound Performance (Decode Speed)

Generation speed (decoding) is limited by memory bandwidth. The M5 Ultra increases this bandwidth significantly:

• M3 Ultra: $819 \text{ GB/s}$
• M5 Ultra: $1.2 \text{ TB/s}$

This improvement results in a 1.6$\times$ increase in tokens per second.

3. Energy Efficiency

The Mac Studio remains remarkably power-efficient compared to enterprise hardware:

• M3 Ultra Peak Power: $\approx 200\text{W}$
• M5 Ultra Peak Power: $\approx 300\text{W}$

As the researchers noted, this power draw is roughly equivalent to a common household toaster, allowing these powerful machines to be deployed in standard office environments without specialized cooling or power infrastructure.