Optimizing Automatic1111 for Apple Metal: Achieving a 40% Boost for SD1.5

By Derek Anderson

I frequently utilize Draw Things on my Apple hardware, but I've always felt a lingering frustration with Automatic1111: it simply feels sluggish. Given that I have a deeply integrated workflow built around the A1111 ecosystem—which is incredibly mature—I wanted to bridge the performance gap.

On my M3 Pro, a standard five-step generation using DPM++ SDE typically took between 8 and 10 seconds. Draw Things proved that Stable Diffusion on Apple Silicon could be significantly more responsive, leading me to wonder: How much of this latency is actually mandatory?

I considered replacing Automatic1111 entirely. Instead, I focused on a specific goal:

🎯 Project Objectives

• Maintain the original WebUI.
• Preserve compatibility with existing checkpoints and LoRAs.
• Keep all samplers, extensions, and API functionality.
• Retain standard prompt syntax and general workflow.
• Avoid converting everything to Core ML or building a new engine.

The mission was narrow: Make Automatic1111 behave like native Apple software in the areas that matter most.

---

The Results: Real-World Performance

Rather than relying on synthetic benchmarks, I measured the specific workloads I actually use. The results were substantial:

Note: These are observed ranges from actual workloads, not a controlled benchmark claiming a universal 2x increase. Additionally, these runtime gains are distinct from NGMS, which reduces the actual amount of guidance computation.

---

Strategy: Workload Over Benchmarks

I focused on a very specific configuration to ensure the optimizations were meaningful:

• Model: Stable Diffusion 1.x
• Sampler: DPM++ SDE Karras (5 steps)
• CFG Scale: $\approx 1.15$
• Resolutions: $384 \times 640$ and $512 \times 512$
• Precision: FP16 UNet on MPS / FP32 VAE (default)

Early on, I tried DPM++ 2M because it was faster, but it altered the visual output too much for my short-step schedule. This established my golden rule: If an optimization increases speed but degrades the specific result I'm targeting, it is discarded.

---

Technical Deep Dive

1. Selective Metal Flash Attention

The PyTorch MPS backend has improved, but certain attention shapes are still more efficient when handled directly by Metal. However, a "one size fits all" approach is a mistake. I implemented a router that only triggers Metal Flash Attention when the specific SD 1.x shapes are detected.

The logic follows this conceptual path: if inference and fp16_mps and query_tokens == 192 and head_dim in (40, 80, 160): return metal_flash_attention(q, k, v)

2. Solving the Command Buffer Bottleneck

Improving the kernel wasn't enough. I discovered that the native extension was committing the MPS command buffer after every single attention call. Since the UNet calls attention repeatedly, these tiny submissions created massive overhead.

The Solution: I integrated the kernel into PyTorch's existing MPS stream. The extension now:

1. Ends PyTorch's current kernel coalescing.
2. Encodes the Metal Flash Attention operation into the current buffer.
3. Allows PyTorch to continue seamlessly.

"The fastest kernel still loses if you submit the command buffer after every call. You aren't just optimizing matrix multiplication; you're optimizing the coordination between Python, PyTorch, MPSGraph, and Metal."

Safety Check: To prevent "green images" (broken tensors), the fork now runs an isolated correctness test before enabling the Metal path.

3. Navigating Unified Memory

Apple Silicon uses a unified memory architecture where the GPU and CPU share the same pool. If the allocator thrashes or macOS begins swapping, performance plummets.

Instead of a static VRAM limit, I implemented a dynamic budget based on total and available memory.

The Math: The estimated peak memory is calculated as: $\text{estimated\_peak} = (\text{batch} \times \text{heads} \times \text{query\_tokens} \times \text{key\_tokens} \times \text{element\_size}) \times 2.5$

The budget is then determined by: $\text{budget} = \min(10\% \text{ of total memory}, 20\% \text{ of available memory}, 1.5\text{ GiB})$

if estimated_peak <= budget:
    run_native_sdpa()
else:
    run_memory_bounded_sub_quadratic_path()

This ensures that an 8 GB Mac doesn't attempt the same operations as a 32 GB Mac, and accounts for other open apps like Chrome or Xcode.

---

Resources & Links

You can find the fine-tuned version of Automatic1111 for Apple Silicon here:

GitHub Repository: dmikey/stable-diffusion-webui-metal