Maximizing Value: The Shift Toward Cost-Efficient Inference

By Ian Ye | July 3, 2026

The appetite for AI inference is expanding at a rate that far exceeds the available hardware supply. With a relentless cycle of frontier model releases—including Minimax M3, GLM5.2, and Claude Fable—the "token gold rush" is intensifying. Consequently, Blackwell GPUs are in short supply, driving up NVIDIA hardware costs and making token generation increasingly expensive.

However, there is a viable alternative. The AMD Instinct MI355X offers comparable hardware specifications while being, on average, $\approx 2.75\times$ cheaper per GPU than the B300. At Wafer, we have long argued that the secret to affordable inference is hiding in plain sight.

The Software Friction Gap

While the MI350 series competes with Blackwell at the silicon level, NVIDIA maintains a significant lead in software ecosystem and "day-0" support. This allows providers to deploy models on NVIDIA hardware with minimal friction. In contrast, achieving State-of-the-Art (SOTA) performance on the MI355X / ROCm stack is rarely a "plug-and-play" experience.

Finding a working image for the latest models is easy. In reality, you're lucky to find an image that even boots the model. This lack of immediate support often forces engineers into weeks of manual optimization, meaning AMD is perpetually playing catch-up.

The good news? As AI agents become more proficient at model and kernel optimization, this gap is evaporating.

---

Benchmarking the MI355X

We tested GLM5.2 on MI355X capacity provided by TensorWave. Using a workload of 20k input / 1k output tokens with a 60% cache hit rate, we achieved an aggregate throughput of 2626 tok/s/node at 2.4 rps, maintaining a Time to First Token (TTFT) $\le 5s$.

While this is roughly 80% of the performance of a B200 (which hit 3192 tok/s/node at 3.0 rps), it is over 2x more cost-effective.

Aggregate Throughput Metrics

Additionally, we recorded 213 tok/s on a single stream (10k in / 1.5k out) following Artificial Analysis standards. While not the absolute top of the leaderboard, it dominates in performance per dollar.

---

Engineering the Solution

Our optimization process followed a specific roadmap to eliminate bottlenecks:

1. Quantization and Framework Selection

We compared our MXFP4 quantization against the official FP8 baseline. The results showed that MXFP4 was effectively lossless.

Evaluation Comparison:

For the framework, we evaluated three candidates:

• vLLM: $\times$ (No working MXFP4 + GlmMoeDsa path).
• ATOM: $\times$ (Output quality degraded at long contexts).
• sglang: $\checkmark$ (Least friction, native support, maintained coherence).

2. Unlocking Speculative Decoding

Speculative decoding is key to throughput, but the sglang ROCm image required two critical fixes:

• Fix MTP Head Prefix Mismatch: The MTP head's shared expert was stored in bf16, but sglang's quantization lookup failed because the prefix was model.decoder.* instead of model.layers.78.mlp.shared_experts.*. This caused a shape mismatch crash.
• Remove CUDA Dependencies: Deep speculative decode (5/1/6 config) was blocked by a fused multi-step metadata kernel that explicitly called #include cuda_runtime.h without a ROCm guard.

// The offending line that needed removal/guarding for ROCm:
#include cuda_runtime.h 

By duplicating the layer 78 entries under the correct decoder name in the Quark un-quantized list, we unblocked speculative decode, resulting in a ~3x gain in single-stream throughput.

3. Prefill and MoE Tuning

To maximize aggregate throughput, we applied these configurations:

• --kv-cache-dtype fp8_e4m3
• --enable-aiter-allreduce-fusion

We also experimented with Tensor Parallelism (TP) and Data Parallelism (DP):

• TP8: 1461 tok/s/node (Optimized for single stream).
• TP4 $\times$ DP2: 1944 tok/s/node (Significant improvement for prefill-bound workloads).

The final jump to 2626 tok/s/node came from tuning the MoE kernel. The sglang image was using a slow FlyDSL fallback for fp4. We manually tuned the selection for the following shapes: $\text{model\_dim} = 6144, \text{moe\_inter} = 2048, E = 256, \text{topk} = 8$

---

Optimization Workflow Summary

Final Thoughts

"Achieving the best performance per dollar ratio on the MI355X wasn’t particularly hard."

Despite the framework bugs and the lack of day-0 support, we reached these numbers without writing a single custom kernel. The hardware is there; we just need the software to catch up.