Deploying Qwen 3.8 Flash Next (125B) on Consumer Gear via Strata

Strata is a high-performance inference engine designed to bring the massive 125-billion-parameter Qwen 3.8 Flash Next model to standard gaming PCs. It provides a local API compatible with OpenAI and Anthropic, supports image inputs, and is fully open-source.

🚀 Performance Benchmarks

Strata allows for incredibly fast interaction. While a high-end card like the RTX 3090 (24 GB) can reach speeds of $\approx 100\text{--}140 \text{ tokens/s}$ , the following table breaks down performance on other hardware:

Speed Metrics (4K Output / 32K Prompt)

Hardware Configuration	Model Quant	Write Speed ( $\text{t/s}$ )	Read Speed ( $\text{t/s}$ )
NVIDIA RTX 5070 (12GB)	`Q2_0`	94	2,650
(Ryzen 5 7600, 64GB RAM)	`IQ2_XS`	79	2,090
	`IQ3_XXS`	62	1,750
	`IQ3_S`	53	1,620
	`Coder`	55	2,180
AMD RX 9070 XT (16GB)	`Q2_0`	60	1,160
(Ryzen 9 3900X, 47GB RAM)	`IQ2_XS`	52	1,110
	`Coder`	44	1,420

Note: NVIDIA results for Q2_0 utilize engine version 0.1.36; others use 0.1.26.

🛠️ System Requirements

Hardware Checklist

GPU (NVIDIA): RTX 20, 30, 40, or 50 series.
GPU (AMD): RX 7900 XT/XTX, 7800 XT/7700 XT, 9060 XT, 9070/9070 XT, AI PRO R9700, or RX 6800/6900 series.
VRAM: Minimum 12 GB.
OS: Windows 10/11 or Linux (with updated drivers).

Unsupported Experimental Hardware

The community has tested Strata on older cards, though results may vary:

Tesla P40 / V100
GTX 10 series
Radeon VII / MI50
RX 6700 XT / 5500 XT

📥 Installation Guide

You can choose between an automated AI-driven setup or a manual installation.

Option 1: AI-Assisted Setup

If you use an AI coding tool (like Cursor, Claude Code, or GitHub Copilot), simply provide the following instruction:

Set up Strata on this PC for me: https://github.com/Niko1221/Strata 
Please follow the guidelines in docs/AI_SETUP.md within the repository.

The AI will evaluate your RAM, disk space, and GPU to select the optimal model and handle the installation.

Option 2: Manual Setup

Download: Clone the repository or download the ZIP.
Execute: Run SETUP.bat (Windows) or ./setup.sh (Linux).
Configure: The installer will prompt you for:
- Model selection and size.
- Context window size (memory for conversation history).
- Vision capabilities (image reading).
Wait: The first launch may cause system instability or lag for 1–3 minutes while 35–55 GB of data is loaded into RAM.

Maintenance: Use UPDATE.bat or ./update.sh to update the engine without a full restart.

🧠 Model Selection & RAM Mapping

Choosing the right version depends on your available system memory.

System RAM	Recommended Model	Notes
32 GB	`Coder`	Optimized for programming; fits 32GB.
48 GB	`IQ2_XS` or `Q2_0`	`Q2_0` is the fastest option here.
64 GB	`IQ2_XS` (Rec) / `IQ3_S`	`IQ3_S` is higher quality but slower.
96 GB+	`IQ3_S` or `UD-IQ4_XS`	Maximum quality; room for other apps.

Special Model Variants

Coder: A streamlined version with half the experts removed. It retains 91% of the full model's SWE-bench Verified score but is less capable in general conversation and CJK (Chinese/Japanese/Korean) languages.
Swift 1.5: A fine-tuned version designed for faster "thinking" and quicker responses.
Unsloth UD-IQ4_XS: A $\approx 4$ -bit version balancing quality between IQ3_S and UD-Q4_K_XL.
Unsloth UD-Q4_K_XL: The highest quality (closest to full model). However, on 64GB systems, it relies heavily on SSD offloading, resulting in slow speeds ( $\approx 7\text{--}8.5 \text{ t/s}$ ).

[!IMPORTANT] If you have less than 80 GB of RAM, Strata will offload parts of the model to your SSD. Using an NVMe SSD is highly recommended to minimize the performance hit.

🖥️ Usage

Once installed, access the interface via your web browser: http://127.0.0.1:8080

Features include:

Chat Interface: Standard LLM interaction.
Live Monitor: Real-time tracking of GPU/CPU usage and model performance.
Agent Integration: Compatible with external coding agents.

If this tool helps you, consider supporting the developer:

Buy Me A Coffee