Token Freedom via APU: AMD Ryzen AI MAX+ 395 Runs DeepSeek V4 Flash at 32 tok/s on a 284B Model

Developer Sandro posted on X a remarkable achievement: running DeepSeek V4 Flash on an AMD Ryzen AI MAX+ 395 (Strix Halo APU) at 32 tok/s. A 284-billion-parameter large model, no discrete GPU required — the CPU and integrated GPU share 128GB of unified memory, and it just works. This is a significant milestone for local LLM inference on consumer hardware.

💡 What You Will Learn

Developer Sandro posted on X a remarkable achievement: running DeepSeek V4 Flash on an AMD Ryzen AI MAX+ 395 (Strix Halo APU) at 32 tok/s. A 284-billion-parameter large model, no discrete GPU required

DeepSeek V4 Flash is a MoE model released by DeepSeek in late April this year. With 284B total parameters but only 13B activated per inference step, it uses 256 routed experts plus 1 shared expert, activating 6 per layer with a 1M-token context window. Its benchmarks rival models three times its size.

Lucebox's Optimization Trio

First, fitting the model: Lucebox used their proprietary ROCmFPX quantization to compress the model to 102.3GB — roughly 2.88 bits per parameter. This isn't a fixed quantization format but a family: routing expert gates and up matrices use ROCmFP2 (2.5 bits/param), expert down projections use ROCmFP3, and sensitive layers retain ROCmFP4 or higher precision. Add 11.3GB for the DSpark draft model, and the whole stack fits in Strix Halo's 128GB unified memory with ~14GB left for the OS and runtime.

For inference acceleration, Lucebox wrote DeepSeek-specific HIP decode paths handling hyperconnections, sparse attention, and routing computation. Pure autoregressive mode already reaches 25.31 tok/s. Adding DSpark speculative decoding: a small 3-layer draft model quickly predicts the next few tokens, while the 284B target model validates 4 positions (including the current seed) in one pass. Correct proposals commit immediately; the first error is corrected by the target model. This q=4 mode pushes decoding to 32.0 tok/s — 26.4% faster than autoregressive.

Kernel-level optimization: in the q=4 validation path, each packed dense weight is decoded only once and applied to all 4 validation columns simultaneously, rather than decoding four times. This contributed 2.1-2.3% performance gain alone. The Geometric team also optimized the HIP routing path's TOP-K and ARGSORT kernels, achieving 3.5-7x speedup in microbenchmarks.

Prefill optimization: using learned indexers for sparse prefill, limiting compressed history attention range. 2K prompts reach 245 tok/s, 8K scenarios hit 246.8-255.9 tok/s, and 24K long texts manage 221.9 tok/s. By comparison, token-by-token exact prefill is only 22.5-23 tok/s — sparse mode is over 10x faster.

Four Old GPUs vs. One APU

TinyComputers.io's A.C. Jokela compared against four NVIDIA Tesla P40s — 48 TFLOPS combined, but unable to load the model. The P40's Pascal architecture lacks native FP8 and FP4 support; PyTorch 2.10+ CUDA kernel compilation targets compute capability 7.5 minimum, while P40 is 6.1. Even if forced to compile, Pascal has no FP8 tensor cores — any FP8 operation would be software-emulated, destroying performance. Combined with 96GB VRAM spread across PCIe 3.0, the PCIe bottleneck from CPU offloading would cripple inference.

In contrast, Strix Halo's RDNA 3.5 architecture natively supports FP8, with CPU and GPU sharing a unified LPDDR5X memory pool — no PCIe middleman. Despite only 40 compute units (Radeon 8060S) providing a quarter of the P40's raw compute, it actually runs while the P40s cannot.

Compared to the previous Strix Halo record (HipFire's 18.99 tok/s), Lucebox's optimization is 68.5% faster — over 2x the DwarfStar record (15.6 tok/s).

Asymmetric Dual-GPU: Even Faster

Lucebox also developed a more aggressive asymmetric setup: a Radeon AI PRO R9700 (32GB GDDR6) paired with Strix Halo (128GB unified memory). The R9700 handles latency-sensitive dense compute paths and popular experts, while Strix Halo provides large-capacity storage for cold experts. Each MoE layer submits routing to both cards simultaneously, merging results on the target device — both cards work in parallel, not serial.

This configuration achieves 50.65 tok/s median on DeepSeek V4 Flash, with short-test peaks of 55 tok/s. The full system costs $6,499 — 31% cheaper than two DGX Sparks ($9,398) — while approaching dual DGX Spark performance. A single DGX Spark averages only 14 tok/s, making Lucebox's single-machine solution 3.6x faster with 2.6x better performance per dollar.

The key insight: leveraging MoE's natural property where 256 experts are independent and can be distributed across devices. Lucebox keeps hot experts on the R9700's fast local GDDR6 and cold experts in Strix Halo's large memory pool — both cards work simultaneously, never waiting on each other.