Google Releases Gemma 4 QAT Quantized: 26B Model Runs on 15GB VRAM with Near-Zero Accuracy Loss
Google DeepMind has released full QAT (Quantization-Aware Training) versions of Gemma 4 on Hugging Face. The 26B MoE model needs only 15GB VRAM — down 72% from 50GB — with Unsloth's dynamic GGUF quantization pushing accuracy even higher. Even the E2B model fits in 2.19GB, making it phone-friendly.
💡 What You Will Learn
Google DeepMind has released full QAT (Quantization-Aware Training) versions of Gemma 4 on Hugging Face. The 26B MoE model needs only 15GB VRAM — down 72% from 50GB — with Unsloth's dynamic GGUF quant
Google Releases Gemma 4 QAT Quantized! 26B Model Needs Only 15GB RAM, Near-Zero Accuracy Loss
Today's news will make local deployment enthusiasts very happy.
Google DeepMind just released the full Gemma 4 QAT (Quantization-Aware Training) series on Hugging Face — from the tiny E2B to the 31B flagship, all optimized for quantization.
Even more exciting: the Unsloth team immediately produced GGUF versions, taking model quantization to new heights.
Let me break down what is in this release.
- 72% VRAM Reduction, Nearly Zero Accuracy Loss
First, the numbers.
Original Gemma 4 BF16: 31B needs 58GB VRAM, 26B MoE needs nearly 50GB. Most users' GPUs could not even load them.
After QAT, everything changes: E2B (2B) to just 3GB RAM — runs on a phone; E4B (4B) to just 5GB RAM — tablet territory; 12B to just 7GB RAM; 26B-A4B (MoE) to just 15GB RAM — an RTX 4060 can handle it; 31B to just 18GB RAM.
Yes, a 26B MoE model in 15GB. That is less than my 16GB 4060 Ti — meaning even my card can run it locally.
The key is that this compression does not sacrifice accuracy. QAT's core idea: bake quantization into the training process, so the model learns to perform well at low precision. Final result: 72% VRAM reduction with accuracy close to BF16 original.
- Unsloth is the real hero
Official QAT weights alone would not be enough. What made this blow up is Unsloth's dynamic quantization approach.
They did something clever: typical community quantization uses Naive Q4_0, which loses a lot of accuracy. For the 26B model, Naive Q4_0 only achieves 70.2% Top-1 accuracy.
Unsloth's 'dynamic method' forces quantized weights to align with the BF16 QAT original distribution. Results: 26B Top-1 accuracy 85.6% — 15.6% higher than Naive; file size is actually smaller than Naive Q4_0 by 200MB; E2B KLD (distribution divergence) is 29x better than Naive, with 22% smaller file size.
Translation: Unsloth not only compresses models further, it makes them significantly smarter than standard quantization.
And they only recommend one quantization type: UD-Q4_K_XL. Other types actually reduce accuracy.
- Mobile Mixture: phones can run LLMs too
There is also a Mobile Mixture QAT variant — extreme compression for E2B and E4B, dropping some layers to 2-bit (TQ2_0): E2B compressed to 2.19GB, 61 tensors at 2-bit; E4B compressed to 3.22GB; near-zero accuracy loss: E2B 97.82% Top-1, E4B 98.76%.
2GB for a multimodal model was unthinkable last year. Google has dropped the entry barrier to phone and edge device level.
What this means for ordinary users
The cost of running local AI models has been slashed to the floor. Previously, running a usable open-source model required at least 24GB VRAM. Now an RTX 4060 (8GB VRAM + system memory sharing to 16GB) can run a 26B MoE model.
And the E2B/E4B variants can run on phones, Raspberry Pis, and laptops.
Deployment is simple — Unsloth Studio installs with one command, or use llama.cpp to download GGUF files.
Personal take
Google DeepMind's move here is a checkmate play on the open-source ecosystem. Gemma 4 is Apache 2.0, fully free for commercial use, with performance matching or exceeding Llama 4. Now with QAT quantization, the hardware barrier drops to consumer-level equipment.
Compare: Llama 4 still does not have an official quantized training release. Google trained quantization awareness directly into the model, paired with Unsloth — creating a perfect pipeline.
One problem: do you have enough disk space? Downloading all five models is nearly 50GB.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.
