70.8% vs 98.1%: the 4-bit quantization your whole phone-AI catalog ships as isn't one number

Quantize a 0.6B-parameter Qwen3 model to 4 bits — the exact bit-width every model in our own catalog ships as — and it keeps 70.8% of its full-precision score. Quantize a 14B model the same way and it keeps 98.1%. Same recipe, same lab, same benchmark suite, a 27.3-point gap, and the only variable that moved is how small the model already was. The pitch behind nearly every on-device AI product, this one included, is "go small, then quantize hard, and it'll still basically work." A signal-noise analysis published August 8 says that bet gets worse, not better, at exactly the size class phones actually run.

The paper, and why it's worth trusting the numbers

"Quantization Degradation in Large Language Models: A Signal-Noise Perspective" (arXiv:2608.08188, Zhou, Cao, Ye, Yu, Yu, Li, Zhao and Liu, Institute of Automation, Chinese Academy of Sciences, submitted August 8, 2026) runs GPTQ quantization at 2-bit, 3-bit and 4-bit against the entire Qwen3 family — 0.6B, 1.7B, 4B, 8B and 14B — across 800-plus individual evaluations spanning commonsense, knowledge and reasoning benchmarks. That's the detail that makes this different from a single model card's quantization footnote: it's one consistent method, one consistent benchmark battery, applied across a 23x parameter-count range, so the only thing that changes row to row is scale.

The headline framing in the abstract is almost soothing: "4-bit quantization usually preserves performance, 2-bit often causes broad degradation, and at 3-bit, degradation becomes apparent but varies markedly with task type, quantization method and model scale." Read the actual table and "usually preserves performance" turns out to be doing a lot of work for the smallest models on the list — the ones that make up the entire top half of our own Generalists catalog.

The table

Table 1b reports retention — the percentage of full-precision (bf16) score a quantized model keeps — for each Qwen3 size at each bit-width:

Source: arXiv:2608.08188, Table 1b (GPTQ, Qwen3 family, retention = quantized score ÷ bf16 score, averaged across the paper's benchmark suite).

Two shapes sit inside that table, and they're different shapes. At 4-bit and 3-bit, retention climbs steadily with scale — 0.6B is worst, 14B is best, every step in between is monotonic. At 2-bit, that relationship doesn't just weaken. It disappears. 8B (2.2%) is worse than 0.6B (6.3%). 14B (8.4%) barely clears 1.7B's already-broken 3.6%. Every single model in the table is functionally destroyed at 2-bit — the paper's own framing, "broad degradation," undersells a Qwen3-8B that's retained less than a fortieth of itself. Scale stops being a shield the moment you go low-bit enough, because by then every model, regardless of size, is operating on noise rather than signal.

Where the damage actually concentrates

The paper breaks 3-bit retention down by task category, and the split is sharper than the overall number suggests. At 0.6B: commonsense retention is 55.3%, reasoning is 20.9%, and knowledge — factual recall — is 15.5%. At 14B those same three categories sit at 93.7%, 86.4% and 83.6%. Commonsense degrades the least at every size; knowledge degrades the most. A small quantized model isn't uniformly "a bit worse" across the board — it's disproportionately losing the facts it knows, while keeping more of its basic situational reasoning intact. That's a specific, checkable prediction: ask a heavily quantized small model something with a definite factual answer, and the failure mode this data predicts is a wrong fact stated fluently, not a garbled sentence.

The mechanism the authors propose is a signal-to-noise framework, not just a benchmark table. Quantization error gets introduced at every layer a weight passes through, and that error either compounds or dampens as it propagates deeper into the network. Their finding: "larger models benefit not only from smaller errors introduced at each layer but also from weaker amplification as errors propagate" — smaller models suffer both a bigger initial hit and a network structure that lets that hit snowball forward through every subsequent layer, while a 14B model's extra width and depth apparently give quantization error more room to get diluted rather than compounded. Size isn't just "more parameters to spread the damage across" — it changes how the damage propagates, structurally.

The 2-bit collapse has an independent second source

A second, unrelated paper landed six days later and lands on the same collapse point from the opposite direction — not measuring the damage, but trying to fix it. QUASAR (arXiv:2608.13966, Counathe, Athiwaratkun, De Sa and Zhang, submitted August 14) targets exactly the mismatch standard quantization-aware training has at low bit-widths: QAT computes its training loss and gradients against a lossy reconstruction of the full-precision weights, then applies the resulting updates to those same latent full-precision weights — a loop that never actually reconstructs against the true quantization error it's supposedly correcting for. QUASAR replaces that with continuous, loss-aware reconstruction during training itself, using an exponential moving average of squared gradients as an online importance signal and fitting the dequantizer with saliency-weighted least squares.

Tested on Qwen3 and Llama-3.1 models, QUASAR's fix delivers its largest gains specifically at 2-bit: a 29% reduction in KL-divergence versus baseline QAT methods and a 3.5–4.3 percentage-point accuracy improvement. At 3-bit and 4-bit, the same method only manages a 10%-plus KL-divergence reduction — real, but a fraction of the 2-bit gain. Two independent teams, different methodology (one measuring degradation, one measuring a proposed fix), converge on the same conclusion: 2-bit is where standard quantization pipelines are most broken, and it's also where a smarter pipeline has the most room to help. Neither number moves the collapse-and-flip pattern in the first table — QUASAR is a training-time fix that would need to be baked into a model before release, not something a Q4_K_M-shaped app can retrofit onto a GGUF someone already quantized the old way.

A third, independent data point: what an edge benchmark actually chose

Neither paper is talking to the on-device industry directly, so here's a data point from somewhere that had to make a real decision instead of a research claim. MLCommons' MLPerf Inference v6.1 introduced an "Edge Agentic Inference" track this cycle, and had to pick one reference model and one quantization format for the whole benchmark. They picked Qwen3.6-27B, served via llama.cpp, quantized to Q4_K_M GGUF — a model roughly double the size of the largest one in the CASIA retention table, at 4-bit, the bit-width where that table shows the curve has already flattened out near its ceiling (95–98% from 4B upward). MLCommons didn't cite the CASIA paper — the timing doesn't even allow it, their spec predates it by weeks — but a benchmark org whose entire job is picking a config that won't embarrass itself under scrutiny landed exactly where the retention curve says quantization stops costing you much: well above the sub-4B range, at the bit-width where the size penalty has already mostly washed out.

What this means for a catalog that's mostly sub-4B, at exactly 4-bit

Here's where we turn the lens on ourselves rather than the industry in the abstract. privateSLM's own Generalists tier — the models most people actually pick — is LFM2.5 1.2B, Qwen2.5 1.5B, Gemma 2 2B, Llama 3.2 3B, Phi-4 Mini 3.8B, G9v3 3B, Nanbeige4.2 3B and LFM2.5 2.6B. Every one of them ships as Q4_K_M or the nearest equivalent (Nanbeige4.2 is IQ4_XS specifically because Q4_K_M missed our size cap). Every one of them sits in the 1.2B–3.8B band the CASIA table shows retaining somewhere between roughly 71% (extrapolating from the 0.6B row) and 95% (at 4B) of full precision — not the 98% ceiling reserved for the 14B tier nobody's phone can run anyway.

Three honest caveats keep this from being a "your app is broken" post. First, this table is GPTQ, not GGUF K-quants — the same caveat we raised about the multilingual-quantization study three days ago applies here too: K-quants use per-block scale factors GPTQ doesn't, and nobody has run this exact battery against K-quants specifically. It's plausible K-quants hold up better at the same nominal bit-width; nobody has measured it. Second, this is Qwen3's architecture and training recipe specifically — LFM2.5, Gemma, Llama and Phi-4 could each sit meaningfully above or below this exact curve; the paper doesn't test them, and the "different architectures fail differently" lesson from our own quantization coverage this month says we shouldn't assume they line up. Third, retention here is averaged across benchmark categories most users never directly probe — a 1.2B model losing 25–30% of its raw eval score doesn't necessarily read as "25% worse" in an ordinary chat, especially on the commonsense tasks the paper shows degrade least.

What survives those caveats is the shape, not the exact number, same as it's been every time this month a new quantization paper has landed on our desk: retention is not one number you get to quote once on a model card and move on from. It's a function of size, bit-width and task type all at once, and the size class an entire industry — us included — is racing to fit onto phones is precisely the size class where that function is at its steepest, most punishing part of the curve. We don't have a fix to ship today. Nobody does; that's what QUASAR is trying to build, and it lives in the training step, not the download. What changes is what "Q4_K_M, verified" should mean in a catalog entry going forward: a confirmed download that loads and runs, not a silent assumption that four bits costs the same wherever you spend them.

Discuss this on the forum → — if anyone has run a K-quant-specific version of this comparison, we want the numbers.