23.6% to 7.5%: the same 4-bit quantization step that ships a model to your phone can push it below a coin flip — in Russian only

Qwen 3.5 2B answers multiple-choice reasoning questions in Russian correctly 23.6% of the time at full precision — already mediocre against a 10% random-chance floor on a ten-choice test, but clearly better than guessing. Quantize the exact same checkpoint to 4 bits, the step nearly every on-device model goes through before it ships to a phone, and that number drops to 7.5%. Below random. The model didn't get slightly worse at Russian. It got quantized into a worse Russian reasoner than a die roll, while its English score on the identical benchmark barely moved. Nobody ships a "Q4_K_M, minus your second language" warning label, because until now nobody had measured the gap language by language.

The paper nobody had written yet

The numbers above come from "The Multilingual Quantization Tax: Structural Collapse and Typological Fragility in Edge SLMs" (arXiv:2608.09941), by independent researcher Mohammad Wathiq Soualhi, posted June 21, 2026 and still uncorroborated by follow-up work as of this writing. It's a real, reproducible, and — as far as we can find — entirely unscooped result: quantization research has evaluated the "quantization tax" (the accuracy a model loses when you shrink its weights) almost exclusively in English. This paper asks what happens to the other seven-plus billion people whose languages a phone-sized model is also trained on.

The setup is deliberately narrow, and that narrowness is what makes the numbers trustworthy rather than a marketing slide. Soualhi ran the 2B and 4B instruction-tuned variants of Google's Gemma 4 and Alibaba's Qwen 3.5 — both current, real, phone-class architectures — through lm-evaluation-harness (v0.4.12) on a vLLM (v0.20.1) backend, comparing native bfloat16 weights against calibration-free 4-bit NormalFloat (nf4) quantization via bitsandbytes (v0.45.3) — the same family of "just shrink the weights, no fancy calibration set" quantization that GGUF's own K-quants belong to, though it isn't the identical algorithm llama.cpp uses (more on that distinction below). Eight typologically different languages — English, Chinese, Russian, Japanese, Arabic, Hindi, Swahili, Yoruba — across two benchmarks: MMLU-Pro X Lite (a 10-choice reasoning test, 10% random baseline) and Global PIQA (binary physical-commonsense questions, 50% baseline, with both translated and untranslated versions of each question). Internal "thinking" modes were switched off on every model, specifically so a chain-of-thought couldn't paper over a broken forward pass. Total compute: about 14 GPU-hours on a single RTX A6000 — modest, but real, and the full evaluation commands are published in the paper's appendix for anyone who wants to rerun it.

The tax isn't a tax. It's a tariff, and it isn't flat.

Here's the full table, straight from the paper, all values in percent, higher is better:

Source: arXiv:2608.09941, Table 1 (MMLU-Pro X Lite, 10-choice, 10% random baseline). Bold = the paper's headline collapse cases.

Two numbers in that table do the real work. Qwen 3.5 2B's Russian score — the single largest drop anywhere in the study, 16.1 points, more than double the next-largest single-language tax in the same row — takes it from a real, if weak, reasoning signal down to a number a coin-weighted die would beat about as often. And Qwen 3.5 2B's Swahili score falls from 11.1% (barely above the 10% floor to begin with) to 1.2% — not a small model doing slightly worse, but a model whose already-thin competence gets erased by weight truncation specifically, on that language, on that architecture.

The paper's most useful move is refusing to treat every low number as the same kind of failure. Several scores near the 10% baseline — Gemma 4 E2B-it's Yoruba edging up 1.0 point after quantization, for instance — are flagged explicitly as statistical noise, not evidence of anything: the paper runs a standard-error bound (|Δ| < √(SE_base² + SE_quant²)) on the closest cases and shows several of them, including that Yoruba uptick, fall inside the margin where you can't tell signal from noise. That's the opposite of overclaiming — the author is actively pruning weak evidence out of the headline results, which is exactly why the ones that survive (Qwen's Russian and Swahili drops, both far outside any plausible noise band) are worth taking seriously.

Two architectures, two completely different failure patterns

If quantization damage were just "big model good, small model bad" or "hard language, easy language," you'd expect Gemma and Qwen to break on the same scripts. They don't. Gemma 4 E4B-it takes its single worst hit on Arabic (−11.4 points) while staying rock-stable on Hindi (−3.9). Qwen 3.5 4B does close to the opposite: it's fine on Arabic (−2.7) but its Hindi score was already sitting at a barely-there 3.6% before quantization even touched it — a base-model weakness, not a quantization-induced one, though quantization still shaves it further to 1.5%. The paper calls this a "double dissociation": the same low-bit truncation attacks a different script depending on which architecture it's hitting, because it's not attacking "foreignness" in the abstract — it's attacking whichever narrow band of high-magnitude outlier weights that specific model's specific pretraining mix happened to allocate to routing that specific script into its reasoning core. Cut those weights down to 4 bits and the routing breaks; the language itself was never the variable, the model's internal wiring for that language was.

That framing also explains a genuinely counterintuitive finding: a model's own dominant, best-resourced language gets no special protection. Qwen 3.5 4B's English score is quantization-proof (−0.2, arguably a noise-level improvement) — but its Chinese score, the language its own technical report leans on hardest, still takes a real 3.6-point hit. Gemma's English tax (−4.3) lands almost exactly where its Hindi tax does (−3.9), on a model built by a company with vastly more English than Hindi training data. Depth of pretraining buys you a higher starting line, not immunity from the truncation itself — what the paper calls the "Home Language Fragility Paradox."

Quantization doesn't erase the knowledge. It severs the bridge to it.

The clearest single piece of evidence for what's actually breaking comes from Global PIQA's split design — the same physical-commonsense questions asked twice, once in natural, untranslated Swahili and once as a machine-translated-from-English version. On the native version, Qwen 3.5 4B's Swahili score drops a negligible 1.00 point after quantization. On the translated version — same language, same underlying commonsense content, but requiring the model to route a query through its English-centric reasoning core first — the same quantized model loses 11.65 points. The knowledge needed to answer "will a glass bottle break if you drop it" in Swahili is still in there; what's gone is the machinery that lets a translated prompt find it. The paper's hypothesis, consistent with prior work on outlier weights in transformer internals (Dettmers et al., 2022), is that cross-lingual routing depends on a small number of high-magnitude weights acting as a bridge between the model's dominant language space and everything else — precisely the weights 4-bit truncation is most likely to clip.

The same pattern shows up domain by domain, not just language by language. Averaged across every model and language, hard sciences that require multi-step symbolic reasoning take a noticeably bigger hit under quantization — chemistry (−7.53 points), physics (−7.45) — than associative, recall-heavy domains like law or psychology (around −3.7). Quantization isn't an even haircut across everything a model knows; it disproportionately damages the multi-step reasoning chains that were already the most fragile part of a small model's competence, whatever language you're asking in.

What this does and doesn't tell us about a GGUF phone app

Here's where we have to be precise rather than alarmist, because the honest caveat matters more than the headline. This study quantizes with calibration-free nf4 via bitsandbytes, run through vLLM on a datacenter GPU — not llama.cpp's Q4_K_M K-quants, which is what every model in privateSLM's own catalog actually ships as. Both are "shrink the weights to roughly 4 bits per parameter without a calibration dataset" approaches, and both are exposed to the same underlying mechanism the paper describes — outlier weights getting clipped by low-bit rounding — but K-quants use per-block scale factors that nf4 doesn't, and we have not seen a study that runs this exact multilingual battery against GGUF K-quants specifically. It is entirely possible K-quants handle some of these cases better, or worse; nobody has measured it yet, us included. It's also worth being clear that this is one independent researcher's paper, posted without institutional affiliation, using two model families at two sizes each — a real, carefully-controlled, fully-reproducible result, not yet corroborated by a second lab, and the author says as much in the paper's own limitations section.

What survives that caveat is the shape of the finding, not the exact numbers: quantization tax is not a single number you can quote once for a model card and consider done. A phone-sized model that "works great" in the English demo every reviewer runs can be measurably, sometimes catastrophically, worse in a language that same demo never tested — and the gap is invisible from the English-only benchmark tables the whole industry, us included, currently leans on. We don't have a fix to ship today; there isn't a GGUF flag that solves this and no model in our catalog claims otherwise. What changes is what we'd want to see before trusting an "N tokens/sec, this good on MMLU" pitch for a model somebody's going to use in Hindi, Swahili, or Yoruba: a benchmark run in that language, post-quantization, not an English number and an assumption that it generalizes.

Discuss this on the forum → — if you've run a non-English benchmark against a quantized GGUF model yourself, we'd like to see the numbers.