SLM Daily #12: our day-one catalog entry just lost to a newcomer by 42x on one benchmark
0.33 versus 14.00 — that's Llama 3.2 1B Instruct against Liquid AI's brand-new LFM2.5-1.2B-Instruct on AIME25, by Liquid's own published numbers. It isn't a cherry-picked outlier: the newcomer beats Llama 3.2 1B on all seven benchmarks Liquid tested it against, in a smaller file. Llama 3.2 1B has been in privateSLM's catalog since before this blog existed. Today it's out, and LFM2.5-1.2B-Instruct is in — the first straight one-for-one replacement this catalog has ever made, rather than a pure addition.
The numbers, from the vendor that has something to lose by inflating them
Liquid AI published a direct head-to-head table on LFM2.5-1.2B-Instruct's own model card, released August 5, comparing it against four "relevant sub-2B models" including the exact Llama 3.2 1B Instruct build we've shipped since launch:
| Model | GPQA | MMLU-Pro | IFEval | IFBench | Multi-IF | AIME25 | BFCLv3 |
|---|---|---|---|---|---|---|---|
| LFM2.5-1.2B-Instruct | 38.89 | 44.35 | 86.23 | 47.33 | 60.98 | 14.00 | 49.12 |
| Qwen3-1.7B (instruct) | 34.85 | 42.91 | 73.68 | 21.33 | 56.48 | 9.33 | 46.30 |
| Granite 4.0-1B | 24.24 | 33.53 | 79.61 | 21.00 | 43.65 | 3.33 | 52.43 |
| Llama 3.2 1B Instruct | 16.57 | 20.80 | 52.37 | 15.93 | 30.16 | 0.33 | 21.44 |
| Gemma 3 1B IT | 24.24 | 14.04 | 63.25 | 20.47 | 44.31 | 1.00 | 16.64 |
Source: LiquidAI/LFM2.5-1.2B-Instruct model card, published August 5, 2026. GPQA/MMLU-Pro/IFBench/AIME25 follow ArtificialAnalysis's methodology; IFEval/Multi-IF average strict and loose scoring; BFCLv3 uses Liquid's tool-use handler.
We didn't just trust the table. We HEAD-checked LiquidAI/LFM2.5-1.2B-Instruct-GGUF's Q4_K_M file ourselves and got back 730,895,168 bytes (730.9 MB) — smaller than Llama 3.2 1B's own Q4_K_M (807.7 MB), which has been in our catalog since it launched. Same size tier, same lfm2 architecture already confirmed compatible in llama.cpp mainline (it's the same family as our existing LFM2.5-2.6B entry), same LFM Open License v1.0 we already ship under. A model beating the incumbent on seven independent metrics, in a smaller file, isn't a marginal upgrade our "replace only when it strictly dominates" rule is built for — it's the exact case that rule exists for.
One honest caveat: these are Liquid's own numbers, not a benchmark we ran ourselves, and Llama 3.2 1B is now nearly two years old — this says less about Meta's engineering than about how far the 1B-parameter class has moved since. We're not aware of any independent leaderboard that disputes the ranking, but if one surfaces showing a different picture, we'll say so.
What's now in the catalog
Generalists: Llama 3.2 3B, Qwen2.5 1.5B, Gemma 2 2B, Phi-4 Mini, G9v3 3B, Nanbeige4.2 3B, LFM2.5 2.6B, and now LFM2.5 1.2B in place of Llama 3.2 1B — still 8, our generalist cap. 14 pre-cap specialist models (medical, legal, finance and similar) are unchanged, for the same standing reason as every prior edition: we haven't unilaterally pulled models people are actively downloading from a catalog with no size cap on that tier.
Engine check
llama.cpp advanced from b10362 (Aug 11) to b10436 (Aug 15) — 74 tags in four days, all optimization and bugfix work (SYCL, server metrics, WASI threading). We diffed llama-arch.h directly: no new architecture tags land in this window that change anything for a phone-sized model.
Also this edition
Our deep-dive digs into a new paper on 4-bit quantization's language-by-language cost — the same nf4 truncation your phone's SLM ships with can push a model's accuracy on some languages below random chance, while barely touching English on the identical checkpoint. Read it here. No Edge in the Wild spotlight this edition — nothing that met our attribution and verification bar surfaced in the window since our last check.