Small language models, Apple Intelligence, private machine learning — explained for humans.
Artificial Analysis and Liquid AI's new mobile-intelligence benchmark ties Nanbeige4.2-3B (already in our catalog) with LFM2.5-2.6B on a real iPhone 17 Pro. The tie comes entirely from one axis: when either model doesn't actually know the answer, one stays quiet and the other confidently guesses wrong two times out of three.
IBM's Granite 4.2 3B beats Llama 3.2 3B Instruct by 22 points on GPQA — but every other benchmark on its card uses a different test version than the one we can check it against. Plus what's new in llama.cpp since our last edition, and four DIY candidates checked and skipped.
A July arXiv paper measured 4-bit quantization making a vision-language model's decode 55.8-55.9% slower on NVIDIA Jetson boards. We quantized a chat model to the same bit-width on our own engine and measured decode 81% faster. Both numbers are real — the difference is the kernel, not the math.
llama.cpp shipped v0.2.0, its first tag to get an actual GitHub Release page instead of a bare pre-release tag — carrying a new 280B/16B-active MoE architecture nowhere near phone-sized. Zero catalog changes; one candidate checked and skipped, and why.
llama.cpp retuned when CUDA switches from one decode kernel to another, worth up to 49% more speed on an RTX 5090. Real Metal and Vulkan correctness fixes shipped too. Zero catalog changes, zero new architectures — our own engine is still three months behind all of it.
Mayukh ran the same 10-question offline-AI test on a $90 handheld board, a Raspberry Pi 5 and a $1,000 Ryzen mini PC. The board used 14.6x less RAM than the Pi for 15% less speed — and the priciest machine's "perfect" score hid a confidently wrong claim the grading rubric never caught.
A fresh signal-noise analysis of 800+ quantized evaluations shows 4-bit retention isn't flat — a 0.6B model keeps 70.8% of its full-precision score, a 14B model keeps 98.1%, and at 2-bit the protection model size buys collapses entirely. What that means for a catalog built almost entirely of sub-4B models shipped at exactly that bit-width.
Andris Gauračs wrote his own ~3,700-line C99 inference engine from scratch to run Cactus Compute's 45M-parameter Needle 2 tool-calling model on a bare ESP32-S3 — no filesystem, no OS, no port of Cactus's own code. Grammar-guaranteed JSON tool calls, entirely offline.
A one-line kernel-launch bug in llama.cpp's SYCL backend took an Intel Arc B70 from 20.21 GB/s to 158.19 GB/s once fixed. Three engine releases since our last edition, zero new architectures, zero catalog changes — the one new GGUF worth naming misses our size cap by nearly 2.5x.
Hugging Face's own Summer 2026 report: not one model released in 2026 cracked the platform's download top 25, and thirteen of the twenty-five are from 2022. What that means for a blog that covers a new small model every few days.
Two llama.cpp fixes closed a silent-corruption path in LoRA adapter loading and GGUF array parsing. Kimi-K3 architecture support landed too, but it's a 2.8T-parameter model. Zero catalog changes this edition — what we checked and skipped, and why.
A fresh llama.cpp server bug lets one finished conversation's content bleed into a different session under concurrent load — and cached_tokens, the field built to flag cache reuse, reads 0 every time it happens. We checked our own engine's code against it.
A new independent study finds 4-bit quantization doesn't apply an even tax across languages. Qwen 3.5 2B's Russian reasoning score falls from 23.6% to 7.5% — worse than random chance — while the identical checkpoint's English score barely moves.
Liquid AI's LFM2.5-1.2B-Instruct beats Llama 3.2 1B Instruct — in our catalog since launch — on all seven of Liquid's own published benchmarks, including a 42x gap on AIME25, at a smaller file size. First straight replacement in the catalog's history.
Rob Porter's torchwright compiler writes exact algorithms directly into a stock Phi-3 checkpoint's weights — zero gradient descent, zero training data — and beats six frontier LLMs at direct-answer multiplication, 100% to as low as 0%. The checkpoint is the same architecture family already in our own catalog.
Edge in the Wild: Salman Farsi's NanoMind-S3 runs a fully dense LLaMA-2-architecture transformer on a single ESP32-S3 — no lookup-table shortcuts, no flash-embedding trick, every one of its 15.2M parameters computed on every token.
Every on-device LLM benchmark quotes a first-reply number. A real sustained-load study found an iPhone 16 Pro loses 44% of its throughput by the eighth reply — and a Samsung Galaxy S24 Ultra doesn't slow down at the sixth. The OS floors the GPU clock and inference stops outright.
Edge in the Wild: Samuel Alexander runs Meta's dense 30B Muse-Glimmer vision-language model entirely on a Qualcomm Dragonwing IQ-9075 edge board, no cloud calls — 2.84 tok/s turns out to be 91% of the board's own peak memory bandwidth, and the model is honest when it hallucinates a part number.
Four candidates checked since our last edition, zero added. LiquidAI's LFM2.5-VL-3B clears every published catalog rule and still doesn't ship — because our own engine has no code path that ever hands a model a photo.
The EU AI Act's new content-marking law took effect August 2 and requires AI-generated text to carry a machine-readable, detectable mark. No model in our catalog does this, neither does llama.cpp's sampler, and it's unclear whose obligation it even is — a compliance gap the whole open-weight ecosystem shares.
Edge in the Wild: independent builder leonickson1 streams a mixture-of-experts model's weights straight off SSD instead of loading them, running Qwen3-Next-80B in 4.3 GB of peak RAM on a Mac — and a 35B model on an iPhone.
A one-line ordering bug in llama.cpp disabled sliding-window attention for EXAONE 4.5, pushing its KV cache from 1952 MiB to 4160 MiB and corrupting output — fixed August 11. Zero catalog additions this edition; six candidates checked and rejected, why.
Edge in the Wild: independent builder Carloscodix trained a transformer entirely on an $8 ESP32-S3 — hand-written backpropagation in C, gradient-checked to 1.07e-08, no GPU or datacenter ever in the loop. Every other DIY build on this blog runs a model born elsewhere; this one was born on the chip.
Diffusion LLMs are pitched as the on-device latency fix — parallel tokens instead of one at a time. The only real phone-NPU benchmark that exists says otherwise: the naive version is still slower than the model it's replacing on all but the newest chips, and the fix that makes it fast collapses GSM8K accuracy 12x.
Edge in the Wild: PrismML's Bonsai 27B shipped MLX-only in July. A solo builder now runs a 26.9B-parameter, 1-bit build of it on a Jetson Orin NX 16GB at 6.76 tok/s on 25 watts — via a native GGUF pack and a private llama.cpp fork that still doesn't touch mainline.
A Metal shader bug that could silently corrupt normalization math on Apple GPUs got fixed the day after our last check. Zero catalog additions this edition — seven candidates checked and ruled out, and why.
AMD just paid an undisclosed sum for Taalas, whose test chip serves Llama 3.1 8B by casting the model's weights directly into silicon instead of loading them from memory. What that means for engines — like ours — built to run whichever model you actually download.
Edge in the Wild: NightRun is a UEFI application that IS the machine's first and only program — no Linux, no scheduler, no network stack — loading a quantized model straight into RAM and serving chat on the bare framebuffer. Real Raspberry Pi 5 hardware, MIT-licensed.
Liquid AI's LFM2.5-2.6B lands in the catalog five days after its own release. llama.cpp hit tag b10290 this week, all of it multi-token-prediction and MoE work irrelevant to a phone.
Independent builder slvDev got a 28.9M-parameter language model running on an $8 ESP32-S3 microcontroller — about 100x the previous record on hardware like this — by keeping 25 million of those parameters in flash instead of RAM.
A byte-identical 355 MB GGUF costs 56 MB of RAM on an iPhone 13 Mini and 330 MB on a Galaxy A56, through the exact same inference engine. On-device memory budgets aren't a model property — they're an OS property.
SLM Daily #7: llama.cpp cut ten releases in one day and picked up a phone-sized architecture. Its default quant missed our size cap — a smaller one didn't, and it's our first catalog addition since July 27.
Nanbeige4.2-3B beats models three to four times its size by looping its own transformer layers twice instead of adding parameters — a computation mainline llama.cpp can't build yet, gated behind a private fork and a closed GitHub issue.
SLM Daily #6: GLM-5.2 and Laguna-S-2.1 support landed in llama.cpp this week — 754B and 118B parameters, neither phone-sized. Zero catalog changes.
Two independent 2026 papers find phone NPUs losing up to 10x their usable throughput and up to 65% of their achievable energy efficiency to scheduling code, not weak silicon — and it's playing out in llama.cpp's own repo right now.
SLM Daily #5: G9v3-3B tops Artificial Analysis's Intelligence Index for open-weight models under 4B params — we verified it and added it anyway. Plus llama.cpp's architecture count jumped from 135 to 171, and a size-cap bug we've now caught twice.
Private Cloud Compute's free tier counts lifetime downloads across every app you've ever shipped, not the one asking for AI. A ten-year App Store veteran got locked out over an unrelated app — and there's no paid tier to buy back in.
G9v3-3B cleared every mechanical bar we check for and posted a genuinely independent benchmark score 5x its peer median — plus what else we checked this edition and why nothing else made the cut.
llama.cpp release b10089 closed a gap that had been there since k-quants shipped: CUDA couldn't run GET_ROWS on quantized embedding tables, so it silently fell back to a CPU round-trip on every token. Fixing it took Qwen3-1.7B from 6.18 to 1.72 ms/token — 3.6x faster. Metal, Vulkan, CPU and SYCL never had the bug.
Edge in the Wild: Hanzo Huang's rkllm-docker packages five chat models for Rockchip's RK3576/RK3588 NPU boards behind a single docker run — Qwen2.5-1.5B hits 19.55 tok/s at 1.6GB RAM, and the smaller W4A16 quant beats W8A8 on both speed and memory, every time.
Liquid AI's July 21 technical report puts a real number on the "tokenizer tax": expanding LFM2.5-8B-A1B's vocabulary from 65K to 128K tokens cut Thai token counts by up to 4x — and slowed decoding for every language, including English, by 7 to 10%, forever, on every token.
Xiaomi just announced Xiaomi-Robotics-1, trained on 100,000+ hours of manipulation data. Its predecessor, the model actually shipping today, is a 4.7B-parameter VLA that runs in 80ms on a single RTX 4090 — smaller than most phone chat models, for a reason that has nothing to do with cost.
A public Apple Silicon benchmark shows llama.cpp costs 0.25 joules per generated token on an iPhone 17 Pro versus 0.11 for Apple's own Foundation Models path — and a naive CoreML/ANE port of the same open model is worse than either. What that means for engines that chase model compatibility instead of Apple's walled garden.
Edge in the Wild: home-lab builder AGmind benchmarked a single Beelink GTR9 Pro (AMD Ryzen AI Max+ 395, 128GB unified memory) against 32 concurrent llama.cpp chat sessions — 236 tok/s aggregate, sustained for 30 minutes, with the numbers that made it look bad too.
Nvidia's July 15 Jetson Thor T2000/T3000 launch cut memory on purpose to dodge a DRAM price spike — and IDC now says the same squeeze is making sub-$100 phones "permanently uneconomical." What that means for every on-device AI app's RAM roadmap.
PrismML's Bonsai 27B shrinks a 54GB model to 3.9GB at 89.5% of full-precision quality — and reports say Apple is in talks to acquire the team that built it. Why the model that might save Siri's on-device story can't run in our engine either.
Edge in the Wild: Tim Schupp's 2-node Kubernetes cluster on AMD Strix Halo mini-PCs — 218ms to first token on a local 32B model, real hardware photos, full credit to the builder.
SLM Daily #3: the license question we left open two posts ago is resolved — it was never the blocker. The only thing keeping a 153 MB model out of the catalog is that it's 23 days old. Zero catalog changes again.
A $249 Jetson Orin Nano out-detected a Mac Mini in a builder's own benchmark. Ten verified, individually-credited Jetson projects, ranked — with a top-5 countdown video built almost entirely from the builders' real photos and footage.
A countdown of five DIY robotics builds running small language models fully on-device — companion video made with MegaBrain, rendered with Kling 3.0.
25 verified DIY robotics builds running small language models on-device, sorted into 10 categories — from a $5 microcontroller to an NVIDIA-award-winning animatronic robot. With a companion video, made with MegaBrain.
Our build script fetches whatever ggml-org/llama.cpp calls "latest" at compile time and never records the tag. Real release-cadence and PR-merge data show why that's a subtler problem than the pinned-old-engine gap we wrote about a week ago.
SLM Daily #2: the engine now tracks llama.cpp's latest release instead of an old pinned copy — Qwen3, Gemma 3, Llama 4 and LFM2 all check out. The catalog didn't grow today anyway, for a specific, boring reason.
Same model, same quant label, three different files — we compared bartowski, Unsloth and AesSedai's Qwen3.5 builds, and Unsloth's Gemma 3 27B quant against Google's own QAT checkpoint.
Every local-AI app's description carries the same kind of number. Ours vendors an engine that recognizes three Qwen-family architectures — and none of them is the one Qwen's current releases actually use.
SLM Weekly #1: what we added to the privateSLM catalog, what we skipped, and the exact grep command that told us which was which.
Zero-download AI chat via Apple Intelligence, a local GGUF fallback, real chat history and hands-free voice — everything on device.
GPT-class models get the headlines, but the most interesting shift in AI is happening at the small end. Here's why 1–3B parameter models matter.
iOS 26 gives every app direct access to Apple's on-device model. What it can do, what it can't, and what we learned shipping with it.
"We don't store your data" is a promise. Physics is a guarantee. The case for AI that never phones home.