Your phone's AI chip is doing up to 10x less than it could — and it's not the chip's fault

A phone's NPU can lose up to 10x its usable throughput to the software scheduling sitting on top of it, according to a paper published this month — not because the silicon is underpowered, but because of thread configuration, uncoordinated sleep latencies, and polling intervals in the framework layer. A second, independent paper found that fixing just the frequency scheduling, no hardware change at all, recovers up to 65% of the energy a phone wastes running local AI. Neither number is about the chip in your pocket. Both are about a few hundred lines of scheduling code standing between it and the model you're actually running.

This matters specifically for the app category privateSLM is in. Every "runs AI on your phone" pitch — ours included — leans on the phone's NPU spec sheet: some number of TOPS, marketed like a benchmark result. These two papers are the first place we've found that spec sheet checked against what local-AI software actually extracts from it. The gap is the story.

What a TOPS number actually promises

Every phone with a dedicated AI accelerator ships a headline number — trillions of operations per second, the same unit chipmakers use to sell GPUs. It's a ceiling: the most the silicon can physically do under ideal conditions. Nothing about that number says what share of it a given app, on a given day, running a given model, actually reaches. Until this year, almost nobody outside chip vendors' own labs had published a direct measurement of that gap for local LLM inference specifically. Two papers did, independently, within two weeks of each other.

Up to 10x — and it's not the silicon

"Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference" (Cai, Tian, Yang, Ren, Yuan, Li, Wang; submitted July 6, 2026) measured framework-to-framework throughput on identical phone hardware running identical models. Its central finding: "framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators." Same chip, same model — the only variable across that 10x range is which software stack is doing the scheduling.

That's the part worth sitting with. A 10x gap from switching frameworks, with the hardware held constant, means the NPU spec on the box was never really in question. What's in question is whether the app between you and that spec is using it.

Where the other 40–55% goes

The same paper breaks the waste down into causes, not just a headline ratio:

Source: arXiv:2607.05475, "Is Your NPU Ready for LLMs?" (submitted 2026-07-06)

None of these three causes — thread config, sleep-latency handling, polling loops — is a model problem or a silicon problem. They're scheduler problems: decisions a framework makes about how often to check for work, how aggressively to let a chip idle down, and how many threads to hand a task, made once by whoever wrote the inference engine and then inherited by every app built on it.

A second team, a different device class, the same story

"EnerInfer: Energy-Aware On-Device LLM Inference" (Zou, Liu, Sun, Mascherin, Roy, Liu, Peng, Jia, Chen; submitted 2026-06-22, revised 2026-06-24) arrived at a compatible conclusion from a different angle: instead of comparing frameworks, it tunes NPU and memory clock frequencies dynamically per inference request, and measures the energy saved with no accuracy or latency penalty ("without QoE violation," in the paper's terms).

Source: arXiv:2606.23001, "EnerInfer: Energy-Aware On-Device LLM Inference"

Two teams, two weeks apart, two different methodologies (framework-comparison vs. dynamic frequency scheduling), and neither cites the other. Both land on the same underlying claim: on a phone specifically, the software layer between the model and the NPU is the dominant lever on both speed and battery life, bigger than anything you'd get from a faster chip.

This isn't hypothetical — it's playing out in llama.cpp's own repo

llama.cpp, the inference engine privateSLM's local-GGUF fallback (and Ollama's, and LM Studio's, and most other local-AI apps') is built on, only shipped a mainline backend for Qualcomm's Hexagon NPU in build b8755 (April 11, 2026). Before that release, llama.cpp builds on Snapdragon phones ran on CPU and general-purpose GPU compute paths — Metal, CUDA, Vulkan — never touching the phone's dedicated NPU silicon at all, regardless of what the phone's spec sheet claimed. Most local AI apps that haven't shipped a rebuild since April are almost certainly still in that position today.

Even after the NPU backend landed, the scheduling gap the two papers describe is visible in real time inside that same project. A community fork of llama.cpp's Hexagon backend (zhouwg/ggml-hexagon) states in its own README that its prefill and decode performance on a Snapdragon 8 Elite phone "surpasses Qualcomm's official ggml-hexagon since 07/21/2026" — six days before this post. Read that plainly: a community reimplementation, running on the exact chip Qualcomm designed and the exact SDK Qualcomm shipped for it, is currently outperforming Qualcomm's own reference software. If the chip vendor's own stack isn't the ceiling, "the NPU is slow" was never a safe conclusion to draw from a slow local AI app.

What this means if you're picking a local AI app

Two honest implications, not one convenient one:

• A TOPS number in an app's marketing describes the chip, not the app. Neither paper found a way to infer software efficiency from hardware specs — you'd need a direct benchmark of that specific app, on that specific phone, which almost nobody publishes.
• A vendor-maintained NPU stack has a structural advantage a community backend has to fight for. Apple's Foundation Models framework — the engine privateSLM uses first when your device supports Apple Intelligence — is written and tuned by the same company that designs the Neural Engine it runs on, with none of the framework-hopping the "up to 10x" figure above describes. A third-party llama.cpp build reaching for a phone's NPU through a community-maintained backend is, per this month's evidence, fighting exactly the scheduling battle these two papers measured — sometimes winning it, as the July 21 result above shows, but starting from behind by default.

Where we actually stand

We haven't benchmarked privateSLM's own llama.cpp fallback against its NPU ceiling on any specific phone — that's a real gap in what we can tell you today, not a reassurance we're going to manufacture. What we can say honestly: our engine-selection logic already prefers Apple's system model when it's available, for reasons that were about capability and zero-download convenience, not NPU efficiency specifically. This month's research is a reason to also care about NPU efficiency, and a concrete thing to go measure rather than assume.

Discuss this on the forum → — if you've profiled a local LLM app's actual NPU utilization on your own phone, we want the numbers.