One skipped code path made small models 3.6x slower to decode on every CUDA GPU

At 19:45 UTC on July 22, llama.cpp shipped release b10089, and one contributor's own benchmark number tells you everything: decoding Qwen3-1.7B on CUDA went from 6.18 milliseconds a token to 1.72 — 3.6x faster, on identical hardware, with no change to the model. The bug wasn't in the model. It was a single missing code path in llama.cpp's CUDA backend, and Metal, Vulkan, CPU and SYCL never had it.

That's the part worth sitting with. This wasn't a new optimization or a clever kernel trick — it was llama.cpp's other three backends already doing the right thing, for who knows how long, while CUDA quietly took the slow path on every single decode step.

What GET_ROWS actually does

Every token you generate starts with a lookup: take the token's ID, pull its row out of the embedding matrix. That operation is called GET_ROWS in GGML's compute graph, and on a quantized GGUF — the Q4_K_M-style files this blog writes about constantly — that embedding matrix (token_embd) is itself stored in a quantized format, commonly q6_K for a Q4_K_M checkpoint. To read a row out of a q6_K tensor, the backend needs a dequantizing kernel that understands the q6_K layout.

CUDA didn't have one — not for q6_K, not for any of the other k-quants (q2_K through q6_K), not for the nine i-quant variants, not for mxfp4. When GGML's scheduler hit a GET_ROWS op it couldn't run on a quantized type, it didn't error out — it fell back, copying the entire embedding matrix to host memory and doing the lookup on the CPU instead, then shipping the result back to the GPU. Every token. On every CUDA card. The PR's own description puts it plainly: these lookups got "kicked out of the graph," forcing a slower host round-trip than even Vulkan managed on the identical hardware.

K-quants aren't new. Contributor ikawrakow introduced the whole q2_K–q6_K family in PR #1684, merged June 5, 2023, and that original PR already shipped CUDA kernels for them — the author's own numbers in that thread benchmark Q2_K and Q3_K token prediction directly on an RTX 4080. CUDA has had fast matmul kernels for k-quants since the format existed, three years before this fix. What it didn't have — for how long exactly, the b10089 release notes don't say — was the much narrower GET_ROWS op used specifically for embedding-table lookups. The PR's own description only establishes that the gap was old enough to have a name and a standing workaround: skip the test on CUDA and move on.

The fix, and the receipts

PR #25962, authored by contributor @pwilkin, factored the existing super-block dequantizer functions out of convert.cu into shared device-side helpers, then wired them into a new k_get_rows_kq kernel with quant-specific thread layouts (32 threads for q4_K, 64 for the others). No new tests were written for the fix — the existing test-backend-ops suite already covered GET_ROWS across every quant type; those cases simply flipped from "skipped on CUDA" to "passing against the CPU reference," which is its own quiet confirmation of how long this specific gap had a name and a known workaround (skip the test) rather than a fix.

Why small models paid the biggest tax

Here's the part that matters for exactly the size of model this blog covers. The cost of that CPU round-trip — copy an entire embedding matrix, do the lookup, copy the result back — is roughly fixed per token, regardless of how big the model is. A 70B model's actual per-token matmul work dwarfs that fixed overhead, so the bug was close to a rounding error for it. A 1-4B phone-and-edge-class model does far less real compute per token in absolute terms — which means the same fixed overhead ate a much bigger slice of its total decode time. The bug didn't just slow CUDA down uniformly. It slowed CUDA down worst for the exact class of model people run because they want it to be fast: small ones.

That's also precisely backwards from the usual small-model pitch. The whole argument for a 1-4B model over a 70B one is that less compute per token should mean more speed. A silent scheduler fallback like this one quietly claws part of that advantage back — not because the small model got slower in any way its own benchmark card would show, but because the engine running it was taking a detour nobody could see from the outside. A tok/s number on a llama-bench run captures the net effect, but nothing in a typical benchmark table would have told you a chunk of that number was CPU-copy overhead rather than GPU compute.

The backend that never had the problem

The part of the release notes easy to skim past: this fix brings CUDA to "complete parity with CPU, Metal, Vulkan, and SYCL backends" on GET_ROWS type coverage. Read that the other way around — CPU, Metal, Vulkan and SYCL were already there. If you've been running a GGUF chat model through llama.cpp's Metal backend on a Mac or an iPhone, this specific detour was never yours to pay. It's a genuinely CUDA-specific gap, which is a useful, narrow claim — not an argument that Nvidia hardware is slower in general, just that this one code path was incomplete on this one backend until yesterday.

What this means if you're running a rig, not a phone

privateSLM's engine runs on Apple Silicon — CPU and Metal, never CUDA — so this specific bug was never in our own users' path. But a big share of this blog's audience isn't reading on an iPhone: it's the same crowd we've written about running a Beelink mini-PC serving 32 concurrent chat sessions or a two-node home-lab cluster, plus anyone on r/LocalLLaMA with an Nvidia card doing the same thing this blog's whole thesis is about — keeping inference local instead of routing it through someone else's API. If that's your box, the fix is a one-line ask: update to llama.cpp b10089 or newer, and re-run whatever benchmark you last trusted. If you built your capacity-planning numbers on a pre-b10089 CUDA build — how many concurrent chat clients your home server can hold, what tok/s you quoted a friend — those numbers were measuring the bug as much as the hardware.

# Check what you're actually running
./llama-cli --version

# Reproduce the before/after yourself on your own GPU + model
./llama-bench -m your-model-Q4_K_M.gguf -p 0 -n 128 -r 5

If llama-cli --version reports a build older than b10089 and you're on CUDA, that single number is the gap between the benchmark you have and the one you'd get after a rebuild.

Discuss this on the forum → — if you've re-run llama-bench before and after b10089 on your own card, we want that number.