One Docker command turns a Rockchip NPU board into a 19.55 tok/s chat server
On July 21, Shenzhen-based edge-AI engineer Hanzo Huang (Hanzo-Huang on GitHub, "AI Sensing Application Engineer" at Seeed Studio) published a Docker image that turns a Rockchip RK3576 or RK3588 board into an OpenAI/Ollama-compatible chat server with one command — no manual RKLLM toolchain setup. His own benchmark table: Qwen2.5-1.5B-Instruct at 19.55 tokens per second, using 1.6 GB of RAM, on a board whose entire acceleration budget is a neural processing unit, not a GPU.
Edge in the Wild is our spotlight on real builders running real models on real edge hardware — credited, sourced, no invented numbers. Source, in full, with credit: Hanzo Huang, rkllm-docker (GitHub), also written up on Hackster.io. The repo's own README describes it plainly as "a personal Docker test bed... not a stable release or complete RKLLM platform" — an honest label from the builder, and one we're keeping intact rather than dressing it up.
What it actually packages
Rockchip's RK3576 and RK3588 SoCs ship with a dedicated NPU, and Rockchip provides rkllm — a separate runtime and conversion toolchain from llama.cpp entirely, built specifically for that NPU rather than for CPU or a general GPU compute API. Getting a model running on it normally means hand-converting weights with Rockchip's own toolkit, matching driver versions to a specific board, and wiring up your own serving layer. Huang's project collapses that into pre-built container images — pick a tag matching your board and quant (e.g. w4a16-rk3576), docker run it, and a FastAPI server exposes a chat completions endpoint immediately, with Prometheus metrics included.
Five models, two quant schemes, one board
The README lists five chat models validated on RK3576, each in two quantization schemes — W4A16 (4-bit weights, 16-bit activations) and W8A8 (8-bit weights, 8-bit activations): Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen3-1.7B, Qwen3-4B, and Gemma 3 4B IT. Every one of them is a model already familiar to anyone following this blog's own catalog — the interesting part isn't the model selection, it's what a purpose-built NPU runtime does with models this shape that a general CPU/GPU engine doesn't.
The quirk worth noticing: the smaller quant wins on both axes
Normally a lower-bit quant trades some quality for speed and memory, and an 8-bit quant is the "safer, slightly slower" option. That's not what Huang's own numbers show. On every single model tested, W4A16 beats W8A8 on both tokens-per-second and RAM at the same time — Qwen2.5-1.5B-Instruct runs 40% faster (19.55 vs 13.99 tok/s) and uses 27% less memory (1.6 GB vs 2.2 GB) at W4A16 than at W8A8. Gemma 3 4B IT shows the same pattern: 8.40 vs 5.33 tok/s, 4.0 GB vs 5.5 GB. There's no free-lunch story where the bigger quant buys you something back — on this specific NPU runtime, for these models, W4A16 is not a tradeoff, it's just better, on every axis this benchmark measured. The README doesn't explain why the NPU's compiler favors that path over W8A8 specifically; we're reporting the empirical result Huang measured, not inventing a mechanism for it.
Why an NPU-first spotlight, not another GPU rig
Every "Edge in the Wild" build we've covered so far has been CPU, GPU, or unified-memory APU hardware running llama.cpp or a llama.cpp-adjacent stack. Rockchip's NPU path is a genuinely different bet: dedicated silicon that only does one kind of math, in exchange for running a real chat model at usable speed on hardware with no discrete GPU line item at all, and a RAM footprint (1.6–4.0 GB across five real chat models) that fits inside what a phone already carries. That's the same trade privateSLM makes by routing to Apple's on-device Foundation Models framework before falling back to a general GGUF engine: purpose-built acceleration paths on the same silicon almost always beat a general-purpose one, when someone bothers to wire them up. Huang did that wiring for Rockchip's NPU and published the receipts; nobody had to trust a spec sheet number over an actual measured tok/s figure.
The project is explicit about what it isn't — a "personal testing project," four stars, one fork, no claimed production hardening. That's the right amount of confidence for what's actually being shown: real hardware, real models, a real benchmark CSV in the repo, and a Docker tag anyone can pull and reproduce today.
Discuss this on the forum → — if you've run an NPU-accelerated chat model on Rockchip, Qualcomm's Hexagon, or another dedicated accelerator, we want your numbers next to Huang's.