The most expensive machine scored 10/10 — and got Mercury's size wrong by 40%

A $1,000 mini PC just aced a 10-question offline-AI benchmark — 10 out of 10 — while stating, in the same breath as its correct diameter figure, that Mercury is smaller than the Moon. It isn't: Mercury's 4,879 km diameter is about 40% larger than the Moon's 3,475 km. The grading rubric only checked whether the model named the right planet, so the false comparison rode along inside a perfect score, undetected. That's the twist inside a much bigger project: an independent builder ran the identical offline test — same ten questions, same temperature-zero determinism, same MD5-verified prompts — across three tiers of hardware, from a $90 handheld board to that $1,000 machine, to find out what running AI locally actually costs in memory, not just in dollars.

Source & credit: the project, code and writeup are the independent work of Mayukh (GitHub @mayukh4, YouTube Mayukh Builds). Full repo, MIT licensed: github.com/mayukh4/unihiker-offline-ai.

Watch "How Little RAM Does Local AI Actually Need?" on YouTube → — channel Mayukh Builds.

Three rungs, one identical test

The ladder runs from a $90 handheld board to a four-figure mini PC, with a Raspberry Pi 5 in between as the "normal person's" reference point. Every rung ran the same ten fixed questions — general knowledge, arithmetic, a telemetry-to-JSON structuring task, a two-turn dialogue memory check — at temperature 0, with prompts MD5-hashed to confirm no drift between runs.

Source: mayukh4/unihiker-offline-ai, README.md and RESULTS.md. *See "the wrong answer inside a perfect score," below — one of the Beelink's ten correct-per-rubric answers contained a false claim the scoring didn't catch. DFRobot lists the UNIHIKER M10 at $89.90 on its own product page.

Getting llama.cpp onto a board that predates it

The UNIHIKER M10 is a $90 handheld from DFRobot built for classrooms — a 2.8" touchscreen, a quad-core Cortex-A35 (the builder identifies the SoC as a Rockchip RK3308) at 1.2 GHz, and 512 MB of RAM, of which about 423 MB is actually usable after the OS takes its cut. It ships Debian 10, with glibc 2.28 and cmake 3.13.4 — a standard llama.cpp build wants glibc 2.35 or newer and cmake 3.14 or newer, and the board doesn't have enough RAM to compile anything itself anyway. So compiling on the board wasn't an option, and neither was compiling normally and copying the binary over.

The workaround: cross-compile a fully static musl binary — no dependency on the board's outdated glibc at all — inside an Alpine Linux ARM64 Docker container on a separate machine, then transfer it over. It's the kind of unglamorous systems work that decides whether "run an LLM on this" is even possible before a single token gets generated, and it's why this project is really two things at once: a build recipe for hardware most on-device-AI coverage skips over, and a benchmark once that recipe works.

14.6x the memory for 15% more speed

Once both boards could run something, the comparison that matters most for anyone deciding how much hardware "local AI" actually requires is the Pi 5 versus the M10, not the Beelink versus either. The Pi's Gemma 4 E2B model used 3,906 MB of peak RAM against the M10's 267 MB — 14.6x more memory — to generate at ~5.4 tok/s against the M10's ~4.7 tok/s, roughly 15% faster. Memory bought almost nothing in raw generation speed.

Where the Pi actually earns its extra RAM is prompt processing, not generation: it reads and encodes an incoming prompt about 7.3x faster than the M10, per the builder's own measurements. That's the real practical gap between the two boards — how long you wait before the model starts answering, not how fast it types once it starts — and it's the opposite of what a single "tokens per second" headline number would tell you.

The wrong answer inside a perfect score

The Beelink's 35B-parameter MoE model (Qwopus3.6-35B-A3B, a community coder-tuned build quantized to IQ4_XS by Jackrong, running via an MTP-enabled llama.cpp build for self-speculative decoding) generated at roughly 4x the Pi's throughput and answered every one of the ten questions in a way the grading rubric marked correct — including the one about the solar system's smallest planet. Its full answer named Mercury correctly, gave its diameter as 4,879 km — also correct — and then added, in the same sentence, that this makes Mercury "slightly smaller than Earth's moon." The Moon's diameter is 3,475 km. Mercury is the larger of the two by roughly 40%, not the smaller.

Because the scoring rule for that question only checked whether the model named the right planet, the fabricated comparison attached to an otherwise-accurate figure went straight through as a correct answer, on the single most capable, most expensive machine in the test. The builder's own line on it, from RESULTS.md: "Bigger models do not stop being wrong; they get better at sounding right." Neither the $90 board nor the Pi made this specific mistake, if only because neither elaborated with an unprompted comparison in the first place — a smaller, more literal model has less room to confidently invent something adjacent to a right answer.

Why it matters here

Two separate lessons sit inside one project, and they cut in opposite directions for anyone deciding how to run AI without shipping their data to a server. The memory number argues for going small: a $90 board that fits in a pocket can hold a working, if limited, offline model in a quarter-gigabyte of RAM, for a real but modest accuracy cost against a $1,000 machine 20x its price. The Mercury answer argues against trusting any single benchmark score, small model or large: a rubric that checks the named answer and nothing else will wave through a confident, fluent, wrong elaboration every time, on any tier of hardware, and the more capable the model, the more convincingly that elaboration reads. Neither finding is an argument this blog gets to make about its own catalog for free — "10/10" and "beats the incumbent on every published benchmark" are exactly the kind of claims this same failure mode can hide inside, which is the whole reason we cite the underlying number and the source, not just the score.

Discuss this on the forum → — run your own offline-AI benchmark on hardware this small? Tell us what broke first.