An 80B-parameter model just ran in 4.3 GB of RAM — on an unmodified Mac, no cluster, no cloud

Qwen3-Next-80B-A3B is an 80-billion-parameter model. Its 4-bit GGUF-equivalent weights take up 42 GB on disk. An independent builder going by leonickson1 got it generating text on an ordinary Apple Silicon Mac using a peak of 4.3 GB of RAM — under a tenth of the file size — by never loading most of the model into memory at all.

Source & credit: the runtime, the benchmarks, and the writeup are the independent work of leonickson1 (GitHub handle; the same person publishes a companion free iOS app, "Priv AI," under the name Monish Soundar Raj). Everything below is our summary and framing — read the original: github.com/leonickson1/Swiftlet (Apache 2.0). Also discussed at length on Hacker News (312 points).

The trick: never hold the model you're not using

Qwen3-Next-80B-A3B is a mixture-of-experts (MoE) model — the "A3B" in its name means roughly 3 billion parameters actually fire per token, even though the full network stores 80 billion across many specialist "expert" sub-networks. Every existing engine we know of, including the one privateSLM ships, still has to memory-map or fully load that entire 80B-parameter file before it can serve a single token, because it doesn't know in advance which experts a given token will need. Swiftlet's approach is to stop assuming that: it streams each token's specific expert weights off SSD on demand, keeping only the currently-needed slice of the model resident in RAM. The README's own numbers, measured on an M5 Mac:

And on a phone: the README states the 35B model "also runs on an iPhone 17 in about 2.5 GB of RAM, at about 1 tok/s today." One token per second isn't fast — for context, it's roughly what a 7B model gets on much older hardware with no streaming trick at all — but the number that matters here isn't the speed, it's that a 35-billion-parameter model produced any tokens whatsoever on a phone without a cluster, a cloud API, or the model ever existing in RAM as a single contiguous 18 GB block.

The honest caveat: this speeds up decode, not the hard part

The Hacker News thread is worth reading specifically because commenters pushed on the weakest part of the pitch, and the author didn't dodge it. Several pointed out that tokens-per-second during generation isn't the real bottleneck for anything beyond short chat messages — prompt processing (prefill) is, because it has to touch far more of the streamed weights up front. leonickson1's own reply: "the honest answer is this is good for chat-length stuff, not for feeding it a 10k token document" — with a 10,000-token prompt reportedly taking on the order of half an hour to process before generation even starts. That's a real, load-bearing limitation, not a footnote: it means Swiftlet currently solves the "can this model exist on this hardware at all" problem without solving the "can you actually use it for long-context work" problem. The author also clarified that tuning the on-disk cache size barely moves decode speed either way — the bottleneck there is GPU dispatch overhead, not SSD read speed, with the cache hitting anywhere from 43% to 70% of requests at similar throughput regardless of size.

Not claiming to be first, and saying so

The README credits prior art directly rather than presenting the idea as novel: "ANEMLL showed a 397B MoE streaming on an iPhone 17 Pro as a proof of concept in early 2026," and a dedicated section describes specific design lessons borrowed from two other projects, drumih/turbo-fieldfare (a Gemma-streaming experiment) and JustVugg/colibri. That kind of explicit sourcing is unusual enough in a 312-point Show HN that it's worth calling out on its own — the pitch here is "a usable, documented implementation of an idea a few people were independently circling," not "we invented weight streaming."

Why this matters more than a clever demo

Every model in privateSLM's own catalog is capped by the same constraint Swiftlet is attacking: to run a model at all today, our engine has to hold its full weights resident, which is exactly why our biggest catalog entries top out around 8 GB of RAM and nothing close to an 80B-class model is on the table. Weight-streaming for MoE architectures is a structurally different answer to that ceiling than "wait for phones to ship more RAM" — it decouples how big a model's total parameter count can be from how much memory running it requires, which is precisely the trade our "the model comes to your data" pitch depends on getting better over time. We're not adding this to privateSLM this week: it's a from-scratch Swift/Metal runtime, not a llama.cpp patch, built specifically for MoE architectures our current engine already loads a different way, and the prefill bottleneck the author flagged himself is a real gap before it's chat-ready. But it's a concrete, working, 488-star proof that the RAM ceiling on how large a genuinely local model can be isn't as fixed as it looks from inside a single-engine catalog.

Discuss this on the forum → — tried MoE weight-streaming yourself, or hit the prefill wall leonickson1 describes? Tell us your numbers.