Five of six frontier LLMs fail 5-digit multiplication. A transformer that was never trained gets it 100% right.

Give six frontier language models the same 500 random five-digit multiplication problems, no scratchpad, no tool calls, just a direct answer at temperature zero, and five of them score below 30%. None reaches 90%. Then run the same 500 problems through a transformer that was never trained on anything — no gradient descent, no dataset, not one training step — and it gets all of them right. Every one, at every operand length the test covered, three digits through seven. The trained models get steadily worse as the numbers get bigger. The untrained one doesn't move, because "trained" is the wrong word for what happened to its weights. They were computed.

This is the work of independent researcher Rob Porter, published across three posts on his blog Out of Distribution between July 23 and August 13, 2026 — the most recent of them landing inside our research window for this edition. The tool behind it, torchwright (MIT license, source on GitHub), is a compiler that turns an ordinary Python computation graph into the weight matrices of a real transformer checkpoint. Not a synthetic training set that teaches a model to imitate an algorithm — the algorithm's own logic, hand-derived and written directly into the weights. The output loads through vanilla Hugging Face `transformers`, no custom model code, because it's built on a stock Phi-3 architecture — the same architecture family privateSLM already ships in its catalog as Phi-4 Mini.

The question: is bad arithmetic a training problem or an architecture problem?

Porter's starting point is a genuine open question in the field: transformers are proven Turing-complete under idealized assumptions, so in principle nothing stops one from computing exact arithmetic. In practice, LLMs are notoriously unreliable at it without a scratchpad or tool call. Is that a limitation of the architecture, or just a limitation of what gradient descent happens to find? Porter's approach sidesteps the debate entirely: instead of asking what a transformer can learn, calculate the exact weights necessary for a transformer to implement a specific algorithm. If the weights exist and the forward pass produces the right answer, the architecture can express the algorithm — full stop, no appeal to training required.

The idea of hand-constructed transformer weights isn't new; Porter cites Tracr, which compiles a language called RASP into transformer weights, as prior art. Porter built his own compiler instead of reusing Tracr because he wanted to express arbitrary computation graphs in ordinary Python rather than RASP's own language. The substrate he started from was deliberately minimal: FFNs with ReLU, a hand-designed position encoding, and — notably — no normalization layers, chosen purely because they were easier to derive exact constructions for. Only later did the compiler target a real, standard architecture.

How a computation graph becomes attention weights

The residual stream — the running sum every transformer layer reads from and adds to — is the only shared memory a transformer has. Torchwright treats it like a whiteboard with numbered columns: every value a computation graph produces claims a block of columns for as long as something downstream still needs it. Because a transformer layer's skip connection means each sublayer can only add to the stream (out = in + f(in)), the compiler has exactly two primitive moves — write a new value into zeroed columns, or free a column by writing that value's negation into it once nothing needs it anymore. Addition of two values falls out for free: route both into the same columns and let the skip connection sum them.

From there Porter built up a library of primitives — comparison via a token-embedding-style dot-product trick, lookup tables via wide FFN layers, one-hot routing — and composed them into full algorithms. The result, from the project's own README:

from transformers import pipeline

generate = pipeline("text-generation", model="binary_increment_hf_bundle")
print(generate("1011\n", return_full_text=False)[0]["generated_text"])
# 1100

An ordinary Phi-3 checkpoint, loaded with the standard Hugging Face pipeline, incrementing a binary number correctly. Every weight in it was computed by the compiler from a Python description of binary addition with carry propagation. Nothing about that checkpoint's file format, tokenizer, or loading path distinguishes it from any other Phi-3 GGUF conversion candidate.

Four calculators, four places to hide the work

Porter's second post pushes the same compiler at multiplication — the operation where trained LLMs degrade fastest. He built four separate calculators, all accepting the same A op B\n input grammar and all producing byte-identical answers, that differ only in where they put the serial work carrying digits requires:

• Grade-school — the standard column-by-column algorithm taught in school, with carries propagating right to left. The published three-digit checkpoint compiles to 27 layers at d_model=2048, d_hidden=4096.
• Hardware-style — borrows carry-lookahead and carry-save reduction from digital circuit design, trading width for depth. Two layers deeper than grade-school at three digits (29 vs. 27), but the trade inverts as operands widen: at ten digits it needs only 36 layers versus grade-school's 43.
• Scratchpad — moves the serial dependency out of the network's depth and into the generated token stream itself, emitting intermediate carry/digit records instead of jumping straight to the answer. Layer count stays essentially flat (18) as operand length grows, because the work is now paid for in tokens, not layers.
• Memorizing — skips computing the answer at all and stores every possible input/output pair directly, trading computation for a lookup table. It works, but the table grows roughly tenfold per added operand digit; Porter's extrapolation puts a three-digit version at around 700 million parameters and 200 layers, so the published checkpoint stops at two digits.

All four were exhaustively checked against their full input domains — the grade-school, hardware-style, and scratchpad checkpoints against all 3,000,000 possible three-digit expressions each, the memorizing checkpoint against its smaller 30,000-expression domain. Every one of them got every expression right, on every check.

Versus the frontier

The number that matters for a claim like "exact multiplication fits inside transformer weights" is how it stacks up against models people actually use. Porter ran the same test — direct-answer multiplication, reasoning disabled, temperature zero, four retry attempts allowed for malformed responses — against six frontier LLMs and his own seven-digit grade-school checkpoint, sampling 500 fresh expressions at each operand length:

Source: ood.dev, "A calculator, compiled into a transformer," Aug 9 2026. † 42% of Claude Opus 5's six-digit attempts never produced a parseable direct answer after four retries and were scored wrong — the model kept showing its work despite instructions not to.

Qwen 3.7 Max is the standout among the trained models, holding on well past the others for reasons Porter says he doesn't know. Every other model is at effectively zero by seven digits. The compiled checkpoint — the same one, unchanged, used for every column — never drops below 100%. Porter is careful about what this does and doesn't prove: it's not a general-capability result, since the compiled model was built for exactly this input grammar and a fixed maximum operand length, and a large enough lookup table could trivially "solve" any finite domain. The actual claim is narrower and, we think, more interesting: exact multiplication can be implemented in transformer weights directly, using an ordinary decoder-only architecture, and none of the six frontier models tested reliably finds that same computation through training, even though the capacity to express it was apparently there the whole time.

It's not a research artifact. It's a Phi-3 checkpoint.

The published three-digit grade-school checkpoint, physicsrob/torchwright-calculator-simple-max-digits-3 on Hugging Face, reports "architectures": ["Phi3ForCausalLM"] in its config — the same model class as Microsoft's Phi-4 Mini, which privateSLM already ships in the Generalists category of our own catalog. It has a genuinely tiny 18-token vocabulary (digits, an operator each, a newline, and a couple of special tokens — this checkpoint has never seen the word "the"), 27 layers, and roughly a billion parameters in fp32, none of it quantized. It is not a chat model, it fails every one of our CATALOG RULES on purpose (single-task, non-instruct, a vocabulary too narrow to hold a sentence), and we're not adding it to anything. What it demonstrates is more useful to us than a catalog entry: the architecture our engine already runs, unmodified, is expressive enough to host exact, guaranteed-correct arithmetic with zero training in the loop — if someone converted this checkpoint to GGUF, the same llama.cpp build behind privateSLM's chat models would very likely load it and produce exactly the digits the compiler put there, on the first try, forever.

Porter didn't stop at calculators. The day before we wrote this — August 13, 2026 — he published a third post, "Doom, compiled into a transformer": a 38-layer Phi3ForCausalLM checkpoint that takes level geometry and a player position/viewing angle as its prompt and generates the pixels of the frame Doom's own renderer would have drawn — the original rendering algorithm translated into transformer weights, not imitated by training on rendered frames. Same compiler, same "no training anywhere" constraint, same stock architecture. It's a stunt, and Porter treats it as one, but the fact that it's the same trick as the calculator — not a special case — is the actual point: if an algorithm can be written down as a fixed computation graph, torchwright can, in principle, put it inside a transformer's weights with a correctness guarantee no amount of fine-tuning gives you.

What this means for us

Nothing here ships in privateSLM this week, and nothing here contradicts anything we've written about training, quantization, or benchmark gaps in trained chat models — this is a different category of object entirely. But it's a genuinely new data point in a debate our audience cares about: when a small on-device model gets arithmetic wrong, is that the ceiling of what a 3B-parameter transformer can do, or the ceiling of what gradient descent happened to find for it this time? Porter's answer, backed by an exhaustively-verified 3,000,000-expression checkpoint and a head-to-head loss against six current frontier models, leans hard toward the second. That's a more optimistic story for small models than it sounds — it means the architecture privateSLM already runs on your phone has more headroom for exact, reliable computation than its current training-derived weights are using. Whether anyone builds a practical hybrid — a trained chat model with compiled subroutines stitched into the same weight space for the operations it can't be trusted to learn — is an open problem. Porter's compiler is, at minimum, proof the substrate can hold it.

Discuss this on the forum → — if you've tried GGUF-converting a torchwright checkpoint yourself, or found another compiled-weights project we should know about, tell us.