Measured on-device inference. Every number reproducible; every failure recorded.
This page exists so you can decide how much to trust the numbers without taking our word for anything. Where a claim can be checked against the data, the data is linked. Where something is unknown, it says so.
Every measurement is at least 5 timed runs after discarded warmups, and the variance is recorded. This is enforced in the type system, not by convention: a record claiming success without a timing distribution of five or more samples fails validation and cannot be written. The aggregate statistics are recomputed from the raw samples on load and rejected if they disagree, so a variance figure that did not come from measured data is not representable.
In this corpus: 10 timed runs per measurement, 3 discarded warmups, median coefficient of variation 1.5%, worst 29.2%. Warmups are discarded because the first inference includes lazy kernel compilation and delegate model caching — a real cost, but a different one from steady-state latency.
Some rows come from Qualcomm AI Hub, which runs a submitted model on real phones in its own racks. They are marked 3P wherever they appear, and the mark's tooltip names the job that produced them. Everything on this page about our gate, our thermal probe and our warmup policy describes our devices and says nothing about theirs.
What is genuinely theirs, and therefore unknown to us:
One thing they give us that no silicon vendor publishes about a rival: per-node compute_unit, so silent fallback is measurable on a second vendor with the same method as the first.
A caveat that cost us a pass: profile jobs synthesize their own random inputs. For a model taking float inputs that is harmless. For one taking integer indices — token ids, token types — a random value is out of bounds for the embedding it indexes, and the job fails with an error that looks like the device refusing the model. It is not. We now refuse to submit such models rather than record a failure against hardware that did nothing wrong, which is why the text models measured on our own devices have no Snapdragon rows here.
Steady-state latency is measured after discarded warmup runs, because the first call includes lazy kernel compilation and delegate caching — a real cost, but a different one from the per-inference cost that dominates a busy app.
That reasoning is standard and it quietly hides something large. A user opening an app cold pays the model load and the first inference, and on a phone the load can be three orders of magnitude bigger than the inference it enables. So both are recorded: cold_load_ms (session creation, delegate initialisation, ahead-of-time compilation), warm_load_ms (the same with caches populated) and first_inference_ms (the first run, timed rather than binned).
The reason this is a column and not a footnote: it is not a constant offset. On one device, the CPU-only path loaded an order of magnitude faster than the path that reached the NPU — so an accelerator's speedup is partly financed by startup cost, and subtracting a fixed number would misrepresent both.
Hosted rows carry load times but not first_inference_ms: the service reports load and steady state, not the duration of the first run specifically, so it is null with that reason rather than derived.
Before every measurement the host is checked, and if it fails the measurement does not happen. The refusal is itself recorded as a row. The reasoning is that a missing number is recoverable and a wrong number is not: a run on a warm, busy, battery-powered laptop produces a figure that looks entirely reasonable and is worthless.
We found the value of this the hard way. Measuring the same recipes on a contended host and then on a quiet one moved median latency by about 10% and moved the coefficient of variation by an order of magnitude — from 2.8–31.7% down to 0.3–3.8%. The contended numbers were not obviously wrong. They were quietly wrong, which is worse.
Apple Silicon exposes no unprivileged temperature reading. pmset -g therm returns nothing, the Intel xcpm sysctls do not exist on ARM, and powermetrics requires root. Rather than publish a number we cannot obtain, we measure the thing we actually care about: a fixed deterministic matmul is timed immediately before each measurement and compared against this unit's own recorded healthy throughput. A machine that has got slower is throttled or contended, whatever any sensor claims.
The baseline is a low percentile of recent healthy samples, not the fastest time ever seen. An all-time minimum turned out to be a ratchet: one unusually quiet moment recorded 7.63 ms against a normal healthy figure near 8.4 ms, and after that the machine could never satisfy its own threshold again. The ratio is recorded on every row as calibration_ratio, so you can re-filter more strictly than we did.
cpu_temperature_c is null on every row, with the reason attached. That is the honest state of the art on this hardware.
When a delegate claims part of a graph, the ops it could not claim quietly run on CPU. Nothing errors. That is the failure mode this project was built to find, and measuring it turned out to be subtler than expected.
We report it two ways, because they answer different questions:
Both are necessary because the two differ materially. On ViT-base, moving from optimisation level disabled to all cut CPU node count from 244 to 86 and lifted the accelerator's time share from 53.7% to 81.5%. A fallback figure taken from the unoptimized graph is a faithful measurement of a graph that never runs.
This matters for reading the matrix: the FLOP-share column does not predict whether the accelerator helps. It reads 97–99.8% for every model we have measured and separates nothing. The as-run time share does separate them. We publish both, labelled, rather than quietly dropping the one that turned out to be the wrong tool.
One further caution: time share is not an efficiency measure. It says where the time went, not whether sending that work to the accelerator was a good idea. A model can spend 70% of its time inside the accelerator and still be twice as slow as plain CPU.
Quantized recipes carry a cosine similarity against the fp32 PyTorch reference captured at export time. It exists so the quantization column has a cost beside its speedup — “int8 is twice as fast” with no cost column is a half-truth.
It is not task accuracy. A model can hold cosine 0.999 globally and fail badly on the one slice a customer cares about. Real eval-set accuracy is a separate tier that is not built yet, and the field is null with that reason attached rather than filled in with this number.
Parameter counts here are the initializers in the exported ONNX graph, which is not
always the number on the model card. Llama-3.2-1B reports 1498.5M
against a card value near 1236M, and the 262.5M difference is exactly one embedding
matrix (128256 × 2048): torch.onnx.export un-ties the tied
embedding, materialising it once as embed_tokens.weight and again as the
output projection's weight.
That is a real property of the artifact, not a measurement error — those bytes are in the file and a deployment pipeline moves them. In fp32 it is about 1.05 GB of the 5.6 GB artifact, roughly a fifth of it, spent on a duplicate. We report the count as exported rather than the card value because the former is what ships.
A decoder has no single latency. Prefill consumes the whole prompt in one compute-bound pass and yields time to first token; decode then emits one token at a time, bound by memory bandwidth as it re-reads the KV cache and the weights. We report both distributions separately and never average them, because the average describes neither phase — and that average is the figure most often quoted.
The KV cache is threaded explicitly through the graph: each step feeds the previous
step’s present tensors back in as past. Without it every
step would reprocess the whole sequence, decode cost would grow quadratically, and any
tokens-per-second figure would be fiction.
Quality is exact token agreement, not a float tolerance. Greedy
decoding through the exported graph must reproduce the identical token sequence to
fp32 PyTorch. That check earned its place immediately: our first working export scored
25% because position_ids was not an explicit graph input, so the tracer
baked prefill’s rotary positions in and every decode step computed them at the
wrong offset. The text stayed fluent; it was simply different text. Every tolerance on
logits would have passed it.
sustained_tok_s_5min is null everywhere. Sustained throughput is a
thermal-soak measurement belonging to the stress bench, and a thirty-second stand-in
would misrepresent it.
Each measurement runs in a fresh child process which reports its own peak resident size, so the figure is attributable to that recipe alone. The parent deliberately does not read the aggregate child usage, which is a running maximum across every child that ever exited and would blame one heavy recipe's memory on every lighter one measured afterwards.
A platform detail worth stating because it is a classic silent error: ru_maxrss is reported in bytes on macOS and in kilobytes on Linux. Nothing in the API says so, and both produce believable numbers. There is a regression test that allocates a known amount and checks the result lands near it.
19 of 229 rows in this corpus are failures, and they are in the data download along with everything else. A recipe that will not lower, or a delegate that aborts the process outright, is a fact about the toolchain that is expensive to rediscover.
Both stages of every measurement run out of process, which is what makes this possible: a delegate that crashes the interpreter becomes a recorded row instead of a dead run with nothing to show. We found this necessary rather than theoretical — one vendor flag combination aborts the process on every model we have tried it on.
The gate that everything else depends on. Each exported model ships with the fp32 PyTorch output for a fixed input, and the runtime's fp32 output must reproduce it to a cosine of 0.9999 or better. This is what catches a silently wrong kernel — the failure that would make every number here worthless while every latency figure still looked plausible.
Stated plainly, because a methodology page that only lists strengths is marketing.
The corpus is append-only. Measurements are never edited: a re-measurement is a new row carrying a new harness_version, and both rows stay. If we get something wrong, the record of having got it wrong stays too.
We have already used that. An earlier version of this atlas explained the accelerator's losses using the FLOP-fallback figure. That explanation was wrong, and the section above says so.