Measured on-device inference.
Every number reproducible; every failure recorded.
Data
The whole corpus, in full, free. Downloading it and disagreeing with us is the
point — a benchmark nobody can check is a benchmark nobody should trust.
measurements — one row per observation, 229 of them,
including the 19 that failed. Carries the full record as canonical
JSON in payload alongside denormalised columns.
recipes — every configuration measured, content-addressed by
recipe_id. The same recipe always hashes to the same id.
graph_fingerprints — structural summaries of each model: op
histogram, dtypes, shapes, attention variant, norm type. No weights, no customer
data.
Reading it
import duckdb
duckdb.sql("SELECT * FROM 'measurements.parquet' WHERE outcome = 'success'").show()
Caveats that travel with the data
Every row records harness_version. Rows from different
versions are not necessarily comparable; that is why the field exists.
Every row records the host conditions it was taken under —
thermal_state, power_source,
load_avg_1m, calibration_ratio —
so you can re-filter on stricter criteria than ours.
stress_profile is clean on every
row so far. The soak and memory-pressure rungs are not built yet.
Absent values are null with a written reason in the record's
unavailable map. Nothing is imputed, ever.