MedGaugeAbout
What MedGauge believes, and what it promises
MedGauge measures medical AI by the work it produces. Not by how it talks about the work, not by whether its answers sound right to another model, not by how long a person would have needed. A task gives a producer static materials and asks for a structured work product. A program decides whether that product meets every requirement. The harness that produced it is named on the record.
What we believe
The subject is a model inside a harness, not a model. A result belongs to the pair, and the same model under two harnesses is two rows. How much the harness moves a result is measured as a paired difference between runs that differ in one element; we assume it neither small nor large.
Correctness does not depend on who produced the work. The expert’s product and the practitioner’s rule stay at the centre of the measurement, converted into checks on declared fields that two experts set independently. The clock does not transfer as a unit: a person’s hours say nothing about what a producer whose attempts run in parallel consumed. Expert time is a property of the task family, not a measure of the producer.
A machine’s inconsistency is part of what is examined. The same input, re-run, gives different output. Every task is run three times, a task passes only when all three runs pass, and the variation between runs is published as a property of the subject.
A grade must be recomputable by someone who was not there. A program reads declared fields of the artifact; no model grades another model and no grader reads prose. We choose this not for accuracy but because it is the same tomorrow as today. Its cost — a requirement that cannot be written on a field is not scored — is published, not hidden.
The reference is a procedure. Metrology allows a quantity to be referenced to a measurement procedure rather than a unit. Ours is a versioned grading program applied to a fixed artifact contract. We invented no units; we moved the reference.
Difficulty belongs to the task. It is computed from the task’s structure before any run and never adjusted to results.
What we promise
Open method, closed set. Every rule that decides a grade is public: methodology, algorithms, schemas, statistics, seeds, all versioned. Task materials, keys, verifiers and trajectories are closed; a hash manifest of each frozen version’s holdout answers, released with its first holdout round, commits us to the keys we held.
An error is a failure. Empty response, timeout, server fault, malformed artifact: each is a failed run. Nothing is re-run and nothing leaves the denominator. If the fault is ours, that task version is voided for everyone alike, repaired and re-issued, and the voided list is published.
One failure means not passed. A subject passes a benchmark version only when every task passes in all three runs and no safety veto is hit anywhere. Not passed is published with the figures showing where marks were lost.
The number of tasks decides how strong a verdict is, not whether it is published. When every task is passed we publish the lower bound the count supports on the probability that a task of this kind is passed three times running — three tasks support about 0.37, fifty-nine support 0.95 — with its conditions beside it: built to specification, screened by difficulty, nothing re-run or excluded, a population shaped by the voiding rule. Version 0.1.0 sets no level threshold and issues no level.
No overall score. Ten dimensions are reported, each on its own; nothing is summed, weighted or averaged across dimensions, families or versions. Rows are ordered by holdout stability, compared in pairs on the same tasks, and grouped by letters that may overlap, because “not shown to differ” is not transitive; cost per success reorders only adjacent rows with identical letters.
Safety stands apart. Fabricated evidence, altered inputs, a smuggled correction, an obeyed planted instruction, an unsafe recommendation: each is a veto. A veto fails the run, is counted under its own heading and marked on the row, and never enters a merit figure.
Cost is reported in units that can be checked. Tokens, actions and wall-clock time are recorded for every run. Where a price table exists the amount is published with its version and compares prices paid, not resources used; self-hosted and open-weight subjects show amount not applicable.
Nothing is published that cannot be recomputed. Every stored artifact can be re-graded; a re-grade that does not match the record exactly withdraws it. Where a link in the chain from a score to its source clause has no measured uncertainty yet, we say structurally traceable and no more.
We say what we have not done. The site is live and every leaderboard is empty; the task sets are small. Standardised semantics for products and scenarios, and their validation, are a plan, not a finding. We will not call a plan a result or a simulation a measurement. How often people fail in the ways machines do was not found in the sources we reviewed, so we make no comparison.
We claim no originality. Every element here is a definition, a rule or a procedure that can be checked against the methodology. A rule found wrong is changed in a new version with a non-comparability notice, never quietly.
A closed set of tasks, an open method, a program that decides, a harness that is named, three runs that must all pass, vetoes kept apart, no totals, and a bound whose strength we state plainly. The rest is work.
Version 0.1.0 · MedGauge Initiative · rests on MedGauge Methodology, version 0.1.0 (medgauge.org/methodology)
See the methodology for the normative rules and updates for how they change.