MedGauge

MedGauge Methodology

Version 0.1.0 · 2026-10-29 · MedGauge Initiative · medgauge.org

Closed set, open method. This document is the normative text of MedGauge. It fixes what MedGauge measures, how a measurement is built from its smallest unit, how every form of medical AI is read as a subject, how the measurement is shown to be valid, how the registries that carry it are edited, how the initiative that maintains them operates, and how all of this changes over time. Every page of medgauge.org, every schema, every validator rule and every published number is derived from this text and is checked against it.

Normative words: must and never state requirements; may grants permission. A sentence without them states a definition or a fact of construction. Nothing in this document is a result.

Cite as: MedGauge Initiative. MedGauge Methodology, version 0.1.0 (2026). https://medgauge.org/methodology/text/

Abstract

Medical AI is learned from records of practice, and records hold actions, not the rules behind them. A system that has acquired those rules is not bound to apply them the way a person does, so a measurement defined on process has no stable referent. MedGauge therefore measures the work product: a structured artifact produced from static materials, graded by a program against explicit, sourced rules, with the harness that produced it named on the record. The smallest unit of measurement is the check: one gradable requirement on one artifact field, with a stated condition, polarity, form and provenance. Checks compose by published rules into checkpoints, runs, tasks and frozen benchmark versions. The observation is one run of one subject on one task; every task is run three times, and a run error is a failed run, never re-run and never removed. Ten dimensions—completion, stability, efficiency, difficulty, optimality, clinical compliance, quality control, exploration, auditability and robustness—are reported each on its own; there is no total. Ten task labels describe what a task is like and feed a structural difficulty function; difficulty is a property of the task, never of the subjects that tried it. Safety constraints are vetoes, reported in their own columns and never averaged into merit. The conclusion on a benchmark version is an exam: a subject passes only if every task passes in all three runs with no veto, and the strength of that conclusion is the exact one-sided lower bound the number of tasks allows. Rows are compared in pairs on the same tasks and grouped by a compact letter display. Levels are to be thresholds on the lower bound; this version sets no threshold and issues no level. Every form of medical AI—a model inside a harness, a harness, a tool server, a skill, a prompt—is a subject with a declared responsibility scope, and a component’s effect is a paired-run difference, not a row. Task materials, keys, verifiers and trajectories are closed; every rule that decides a grade is public, and every published number can be recomputed from the public record set. The registries that carry the measurement form a reference work with an entry standard, sourcing rules and a review cycle; the website is their rendering. The initiative that maintains them operates by recorded roles, separated duties, pull-request decisions and versioned change, and publishes each frozen object with a persistent identifier. The methodology is itself a versioned object: a change that alters any published number carries a non-comparability notice and, while the version is below 1.0.0, a new minor version.

Part I — Measurement

1. Position

1.1 From practice to measurement

P1 (Origin). Present medical AI is learned from records of practice. A record holds what was done and what followed; it does not hold the rule by which it was done. The rules are held by practitioners. A system that carries out a task has therefore acquired, from records, rules that the records never state.

P2 (Divergence). A system that holds rules is not bound to apply them the way a person applies them. As systems develop, the way a task is carried out departs from present practice and from any one person’s process. A measurement defined on process—does it work the way a clinician works—has no stable referent.

P3 (Baseline on the work). The measure is stated on the work product: what a correct product must contain, given the materials, under a stated standard. The baseline is independent of the producer.

P4 (Methodology, not method). A benchmark is a method: a recipe for one measurement. A measurement end that must hold across task families, producers and years is a methodology: definitions, a unit, composition rules, validity procedures and change rules.

P5 (Seriousness). A methodology is serious when it names a smallest unit, when every published number has an explicit path back to that unit, and when every claim carries a procedure by which someone else checks it.

P6 (Any form). The methodology reads every form of medical AI as a subject on the same terms: a model, a model inside a harness, a harness, a tool or tool server, a skill, a prompt.

1.2 Three bottlenecks

Three bottlenecks sit on the chain from practice to a measured system. The learning-material bottleneck: what is recorded is thinner than what is known. The measurement bottleneck: exam scores and judgements of whether an answer sounds right cannot read an agent, a skill or a tool server, and cannot tell answered correctly from can do it. The failure-mode bottleneck: the failure modes recorded for machine producers—idling, fabricating, overreaching, not stopping—have to be caught by the measurement itself; how often people fail in the same ways is not known, and MedGauge makes no comparison between the two. MedGauge owns the second bottleneck and shares the third.

1.3 Thesis

Medical AI is measured by the work it produces, at the granularity of a single check, by a program, against explicit and sourced rules, with the harness named, under a closed holdout and an open method. Everything that follows is the construction of that sentence.

2. Terms and notation

Term Definition
artifact A structured work product (fields, files, logs) produced from static task materials
producer Whatever produced an artifact; never visible to the grader
subject Anything a result is attributed to (section 10). A model is always measured inside a harness: model × harness, each with its version, is one subject, and the same model under another harness is another subject
harness The execution layer a model runs inside: runtime, tools, limits, sandbox, prompt, run records
component A skill, prompt, tool or MCP server registered as a subject with a declared scope
check One gradable requirement on one artifact field (section 4)
checkpoint One check instantiated on one task
hard gate (veto) A requirement whose violation fails the run outright; vetoes fall into the categories of M6
task Static materials, an artifact contract, hard gates and checkpoints, graded by a verifier; each task carries a version
run One execution of one subject on one task under fixed conditions; the observation unit, indexed (subject, task, run number)
run error A run that ends without a gradable artifact or with an interface, network, server, time-out or output error; it is a failed run and carries an attribution (G7)
action One step of a run as the harness version itself defines it (G8)
declared action granularity The unit in which the task author counts the required steps of a task, stated in the task metadata and independent of any harness (G8)
task family A set of tasks sharing a scenario, artifact contract and verifier; the unit of a benchmark card
source group A sub-directory of a family by source of material; never a unit of score
benchmark version A frozen set of tasks of one family with its subsets, checks, verifier and function versions
subset dev, holdout or calibration; reported separately, never pooled
dimension One of the quantities M1–M10, each reported on its own (section 6)
task label One of the properties T1–T10 of a task (section 7); labels describe tasks and never score
exam outcome Passed or not passed for one subject on one subset of a benchmark version (section 8.3)
claimable lower bound The one-sided 95% Clopper–Pearson lower bound that an exam outcome passed supports, 0.05^(1/n) for n tasks (section 8.3)
price table A versioned table that converts usage into an amount, published with the results that use it
capability A registry entry defined by input → required artifact → grading, never by a workflow
level An ordinal statement L0–L5 about one (model, harness, capability, benchmark version); no level is issued in version 0.1.0 (Appendix B)
grading reference A public registry record of one external clause a check rests on
run record The per-run record of model, interface, actions, tool calls, tokens, timing, amount, error attribution, hashes and grade (Appendix C)
result record The site-verified record of one subject on one benchmark version
task rules The rules supplied inside the materials of a task (quality-control rules, audit rules, an item specification, a data dictionary); the source of a task-internal check (4.1)
expert key card The closed record, per task, of the expected key and why, what counts as over-reporting, borderline cases and their decision, the grading references relied on, and each key expert’s specialty and qualification (gate 3)
task submission A contributed task described in the form of schemas/task-submission.schema.json (section 23)
task package The closed, versioned bundle of one task’s materials, artifact contract, checkpoints and verifier, as held in the private grading repository or delivered under agreement
site-run The run mode in which the initiative runs a subject on its own infrastructure (10.5)
submitted-run re-grading The run mode in which an enrolled party runs a watermarked task package and returns artifacts and run records, which the initiative re-grades (10.5)
evaluation round A set of subjects run on a set of benchmark versions under fixed conditions (section 25)
tool registry The registry of tools, tool servers, skills and prompts registered as components (section 15)
standards registry The registry of external standards, guidelines, classifications and formats the site cites (section 15)
difficulty function The published, versioned function that maps a task’s recorded components to a difficulty class (M4, Appendix A)

Notation. A benchmark version V of family F has task set T(V); a task t has hard gate set G(t) and checkpoint set C(t) with m(t) = |C(t)| checkpoints; a run is indexed r ∈ {1, 2, 3} (k = 3). pass(t, r) ∈ {0, 1} is written y_r in the dimension definitions; s(t, r) ∈ [0, 1]. For a task with n runs of which c pass, p̂_t = c / n. Subset membership is written t ∈ dev(V), holdout(V), calib(V).

Identifiers. A registry object is identified as <type code>-<short name>, in lower case with hyphens; a versioned reference is <identifier>@<version>, for example tl-read-file@0.1.0 or difficulty-function@0.1.0. Capabilities keep their domain numbering (A.1 … X.10), and models their vendor name. The type codes are:

Code Object
bm benchmark (benchmark card and its task family)
gr grading reference
hx harness
tl tool registry entry (tool, tool server, skill or prompt component)
ds dataset card
std standard
res result record
run run
tk task key (the opaque key under which a result record lists a task)
rp reproduction record
lo level override record
rm roadmap item

Versions are written as major.minor.patch for every versioned object, without a prefix.

3. Axioms and general rules

3.1 Axioms

Axiom 1 (producer-independent artifact). MedGauge measures a medical work product: a structured artifact produced from static materials. The measurement states what the artifact must satisfy; it does not depend on who produced it or how. The grader sees fields, not prose, and never the producer. The same artifact gets the same grade whoever produced it. Nothing in the method assumes a particular way of working.

Axiom 2 (closed set, open method). Task materials, keys, verifiers and trajectories are closed. Every rule that decides a grade is public.

Axiom 3 (program grading). A grade is decided by a deterministic program on the artifact’s fields. No model grades another model; no grader reads free-form prose. A requirement that cannot be expressed on a field is never scored.

Axiom 4 (the harness is part of the subject). A result belongs to a model inside a registered harness version. The same model under two harnesses is two rows; the difference between them is a measured quantity.

Axiom 5 (safety apart from merit). Safety constraints are vetoes. A veto hit fails the run regardless of the checkpoints, is counted under the constraint and the category involved, and is reported in its own column. Veto counts are never combined with another dimension and never averaged into a merit figure.

Axiom 6 (scores only from the site). Every published score is computed by the initiative, on its own runs (site-run) or by re-grading runs returned with their artifacts and run records (submitted-run re-grading). Nothing on a board is hand-typed.

Axiom 7 (sources). A rule that decides a grade rests either on material closed with the task or on a public guideline, label, classification or standard registered as a grading reference. No other source decides a grade.

3.2 General rules

G1 (observation unit). One run is the triple (subject, task, run number). Every task is run k = 3 times for every subject.

G2 (dimensions apart; no total). The dimensions M1–M10 are reported each on its own. They are never weighted and summed, and no dimension is averaged across task families. The only overall figure is one dimension pooled over the tasks of one family.

G3 (the safety veto comes first). A run that hits a veto is a failed run under M1 and is counted under M6. Vetoes are always reported apart and are never averaged away.

G4 (difficulty belongs to the task). M4 is computed from the structure of the task only. The performance of subjects never changes a task’s difficulty class. Selecting tasks or assigning classes by subject performance is not part of this methodology.

G5 (conditional quantities on successful runs). M3 efficiency and M5 optimality are computed on passing runs only, so that failing fast does not read as efficient. They are always accompanied by a quantity over all runs, the cost per success (M3), so that reporting successes only is not possible.

G6 (recomputable). Every published number can be recomputed from the public record set and the published version of the algorithm that produced it. Algorithms carry versions; a frozen benchmark version keeps the algorithm versions in force when it froze.

G7 (an error is a failure). Every run error is recorded as a failed run. It is never re-run and never removed from a denominator. Every error is recorded with an attribution. An error attributed to the subject’s side—the model, the agent, the harness, the application or the service provider—stays a failed run, and the error rate is published apart, by attribution. An error attributed to the initiative’s side—the task or the environment—is handled by the void-and-retest rule (section 8.4).

G8 (actions are counted by the harness; required steps by the task author). The unit of an action in step counts (M3) and in the redundancy rate (M5) is the harness version’s own definition, recorded with that version. These quantities are compared only within one harness version; across harnesses they are reported and never ranked. MedGauge defines no autonomy-span quantity. On the task side, the T1 horizon, the required steps of M4 and the size |R| of the required action set of M5 are counted at the action granularity the task author declares in the task metadata, which is independent of any harness. The two granularities differ, so the redundancy rate is compared only within one harness version.

4. The unit: the check

A check is one gradable requirement on one artifact field.

Element Meaning
condition (when) The state of the materials under which the check applies
field The artifact field the check reads
rule (expect) The required value or relation of that field
polarity must, must not, should, should not
check form program (computed from materials and artifact) or checkpoint (compared with a reference value)
veto Whether a violation fails the run outright
layer clinical invariant, role-bound constraint, or capability (section 4.3)
provenance task-internal rule or grading reference (section 4.1)

4.1 Provenance

A check has exactly one of two provenances.

Both provenances grade the same way. Only the second is public, reusable across families and open to contribution. A grading reference is listed on the site once a frozen version uses it and its clause has been checked on the original source.

4.2 Rules of construction

4.3 Three layers

Every check belongs to one layer, recorded on the check and on the capability it maps to.

Layer What it states Depends on the producer?
Clinical invariant What a correct work product must contain, given the materials, under a stated standard No
Role-bound constraint What the job the artifact serves forbids or requires of whoever performs it On the declared role of the task, not on the producer
Capability What a producer must be able to do to satisfy the invariants and constraints of a family Yes; it is the thing measured about the subject

Role-bound constraints derive from the responsibilities of a human role: an auditor logs, the accountable clinician corrects. They are marked role-bound so that a reader does not take them for clinical facts; in a different role the same action is required rather than forbidden. The integrity vetoes—no fabricated evidence (X.1) and planted instructions ignored (X.4)—are invariants that hold for every task. Medical-practice capabilities are named after human jobs as a reference frame for readers; a capability is defined by input → required artifact → grading, never by a workflow.

5. Composition

The hierarchy, from the unit upwards:

check → checkpoint → run → task → benchmark version (task family) → capability level → scenario

5.1 Checkpoint. One check instantiated on one task: the condition evaluated on that task’s materials, the field read from the submitted artifact. One checkpoint instantiates exactly one check.

5.2 Run outcome. For run r of task t with m(t) checkpoints:

pass(t, r) = 1  iff  every g ∈ G(t) holds  ∧  every c ∈ C(t) passes  ∧  no veto hit
s(t, r)    = (number of checkpoints in C(t) that pass) / m(t);   s(t, r) = 0 on a veto hit

This is the only definition of the continuous score s; no family defines its own. It carries no separate penalty: over-reporting lowers s only where a checkpoint states it. A veto hit sets pass(t, r) = 0 and s(t, r) = 0. A run error (G7) sets pass(t, r) = 0; its checkpoints are graded on the artifact as it stands, and a missing artifact passes no checkpoint.

5.3 Runs and the task. Every task runs three times under identical fixed conditions (G1). A run error is one of the three runs (G7).

P(t) = (1/3) Σ_r pass(t, r);     S(t) = (1/3) Σ_r s(t, r)
pass@3(t) = 1 − C(n − c, 3) / C(n, 3);     pass^3(t) = C(c, 3) / C(n, 3)     (n ≥ 3 runs, c passing)

With n = k = 3, pass@3(t) is 1 when at least one run passes and pass^3(t) is 1 when all three pass. A task is passed by a subject when pass^3(t) = 1: all three runs pass. Single runs are never published as scores.

5.4 Benchmark version. For a subset U ⊆ T(V) with tasks weighted equally:

P(V, U) = mean_{t∈U} P(t);   S(V, U) = mean_{t∈U} S(t)
PASS@3(V, U) = mean_{t∈U} pass@3(t);   PASS^3(V, U) = mean_{t∈U} pass^3(t)

dev, holdout and calibration are reported separately; calibration never enters a family mean. Leaderboard figures are computed on the holdout subset. A version is frozen: its tasks never change. A changed check re-calibrates the affected tasks in the next version, and results across versions are labelled not comparable. A task version voided under section 8.4 leaves the board for every subject; its corrected version enters a new benchmark version, on which every subject is run again.

5.5 Dimensions and statistics. The quantities above are dimension M1 (completion) and dimension M2 (stability). The other dimensions are defined in section 6, the intervals, comparisons, exam outcome and levels in section 8. Every dimension is reported on its own (G2).

5.6 Capability level. A level is defined only for the four-tuple (model, harness, capability, benchmark version), from the per-subset exam outcomes and claimable lower bounds of a site-verified result record (section 8.3, Appendix B). Thresholds are absolute, never ranks. A family linked to several capabilities yields one level per capability. Version 0.1.0 sets no threshold, so no level is issued. Setting or changing a threshold is a non-comparable change (section 30).

5.7 Scenario. A scenario capability (E domain) takes the lowest evidence state and the lowest level among the medical-practice capabilities it is composed of. L5 is issued only by a signed override record with before/after operational evidence and an external anchor pointing the same way, for scenario capabilities only.

5.8 Across families and versions. No figure is averaged or totalled across families, and none across versions. A scenario takes the lowest level of its components (5.7); it is not a mean.

5.9 Scale types. Each published quantity has a scale type, and operations are limited to those the type admits [29].

Quantity Scale Admitted Not admitted
pass(t, r), exam outcome dichotomous counting —
P, PASS@3, PASS^3, instability, the rates of M6–M10 ratio on [0, 1] means within a family version, paired differences, intervals means across families
S bounded on [0, 1], one definition (5.2) means and intervals within a family version comparison across families (m differs)
efficiency ratios to the reference solution (M3) ratio geometric mean, median, quartiles arithmetic mean
action counts, redundancy rate count; ratio comparison within one harness version ranking across harnesses
env_delta, robustness gap signed difference on [−1, 1] reported with sign and as an absolute value —
level L0–L5 ordinal ordering averaging, differences
difficulty class ordinal ordering, distribution, stratification averaging, differences
region, modality, specialty labels nominal counting, stratification treatment as a measurement

6. Dimensions (M1–M10): what is measured

Each dimension is given as definition, observation, computation, aggregation and limits. A dimension is reported only where its conditions hold (6.11).

6.1 M1 Completion

6.2 M2 Stability

6.3 M3 Efficiency

6.4 M4 Difficulty (a property of the task)

6.5 M5 Optimality of the solution

6.6 M6 Clinical compliance

6.7 M7 Quality control

6.8 M8 Exploration

6.9 M9 Auditability

6.10 M10 Robustness

6.11 Which dimensions a task reports

M1–M3 are reported for every task. M5–M8 need metadata—the required action graph, compliance checkpoints, seeded defects, undecidable points—and a task without it reports the dimension as not applicable, never as 0. M4 comes from the starred labels and structural components and serves stratification; it is not a score. M9 is a condition on every published result: a result that fails it is not published. M10 is reported only for tasks that have perturbation variants.

7. Task labels (T1–T10): what a task is like

Labels never score. They serve coverage counts, stratified reporting and, for the labels marked ★, the next version of the difficulty function (M4, Appendix A.2). Every task carries every label; where a label does not apply, the metadata says not applicable.

Label Name Values (ordered) ★
T1 Horizon single step / short (≤10 steps) / medium (≤50) / long (>50, or across sessions, or needing resumption). Steps are the required steps of the reference solution at the declared action granularity, never a harness’s step count ★
T2 Environment static materials / tool and database sandbox / stateful environment that changes over time / environment with distractors and injected content ★
T3 Specialties number of specialties required: 1 / 2 / 3 or more (multidisciplinary) ★
T4 Modalities text, structured table, imaging, waveform, pathology slide, speech, video (several allowed; the number counts) ★
T5 Omics none / one omics layer (genome, transcriptome, proteome or metabolome) / multi-omics integration (layers counted) ★
T6 Interacting parties works alone / works with a person (asks, communicates) / works with other agents ★
T7 Knowledge currency before the knowledge cutoff / spans a guideline revision / needs evidence after the cutoff
T8 Region and standard the national or regional guidelines and coding systems the task rests on; always required; where sources conflict, the version the task is graded against
T9 Scale single case / small batch (≤100 items) / large batch (>100 items) ★
T10 Risk low (documents, coding) / medium (quality control, follow-up) / high (prescribing, diagnosis, treatment advice)

8. Statistics

8.1 Sampling and intervals

8.2 Tiers and the order of rows

8.3 The exam, the claimable lower bound and levels

Tasks all passed, n Claimable lower bound (95%) Equivalent to a failure rate of at most
3 0.368 63%
10 0.741 26%
13 0.794 21%
29 0.902 9.8%
59 0.950 5%
100 0.970 3%
299 0.990 1%

The table holds under three conditions, which are published with it:

  1. Independence. The bound treats the n tasks as independent binomial trials. One trial is one task, and success is that task passing in all three runs. The bound therefore constrains the probability that a task of this kind is passed three times in a row by the subject, not the reliability of a single run, and sentences that report it say so.
  2. Scope. The conclusion is stated for tasks of this kind; the qualifying sentence states that the tasks were built to MedGauge specifications and screened by difficulty class, not drawn at random from practice. The bound extends no further.
  3. No re-runs, no removals. G7 is the condition under which the table holds, not an addition to it; any removal after the fact invalidates the bound. Errors attributed to the initiative’s side are handled by the void-and-retest rule (8.4). That rule has a selection effect on the task population—the tasks that remain are those whose failures were attributed to subjects—and the qualifying sentence states it.

The task population is finite and small; whether the infinite-population formula is conservative for it is not established here, and the question does not change the order of magnitude.

8.4 Run errors and the void-and-retest rule

8.5 Agreement between two environments

A statement that env_delta ≤ 0.05 is an equivalence claim. It is tested in the two one-sided tests framework [59] with a stated equivalence bound and enough tasks; for a bound of 0.05 MedGauge’s own computation gives 309–734 tasks, which no benchmark version holds. Until a threshold is set, env_delta is reported with its sign and as an absolute value and is not judged. A threshold is set by a methodology change once the number of tasks supports it, or with a changed equivalence bound.

8.6 Number of tasks and the strength of a conclusion

The table below is for reference. Under the exam, it is not a condition for publication.

Purpose Tasks
Tiers or pairwise comparisons with useful resolution ≥ 30
A claimable lower bound of 0.90 / 0.95 / 0.99 29 / 59 / 299 tasks all passed (exact bound, 8.3)
A calibration subset ≥ 5; 29 all passed for a claimable lower bound of 0.90 (exact bound)
The effect of a component or harness ≥ 30 for a large effect; about 200 for an effect of 0.075
Agreement between two environments (difference ≤ 0.05) 309–734; not attainable (8.5)

Pages show the counts, the exam outcome and the lower bound at the actual number of tasks. While no level threshold is set, the level column shows not issued. Once thresholds are set, a passed subset whose number of tasks cannot reach the next threshold shows too few tasks against it.

9. Traceability and its uncertainty budget

The chain from a number back to its source is: artifact field → checkpoint → check → grading reference → public clause, with its locator and check date. The chain is unbroken and every link is recorded. The table states, for each link, where uncertainty enters and how it is quantified.

Link Source of uncertainty How it is quantified State
Public clause Version of the clause; regional ambiguity Not quantified; T8 records the version a task is graded against Labelled
Grading reference (key) Key error rate The answer-key audit after every round (12.4); the share of keys changed is published per round Existing procedure; publication starts with the first round
Grading reference (key) Agreement between the experts who set the key Two experts set each key independently (gate 3); agreement is reported as Gwet’s AC1 per family version, because kappa-type coefficients behave paradoxically at high agreement [60] Gate 3
Check → score Re-grading agreement Required to be 100% (M9); contributes zero Existing
Task sampling Sampling of tasks The intervals of section 8 Existing

Until the quantified items are published, MedGauge describes the chain as structurally traceable. Whether it may then be called traceable in the metrological sense, which presupposes a calibration hierarchy, is decided separately by the maintainers; until they decide, MedGauge does not describe it so.

10. Subjects and attribution

A subject is anything a result is attributed to. Every form of medical AI is read as one of the kinds below, and every kind is registered with the same four elements: kind, version, responsibility_scope, and requires (the model and harness it is measured with, where it is not itself a full producer).

Subject kind Registry Scope Measured with
model × harness model registry × harness registry the whole artifact — (a leaderboard row)
harness version harness registry the whole artifact, as a paired difference with the same model model
tool, MCP server tool registry the checks of the classes it serves model, harness
skill tool registry (component) the checks it is meant to move model, harness
prompt tool registry (component) the checks it is meant to move model, harness

10.1 Responsibility scope. A declared subset of checks or artifact fields: the full artifact, a list of check classes, or a list of fields. A subject is scored on its scope; its effect outside the scope is reported as a side effect.

10.2 Attribution by paired run. A component’s effect is measured on a pair of runs: the same model, the same harness version, the same benchmark version, with and without the component, three runs each. The paired difference in pass rate on the component’s scope, over the same tasks, is its measured effect (8.1). A single row without a pair attributes nothing to a component. The result record of the component run names its control run (paired_with); the validator requires that the two subjects differ by exactly one component.

10.3 Fixed run conditions. Every result record fixes seven conditions (Appendix C); changing any of them makes a different record.

10.4 Run records and re-grading. Each run writes a run record (Appendix C), including run errors and their attribution (G7). The stored artifact can be re-graded by the verifier at any time, and re-grading must reproduce the recorded grade on every checkpoint (M9). A result record carries the hash of its run records and the outcome of every run (G6).

10.5 Run modes. Site-run: the initiative runs the subject on its own infrastructure; tasks never leave the site. Submitted-run re-grading: an enrolled party receives a watermarked task package under agreement, runs it, and returns artifacts and run records; the initiative re-grades them and only its figures are recorded. In both modes the score is computed by the initiative.

10.6 Reproduction. Reproduction has a narrow meaning: the same harness in a second environment, on calibration tasks, with the reference solution at full marks; env_delta is the difference in PASS^3 between the two environments, reported with its sign and as an absolute value (M9). Until a threshold is set (8.5), env_delta is reported and not judged. A difference between harnesses is not a reproducibility failure; it is what MedGauge measures.

10.7 Autonomy. A harness declares an autonomy class A1–A4 (assistant; co-pilot; executes with run records; executes and self-verifies). The A scale describes the harness and is distinct from capability levels.

11. Task construction and baseline

11.1 Task forms. Every scored task is an artifact task: static materials in, a structured artifact out, a program that decides the end state. The forms are detect and log, item-by-item judgement, structured answer, transform and output and multi-artifact checkpoints. Clinical families add structured forms—an assessment form, a triage form produced from a static record of a past consultation, a regimen assessment form—each graded field by field. Conversational evaluation by rubric is outside the scored track.

11.2 Materials. Materials are described by type only. Real material is de-identified by removing identifiers in the eighteen categories listed by the HIPAA Safe Harbor method [43] and shifting dates by one consistent offset per patient. Safe Harbor itself removes every date element except the year, so the method as a whole is not Safe Harbor; it rests on technical registration and re-identification risk assessment per the national de-identification guides registered in the standards registry. Synthetic material is labelled synthetic. De-identified is not anonymised: institutional de-identified data is used only inside the sandbox and is never offered for download. An identifiability scan is supporting evidence, not a conclusion.

11.3 Gates. A task enters a frozen benchmark version only when every gate below is passed and recorded. Gate records live in the private grading repository; a benchmark card shows a gate only when a record reference is published.

# Gate Criterion Verified by
1 Reference solution scores full marks Graded by the same verifier subjects face, inside the task container; any checkpoint missed fails Builder; independent acceptor re-runs on the delivered package
2 Constructed wrong artifacts all fail Wrong artifacts built by class: empty, flag everything, one extra item, fabricated value, corrected value smuggled in, modified inputs, injected instruction executed; any one that passes fails Builder; acceptor re-runs
3 Two expert keys and the expert key card Two experts from different institutions each set the key independently; their agreement is recorded for Gwet’s AC1 (section 9) and each disagreement is resolved and recorded. The card states the expected key and why; what counts as over-reporting; borderline cases and their decision; grading reference ids relied on; each expert’s specialty and qualification Two experts from different institutions, each reviewing the card
4 Pre-test prediction A predicted pass rate per model tier with a one-line reason, dated before the first run; published beside the measured figure; never an entry criterion and never a change of class (G4) Builder writes; acceptor checks the date
5 Structural difficulty The components and the starred labels recorded and the class computed by the difficulty function in force (M4, Appendix A); holdout entry at class 2 or the family’s higher threshold; measured pass rates never change a class Builder records; acceptor recomputes at freeze
6 Independent acceptance A reviewer who receives only the package and the checklist re-runs gates 1–2 and reads the card; borderline decisions take three independent votes; acceptance by the builder fails Independent acceptor
7 Every checkpoint names its check and provenance; the metadata is complete Task-internal rule with its task rules, or a registered grading reference checked on the original; every label T1–T10 filled; the declared action granularity stated; where the family reports M5–M8 or M10, the required action set, compliance checkpoints, seeded defects, judgement points or variant relation recorded Reference engineer; acceptor checks the map
8 Materials de-identified or synthetic Per 11.2; any residual identifier fails Data steward; acceptor spot-checks
9 Separation of roles Builder, key experts and acceptor are different people; the author of a grading reference builds no task that tests it Maintainers
10 Conformant-alternative self-test At least one correct answer written differently from the reference solution (alternative valid category name, alternative accepted treatment, the other region’s standard, another order that satisfies the precedence constraints) passes; a perturbation variant passes gates 1 and 10 on its own Builder; acceptor re-runs
11 Expert baseline for the version The procedure in 11.5 has been run before the version is used for levels Maintainers; acceptor checks qualification and separation

11.4 Structural difficulty. Difficulty is a property of the task, not of the subjects that have tried it (G4). It is computed by the published difficulty function in force (M4, Appendix A) from the components recorded in the closed task metadata. Cut points are set by design and never fitted to results. Before a round, the expected PASS^3 per class and model tier is written down (gate 4); after the round the measured figure is published beside it, with the monotonicity check of M4. Neither ever changes a task’s class. The function is versioned; a frozen benchmark version keeps the function version in force when it froze. The class and label distributions per version are public; per-task components and labels stay closed. Until a family’s tasks carry recorded components, no page states a class for that family.

11.5 Expert baseline. For each family and version: the median time a qualified expert needs per task, the expert pass rate under the same verifier, the number of experts and the measurement date. A person qualifies who holds the professional qualification of the role the family serves, is in current practice in that role, and took no part in building the task, its key or its card. At least two experts per task; they are not the experts who set the task’s key (gate 3). The expert receives the same materials and artifact contract a subject receives, with no access to key or verifier, and submits an artifact; time to submission is recorded without a limit. An expert artifact that fails a checkpoint triggers gate 10 on that check before the result is recorded. No number is published before the procedure has run.

11.6 Calibration items. Each family carries designated calibration items with public keys. They prove that a task is solvable and that grading does not mark a correct answer wrong, serve as the material for reproduction (10.6) and for training of new acceptors (section 21), and are the only items on which a check may merely ask for a reference to be restated.

12. Integrity and isolation

12.1 The boundary.

Public Closed
Methodology, scoring rules, task forms Task statements, case data, environment instances
Synthetic illustrations of artifacts Answers, reference solutions, verifiers
Schemas for every registry, including task metadata Run trajectories
Hash manifest of holdout answers per frozen version, released with the first holdout round Which grading references each task tests; difficulty components and labels per task
Benchmark cards, capability, harness, tool, standards and dataset registries Expert key cards, gate runs, review records
Grading references: standard, locator, rule applied Per-checkpoint grading detail (public records carry only a hash)
Site-verified result records with every run under an opaque task key; per-reference summaries; class and label distributions per version; voided task versions with their attribution Role and standing records of members

12.2 Contamination control. Holdout answers never ship with a package; a hash manifest commits to them. One container per task with networking disabled; the verifier is mounted only at grading time. Canary strings are embedded in task material; a canary in a subject’s output voids the result [16]. A public task set can overlap with training data, so that its score reflects recall rather than skill [5]; this is why the holdout is closed. A task that enters any training set or targeted debugging is retired and replaced in the next version. Instructions planted in data test whether a subject follows them (X.4) [25, 26].

12.3 Training isolation. Training data built for the same capability uses independent source material, stored and licensed separately from evaluation sets. Before release, a similarity check covers six dimensions: patient, data source, original document, task-statement template, answer and environment state. The result and date are recorded on the dataset card.

12.4 Answer-key audit. After every round the keys of the tasks run are audited against the round’s failure attribution and expert artifacts. The share of keys changed is published per round (section 9). A key found wrong is corrected in the next version; results across versions are labelled not comparable; historical results are never deleted [15, 18]. A key error found during a round is an error on the initiative’s side and falls under the void-and-retest rule (8.4).

12.5 Retirement. A task is retired on contamination or on a key defect that cannot be corrected in place. A capability is retired when three frontier tiers reach PASS^3 ≥ 0.95 for two consecutive versions, or when it is merged. Retired entries keep their pages.

13. Validity

Each property has a procedure. A property without a procedure is not claimed.

Property Claim Procedure
Attributable Every pass or fail traces to a field, a checkpoint and a check Per-checkpoint detail on the closed side, hashed into the public record; scope declarations; paired runs for components
Recomputable Any stored artifact re-graded by the verifier reproduces the recorded grade, and every published figure follows from the public record set Re-grading of every stored run (M9); every run outcome in the result record; published seeds and quantile rules; run records hashed into the record; the same task package copy delivered and re-gated
Composable Scores at every level follow from the level below by the published rules Sections 5 and 6; equal weights within a family; subsets kept separate; dimensions never combined; versions labelled not comparable
Falsifiable A wrong key or an over-strict check can be shown wrong and is then corrected Gates 1, 2 and 10; expert artifacts that fail a checkpoint trigger review; answer-key audit after each round; correction in the next version
Contributable Outside parties add to the measurement without seeing tasks Section 23

Two further properties are measured rather than claimed: reliability (M2: PASS^3 beside PASS@3 and instability; re-grading identity and env_delta under M9; the key-error rate and expert agreement of section 9) and fairness (paired population cases under X.10, graded as pairs).

13.1 Validity argument. In the vocabulary of measurement validity [30, 31]: content validity rests on checks derived from the source’s own terms and from the artifacts of named jobs, with an expert key card per task; construct validity rests on capabilities defined by input → artifact → grading, on the three-layer marking that keeps clinical facts apart from role-bound constraints, and on external anchors recorded per capability; criterion validity rests on the expert baseline per version and, for L5, on before/after operational evidence; consequential validity rests on levels being absolute and defined only for the four-tuple, so that no reading of a level as a clinician grade is licensed by the site.

13.2 Threats and the procedure that meets each.

Threat Procedure
Contamination of the holdout Closed set; hash manifest; one container, no network; canary; retirement; training isolation
Defective key Gates 1, 2, 10; expert key card; independent acceptance; post-round audit; versioned correction
Harness confound Harness in the subject; paired runs; env_delta for reproduction; harness_delta reported, never a pass line
Judge bias (position, verbosity, self-preference) Eliminated by construction: program grading only [19]
Style and over-reporting Fields, not prose; over-reporting stated as checkpoints; “one extra item” in gate 2; precision beside recall (M7); over-abstention beside abstention (M8)
Ceiling and saturation Structural difficulty; pass rates published per class (M4); capability retirement rule
Small samples Three runs; errors never removed; Clopper–Pearson and task-level BCa intervals; paired comparisons with Holm adjustment and compact letters; exam conclusions stated with their lower bound and number of tasks (section 8)
Regional parochialism Rule 4.2.4; jurisdiction statement; conformant-alternative test with the other region’s standard
Retrofitted difficulty Difficulty from structure only (G4); cut points fixed before results; gate 4 predictions dated before the first run and published beside the measurement
Identity leakage into grading Task-submission form with no origin fields; producer never visible to the grader
Score laundering Axiom 6; submitted runs re-graded by the initiative; canary; trajectory sample
Selective reporting Errors are failures (G7); cost per success over all runs beside conditional efficiency (G5); voided task versions listed with their attribution (8.4)

14. Reporting

14.1 Result record. Methodology version; benchmark version; model and snapshot fingerprint; harness id and version; components and paired_with where applicable; the seven fixed conditions; every run of every task under an opaque task key, with its checkpoint counts, s, veto categories and error attribution; by subset, the exam outcome with its number of tasks and claimable lower bound, PASS@3, PASS^3, instability, task pass rate and S, with their intervals, errors by attribution, veto rate by category and compliance rate; the other dimensions where the family supports them (6.11); the board position with its group letters; veto hits keyed by constraint; failure attribution (missed findings, over-reporting, fabricated evidence, format, step limit, other); amounts with the price table version, or amount not applicable; voided task versions; run records hash; verification state; date. The record follows schemas/result.schema.json.

14.2 Per-reference summary. Per benchmark version, for each grading reference: how many tasks test it and the pass rate over site-verified rows, without naming tasks.

14.3 What is never reported. A level while no level threshold is set; a level above L0 without a calibration subset; a class for a family without recorded components; a number for an expert baseline not yet measured; a single run as a score; a figure pooled across subsets; a combination of dimensions; a cross-family or cross-version total; an action or redundancy figure ranked across harness versions; an amount for a self-hosted or open-weight subject; a figure from which an errored run was removed; the labels of a single task; a producer’s identity inside a grade.

14.4 Non-comparability. A change of benchmark version, difficulty function version or level threshold, and a methodology version that makes a non-comparable change (section 30), produce a notice on every affected board; earlier figures stay published under their own version.

Part II — The reference work

15. Registries

The registries are the reference work. The website is their current rendering.

Registry Entry Unit of the reference work
Capability registry capability The indexed competence: definition, domain, layer, dependencies, grading form, evidence state
Harness registry harness version The execution layer: runtime, slots, tools, limits, sandbox, autonomy class, adapter check
Tool registry tool, component Protocol, scope, side effects, conformance, requires
Standards registry standard The cited source: publisher, version where confirmed, locator, source check, how the site uses it
Benchmark registry benchmark card The chapter: scenario, capabilities, task form, materials by type, scoring, integrity, versions, access
Result records result record The evidence line: one subject on one version, site-verified
Dataset registry dataset card The material: type, provenance, legal status, isolation, access tier
Model registry model The named producer: vendor, snapshots, access route; listed only when a reviewed result exists
Grading-reference registry grading reference The cited rule: one clause, its locator, check date, the rule applied, layer
Roadmap, reproductions, level overrides roadmap item, reproduction record, level override Planned work; second-environment records; signed L5 records

16. Entry standard

Every entry carries an identifier, a version, the specification revision that introduced it, and the dates on which it was introduced and last reviewed. Frozen objects are immutable. Every entry is validated against a published JSON Schema at build time; an entry that fails validation does not exist on the site.

A benchmark card states: scenario; capabilities served; task types; artifact contract and fields; materials by type; defect and verdict classes; scoring form, pass rule, vetoes and stability reporting; the dimensions the family reports (6.11); integrity records by reference; de-identification basis; external anchors; the public/closed boundary; versions with subsets, difficulty and label distributions, expert baseline and hash manifest; a synthetic illustration of the task shape; and a citation form. It states no task, no answer and no result other than site-verified records.

A grading reference states: the standard (a standards registry id); the locator inside it; the check date; the condition; the rule on a named field; the polarity; the check form; the veto flag; the layer; the capabilities it maps to; and, where sources disagree, each source with its locator and the jurisdiction statement.

A capability entry states: definition by input → required artifact → grading; domain; layer; dependencies; grading form; applicable vetoes; external anchors; and an evidence state derived at build time and never written in source.

17. Editorial rules

18. The website

medgauge.org renders the registries, the methodology and the articles. Its build validates every record, resolves every cross-reference, checks every internal link, enforces English-only text and runs the red-line scan before anything is published. The site carries no scripts, no third-party resources and no tracking. The build identifies the registry commit it was made from on every page. The site is the public face of the reference work; it is never its source.

Part III — Operation

19. Nature of the initiative

MedGauge is an initiative: a body of maintainers operating a GitHub organisation, the website and a private grading repository under this methodology. It is not an institute, a society, an accreditation body, a certification scheme or a regulator. It issues no certificate, and nothing it publishes constitutes a basis for the regulatory approval of a medical device or a substitute for registration testing or clinical trials. Its standing rests on this text, the records it publishes and the reproductions others make of them. MedGauge is independent of the parties it measures and of any supplier of evaluation material; tasks are built, held and graded by the initiative.

20. Roles and separation

Role Does Leaves
Maintainer Operates the repositories, the site and the runs; approves changes under section 22 Approvals, release records
Reference engineer Builds grading references by the reverse path (4.2.6); re-checks clauses on the original Reference records with check dates
Task builder Builds tasks, keys and verifiers; runs gates 1, 2, 4, 5, 10 Gate records; task submissions
Expert reviewer Prepares or reviews expert key cards; takes part in the expert baseline Expert key cards; baseline records
Independent acceptor Re-runs gates on the delivered package with only the checklist; signs acceptance Acceptance records
Data steward De-identification and synthetic-generation records; legal status of material Gate 8 records; dataset cards
Results reviewer Applies the review checklist to runs before a record is merged Review records
Reproducer Runs calibration tasks in a second environment Reproduction records
Contributor Supplies a contribution unit (section 23) The unit and its acceptance record

Separation: the builder, the experts who set the key and the acceptor of one task are different people; the author of a grading reference builds no task that tests it; a party measured on a family contributes no holdout task to it; whoever submits a subject computes no score; the grader never sees a producer.

21. Standing and progression

Standing is earned by recorded acts, not by title. Each role names the records that qualify a person for it; the records are kept in the private repository, and public attribution follows section 27.

Standing Qualifying records Confirmed by
Contributor One contribution unit accepted The accepting maintainer
Reviewer Three contribution units accepted, or three clause checks on originals confirmed by a second person One maintainer
Acceptor Gates 1, 2 and 10 re-run on the calibration reproduction pack with outcomes matching the record; second acceptor on three packages Two maintainers
Expert reviewer Professional qualification of the role the family serves; current practice in that role; no part in building the tasks reviewed Two maintainers
Maintainer Acceptor standing; acceptor or builder of one frozen benchmark version; appointment by the maintainers The maintainers

Training is done on the reference work itself: the calibration reproduction pack (tasks with public keys), the methodology articles and the gate checklist. The first act of every new acceptor is a gate re-run on the pack that matches the recorded outcome. Standing lapses after two consecutive benchmark versions without a qualifying record and is regained by the same records.

22. Decisions

Every change to the public repository is a pull request that passes the build checks (section 18).

Object Procedure Approval
Registry record Pull request; schema and reference checks One maintainer
Grading reference listing Second-person clause check on the original; conformant-alternative test; a frozen version uses it Reference engineer and one reviewer
Benchmark version freeze Gates 1–11; independent acceptance Independent acceptor and one maintainer
Result record Site run or site re-grade; review checklist (section 25.4) Results reviewer and one maintainer
Harness, tool or component registration Schema; adapter check or smoke test; scope declaration One maintainer
Capability added or retired Request for comment with a gradable task summary, an evidence form and an external anchor Two maintainers
Level thresholds, aggregation rules, difficulty cut points Request for comment; non-comparable change under section 30; site-wide notice Two maintainers
Methodology text Request for comment; version per section 30 Two maintainers
Charter Pull request; version Two maintainers
Dispute on any of the above Three independent votes; outcome and reasons published —

23. Contribution

Unit The contributor supplies Acceptance
Grading reference Standard, locator, rule on a named field, polarity, check form, layer; for regional splits, each source and the jurisdiction statement A second person checks the clause on the original and records the date; the conformant-alternative test is run; listed once a frozen version uses it
Check with provenance A check (section 4) with its provenance; for task-internal rules, the task rules it belongs to Reviewed with the family it serves; enters a version through the gates
Task naming its checks A task submission in the form of schemas/task-submission.schema.json, carrying only checks with provenance, difficulty components, verifier reference, capability ids, family, materials hash, the task labels T1–T10, the declared action granularity and, where the family reports them, the required action set, judgement points, seeded defects, compliance checkpoints and variant relation; with the contributor’s declarations (no individually identifiable information, no origin identifier, not a holdout task from a measured party)—the task-submission schema admits nothing else The eleven gates, independent acceptance, freezing; the task enters the private grading repository and never the public one
Paired run A request to run a subject with and without a component under a fixed model and harness Site-run or submitted-run re-grading; only site-computed figures are recorded

Never accepted: an answer key or a correction for an existing task; a holdout task from a party that will be measured on that family; a score computed outside the initiative; any task, check or reference that carries the identity of its origin; any material containing individually identifiable information.

Contributors keep copyright in what they contribute and license it under the repository’s licences: Apache License 2.0 for code and schemas, Creative Commons Attribution 4.0 International for content and registries. Contributions are made under the contributor’s own name and identifier.

24. Registration of harnesses, tools and components

A harness version is registered by a file per version that states runtime, model slots, tool components pinned to tool registry versions, limits, sandbox, autonomy class, reproduction command and system-prompt hash where the prompt is not public. It becomes active after the adapter check: the reference solution scores full marks through it on a calibration task. A tool or component is registered with its protocol, medical scope, side effects, responsibility scope and requires; a tool server’s exposed tool list is frozen as a schema hash; a skill is validated against the Agent Skills specification; a smoke test (start, list tools, call one read-only tool) is recorded. A published version is never edited; a change is a new version. A version with no result record for two consecutive benchmark versions, or on its maintainer’s request, is marked deprecated; its page stays.

25. Evaluation rounds

25.1 An evaluation round is a set of subjects run on a set of benchmark versions under fixed conditions.

25.2 Registration. Before the first run, the round’s protocol is registered with a timestamp on a public registry of study protocols (OSF Registries) and cited in the round report [33]: subjects; benchmark, verifier, difficulty function and statistical algorithm versions, with their random seeds and quantile rules; the price table version; the seven fixed conditions; the predicted PASS^3 per difficulty class and model tier (gate 4); the review checklist version.

25.3 Execution. Site-run by default; submitted-run re-grading under agreement. Three runs per task; one container per task, networking disabled. Every run error—time-out, empty answer, interface, network, server or output error, forbidden action—is a failed run, recorded with its attribution and never re-run (G7). An error attributed to the initiative’s task or environment voids that task version for every subject (8.4).

25.4 Review checklist. A record is verified only when all hold: the reference solution scores full marks through the harness; re-grading agrees with the recorded grade on every checkpoint (M9); every run is in the record and every run error carries an attribution; where a second environment was used, env_delta is recorded with its sign and judged only once a threshold is set (8.5); run records are verifiable (request identifiers and usage self-consistent); amounts name their price table version, or the subject is marked amount not applicable; a trajectory sample of at least ten percent of tasks is checked for vetoes and for access to the site, the repository or the network; no canary appears in any output.

25.5 Report. Per family and subset: the exam outcome of every subject with its number of tasks and claimable lower bound; each dimension on its own, with its interval; compact letters for the holdout board; veto columns by category; error rates by attribution; voided task versions; amounts with the price table version; figures per difficulty class and the monotonicity check (M4); predicted against measured PASS^3 per class; the answer-key audit with the share of keys changed and the expert agreement (section 9); the harness_delta view where the same model ran under several harnesses. A round report is published with a persistent identifier and follows the reporting items of the registered guidelines for studies of language models in health care [38] where they apply.

25.6 Applications. A party applying to an evaluation round states purpose (research, internal product evaluation, or regulatory submission testing), the subject (model, snapshot, available harnesses), the agreement for submitted-run re-grading where used, and the destruction date for any task package. Separation under section 20 applies to every applicant.

26. Reproduction

An external reproduction runs calibration tasks in the reproducer’s own environment with a registered harness. A reproduction record states organisation, harness and version, model, reference-solution outcome and env_delta with its sign; pass or fail is stated only once a threshold is set (8.5). A reproduction whose reference solution does not score full marks, or whose env_delta exceeds the threshold once one is set, is investigated first on the adapter (reference solution, container, step limit, service snapshot); the task is never changed to fit. Reproduction records are public.

27. Publication and attribution

28. Independence and name use

Third parties may cite MedGauge results with the four-tuple and date. The name implies no endorsement, certification or accreditation. Benchmark names and identifiers are stable; a rename keeps the old identifier as a redirect. No party may present a MedGauge figure without its version and date, or a level without its four-tuple.

Part IV — Evolution

29. Versioned objects

Object Versioning Carried on
Methodology (this text) major.minor.patch Every result record and level
Charter major.minor.patch The repository
Schemas and validator rules With the methodology version, which each schema’s $id carries Build
Statistical algorithms (intervals, paired test, letter display) Version, with published random seeds and quantile rules Result records, round protocols
Price table Version; published with the results that use it Result records
Task Version; a voided version is listed with its attribution Task metadata, result records
Benchmark version major.minor.patch per family; frozen Result records, cards
Difficulty function major.minor.patch, cited as difficulty-function@<version> Benchmark versions, task metadata
Level rules With the methodology version Levels
Registry entries version field; introduced_in Entry
Harness, tool, component One immutable file per version Result records
Verifier Version and hash, closed Task package; result record
Methodology article Date; supersedes link Article

30. Change classes

Every change enters by request for comment and pull request (section 22), takes effect at the next benchmark version freeze unless it is a patch, and is recorded in Appendix D.

31. Review cycle

The methodology is reviewed after every holdout round against the round’s inputs: the answer-key audit and the expert agreement; predicted against measured PASS^3 per class and the monotonicity check of M4; the env_delta distribution; error rates by attribution and voided task versions; saturation; veto patterns; disputes raised. It is reviewed in any case once every twelve months. Each review ends in an entry in Appendix D: either no change, or a versioned change under section 30.

32. Retirement and deprecation

Tasks retire under 12.5; capabilities under 12.5; harness, tool and component versions under section 24. A methodology version is superseded, never deleted: every result record names the version in force when it was made, and comparisons across a non-comparable change carry the non-comparability notice.

33. Relation to other measurement programmes

MedGauge takes proven structures from general and medical evaluation programmes and adds the parts specific to medical work products.

Programme What it established What MedGauge takes, and adds
HELM [1]; MedHELM [2] Scenario-by-metric reporting with a published taxonomy A capability taxonomy every task attaches to; medical-practice capabilities graded by artifact rather than by answer text
LMArena [3] Large-scale human preference ranking Nothing from the method: MedGauge scores are program-decided, not preference-decided
SWE-bench [4]; the study of its verified subset [5] Execution-graded tasks; agent scaffolds named per entry; a public task set can overlap with training data, so that a score can reflect recall rather than skill Model × harness as the row; end-state grading; a closed holdout
τ-bench [6]; the code-evaluation study [7] pass^k as the reliability figure; the unbiased estimators of pass^k and pass@k PASS^3 beside PASS@3 (M2); PASS^3 as the primary ordering key
Terminal-Bench and the Harbor task format [8] A benchmark and a task format separating dataset from harness The packaging of task families; trajectories recorded in a format registered in the standards registry with its locator and source check
HealthBench [9] Physician-written rubrics for health conversations The complementary track; conversational rubric evaluation stays outside the scored track
MedAgentBench [10]; AgentClinic [11]; MedCalc-Bench [12] Agent tasks in a records environment; sequential clinical decisions; medical calculation Artifact tasks in a records setting; calculation and recomputation as checks with X.3 in force
FrontierMath and the Epoch AI hub [13]; Humanity’s Last Exam [15] Held-out sets with few public problems; expert-sourced items and the cost of key errors Closed holdout with a hash manifest per version; network-disabled containers; calibration items public only
On the Measure of Intelligence [14] A definition and measure of intelligence, with the abstraction-and-reasoning task set built to it The construct defined before the tasks: a capability is defined by input → artifact → grading before any task is built for it
BIG-bench [16]; contamination studies [17] Canary strings; measurement of contamination per benchmark Canary in task material; retirement on contamination
Label-error studies [18]; judge-model studies [19] Test-set errors destabilise rankings; judge models carry position and verbosity bias Key validation before any run; program grading only
Agent evaluation practice [20, 21, 22, 23, 24] Cost reporting; benchmark design criteria; reproducible evaluation; the limits of a benchmark’s claim Cost as a reported constraint (X.8); the non-claims of section 35
Harness-effect studies [27, 28] The same model under different harnesses moves by many points The harness in the subject; harness_delta as a measured quantity; paired-run attribution; actions counted by each harness’s own definition (G8)
Statistical practice for evaluations [52–60] Exact binomial bounds; cluster bootstrap; paired comparison; multiple-comparison adjustment; limits of rank bootstraps; agreement coefficients Section 8 and section 9
Medical agent and clinician benchmarks [49, 50, 51] Long-horizon clinical tasks graded by checkpoints; records-based temporal tasks External anchors recorded per capability
Exam-style benchmarks [44, 45, 46] Knowledge measured by multiple choice Not used for scoring; such items are confined to the calibration subset

34. Relation to standards

34.1 Risk management. The vocabulary of ISO 14971 [39] (adopted identically in the national standard [42]) maps onto MedGauge elements as follows. The mapping is definitional; MedGauge claims no conformity assessment.

ISO 14971 term MedGauge element
Hazard (potential source of harm) A cross-cutting constraint: fabricated evidence (X.1), altered inputs (X.2), corrected values (X.3), executed planted instruction (X.4), unlogged information gap (X.5), identifiable output (X.6)
Hazardous situation A veto hit or a failed X.5 checkpoint: the hazard realised in a work product that would be used
Harm Outside the measurement; determined by the deploying organisation’s risk analysis
Risk control verification Veto counts per subject and version, reported separately and never averaged
Production and post-production information Site-verified result records with four-tuple and date; non-comparability across versions

The general guidance on applying ISO 14971 is [40]. The machine-learning guidance [41] names risk categories—data quality, bias, overtrust, adaptive behaviour—that correspond to isolation and de-identification (section 12, dataset cards), paired population cases (X.10), vetoes kept apart from merit with absolute levels (Axioms 5 and 6, section 5.6), and fixed run conditions with snapshot fingerprints and version non-comparability (10.3, 14.4). ISO/TS 24971-2:2026 states its scope as machine-learning-enabled devices other than those employing large language models or generative AI; MedGauge uses its risk vocabulary as a reference frame.

34.2 Reporting. TRIPOD-LLM [38] for round reports and benchmark write-ups; CONSORT-AI [37] where a round is designed as a trial.

34.3 Privacy. The HIPAA Safe Harbor identifier list [43] and the national de-identification and evaluation guides registered in the standards registry for materials; the legal status of every dataset is stated on its card.

34.4 Terminology and data. FHIR, OMOP CDM, SNOMED CT, LOINC, ICD and RxNorm entries in the standards registry are cited by grading references and dataset cards by identifier; no terminology content is reproduced.

34.5 Agent packaging. The Harbor task format for tasks [8]; for trajectories, the format registered in the standards registry with its locator and source check; the Agent Skills specification for skills; the Model Context Protocol for tool servers (standards registry).

35. Non-claims

References

  1. Liang, P., Bommasani, R., Lee, T., et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research (2023). arXiv:2211.09110.
  2. Bedi, S., Cui, H., Fuentes, M., et al. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks (2025). arXiv:2505.23802.
  3. Chiang, W.-L., Zheng, L., Sheng, Y., et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML (2024). arXiv:2403.04132.
  4. Jimenez, C. E., Yang, J., Wettig, A., et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR (2024). arXiv:2310.06770.
  5. Prathifkumar, T., Mathews, N. S., Nagappan, M. Does SWE-Bench-Verified Test Agent Ability or Model Memory? (2025). arXiv:2512.10218.
  6. Yao, S., Shinn, N., Razavi, P., Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024). arXiv:2406.12045.
  7. Chen, M., Tworek, J., Jun, H., et al. Evaluating Large Language Models Trained on Code (2021). arXiv:2107.03374.
  8. Merrill, M. A., Shaw, A. G., Carlini, N., et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026). arXiv:2601.11868.
  9. Arora, R. K., Wei, J., Soskin Hicks, R., et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025). arXiv:2505.08775.
  10. Jiang, Y., Black, K. C., Geng, G., et al. MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents (2025). arXiv:2501.14654.
  11. Schmidgall, S., Ziaei, R., Harris, C., et al. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments (2024). arXiv:2405.07960.
  12. Khandekar, N., Jin, Q., Xiong, G., et al. MedCalc-Bench: Evaluating Large Language Models for Medical Calculations. NeurIPS Datasets and Benchmarks (2024). arXiv:2406.12036.
  13. Glazer, E., Erdil, E., Besiroglu, T., et al. FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (2024). arXiv:2411.04872. Epoch AI Benchmarking Hub: https://epoch.ai/benchmarks
  14. Chollet, F. On the Measure of Intelligence (2019). arXiv:1911.01547.
  15. Phan, L., Gatti, A., Han, Z., et al. Humanity’s Last Exam (2025). arXiv:2501.14249. Analysis of its chemistry and biology answers: https://www.futurehouse.org/research-announcements/hle-exam
  16. Srivastava, A., Rastogi, A., Rao, A., et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (2023). arXiv:2206.04615.
  17. Sainz, O., Campos, J. A., García-Ferrero, I., et al. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP (2023). arXiv:2310.18018.
  18. Northcutt, C. G., Athalye, A., Mueller, J. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS Datasets and Benchmarks (2021). arXiv:2103.14749.
  19. Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS (2023). arXiv:2306.05685.
  20. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., Narayanan, A. AI Agents That Matter (2024). arXiv:2407.01502.
  21. Reuel, A., Hardy, A., Smith, C., et al. BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS Datasets and Benchmarks (2024). arXiv:2411.12990.
  22. Biderman, S., Schoelkopf, H., Sutawika, L., et al. Lessons from the Trenches on Reproducible Evaluation of Language Models (2024). arXiv:2405.14782.
  23. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., Hanna, A. AI and the Everything in the Whole Wide World Benchmark. NeurIPS Datasets and Benchmarks (2021). arXiv:2111.15366.
  24. Bowman, S. R., Dahl, G. E. What Will it Take to Fix Benchmarking in Natural Language Understanding? NAACL (2021). arXiv:2104.02145.
  25. Debenedetti, E., Zhang, J., Balunović, M., et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS Datasets and Benchmarks (2024). arXiv:2406.13352.
  26. Greshake, K., Abdelnabi, S., Mishra, S., et al. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023). arXiv:2302.12173.
  27. Stop Comparing LLM Agents Without Disclosing the Harness (2026). arXiv:2605.23950.
  28. Yao, Y., Tan, X., Liu, C.-H., et al. Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows (2026). arXiv:2605.27922.
  29. Stevens, S. S. On the Theory of Scales of Measurement. Science 103(2684), 677–680 (1946).
  30. Messick, S. Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. American Psychologist 50(9), 741–749 (1995).
  31. Cronbach, L. J., Meehl, P. E. Construct Validity in Psychological Tests. Psychological Bulletin 52(4), 281–302 (1955).
  32. Efron, B. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7(1), 1–26 (1979). Cited for the bootstrap; not for any interval procedure.
  33. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., Mellor, D. T. The Preregistration Revolution. Proceedings of the National Academy of Sciences 115(11), 2600–2606 (2018).
  34. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3, 160018 (2016).
  35. Gebru, T., Morgenstern, J., Vecchione, B., et al. Datasheets for Datasets. Communications of the ACM 64(12), 86–92 (2021). arXiv:1803.09010.
  36. Mitchell, M., Wu, S., Zaldivar, A., et al. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (2019). arXiv:1810.03993.
  37. Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nature Medicine 26, 1364–1374 (2020).
  38. Gallifant, J., Afshar, M., Ameen, S., et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine 31, 60–69 (2025).
  39. ISO 14971:2019. Medical devices — Application of risk management to medical devices. International Organization for Standardization.
  40. ISO/TR 24971:2020. Medical devices — Guidance on the application of ISO 14971. International Organization for Standardization.
  41. ISO/TS 24971-2:2026. Medical devices — Guidance on the application of ISO 14971 — Part 2: Machine learning in artificial intelligence. International Organization for Standardization.
  42. GB/T 42062-2022. Medical devices — Application of risk management to medical devices (identical adoption of ISO 14971:2019). State Administration for Market Regulation; Standardization Administration of China.
  43. U.S. Department of Health and Human Services, Office for Civil Rights. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule (2012). https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html
  44. Singhal, K., Azizi, S., Tu, T., et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023).
  45. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences 11(14), 6421 (2021). arXiv:2009.13081.
  46. Hendrycks, D., Burns, C., Basart, S., et al. Measuring Massive Multitask Language Understanding. ICLR (2021). arXiv:2009.03300.
  47. Brand, A., Allen, L., Altman, M., Hlava, M., Scott, J. Beyond authorship: attribution, contribution, collaboration, and credit. Learned Publishing 28(2), 151–155 (2015).
  48. MLCommons. Croissant: a metadata format for machine learning datasets. https://mlcommons.org/working-groups/data/croissant/
  49. PhysicianBench (2026). arXiv:2605.02240.
  50. HealthAgentBench (2026). arXiv:2606.31179.
  51. LongMedBench (2026). arXiv:2607.09322.
  52. Clopper, C. J., Pearson, E. S. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika 26(4), 404–413 (1934).
  53. Wilson, E. B. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158), 209–212 (1927).
  54. Field, C. A., Welsh, A. H. Bootstrapping Clustered Data. Journal of the Royal Statistical Society Series B 69(3), 369–390 (2007).
  55. Miller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024). arXiv:2411.00640.
  56. Holm, S. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics (1979). https://www.jstor.org/stable/4615733
  57. Hall, P., Miller, H. Using the bootstrap to quantify the authority of an empirical ranking. The Annals of Statistics 37(6B), 3929–3959 (2009).
  58. McNemar, Q. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12(2), 153–157 (1947).
  59. Schuirmann, D. J. A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. Journal of Pharmacokinetics and Biopharmaceutics 15(6), 657–680 (1987).
  60. Gwet, K. L. Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement. British Journal of Mathematical and Statistical Psychology 61(1), 29–48 (2008).

Appendix A — Difficulty functions

A benchmark version keeps the difficulty function in force when it froze (G6). Every benchmark version frozen so far uses difficulty-function@0.1.0 (A.1). The framework in A.2 is the next version of the function; its points and cut points are published in this appendix, by a change under section 22, before any benchmark version freezes under it, and until they are no class is stated under it.

A.1 difficulty-function@0.1.0

Component Measure Points
breadth number of checkpoints 1–5 → 0; 6–12 → 1; 13 or more → 2
crossing number of distinct materials that must be joined to decide at least one checkpoint 1 → 0; 2 → 1; 3 or more → 2
gaps number of required information-gap logs (X.5) 0 → 0; 1–2 → 1; 3 or more → 2
planted content materials carry injected instructions or decoy items no → 0; yes → 1
regional split at least one checkpoint depends on a stated regional standard no → 0; yes → 1
controls no-defect or no-adjustment controls are present no → 0; yes → 1

score = sum of points (0–9). class 1 = 0–2; class 2 = 3–4; class 3 = 5–6; class 4 = 7–9. Holdout entry at class 2 or the family’s higher holdout_entry_class; class 1 tasks go to the dev or calibration subset.

A.2 Next version: components from structure and labels

Component Measure
required steps required steps of the reference solution at the declared action granularity (G8)
dependency depth the longest chain in the directed acyclic graph of required actions
information sources number of sources that must be compared
distractors number of distractor materials
ambiguous judgement points number of judgement points with ambiguity
T1 horizon ordered label (section 7)
T2 environment ordered label
T3 specialties ordered label
T4 modalities number of modalities
T5 omics number of omics layers
T6 interacting parties ordered label
T9 scale ordered label

The function maps the components to classes 1–4 by points and cut points fixed by design. Each structural feature is scored once. The length of the reference solution is scored through T1 horizon only, which is the banded form of the required steps; the count of required steps is recorded, and enters the function through T1 alone. Dependency depth is scored as its own component: it is bounded by the required steps but measures the longest chain of required actions, not their number. The points and cut points are never fitted to results. Holdout entry is at class 2 or the family’s higher holdout_entry_class; class 1 tasks go to the dev or calibration subset. Points, cut points and components change only in a new function version, published with the comparison that motivated it.

Appendix B — Level rules

A level above L0 requires the exam outcome passed (section 8.3) on the subset the rule names, and a claimable lower bound, 0.05^(1/n) for n tasks, at or above the level’s threshold. Any veto means not passed. Thresholds are absolute, never ranks. Every level is published with its number of tasks. The rule reads the per-subset exam fields of a site-verified result record.

Level Name Rule Threshold on the claimable lower bound Derived by program
L0 Unproven No site-verified result record for the four-tuple, or no level above L0 reached — Yes
L1 Calibrated Calibration subset passed Not set in version 0.1.0 Yes
L2 Usable L1 and holdout subset passed Not set in version 0.1.0 Yes
L3 Stable L2, with the holdout bound at the L3 threshold Not set in version 0.1.0 Yes
L4 Strong L3, with the holdout bound at the L4 threshold Not set in version 0.1.0 Yes
L5 Role candidate L4, written before/after scenario evidence, an external anchor pointing the same way; signed override record; scenario capabilities only As L4 No (signed)

Version 0.1.0 sets no level threshold and issues no level. A level whose threshold is not set is not issued, and every level above it depends on it; level columns and capability pages show not issued. A threshold is set by a change under section 22 (a non-comparable change, section 30). The claimable lower bound shows what a threshold would demand: a threshold of 0.95, for example, would need 59 tasks all passed (8.3). That figure illustrates how the bound is computed; it is not a level rule. Once thresholds are set, where the number of tasks is below the number a threshold needs, the level column shows too few tasks (8.6). With no calibration subset, L1 is not computable and the capability page says so instead of showing L0.

Appendix C — Fixed run conditions and run record fields

Seven fixed conditions: model and version; service snapshot or fingerprint returned by the interface; reasoning setting; runtime environment (container image digest); tool permissions and tool list; harness id and version; budget (step limit, output tokens, timeouts, token budget).

Run record fields: model; interface; actions under the harness version’s own definition, and the step limit; tool-call counts; token use; timing; the amount and its price table version, where amounts apply; temperature; system-prompt hash; whether the artifact exists; whether inputs were modified; checkpoints passed and m; veto categories; any run error with its kind and attribution; grade.

Appendix D — Methodology change log

Version Date Change
0.1.0 2026-10-29 First published version. Defines the check as the unit of measurement and its composition into checkpoints, runs, tasks and frozen benchmark versions; the general rules G1–G8; the dimensions M1–M10, reported each on its own with no total; the task labels T1–T10; three runs per task, a run error counted as a failed run with its attribution, and the void-and-retest rule; the exam conclusion with its exact claimable lower bound; Clopper–Pearson and task-level BCa intervals; paired, Holm-adjusted comparisons with a compact letter display and the order of rows; the uncertainty budget of the traceability chain; difficulty-function@0.1.0 and the framework of its next version; the identifier type codes; the registries, editorial rules and evidence grades; operation, contribution, publication and change rules. No level threshold is set and no level is issued.