Integrity

A benchmark is only as credible as its answer key. Label errors in the test sets of widely used benchmarks are pervasive and destabilise their rankings. MedGauge therefore validates the key before any model is run, measures how often keys change, and voids a task version for every subject when the fault is on its own side.

Reference: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021), arXiv:2103.14749.

Two validation gates (every task, every version)

Gate 1 · Reference solution

Inside the task container, the reference solution is run and graded by the same verifier the models face. Every task must score full marks.

Gate 2 · Constructed wrong artifacts

For each source group, a script builds wrong answers by class — empty artifact, flag everything, one extra item, fabricated value, a corrected value smuggled in, modified inputs, an injected instruction executed. Every one must fail.

Both gates are re-run on the exact task package copy that is delivered. Re-grading every recorded run from its stored artifact must reproduce the original grade.

Independent expert keys and expert key cards

Two experts from different institutions each set the key of a task independently; each disagreement is resolved and recorded. Their agreement is published per family version as Gwet's AC1, a coefficient that stays stable at high agreement, where kappa-type coefficients behave paradoxically. The expert key card records the expected key and why, which findings count as over-reporting, borderline cases and how they are decided, and the grading reference ids the key relies on, and each key expert's specialty and qualification. Expert key cards are part of the closed side; a benchmark card shows a card record only when its reference is published.

Answer-key audit

After every round, the keys of the tasks run are audited against the round's failure attribution and expert artifacts, and the share of keys changed is published. A key found wrong is corrected in the next version; results across versions are labelled not comparable; historical results are never deleted. A fault found during a round on the initiative's side—a key, a task or the environment—voids that task version for every subject: the results on it are voided together, every subject is run again on the corrected version, and the voided task versions are listed with their attribution.

Independent acceptance

Acceptance is done by a reviewer who receives only the package and the acceptance checklist — not the builder's notes or self-assessment. Borderline decisions are taken by three independent votes.

Structural difficulty

Difficulty is a property of the task. Its class is computed by a published, versioned function from the task's structure. Every frozen benchmark version uses [email protected]: breadth, crossing, gaps, planted content, regional split and controls. Its next version takes required steps at the declared granularity, dependency depth, information sources, distractors, ambiguous judgement points and the starred task labels; its cut points are published before any benchmark version freezes under it. Cut points are fixed before any result.
A task enters the holdout only if every gate passes and its class is 2 or higher, or at least the family's higher holdout entry class where it sets one.
Pass-rate predictions per model tier are written down before testing and published beside the measured rates. They never decide entry and never change a class.
After every round the measured pass rate per class is published and checked for a monotonic fall. The check never changes a class; a function that fails it is revised in a new function version.

Retirement

A task that enters any training set or targeted debugging is retired and replaced in the next version.
A capability is retired when three frontier tiers reach pass^3 ≥ 0.95 for two consecutive versions, or when it is merged.
Retired items keep their pages; historical results are never deleted.

Grading reference integrity

Grading references carry their own rules, enforced at build time: every reference names a registered standard, a locator and a check date; a reference is listed only once a frozen version uses it; a veto requires a source checked on the original; where regional standards differ, the reference states the regional standard a task must declare. Each reference is also tried against a conformant answer written differently, so that grading never marks a correct alternative wrong.