MedGaugeMethodology
MedGauge publishes everything about how it measures — task forms, scoring rules, integrity checks, schemas, hashes — and nothing that would let a model be tuned to the test: task statements, data, answers, environments and trajectories stay closed.
Five commitments
- Artifact tasks, graded by program. A task gives static materials and requires a structured artifact; a program grades the end state. No conversation, no judge model.
- The harness is part of the subject. A result belongs to a model inside a registered harness version.
- The grading basis is explicit. Where a checkpoint rests on an external clause, the clause is registered as a grading reference with its locator and the rule applied.
- Scores come only from the site. Results are produced by site runs (or site re-grading of submitted runs), reviewed, and merged as records. Nothing on a board is hand-typed, and every number can be recomputed from the public record set.
- Dimensions apart; the conclusion is an exam. Ten dimensions are reported each on its own, with no total. A subject passes a benchmark version only if every task passes in all three runs with no veto; a run error is a failed run. The conclusion is stated with the lower bound its number of tasks allows.
The normative text is the MedGauge Methodology, version 0.1.0 (2026-10-29), its first published version. While the version is below 1.0.0, a change that alters a published number is a new minor version with a non-comparability notice. See the changelog.
What MedGauge does not do
| Not done | Why | Status |
|---|---|---|
| No "AI doctor" grading. | Levels are defined only for model × harness × capability × benchmark version. No page rates a model as a clinician, by specialty, or by disease. | Permanent |
| Main track is artifact tasks. | Every scored task gives materials and grades a structured end state. Requirements that can only be judged by rubric are never scored. | Current |
| Conversational evaluation | Multi-turn dialogue graded by rubric needs a judge model plus physician consensus calibration. Where a task family needs a dialogue-like setting, MedGauge uses a static record of a conversation that has already happened and grades the triage or assessment artifact produced from it. | Not open · planned |
Also out of scope: knowledge-quiz leaderboards (saturated), note-generation leaderboards, and substituting for regulatory testing of medical devices. Planned additions, including a separate physician-rated output study, are listed on the roadmap.
How MedGauge relates to other evaluation sites
MedGauge borrows proven structures from general and medical evaluation sites and adds the parts specific to medical agent work.
| Site | What it established | What MedGauge takes, and adds |
|---|---|---|
| HELM · MedHELM | Holistic, scenario-by-metric reporting with a published taxonomy | A capability taxonomy that every task attaches to — with medical-practice capabilities graded by artifact rather than by answer text |
| LMArena | Large-scale human preference rankings | Nothing from the method: MedGauge scores are program-decided, not preference-decided. The difference is deliberate. |
| Artificial Analysis | Independent re-runs with disclosed settings and cost | Fixed, disclosed run conditions per record and a cost column |
| SWE-bench | Execution-graded tasks; submissions with trajectories; agent scaffolds named per entry | Model × harness as the row; end-state grading; and a closed holdout, because a public task set can overlap with training data (a study of SWE-bench Verified) |
| HealthBench | Physician-written rubrics for health conversations | The complementary track: MedGauge grades work products, not conversations, and keeps conversational rubric evaluation out of its scored track |
| Epoch AI · FrontierMath | Independent benchmarking hub; a held-out problem set with only a few problems public | Closed holdout with a hash manifest per frozen version; public calibration tasks only |
| ARC Prize | Public / semi-private / private tiers; offline evaluation | Network-disabled containers; tiered access |
Architecture
Sections
- Scoring — general rules, gates and vetoes, run errors, the ten dimensions and ten task labels, the exam and its lower bound, intervals and the order of rows.
- Integrity — two validation gates, independent expert keys, independent review, answer-key audit, structural difficulty, retirement.
- Isolation and closed set — what is public and closed, contamination control, training isolation, de-identification.
- Reproducibility — fixed run conditions, run records, re-grading, recomputation, adapter checks, run modes.
- Articles — shorter pieces on method and reasoning, each with an evidence grade (verified, analysis or hypothesis).