Planned work

Roadmap

What MedGauge plans to add, and why. Every item on this page is planned. None has results, and none counts toward any score or capability level until it is built, validated and frozen.

Planned, not done. Planned benchmark cards have no frozen version and no tasks have passed the validation gates. Planned studies have no data.

Planned benchmark families

Benchmark	Capabilities	Stage	Next steps
Annotation QA Audit Planned	C.4 C.7	Planned	See card
Clinical Coding Audit Planned D.9 is defined, but no frozen benchmark tests it yet; this card plans one. Terminology and coding are core medical NLP evaluation areas; MedGauge needs an audit-form family that grades rule identification, not code generation.	D.9 A.1	Design	Select code systems and editions per region and register them in the standards registry. Link the family to grading references for the coding rules of each code system. Build items, pass both validation gates, write expert key cards, freeze version 0.1.0.
De-identification Audit Planned D.8 is defined in the capability registry, but no frozen benchmark tests it yet; this card plans one. De-identification is a standard evaluation area in medical NLP, so MedGauge needs an artifact-task family for it.	D.8 X.6	Design	Write the identifier specification and the seeded-defect generator (synthetic records only). Build reference solution and constructed wrong artifacts; pass the two validation gates. Write expert key cards; freeze version 0.1.0 with a designated calibration item.
Paired Population Cases Planned X.10 is proposed as a cross-cutting constraint. Population-adjustment grading references specify paired cases; this family turns them into graded pairs.	X.10 D.18 D.19	Design	Derive pairs from population-adjustment grading references of the linked families. Add no-adjustment controls for every attribute. Accept X.10 through RFC before any version of this family can be frozen.
Prescription and Reimbursement Review Planned	D.2 X.5	Planned	See card
Specialty Registry QC Planned	D.3 D.13 D.14	Planned	See card

Planned studies and infrastructure

Study · added 2026-10-23

Physician-rated output study Planned

A separate, clearly labelled study in which physicians rate selected artifacts and free-text outputs, to measure qualities that program grading cannot reach, such as the usefulness of an explanation or the appropriateness of a recommendation in context.

Program grading decides correctness of structured fields. Blind physician rating answers a different question and complements it. It will not replace program grading, and its results will not enter benchmark scores or capability levels.

Design

Blind, randomised presentation; raters do not see which model or harness produced an output.
Rating dimensions defined in advance, with written anchors for each scale point.
Inter-rater agreement measured and published with its values, not only described.
A held-out subset re-rated after a washout period to measure rater consistency.
Raters' specialties and conflict-of-interest declarations published with the results.
Results published as a separate study record, never mixed into leaderboard rows.

Not in scope

Grading any model as a clinician or by specialty.
Using a judge model in place of physicians.

Next steps

Write the study protocol and rating anchors; submit for method review.
Recruit raters from at least two institutions and specialties.
Pilot on synthetic material before any real case material is used.

Content · added 2026-10-23

Designate calibration items for frozen benchmarks Planned

Each frozen benchmark family needs at least one designated calibration item, published with its key, so that L1 can be derived once level thresholds are set.

Next steps

Select or build one calibration item per family; publish it in the next benchmark version.

Infrastructure · added 2026-10-23

Second-environment reproduction for registered harnesses Planned

Measure env_delta for each registered harness by running calibration items in a second environment, and record the result as a reproduction record.

Next steps

Run after calibration items exist; publish reproduction records.