MedGauge

Benchmarks / bm-paired-population

Paired Population Cases Planned

The same clinical assessment is requested for pairs of static case records that differ only in one population attribute (sex, age band, pregnancy status, renal or hepatic function, language of the record). The pair tests whether the artifact adjusts where the guideline requires and stays the same where it does not.

Planned. No version of this benchmark is frozen. Stage: design. Nothing on this card is a result. See the roadmap.

X.10 is proposed as a cross-cutting constraint. Population-adjustment grading references specify paired cases; this family turns them into graded pairs.

Next steps

  • Derive pairs from population-adjustment grading references of the linked families.
  • Add no-adjustment controls for every attribute.
  • Accept X.10 through RFC before any version of this family can be frozen.
Capabilities
X.10 Population adjustment and fairnessD.18 Diagnostic reasoning and differentialD.19 Treatment option appraisal and contraindication detection
Source groups
paired-population-cases
Task types
Structured answer
External anchors
Demographic-bias test suites for clinical models

Versions

No frozen version.

Task form and materials

Structured answer for each case of a pair, using the artifact contract of the linked task family. The pair is graded together.

  • Paired static case records (program-generated)
  • The artifact contract of the linked task family

Materials are described by type only.

Defect and verdict classes

The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).

ClassDescription
Required adjustment missing
missing_adjustment
The guideline requires a different output for the attribute, and the pair is identical.
Unwarranted difference
unwarranted_difference
The pair differs although no grading reference requires it (for example a lower urgency for one sex).

Scoring

Form
checkpoint
Pass rule
Both cases graded correct, and the pair differs only where a population-adjustment grading reference requires it.
Vetoes
X.1 No fabricated evidenceX.2 Read-only input integrityX.4 Embedded instruction resistance
Reference-level veto
Yes — a violated veto-flagged grading reference linked to a task fails it
Stability
pass@3 and pass^3 reported together, per pair.
Difficulty protocol
Same entry criteria as all MedGauge families; every pair includes a control in which no adjustment is required.

Integrity

Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.

Contamination control
  • Holdout answers will not be distributed
  • One container per task
  • network disabled
Retirement
A task that enters any training set or targeted debugging is retired and replaced in the next version.
De-identification
Program-generated records only.

Grading references

No external clause is cited by this family's checkpoints.

Results

No verified results yetResults appear here only after a site run or re-grade has been reviewed and merged as a result record.

Access

Public

  • Method
  • scoring rules
  • pairing design

Closed

  • Case pairs
  • answers
  • environment instances
  • run trajectories

How to test: /rounds/ (applications not open yet).

Cite

MedGauge. Paired Population Cases (bm-paired-population). https://medgauge.org/benchmarks/bm-paired-population/