Why a program, not a judge model

The choice

Every scored MedGauge task is an artifact task: static materials in, a structured artifact out, and a program that decides whether the artifact is correct. No model grades another model, and no grader reads free-form prose.

What it buys

• Repeatability. The same artifact always gets the same grade. Every stored run is re-graded, and the result must agree with the recorded grade on every checkpoint, or the result is not published.
• Auditability. A disputed grade can be traced to a field, a checkpoint and a rule — not to a judge’s reading of a paragraph.
• No style bias. A verbose, confident answer gets no credit for tone. The grader sees the fields, not the rhetoric.
• Hard safety gates. Fabricated evidence, altered inputs and corrected values are checked mechanically and fail the run.

What it costs

Program grading needs work to be expressible as fields. Many clinical requirements can be: an urgency level, a per-cause status in a differential, a contraindication flag, a logged missing fact. Some cannot — the quality of an explanation to a patient, for example. MedGauge does not force those into the scored track; they are never scored.

Where it stops

Conversational evaluation with rubric grading needs a calibrated judge and physician consensus on the rubric. On MedGauge it is listed as not open, planned. Where a task family needs a dialogue-like setting today, the task gives a static record of a conversation that has already happened and grades the triage or assessment form produced from it.

Two validation gates keep the program honest

A program grader is only as good as its key. Before any model is run, the reference solution must score full marks, and a set of deliberately wrong artifacts must all fail: empty, flag-everything, one extra item, fabricated value, smuggled correction, modified input, executed injection. If any wrong artifact passes, the grader is fixed before the task is used.