Predict before you test

The practice

Before a task is first run, its builders write down a predicted pass rate for each model tier, with a one-line reason, dated. After the round, the predictions are published beside the measured rates on the benchmark card.

What the prediction is not

The prediction does not decide whether a task enters the holdout, and neither the prediction nor the measured rate ever changes a task’s difficulty class. Difficulty is computed from the task’s structure by a published function whose cut points are fixed before any result. A task enters the holdout when it passes every gate and its structural class is 2 or higher.

Why we expect it to help

• It prevents retrofitted difficulty. It is easy to call a task “hard” after seeing which models failed it. A prediction made earlier cannot be adjusted.
• It calibrates builders. Builders who consistently over- or under-estimate difficulty should learn where their intuition is off.
• It exposes leakage. A task that frontier models pass far more often than predicted is a candidate for contamination review.

These benefits are expected, not yet shown on MedGauge results. The prediction sits beside a second check that is part of the method: after every round, the measured pass rate per difficulty class is published and checked for a monotonic fall; a difficulty function that fails the check is revised in a new version, and no task’s class is changed by it.