Benchmarks / bm-annotation-qa-audit

Annotation QA Audit Planned

A data-quality auditor inspects the version history of an annotated evaluation suite and finds defects in the gold labels, the grader and the reference solutions, such as a grader that logs but never vetoes, or a reference solution that has gone stale.

Planned. No version of this benchmark is frozen. Nothing on this card is a result. See the roadmap.

Capabilities: C.4 Data validation C.7 Program verification
Source groups: suite-version-history-audit
Task types: Detect and log
External anchors: BixBench

Versions

No frozen version.

Task form and materials

Detect-and-log over a suite snapshot; each finding names the defect class and the file evidence.

Snapshot of an evaluation suite with its version history
Grader source and reference solutions (read-only)

Materials are described by type only.

Artifact fields

Field	Type	Meaning
defects[].location	string	File and version where the defect is observed
defects[].class	enum	One of the defect classes below
defects[].evidence	string	Verbatim excerpt or diff hunk supporting the finding

Defect and verdict classes

The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).

Class	Description
Gold-label error label_error	A gold label contradicts the source material.
Grader records but does not enforce grader_non_enforcing	A grader computes a check but never uses it to fail a submission.
Stale reference solution stale_reference	The reference solution no longer matches the current task version.
Version drift version_drift	Task, answer and grader versions are out of step across the history.

Illustration of the task shape

Synthetic illustration of the shape only. It is not taken from any task.

Inputs (synthetic)

suite/          tasks/, answers/, grader/, CHANGELOG.md
history.log     ordered list of commits touching the suite

Artifact (synthetic)

{"defects": [{"location": "grader/check_fields.py@v3", "class": "grader_non_enforcing",
              "evidence": "result['format_ok'] = ok  # never read"}]}

Scoring

Form: Program-graded end state
Pass rule: All seeded defects logged with correct class and evidence; zero over-reporting.
Vetoes: X.1 No fabricated evidence X.2 Read-only input integrity X.4 Embedded instruction resistance
Reference-level veto: Not applied to this family
Stability: pass@3 and pass^3 reported together.
Difficulty protocol: Same entry criteria as all MedGauge families.

Scoring details

Findings are matched by location and class; evidence must be verbatim.
Reporting a design choice documented in the suite as a defect counts as over-reporting.

Integrity

Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.

Contamination control

Holdout answers will not be distributed
One container per task
network disabled

Retirement

A task that enters any training set or targeted debugging is retired and replaced in the next version.

De-identification

Not applicable; materials contain no patient data.

Grading references

No external clause is cited by this family's checkpoints.

Results

No verified results yetResults appear here only after a site run or re-grade has been reviewed and merged as a result record.

Access

Public

Method
scoring rules
task form with synthetic illustration

Closed

Suite snapshots
answers
environment instances
run trajectories

How to test: /rounds/ (applications not open yet).

Cite

MedGauge. Annotation QA Audit (bm-annotation-qa-audit). https://medgauge.org/benchmarks/bm-annotation-qa-audit/