Clinical Temporal Audit Active

Capabilities

D.1 Clinical temporal auditA.5 Temporal and knowability reasoningA.4 Multi-source cross-analysis

Source groups

inpatient-record-timing

Task types

Detect and log · Structured answer

External anchors

LongMedBench, PhysicianBench

Versions

What a version reports depends on its subsets. Leaderboard metrics (pass^3, pass@3) are computed on holdout tasks only, so a version with 0 holdout tasks produces no leaderboard row. No level is issued under methodology 0.1.0, which sets no level threshold; once thresholds are set, L1 and above require a calibration subset. Holdout and calibration tasks are to be designated in a later version before the first round (see the roadmap).

Task form and materials

Directory-style task in the Harbor task format. Inputs are read-only. The agent writes one structured findings file at a fixed submission path; the end state is graded by program.

• Inpatient record bundle split across several tables (progress notes, operation notes, test records)
• Event and report timestamp tables
• Data dictionary and audit task rules

Materials are described by type only.

Artifact fields

Defect and verdict classes

The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).

Illustration of the task shape

progress_notes.csv   note_id, written_at, text
  N-12, 2031-03-04 09:10, "... CT on admission showed ... (see report R-7) ..."
reports.csv          report_id, issued_at
  R-7, 2031-03-04 15:42

{"findings": [{"record_ref": "progress_notes/N-12",
               "defect_class": "cited_before_available",
               "evidence": "CT on admission showed",
               "available_at": "2031-03-04T15:42"}],
 "undeterminable": []}

Scoring

Form

Program-graded end state

Pass rule

Every seeded defect cluster recalled, zero over-reported items, zero fabricated evidence, and zero veto hits.

Vetoes

X.1 No fabricated evidenceX.2 Read-only input integrityX.3 No corrected valuesX.4 Embedded instruction resistance

Reference-level veto

Not applied to this family

Stability

pass@3 and pass^3 reported together; single-run scores are not published.

Metrics reported

recall_by_defect_type, precision, evidence_error_rate, steps, tokens

Difficulty protocol

A task enters the holdout set only if it passes every gate, including a full-score reference solution and every constructed wrong artifact blocked, and its structural difficulty class is 2 or higher. Pre-test predictions are dated and published beside the measured pass rates; they never decide entry or change a class.

Scoring details

1. The verifier matches findings to seeded defect clusters by record reference and defect class; wording is not graded.
2. A task passes only if every cluster is recalled and no finding falls outside a cluster (zero over-reporting).
3. Evidence must occur verbatim in the inputs; a non-verbatim quotation is a fabricated-evidence veto (X.1).
4. Input files are hashed before and after the run; any change is a veto (X.2). Writing a corrected time into the record is a veto (X.3).
5. Items that cannot be decided from the inputs must be listed under undeterminable; guessing them is scored as over-reporting.
6. The continuous score is the share of checkpoints passed, as the methodology defines it for every family; it credits partial recall, falls where an over-reporting checkpoint fails, and is reported alongside, never used for pass/fail.

Integrity

Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.

Contamination control

• Holdout answers are not distributed with any package
• One container per task, network disabled
• Task materials never enter training or targeted tuning
• Similarity checks against any training data built for the same capability

Retirement

A task that enters any training set or targeted debugging is retired and replaced in the next version.

De-identification

Identifiers in the 18 HIPAA Safe Harbor categories removed, dates shifted by a consistent per-patient offset (not the Safe Harbor method as such), technical registration per GB/T 37964 and risk assessment per GB/T 42460.

Grading references

No external clause is cited by this family's checkpoints.

Results

Access

How to test: /rounds/ (applications not open yet).

Cite

MedGauge. Clinical Temporal Audit (bm-clinical-temporal-audit), version 0.1.0. https://medgauge.org/benchmarks/bm-clinical-temporal-audit/