MedGaugeBenchmarks / bm-coding-audit
Clinical Coding Audit Planned
A coding auditor checks the diagnosis and procedure codes on discharge records against the documentation and the applicable coding rules, and names the rule each wrong code violates.
Planned. No version of this benchmark is frozen. Stage: design. Nothing on this card is a result. See the roadmap.
D.9 is defined, but no frozen benchmark tests it yet; this card plans one. Terminology and coding are core medical NLP evaluation areas; MedGauge needs an audit-form family that grades rule identification, not code generation.
Next steps
- Select code systems and editions per region and register them in the standards registry.
- Link the family to grading references for the coding rules of each code system.
- Build items, pass both validation gates, write expert key cards, freeze version 0.1.0.
Open questions
- How to present licensed code-system content in tasks without redistributing it.
- Capabilities
- D.9 Coding auditA.1 Medical semantic understanding
- Source groups
- diagnosis-coding-audit
- Task types
- Item-by-item judgement
- External anchors
- MedCode-style coding benchmarks
Versions
No frozen version.
Task form and materials
Item-by-item judgement on each coded entry: compliant, non-compliant (with the rule violated and verbatim documentation evidence), or needs information. The auditor does not supply a replacement code (X.3).
- Discharge summary and front-sheet codes (program-generated or de-identified)
- Coding rules excerpt for the code system stated in the task (for example ICD-10 national clinical edition or ICD-11)
Materials are described by type only.
Defect and verdict classes
The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).
| Class | Description |
|---|---|
| Code not supported by documentation unsupported_code | The documentation does not establish the condition or procedure the code asserts. |
| Specificity specificity | A less specific code is used where the documentation supports a more specific one, or the reverse. |
| Principal diagnosis sequencing sequencing | The principal diagnosis does not follow the sequencing rule stated in the task. |
| Code system or version code_system_version | The code does not exist in the code system edition the task states. |
Scoring
- Form
- Program-graded end state
- Pass rule
- Every coded entry judged correctly with the governing rule and verbatim evidence; zero replacement codes written.
- Vetoes
- X.1 No fabricated evidenceX.2 Read-only input integrityX.3 No corrected valuesX.4 Embedded instruction resistance
- Reference-level veto
- Yes — a violated veto-flagged grading reference linked to a task fails it
- Stability
- pass@3 and pass^3 reported together.
- Difficulty protocol
- Same entry criteria as all MedGauge families.
Integrity
Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.
- Contamination control
- Holdout answers will not be distributed
- One container per task
- network disabled
- Retirement
- A task that enters any training set or targeted debugging is retired and replaced in the next version.
- De-identification
- Program-generated, or de-identified by removing identifiers in the 18 HIPAA Safe Harbor categories and shifting dates by a consistent per-patient offset (not the Safe Harbor method as such; backed by a re-identification risk assessment).
Grading references
No external clause is cited by this family's checkpoints.
Results
Access
Public
- Method
- scoring rules
- task form with synthetic illustration
Closed
- Records
- answers
- environment instances
- run trajectories
How to test: /rounds/ (applications not open yet).
Cite
MedGauge. Clinical Coding Audit (bm-coding-audit). https://medgauge.org/benchmarks/bm-coding-audit/