MedGauge

Benchmarks / bm-deid-audit

De-identification Audit Planned

A data-protection officer checks a de-identified clinical record set before release: residual identifiers, inconsistent date shifting across a patient's records, and quasi-identifier combinations that raise re-identification risk.

Planned. No version of this benchmark is frozen. Stage: design. Nothing on this card is a result. See the roadmap.

D.8 is defined in the capability registry, but no frozen benchmark tests it yet; this card plans one. De-identification is a standard evaluation area in medical NLP, so MedGauge needs an artifact-task family for it.

Next steps

  • Write the identifier specification and the seeded-defect generator (synthetic records only).
  • Build reference solution and constructed wrong artifacts; pass the two validation gates.
  • Write expert key cards; freeze version 0.1.0 with a designated calibration item.

Open questions

  • Whether to include a transform variant scored by exact comparison, or keep detect-and-log only.
Capabilities
D.8 De-identification output verificationX.6 No individually identifiable output
Source groups
deid-output-verification
Task types
Detect and log · Transform and output
External anchors
i2b2/n2c2 de-identification tasks

Versions

No frozen version.

Task form and materials

Detect-and-log over a de-identified record set, with an optional transform variant in which the agent de-identifies a synthetic record set and its output is compared with the expected output. Identifiers are reported by location, never by value.

  • De-identified record set (program-generated; no real patient data)
  • De-identification specification (identifier categories, date-shift rule)
  • Release checklist

Materials are described by type only.

Defect and verdict classes

The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).

ClassDescription
Residual direct identifier
residual_identifier
An identifier of a category the specification requires removing is still present.
Inconsistent date shift
inconsistent_date_shift
Dates for the same patient are shifted by different offsets, breaking intervals.
Quasi-identifier combination
quasi_identifier_combination
A combination of retained fields narrows a record to a very small group.

Scoring

Form
Program-graded end state
Pass rule
Every seeded residual identifier and date-shift inconsistency located, zero over-reporting, and no identifier value reproduced in the artifact.
Vetoes
X.1 No fabricated evidenceX.2 Read-only input integrityX.3 No corrected valuesX.4 Embedded instruction resistanceX.6 No individually identifiable output
Reference-level veto
Not applied to this family
Stability
pass@3 and pass^3 reported together.
Difficulty protocol
Same entry criteria as all MedGauge families.

Integrity

Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.

Contamination control
  • Holdout answers will not be distributed
  • One container per task
  • network disabled
Retirement
A task that enters any training set or targeted debugging is retired and replaced in the next version.
De-identification
Program-generated records only; seeded identifiers are synthetic.

Grading references

No external clause is cited by this family's checkpoints.

Results

No verified results yetResults appear here only after a site run or re-grade has been reviewed and merged as a result record.

Access

Public

  • Method
  • scoring rules
  • task form with synthetic illustration

Closed

  • Record sets
  • answers
  • environment instances
  • run trajectories

How to test: /rounds/ (applications not open yet).

Cite

MedGauge. De-identification Audit (bm-deid-audit). https://medgauge.org/benchmarks/bm-deid-audit/