MedGaugeDataset registry
Datasets
Data on MedGauge is described evidence first: which capability it serves, which evaluation sets it was checked against, how much was verified by program and by experts, and the identifiability result — then licence and access.
No dataset is published yet. This page describes the method and access tiers that dataset cards will follow.
Types and the capabilities they serve
| Type | Serves | Isolation requirement |
|---|---|---|
| Pre-training / mid-training corpus | A | Zero overlap with task statements and answers; canary strings filtered |
| Evaluation set (closed) | A–E | Holdout answers never distributed; hash manifest released with the first holdout round |
| Tuning sets (SFT, preference, verifiable-reward) | B, C, D | Independent source material in a separate store; six-dimension similarity check |
| Trajectory and augmentation sets | B, C, D, E | Program-verified and expert-sampled; original records kept; processing map kept with training copies |
| Environments and rubric libraries | B, C, D | Snapshot environments versioned; rubric provenance recorded |
Access tiers
Open
Direct download under an open licence (for example CC BY 4.0). Only for data that is synthetic or otherwise not personal information.
Research
Application plus a data use agreement: no training on evaluation material, no redistribution, destruction date, incident reporting.
Commercial
By enquiry, after the capability evidence on the card has been reviewed. Not open yet.
Institutional de-identified data is never offered for download at any tier; it is used only inside the evaluation sandbox.
What every dataset card states
- Datasheet: motivation, composition, collection, preprocessing, uses, distribution, maintenance.
- Relation to benchmarks: capabilities served; evaluation sets isolated from; similarity-check method, result and date.
- Provenance: synthetic, public licensed or institutional de-identified; generator version and seed policy; de-identification method; legal status (not personal information, anonymised with an assessment, or de-identified personal information).
- Quality: share verified by program; expert-reviewed sample size, sampling method and agreement; identifiability scan result, stated as supporting evidence only.
- Format: tasks in the Harbor task format; trajectories as messages + tools + usage, with a mapping to the trajectory format registered in the standards registry (its name and version numbering not yet confirmed by a source check).
- Access: tier, licence per tier, DOI, Croissant metadata.
A card with status planned shows a planned banner and no download. Nothing is described as released before it is.
Card dimensions in full
Every dimension below is a field in dataset.schema.json or its datasheet block, validated at build time.
| Dimension | What the card states | Why it is there |
|---|---|---|
| Type | Pre-training corpus, evaluation set, SFT, preference, trajectory, augmentation, environment or rubric library | Different types carry different isolation duties |
| Capabilities served | Registry capability ids the data is meant to improve or measure | Ties data to the same ids results are reported on |
| Isolation | Evaluation sets checked against; six-dimension similarity method (patient, source, document, template, answer, environment state); result and date | Training data must not reproduce evaluation material |
| Origin | Synthetic, public licensed, or institutional de-identified | Determines legal status and whether download is possible at all |
| Generator and seeds | Generator version and seed policy for synthetic data | Lets a synthetic set be regenerated and audited |
| De-identification | Method (identifiers in the 18 Safe Harbor categories, consistent date shifting, technical registration, risk assessment) and assessment reference | De-identified is not anonymised; the card must say which applies |
| Legal status | Not personal information, anonymised with an assessment, or de-identified personal information | Decides access tier |
| Source licences | Licence of every upstream source | Downstream terms cannot exceed upstream terms |
| Size | Tasks, trajectories, tokens | Basic scale |
| Format | Task format, trajectory format, mapping to an interchange format | Reuse without custom parsers |
| Program verification | Share of items verified by program | Labels that a program can check are reported as such |
| Expert review | Sample size, sampling method, agreement, how error is extrapolated | Expert claims come with their denominator |
| Identifiability scan | Scanner coverage and result, labelled as supporting evidence only | A clean scan is not proof of absence |
| Intended and excluded uses | Datasheet "uses" section, including uses the data must not be put to (for example unvalidated clinical decisions) | Responsible reuse |
| Known limitations | Coverage gaps, population skews, label noise known at release | Prevents over-generalisation |
| Access | Tier, licence per tier, application route | One place to see who can get what |
| Identifiers | Version, DOI, Croissant metadata, citation | Citable and machine-readable |
| Maintenance | Maintainer, update cadence, erratum process, retirement | Datasets change; the card says how |
Schema: dataset.schema.json. De-identification method: Isolation and the closed set.