MedGaugeContent as code
Schemas
Every registry record is validated against one of these JSON Schemas (draft 2020-12) at build time, followed by cross-table reference checks. A failing record fails the build.
| Schema | Description | Required fields |
|---|---|---|
| benchmark.schema.json | A benchmark card (benchmark registry): scenario, capabilities served, task form, scoring, integrity controls, frozen versions and the public/closed boundary. | id, name_en, capabilities, status, versions, scenario_en, task_form, materials, scoring, difficulty_protocol, integrity, deidentification, access, source_groups |
| capability.schema.json | A capability or cross-cutting constraint (capability registry): definition, domain, dependencies and the artifact that demonstrates it. | id, name_en, domain, definition_en, evidence_grade, status, version |
| dataset.schema.json | A dataset card (dataset registry): type, provenance, quality checks, access tier, relation to benchmarks and maintainer. | id, name_en, type, status, relation_to_benchmarks, provenance, quality, access, maintainer, version |
| grading-reference.schema.json | One external clause that a verifier checkpoint is built on: the standard, its locator, and the rule the checkpoint applies. A reference is listed only once a frozen benchmark version uses it and the clause has been checked on the original source. | id, benchmarks, standard, locator, checked_on, when, rule, polarity, check_form, veto, version |
| harness.schema.json | One version of a registered harness (harness registry): runtime, model slots, tools, limits, sandbox, autonomy class and reproduction command. | id, name, kind, base_runtime, limits, sandbox, autonomy, reproduction, maintainer, license, status, version |
| level-override.schema.json | A signed level assignment that a program cannot derive (L5): the model and harness, the capability, the evidence, the signer and an expiry date. | id, model, harness, harness_version, capability, benchmark_version, level, evidence, signed_by, signed_on |
| model.schema.json | A model entry (model registry): vendor, display name, snapshots and access route. | id, vendor, display_name, snapshots, access |
| reproduction.schema.json | A second-environment reproduction: the benchmark version, harness and model re-run elsewhere, whether the reference solution still scores full marks, and env_delta with its sign. | id, org, benchmark, benchmark_version, harness, harness_version, model, oracle_full_score, env_delta, date |
| result.schema.json | A site-reviewed result record under methodology 0.1.0: benchmark version, model snapshot, harness version, run conditions, the three runs of every task with error attribution, scores by subset and dimension with their intervals, the exam outcome and its lower bound, vetoes by category, amounts with the price table version, and the hash of the run records. | id, run_outcomes, methodology_version, benchmark, benchmark_version, model, model_snapshot, harness, harness_version, run_conditions, scores, cost, run_records_hash, submitted_by, verified_by_site, created_at, run_mode |
| roadmap.schema.json | Planned work that is not a benchmark card (studies, infrastructure). Every item is planned; nothing here is a result. | items |
| standard.schema.json | A standard or guideline the site cites (standards registry): publisher, version, URL, scope and how MedGauge uses it. | id, name, publisher, url, scope, how_we_use_en, region |
| task-metadata.schema.json | Structure of the [metadata] table in each task's task.toml under methodology 0.1.0. Task files live on the closed side (private grading repository and task packages); this schema is published so that harness builders, contributors and auditors know which fields exist. It describes format only and carries no task content. It adds the task labels T1–T10, the declared action granularity, difficulty components, the required action set with its precedence, judgement points, seeded defects, compliance checkpoints, veto categories and perturbation variants. | source_group, task_id, task_version, family, capabilities, split, source, submission_path, labels, action_granularity, difficulty_components, checkpoints_total, veto_categories, difficulty_function, difficulty_class |
| task-submission.schema.json | Task-submission form for a contributed task (contribution unit 'task naming its checks', methodology section 23). The task arrives as a task package and enters the private grading repository, never the public one. The form admits only checks with provenance, difficulty components, the verifier reference, capability ids, the family, the materials hash, the task fields of methodology 0.1.0 and the contributor's declarations; it admits no origin field. This schema describes format only and carries no task content. | methodology_version, family, capabilities, intended_subset, materials, artifact_contract_sha256, checks, verifier, reference_solution_sha256, conformant_alternative_sha256, difficulty_components, labels, action_granularity, checkpoints_total, veto_categories, declarations |
| tool.schema.json | A tool, skill, plugin or MCP server in the tool registry: protocol, medical scope, side effects, conformance testing and licence. | id, kind, name, summary_en, spec, source, medical_scope, conformance, safety, license, maintainer, status, version |