Contribution

Contribute tasks and checks

MedGauge accepts tasks, checks with provenance and grading references from clinical and research staff, institutions and individual experts. A contributed task enters the private grading repository and is never published on this site. The rules are those of the MedGauge Methodology, version 0.1.0, sections 20, 21, 23 and 27.

Contributions are sent by e-mail. This site runs no submission server and nothing on it collects information. Send contributions to [email protected].

Who can contribute

Clinical and research staff, with work at any of four points: task design, preparation of materials, reference solutions, and professional review (setting a key independently, reviewing an expert key card, taking part in the expert baseline).
Institutions. Name one responsible contact, the scope of the materials concerned, and the hand-over requirements that apply to them.
Individual experts. Write first with the direction you have in mind, the time you can give and the kind of work that suits you.

Roles are kept apart for every task: its builder, the two experts who set its key and its independent acceptor are different people, and the author of a grading reference builds no task that tests it. Expert-reviewer standing requires the professional qualification of the role the family serves and current practice in that role.

How a contribution starts

Propose. Describe the scenario, the benchmark family and the capabilities it would serve, in a few lines and without task materials.
Check feasibility together. The maintainers confirm the family, the intended subset, the contribution unit and the basis of the materials (synthetic, de-identified or public).
Build one task and review it across. One small task first calibrates the requirements. It is built in full, run through the submission check, sent, and taken through the gates and independent acceptance; further tasks follow the same route.

What can be contributed

Unit	What to supply	How to send it	How it is accepted
Task naming its checks	A task submission in the form task-submission.schema.json: checks with provenance, difficulty components, verifier reference, capability ids, family, materials hash, the task labels T1–T10, the declared action granularity and, where the family reports them, the required action set, judgement points, seeded defects, compliance checkpoints and variant relation, with the six declarations—and nothing else	The filled template and the task package, by e-mail (below)	The eleven gates, independent acceptance and freezing; the task enters the private grading repository, never the public one
Check with provenance	A check with its provenance; for a task-internal rule, the task rules it belongs to	The template with only `methodology_version`, `family`, `capabilities` (optional), `checks` and `declarations` filled; submission check with `--unit check`	Reviewed with the family it serves; enters a version through the task gates
Grading reference	The standard (a standards registry id), the locator inside it, the rule on a named artifact field, the polarity, the check form, the layer; for a regional split, each source with its locator and the jurisdiction statement	A record under grading-reference.schema.json, by e-mail until the repository is public, then by pull request	A second person checks the clause on the original source and records the date; the conformant-alternative test is run; listed once a frozen benchmark version uses it
Paired run	A request to run a subject with and without a component under a fixed model and harness version	A request by e-mail	Run by the initiative; only site-computed figures are recorded

Registry records for harnesses, tools, components, standards and datasets, corrections to existing records, and methodology articles carrying an evidence grade are also accepted; see CONTRIBUTING.md in the repository.

Never accepted

An answer key or a correction for an existing task.
A holdout task from a party that will be measured on that family.
A score computed outside the initiative.
Any task, check or reference that carries the identity of its origin.
Any material containing individually identifiable information.

Prepare a task

Download	Use
medgauge-task-submission-template.yaml	The task-submission form for one task, every field with a comment. Fill it in and send it.
medgauge-task-submission-fields.csv	The same fields as a table (field, required, type, allowed values, description, example), for working in a spreadsheet.
medgauge-task-submission-example.yaml	A filled form for a toy, non-medical task, with placeholder ids and hashes. It passes the submission check.
medgauge-submission-check.mjs	The submission check, which you run on your own computer with Node.js 18 or later. Built-in modules only; it reads the files you name and makes no network connection.

Assemble the task package in one directory: the materials directory (what the subject receives), the artifact contract file, the reference solution artifact, at least one correct artifact written differently from it (gate 10), the verifier, and the task rules every check names. Every hash in the form must be reproducible from the files in the package.
Compute the hashes with the submission check: hash-dir for the materials directory (the sha256 of its hash manifest), hash-file for each single file.
Fill the template. One form per task. The form admits no name, institution, source or other origin. If you cannot produce YAML or JSON, fill the field table instead; the maintainers transcribe it into the form and send it back for your confirmation before it is registered.
Run the submission check and resolve every problem it reports. A pattern scan finds only obvious cases; a clean scan is not evidence that material is free of identifiable information.

node medgauge-submission-check.mjs hash-dir package/materials
node medgauge-submission-check.mjs hash-file package/reference/artifact.json
node medgauge-submission-check.mjs check form.yaml --scan package
node medgauge-submission-check.mjs check checks.yaml --unit check

For full JSON Schema validation with another tool, the form's schema and the two schemas it refers to are published at /schemas/; the submission check carries the same rules.

Send it

To: [email protected]
Subject: MedGauge contribution: <unit> <family>, for example MedGauge contribution: task bm-example-toy. Units: task, check, grading-reference, paired-run.
Attachments: The filled form (.yaml or .json), and the task package as one archive (.zip or .tar.gz).
In the message: The archive's sha256 (hash-file on the archive); whether your side is measured on the family, which decides whether a holdout task can be accepted; for de-identified material, the de-identification basis (methodology section 11.2); and how you wish to be named in publications (name and ORCID, or the collective name).
Large archives: Say so in the message without attaching the archive; the maintainers reply with a transfer method.
Encryption: To send encrypted, write first without attachments; the maintainers reply with the method to use.

What happens after sending

Each submission receives a submission number, and from then on is known only by it. It moves through a logged sequence of states:

submitted → screening → under review → approved / rejected / more information → frozen → closed

State	What it means
submitted	Received and registered against the task-submission schema. Arrival is confirmed with the submission number only.
screening	Form valid, six declarations true, archive and form hashes match the package, holdout rule, not a key for an existing task, identifiable-information and origin scan of the materials (gate 8).
under review	The gates are run, two experts from different institutions each set the key independently, and the expert key card is reviewed.
more information	Something the contributor can repair is missing or failed; the notice names it. The submission returns to screening when the answer arrives.
approved	An independent acceptor, given only the task package and the acceptance checklist, re-runs gates 1 and 2, reads the expert key card and signs acceptance. The task receives a task id that is not derived from its origin.
rejected	A rule above is not met, or a gate failed that the contributor cannot repair; the notice gives the reason.
frozen	The task is part of a frozen benchmark version. The contributor is told the version, not the subset or the task id.
closed	After freezing, rejection or withdrawal. A submission can be withdrawn in any state before it is frozen.

The contributor is notified on more information, approved, rejected and frozen. A notice states the decision and its reason; it never discloses grading detail, keys or other tasks. The declarations are the contributor's undertaking and replace none of the gates or the independent acceptance.

Closed set

Task statements, materials, answers, reference solutions, verifiers and run trajectories stay in the private grading repository. This site shows the methodology, schemas, benchmark cards, hash manifests and site-verified records.
Per-task labels, difficulty components and the grading references a task tests are closed; class and label distributions per version are public.
A party measured on a family contributes no holdout task to it. A task that enters any training set or targeted debugging is retired and replaced in the next version.

Attribution and standing

Contributors keep copyright in what they contribute and license it under the repository's licences: Apache License 2.0 for code and schemas, CC BY 4.0 for content and registries. Contributions are made under the contributor's own name and identifier.
No task, check or reference carries the identity of its origin. Attribution is made in publications: they are authored by the MedGauge Initiative, with contributors named with their ORCID and contributor role (CRediT) and a contribution statement; a contributor who does not consent to be named is recorded under the collective name.
Accepted contributions are recorded. One accepted unit gives contributor standing; three accepted units, or three clause checks on originals confirmed by a second person, give reviewer standing. Standing lapses after two consecutive benchmark versions without a qualifying record.
This site makes no offer of remuneration. Any arrangement is agreed for the specific project.

Privacy

Only institutional and contact information is collected: the details you put in your e-mail. It is kept by the submissions coordinator outside every repository. The submission register knows a submission only by its submission number and a pseudonymous contact key.
Task materials must contain no individually identifiable information. De-identified material follows methodology section 11.2, and any residual identifier fails gate 8. Do not send health information about any person in an e-mail.
How long contact information and rejected packages are kept: not set.