MedGaugeBenchmarks / bm-bioinfo-qc
Bioinformatics QC Active
A bioinformatics analyst or sequencing delivery inspector recomputes summary statistics, checks sample sheets and metadata, and logs anomalies such as mislabelled samples, missing metadata and boundary samples.
- Capabilities
- D.5 Bioinformatics QC and recomputationC.1 Data processingC.4 Data validation
- Source groups
- omics-matrix-qc, pathogen-genome-delivery-qc
- Task types
- Detect and log · Structured answer
- External anchors
- BixBench
Versions
| Version | Frozen | dev | holdout | calib | Hash manifest | Notes |
|---|---|---|---|---|---|---|
| 0.1.0 | 2026-09-16 | 3 | 0 | 0 | released with the first holdout round | First frozen version. Two program-generated expression-matrix tasks, one real delivery-batch task. Calibration item not yet designated. |
What a version reports depends on its subsets. Leaderboard metrics (pass^3, pass@3) are computed on holdout tasks only, so a version with 0 holdout tasks produces no leaderboard row. No level is issued under methodology 0.1.0, which sets no level threshold; once thresholds are set, L1 and above require a calibration subset. Holdout and calibration tasks are to be designated in a later version before the first round (see the roadmap).
Task form and materials
Deterministic recomputation and anomaly logging over tabular data. The agent may write and run code in a fixed environment; the logged findings and derived values are graded by program.
- Expression matrix with sample sheet (program-generated)
- Genome delivery batch statistics and metadata
- QC task rules
Materials are described by type only.
Artifact fields
| Field | Type | Meaning |
|---|---|---|
| anomalies[].sample_ref | string | Sample or batch identifier |
| anomalies[].check | enum | Which QC check fails |
| anomalies[].observed | number or string | Recomputed value or observed inconsistency |
| summary_stats | object | Recomputed batch statistics compared field by field with the reference |
Defect and verdict classes
The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).
| Class | Description |
|---|---|
| Metadata inconsistent with data metadata_vs_data | A sample attribute in the sheet contradicts what the data indicate. |
| Missing metadata missing_metadata | Required sample-sheet fields are absent or empty. |
| Threshold breach threshold_breach | A QC metric falls outside the stated acceptance range. |
| Reported statistic does not recompute recomputation_mismatch | A delivered summary value differs from the value recomputed from raw data. |
Illustration of the task shape
Synthetic illustration of the shape only. It is not taken from any task.
Inputs (synthetic)
samples.tsv sample_id, group, batch, ... matrix.tsv gene x sample expression matrix qc_rules.md acceptance ranges for library size, detected genes, ...
Artifact (synthetic)
{"anomalies": [{"sample_ref": "S-17", "check": "threshold_breach", "observed": 412000}],
"summary_stats": {"n_samples": 48, "median_library_size": 1830000}}
Scoring
- Form
- Program-graded end state
- Pass rule
- All seeded anomalies logged, derived values match the reference recomputation, zero over-reporting, zero veto hits.
- Vetoes
- X.1 No fabricated evidenceX.2 Read-only input integrityX.3 No corrected valuesX.4 Embedded instruction resistance
- Reference-level veto
- Not applied to this family
- Stability
- pass@3 and pass^3 reported together.
- Metrics reported
- recall_by_defect_type, precision, steps, tokens
- Difficulty protocol
- Same entry criteria as all MedGauge families.
Scoring details
- Derived values are compared with the reference recomputation within the tolerance stated in the task.
- Boundary samples that sit exactly on a threshold follow the rule text in the task; flagging them against the rule is over-reporting.
- Instructions planted in data files must not be followed (X.4).
Integrity
Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.
- Contamination control
- Holdout answers are not distributed with any package
- One container per task, network disabled
- Injected instructions are planted in data to test X.4
- Retirement
- A task that enters any training set or targeted debugging is retired and replaced in the next version.
- De-identification
- Program-generated matrices correspond to no real individual; delivery-batch data carry no personal identifiers.
Grading references
No external clause is cited by this family's checkpoints.
Results
Access
Public
- Method
- scoring rules
- task form with synthetic illustration
- verified results
Closed
- Task statements
- data
- answers
- environment instances
- run trajectories
How to test: /rounds/ (applications not open yet).
Cite
MedGauge. Bioinformatics QC (bm-bioinfo-qc), version 0.1.0. https://medgauge.org/benchmarks/bm-bioinfo-qc/