MedGaugeCapabilities / A · Foundational
A.6 Research evidence semantics Active
Given a statement and its cited sources, the agent classifies each citation as supported, altered, unsupported, mis-cited or unverifiable, covering existence, version, numeric values and conditions of applicability. Graded by pairing the five-class labels against the reference.
Evidence: frozen. A frozen benchmark version tests this capability. No site-verified result exists yet.
- Domain
- A · Foundational
- Evidence form
- program
- Veto constraints
- —
- Depends on
- —
- Required by
- D.10D.12D.19D.2D.4D.6
- Internal task families
- evidence_check
- External anchors
- MedQA, HealthBench
- Version
- 0.1.0 · 0.1.0
Definition in detail
Medical claims are only as reliable as the sources they cite. A citation that exists but states a different value, version or population can carry a wrong recommendation into practice.
Inputs a task gives
- Statements with their citations
- Excerpts or full text of the cited sources, sometimes in more than one version
- Bibliographic metadata for each citation
Artifact required
A citation verdict table: statement_id, citation_id, label (supported, altered, unsupported, mis-cited, unverifiable), dimension checked (existence, version, numeric value, applicability condition) and evidence as a verbatim excerpt from the cited source.
How it is graded
A program pairs each label with the reference label, exact match over the five classes. Unverifiable is correct only when the supplied materials do not allow a decision; knowledge from outside the supplied sources does not count as verification. Evidence excerpts must exist verbatim in the supplied sources.
In scope
- Checking whether a cited source exists and is the cited version
- Comparing numeric values, units and effect directions between statement and source
- Checking that the population and conditions of applicability match
- Separating unverifiable citations from unsupported ones
Out of scope
- The guideline verification workflow with its own list format - graded under D.6
- Building literature catalogues or search strategies - graded under D.10
- Fabricated quotations in the agent's own artifact - X.1
Typical failures
Expected failure patterns; not measured rates.
- Labelling a statement supported when the source reports the value for a different subgroup or unit
- Confusing altered (the source says something different) with unsupported (the source does not address the claim)
- Answering from memory of a source that was not supplied instead of labelling it unverifiable
Benchmarks
- Guideline Evidence Check Active
Grading references
No grading reference is mapped to this capability.