ClinVar expert-review kit¶
Status: ready for reviewers; no reviews collected yet.
The ClinVar case found 95 sampled variants where the validator and NCBI's own 2023-09 review status disagree, almost always because the validator holds back a pathogenic classification that carries a dissenting submission. Three years later those classifications were markedly less stable, but stability is not correctness. This kit asks domain experts directly, following the project's gold-standard protocol:
Given only the submissions ClinVar held in September 2023, is a pathogenic / likely pathogenic classification sufficient for each intended use?
What reviewers get, and what they do not¶
| Reviewers see | Hidden from reviewers |
|---|---|
An opaque case code (e.g. C3F9A21) and the gene symbol |
VariationID and submission accessions (they would reveal the current ClinVar record) |
| Every 2023-09 submission: submitter, assertion criteria / expert panel / practice guideline, classification, collection method, date last evaluated | ClinVar's aggregate review status and stars |
| The rubric defining the three uses in curation terms | Validator decisions, reason codes and which cases are "divergent" |
| 2026-09 outcomes |
Cases: all 95 divergent variants plus 95 agreeing controls matched by review-status
stratum, so reviewers cannot tell which is which. The first 20 cases in code order form a
calibration set (split development), which is discussed and excluded from scoring; the
other 170 are the test set. Each case needs one statement label and three use labels.
Time. Plan for roughly 3–6 minutes per case, i.e. about 10–19 hours per reviewer for all
190. Sizes are adjustable (--controls-per-divergent, --calibration).
Steps¶
Run from the repository root after uv sync --frozen --extra dev.
1. Prepare packets (once):
uv run --frozen python evaluation/clinvar_review/prepare_packets.py --output artifacts/clinvar-review
artifacts/clinvar-review/reviewer_packet/: send the same folder to every reviewer (review_workbook.xlsxwith dropdowns,labels.csv/evidence.csvas a fallback,RUBRIC.md).artifacts/clinvar-review/maintainer/: keep private until all reviews are in.key.jsonmaps codes to variants and holds the secret salt;predictions.csvholds the validator and NCBI decisions to be scored;manifest.draft.jsonstarts the reference-set manifest.
2. Calibration round. Reviewers label only the calibration rows, independently, and
return their workbooks. Import them (step 4; blank rows are skipped), run the agreement
report (step 5), then meet to discuss the calibration cases and clarify the rubric. If the
rubric wording changes, raise its version in RUBRIC.md and prepare_packets.py.
3. Test round. Reviewers complete the test rows independently, without discussion.
4. Import each reviewer's workbook:
uv run --frozen python evaluation/clinvar_review/import_sheets.py reviewer_R1.xlsx \
--key artifacts/clinvar-review/maintainer/key.json --reviewer-id R1 \
--qualification "Clinical molecular geneticist, 8 years of variant curation" \
--annotated-at 2026-10-15 --output artifacts/clinvar-review/annotations.csv
Each case becomes three rows (one per use) in the protocol's annotation format. A half-filled row, an unknown code or a second import of the same reviewer is rejected, and the file is left unchanged.
5. Agreement before adjudication:
uv run --frozen bioevidence review agreement artifacts/clinvar-review/annotations.csv \
--output artifacts/clinvar-review/agreement.json
Krippendorff's α (with a bootstrap 95% interval), Cohen's κ for two reviewers, and raw agreement, per use. Report these whatever they are.
6. Adjudicate disagreements:
uv run --frozen bioevidence review adjudication-sheet artifacts/clinvar-review/annotations.csv \
--output artifacts/clinvar-review/adjudications.csv
One blank row per disagreeing case-use. A third expert, or the reviewers together, fills in
the final labels, adjudicator ID, time and reasons, using both reviewers' rows in
annotations.csv (linked by annotation_ids_json). Original reviews are never edited.
7. Score the validator and NCBI's review status against the resolved labels (test split):
uv run --frozen bioevidence review score --annotations artifacts/clinvar-review/annotations.csv \
--adjudications artifacts/clinvar-review/adjudications.csv \
--predictions artifacts/clinvar-review/maintainer/predictions.csv \
--output artifacts/clinvar-review/score.json
Results are reported for all cases and separately for divergent and control cases. The
key number is on the divergent subset: where the two disagree, which one do experts side with?
8. Freeze and publish:
uv run --frozen bioevidence review freeze --manifest artifacts/clinvar-review/maintainer/manifest.draft.json \
--annotations artifacts/clinvar-review/annotations.csv \
--adjudications artifacts/clinvar-review/adjudications.csv \
--file artifacts/clinvar-review/maintainer/predictions.csv --file artifacts/clinvar-review/maintainer/key.json \
--dataset-id clinvar-germline-expert-review --version 1 --frozen-at 2026-11-01T00:00:00Z \
--output artifacts/clinvar-review/manifest.json
Then commit annotations.csv, adjudications.csv, agreement.json, score.json,
manifest.json and the key into evaluation/clinvar_review/results/, and update the ClinVar
case README, which currently states that no independent expert annotation exists.
Model reviewers (#22)¶
The same packet goes to the six models of the LLM benchmark, to measure which parts of expert review they could
take over. model_reviewers.py drives three command-line agents: Claude Code (claude), Codex (codex) and
Antigravity (agy, for Gemini). Each vendor contributes a frontier model and a fast one:
- Claude Opus 5.5 and Claude Haiku 4.5;
- GPT-6-Astra and GPT-5.6-Luna;
- Gemini 3.1 Pro and Gemini 3.8 Flash.
Each model gets exactly what a human reviewer gets, the rubric and one case under its code, and answers through a JSON schema.
# Where the CLIs are installed; reads only reviewer_packet/, never the key
python3 evaluation/clinvar_review/model_reviewers.py run \
--packet artifacts/clinvar-review/reviewer_packet --output artifacts/clinvar-review/models \
--set calibration # start small; --limit N, --codes, --reviewers
# Maintainer, with the key: protocol annotations and predictions, then the analysis
uv run --frozen python evaluation/clinvar_review/model_reviewers.py export \
--output artifacts/clinvar-review/models --key artifacts/clinvar-review/maintainer/key.json
uv run --frozen python evaluation/clinvar_review/analyze_models.py \
--models artifacts/clinvar-review/models --output artifacts/clinvar-review/models/analysis
- Isolation. One case per call. Every call is a fresh, non-interactive session in an empty temporary directory, so the model cannot read the key or other cases.
- No lookups.
- Claude Code runs with every tool disabled.
- Codex runs without browser tools or your user config and MCP servers, in a read-only sandbox.
- Antigravity cannot switch off web search. Its tool calls are detected from the event stream.
- A case answered after a tool call is retried once. If a tool is used again, the case is kept but marked
later_information_seen = yes. - Quota. A CLI whose account is out of quota stops that reviewer; nothing is recorded for its remaining cases,
and the next run resumes them. The Claude models share one call slot. Several installs can write to one output
folder (for example Claude on Windows and Codex and Antigravity in WSL);
manifest.jsonrecords the CLI version and path that answered for each model. - Record.
manifest.jsonrecords model IDs, CLI versions and hashes of the rubric, prompt and schema.raw/keeps every answer.runs.csvlists time, tokens and tool use per call. Re-running resumes.
What is published, and when¶
Model labels could steer a human reviewer, so they follow the same rule as the key: per-case model labels stay
private until all expert reviews are in. So does anything that compares models with the key, the validator or
NCBI's review status. Until then analyze_models.py --public writes only:
- how each model ran: answers, tool use, later information seen, time;
- how much the models agree with each other, per use (Krippendorff's alpha, bootstrap interval);
- the decision rule below;
- the SHA-256 of the withheld per-case files.
The published summary is in results/models.
Results so far¶
All six models reviewed all 190 cases. - Clean runs. No model used a tool or said it recognised later information. - Two runs. The calibration round (20 cases) ran on 2026-09-28; the rest ran on 2026-10-08. GPT-6-Astra's Codex CLI was updated in between, from 0.153.4 to 0.160.0. - Two accounts' quotas. Claude ran on the Windows install's account. Antigravity's account ran out of quota partway through, and its 35 unanswered cases were rerun after the reset.
On the 170 test cases, before any expert label:
| Label | All six agree | Krippendorff's alpha (95% CI) |
|---|---|---|
| Research summary | 91.8% | 0.12 (0.03–0.21) |
| Clinical reference | 55.3% | 0.53 (0.43–0.62) |
| Expert reference | 38.2% | 0.49 (0.43–0.55) |
| Statement correct? | 86.5% | 0.47 (0.24–0.62) |
- High raw agreement means little where nearly every case gets one label. For research summaries the six models agree on 92% of cases, but alpha is close to chance.
- For the uses that depend on the evidence, the models often split. All six agree on about half the cases or fewer.
So their consensus cannot stand in for an expert's decision. Whether any of them is right is what the expert labels will show (#23).
Analysis against the expert labels¶
Once the expert labels are resolved, analyze_models.py --annotations … --adjudications … --predictions
maintainer/predictions.csv adds, on the test split:
- Scores for each model, the models' majority (at least four of six), the validator and NCBI's review status,
per use and per subset (divergent, control).
- Error correlation. For each pair of models: their misses, the misses they share, P(B misses | A misses), and
Cohen's kappa of the miss indicators. Shared misses make a second model's opinion worth little.
- Does agreement mean correctness? How often the six models are wrong when they all agree, and when they split.
- Decision statement, by a rule fixed before any expert label exists. Per use:
- Models alone, if the models' majority wrongly admits cases that experts would not admit at a rate whose
exact one-sided 95% upper bound is below 5%, and wrongly blocks fewer than 20% of the cases experts would
admit, on the divergent and the control subset each.
- Deterministic rules, if the validator meets the same two conditions.
- Experts otherwise.
Model labels are not independent human annotations. Keep them in their own files, never in annotations.csv,
and never use them to resolve an expert disagreement.
Interpreting the result¶
- With two independent reviewers and adjudication this is an independently reviewed
reference set for this task. With one reviewer,
freezelabels it a single-reviewer reference set; say so wherever the numbers are quoted. - It measures agreement with expert judgment of evidence sufficiency for a use, on one sample of one registry. It does not establish variant pathogenicity or clinical validity.
- Do not change the profile in response to the test-set labels. Any later profile version needs a separately reported evaluation.
- Publish reviewer IDs and one-line qualifications only with each reviewer's consent.