Real-source case: single-cell cell-type annotation¶
The third real-data case, after ontology names (vbo_canine) and clinical genetics (clinvar_germline),
moves to data: a cluster of a single-cell RNA-seq dataset, the marker genes that characterise it, and the
Cell Ontology term an annotator gives it. It asks what bioevidence adds when a model annotates clusters:
which of the model's errors can be caught without knowing the answer, and which cannot.
Not a benchmark of cell-type annotation methods. Author labels are the reference; they are themselves judgments (see Limits).
Sources¶
Six normal human datasets from CZ CELLxGENE Discover (CC BY 4.0), each
annotated by its authors with Cell Ontology terms: liver, kidney, adult duodenum, pancreas, skin, and the T,
NK and ILC compartment of a fetal lung atlas (a fine-grained immune subset). Reference releases: the Cell
Ontology (2026-06-08, full cl.obo, which keeps the disjointness axioms), the HGNC complete set
(2026-09-30) and the HuBMAP ASCT+B tables for liver, kidney, pancreas, skin and lung (CCF v2.0). See
sources/ATTRIBUTION.md.
prepare_source.py derives, from files downloaded once and recorded with their SHA-256 in
sources/manifest.json:
- Marker tables. Each author cell type with at least 30 cells is a "cluster". From raw counts
(normalised to 10,000 per cell, log1p), each gene gets
logfc(mean inside minus mean outside the cluster),pct_inandpct_out. The 50 genes with the highest positivelogfcamong those expressed in at least a quarter of the cluster form its table, named by current HGNC symbols. One table per dataset. - Truth. The authors' term for each cluster (
truth.jsonl), never shown to a model. - Reference. ASCT+B biomarker genes as
gene marker_of cell typeassertions (1,027). - Tasks. 70 clusters, one task each, showing the tissue, the assay and the top 20 markers with their statistics. Four per dataset (24) form the pilot split; the other 46 are held out.
The derived tables, the two reference releases and the ASCT+B projection are committed as gzip-compressed snapshots named by their SHA-256; the expression matrices are not.
Records and grounders¶
An annotation is a record under profile.yaml: cluster has_cell_type <CL term>, with one
marker_gene evidence item per cited gene, located as cluster=c03;gene=CD8A in the dataset's marker
table, in a supporting or contradicting evidence line. Five grounders check it, all generic:
| Grounder | What it checks here |
|---|---|
TableGrounder |
Each cited marker is one of this cluster's markers in the pinned table (BEV016 otherwise, naming the closest genes). A marker counts as verified only then, and the profile requires a verified marker. |
OntologyGrounder |
The term exists (BEV016), is current (BEV023), is a cell type under CL:0000000 (BEV024) and matches its label (BEV017, naming the term the label belongs to). |
GeneGrounder |
Cited genes are approved HGNC symbols (aliases and previous symbols: BEV023, naming the approved one). |
ReferenceGrounder |
A supporting marker that ASCT+B lists only for cell types the Cell Ontology declares disjoint from the claimed one (e.g. a B cell marker for a T cell) sends the record to an expert (BEV025). Used in protocol 1 (the pilot) only: ASCT+B biomarker lists are not specificity statements, and the pilot showed its alarms to be mostly spurious. |
SourceBytesGrounder |
The marker table's bytes hash to the record's frozen hash. |
The AI experiment is in evaluation/llm_benchmark/celltype_loop.py,
with results in the benchmark README.
On the 46 held-out clusters, six models working alone gave a Cell Ontology ID that does not match its label in about one answer in five. Bioevidence admitted none of these errors. In its feedback loop: - answers compatible with the authors' term rose from 176 to 205 of 276; - 8% of the annotations went to a person.
The external split, six datasets from other studies (81 clusters, mostly immune), tested the Cell Ontology definition check (protocol 3). The result was negative. Protein-level definitions such as "a mast cell has CCR3" or "an NK cell lacks CD3 epsilon" do not hold for transcripts. The check therefore flagged correct annotations as often as wrong ones, and fed back to the models it turned correct answers into wrong ones. The identifier layer held there too: 70 of 479 answers had an identifier error, and none was admitted.
Limits¶
- Author labels are the reference, at the granularity each study chose ("myeloid leukocyte", "stem cell"). Some are debatable from the markers alone (a duodenal "B cell" cluster led by JCHAIN and MZB1 looks like plasma cells). Scoring is ontology-aware (exact, coarser, finer, wrong), not an expert judgment.
- The marker table holds the top 50 upregulated genes of each cluster. A claim that a gene is absent cannot be checked against it, and is reported as not in the table.
- ASCT+B covers some organs and some cell types, and a gene it does not list is never reported. The disjointness axioms of the Cell Ontology cover major lineages only (e.g. epithelial cell and leukocyte, T cell and B cell lineage, lymphocyte and myeloid leukocyte), so a marker cited for the wrong type within a lineage is not caught this way.