SARANGA Database

Home › Benchmark

Benchmark: dependency extraction against human-made gold

How well does an extractor recover the dependencies between the results of a paper? We score extractors against two kinds of dependency structure made by people:

Node R is the share of the gold's statements found; edge precision, recall and F1 are over the dependencies between them, pooled over items, with 95% bootstrap intervals over items.

Extractor · goldItemsNode REdge PEdge REdge F1
LLM extractor (Claude Opus 5.5, the silver prompt)
Author diagrams1460.9730.8130.7580.784 [0.752, 0.812]
Lean blueprints910.9380.4580.7400.566 [0.528, 0.600]
Rule-based extractor of the arXiv corpus (distilled sentence rules)
Author diagrams1460.9740.7700.5200.621 [0.579, 0.663]
Lean blueprints910.9480.4180.3900.403 [0.306, 0.494]
Rule-based, learned tagger and edge model trained on the silver
Author diagrams1460.9740.7690.5540.644 [0.605, 0.682]
Lean blueprints910.9480.4340.5730.494 [0.447, 0.534]

Against TheoremGraph

On the 77 author-diagram papers that are also in the release of TheoremGraph (Kurgan et al., 2026):

ExtractorItemsNode REdge PEdge REdge F1
SARANGA, LLM extractor770.9790.8390.7590.797 [0.747, 0.838]
SARANGA, rule-based (learned tagger and edges)770.9780.7730.5100.614 [0.554, 0.669]
SARANGA, rule-based (sentence rules)770.9770.8060.4810.603 [0.537, 0.667]
TheoremGraph, all edges770.8940.4680.3840.422 [0.361, 0.474]
TheoremGraph, deterministic edges only770.8940.7900.2020.322 [0.223, 0.418]

Author diagrams usually draw only part of a paper's dependencies, so their precision counts only edges between results the diagram contains. Blueprint precision is lower partly because blueprints declare dependencies inside empty or placeholder proofs, which no reader of the text can recover. To score your own extractor on these gold sets, write to us (addresses below).