Benchmark: dependency extraction against human-made gold
How well does an extractor recover the dependencies between the results of a paper? We score extractors against two kinds of dependency structure made by people:
- Author diagrams (146 papers): figures in which a paper's authors drew how its results depend on each other. Found by searching the LaTeX source of all of arXiv; the extractor reads the paper with the diagram removed.
- Lean blueprints (91 blueprints): the informal plans of Lean formalisation projects, where every statement lists the results it uses. The extractor reads the blueprint with these markers removed.
Node R is the share of the gold's statements found; edge precision, recall and F1 are over the dependencies between them, pooled over items, with 95% bootstrap intervals over items.
| Extractor · gold | Items | Node R | Edge P | Edge R | Edge F1 |
|---|---|---|---|---|---|
| LLM extractor (Claude Opus 5.5, the silver prompt) | |||||
| Author diagrams | 146 | 0.973 | 0.813 | 0.758 | 0.784 [0.752, 0.812] |
| Lean blueprints | 91 | 0.938 | 0.458 | 0.740 | 0.566 [0.528, 0.600] |
| Rule-based extractor of the arXiv corpus (distilled sentence rules) | |||||
| Author diagrams | 146 | 0.974 | 0.770 | 0.520 | 0.621 [0.579, 0.663] |
| Lean blueprints | 91 | 0.948 | 0.418 | 0.390 | 0.403 [0.306, 0.494] |
| Rule-based, learned tagger and edge model trained on the silver | |||||
| Author diagrams | 146 | 0.974 | 0.769 | 0.554 | 0.644 [0.605, 0.682] |
| Lean blueprints | 91 | 0.948 | 0.434 | 0.573 | 0.494 [0.447, 0.534] |
Against TheoremGraph
On the 77 author-diagram papers that are also in the release of TheoremGraph (Kurgan et al., 2026):
| Extractor | Items | Node R | Edge P | Edge R | Edge F1 |
|---|---|---|---|---|---|
| SARANGA, LLM extractor | 77 | 0.979 | 0.839 | 0.759 | 0.797 [0.747, 0.838] |
| SARANGA, rule-based (learned tagger and edges) | 77 | 0.978 | 0.773 | 0.510 | 0.614 [0.554, 0.669] |
| SARANGA, rule-based (sentence rules) | 77 | 0.977 | 0.806 | 0.481 | 0.603 [0.537, 0.667] |
| TheoremGraph, all edges | 77 | 0.894 | 0.468 | 0.384 | 0.422 [0.361, 0.474] |
| TheoremGraph, deterministic edges only | 77 | 0.894 | 0.790 | 0.202 | 0.322 [0.223, 0.418] |
Author diagrams usually draw only part of a paper's dependencies, so their precision counts only edges between results the diagram contains. Blueprint precision is lower partly because blueprints declare dependencies inside empty or placeholder proofs, which no reader of the text can recover. To score your own extractor on these gold sets, write to us (addresses below).