- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces LiteratureRank, a literature-based framework for constructing model leaderboards in scientific domains where standardized benchmarks are unavailable. The proposed pipeline extracts reported evaluation results from scientific papers, defines evaluation cells to avoid direct comparison across incompatible settings, removes each paper’s own proposed model through a self-report exclusion strategy to reduce author bias, and converts extracted comparisons into pairwise battles for ranking using a weighted Bradley–Terry model with uncertainty estimation. The authors release 14 literature-derived leaderboards covering 1,082 models and approximately 401K retained comparisons across general LLM/VLM capabilities and several scientific domains. The paper validates the framework through analyses of ranking uncertainty, comparison graph properties, evidence scaling behavior, and temporal stability when extending the literature collection window. The results show that the generated leaderboards achieve substantial pairwise separation and remain relatively stable under additional literature accumulation. The main contribution of the paper is a systematic pipeline for transforming fragmented published evaluation results into auditable comparison graphs and model rankings. However, the current validation mainly focuses on the internal consistency and stability of the constructed rankings, while the extent to which these rankings accurately reflect real-world model capability or successfully reduce evaluation bias remains less directly evaluated.
Critical strengths
- A well-designed and auditable framework for literature-based leaderboard construction The paper introduces a comprehensive pipeline that transforms fragmented scientific evaluation results into structured comparison graphs and model rankings. The decomposition into evaluation cells, self-report exclusion, pairwise battle construction, and uncertainty-aware Bradley–Terry modeling provides a clear and reproducible methodology (see Figure 1; Sec. 3–4). This is consequential because existing literature evaluations are often difficult to aggregate due to heterogeneous datasets, metrics, and protocols, and the proposed framework offers a systematic way to organize such evidence without directly averaging incompatible scores.
- Large-scale empirical study and analysis across diverse domains The paper provides extensive experiments covering 14 leaderboards, 1,082 models, and more than 400K retained comparisons across LLM/VLM capabilities and scientific domains (see Table 1). Beyond producing rankings, the authors analyze uncertainty, comparison graph properties, and temporal robustness (see Figure 3; Figure 5; Table 2). This is consequential because it demonstrates the feasibility of applying literature-based aggregation at a scale beyond individual benchmarks and provides useful insights into how scientific evaluation evidence accumulates.
Critical weaknesses
- The paper does not sufficiently validate whether the proposed literature-based rankings reflect actual model capability The central contribution of the paper is to construct bias-reduced leaderboards from published literature, but the validation mainly focuses on internal properties of the generated rankings, such as uncertainty, graph connectivity, and temporal stability (see Figure 3, Figure 5, and Table 2). The paper explicitly states that it performs no ground-truth evaluation (Sec. 7). Therefore, it remains unclear whether a stable LiteratureRank leaderboard corresponds to the actual relative capability of models or simply reflects existing publication and evaluation patterns. This is consequential because ranking stability alone does not establish ranking correctness, and the lack of external validation weakens the main scientific claim of the paper.
- The effectiveness of the proposed self-report exclusion strategy is not empirically demonstrated The paper introduces self-report exclusion as a key mechanism for reducing author advocacy bias by removing evaluations of each paper’s own proposed model (see Sec. 3.2). However, the paper does not provide controlled ablation studies comparing rankings before and after applying this strategy, nor does it quantitatively measure the amount of bias reduction achieved. Moreover, the authors acknowledge that residual advocacy effects may remain because authors still select baselines and experimental settings (see Sec. 7). This is consequential because reducing evaluation bias is a central motivation of the work, but the effectiveness of the proposed debiasing mechanism is not directly established.
Questions for authors
- How do the authors validate that LiteratureRank rankings correspond to actual model capabilities rather than merely reflecting publication and evaluation patterns? The paper evaluates ranking stability, uncertainty, and evidence coverage (see Figure 3, Figure 5, and Table 2), but does not provide direct validation against independent evaluations or established benchmark rankings. Since the main goal is to construct reliable literature-based leaderboards, could the authors provide evidence that LiteratureRank scores correlate with independently measured model performance or explain why internal stability is a sufficient proxy for ranking validity?
- What quantitative evidence supports the effectiveness of self-report exclusion in reducing author bias? Self-report exclusion is presented as a key contribution of the paper (see Sec. 3.2), but the paper does not appear to provide an ablation comparing rankings with and without this mechanism. Could the authors report how much the leaderboard changes after removing self-reported results and provide quantitative evidence that the resulting rankings are less affected by advocacy bias?
Thank you. You point to Figures 3 and 5 and Table 2, and you are right that they describe internal properties of the boards, not whether the boards are right. Section 6.1 now starts from that point.
W1/Q1 (capability vs. publication patterns). We now test the boards against evidence they were not fitted on (§6.1; App. H).
Chatbot Arena is built from anonymous human votes and shares no data with us. It agrees with our text boards at ρ = 0.78–0.93 and with the vision boards at 0.79 and 0.85, and still agrees when the boards are refit on the literature alone. Epoch AI's own math evaluations agree with the Math board at 0.90.
The boards also predict unseen papers. Fitted without a paper, they pick its winners above chance on all 14 boards; fitted on earlier papers, they predict later ones on 11 of 14.
For the scientific fields we found a single external reference, ProteinGym (ρ = 0.82). The nested-window comparison in Figure 5 is now presented as evidence of stability only.
W2/Q2 (does exclusion reduce bias?). We re-extracted four boards keeping each paper's own model (§6.2; App. G). On the agentic literature, that model wins as if it were about 175 Elo stronger than when others evaluate it, and 80% of agentic models do better in their own reports than elsewhere. Keeping these results would change three of the agentic top ten and add seven models that only their developers have evaluated. On remote sensing, it would add 75 of 164 models. Exclusion keeps them out.
Showing that the excluded ranking is also more accurate is harder. Against Arena, Epoch AI and ProteinGym, the two versions do not differ significantly (Table 6). These references only cover well-known models, which self-reports affect least, and the models that enter through self-reports alone cannot be checked at all.
We say this openly in the paper and present exclusion as a conservative guard against a bias we can measure. That is also the sense in which we kept "Bias-Reduced" in the title: the bias is measured, and the released boards leave out the self-reported comparisons that carry it.