acceptodds
Under review as a conference paper at ICLR 2027

LiteratureRank: Bias-Reduced Leaderboards from Scientific Literature

Abstract

We introduce LiteratureRank, a literature-based approach for ranking methods in scientific domains that lack standardized leaderboards. The needed comparisons are often already published, but scores from different papers are not comparable, and most tables come from papers that propose one of the models they compare. We compare scores only within an evaluation cell, the scores that share a paper, a dataset and a metric, and expand each cell into pairwise wins and losses. To reduce author bias, self-report exclusion removes each paper's own proposed model from its tables during LLM-based extraction, so a model is ranked only from comparisons reported by other papers. A pair-weighted Bradley–Terry model turns these battles into scores with uncertainty intervals. We release 14 leaderboards and their comparison datasets, spanning general LLM and VLM capabilities and six scientific domains, which rank 1,082 models from 401k comparisons. A median of 74% of model pairs per board have non-overlapping 95% intervals, interval width shrinks as in the number of comparisons, and widening one board's collection window by a year leaves the order of its existing models nearly unchanged (Spearman ). The rankings can be updated as new papers appear with little additional computation.

Primary area:
datasets and benchmarks
Keywords:
method comparison, metascience

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rY69OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LiteratureRank, a literature-based framework for constructing model leaderboards in scientific domains where standardized benchmarks are unavailable. The proposed pipeline extracts reported evaluation results from scientific papers, defines evaluation cells to avoid direct comparison across incompatible settings, removes each paper’s own proposed model through a self-report exclusion strategy to reduce author bias, and converts extracted comparisons into pairwise battles for ranking using a weighted Bradley–Terry model with uncertainty estimation. The authors release 14 literature-derived leaderboards covering 1,082 models and approximately 401K retained comparisons across general LLM/VLM capabilities and several scientific domains. The paper validates the framework through analyses of ranking uncertainty, comparison graph properties, evidence scaling behavior, and temporal stability when extending the literature collection window. The results show that the generated leaderboards achieve substantial pairwise separation and remain relatively stable under additional literature accumulation. The main contribution of the paper is a systematic pipeline for transforming fragmented published evaluation results into auditable comparison graphs and model rankings. However, the current validation mainly focuses on the internal consistency and stability of the constructed rankings, while the extent to which these rankings accurately reflect real-world model capability or successfully reduce evaluation bias remains less directly evaluated.

Critical strengths
  1. A well-designed and auditable framework for literature-based leaderboard construction The paper introduces a comprehensive pipeline that transforms fragmented scientific evaluation results into structured comparison graphs and model rankings. The decomposition into evaluation cells, self-report exclusion, pairwise battle construction, and uncertainty-aware Bradley–Terry modeling provides a clear and reproducible methodology (see Figure 1; Sec. 3–4). This is consequential because existing literature evaluations are often difficult to aggregate due to heterogeneous datasets, metrics, and protocols, and the proposed framework offers a systematic way to organize such evidence without directly averaging incompatible scores.
  2. Large-scale empirical study and analysis across diverse domains The paper provides extensive experiments covering 14 leaderboards, 1,082 models, and more than 400K retained comparisons across LLM/VLM capabilities and scientific domains (see Table 1). Beyond producing rankings, the authors analyze uncertainty, comparison graph properties, and temporal robustness (see Figure 3; Figure 5; Table 2). This is consequential because it demonstrates the feasibility of applying literature-based aggregation at a scale beyond individual benchmarks and provides useful insights into how scientific evaluation evidence accumulates.
Critical weaknesses
  1. The paper does not sufficiently validate whether the proposed literature-based rankings reflect actual model capability The central contribution of the paper is to construct bias-reduced leaderboards from published literature, but the validation mainly focuses on internal properties of the generated rankings, such as uncertainty, graph connectivity, and temporal stability (see Figure 3, Figure 5, and Table 2). The paper explicitly states that it performs no ground-truth evaluation (Sec. 7). Therefore, it remains unclear whether a stable LiteratureRank leaderboard corresponds to the actual relative capability of models or simply reflects existing publication and evaluation patterns. This is consequential because ranking stability alone does not establish ranking correctness, and the lack of external validation weakens the main scientific claim of the paper.
  2. The effectiveness of the proposed self-report exclusion strategy is not empirically demonstrated The paper introduces self-report exclusion as a key mechanism for reducing author advocacy bias by removing evaluations of each paper’s own proposed model (see Sec. 3.2). However, the paper does not provide controlled ablation studies comparing rankings before and after applying this strategy, nor does it quantitatively measure the amount of bias reduction achieved. Moreover, the authors acknowledge that residual advocacy effects may remain because authors still select baselines and experimental settings (see Sec. 7). This is consequential because reducing evaluation bias is a central motivation of the work, but the effectiveness of the proposed debiasing mechanism is not directly established.
Questions for authors
  1. How do the authors validate that LiteratureRank rankings correspond to actual model capabilities rather than merely reflecting publication and evaluation patterns? The paper evaluates ranking stability, uncertainty, and evidence coverage (see Figure 3, Figure 5, and Table 2), but does not provide direct validation against independent evaluations or established benchmark rankings. Since the main goal is to construct reliable literature-based leaderboards, could the authors provide evidence that LiteratureRank scores correlate with independently measured model performance or explain why internal stability is a sufficient proxy for ranking validity?
  2. What quantitative evidence supports the effectiveness of self-report exclusion in reducing author bias? Self-report exclusion is presented as a key contribution of the paper (see Sec. 3.2), but the paper does not appear to provide an ablation comparing rankings with and without this mechanism. Could the authors report how much the leaderboard changes after removing self-reported results and provide quantitative evidence that the resulting rankings are less affected by advocacy bias?
1 comment

Thank you. You point to Figures 3 and 5 and Table 2, and you are right that they describe internal properties of the boards, not whether the boards are right. Section 6.1 now starts from that point.

W1/Q1 (capability vs. publication patterns). We now test the boards against evidence they were not fitted on (§6.1; App. H).

Chatbot Arena is built from anonymous human votes and shares no data with us. It agrees with our text boards at ρ = 0.78–0.93 and with the vision boards at 0.79 and 0.85, and still agrees when the boards are refit on the literature alone. Epoch AI's own math evaluations agree with the Math board at 0.90.

The boards also predict unseen papers. Fitted without a paper, they pick its winners above chance on all 14 boards; fitted on earlier papers, they predict later ones on 11 of 14.

For the scientific fields we found a single external reference, ProteinGym (ρ = 0.82). The nested-window comparison in Figure 5 is now presented as evidence of stability only.

W2/Q2 (does exclusion reduce bias?). We re-extracted four boards keeping each paper's own model (§6.2; App. G). On the agentic literature, that model wins as if it were about 175 Elo stronger than when others evaluate it, and 80% of agentic models do better in their own reports than elsewhere. Keeping these results would change three of the agentic top ten and add seven models that only their developers have evaluated. On remote sensing, it would add 75 of 164 models. Exclusion keeps them out.

Showing that the excluded ranking is also more accurate is harder. Against Arena, Epoch AI and ProteinGym, the two versions do not differ significantly (Table 6). These references only cover well-known models, which self-reports affect least, and the models that enter through self-reports alone cannot be checked at all.

We say this openly in the paper and present exclusion as a conservative guard against a bias we can measure. That is also the sense in which we kept "Bias-Reduced" in the title: the bias is measured, and the released boards leave out the self-reported comparisons that carry it.

Reviewer_HFt2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes LiteratureRank, a method for building leaderboards from evaluation results reported across scientific papers. It uses an agent-assisted search to collect papers and LLMs to extract results. Scores within the same evaluation cell are converted into pairwise wins and losses. The proposed model's own results are excluded to reduce self-report bias, and a weighted Bradley-Terry model produces the final rankings with confidence intervals. The authors release 14 leaderboards covering 1,082 models and approximately 401K retained comparisons across general LLM/VLM capabilities and six scientific domains. They report three main findings: (1) a median of 73.7% of model pairs have non-overlapping 95% confidence intervals, (2) interval width decreases approximately as n^(-0.38) with the number of comparisons, and (3) expanding the literature collection window for the vision leaderboard largely preserves the existing rankings (Spearman correlation = 0.991).

Critical strengths

The main strength is the effort to bring together a large collection of published evaluation results and make the comparisons traceable to their original sources. The 14 leaderboards and roughly 401K retained comparisons could be a useful research resource, even independently of the proposed ranking method. However, without sufficient validation of extraction accuracy and whether the comparisons are actually valid, it is difficult to judge how reliable this resource is.

Critical weaknesses
  1. It is unclear whether one overall ranking makes sense, or what is really new here. The paper combines results from very different tasks and gives each model one overall score. I am wondering why this work. For example, suppose A beats B on classification, B beats C on retrieval, and C beats A on generation. All three results may be perfectly correct because different models are good at different things. Yet the paper combines them into one ranking, without checking whether such a ranking is meaningful.

This problem is also not new. Network meta-analysis (NMA) has studied how to combine direct and indirect comparisons for many years. It has established ways to check whether results from different studies can reasonably be combined and whether indirect conclusions agree with direct evidence. The paper does not perform these checks. Even using Bradley-Terry to produce an overall ranking from pairwise results has been explored in NMA. Tbh, I struggle to see what is new in the ranking method itself, beyond automatically collecting results from scientific papers.

  1. The paper does not show that its rankings are less biased, and the reported confidence in them may be misleading. It claims to reduce bias by removing results reported by a model's own authors. But does this actually make the final ranking less biased? I do not see an ablation that answers this. For example, suppose a paper proposes model X and compares it with A and B. Removing X does not fix the problem if A was poorly tuned or the authors selected baselines that made their method look good. Worse, the paper's reported search rules can leave out independent evaluation papers that introduce no new model or benchmark. Such papers could provide valuable evidence for checking the rankings.

There are also basic checks missing from the pipeline. The authors do not report how accurately the LLM extracts results from papers. They also assume that results appearing together in a table are directly comparable, even though some numbers may have been copied from other papers or obtained under different settings. These errors can directly affect the final ranking. Finally, there is a serious problem with the reported confidence intervals. One table with 10 models produces 45 pairwise comparisons, but these are not 45 independent experiments. They all come from the same underlying results. The paper recognizes this dependence, yet its HC1 calculation does not account for the shared errors within a table or paper. This puts the reported confidence intervals, the claim that around 74% of model pairs can be clearly separated, and the conclusions about how confidence improves with more comparisons into question.

Questions for authors

Questions

  1. Appendix A excludes papers that only compare existing models on existing benchmarks, but allows manual inclusions. Were such independent evaluation papers included through manual exceptions? If so, how many, and what share of the final comparisons came from them?

  2. Does the implementation check whether results grouped into the same evaluation cell were actually obtained under the same experimental protocol, rather than copied from different papers? If so, how is this checked? If not, are such comparisons retained?

Suggestions

I think the problem of building leaderboards from published research is worth studying. However, the following evidence is needed to support the paper's current claims.

  1. Check whether the overall rankings are meaningful. Test whether indirect comparisons correctly predict direct comparisons, and whether the rankings remain reasonably consistent across different tasks.

  2. Demonstrate that bias is actually reduced. Compare rankings before and after self-report exclusion against independent evaluation results. This would directly test the paper's main proposed contribution.

  3. Check the reliability of the extracted evidence. Check whether comparisons within evaluation cells were actually made under comparable conditions.

  4. Recalculate the confidence intervals. Account for comparisons coming from the same paper or evaluation table, and check whether the main findings about ranking confidence still hold.

I would also encourage the authors to explain more clearly how their contribution differs from existing network meta-analysis methods.

5 comments

We are grateful for this review. The point about clustering changed several of our conclusions, and the request for a before/after comparison with independent evaluations shaped the new self-report analysis.

Confidence intervals. As you put it, a ten-model table gives 45 comparisons, not 45 experiments, and HC1 ignored that. Section 4.3 now clusters by source paper (CR1), and App. E cross-checks the result with a paper-level bootstrap. The intervals widen by a factor of about 2.4. The share of separated pairs falls from 73.7% to 48.1% (34% under the bootstrap), and the precision slope flattens from n^-0.38 to n^-0.18. We rewrote the claims that depended on these numbers (§6.3–6.4, conclusion).

Relation to NMA. We agree that the ranking model is not new, and the paper now says so (§2; App. F). Evrenoglou et al. (2026) already fit an extended Bradley–Terry model to study-level wins, losses and ties. What is new here is the setting:

  • networks are extracted automatically at up to 10^5 comparisons per board;
  • there is no common effect scale across hundreds of metrics;
  • author advocacy plays the role of sponsorship bias, and we both exclude and measure it.

We also ran a cross-paper node-split in the NMA sense. On the general boards, direct and indirect evidence agree about as well as noise allows; three scientific boards fall short (App. H).

One overall ranking. Your A > B > C > A example is the right worry, so we measured it directly (App. H, p. 28). Only 1–7% of decided model triples are cyclic, against 25% at random. Rankings fitted on disjoint halves of a board's benchmarks agree at ρ = 0.70–0.95 on the general boards, pathology and spatial transcriptomics.

Protein was the clear exception, and subsampling showed that this is not a matter of sample size. The protein board is therefore now restricted to fitness prediction. We also say plainly what the score is: average propensity to win over the comparisons a field actually reports, not general ability.

Does exclusion reduce bias? The re-extraction in §6.2 keeps each paper's own model. On Agentic, that model wins as if it were about 175 Elo stronger than when others evaluate it, and keeping these results would change three of the top ten.

Following your suggestion, we compared the rankings before and after exclusion with Arena, Epoch AI and ProteinGym (Table 6). No difference is significant, and the directions disagree. These references only cover well-known models, while the models most affected by self-reports have no outside evaluation, and other leaderboards such as BenchLM are built from developer numbers. So we can show the bias and its effect on the board, but not a gain in accuracy, and the paper states this.

Baseline tuning and selection. Agreed: removing the proposed model's rows does not stop authors from choosing weak baselines or under-tuning them. A ranking fitted only on independent sources (benchmark papers plus leaderboard exports) agrees with the full one at τ = 0.98 on Math and 0.99 on Agentic (App. G), so one author's choices do not move a board much. The two-tier display also keeps single-paper models out of the main ranking. Under-tuning that is systematic across many papers cannot be measured from the literature, and §7 lists it as a limitation.

Independent evaluation papers (Q1). On the general boards, every paper comes in through the benchmark track, so independent evaluation papers are the main source (Table 7). On protein fitness, benchmark papers supply 73.6% of comparisons. What the workflow leaves out are papers that only re-run existing models on existing benchmarks, which cannot be enumerated reliably and often reuse earlier numbers.

No papers were added through manual exceptions; those lists are empty. Checking this also showed that the submission described the pipeline incompletely. Section 3 and App. A now describe the official reports and leaderboard exports used on the general boards.

Extraction accuracy. In a 432-comparison audit against the PDFs, 94.2% were fully correct and 98.8% of scores were right (App. J). Rebuilding the boards without the resulting fixes changes little (τ ≥ 0.97 on 13 of 14 boards).

Copied numbers and mixed settings (Q2). There was no automatic per-table protocol check, and §3.4 now says that comparability within a table rests on the authors' account. The audit measured how often that fails: both scores came from the same protocol in 93.3% of comparisons, and the four sampled documents whose tables mixed conditions were removed. Verbatim duplicates and scores the papers mark as cited are rare, and removing them changes no top ten.

Unsampled papers may still contain mixed tables. We therefore injected errors at the measured rates, including the 6.7% protocol mismatch, and the main-tier rankings stayed stable (τ = 0.76–0.95). Remaining sources of mixing, mainly different agent harnesses and cited baselines, are listed in §7.

Thank you for the detailed clarifications and the additional supportive evidence. However, I still believe that the core concerns remain unresolved as of now.

  1. My original concern was not about ranking stability across different benchmarks (or subsets). I primarily questioned whether results from fundamentally different tasks can meaningfully support one common model-strength scale in the first place. The new consistency experiments do not address that.

  2. It seems from the newer results that LiteratureRank appears most convincing on some boards where existing independent benchmarking infrastructure already supplies much of the evidence. I still need to figure out how much of the claimed success demonstrates the new literature-extraction framework, and how much comes from reusing already organized evaluation results.

  3. Also, regarding the new repositioning in contrast to works on NMA, I would like to point out that combining existing benchmark results into an overall ranking is already established. So is collecting externally reported scores from multiple sources. See Artificial Analysis, Epoch AI's Capabilities Index, and BenchLM. Automatic extraction of evaluation results from scientific PDFs for leaderboard construction is also an established research problem within the "Automated Leaderboard Generation (ALG)" community (as one of several examples, checkout SciLead, EMNLP, 2024 or this position paper at EMNLP 2025 is also useful "A Position Paper on the Automatic Generation of Machine Learning Leaderboards"). Given all these precedents, I still find it difficult to identify the distinct methodological contribution.

  4. Also, I am still concerned regarding the advantage of excluding self-report. I can see that the new experiments show that models tend to perform better in their own reports, and that excluding these results changes the rankings. However, this does not establish that the resulting rankings are less biased. In fact, the results show that there is no significant improvement when compared against independent evaluations, and exclusion performs slightly worse in one held-out evaluation.

    What starts becoming interesting is when, in the additional results, the authors also show that explicitly modeling the self-report advantage is a viable alternative to discarding the observations. This seems to be competing with the primary contribution of the paper. Interestingly, the resuls are also never significantly worse than either alternative (i.e., full exclusion and full inclusion) w.r.t raw log-loss tests. So, why should exclusion be preferred over modeling and adjusting for self-report advantage? As of now, it is not very clear.

I would hence like to retain my original score. However, I think the soundness has improved in the revised version.

Thank you for the follow-up, and for noting that the soundness has improved. This round adds two experiments and revises three passages; new text is in blue.

One common scale across different tasks. You are right that split-half and cycle analyses show transfer, not that a single scale is justified. We tested that question directly (App. H, pp. 29–30, Table 10, Figure 6). We gave the model a separate strength for every (model, benchmark) pair, shrunk towards the common strength, and compared it with the one-dimensional board on held-out papers. The shrinkage was tuned on the test rows, which favours the richer model.

Benchmark-specific strengths do exist. Where other papers allow estimating them, they improve prediction on Coding and Agentic, and slightly on chemistry; on the other boards they make no significant difference. The literature, however, rarely provides what they need. On the general boards, 78–94% of a held-out paper's comparisons are on benchmarks that no other paper uses, and a per-task ranking cannot say anything about them. The common scale predicts these comparisons above chance on 13 of 14 boards, within 2.2 points of its overall held-out accuracy on the general boards and 4.2 on the scientific ones. Of the few comparisons a per-task model could use, 56–100% repeat a comparison already in the training papers.

The score therefore does not assume that ability is one-dimensional. It is the scale that lets results on different tasks inform each other, and the paper reports what it loses. Epoch's ECI takes the same step with an item-response model. Where our data do not support a shared scale (protein and chemistry, split-half ρ = 0.31 and 0.23), the paper says so.

Framework versus reused evaluations. Refitted on the literature alone, without any leaderboard export, the boards still agree with Arena at ρ = 0.74–0.89 (0.61 on Instruction Following), compared with 0.78–0.93 with exports (Table 9, "lit."). The six scientific boards contain no exports, and every one of them predicts held-out papers above chance. External validation is strongest where evaluation infrastructure exists because only there is a referee available. The agreement does not come from the exports.

Prior work. We agree that aggregating reported scores and generating leaderboards automatically are established problems. §2 now cites SciLead (Şahinuç et al., 2024), the ALG position paper (Timmer et al., 2025), Artificial Analysis, BenchLM and Epoch's ECI (Ho et al., 2025). ALG systems produce one table of absolute scores per (task, dataset, metric), and index builders combine scores over a curated, shared set of benchmarks. LiteratureRank compares only numbers measured in the same table under the same protocol. This lets hundreds of one-off benchmarks, in fields that have no shared set, enter a single board, and every comparison remains traceable to its source. Our contribution is this pipeline, its validation and the released boards, not a new ranking model.

Why exclude rather than model the self-report advantage. The γ-adjusted fit is not worse in log-loss, but that test scores comparisons between third-party models, which the adjustment barely changes. The adjustment matters for the promoted models, and those are exactly the models the test leaves out. We have now tested them (App. G, p. 26, Table 7; §6.2). For each model evaluated both by its developers and by others, we removed its independent comparisons, placed it using its developers' reports alone, and compared the result with its independent position. As a control, we placed the same model from an equal number of independent comparisons.

Unadjusted, the developers' reports put such models too high: on Agentic, 10 of the 11 models whose evidence is almost entirely their own reports, by 10.1 places on average. γ removes this average shift, but not the error for individual models. Even after adjustment, these placements are further from the independent position than the control's. The gap is 1.6 places on Agentic (3.1 [0.7, 5.2] at τ = 20) and 15 places on protein (26 [12, 41] at τ = 20).

This follows from what γ is. It is a single constant, estimated from the models that have both kinds of evidence, and then applied by assumption to models that have none. It cannot correct a bias that varies from paper to paper, and it cannot undo authors' choices of benchmarks, baselines and tuning. It also needs a large overlap to be estimated at all. On remote sensing, where 75 of 164 models enter only through self-reports, γ = 1.05 [−0.74, 4.48], so the correction for half the board would have an unknown size. Exclusion needs no such estimate, and the external references detect no accuracy cost from it. For these reasons the paper uses γ to measure the bias, and the released boards exclude self-reports.

Reviewer_nzuFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for creating domain-specific consolidated leaderboards by compiling model evaluation results scattered across multiple papers, without having to rerun the models. It uses a large language model (LLM) to extract the evaluation results, excludes the results of models proposed in the respective papers, and then aggregates the head-to-head comparisons between models—obtained under identical evaluation conditions—using a weighted Bradley–Terry model. The authors present model-specific scores, confidence intervals, and rank ranges across 14 leaderboards and analyze how the uncertainty of rankings varies depending on the amount of comparative data available for each domain.

Critical strengths

S1. Rather than manually averaging scores from different papers, this approach uses the relative performance of models evaluated under the same conditions. This is a reasonable approach to minimize the issues that arise when aggregating results with different metrics or score ranges.

S2. Results from the author’s proposed model are excluded, but results evaluated in other papers are included. The authors considered the impact that evaluations favorable to their own model might have on the rankings and cited the sources of the extracted results to allow readers to verify the original papers.

S3. Fourteen leaderboards were created, presenting confidence intervals and ranking ranges along with the scores. In particular, it highlights cases where it is difficult to distinguish the relative merits of top models depending on the field, thereby cautioning against accepting the rankings as definitive results.

Critical weaknesses

W1. Unverified Bias Reduction. There are no experiments that directly verify the key claim of reduced bias. It is necessary to compare how much the rankings differ when the model’s own results are included versus excluded, and to determine whether the results obtained by excluding them are closer to external evaluations. Currently, while there are rules in place to reduce bias, it is difficult to know to what extent it has actually been reduced.

W2. Insufficient Validation of Ranking Accuracy. There is insufficient verification that the final rankings accurately reflect actual performance. While the finding that the rankings of common Vision models remain similar even when the data collection period is changed demonstrates stability, it does not serve as evidence that the rankings are correct. It is necessary to compare these rankings with those evaluated directly under common conditions or to conduct experiments predicting the wins and losses of papers not used in the paper.

W3. Lack of Baseline Comparisons and Sensitivity Analysis. There is insufficient comparison to determine how necessary the proposed design is. While the appendix shows examples where filtering prevents extreme score estimates, it does not compare the approach to simple win rates or the unweighted Bradley–Terry method. The minimum comparison count of 50 should also be varied to determine how the excluded models and the final rankings are affected by this criterion.

W4. Unverified LLM Extraction Accuracy. The accuracy of results extracted using LLMs is not measured. Even if model names are standardized according to rules after extraction, there remains a possibility that scores or evaluation conditions could be misread, or that different models could be merged into one.

W5. Limited Comparability Across Papers. Further explanation is needed regarding the extent to which differences in evaluation across papers can be resolved through a consolidated ranking. Even when comparisons are made under identical conditions within a single paper, differences in evaluation conditions can become an issue again when combining these results with the win-loss records of other papers.

Questions for authors

Q1. How does the ranking change when the results of a specific model are included or excluded? Is there a criterion for determining which approach has less bias?

Q2. Are there any comparative results showing that the proposed method outperforms simple win rates or the unweighted Bradley–Terry method?

Q3. If the minimum number of comparisons (50) is changed, how much do the number of remaining models and the top rankings change?

Q4. Compared to human-reviewed results, how accurate are the extraction and model name integration? Do the rankings hold up even after errors are corrected?

3 comments

Thank you for the review. You were right on two counts: the submission showed that the rankings are stable without showing that they are correct, and it never tested the self-report rule. Most of the revision answers these two points. All boards have been regenerated with the corrections described below, so several numbers have changed. New or changed text is marked in blue, and page numbers refer to the revised PDF.

W1/Q1 (with vs. without self-reports). We re-extracted four boards (Agentic, Math, protein, remote sensing) with a prompt that keeps each paper's own model and labels it (§6.2, p. 9; App. G).

The bias shows up clearly. On the agentic literature, a paper's own model wins as if it were about 175 Elo stronger than when other authors evaluate it (γ = 1.01, 95% CI [0.62, 1.51]). If these results were kept:

  • three models in the agentic top ten would change;
  • seven models would reach the board only on their developers' own reports;
  • on remote sensing, 75 of 164 models would be ranked only through such comparisons.

Math and protein move less (Table 5).

For a criterion of which version is less biased, we compared both against references that score the models themselves: Arena, Epoch AI and ProteinGym (Table 6, p. 26). None of them separates the two versions, and they lean in different directions. Held-out papers do not separate them either (App. G, p. 25).

We do not think this argues against exclusion. These references cover well-known models that many papers already evaluate, while self-reports matter most for models that nobody else has tested, and those cannot be checked at all. Other public leaderboards, such as BenchLM, are themselves built from developer-reported numbers. We therefore present exclusion as a conservative guard against a measured bias, not as a proven gain in accuracy. Where outside references exist, our boards agree with them well. Outside protein, the scientific fields have none, which is the gap these boards are meant to fill.

W2 (direct evaluations and unseen papers). Section 6.1 (p. 8) is new and tests the boards against evidence they were not fitted on.

Chatbot Arena rates every model under the same voting protocol. It agrees with our six text boards at ρ = 0.78–0.93 and with the two vision boards at 0.79 and 0.85, and the agreement holds when the boards are refit on the literature alone. Epoch AI's own math runs agree with the Math board at 0.90. ProteinGym agrees with the protein board at 0.82, although partly circularly (App. H).

The boards also predict papers they have not seen. Fitted without a paper, a board predicts the winners in that paper above chance on all 14 boards. Fitted only on earlier papers, it predicts later ones on 11 of 14; Table 8 lists the three exceptions.

W3/Q3 (the threshold of 50). App. I reruns every board at thresholds from 10 to 200. The general boards hardly change between 20 and 100 (τ ≥ 0.94). The heads of the scientific boards do change, mostly because single-paper models drop in or out. This is why the boards are now shown in two tiers, with only models supported by at least two independent sources in the main ranking (§5).

Q2 (win rate, unweighted BT). Weighted BT matches Arena clearly better than the raw win rate (+0.074, p = 0.001) and about as well as unweighted BT. On held-out papers it beats the win rate on 13 of 14 boards (App. H). We now describe the weights as a choice of estimand rather than an accuracy gain.

W4/Q4 (extraction accuracy). We audited 432 sampled comparisons. Each was first checked against the PDF by an LLM and then verified by the authors (App. J, Table 10). 94.2% were fully correct in model identity, name merge, scores, metric and direction, and 98.8% of the scores themselves were correct.

The errors we found are fixed: a few bad name merges, leaked self-variants, standard deviations read as scores, and four documents that mix conditions. To see whether the rankings hold after these fixes, we rebuilt every board with all corrections turned off. Against the corrected boards, τ is at least 0.97 on 13 boards and 0.90 on chemistry. The leader changes only on Multilingual, where the top two are tied within error.

W5 (comparability across papers). We added a cross-paper node-split, which asks whether papers that never compare a pair imply the same ordering as the papers that do (App. H, p. 27). On the general boards they agree about as often as noise allows. Protein, remote sensing and agriculture fall short, and the paper says so. Copied numbers are rare: verbatim duplicates across papers are at most 2% on 11 boards, and removing them changes no top ten.

We would like to update you on two points from your review, because the other reviewers raised them again in the discussion and we have run new experiments on both. New text in the revised PDF is in blue.

W1/Q1, a criterion for the self-report rule. In our first reply we said that Arena, Epoch AI and ProteinGym do not separate the boards with and without self-reports. The reason is that they cover well-known models, while the models that only self-reports would admit to a board are evaluated by nobody else. We have now tested those models directly by simulation (§6.2; App. G, p. 26, Table 7).

For each model evaluated both by its developers and by other authors, we removed all of its independent comparisons, so that it reaches the board only through its developers' reports. We then compared its position with the one its independent evidence gives. As a control, the same model was also placed from an equal number of independent comparisons.

On the agentic literature at the paper's threshold, 10 of the 11 models whose evidence is almost entirely their own reports are ranked too high, by 10.1 places on average. The board would predict them to beat their independent opponents 14 points more often than they do. Placed by self-reports, a model ends up further from its independent position than when placed by the same amount of independent evidence. The gap is 2.4 places on Agentic (2.9 [0.6, 5.1] at τ = 20) and 21 places on the unrestricted protein board ([3, 38]). Adjusting for the measured home advantage γ removes the average shift but not this per-model error (1.6 places on Agentic, 15 on protein; significant at τ = 20).

This gives the criterion you asked for, applied where the two rules actually differ: the models that exclusion keeps out are the ones self-reports place worst. For the models that both rules rank, the external references show no accuracy cost (Table 6).

W5, comparability across papers. We also tested whether results on different benchmarks support one strength per model (App. H, pp. 29–30, Table 10, Figure 6). We compared the board with a model that adds a separate strength for every (model, benchmark) pair. On the general boards, 78–94% of a held-out paper's comparisons are on benchmarks that no other paper uses. The common scale predicts these above chance on 13 of 14 boards, within 2.2 points of its overall held-out accuracy on the general boards and 4.2 on the scientific ones. Benchmark-specific strengths improve prediction significantly only on Coding and Agentic (marginally on chemistry), and only on the few comparisons where they can be estimated.

We hope these additions address your remaining concerns, and we are glad to answer any further questions.

Thank you for the additional experiments and your response. The concerns regarding the reason for excluding self-model evaluation and the accuracy of the combined rankings have been largely addressed by the additional results.

However, the benefits of using weights are still unclear. If the performance is similar to that of the unweighted Bradley–Terry method, it would be helpful to provide further explanation of the situations in which weights are necessary. Please demonstrate, through experiments or theoretical justification, what benefits are gained when considering the added complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.