acceptodds
Under review as a conference paper at ICLR 2027

Same Results, Different Rankings: Measuring Evaluation-View Dependence in LLM Agent Benchmarks

Abstract

A benchmark can select different winners without changing any saved result. In DeepSWE, two configurations exchange first and twelfth place when the evaluation changes from single-attempt success to four-attempt task coverage. Such changes raise a measurement question: how much of a leaderboard depends on the evaluation view, and where does that dependence occur? We introduce the Structured Leaderboard Ambiguity Index (SLAI), which combines changes in model-pair relations with their ranking position and relative rank-gap amplitude across a declared family of views. Each contribution retains evidence from an actual pair of views. ParetoSweep computes the index exactly and returns these witnesses. Two theoretical controls establish information that selected conventional summaries omit: identical ranking summaries can accompany an SLAI ratio of \(n-2\), and duplicating an existing view leaves the entire ambiguity surface unchanged. Across ten saved evaluation panels, the full family can reveal substantially more weighted change than any fixed pair: BFCL yields 27.93% versus 9.36%. Four additional source panels retain both low-change cases and further family-coverage gaps. SLAI describes where view-dependent comparisons occur and preserves the evaluation conditions needed to inspect them.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, leaderboard sensitivity, evaluation choices, ranking ambiguity, ranking stability, model comparison

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SEH7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses an interesting question: how much does the leaderboard ranking of an AI model depend on the concete evaluation view of the person performing the benchmark experiments. Studying this question is also motivated by a convincing example: switching from a one-attempt to a four-attempt success rate analysis makes the #1 and #12 reverse for DeepSWE, indicating that leaderboards should always be taken with a pinch of salt and that the underlying evaluation view has to be made transparent.

To address this issue, the paper introduces a couple of measures (LAI, SLAI,d_ha) that are more robust under different evaluation views, in essence by relying on a worst case view over all different such views. The paper demonstrates the proposed measures on real-world benchmark suites and theoretically connects them with each other and also with alternative measures known from the related literaturs.

Critical strengths
  • The paper addresses an interesting and timely topic, which is well-motivated in its introduction.

  • The idea of introducing a measure that quantifies the robustness / stability of benchmark results / leaderboards seems promising and convincing.

Critical weaknesses
  • The paper introduces several different measures without giving enough contextual interpretation of those measures. In this way, at least for me, it was hard to identify the benefits of thoe measures over existing ones (and also among the introduced ones).

  • The experiments and results sections (Sections 4 and 5) are very confusing to me. (1) The description of the experimental setup is unclear to me: why the distinction between design panels and additional panels? what different evaluayion views ar concretely considered? I think the paper really should make an effort to explain this setup more carefully and to relate it to the formal model introduced in its previous sections. (2) I find it rather uncommon to mix the experimental results in Sections 5 with the theoretical results (Prop 1, Theos 1 and 2). Would it make sense to clearly separate those parts?

Questions for authors
  • Why not stop with LAI? What's the advantage of using SLAI rather than LAI directly? It would be nice to have this clearly explained in the main text. (In general, some intuition/explanation next to the introduced formulas would be nice).

  • I like the idea of checking leaderboards for robustness towards the evaluation view. Another aspect coming to mind here is the "sample dependence" of leaderboards, i.e., leaderboards changing under (slight) changes in the underlying benchmark suite. Is there any possibilities to obtain statistical guarantees for the proposed measures that apply beyond the concrete suite considered? In the spirit of recent works such as: https://doi.org/10.52202/079017-3115

Reviewer_8Gk9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Structured Leaderboard Ambiguity Index (SLAI) to answer this measurement question: “how much of a leaderboard depends on the evaluation view, and where does that dependence occur?“ by tracking whether the relative order of two models flips between two evaluation views by checking order reversal, tie, and when the view remains unchanged. The authors state that in a benchmark ranking table (leaderboard) rank reversals near the top of the ranking table are more important than those appearing towards the bottom and so based on this, they propose a formula that scores the rank flips appearing relatively on top of the table, higher than those towards the bottom. This decision stems from how ranking tables are used in practice and based on ranking preference modeling (Vigna, 2015; Webber et al., 2010).

The paper cites non-peer-reviewed pre-prints, code snapshots, and website blog posts dated as late as Sep 19, 2026. The information “The first local protocol was written on 19 September 2026 at 01:47:09.886 (UTC+8)“ provides traceable and/or verifiable information about the authors which is in violation of double-blind review. With the exact log temporal information, timezone combined with the zip file uploaded , the author information becomes traceable , violating the double-blind review requirement. Here are the examples in the paper where traceable information is provided:

While the paper addresses an important problem and provides a detailed solution, the theoretical framing still needs a lot of improvements. How do the authors ensure the stability of the proposed metric under varying configurations such as changing the list of models in a ranking table? This mainly includes those models that consistently appear low in the ranking, or minor variants of those models. Lets’ say we add 10 or 20 bad extra bad models to the bottom of the ranking table without changing the top ones. Does SLAI remain stable? Does the current formula handle situation like this?

The paper gives a useful account of how rankings depend on evaluation choices, and the DeepSWE example makes clear why this deserves attention. The analysis of SLAI is also reasonably thorough. What holds me back from a stronger recommendation is the lack of a worked example showing how the metric helps someone choose a model. I would like the authors to address that in the response.

Critical strengths

the paper addresses an important problem and provides a detailed solution.

  1. The DeepSWE example is compelling. A configuration can move from first to twelfth place just by changing the success criterion, without changing any underlying results. This is a good reason to look beyond a single leaderboard. The other benchmark panels show that the issue is not confined to this example.
  2. The paper explains what SLAI measures and backs this up with proofs. I found the results on ranking-position sensitivity and duplicated views useful for understanding how it differs from existing metrics. The independent checks of the computation are also reassuring.
Critical weaknesses

the theoretical framing still needs a lot of improvements. How do the authors ensure the stability of the proposed metric under varying configurations such as changing the list of models in a ranking table? This mainly includes those models that consistently appear low in the ranking, or minor variants of those models. Lets’ say we add 10 or 20 bad extra bad models to the bottom of the ranking table without changing the top ones. Does SLAI remain stable? Does the current formula handle situation like this? My main concern is what a user should do with the SLAI value. The paper shows cases where SLAI picks up differences that other metrics miss, but I wanted to see a case where knowing this would actually change which model someone chooses. Without that example, the practical case for using SLAI is less clear than the mathematical one. The choice of views and weights needs more justification. The sensitivity analysis helps, but it does not explain why the default weighting is appropriate for a particular use. I also had trouble judging what counts as a high SLAI value, or whether a higher value on one benchmark means more than a lower value on another.

Questions for authors

How do the authors ensure the stability of the proposed metric under varying configurations such as changing the list of models in a ranking table? This mainly includes those models that consistently appear low in the ranking, or minor variants of those models. Lets’ say we add 10 or 20 bad extra bad models to the bottom of the ranking table without changing the top ones. Does SLAI remain stable? Does the current formula handle situation like this?

Is there a case in your experiments where SLAI would lead a user to choose a different model than they would using conventional ranking metrics? If so, please walk through that decision.

Can SLAI values be compared across benchmarks with different numbers of models and views? If only some comparisons are valid, what conditions are needed?

How would you choose the views for a new benchmark? In particular, how do you avoid inflating ambiguity by including views that are distinct but not relevant to the intended use?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8Gk9: Appreciation and Clarification on Anonymity

Thank you for your thoughtful review and positive assessment. We sincerely appreciate your recognition of the problem’s importance, the compelling DeepSWE example, our theoretical results on ranking-position sensitivity and duplicate views, and the independent computational checks.

Regarding the anonymity concern, the cited passages record data provenance and analytical chronology rather than author identity. The bibliography entry cites Datacurve’s public third-party DeepSWE v1.1 benchmark, noting its release and snapshot dates to identify the exact public dataset evaluated. Similarly, the local protocol and public-data retrieval timestamps in Appendix B.2, with the UTC+8 offset specifying the time convention of those records, document the chronology of operations on previously public data, establishing that the SLAI definition, default weights, comparison measures, and verification approach were fixed before the additional numerical analyses.

The quoted material contains no author names, affiliations, or personal contact details, and the cited URL points to a public third-party benchmark resource. Together, these records serve to make the evidence and analysis chronology verifiable. We therefore respectfully clarify that the cited examples document research provenance and do not disclose the authors’ identities.

AuthorsOpenReview ↗

Response to Reviewer 8Gk9 (1/2): Configuration Stability and Model-Selection Walkthrough

Thank you again for your positive assessment and constructive technical questions. Following our earlier note on anonymity, this first part addresses configuration-pool stability and worked model-selection decisions; the second part covers cross-benchmark comparability, view selection, and weights.

1. Stability when adding 10–20 low-ranked models or minor variants

As stated explicitly in Appendix A.8 (Dependence on the Configuration Pool) and Section 6, SLAI is not invariant to adding models to the configuration pool nn—just as normalized Kendall (1−τ)/2(1-\tau)/2, Spearman (1−ρ)/2(1-\rho)/2, and LAI all change when N=(n2)N = \binom{n}{2} changes. Three mechanisms in the paper address this directly:

  • Bottom-tier noise is strongly suppressed: By Theorem 1 and Appendix A.6 (Eq. 20), an adjacent swap at (k,k+1)(k, k+1) contributes SLAI(k)=n−kN(n−1)2\text{SLAI}(k) = \frac{n-k}{N(n-1)^2}. Swaps among models at the bottom (k=n−1k=n-1) are downweighted by (n−2)(n-2) relative to swaps near the top (k=2k=2), scaling as Θ(n−4)\Theta(n^{-4}) versus Θ(n−3)\Theta(n^{-3}). Erratic fluctuations among weak bottom models therefore have minimal impact on leading-rank signals.
  • Dilution by stable bottom padding: Adding mbadm_{\text{bad}} consistently worse models leaves top-tier reversal numerators unchanged while growing NN quadratically and (n−1)(n-1) linearly, diluting the global panel average (the same dilution affects LAI=D/N\text{LAI} = D/N and Kendall distance). Appendix A.8 therefore requires reporting nn, the view family, and the support mapping alongside SLAI.
  • Shortlist and pair-level isolation: Rather than relying solely on a diluted all-pool scalar, the framework computes exact pair contributions eije_{ij} and configuration burdens bib_i (Proposition 2, Appendix A.9) and supports evaluating SLAI on a declared restricted complete panel / shortlist (Section 3.1 and Section 6, "making shortlist-specific contributions useful"). Evaluating on the competitive shortlist or collapsing minor variants removes distortion from uninformative bottom models.

2. Worked examples where SLAI changes model selection

SLAI is a diagnostic and witness-retrieval index rather than an accuracy score (Section 6). It changes model selection by alerting users when high aggregate agreement masks a consequential reversal and returning the exact view-pair witnesses:

  • DeepSWE (Section 1, Fig. 1a, Appendix C.1 Tables 9–11, Appendix E.4): Conventional summaries indicate strong overall consistency—mean normalized Kendall agreement is 0.85717, mean Spearman correlation is 0.85641 (Appendix E.4), 546/666 pairs are stable, and global SLAI is 2.19%. A user relying on a single ranking or high correlation would simply pick #1. However, ParetoSweep surfaces a #1 vs. #12 strict reversal between GPT-6 Astra (xhigh) and GPT-5.5 (high) with top-position weight h=1h=1, and Tables 9–10 trace why: Astra wins more total attempts (335/452, concentrated on 74 all-4-success tasks) but covers fewer tasks (91/113), whereas GPT-5.5 solves fewer total attempts (291/452) yet covers more tasks (102/113, including 14 tasks Astra misses). A user needing single-shot reliability chooses Astra (74.12% vs. 64.38%), whereas one allowing 4 retries switches to GPT-5.5 (90.27% vs. 80.53%).
  • TUA-Bench (Section 5.2, Fig. 4b, Appendix C.2 Table 12): Under the headline mean reward view, GPT-5.5 xhigh (60.123%) ranks ahead of Opus 4.8 max (59.700%). SLAI (2.82%) and its witnesses expose that this lead comes solely from partial credit (7.790 vs. 5.866), while Opus 4.8 max leads in full single-attempt successes (53.833% vs. 52.333%) and all-5 reliability (51/120 tasks vs. 38/120 tasks). A user requiring reliable full completion therefore selects Opus 4.8 max over the mean-reward #1. (Appendix D.6 Table 13 similarly walks through how format witnesses F03/F04 reverse Qwen3-30B-A3B-Instruct-2507 and ToolACE-2-8B from 4.5 vs. 33.5 to 15.5 vs. 13 on BFCL.)
AuthorsOpenReview ↗

Response to Reviewer 8Gk9 (2/2): Cross-Benchmark Comparability, View Selection, and Weights

Continuing from Part 1/2, we address cross-benchmark comparability, view selection, and the weighting choice.

3. Cross-benchmark comparability and judging "high" vs. "low" SLAI

  • When comparisons are valid: As emphasized in Section 6 and Appendix A.8, raw SLAI values are not universal benchmark-quality scores across arbitrary nn and ∣Q∣|\mathcal{Q}|. Comparisons are strictly valid (i) on a fixed configuration pool (e.g., full-family SLAI vs. max d_ha in Table 1 / Fig. 5b, or across weight/tie policies in Appendix F), and (ii) across panels only when nn, the view semantics (same-task reinterpretation vs. cross-task coverage), and support mapping are reported alongside the index (Appendix A.8), excluding degenerate small-nn cases like Banking (n=2n=2, SLAI = 100%) and separating cross-task panels (LongBench, BBEH) from retry/format panels. Under these conditions, Table 3 and Appendix F.6 show that the ordering of major panels (BFCL > CAR > AA > APEX > ... > LiveBench) holds across all 9 positive (α,β)(\alpha,\beta) weightings and 25 tie policies.
  • What counts as "high" vs. "low": Since 0≤SLAI≤LAI≤10 \le \text{SLAI} \le \text{LAI} \le 1 (Appendix A.2), with 0 indicating complete weak-order agreement (Appendix A.1), our panels establish clear empirical reference bands: low-change controls sit below ~1.3% (EvalPlus 0.030%, LiveBench 0.033%, CRUXEval 0.69%, MLE 1.27%), localized/moderate panels span 2%–9% (DeepSWE 2.19% to CAR 8.37%), and broad multi-view instability reaches 27.93% (BFCL, 2.98x its strongest pair) or 53%–57% across distinct tasks (LongBench, BBEH). Crucially, Section 6 stresses that even a low global SLAI (DeepSWE's 2.19%) can accompany a #1 vs. #12 flip, so the scalar should always be interpreted alongside LAI, max d_ha, and shortlist witnesses.

4. Choosing views and preventing artificial inflation in a new benchmark

  • View selection: Following multiverse/specification-curve principles (Section 2), Q\mathcal{Q} should include publisher-declared or deployment-relevant evaluation views (e.g., retry rules, full-vs-partial scoring, official complexity splits, or valid format specs) that pass complete-panel comparability checks (Appendix B.2–B.3, where L-Eval was excluded from the main set due to unequal input lengths, and LiveBench equal-task weighting was isolated as an analytical control).
  • Avoiding artificial inflation: Because adding distinct views is monotone (Appendix A.2), out-of-scope views irrelevant to the intended deployment should be omitted from Q\mathcal{Q} (e.g., restricting BFCL to formats used by the target serving stack). Two built-in safeguards further prevent spurious inflation: (i) duplication invariance (Theorem 2 / Appendix A.2) ensures that redundant or weak-order-equivalent views add zero to SLAI (unlike mean pairwise distance), and (ii) witness provenance guarantees that if an irrelevant view raises SLAI, ParetoSweep directly identifies the responsible view pair (q, q') for inspection and removal.

5. Justification of the default weights (α,β)=(1,1)(\alpha, \beta) = (1, 1)

As explained in Section 3.2–3.3, the linear rank proximity hh prioritizes leading-rank comparisons without an arbitrary discrete top-kk cutoff, the directed gap change aa maintains invariance to monotone score transforms across disparate scales, and uniform integration (α,β)=(1,1)(\alpha,\beta)=(1,1) serves as a neutral default rather than an empirically optimal utility model. When a specific use case calls for different emphasis, Appendix F.1 (Eq. 50) provides the parameterized moment family Mα,βM_{\alpha,\beta} (e.g., increasing α\alpha to concentrate on top ranks, or setting β=0\beta=0 for location-only sensitivity), while Table 28 confirms that cross-panel ordering is already stable (ρ≥0.950\rho \ge 0.950 excluding Banking) across all tested positive (α,β)(\alpha,\beta) settings.

We hope these responses fully address your technical questions and clarify how SLAI is interpreted and applied in practice.

Reviewer_HorPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Benchmarks under different evaluation setups can give different model rankings. The paper introduces a metric, the Structured Leaderboard Ambiguity Index (SLAI), to measure how sensitive a benchmark is to different setups. The ParetoSweep algorithm is introduced to calculate it. SLAI can capture some benchmark ranking sensitivities that other classical metrics miss.

Critical strengths
  • The proposed metric SLAI integrates multiple ranking variation information, which a single traditional ranking metric cannot capture.
Critical weaknesses
  • Poor clarity and presentation. Overall, I find the paper hard to follow. To give a sense, let's take a look at a single paragraph in lines 125-127. The paragraph starts with "Let n≥2n \geq2 "; however, the meaning of nn is never defined. Based on the context, I suppose it's the number of models. Then "Scores si(q)s_i(q) ", but what's ii and qq here? Then "Configuration identities stay fixed; task support may vary as declared, as in MLE’s complexity groups"; I unfortunately cannot understand what "configuration identities", "task support," and "MLE's complexity groups" are, as well as "missing cells" and "restricted complete panel" in the following sentence: "Missing cells require an explicitly restricted complete panel". Friction like this is everywhere across the whole reading process.
  • Incremental novelty. The proposed metric is more like a summary of three ranking measures in equation 3. I don't see a strong justification for the integration approach as in equation 4. The complicated calculation only hinders the interpretation of the measure.
  • Limited evidence of benefits. It's unclear to me how the additional information from the proposed measure can be used. To me, the utility lies in showing that some benchmarks are even worse than they already appear to be.
Questions for authors
  • What's the information that can be exposed by SLAI but cannot be shown by a set of traditional measurements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.