acceptodds
Under review as a conference paper at ICLR 2027

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Abstract

Agent evaluations are increasingly used to compare models, assess capabilities, and inform deployment decisions, yet observed scores can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models or produce reliable absolute scores. We ask which conclusions current agent evaluations reliably support and what additional evaluation would actually improve them. Using Generalizability theory, we develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change scores or rankings. We find four practical results: (1) Reliability depends on the measurement goal. Fixed model–scaffold systems are ranked reliably (–), while underlying-model reliability is substantially lower (–). (2) Scaffold choice can change evaluation conclusions. We introduce inter-scaffold reliability, measuring whether scaffolds preserve model rankings, and show that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability by at most when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from to at the same task budget and can reduce projected cost by up to . Evaluation design should therefore follow the intended claim: practitioners should identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Primary area:
datasets and benchmarks
Keywords:
datasets, benchmark, agent, AI, leaderboard, harness, scaffold, reliability, psychometrics, IRT, Generalizability theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SKHvOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of understanding what agent evaluations are truly telling us given the choices of model, scaffold, tasks, and benchmark used. Inspired by generalizability theory, they decompose the variance in agent performance into variances from the model, scaffold, and tasks used, as well as all the pairwise interaction terms, and use these variances to calculate the signal-to-noise ratio of the benchmark for its target measurement of choice; for instance, if the benchmark is measuring specific model-scaffold combinations, then the model variance, scaffold variance, and model-scaffold variance are all part of the signal, and everything else is noise. They fit a Bayesian Bernoulli-Logit fixed model across evaluation data from over 9 agentic benchmarks, but only spanning about 13-17 distinct scaffolds. They validate their model by comparing their Bayesian bernoulli-logit model’s predicted rankings of models to held-out benchmarks’ model rankings, and show that their method beats raw averaging over the model’s performance, albeit with small sample size. Their main claims, though, are about how evaluations should be done and presented: mainly that there is a lot of variance in agent performance due to scaffolds, so benchmarks should say whether they are trying to evaluate entire identical systems or individual models, and for the latter, they should use multiple distinct scaffolds.

Critical strengths

The paper reads smoothly, and the AI revisions made aren’t noticeable. ​It has a clear narrative, the contributions are clear, the methods are understandable, and there’s a very detailed appendix. Kudos to the authors for writing 100% of their code by hand too.

Critical weaknesses
  1. The paper does not have any new insights. It's well-known, and maybe even obvious, that using a different scaffold can change the tasks that an agent is able to solve. Epoch AI’s Capabilites Index https://epoch.ai/eci already aggregates across benchmarks when evaluating models independent of scaffolds. The paper would have to show (it does not currently do this) that there is a systemic pattern of benchmarks incorrectly claiming that they measure model ability when they only use a single fixed scaffold.
  2. The paper has a very low sample size on its number of scaffolds, and in some cases the number of models. For instance, for their external validation of their model in Table 3, most of the benchmarks only have 6 or 7 distinct LLMs, which results in massive confidence intervals containing 0.
Questions for authors
  1. Are there many agent benchmarks which claim to evaluate models instead of agents, and use only a single fixed scaffold?
  2. Do you have any experiments showing that model ranking is improved by averaging over many scaffolds, in the same held-out benchmark rank-prediction sense as the external validation?
10 comments
AuthorsOpenReview ↗

(part 1/2)

Thank you for your helpful response!

Insights into Scaffolds, Aggregations, and Practice

We’re grateful for the opportunity to clarify our scope and claims. We very much agree with you: “changing the scaffold changes LLM performance” is not a meaningful insight. We also strongly agree that aggregating benchmarks in leaderboards isn’t novel.

Thankfully, those findings are not our primary contributions or claims. Those findings act as confirmation and serve to illustrate our contributions: tools that can measurably improve agentic evaluation. We believe that shortcomings in our presentation likely obscured them.

We’d love your feedback on the order, clarity, and coherence of our contributed tools, in contrast to the one we had chosen in the paper.

Context: Problems shared by the field

Since our primary audience are those who design and use agent evaluations, we set out to address unsolved problems and needs identified by colleagues. They include [as named in future work / limitations of references below]:

  1. agent evaluations are often unreliable measures of capability and methods for quantifying the reliability across evaluation settings are needed [2], [3]
  2. decoupling scaffold effects from the LLM is both challenging and needed [2],[4], [5]
  3. Even rankings of models are harness-confounded, restricting the usefulness of current methods [2], [5]
  4. Cost is a huge limitation, making simple things like CIs difficult to estimate, let alone factorial evaluations typically needed for the above [1],[2],[3],[4],[5]

The design of our study is meant to help solve these challenges by offering tools that quantify the reliability of decision-relevant information produced in agent evals and that show how to preserve that information with less cost in researchers’ own real-world evaluation design challenges.

Corresponding Methodological Contributions

To help move the field forward, our methods need to answer how evals can deliver more relevant evidence for less cost, when more tasks are valuable for a decision, when scaffold coverage limits intended claims, when a LLM–harness system comparison is already sufficient or should be deconfounded, etc. Our study introduces:

  1. a rigorous, generalizable definition of agent evaluation reliability
  2. methods to quantify and decouple the sources of variation in agentic benchmarking
  3. methods to construct rank- and decision-relevant reliability estimands for more complex evaluations with more sources of variance
  4. analyses which can be used to design agentic evaluations: how to predict design changes that will help most and at the lowest cost and flag where increased cost or tasks won’t help.

Core theme: designing evaluations for decision-making

A key idea we emphasize is that evaluation designs should change to preserve decision-relevant information. Examples of how decisions would change the measurements:

  • A company buying a combined model–scaffold product cares about the complete system.
  • A team choosing 1 LLM for several settings cares about performance across its scaffolds.
  • A developer replacing a LLM inside an existing agent needs a comparison on that specific scaffold.

Our tools explain when evaluators have sufficient information or when a claim requires better evidence: more evaluations can make a score more precise without making the intended claim about model capability more reliable.

Cost-savings without sacrificing the good stuff

We quantify multiple routes to building cheaper evaluation suites that minimize loss of decision-making information. Knowing that scaffolds change scores doesn’t tell us whether the next 1000 rollouts will help. Our decomposition approach (§3.2) can distinguish evaluation problems that require different solutions: more tasks, more scaffolds, broader benchmarks, repeated runs, etc. We separate raw performance differences from ranking error and then derive design-dependent reliability limits (§3.3). The primary estimates and illustrations of the paper quantify these relationships and predict the added gain for varying contexts.

AuthorsOpenReview ↗

(part 2/2)

Aggregation to summarize vs aggregation to stabilize relevant information

Benchmark score aggregation produces a summary score. This is different from the aggregations in our analyses which are being used to identify which additional rollouts are worth buying to help a score support the intended decision. Within a fixed budget, we can compare alternative evaluation designs using their projected reliability and cost, rather than choosing suite size by convention or vibes.

When we quantify (e.g. Fig 3) how increasing task diversity for X rollouts can improve decision-relevant signal more than a budget using 5X more rollouts (§5.4), we illustrate a particular eval configuration set to measure stable, cross-context tool-use capabilities of LLMs.

Uncertainty and what external validation represents

Much of the work calls out the large decision-relevant uncertainty found in many evaluation designs. As you noted, some estimates like in Table 3 have intervals containing 0. Importantly, the method we are showcasing, 𝜃, does not cross 0 in Tab 3 (while conventional aggregation does). We think that is a meaningful distinction.

We agree that the external samples are small. We debated whether to even include Table 3, since it is immaterial to the contributions, worrying that small sample sizes would distract our main message. Since it isn’t needed, we can re/move it to improve clarity, and would appreciate your thoughts. At present, it shows that controlling for noise (shown in 3.2.2) captures a transferable signal. However, this cool result is only a byproduct of our methods which are specifically designed to improve evaluation design (a different objective from performance prediction on unseen benchmarks).

Scaffolds in current practice

Averaging scaffolds & ranking advantage

Thanks for this question. Averaging over many scaffolds will meaningfully improve rank reliability when scaffold interaction effects are a significant source of the measurement error (our tools can quantify that). If certain scaffolds change performance in certain LLMs, averaging over them will always reduce their individual noise contributions proportionally (Eq 3, 7), improving cross-scaffold rank reliability. This is different from a universal guarantee that the amount of noise reduction will be worth the cost (our tools help estimate whether that’s true).

Fixed-scaffold comparisons

You raise an interesting case with fixed-scaffolds. We distinguish reliably ranking fixed model–scaffold systems from ranking across scaffolds. We don’t claim that fixed-scaffold evaluations are defective or that scaffolds inherently undermine rankings (Eq 4). A scaffold that improves all LLMs can change scores a lot while preserving their ordering. Model–scaffold interactions, by contrast, can change relative model advantages. Our work separates these effects and estimates their value for decision-making.

We interpret the question/comments about fixed-scaffold agent benchmarks measuring models as arising because of our weak presentation of the work. In case we misunderstood the intent of the question and to be helpful, some examples of benchmarks that rank LLMs using a single fixed-scaffold from the Epoch AI ECI link provided include FrontierSWE, MirrorCode, APEX-Agents, Furniture Assembly, Vending Bench 2, etc.

We hope we’ve clarified that the study is a tool for designing and interpreting evaluations, not another warning that benchmarks are imperfect. For fellow researchers who are designing agentic evaluations, we hope our study enables quantifying how well conclusions are evidence-supported and which additional evaluations would strengthen those claims.

Thanks again for your feedback! We’d love to learn whether this was clearer than the paper, and whether you have additional questions or feedback.

References

[1] Kapoor et al, 2024, https://arxiv.org/abs/2407.01502

[2] Rabanser et al, 2026, https://arxiv.org/abs/2602.16666

[3] Shi et al, 2026, https://arxiv.org/abs/2609.04298

[4] Yehudai, et al, 2026, https://arxiv.org/abs/2503.16416

[5] Zhang et al., 2026, https://arxiv.org/abs/2605.23950

Thank you to the authors for the response. If I understand correctly, the main idea of the rebuttal is to frame the paper as a tool for figuring out when an evaluation is reliable for the decision to be made (about if the underlying LLM is good with many scaffolds, or if the LLM + scaffold combination is good, or if the LLM is good with one particular scaffold).

The problem still seems to me that the authors do not provide a good empirical validation grounded in real benchmark data that demonstrates that their tool is capable of helping to better make these decisions. The theoretical model makes many assumptions, like scaffolds being drawn at random from a population of scaffolds (instead of being hand-created for specific benchmarks), missing LLM-scaffold combinations not biasing the data, random effects being Gaussian, etc. My point is not that these assumptions are necessarily wrong, just that it is not obvious that the tool's assumptions hold in real data in general, and an empirical validation would inspire more confidence in benchmark designers that the conclusions of the tools are reliable.

Unless the authors can provide quantitative evidence from real benchmark runs that their theoretical model effectively determines evaluation reliability and fixes incorrect claims made in benchmarks due to running an LLM on a single fixed scaffold, I will maintain my score.

The rebuttal does satisfactorily answer my question 2, with listing out FrontierSWE, MirrorCode, APEX-Agents, Furniture Assembly, Vending Bench 2. FrontierSWE and APEX-Agents still say that they are evaluating "agents", so it's not clear that they're making claims about specific LLMs being better with many different scaffolds, but MirrorCode, Furniture Assembly, and Vending-Bench 2 do make claims about AI model performance.

Reviewer_wAKLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies Generalizability Theory to study what agent leaderboards can reliably tell us about models and model–scaffold systems. Using a Bayesian variance decomposition, it finds that adding more tasks may offer limited gains when uncertainty comes from scaffold choice or benchmark coverage, and examines how evaluation budgets could be allocated more effectively.

Critical strengths
  • The distinction between ranking underlying models and ranking complete systems is useful and well motivated. It explains why the same leaderboard can support one claim much more reliably than another.

  • The framework goes beyond reporting unstable rankings: it separates sources of variation and identifies which additional evaluations could actually reduce uncertainty.

  • The empirical analysis is substantial, with results from HAL and Harbor, uncertainty estimates, and extensive sensitivity checks. I also appreciate the explicit discussion of what the sparse evaluation design can and cannot identify.

  • The budget allocation analysis gives the work practical value. The subsampling results and improved agreement with external benchmarks provide encouraging evidence that exhaustive evaluation is not always necessary.

Critical weaknesses
  • The quantitative conclusions depend on sparse and selectively observed model–scaffold combinations. With only two or three scaffolds per benchmark, the estimated scaffold interactions and reliability ceilings may be sensitive to which combinations were evaluated. Connectivity alone does not resolve this concern.

  • The cost-saving recommendations remain partly model-based projections. Task subsampling supports retaining information from the existing leaderboard, but does not directly validate the predicted benefit of collecting evaluations across additional scaffolds or benchmarks.

  • The relationship between the latent-scale reliability coefficients and practical ranking errors could be clearer. The appendix provides useful ranking analyses, but does not fully establish how a given coefficient translates into rank reversals or incorrect selection of the best model.

  • The external validation is promising but relatively small: three comparisons involve only six or seven models. Comparisons against stronger aggregation baselines would also help establish how much improvement comes specifically from the proposed decomposition.

Questions for authors
  1. How sensitive are the estimated reliability ceilings to the variance priors and selective model–scaffold coverage? Can a controlled recovery experiment using the observed missingness pattern help quantify this?

  2. Can the allocation recommendations be tested by fitting on one subset of evaluation conditions and checking the predicted gains on held-out conditions at a matched budget?

  3. How closely do the latent reliability coefficients track observed rank reversals or top-model selection errors across independently sampled evaluation conditions?

2 comments
AuthorsOpenReview ↗

(part 1/2)

We sincerely thank you for your very helpful feedback!

Sensitivity Analyses

Prior sensitivity

Your question about prior sensitivity is great. We provide the details for it in Reviewer obhP’s response, where we describe several variance prior sensitivity experiments conducted, report findings for task ceiling sensitivity (Eq 8) and other coefficients. Overall, the estimations are generally robust to variance priors, and detectable changes do not meaningfully change conclusions.

Selective System Coverage

Thanks for this insightful idea. For system coverage effects, we jackknife by removing each scaffold-bridging model's second scaffold and refitting, and refit again by removing half of the bridges and refitting.

The results confirm our claims (which we will make clearer) that more within-benchmark bridges create more stable ceiling estimates: CORE-Bench (most bridges) changes are < 0.03 under jackknifing. For most benchmarks, removing half of the bridges moves estimates by only 0.06–0.15. But not all: we will explicitly label the benchmarks with < 3 bridges, SWE-bench and USACO, as weakly identified for ceiling & interscaffold reliability under selective coverage; their estimates are sensitive to prior beliefs about σMA\sigma_{MA} between bridge models and can’t be recovered when their bridge is removed.

Pattern Missingness Recoverability

Thank you for this suggestion. We simulated from fitted estimates, then applied 4 coverage patterns and refit with our estimator:

  • exact HAL coverage pattern
  • random scaffold assignment (MCAR)
  • selective assignment: models mainly use scaffolds that suit them (MNAR)
  • full crossing

The same conclusions emerge: well-bridged benchmarks are recovered under every pattern (e.g., CORE: truth 0.90, est. 0.84–0.90; τ-bench: truth 0.65, est. 0.59–0.67). Selective coverage adds little bias beyond sparsity, highlighting that what matters in missingness is how many models bridge scaffolds, not which bridges were chosen.

As before with SWE-bench, the observed pattern returns estimates that are roughly 0.78 whether the truth is 0.50 or 1.0. Our reported credible intervals include the truth, so the uncertainty we reported is honest. We will clearly describe SWE’s weak identification under pattern missingness, adding these analyses to the study. These findings give a clearer understanding of the wide HDIs we report.

Allocation recommendation tests

We love this experiment suggestion. For each benchmark, a fit that never saw it (similar to Apx. E.4 on a reduced sample) predicted how much a fixed budget (in tasks or in dollars) would gain over more of the other benchmarks. We then measured the gain on held-out tasks.

Task diversity improved agreement with the unseen benchmark under various budgets tested. For example, 32 tasks spread over 8 benchmarks rather than 1 gives a predicted +0.07 τ and an observed +0.09. However, small task budgets (<20 tasks) and weak held-out benchmarks have lower reliability baselines, and the fit can overstate the gains with unseen benchmarks: Eq. 6 treats benchmarks as exchangeable draws, so it can’t anticipate that some held-out benchmarks share much less with other benchmarks. Between now and camera-ready, we will be expanding these analyses to provide added clearer insights. (These results are currently based on subsets of the data using only highly connected models and with specific held out sets, and we are setting up a more careful analysis that can more appropriately represent the estimates in the paper.) We believe this is a great question and want to get it right.

Until any changes in those analyses, we will present further D-study projections as upper-end expected gains and report the sampled held-out test with any calibration shortfall, emphasizing our current cost-saving figures (“up to 83%”) as full model-based.

AuthorsOpenReview ↗

(part 2/2)

Rank Reversals and Top Model Identification

Thank you for this feedback: it improves our ability to communicate about decision-relevant findings. 𝜃 values track reversals and top-1 agreement just as would expected for a given benchmark’s reliability. We will add (1) a reliability translation (theoretical simulation), (2) empirical mapping measurements from HAL, and (3) replication analysis on held out tasks. The findings are consistent and illustrate how reliability is related to rank-based decisions, adding to our core findings.

(1) Reliability-rank map, simulation (this table represents populations of 20 models, an expanded table will be added to the paper):

Ep2P(pair reversed)P(top-1 wrong)
.15.37.84
.44.27.69
.75.17.47
.90.10.31
.99.03.10

At E2 0.75, about 1/6 pairs are ordered incorrectly; choosing the best model worsens further with lower reliability.

(2) These trends are matched empirically (Eq 3 shown). The rank-based framing restates & strengthens our reliability findings in decision-relevant terms. Empirical reliability-rank map:

Benchn_ModelsEq. 3 ReliabilityP(pair wrong)P(top-1 wrong)
CORE34.85.13.40
SciCode17.82.14.40
SciAgent19.61.21.57
tau-bench23.57.23.62
SWE24.52.24.65
USACO13.29.32.72
GAIA21.27.32.78
Assist18.20.35.80
Mind2Web13.14.38.80

(3) Independent re-evaluations: we ran 93 replicate pairs on held-out tasks, a scaffold, or both. Our 90% posterior-predictive intervals contained the observed Kendall τ 90–91% of the time, the reversal rate 90–96%, and the regret 93–99%. For new tasks the predictions are unbiased. However, for unseen scaffolds, they are slightly conservative (predict more scaffold reranking than observed) which is expected given the coverage patterns. We will add a user-oriented appendix to help make decision-scale translations.

Thanks again! Your thoughts have already improved our study, and we will add the analyses, additional evidence, qualifications, and tools to the paper and code to the public repo. We’re happy to engage further.

Reviewer_obhPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses Generalizability Theory to argue that you can't say whether an agent benchmark is reliable until you say what it is meant to rank. The options are fixed model–scaffold systems, the underlying models, or models across tasks. The authors fit Bayesian crossed random-effects models to about 30k HAL rollouts, plus a subset of the Harbor Index. They report that system rankings are highly reliable (Eρ² ≈ 0.94–0.99) and model rankings much less so (0.15–0.84). With scaffold coverage held fixed, more tasks raise model-ranking reliability by at most about 0.10. Spreading a task budget across benchmarks could cut evaluation cost by up to 83%.

Critical strengths

1.The model-vs-system distinction is the right framing, and agent evaluation needs it. Online-Mind2Web makes the point well: 0.974 as a system ranking, 0.148 as a model ranking. 2.The modeling choices are sound. Outcomes are modeled as binary on the logit scale, and reliability is computed for each posterior draw instead of from point estimates. 3.The identifiability appendix (App. F) and the full leave-one-benchmark-out refits is carefully investigated.

Critical weaknesses

W1.The scaffold claim rests on very little data. The "more tasks won't fix it" ceiling depends on only 2–3 hand-picked scaffolds (agent harnesses) per benchmark (based on table 2). W2:The 54 "models" count reasoning-effort settings as separate models.Since σ²_M is the numerator of every G-coefficient, the authors should treat reasoning effort as its own facet or nest it within model.

Questions for authors
  1. How do the main G-coefficients change if you treat reasoning effort as a separate facet?
  2. How sensitive is the task-only ceiling (Eq. 8, Table 15) to the prior on σ_A and σ_MA? Could you report it under at least two alternative half-t scales? 3.With n_a = 2 and scaffolds chosen as generalist-vs-specialist, what population of scaffolds do you think ρ_AA′ generalizes to? Would you accept reframing it as a "generalist–specialist agreement" coefficient?
2 comments
AuthorsOpenReview ↗

(part 1/2)

Thank you for your insightful feedback. It has already strengthened our work!

Prior sensitivity

Thanks for your question re priors: the following experiments enhance our findings and will be added. We refit benchmarks with 6 half-priors: original t₃(0, 2.5), t₃(0,1), t₃(0,5), t₇(0,1.5), N(0, 2.5) and t1t_1/Cauchy(0, 2.5). The table below asks whether the most extreme spread of posterior medians (for Eq 8) across priors is larger than Monte Carlo error alone would produce.

Benchmedian range across priorsCombined MCSErange width / MCSE
scicode.84–.86.011.6
GAIA.27–.30.031.2
tau.65–.68.021.6
SWE.51–.58.051.6
USACO.29–.34.041.4
AssistB.21–.25.013.5
CORE.90–.91.011.6
mind2web.13–.17.022.1
sciagent.64–.67.031.5

Ceilings show little change across priors (.01–.07). For 8/9 benchmarks that spread is indistinguishable from Monte Carlo error: width / MCSE <3. AssistB’s detectable effect is small: .16 posterior SDs, practically not affecting interpretation or conclusions.

Other Estimands

Other coefficients are also stable. No model- or inter-scaffold-reliability min-max contrast across priors (Eq 3,5) exceeds MC error. Similarly, system reliability (Eq 4) changes are <.025 in all benchmarks except AssistB (.9-.97), again not affecting conclusions. Estimated variance components (Eq 2) show a little more change with the prior, but as scale priors move them together, their ratios (our estimands) don’t change. Pooled-model estimates are robust to priors. Across priors, changes to any estimates are all ≤ .025. We have more sensitivity tests in our response to wAKL.

ρ_AA′ generalization

Thanks for the insightful question! For 5/9, benchmarks, “generalist–[benchmark-specific]” reliability is exactly our interpretation; we’ll name that in the text. For the others (>2 scaffolds or no benchmark-specific scaffold), ρ_AA′s interpretation would be reliability among HAL’s “benchmark-compatible” scaffolds (this interpretation is a reason why we don’t report it for the pooled model, and why we do not discuss scaffolds as object of measurement). While our estimation of ρ_AA′ works for any design (like interrater metrics), its interpretation would generalize based on observed populations. We will make this more explicit in the text.

AuthorsOpenReview ↗

(part 2/2)

Reasoning as a Facet

We love your observations about reasoning nesting in models (Reasoning-nested-Facet-Design, “RFD”). RFD can provide a rich understanding of LLMs and can answer more evaluative questions. We had dropped these analyses to better focus on what we saw as more common reporting practices (separate benchmark scores for each setting) and to decrease interpretation complexity within the data’s constraints, so we’re grateful and agree it is more conceptually meaty. It’s an example of other useful facets that could be included (e.g., provider, generation), based on the intended decision.

In column order, the median posterior G coefficients using all tasks in each benchmark: original Eq 3’s Model-Reason as object of measurement (OOM), RFD’s LLM as OOM [M] tested across R={1, 3} reasoning setting(s), respectively, and RFD’s ModelxReason [MR] as OOM:

BenchEq 3G[M]{R=1}G[M]{R=3}G[MR]
scicode.82.61.72.77
GAIA.27.15.22.44
𝛕-bench.57.45.62.59
SWE.50.27.30.35
usaco.31.10.19.45
assist.19.12.17.28
CORE.85.81.90.86
mind2web.14.16.17.22
sciagent.61.32.39.49

Comparing the various coefficients across/between Eqs (3,5,8) and related RFD counterparts, benchmark estimates stay within relative grouping orders {CORE, SciCode} > {SciAgent, tau} > {SWE, USACO, GAIA} > {Assist, Mind2Web}. No increases in related coefficients move benchmarks with reliabilities <0.75 to >0.75. However, we note that cross design estimate differences also reflect a change in what is “signal” (not that one design is better).

RFD improves estimation of the poorly connected benches: for USACO & SWE, extra partial pooling reduces their wide HDIs by ~20% for Eq 3->G[MR]. USACO is the one benchmark where reasoning variation is much higher than stable LLM variation (even more true at the LLM-task-level). For SWE, RFD adds more evidence that the model-harness interaction is still the bottleneck. It’s the opposite of 𝛕-bench, which benefits from more reasoning runs: R=1->3, G=.45->.62 (note that the change from R=1->3 implies a three-fold cost increase). A key headline is bolstered: many tasks do not make up for missed sampling of other influential facets—task ceilings are .16 lower on average for OOM = LLM than for LLMxReason.

We’ll add a helpful appendix and would love your input on whether RFD should appear in the body. Some drawbacks: RFD can further increase generalization uncertainty or sensitivity to priors (which we’ll test). In HAL data in particular, reasoning configurations are observed for only subsets of LLMs on a subset of possible settings, so separating their effects depends a lot on the hierarchical assumptions of observed overlap. While partial pooling helps estimate missing cells, it can’t create the information we’d gain from actually observing those cells. If certain LLMs are preferentially evaluated at different configurations (as here), the interpretation of reasoning variation may not generalize helpfully to the question intended to be answered. As reasoning levels vary in presence and meaning across LLMs (e.g., only GPT-5 uses “minimal”), unnesting reasoning from models can exacerbate problems with interpretation and usefulness. RFD’s conceptual strength is checked by the practical limitations of the data. We’d value your input.

Thank you again. We’ll add these extra analyses, evidence, and guidance to the paper and code to the repo. We are happy to engage further!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.