- Rating
- 2(+1)
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of understanding what agent evaluations are truly telling us given the choices of model, scaffold, tasks, and benchmark used. Inspired by generalizability theory, they decompose the variance in agent performance into variances from the model, scaffold, and tasks used, as well as all the pairwise interaction terms, and use these variances to calculate the signal-to-noise ratio of the benchmark for its target measurement of choice; for instance, if the benchmark is measuring specific model-scaffold combinations, then the model variance, scaffold variance, and model-scaffold variance are all part of the signal, and everything else is noise. They fit a Bayesian Bernoulli-Logit fixed model across evaluation data from over 9 agentic benchmarks, but only spanning about 13-17 distinct scaffolds. They validate their model by comparing their Bayesian bernoulli-logit model’s predicted rankings of models to held-out benchmarks’ model rankings, and show that their method beats raw averaging over the model’s performance, albeit with small sample size. Their main claims, though, are about how evaluations should be done and presented: mainly that there is a lot of variance in agent performance due to scaffolds, so benchmarks should say whether they are trying to evaluate entire identical systems or individual models, and for the latter, they should use multiple distinct scaffolds.
Critical strengths
The paper reads smoothly, and the AI revisions made aren’t noticeable. It has a clear narrative, the contributions are clear, the methods are understandable, and there’s a very detailed appendix. Kudos to the authors for writing 100% of their code by hand too.
Critical weaknesses
- The paper does not have any new insights. It's well-known, and maybe even obvious, that using a different scaffold can change the tasks that an agent is able to solve. Epoch AI’s Capabilites Index https://epoch.ai/eci already aggregates across benchmarks when evaluating models independent of scaffolds. The paper would have to show (it does not currently do this) that there is a systemic pattern of benchmarks incorrectly claiming that they measure model ability when they only use a single fixed scaffold.
- The paper has a very low sample size on its number of scaffolds, and in some cases the number of models. For instance, for their external validation of their model in Table 3, most of the benchmarks only have 6 or 7 distinct LLMs, which results in massive confidence intervals containing 0.
Questions for authors
- Are there many agent benchmarks which claim to evaluate models instead of agents, and use only a single fixed scaffold?
- Do you have any experiments showing that model ranking is improved by averaging over many scaffolds, in the same held-out benchmark rank-prediction sense as the external validation?
10 comments
(part 2/2)
Aggregation to summarize vs aggregation to stabilize relevant information
Benchmark score aggregation produces a summary score. This is different from the aggregations in our analyses which are being used to identify which additional rollouts are worth buying to help a score support the intended decision. Within a fixed budget, we can compare alternative evaluation designs using their projected reliability and cost, rather than choosing suite size by convention or vibes.
When we quantify (e.g. Fig 3) how increasing task diversity for X rollouts can improve decision-relevant signal more than a budget using 5X more rollouts (§5.4), we illustrate a particular eval configuration set to measure stable, cross-context tool-use capabilities of LLMs.
Uncertainty and what external validation represents
Much of the work calls out the large decision-relevant uncertainty found in many evaluation designs. As you noted, some estimates like in Table 3 have intervals containing 0. Importantly, the method we are showcasing, 𝜃, does not cross 0 in Tab 3 (while conventional aggregation does). We think that is a meaningful distinction.
We agree that the external samples are small. We debated whether to even include Table 3, since it is immaterial to the contributions, worrying that small sample sizes would distract our main message. Since it isn’t needed, we can re/move it to improve clarity, and would appreciate your thoughts. At present, it shows that controlling for noise (shown in 3.2.2) captures a transferable signal. However, this cool result is only a byproduct of our methods which are specifically designed to improve evaluation design (a different objective from performance prediction on unseen benchmarks).
Scaffolds in current practice
Averaging scaffolds & ranking advantage
Thanks for this question. Averaging over many scaffolds will meaningfully improve rank reliability when scaffold interaction effects are a significant source of the measurement error (our tools can quantify that). If certain scaffolds change performance in certain LLMs, averaging over them will always reduce their individual noise contributions proportionally (Eq 3, 7), improving cross-scaffold rank reliability. This is different from a universal guarantee that the amount of noise reduction will be worth the cost (our tools help estimate whether that’s true).
Fixed-scaffold comparisons
You raise an interesting case with fixed-scaffolds. We distinguish reliably ranking fixed model–scaffold systems from ranking across scaffolds. We don’t claim that fixed-scaffold evaluations are defective or that scaffolds inherently undermine rankings (Eq 4). A scaffold that improves all LLMs can change scores a lot while preserving their ordering. Model–scaffold interactions, by contrast, can change relative model advantages. Our work separates these effects and estimates their value for decision-making.
We interpret the question/comments about fixed-scaffold agent benchmarks measuring models as arising because of our weak presentation of the work. In case we misunderstood the intent of the question and to be helpful, some examples of benchmarks that rank LLMs using a single fixed-scaffold from the Epoch AI ECI link provided include FrontierSWE, MirrorCode, APEX-Agents, Furniture Assembly, Vending Bench 2, etc.
We hope we’ve clarified that the study is a tool for designing and interpreting evaluations, not another warning that benchmarks are imperfect. For fellow researchers who are designing agentic evaluations, we hope our study enables quantifying how well conclusions are evidence-supported and which additional evaluations would strengthen those claims.
Thanks again for your feedback! We’d love to learn whether this was clearer than the paper, and whether you have additional questions or feedback.
References
[1] Kapoor et al, 2024, https://arxiv.org/abs/2407.01502
[2] Rabanser et al, 2026, https://arxiv.org/abs/2602.16666
[3] Shi et al, 2026, https://arxiv.org/abs/2609.04298
[4] Yehudai, et al, 2026, https://arxiv.org/abs/2503.16416
[5] Zhang et al., 2026, https://arxiv.org/abs/2605.23950
Thank you to the authors for the response. If I understand correctly, the main idea of the rebuttal is to frame the paper as a tool for figuring out when an evaluation is reliable for the decision to be made (about if the underlying LLM is good with many scaffolds, or if the LLM + scaffold combination is good, or if the LLM is good with one particular scaffold).
The problem still seems to me that the authors do not provide a good empirical validation grounded in real benchmark data that demonstrates that their tool is capable of helping to better make these decisions. The theoretical model makes many assumptions, like scaffolds being drawn at random from a population of scaffolds (instead of being hand-created for specific benchmarks), missing LLM-scaffold combinations not biasing the data, random effects being Gaussian, etc. My point is not that these assumptions are necessarily wrong, just that it is not obvious that the tool's assumptions hold in real data in general, and an empirical validation would inspire more confidence in benchmark designers that the conclusions of the tools are reliable.
Unless the authors can provide quantitative evidence from real benchmark runs that their theoretical model effectively determines evaluation reliability and fixes incorrect claims made in benchmarks due to running an LLM on a single fixed scaffold, I will maintain my score.
The rebuttal does satisfactorily answer my question 2, with listing out FrontierSWE, MirrorCode, APEX-Agents, Furniture Assembly, Vending Bench 2. FrontierSWE and APEX-Agents still say that they are evaluating "agents", so it's not clear that they're making claims about specific LLMs being better with many different scaffolds, but MirrorCode, Furniture Assembly, and Vending-Bench 2 do make claims about AI model performance.
(part 1/2)
Thank you for your helpful response!
Insights into Scaffolds, Aggregations, and Practice
We’re grateful for the opportunity to clarify our scope and claims. We very much agree with you: “changing the scaffold changes LLM performance” is not a meaningful insight. We also strongly agree that aggregating benchmarks in leaderboards isn’t novel.
Thankfully, those findings are not our primary contributions or claims. Those findings act as confirmation and serve to illustrate our contributions: tools that can measurably improve agentic evaluation. We believe that shortcomings in our presentation likely obscured them.
We’d love your feedback on the order, clarity, and coherence of our contributed tools, in contrast to the one we had chosen in the paper.
Context: Problems shared by the field
Since our primary audience are those who design and use agent evaluations, we set out to address unsolved problems and needs identified by colleagues. They include [as named in future work / limitations of references below]:
The design of our study is meant to help solve these challenges by offering tools that quantify the reliability of decision-relevant information produced in agent evals and that show how to preserve that information with less cost in researchers’ own real-world evaluation design challenges.
Corresponding Methodological Contributions
To help move the field forward, our methods need to answer how evals can deliver more relevant evidence for less cost, when more tasks are valuable for a decision, when scaffold coverage limits intended claims, when a LLM–harness system comparison is already sufficient or should be deconfounded, etc. Our study introduces:
Core theme: designing evaluations for decision-making
A key idea we emphasize is that evaluation designs should change to preserve decision-relevant information. Examples of how decisions would change the measurements:
Our tools explain when evaluators have sufficient information or when a claim requires better evidence: more evaluations can make a score more precise without making the intended claim about model capability more reliable.
Cost-savings without sacrificing the good stuff
We quantify multiple routes to building cheaper evaluation suites that minimize loss of decision-making information. Knowing that scaffolds change scores doesn’t tell us whether the next 1000 rollouts will help. Our decomposition approach (§3.2) can distinguish evaluation problems that require different solutions: more tasks, more scaffolds, broader benchmarks, repeated runs, etc. We separate raw performance differences from ranking error and then derive design-dependent reliability limits (§3.3). The primary estimates and illustrations of the paper quantify these relationships and predict the added gain for varying contexts.