- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks when a leaderboard of model–harness configurations with uneven coverage can support a model ranking. It argues that three questions must be answered before ranking:
- Identity: do shared task names refer to the same versioned task and scorer?
- Support: how much of a declared common harness–task target mixture (q, r) was actually observed for each model?
- Identification: under a task-additive logit model, which contrasts are recovered by the observed design?
These map to three reporting actions: certify an ordering when sharp probability-scale missing-outcome bounds (Eq. 6, Manski-type) exclude zero; label a model-assisted extrapolation; or defer, naming the next cell to acquire.
The technical results are as follows. Proposition 1 shows that within-component contrasts in the configuration–task incidence graph are identified, while model main effects are not identified under free model–harness interactions. Proposition 2 states that cross-component comparisons need extra restrictions. Model–task interactions break the logit certificate (Eq. 5).
The paper names a real and under-discussed problem in agent evaluation. Its insistence on separating identity, support and identification is a valuable framing for practitioners who publish leaderboards.
Critical strengths
S1. A useful conceptual separation. The paper makes precise three things that leaderboard discussions routinely conflate: (i) task identity, (ii) target support, and (iii) likelihood identification. The point that "connected graph ≠ common-target support" (Fig. 1, Fig. 8) is well made. Candidate label joins raise identified observed-configuration contrasts from 1,138 to all 1,378 (l. 295–297) while adding zero observations, and target support stays at most 39.89%. This is a clear, reusable lesson.
S2. A principled, actionable reporting rule with a clean guarantee. The certificate reports a sign only when every bounded completion agrees. Because it compares per-model intervals [L_i, U_i], certified pairs form an interval order, which is transitive by construction. The proposed release format (supported sign, point estimate, deferred pairs plus next cell; §9) is concrete and could be adopted by leaderboard maintainers. A support-only gate is clearly worse: ridge with minimum pair support 0.9 reports 30.45% of pairs with 2.96% error, while the certificate reports about 61.65% with zero error (l. 425–427).
S3. A concrete demonstration that the target matters. The DeepSeek V3.2 / GPT-5.2 reversal between published weightings (Table 5) and the measured gain from a target-mass-based next-cell rule (Table 6) make the abstract argument tangible for practitioners.
Critical weaknesses
W1 (major). The technical core is classical, and the paper's own advance over it is thin.
- Identifiability of within-component contrasts in additive two-way designs is the textbook connectedness condition. It goes back to connectedness in incomplete cross-classifications, e.g. Weeks & Williams (1964), and to Ford's (1957) condition for paired-comparison (Bradley–Terry) estimation.
- The bounds are standard Manski missing-outcome bounds.
- The authors acknowledge both (l. 111–113, l. 118–119).
What remains is the mapping of these tools onto three reporting actions, plus the case studies. That can be a contribution, but then the empirical evidence must show that the protocol changes conclusions in ways that matter and outperforms simpler practice. W2 and W3 argue that it does not yet.
W2 (major). On the real sparse leaderboard, the headline result is determined by arithmetic, and the unrestricted certificate is uninformative at realistic sparsity.
- On Terminal-Bench, every model's joint support under the uniform target is below 0.4. If s_m < 1/2, every model's interval contains 1/2, so no pair can ever be certified. This follows from coverage alone, without looking at any outcome. It is not an empirical finding about agents.
- The uniform 30-harness × ~90-task target is, by the authors' own description, a "coverage stress test", not a deployment target.
- On the public panel, certificate coverage is about 0–2.5% at 60% observation, 25.50% at 80%, and about 60% at 90%.
W3. The certificate has no measured accuracy advantage, and the evidence comes from a single small table.
W4. The positioning in related work is incomplete. Closely related work on missing scores and aggregation in benchmarks is not cited:
- Himmi et al., "Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks" (EMNLP Findings 2024);
- Colombo et al., "What are the best systems? New perspectives on NLP benchmarking" (NeurIPS 2022);
- Rofin et al., "Vote'n'Rank" (EACL 2023), which studies how the choice of aggregation rule changes rankings — directly relevant to the weighting reversal.
Questions for authors
1.Restricted bounds. Under a sensitivity parameter, e.g. |ζ_mt| ≤ Γ, bounded harness effects, or benchmark-specific score ranges, how many Terminal-Bench pairs become certifiable as a function of Γ? At what Γ does the first certificate appear?
2.Deployment targets. On Terminal-Bench, what happens under a single-harness or a small prespecified harness-mixture target (e.g. the 2–3 most-used harnesses)? How many pairs are certified or deferred then?
3.Sampling uncertainty. Applying Eq. 7 with valid simultaneous cell intervals, how much certificate coverage remains on the public panel at 90% observation? Is the DeepSeek V3.2 vs. GPT-5.2 gap (0.005) distinguishable from sampling noise under either target?
4.Second panel. Can the masking and next-cell evaluation be repeated on another crossed panel, e.g. HAL, or a Terminal-Bench subset with dense coverage?
5.Next-cell rule vs. stronger baselines. How does it compare with an exact bound-width-greedy rule, a cost-weighted variant (cells differ greatly in evaluation cost), or a set-cover formulation over deferred pairs?
3 comments
Response to Reviewer GeF8 (2/3)
Q2. Which comparisons survive on a single harness or a small mixture?
We selected harnesses by archived trial count before comparing scores. Table 21 weights the selected harnesses equally and the 89 literal tasks in their union uniformly. All 17 models are included, assuming scores are defined on every positive-weight configuration. The table holds observed success rates fixed.
| Deployment target | Certified pairs | Deferred pairs |
|---|---|---|
| Terminus-2 | 21 | 115 |
| Terminus-2 + Forge | 4 | 132 |
| Terminus-2 + Forge + Ruley | 0 | 136 |
All four two-harness orderings use partial coverage. The 21 on Terminus-2 come from seven fully observed models. Extending any row to all 179 literal tasks leaves every pair deferred.
For the underlying success probabilities, the two 95% simultaneous Hoeffding constructions both defer all 136 pairs on the equal mixture. On the complete seven-model Terminus-2 subset, the model-level construction retains 9 of 21 orderings assuming independent bounded trials within each model. The cellwise construction corrects over observed cells, requires independence within each cell, permits dependence across cells, and retains none (Appendix Q.1).
Q3. Is the DeepSeek–GPT gap distinguishable from sampling noise?
The archive leaves the sign of the DeepSeek V3.2–GPT-5.2 difference open under both weightings. Under each of the two count reconstructions, the simultaneous interval for that pair includes zero even on the full table. At 90% observation, Eq. (7) certifies no pair on any of the 200 random masks under either weighting. The roughly 0.005 lead under the four-domain weights and its reversal under equal benchmark weights describe the published scores.
Successful and total session counts reproduce 125 of 150 primary scores. For another 23 SWE-bench cells, primary score times completed sessions and average score times total sessions give the same integer numerator. We analyzed the 125- and 148-cell reconstructions separately using 95% Bonferroni-adjusted exact binomial intervals under within-cell binomial sampling. Masked or unresolved scores range from zero to one. A separate deterministic sensitivity check allowing ±0.25 percentage points per score also puts zero within the difference under both weightings (Appendix Q.3).
Q4. Does the acquisition result replicate on another crossed panel?
We repeated the replay on seven complete panels from the independent HAL release, selecting rectangles from observation patterns alone. Each crosses three model settings with two scaffolds, with reasoning-effort settings kept distinct. Each policy started from the same 90% random masks and revealed ten cells under uniform scaffold–task weights.
| HAL benchmark | Random (%) | Deferred-weight (%) | Width-greedy (%) | Paired gain (pp) |
|---|---|---|---|---|
| OnlineMind2Web | 0.00 | 0.00 | 0.00 | 0.00 ± 0.00 |
| GAIA | 0.00 | 0.00 | 0.00 | 0.00 ± 0.00 |
| ScienceAgentBench | 46.94 | 47.17 | 58.46 | 11.52 ± 17.68 |
| SciCode | 0.00 | 0.00 | 0.00 | 0.00 ± 0.00 |
| τ-bench Airline | 15.48 | 15.48 | 16.94 | 1.46 ± 4.67 |
| CORE-Bench Hard | 4.15 | 4.15 | 4.46 | 0.31 ± 1.91 |
| AssistantBench | 17.12 | 17.12 | 21.98 | 4.85 ± 7.05 |
The last column gives width-greedy minus random in percentage points, as paired mean ± sample SD over 200 masks after averaging eight shared randomized tie orders per mask. Gains use unrounded values. The SD describes variation across masks on fixed archived scores.
ScienceAgentBench shows what the choice of ten cells can buy. Mean certificate coverage rose from 46.94% with random selection to 58.46% with width-greedy, a paired gain of 11.52 percentage points. Width-greedy improved mean coverage on four of the seven panels. The original deferred-weight rule matched random on six. In a separate 90-reveal OnlineMind2Web replay, its apparent advantage under fixed lexical ties disappeared with shared randomized ties. On the 623-cell Terminus-2 block, ten reveals gave 68.48% coverage for width-greedy, 66.74% for deferred-weight and 66.65% for random (Tables 23–24).
Response to Reviewer GeF8 (3/3)
Q5. What changes when the rule optimizes width or accounts for cost?
In the fixed-score replay, each missing score ranges from zero to one and is known once revealed. If model m belongs to d currently deferred pairs, revealing one of its cells of workload weight w narrows each of those d intervals by w. Their total width falls by exactly w × d, whatever score is revealed. Width-greedy chooses the largest such reduction and updates the deferred set after each reveal. We use shared randomized priorities for ties. ATLAS selects items for their Fisher information about latent ability (Li et al., 2026). Our objective measures the remaining uncertainty in specified workload comparisons.
On the four-domain public panel, after ten reveals from the same 200 masks at 90% observation, mean coverage was 98.20% for width-greedy, 97.53% for the original rule and 83.69% for random. This gives a 14.51 percentage point gain over random and a 0.67 point gain over the original rule.
At a retrospective USD 100 budget, dividing width reduction by quoted cost gave 74.95% mean coverage, compared with 72.71% for raw width and 70.19% for random. Accounting for cost therefore gained 2.24 percentage points over raw width and 4.76 over random. The replay treats historical costs as known quotes. For zero quotes we used the median positive cost among initially observed cells on the same benchmark, falling back to the observed global median (Table 23).
For set cover, we would need to know which pairs each candidate cell settles. The hidden score determines that set. A reveal can shrink an interval while leaving zero inside it. Width gives a quantity we can optimize before the reveal, and the replay then counts the pairs actually settled. In a live evaluation, estimating the chosen cell's success probability may require repeated trials and simultaneous bounds as in Eq. (7).
References
Colombo, P., Noiry, N., Irurozki, E., and Clémençon, S. (2022). What are the best Systems? New Perspectives on NLP Benchmarking. Advances in Neural Information Processing Systems, 35.
Rofin, M., Mikhailov, V., Florinsky, M., Kravchenko, A., Shavrina, T., Tutubalina, E., Karabekyan, D., and Artemova, E. (2023). Vote’n’Rank: Revision of Benchmarking with Social Choice Theory. EACL, 670–686.
Himmi, A., Irurozki, E., Noiry, N., Clémençon, S., and Colombo, P. (2024). Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks. Findings of EMNLP, 11759–11785.
Gao, S., and Zhou, L. (2026). Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation. arXiv preprint, arXiv:2605.10448.
Li, P., Tang, X., Chen, S., Cheng, Y., Metoyer, R., Hua, T., and Chawla, N. V. (2026). Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. ICML, PMLR 306, 70246–70269.
Response to Reviewer GeF8 (1/3)
Thank you for pointing us to the ranking literature and for suggesting the width and cost comparisons. We tried the smaller workload you suggested. On an equal mixture of Terminus-2 and Forge, only one model has runs under both harnesses. Keeping all 17 yields four pairwise orderings. With the observed cell rates held fixed, Claude Opus 4.6 leads GLM-4.7 by at least 5.66 percentage points under every completion of the missing scores. Q2 gives the sampling result for this same mixture.
W1. Methodological advance over classical results
A maintainer considering a task-label merge faces two decisions. The classical connectedness condition tells us which configuration contrasts become estimable if the join is valid and task effects are additive. In the archive, candidate joins raise that count from 1,138 to 1,378 on the same 4,702 cells. The merge supplies no new observations. To publish an ordering on a specified workload, we check its weighted score difference over all admissible completions. For pairs still open, the same intervals tell us which missing evaluation would narrow them most in one step (Sections 5–6 and Q5).
Gao and Zhou (2026, Table 3) also retain model orders when bounds separate. Their unknowns are outcomes of completed runs with incomplete verification evidence. Here the maintainer can choose which missing model–harness–task combination to evaluate.
W2. What the global zero tells us about sparse coverage
We agree with your arithmetic. On the global uniform target each model has less than 40% of its target mass observed. Its unrestricted score interval must therefore contain one half, whatever the observed scores. This diagnoses a gap in workload coverage. The two-harness example above shows why the choice of target matters even for a maintainer working with exactly the same archive. Q1 tests the score-range restriction you suggested.
W3. Prediction accuracy and evidence from a second panel
The available-score and ridge baselines matched the certificate's zero sign errors on the four-domain target at 80% and 90% observation. On the six-benchmark uniform target at 90% observation, the configuration-block masks gave each method 1,242 decisions. Each baseline made one error and the certificate made none. These counts use the complete archived scores as the reference. For an individual decision, the certificate establishes whether the order holds for every completion. Q4 gives the acquisition results on Terminus-2 and seven panels from the independent HAL release.
W4. Missing scores and rank aggregation
Section 2 now discusses the three papers you suggested. Colombo et al. (2022) and Rofin et al. (2023) aggregate task rankings. Himmi et al. (2024) use rankings compatible with partial observations in Borda aggregation. Our comparison uses the difference in two models' weighted success rates. Both the size of an observed score gap and the workload weight of a missing cell affect which order can be published. Changing the weights changes the target of that comparison.
Q1. At what restriction does the first historical pair become certifiable?
There is no first certificate in the score-range family we tested. At Γ = 0, a missing score lies between the observed minimum and maximum for that harness and task if at least two models have observations there. Elsewhere it ranges from zero to one. Holding observed rates fixed, this tightest restriction certifies 0 of 136 pairs under each of the three global task-label policies. Increasing Γ expands each informed interval by Γ at both ends, clipped to zero and one. Each pair's difference interval therefore continues to contain zero (Appendix Q.1).