acceptodds
Under review as a conference paper at ICLR 2027

Sparse Agent Leaderboards: Identity, Support, and Identifiability

Abstract

Sparse agent leaderboards compare model-harness configurations under uneven coverage. A model ranking still needs one shared evaluation target. We map task identity, common-target support, and design identification onto three actions: certify a supported ordering, label a model-assisted extrapolation, or defer the comparison. Under task additivity, an incidence graph characterizes identifiable configuration contrasts, while free model-harness interactions leave model main effects unidentified. Sharp probability-scale bounds quantify what missing cells alone permit. In 22,818 archived Terminal-Bench trials spanning 53 configurations, candidate task-label joins connect a disconnected graph without adding any of the 4,702 observed cells, and a 55-record provenance sample contains four reported signatures within one name family. Under every label policy the best-covered model still lacks more than 60% of a uniform harness-task target, so no model pair can be certified against every bounded completion. On a separate public panel of 150 published aggregates, DeepSeek V3.2 beats GPT-5.2 under the source four-domain weights and loses under equal benchmark weights. At 90% observed cells the certificates cover 57.75-61.85% of pairwise orderings, while available-score ranking still publishes all ten pairs. Point estimates can match that coverage with almost no errors on this fixed table. The certificate is a worst-case column and a next-cell rule, not a rival accuracy score. Revealing ten additional deferred-weight cells on random 90% masks raises four-domain certificate coverage from 61.65% to 97.75%, versus 84.25% for ten random extra cells. A table-agnostic CSV tool returns the same three actions from observed cells.

Primary area:
datasets and benchmarks
Keywords:
Sparse leaderboards, Agent evaluation, Partial identification, Identifiability, Missing data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GeF8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks when a leaderboard of model–harness configurations with uneven coverage can support a model ranking. It argues that three questions must be answered before ranking:

  • Identity: do shared task names refer to the same versioned task and scorer?
  • Support: how much of a declared common harness–task target mixture (q, r) was actually observed for each model?
  • Identification: under a task-additive logit model, which contrasts are recovered by the observed design?

These map to three reporting actions: certify an ordering when sharp probability-scale missing-outcome bounds (Eq. 6, Manski-type) exclude zero; label a model-assisted extrapolation; or defer, naming the next cell to acquire.

The technical results are as follows. Proposition 1 shows that within-component contrasts in the configuration–task incidence graph are identified, while model main effects are not identified under free model–harness interactions. Proposition 2 states that cross-component comparisons need extra restrictions. Model–task interactions break the logit certificate (Eq. 5).

The paper names a real and under-discussed problem in agent evaluation. Its insistence on separating identity, support and identification is a valuable framing for practitioners who publish leaderboards.

Critical strengths

S1. A useful conceptual separation. The paper makes precise three things that leaderboard discussions routinely conflate: (i) task identity, (ii) target support, and (iii) likelihood identification. The point that "connected graph ≠ common-target support" (Fig. 1, Fig. 8) is well made. Candidate label joins raise identified observed-configuration contrasts from 1,138 to all 1,378 (l. 295–297) while adding zero observations, and target support stays at most 39.89%. This is a clear, reusable lesson.

S2. A principled, actionable reporting rule with a clean guarantee. The certificate reports a sign only when every bounded completion agrees. Because it compares per-model intervals [L_i, U_i], certified pairs form an interval order, which is transitive by construction. The proposed release format (supported sign, point estimate, deferred pairs plus next cell; §9) is concrete and could be adopted by leaderboard maintainers. A support-only gate is clearly worse: ridge with minimum pair support 0.9 reports 30.45% of pairs with 2.96% error, while the certificate reports about 61.65% with zero error (l. 425–427).

S3. A concrete demonstration that the target matters. The DeepSeek V3.2 / GPT-5.2 reversal between published weightings (Table 5) and the measured gain from a target-mass-based next-cell rule (Table 6) make the abstract argument tangible for practitioners.

Critical weaknesses

W1 (major). The technical core is classical, and the paper's own advance over it is thin.

  • Identifiability of within-component contrasts in additive two-way designs is the textbook connectedness condition. It goes back to connectedness in incomplete cross-classifications, e.g. Weeks & Williams (1964), and to Ford's (1957) condition for paired-comparison (Bradley–Terry) estimation.
  • The bounds are standard Manski missing-outcome bounds.
  • The authors acknowledge both (l. 111–113, l. 118–119).

What remains is the mapping of these tools onto three reporting actions, plus the case studies. That can be a contribution, but then the empirical evidence must show that the protocol changes conclusions in ways that matter and outperforms simpler practice. W2 and W3 argue that it does not yet.

W2 (major). On the real sparse leaderboard, the headline result is determined by arithmetic, and the unrestricted certificate is uninformative at realistic sparsity.

  • On Terminal-Bench, every model's joint support under the uniform target is below 0.4. If s_m < 1/2, every model's interval contains 1/2, so no pair can ever be certified. This follows from coverage alone, without looking at any outcome. It is not an empirical finding about agents.
  • The uniform 30-harness × ~90-task target is, by the authors' own description, a "coverage stress test", not a deployment target.
  • On the public panel, certificate coverage is about 0–2.5% at 60% observation, 25.50% at 80%, and about 60% at 90%.

W3. The certificate has no measured accuracy advantage, and the evidence comes from a single small table.

W4. The positioning in related work is incomplete. Closely related work on missing scores and aggregation in benchmarks is not cited:

  • Himmi et al., "Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks" (EMNLP Findings 2024);
  • Colombo et al., "What are the best systems? New perspectives on NLP benchmarking" (NeurIPS 2022);
  • Rofin et al., "Vote'n'Rank" (EACL 2023), which studies how the choice of aggregation rule changes rankings — directly relevant to the weighting reversal.
Questions for authors

1.Restricted bounds. Under a sensitivity parameter, e.g. |ζ_mt| ≤ Γ, bounded harness effects, or benchmark-specific score ranges, how many Terminal-Bench pairs become certifiable as a function of Γ? At what Γ does the first certificate appear?

2.Deployment targets. On Terminal-Bench, what happens under a single-harness or a small prespecified harness-mixture target (e.g. the 2–3 most-used harnesses)? How many pairs are certified or deferred then?

3.Sampling uncertainty. Applying Eq. 7 with valid simultaneous cell intervals, how much certificate coverage remains on the public panel at 90% observation? Is the DeepSeek V3.2 vs. GPT-5.2 gap (0.005) distinguishable from sampling noise under either target?

4.Second panel. Can the masking and next-cell evaluation be repeated on another crossed panel, e.g. HAL, or a Terminal-Bench subset with dense coverage?

5.Next-cell rule vs. stronger baselines. How does it compare with an exact bound-width-greedy rule, a cost-weighted variant (cells differ greatly in evaluation cost), or a set-cover formulation over deferred pairs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GeF8 (1/3)

Thank you for pointing us to the ranking literature and for suggesting the width and cost comparisons. We tried the smaller workload you suggested. On an equal mixture of Terminus-2 and Forge, only one model has runs under both harnesses. Keeping all 17 yields four pairwise orderings. With the observed cell rates held fixed, Claude Opus 4.6 leads GLM-4.7 by at least 5.66 percentage points under every completion of the missing scores. Q2 gives the sampling result for this same mixture.

W1. Methodological advance over classical results

A maintainer considering a task-label merge faces two decisions. The classical connectedness condition tells us which configuration contrasts become estimable if the join is valid and task effects are additive. In the archive, candidate joins raise that count from 1,138 to 1,378 on the same 4,702 cells. The merge supplies no new observations. To publish an ordering on a specified workload, we check its weighted score difference over all admissible completions. For pairs still open, the same intervals tell us which missing evaluation would narrow them most in one step (Sections 5–6 and Q5).

Gao and Zhou (2026, Table 3) also retain model orders when bounds separate. Their unknowns are outcomes of completed runs with incomplete verification evidence. Here the maintainer can choose which missing model–harness–task combination to evaluate.

W2. What the global zero tells us about sparse coverage

We agree with your arithmetic. On the global uniform target each model has less than 40% of its target mass observed. Its unrestricted score interval must therefore contain one half, whatever the observed scores. This diagnoses a gap in workload coverage. The two-harness example above shows why the choice of target matters even for a maintainer working with exactly the same archive. Q1 tests the score-range restriction you suggested.

W3. Prediction accuracy and evidence from a second panel

The available-score and ridge baselines matched the certificate's zero sign errors on the four-domain target at 80% and 90% observation. On the six-benchmark uniform target at 90% observation, the configuration-block masks gave each method 1,242 decisions. Each baseline made one error and the certificate made none. These counts use the complete archived scores as the reference. For an individual decision, the certificate establishes whether the order holds for every completion. Q4 gives the acquisition results on Terminus-2 and seven panels from the independent HAL release.

W4. Missing scores and rank aggregation

Section 2 now discusses the three papers you suggested. Colombo et al. (2022) and Rofin et al. (2023) aggregate task rankings. Himmi et al. (2024) use rankings compatible with partial observations in Borda aggregation. Our comparison uses the difference in two models' weighted success rates. Both the size of an observed score gap and the workload weight of a missing cell affect which order can be published. Changing the weights changes the target of that comparison.

Q1. At what restriction does the first historical pair become certifiable?

There is no first certificate in the score-range family we tested. At Γ = 0, a missing score lies between the observed minimum and maximum for that harness and task if at least two models have observations there. Elsewhere it ranges from zero to one. Holding observed rates fixed, this tightest restriction certifies 0 of 136 pairs under each of the three global task-label policies. Increasing Γ expands each informed interval by Γ at both ends, clipped to zero and one. Each pair's difference interval therefore continues to contain zero (Appendix Q.1).

AuthorsOpenReview ↗

Response to Reviewer GeF8 (2/3)

Q2. Which comparisons survive on a single harness or a small mixture?

We selected harnesses by archived trial count before comparing scores. Table 21 weights the selected harnesses equally and the 89 literal tasks in their union uniformly. All 17 models are included, assuming scores are defined on every positive-weight configuration. The table holds observed success rates fixed.

Deployment targetCertified pairsDeferred pairs
Terminus-221115
Terminus-2 + Forge4132
Terminus-2 + Forge + Ruley0136

All four two-harness orderings use partial coverage. The 21 on Terminus-2 come from seven fully observed models. Extending any row to all 179 literal tasks leaves every pair deferred.

For the underlying success probabilities, the two 95% simultaneous Hoeffding constructions both defer all 136 pairs on the equal mixture. On the complete seven-model Terminus-2 subset, the model-level construction retains 9 of 21 orderings assuming independent bounded trials within each model. The cellwise construction corrects over observed cells, requires independence within each cell, permits dependence across cells, and retains none (Appendix Q.1).

Q3. Is the DeepSeek–GPT gap distinguishable from sampling noise?

The archive leaves the sign of the DeepSeek V3.2–GPT-5.2 difference open under both weightings. Under each of the two count reconstructions, the simultaneous interval for that pair includes zero even on the full table. At 90% observation, Eq. (7) certifies no pair on any of the 200 random masks under either weighting. The roughly 0.005 lead under the four-domain weights and its reversal under equal benchmark weights describe the published scores.

Successful and total session counts reproduce 125 of 150 primary scores. For another 23 SWE-bench cells, primary score times completed sessions and average score times total sessions give the same integer numerator. We analyzed the 125- and 148-cell reconstructions separately using 95% Bonferroni-adjusted exact binomial intervals under within-cell binomial sampling. Masked or unresolved scores range from zero to one. A separate deterministic sensitivity check allowing ±0.25 percentage points per score also puts zero within the difference under both weightings (Appendix Q.3).

Q4. Does the acquisition result replicate on another crossed panel?

We repeated the replay on seven complete panels from the independent HAL release, selecting rectangles from observation patterns alone. Each crosses three model settings with two scaffolds, with reasoning-effort settings kept distinct. Each policy started from the same 90% random masks and revealed ten cells under uniform scaffold–task weights.

HAL benchmarkRandom (%)Deferred-weight (%)Width-greedy (%)Paired gain (pp)
OnlineMind2Web0.000.000.000.00 ± 0.00
GAIA0.000.000.000.00 ± 0.00
ScienceAgentBench46.9447.1758.4611.52 ± 17.68
SciCode0.000.000.000.00 ± 0.00
τ-bench Airline15.4815.4816.941.46 ± 4.67
CORE-Bench Hard4.154.154.460.31 ± 1.91
AssistantBench17.1217.1221.984.85 ± 7.05

The last column gives width-greedy minus random in percentage points, as paired mean ± sample SD over 200 masks after averaging eight shared randomized tie orders per mask. Gains use unrounded values. The SD describes variation across masks on fixed archived scores.

ScienceAgentBench shows what the choice of ten cells can buy. Mean certificate coverage rose from 46.94% with random selection to 58.46% with width-greedy, a paired gain of 11.52 percentage points. Width-greedy improved mean coverage on four of the seven panels. The original deferred-weight rule matched random on six. In a separate 90-reveal OnlineMind2Web replay, its apparent advantage under fixed lexical ties disappeared with shared randomized ties. On the 623-cell Terminus-2 block, ten reveals gave 68.48% coverage for width-greedy, 66.74% for deferred-weight and 66.65% for random (Tables 23–24).

AuthorsOpenReview ↗

Response to Reviewer GeF8 (3/3)

Q5. What changes when the rule optimizes width or accounts for cost?

In the fixed-score replay, each missing score ranges from zero to one and is known once revealed. If model m belongs to d currently deferred pairs, revealing one of its cells of workload weight w narrows each of those d intervals by w. Their total width falls by exactly w × d, whatever score is revealed. Width-greedy chooses the largest such reduction and updates the deferred set after each reveal. We use shared randomized priorities for ties. ATLAS selects items for their Fisher information about latent ability (Li et al., 2026). Our objective measures the remaining uncertainty in specified workload comparisons.

On the four-domain public panel, after ten reveals from the same 200 masks at 90% observation, mean coverage was 98.20% for width-greedy, 97.53% for the original rule and 83.69% for random. This gives a 14.51 percentage point gain over random and a 0.67 point gain over the original rule.

At a retrospective USD 100 budget, dividing width reduction by quoted cost gave 74.95% mean coverage, compared with 72.71% for raw width and 70.19% for random. Accounting for cost therefore gained 2.24 percentage points over raw width and 4.76 over random. The replay treats historical costs as known quotes. For zero quotes we used the median positive cost among initially observed cells on the same benchmark, falling back to the observed global median (Table 23).

For set cover, we would need to know which pairs each candidate cell settles. The hidden score determines that set. A reveal can shrink an interval while leaving zero inside it. Width gives a quantity we can optimize before the reveal, and the replay then counts the pairs actually settled. In a live evaluation, estimating the chosen cell's success probability may require repeated trials and simultaneous bounds as in Eq. (7).

References

Colombo, P., Noiry, N., Irurozki, E., and Clémençon, S. (2022). What are the best Systems? New Perspectives on NLP Benchmarking. Advances in Neural Information Processing Systems, 35.

Rofin, M., Mikhailov, V., Florinsky, M., Kravchenko, A., Shavrina, T., Tutubalina, E., Karabekyan, D., and Artemova, E. (2023). Vote’n’Rank: Revision of Benchmarking with Social Choice Theory. EACL, 670–686.

Himmi, A., Irurozki, E., Noiry, N., Clémençon, S., and Colombo, P. (2024). Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks. Findings of EMNLP, 11759–11785.

Gao, S., and Zhou, L. (2026). Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation. arXiv preprint, arXiv:2605.10448.

Li, P., Tang, X., Chen, S., Cheng, Y., Metoyer, R., Hua, T., and Chawla, N. V. (2026). Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. ICML, PMLR 306, 70246–70269.

Reviewer_w7xjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for comparing agents under sparse model–harness–task coverage. It separates task identity, common-target support, and model-dependent identifiability, and recommends certifying an ordering, labeling an extrapolation, or deferring comparison. The framework combines graph-based identification analysis with bounded-outcome intervals and a next-cell acquisition rule.

Critical strengths

The formulation is useful and well motivated. Separating identity, support, and identifiability clarifies why a fitted ranking may not be supported by the evaluation design. Explicitly diagnosing insufficient evidence is itself a meaningful contribution.

The empirical analysis is transparent about its limitations, including unresolved task identities, predictive misfit, and the strong performance of matched-coverage baselines.

Critical weaknesses

The demonstrated practical benefit remains limited. The historical analysis certifies none of the 136 model pairs, while matched-coverage predictive baselines make no errors in the reported four-domain experiments at 80% and 90% observation. Acquisition improves certificate coverage, but the evidence comes from revealing fixed aggregates in one table, with the highlighted experiment increasing observation from 135 to 145 of 150 cells. Given the classical methodological ingredients, stronger evidence of useful decisions under realistic sparsity and sampling uncertainty would strengthen the contribution.

The graph-based identification component relies on restrictive task additivity, which excludes configuration-specific task effects such as a harness selectively refusing a task category. The historical predictive discrepancies reinforce concerns about applicability. This does not invalidate the bounded-outcome certificates, but leaves the practical contribution of the model-based component less clear.

Questions for authors

Do plausible deployment-oriented targets yield informative certificates on the historical data, unlike the uniform target? Results or existing evidence here could materially strengthen the case for practical usefulness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer w7xj

Thanks for the review. We followed your suggestion to examine a workload a team might actually use. The equal mixture of Terminus-2 and Forge gives a concrete example.

W1. Practical value on the historical archive

Restricting the comparison to models evaluated under both harnesses leaves one eligible model. Keeping all 17 models yields four pairwise orderings on the mixture. With the observed cell rates held fixed, Claude Opus 4.6 leads GLM-4.7 by at least 5.66 percentage points even if every missing score favors GLM-4.7. Q1 gives the workload weights and the result when we also account for sampling uncertainty.

W2. Predictive baselines at matched coverage

At matched coverage, available-score and ridge predictions recovered the archived signs on the four-domain target at 80% and 90% observation. Both also did so on the 623-cell Terminus-2 block under random and score-selected masks. With the six public benchmarks weighted equally, the 90% configuration-block masks gave each method 1,242 decisions. Available-score and ridge each made one sign error, and the certificate made none. Errors were rare in this replay. For any particular order, the certificate also tells the maintainer whether admissible missing scores could reverse it.

W3. Acquisition evidence beyond the public table

Starting from the same 90% random masks on Terminus-2, ten reveals raised mean certificate coverage from 57.69% to 68.48% with width-greedy, 66.74% with deferred-weight and 66.65% with random selection. The paired gain of width-greedy over random was 1.82 percentage points before rounding. On seven crossed panels from the independent HAL release, width-greedy improved mean coverage on four. ScienceAgentBench gained 11.52 percentage points over random at the same ten-cell budget. Three panels yielded no certificate under any of the rules. Table 24 reports the full comparison with shared randomized tie orders.

The starting coverage matters. On the Terminus-2 block, ten reveals from 60% observation yielded no certified order under the fixed-order policies studied. These replays reveal known archived scores. A fresh run would supply another trial, and the resulting success probability would still need a sampling bound (Appendix Q.4).

W4. When is the graph diagnostic applicable?

Under task additivity, the graph identifies which configuration contrasts can be estimated and where observations would have to link components. Your example of a harness refusing a particular task category illustrates when this interpretation needs care. Our held-out check examined model–task interactions. Adding them reduced log loss from 0.3827 to 0.3734. We therefore report graph contrasts conditional on additivity. The workload comparison in Eq. (6) already allows configuration-specific task outcomes (Sections 4 and 7, Table 22).

Q1. Do deployment-oriented targets yield informative historical comparisons?

Yes. The table shows how many orderings hold across all completions of the missing scores at the archived observed-cell rates. We selected harnesses by trial count before comparing scores, weighted the selected harnesses equally, and gave equal weight to the 89 literal tasks in their union. Table 21 keeps all 17 models and assumes a score is defined for each positive-weight configuration.

Deployment targetCertified pairsDeferred pairs
Terminus-221115
Terminus-2 + Forge4132
Terminus-2 + Forge + Ruley0136

All four orderings on the two-harness workload use partial coverage. The 21 on Terminus-2 come from seven fully observed models. Extending any of these workloads to all 179 literal task labels leaves every pair deferred.

For inference about underlying success probabilities, both 95% simultaneous Hoeffding constructions defer all 136 pairs on the mixture. On the complete seven-model Terminus-2 subset, nine of 21 orderings survive when we aggregate independent bounded trials within each model and correct across models. Correcting across cells, with independence required only within each cell, retains none. The archive contains one or two submission–run groups per model on this subset, so the nine orderings depend on how trials within a model were generated (Appendix Q.1).

Reviewer_aE6SOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that considers task identity, evaluation coverage, and statistical identifiability to decide whether to certify a ranking, report a model-assisted extrapolation, or defer the comparison. Experiments on Terminal-Bench and a public leaderboard show that task-label choices affect identifiability, sparse coverage can prevent reliable rankings, and targeted evaluations can increase the number of certified comparisons.

Critical strengths

The paper addresses an important problem and presents a thoughtful, reproducible framework. The empirical gains are demonstrated on a limited set of benchmark structures.

Critical weaknesses

My concern is that the framework may be a valuable combination of existing methods rather than a substantial new algorithm or theoretical result. For ICLR, the authors need to make the incremental contribution over partial-identification methods, selective classification, and existing benchmark-analysis tools particularly clear. It seems that the theoretical novelty is limited.

The method assumes that tasks affect all models in a similar way, but the results suggest this may not always be true. The authors should test whether their method remains reliable when different models perform differently across tasks.

Questions for authors

What is the main methodological novelty of this work beyond the integration of existing techniques into a system framework?

Could you clarify how robust the proposed method is when different models perform differently across tasks, given that the results suggest the task-additive assumption may not adequately capture the observed variation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer aE6S (1/2)

Thank you for pressing us on the methodological contribution and on what happens when different models find different tasks difficult. We have made the release decision concrete and compared the additive fit with one that allows model-specific task effects.

W1. Contribution beyond existing methods

A maintainer needs to decide which model pairs to order on a given workload and, for the pairs still open, which missing evaluation to run next. We use the same pairwise score intervals for both decisions. We can publish an ordering if every admissible completion gives the same sign. For the remaining pairs, we can calculate how much revealing a candidate cell's score would narrow their intervals in a fixed-score replay. Q1 shows what the maintainer gains at each step.

This uses classical missing-outcome bounds (Manski, 2003). El-Yaniv and Wiener (2010) study the trade-off between risk and coverage in selective classification, and Han and Sun (2026) estimate model and harness effects on Terminal-Bench. The closest use of bounds for ranking is Gao and Zhou (2026, Section 3 and Table 3). Their bounds reflect uncertainty about outcomes of completed runs. Here the missing quantity belongs to a model–harness–task combination the maintainer can still evaluate.

W2. Task difficulty can vary across models

Task additivity enters the graph result through a common task effect on the logit scale. The certificate for a workload allows each model–harness–task probability to vary within its own bounds. Q2 shows what changes when we relax additivity in the predictive model (Sections 4, 5 and 7).

Q1. What does a maintainer gain from the procedure?

Consider an equal mixture of Terminus-2 and Forge on their 89 archived tasks. Requiring each model to have runs under both harnesses leaves one eligible model and no pair to compare. Keeping all 17 models and the declared weights yields four orderings that survive every completion of the missing scores, with the observed success rates held fixed. The other 132 pairs remain open. We selected the harnesses by archived trial count and assumed scores are defined for all 17 models on the workload (Section 6, Table 21). Both simultaneous sampling constructions leave all pairs open on this mixture when the observed probabilities also need to be estimated (Appendix Q.1).

Joining task labels raises a different decision. On the same 4,702 observed cells, candidate joins connect the graph and increase the number of identifiable configuration contrasts from 1,138 to 1,378 under task additivity. Whether the labels describe the same task version requires separate evidence. In a provenance sample, 55 records from one name family carry four reported task signatures. Even under the joins, each model lacks observations for over 60% of the global uniform target. A maintainer can report the additional contrasts conditional on the joins and additivity, while deferring all 136 model comparisons on that target.

For a pair that remains open, revealing a missing cell's archived score removes its workload weight from the width of the interval if the pair includes that model. Choosing the largest total reduction across open pairs gave 98.20% certificate coverage on the four-domain public panel after ten reveals from 90% random observation. On the same masks and shared tie orders, random selection gave 83.69% and the original deferred-weight rule gave 97.53%. The gain over random amounts to about 1.45 additional reportable orderings out of ten on average (14.51 percentage points). The smaller gain over deferred-weight was 0.67 points. We know the width reduction before choosing a cell. Which pairs become reportable depends on its score (Table 23).

AuthorsOpenReview ↗

Response to Reviewer aE6S (2/2)

Q2. How robust are the results to model-specific task effects?

With observed-cell probabilities fixed, a sign certified by Eq. (6) survives arbitrary model-specific task effects because each missing probability can already take any value from zero to one. For the graph result, Eq. (5) gives the ambiguity explicitly. A shift in a model's configuration effects can be offset by its task interactions while preserving every observed probability.

We compared two fits on the same five held-out-cell folds. Adding regularized model–task effects reduced per-trial log loss in every fold, from 0.3827 to 0.3734 when pooled across 4,701 cells. On the 3,565 cells whose model and task appeared together under another harness in training, loss fell from 0.3696 to 0.3572. These are interpolation results. They show that the interaction terms improve prediction within the archive, which is why we report graph contrasts conditional on additivity (Section 7, Table 22).

For unknown observed-cell probabilities, Eq. (7) requires simultaneous bounds valid for the sampling process. On the complete seven-model Terminus-2 subset, the model-level construction retains 9 of 21 orderings under independent bounded trials within each model. The cellwise construction permits dependence across cells and retains none (Appendix Q.1).

References

Manski, C. F. (2003). Partial Identification of Probability Distributions. Springer.

El-Yaniv, R., and Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53), 1605–1641.

Han, J., and Sun, Y. (2026). How good is your harness? Statistical evaluation of coding harnesses. CTB Workshop at ICML 2026.

Gao, S., and Zhou, L. (2026). Can agent benchmarks support their scores? Evidence-supported bounds for interactive-agent evaluation. arXiv preprint, arXiv:2605.10448.

Comments on the paper

1 comment
Area_Chair_vTPwOpenReview ↗

Reminder: Please respond to the author rebuttal

Dear Reviewers, the authors have responded to your reviews with additional experiments, and the author-reviewer discussion ends soon. Please read their response, post a brief reply on whether your concerns are addressed, and update your scores if your assessment has changed. Thank you! AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.