acceptodds
Under review as a conference paper at ICLR 2027

Is This Agent Evaluable? Certifying Verdicts About LLM Agents Under Nondeterminism, Oracle and Coverage Obstructions

Abstract

A benchmark score is an instrument reading. The instrument has four stages: a policy that samples actions, an environment that carries state, an oracle that renders a verdict, and a task suite somebody chose. We define evaluability as the existence of a procedure that settles a claim from the available evidence at a stated error level. We name five ways it fails, and measure each on live agents across three domains with unrelated checker designs. Four results. Swapping the tasks moves a score 6.2 times more than rerunning them, so a three-seed error bar prices the cheaper axis. Whether reruns settle a task is a fact about the pass rates, not the checker, so a fixed suite settles less as agents improve. Judges agree at 0.761 on benchmark obligations but only 0.513 on capabilities that real agents declare. And cheatability belongs to the interface: our own harness leaks every reference answer, making 0.608 of our suite passable with no task competence, none of it reachable by the five browser primitives our agent had. Of nine pre-registered predictions, 3 survive. Where an obstruction binds, the honest output is undecided with the reason named, rather than a score.

Primary area:
interpretability and explainable AI
Keywords:
LLM agents, agent evaluation, agentic benchmarks, web agents, tool-use agents, agent benchmark gaming, agent nondeterminism, LLM-as-a-judge, agent capability discovery, agent scaffolds, evaluation reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NYHKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an “evaluability” framework for LLM agents, identifying five obstacles: nondeterminism, judgement, task grouping, capability coverage, and shortcuts. Experiments across three benchmark domains suggest that task selection typically contributes more uncertainty than repeated execution, motivating an explicit “undecided” outcome when evidence is insufficient.

Critical strengths
  • Relevant problem and integrative framework. The paper shifts attention from benchmark scores to the claims those scores support, connecting statistical uncertainty, evaluator validity, and interface permissions. Its novelty lies primarily in conceptual synthesis and empirical diagnosis.
  • Cross-domain support for the central finding. Task-level uncertainty exceeds rerun uncertainty across different evaluation mechanisms. Controlled experiments investigating nondeterminism and exploitability further strengthen the evidence.
Critical weaknesses
  • The formal definition and empirical measures are not fully aligned. Settleability measures whether success probabilities approach zero or one, which differs from whether a capability claim can be reliably tested. The evidence supports an evaluation diagnostic framework more strongly than a unified certification framework.
  • The central statistical conclusions depend on the estimand. Performance on a fixed benchmark and generalization to a task population require different interpretations of uncertainty. The benefits of adding tasks also depend on their representativeness, dependence structure, and acquisition costs.
Questions for authors
  • Could the authors provide a complete certification example specifying the capability claim, error probability, and accept/reject/undecided decision rules, including which components carry statistical guarantees?
  • Does the advantage of adding tasks over reruns persist after distinguishing fixed-suite evaluation from population-level inference and accounting for template dependence and task acquisition costs?
2 comments
AuthorsOpenReview ↗

Complete certification example and when the task advantage holds

Thank you for the questions in the reviews.

W1 / Q1. Definition versus measures, and a complete example. Agreed: settleability is not testability. In the revision, settleability is a per-task diagnostic of how much rerun information is left. The certificate is a separate procedure (general response, point 1). Its parts carry guarantees as follows:

  • Statistical guarantee (frequentist, level α): the O1 bound. Clopper-Pearson on pooled runs for a fixed suite. A betting confidence sequence for task or family populations, valid under optional stopping.
  • No assumption about the judge: the O2 treatment. Bounds are taken over every way the judge could have ruled.
  • Logical gates with no statistical content: O4 (is the capability exercised?), O5 (is a verdict-fixing object reachable, or does a no-competence control already clear the threshold?), and O3 (families force family units).

Here is a complete example. Claim: "haiku-4-5 passes at least 50% of retail customer-service tasks," at α = 0.05, with the τ-bench retail tools as the action space.

  1. Gates: no reference is in reach, the capability is exercised, and there are no shipped families, so the task is the unit.
  2. Bound: a betting sequence over the 115 per-task rates gives [0.482, 0.677].
  3. Decision: the bound straddles 0.5, so the result is UNDECIDED (O1). About 460 tasks are projected to decide it.

The same traces accept "at least 50% of this suite," with bound [0.539, 0.604]. The table of 14 claims is in our response to sNcn.

W2 / Q2. Does the task advantage survive? We give three separate answers.

Fixed suite versus population. For a fixed-suite claim, the task axis is zero by definition, and reruns are the only lever. For population claims, the task component is 3.3 to 9.1 times the rerun component under every design we tested. The designs are tasks i.i.d., finite-population correction, families fixed (stratified) and families i.i.d. The table is in our response to MVZJ.

Template dependence. We fit a three-level model (template, task within template, run) on VisualWebArena's 32 intent templates. The standard error of sonnet-5's score, in points, is:

now2× reruns2× tasks, same templates2× templates
5.065.034.163.58

New tasks from existing templates still beat reruns, but they reach their own floor: 3.0 points, against a rerun floor of 5.0. The best spend is new templates. On CRMArena-Pro (17 categories) the effect is stronger: 7.45 now, 7.42 with 2× reruns, 7.17 with 2× tasks, and 5.27 with 2× categories.

Acquisition cost. Suppose a new task costs c runs. Then the optimal number of reruns per task is k* = sqrt(c · E[p(1 − p)] / Var_between(p)), Cochran's two-stage optimum. Reruns beyond one per task pay only when a new task costs more than 8 to 15 runs on VisualWebArena and CRMArena-Pro, or 3 to 4 runs on τ-bench. Our k = 8 is optimal only if a task costs 131 to 241 runs (56 to 59 on τ-bench). The revision therefore states the result as a cost condition and adds this table.

If these responses address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

Thank you for your response; my concerns have been fully addressed. However, I believe the paper itself still has significant room for improvement—for instance, regarding the poor quality of the presentation, the rudimentary charts and figures, and the pacing of the narrative. These issues cannot be resolved through the rebuttal discussion alone; in my view, the authors need to undertake further revisions to address them.

Reviewer_sNcnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at whether benchmark scores really tell us what an agent can do. The authors introduce the idea of “evaluability,” which asks whether a claim about an agent can actually be checked using the available traces. They identify five problems that can make this difficult, including variation across tasks, disagreement between judges, errors concentrated in certain task types, skills that are never tested, and shortcuts that let agents pass without demonstrating real ability. They study these issues across several web, tool-use, and CRM benchmarks, as well as agent registries. Their results suggest that the choice of tasks affects scores much more than rerunning the same tasks, and that standard error bars can miss this source of variation. They also find disagreements in judging declared agent capabilities and show that some benchmarks can be passed through shortcuts. The study was pre-registered, and the authors openly report that most of their predictions did not hold. They argue that evaluations should sometimes be allowed to return “undecided” rather than forcing a score.

Critical strengths
  • The comparison between variation across tasks and variation across reruns is useful. Most papers focus on seed-based error bars, but this work shows that the choice of tasks can matter much more. This has clear implications for how evaluation budgets should be spent.

  • I appreciate that the authors report failed predictions instead of focusing only on positive results. The pre-registration and discussion of mistakes in their own evaluation setup make the study more transparent.

Critical weaknesses
  • The entire abstract is fully AI generated. The second paragraph in abstract starts with "Four results. ".
  • There are too many numbers and newly introduced terms without enough explanation. Some statements are hard to understand on the first read, and the main text often feels like a collection of results rather than a clear argument. The connection between the five problems is also not very clear.
  • The paper talks about certifying evaluation results, but does not provide a clear procedure that takes evidence and returns a verdict such as accept, reject, or undecided. Instead, the five problems are studied separately, sometimes using small samples or LLM-based judgments without human validation. Some findings also seem fairly intuitive. For example, task-to-task variation can be larger than rerun variation, and agents can exploit benchmarks when answers are accessible through files. Given that most pre-registered predictions also fail, I am not fully convinced about the paper's main contribution beyond these observations.
Questions for authors
  • Can you show a concrete procedure that takes evaluation traces and returns accept, reject, or undecided? Even one worked example would help clarify the main contribution.

  • What does the variance analysis add beyond standard methods for comparing variation across tasks and repeated runs?

  • The judgments about checkability and declared skills rely on LLM panels. Did you have humans evaluate a sample, and how closely did their judgments agree?

  • Some experiments use relatively small sets of tasks or servers. How confident are you that the findings would hold with larger samples?

  • Why is interpretability listed as the primary area? The paper seems more closely related to agent evaluation and benchmarking.

3 comments
AuthorsOpenReview ↗

Rewritten abstract, worked certification table, variance analysis

Thank you. We take the presentation criticism seriously. The revision rewrites the abstract and the introduction around one argument.

The abstract. We have rewritten it. The new abstract states the question, gives three results with their numbers, and stops. The "Four results." list is gone.

Q1. A concrete accept, reject or undecided procedure. The rules are in point 1 of the general response. Here is the procedure run on our own traces at α = 0.05:

claimestimanddecisionbinding obstruction
sonnet-5 passes ≥ 20% of this VWA suitesuiteACCEPT, [0.227, 0.297]none
same, for VWA-shopping-like tasksfamiliesUNDECIDED, [0.015, 0.547]O1+O3, about 256 templates needed
opus-5 beats sonnet-5 on this suitesuiteUNDECIDED, [−0.084, 0.068]O1
gemini-3.7-flash beats haiku-4-5 on this suitesuiteACCEPT, [0.060, 0.212]none
same, on VWA-shopping-like tasksfamiliesUNDECIDEDO1+O3
sonnet-5 ≥ 20% of this suite, shell interfacesuiteUNDECIDEDO5: answer key reachable
sonnet-5 ≥ 50% of the CRMArena-Pro splitsuiteACCEPT, [0.542, 0.620]none
sonnet-5 ≥ 55% of the CRMArena-Pro splitsuiteUNDECIDEDO2: accepts only if 13 judge-scored passes are trusted
sonnet-5 ≥ 45% of CRMArena-Pro tasksfamiliesUNDECIDED, [0.070, 1.000]O1+O3: only 17 categories exist
sonnet-5 ≥ 30% of CRMArena-Pro taskspopulationUNDECIDEDO5: a no-competence control scores 0.328
sonnet-5 ≥ 50%, dataset readablesuiteUNDECIDEDO5
haiku-4-5 ≥ 50% of τ-bench retailsuiteACCEPT, [0.539, 0.604]none
same, for retail-like taskspopulationUNDECIDED, [0.482, 0.677]O1, about 460 tasks needed
haiku-4-5 handles airline refund policy, from retail tracesanyUNDECIDEDO4

The same evidence accepts a claim about the suite and leaves the matching capability claim undecided. That is the paper's point in one table. We also tested the error rate at the boundary μ = τ. It is 0 of 2,000 simulations for suite claims and 6 of 500 (0.012) for population claims, both below 0.05.

Q2. What the variance analysis adds. The estimator is not new, and the paper says so. It is a two-stage variance decomposition (Cochran, 1977). Three estimators agree on the task-to-rerun ratio: the bootstrap (3.2 to 6.1), a method-of-moments ANOVA (2.9 to 5.6) and the beta-binomial fit (2.8 to 5.6).

What we add is the measurement across three checker designs and its consequence for practice. A rerun error bar is the right interval for a claim about the suite. It is the wrong one for a capability claim, which is the claim agent papers usually make. We also add the cost question that standard treatments leave open: when is a rerun worth more than a new task? Our response to NYHK gives the answer.

AuthorsOpenReview ↗

Human validation, larger-sample replication, primary area

Q3. Human validation of the LLM panels. We have not yet collected human labels, and we will not report numbers we do not have. We have prepared a blind census kit: all 44 declared skills and all 40 residue obligations, each with exactly the prompt the panel saw. Two annotators label each sheet independently. The revision will report human-human alpha and human-panel kappa.

Two points bound the concern meanwhile. First, the P3 share (0.140) uses no LLM at all. It is a mechanical rule over the benchmarks' own checker types. Second, the P10 result was replicated with a different panel, and tested on a larger population (below).

Q4. Small samples. We ran two new arms with a rebuilt panel: five models from four vendors (Anthropic, Zhipu, Alibaba, DeepSeek), three samples each, with the identical prompt and parser. The panel shares only one vendor with the original.

Bridge arm: does a different panel reproduce the result? On the original 44 skills, the new panel finds 14 of 44 decidable: 0.318, Wilson interval [0.200, 0.466]. The paper's panel found 11 of 44 (0.250). The two panels agree on 41 of 44 skills (alpha 0.834 between their labels). The upper bound stays below one half, so P10's direction holds. Vendor dependence also replicates: per-vendor rates run from 0.250 to 0.364.

Scale arm: does it hold on a larger population? We froze a stratified sample of 300 servers from the public MCP registry. The frame is all 15,042 current servers. The strata are the declaration surfaces, allocated in proportion. The seed and content hash are recorded in the split manifest. This is about seven times the original sample. Each server's registry description is the declaration judged.

Only 1 of 300 servers is decidable: 0.003, Wilson interval [0.001, 0.019]. On average 0.913 of annotations flag the description as too vague to derive any check. The judges are unanimous on 0.923 of servers. Alpha is low (0.104) only because nearly every server falls in one class. This is the known prevalence paradox of chance-corrected agreement.

The host withdrew one Alibaba model partway through the first run of this arm. We therefore reran the Alibaba seat in full with a single model, so every judge covers every unit of both arms (parse rate 0.999). The numbers above come from that complete panel. The first, partial panel gives identical majority labels on all 44 skills and all 300 servers.

A one-line description is a thinner declaration than a skill card. So this supports P10's direction at seven times the sample size, not its exact rate.

For the statistical claims, the power curve in our response to MVZJ shows what the sample sizes support. About twice our suite separates the widest pair with probability near 0.9.

Some findings are intuitive, as you say. That task variation exceeds rerun variation is expected in direction. Its size is not standard knowledge: 3 to 6 times on every domain. Neither is the fact that a no-competence agent beats a genuine one on three CRMArena-Pro categories even with the answers hidden.

Q5. Primary area. Interpretability was a mis-selection on our part. The paper belongs under datasets and benchmarks. We will ask the area chair whether it can be corrected.

If these responses address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

I have read the author responses. They seem to have addressed some of the concerns I had pointed out. But in light of the initial stage of the paper how it was submitted with AI written text at many places, i am not sure if i can trust all aspects of the paper to its full human judgement. I decide to keep my score.

Reviewer_MVZJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames agent evaluability through five obstructions: nondeterminism, judgement, task families, capability reach, and shortcuts. Repeated evaluations across VisualWebArena, CRMArena-Pro, and τ-bench suggest that task composition contributes substantially more score uncertainty than rerunning fixed tasks. Further audits distinguish judge agreement from oracle validity, expose incomplete capability discovery, and demonstrate answer-key exploits whose reach depends on the interface. The study also reports failed predictions and corrections to its own instruments, with three of nine confirmatory predictions surviving its stated correction (§§3–14, pp. 2–9; Tables 2–7).

Critical strengths

A concrete, cross-domain account of the two uncertainty axes. Tables 2–3 (p. 3) report joint task/run interval widths versus run-only widths across three checker designs and two extensions. Equation (1) and §8 (p. 5) explain why additional tasks and reruns buy different information. The authors also explain why their own repeated-run design is inefficient for estimating an aggregate score but necessary for studying within-task variation.

Evidence-first instrument auditing with meaningful controls and unfavorable outcomes. The paper does not rescue failed predictions by silently substituting later analyses (Table 5, p. 8; A.2, p. 13). It distinguishes validity from agreement and tests an expected-invalid vision checker with a negative arm (A.23–A.26, pp. 22–24). The shortcut audit names the browser-only interface that excluded answer-key access in the main grid, and the CRMArena constant-“None” control prevents overstating its exploit margin (§13, pp. 8–9; A.19–A.20, pp. 20–21).

Critical weaknesses

The central error-controlled certification claim needs a defined inferential contract. Definition 1 fixes an agent, claim, and evidence set but does not specify the admissible data-generating class, competing hypotheses, or whether the error guarantee is frequentist or posterior (§3, p. 2). Posterior settleability uses an empirically fitted Beta prior; the manuscript does not explain how hyperparameter uncertainty or model calibration supports that guarantee (A.1, p. 12; A.11, p. 17). A.31 (pp. 26–27) does report a hedged betting confidence sequence and prior coverage simulations, so the concern is not an absent sequential experiment. It is that the betting construction, coverage results, task-level sampling assumptions, and meaning of “replicating the task distribution” for the roughly 2× task claim are not supplied. These distinctions matter when the main web set is also described as the entire eligible population (A.6, p. 15). Moreover, Table 12 labels Beta(0.622, 0.465) “interior-peaked,” although both parameters below one imply a U-shaped density; Beta posterior means shrink toward the prior mean, not outward as a consequence of concentration below one. The reported posterior tail probabilities nevertheless reproduce. Please define the guarantee and estimand, give the sequential construction and simulation evidence, distinguish finite-suite from population inference, and correct the prior-shape explanation.

The shortcut evidence supports interface-dependent leakage, but not the stronger claim that reference isolation cannot repair it. The CRMArena exploit explicitly reads the reference from the dataset (Table 13, p. 21). The inference that hiding that dataset cannot repair the checker because the evaluator must contain the grading value conflates evaluator access with agent access (§13, p. 9; A.17, p. 20; A.20, p. 21). An evaluator can retain a private reference while denying the agent access; the paper's own browser-only main grid illustrates the importance of this boundary. Exact matching is not intrinsically an invalid grading rule for an answer task. This counterargument leaves the demonstrated 113/119 exploit and the 22/119 degenerate control intact, but challenges the stronger prescription that the verdict must read different state. Please specify the threat model and either test an isolated-reference condition or narrow the claim to the exposed-reference interface.

Questions for authors

What hypotheses, sampling population, and error notion make Definition 1 operational, and how do fitted-prior settleability and the task-level betting sequence implement that same guarantee? Please include the sequence construction, coverage-simulation design/results, and the independence or finite-population treatment behind the 2× task extrapolation (A.1, A.6, A.11, A.31). Does the CRMArena exploit remain effective when the evaluator can read the reference but the agent cannot? If that condition was not tested, what evidence excludes isolation as a repair (Table 13; A.17, A.20)? What is the intended claim for O2: computational undecidability, insufficient specification, or empirical judge dependence? The manuscript's wording “no program can decide” is stronger than residue labelling and agreement measurements establish. How should readers interpret these measurements given the acknowledged failed validity audit and the deliberately text-only image-referencing cases (Table 8; A.4, A.23, A.25–A.26)?

3 comments
AuthorsOpenReview ↗

Inferential contract, prior uncertainty, confidence-sequence coverage

Thank you for a careful and technical review. You found a real error (the prior shape) and an overstated claim (isolation). We fix both, and we answer each concern with a measurement.

W1. The inferential contract of Definition 1. See points 1 and 2 of the general response. Concretely:

  • Hypotheses: H_accept: μ > τ against H_reject: μ < τ. For comparisons, μ_A − μ_B > 0 against < 0.
  • Data-generating class: runs are independent Bernoulli trials given the task. For population claims, tasks (or families) are also i.i.d. from a stated distribution.
  • Error notion: frequentist, uniform over that class. P(accept and false) ≤ α and P(reject and true) ≤ α. Undecided is never an error.
  • Suite versus population: the estimand is part of the claim. Each has its own bound (general response, point 2).

The fitted Beta prior plays no role in any certificate.

Settleability and hyperparameter uncertainty. You are right that a fitted prior cannot carry an error guarantee. We measured how much its uncertainty matters by refitting the prior on 1,000 task bootstraps:

domainsettleability95% interval from prior uncertainty
VisualWebArena0.755[0.614, 0.755]
CRMArena-Pro0.395[0.000, 0.605]
τ-bench retail0.000[0.000, 0.000]

CRMArena-Pro's value is fragile. Its 47 all-pass tasks sit just above the 0.9 posterior bar (mass 0.933), so a small change in the prior moves all of them at once.

We also checked calibration on held-out runs. We fit on 4 runs per task and test on the other 4, over all 70 splits. Predictive calibration is good: ECE is 0.004, 0.013 and 0.026. But "settled fail" calls are miscalibrated off the web domain. Their held-out runs fail at 0.981 on VisualWebArena, 0.766 on CRMArena-Pro and 0.620 on τ-bench, against a nominal 0.9.

So the revision reports settleability as a descriptive diagnostic with its bootstrap interval. P1's direction (below 0.70) survives prior uncertainty on CRMArena-Pro and τ-bench. On VisualWebArena the interval crosses 0.70. This agrees with the convention dependence the paper already reports.

The betting sequence: construction and coverage. For a candidate mean m, wealth is K_t(m) = ∏_{i≤t} (1 + λ_i (x_i − m)). The bet λ_i is predictable: it uses a running variance estimate and is capped at 0.5. Two processes bet up and down, each at level δ/2. The value m is excluded once either wealth exceeds 2/δ. Ville's inequality gives time-uniform coverage.

New simulations count the probability that the true mean is ever excluded along the path, at nominal 0.05:

targeti.i.d. taskswithout replacementfamilies, task unitfamilies, family unitpeeking t-interval
VWA widest-pair difference0.0000.0000.0000.0000.18 to 0.31
VWA sonnet-5 score0.0030.0000.0080.0000.16 to 0.34
CRMArena-Pro score0.0070.0000.1660.0000.17 to 0.56
τ-bench retail score0.0080.000n/an/a0.16 to 0.28

Two results follow. The sequence holds under sampling without replacement, so it is valid for a finite pool. It fails at the task level when tasks come in families (CRMArena-Pro, 0.166), and recovers once the family is the unit. This is why the procedure moves to family units (gate O3).

AuthorsOpenReview ↗

Power curve, estimands side by side, corrected prior shape

The "roughly 2×" figure. You are right that "replicating the task distribution" was undefined. The original computation repeated the same 79 differences in one fixed order. That is a single path, not a probability. We replace it with a power curve. Units are drawn i.i.d. from the empirical distribution, with 300 simulations per point:

suite sizeP(decide), tasks i.i.d.P(decide), templates i.i.d.
1×0.020.04
1.5×0.450.48
2×0.920.88
3×1.001.00

So "about 2×" survives as a statement with power near 0.9. It assumes new tasks resemble the observed ones, and the revision states that assumption.

Finite suite versus population. On VisualWebArena, "entire eligible population" meant every task our harness supports (79 of 466 shopping tasks). It was not a random sample. We now report all estimands side by side (95% interval widths, in points):

domainfixed suitetasks i.i.d.+ FPC to framefamilies fixed (stratified)families i.i.d.
VWA (5 configs)2.8 to 3.614.7 to 18.013.4 to 16.510.6 to 13.118.7 to 24.4
CRMArena-Pro3.415.514.811.030.5
τ-bench retail4.314.44.3 (frame = released set)n/an/a

CRMArena-Pro was drawn stratified, 7 of 100 tasks per category. Its correct population interval is therefore the stratified one, 11.0 points. The paper's 15.0 overstated it. The task-to-rerun ratio is still 3.3. On τ-bench we ran every released task, so the task axis is zero for the released set. It is nonzero only for retail tasks in general.

The prior shape (Table 12). You are right, and we thank you. Beta(0.622, 0.465) is U-shaped. Our code classified shape by the sign of α + β − 1, which is wrong. All three fitted priors are U-shaped. This holds in every bootstrap refit on VisualWebArena and CRMArena-Pro, and in 99% of refits on τ-bench.

The "outward shrinkage" explanation was also wrong. After 8 of 8 passes, posterior means shrink toward the prior mean on both domains (0.977 toward 0.595; 0.949 toward 0.572). The real mechanism is the β parameter, the prior's weight near p = 1. After k of k passes the posterior is Beta(α + k, β), and its mass above 0.9 is governed by β. Swapping parameters between the domains shows it:

α fromβ frommass above 0.9 after 8/8
CRMArena-ProCRMArena-Pro0.933
τ-benchτ-bench0.831
τ-benchCRMArena-Pro0.937 (settled)
CRMArena-Proτ-bench0.822 (not settled)

The tail probabilities in the paper are unchanged, as you found.

AuthorsOpenReview ↗

Reference isolation experiment and the scope of O2

W2. Shortcuts and reference isolation. You are right that we conflated evaluator access with agent access. We tested the isolated condition directly, on the same frozen 119-task split and the same short-circuit. No isolated arm reads any test reference, calls a model or queries the org:

threat modelarmpasses
reference exposedread the dataset, echo the answer113/119 (0.950)
reference isolatedalways answer "None"22/119 (0.185)
reference isolatedper-category mode from the public B2C split14/119 (0.118)
reference isolatedper-category mode from B2B tasks outside the split39/119 (0.328)
reference isolatedgenuine sonnet-5, k = 80.588 of runs

Isolation does repair the read-the-key exploit, from 0.950 down to a floor of 0.328. We retract "hiding the dataset would not repair this checker."

Two weaker problems remain, and the revision states these instead. First, null references become the guessable string "None" (22 tasks). Second, the answer distribution is skewed enough that guessing the base rate beats the genuine agent in three categories. For example, on wrong_stage_rectification the base-rate guess passes 6 of 7 tasks, while the agent passes 2% of runs.

Exact matching is a valid rule for an answer task, as you say. The repair is a no-competence control reported per category. The certification procedure now refuses any claim whose threshold that control already clears (gate O5). The threat model is now explicit: the agent controls its returned string and every object its action space reaches, and the evaluator holds the reference.

Q3. What O2 claims. "No program can decide" overstates it, and we withdraw the phrase. O2 is insufficient specification: the declaration or reference does not fix a program check, so a judge is needed. Our classification is a stated rule (Appendix P3), not a computability claim. Judge dependence is its measured consequence, and the between-vendor alpha is that measurement.

On the image-referencing obligations: the text-only alpha of 0.357 on 13 obligations measures a panel that could not see what it judged. We now report it only to show that an agreement number needs its population named. It is not evidence about the benchmark. The vision-panel arm (8 judges, 3 vendors) is the measurement of those obligations. The failed validity audit bounds O2's validity half. We state that half as shown by example, not measured at scale.

Thank you again. The isolation and prior-shape corrections improve the paper directly.

If these responses address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.