- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an “evaluability” framework for LLM agents, identifying five obstacles: nondeterminism, judgement, task grouping, capability coverage, and shortcuts. Experiments across three benchmark domains suggest that task selection typically contributes more uncertainty than repeated execution, motivating an explicit “undecided” outcome when evidence is insufficient.
Critical strengths
- Relevant problem and integrative framework. The paper shifts attention from benchmark scores to the claims those scores support, connecting statistical uncertainty, evaluator validity, and interface permissions. Its novelty lies primarily in conceptual synthesis and empirical diagnosis.
- Cross-domain support for the central finding. Task-level uncertainty exceeds rerun uncertainty across different evaluation mechanisms. Controlled experiments investigating nondeterminism and exploitability further strengthen the evidence.
Critical weaknesses
- The formal definition and empirical measures are not fully aligned. Settleability measures whether success probabilities approach zero or one, which differs from whether a capability claim can be reliably tested. The evidence supports an evaluation diagnostic framework more strongly than a unified certification framework.
- The central statistical conclusions depend on the estimand. Performance on a fixed benchmark and generalization to a task population require different interpretations of uncertainty. The benefits of adding tasks also depend on their representativeness, dependence structure, and acquisition costs.
Questions for authors
- Could the authors provide a complete certification example specifying the capability claim, error probability, and accept/reject/undecided decision rules, including which components carry statistical guarantees?
- Does the advantage of adding tasks over reruns persist after distinguishing fixed-suite evaluation from population-level inference and accounting for template dependence and task acquisition costs?
2 comments
Thank you for your response; my concerns have been fully addressed. However, I believe the paper itself still has significant room for improvement—for instance, regarding the poor quality of the presentation, the rudimentary charts and figures, and the pacing of the narrative. These issues cannot be resolved through the rebuttal discussion alone; in my view, the authors need to undertake further revisions to address them.
Complete certification example and when the task advantage holds
Thank you for the questions in the reviews.
W1 / Q1. Definition versus measures, and a complete example. Agreed: settleability is not testability. In the revision, settleability is a per-task diagnostic of how much rerun information is left. The certificate is a separate procedure (general response, point 1). Its parts carry guarantees as follows:
Here is a complete example. Claim: "haiku-4-5 passes at least 50% of retail customer-service tasks," at α = 0.05, with the τ-bench retail tools as the action space.
The same traces accept "at least 50% of this suite," with bound [0.539, 0.604]. The table of 14 claims is in our response to sNcn.
W2 / Q2. Does the task advantage survive? We give three separate answers.
Fixed suite versus population. For a fixed-suite claim, the task axis is zero by definition, and reruns are the only lever. For population claims, the task component is 3.3 to 9.1 times the rerun component under every design we tested. The designs are tasks i.i.d., finite-population correction, families fixed (stratified) and families i.i.d. The table is in our response to MVZJ.
Template dependence. We fit a three-level model (template, task within template, run) on VisualWebArena's 32 intent templates. The standard error of sonnet-5's score, in points, is:
New tasks from existing templates still beat reruns, but they reach their own floor: 3.0 points, against a rerun floor of 5.0. The best spend is new templates. On CRMArena-Pro (17 categories) the effect is stronger: 7.45 now, 7.42 with 2× reruns, 7.17 with 2× tasks, and 5.27 with 2× categories.
Acquisition cost. Suppose a new task costs c runs. Then the optimal number of reruns per task is k* = sqrt(c · E[p(1 − p)] / Var_between(p)), Cochran's two-stage optimum. Reruns beyond one per task pay only when a new task costs more than 8 to 15 runs on VisualWebArena and CRMArena-Pro, or 3 to 4 runs on τ-bench. Our k = 8 is optimal only if a task costs 131 to 241 runs (56 to 59 on τ-bench). The revision therefore states the result as a cost condition and adds this table.
If these responses address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any further questions.