acceptodds
Under review as a conference paper at ICLR 2027

Agentifying AI Evaluation: What to Evaluate, When to Stop, and How to Estimate

Abstract

Evaluating modern AI agents is increasingly costly and time-consuming, as benchmarking tasks can require long sequences of model calls and tool interactions. Can we reduce evaluation time and token cost without sacrificing model evaluation accuracy? Recent methods save cost either by selecting a subset of tasks to run or by stopping trajectories early based on partial execution histories, but generally trade lower cost for reduced evaluation accuracy. We introduce an agentic evaluation framework that jointly and adaptively learns what to evaluate and when to stop, with a tailored estimator that preserves evaluation accuracy. The system jointly learns what tasks to evaluate and when to stop execution, while probabilistically randomizing and adapting these decisions to enable statistical correction. By "learning what tasks to evaluate," it uses historical evaluation traces to identify which benchmark tasks are most informative for estimating agent performance. By "deciding when to stop," it actively monitors each trajectory and probabilistically continues only when further execution is expected to provide sufficient additional information about the eventual outcome. We pair this agentic system with a tailored estimator that corrects for adaptive task selection and stopping. Our theoretical analysis shows that the resulting framework preserves model evaluation accuracy while improving precision for a given evaluation budget. Experiments on benchmarking data show that our approach reduces both evaluation time and token cost while preserving accurate model evaluation results.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Agent evaluation, agent trajectories, optimal stopping, adaptive sampling, prediction-powered inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PgUjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ACES, an agentic evaluation framework designed to reduce the high computational cost of benchmarking AI agents on long-horizon tasks. ACES dynamically learns what tasks to evaluate and when to terminate trajectories early by utilizing historical traces and partial execution histories. The framework employs a closed-loop system that pairs a randomized, budget-constrained stopping policy with an inverse-probability-weighted estimator. This design aims to ensure unbiased performance estimation and valid uncertainty quantification while minimizing token expenditure. Core contributions include a theoretical characterization of conditions where early stopping reduces variance compared to task selection alone, alongside empirical evaluations on ALFWorld, ScienceWorld, and SWE-bench. Results indicate that ACES reduces evaluation costs substantially while preserving accurate performance estimates relative to uniform sampling and active querying.

Critical strengths
  1. The background of this study, which concerns the efficiency and cost of large model evaluation as well as the information gain obtained from the evaluations, is of great significance.
  2. A novel agentic evaluation framework, ACES, is proposed. Decisions about what to evaluate and when to stop are governed by a learned randomized policy under explicit evaluation-cost constraints.
  3. Experiments show that ACES reduces the cost required to reach a fixed estimation accuracy by 36% on ALFWorld and 26.7% on ScienceWorld, with consistent evidence across three agent benchmarks.
Critical weaknesses
  1. Only four benchmarks were evaluated, which limits the reliability of the results. I recommend including additional benchmarks that target diverse capability dimensions, such as coding, long-context understanding, and instruction following.
  2. In the SWE-bench study, the cost proxy simplifies by dividing character counts uniformly by four (Sec. C.1), an assumption that may not map accurately to the asymmetric pricing of prompt versus completion tokens in modern LLM APIs, negatively impacting real-world generalizability.
  3. Assumption 1 presumes that trajectories and costs are identically distributed across tasks (Sec. 4), which is technically flawed for highly heterogeneous software engineering benchmarks like SWE-bench where task difficulty and sequence lengths vary dramatically.
  4. The methodology heavily relies on possessing a substantial dataset of completed historical runs to train the initial representation (Sec. 3.1), meaning the framework suffers from a cold-start problem when applied to entirely novel, un-benchmarked domains.
  5. The analysis assumes that the "initial task information" (Sec. 3.1) is robust enough to form a meaningful starting probability, but the SWE-bench results reveal that initial selection provides the bulk of the efficiency gains (Sec. C.1), suggesting the method's efficacy degrades severely if initial task descriptors are sparse.
Questions for authors
  1. Please provide a quantitative ablation detailing the total compute overhead (in FLOPs or GPU-hours) required to train the models and MLPs at each batch step, proving that this overhead does not offset the trajectory execution savings.
  2. How do different agent models affect the results? Need to conduct ablation experiments.
1 comment

We thank the reviewer for emphasizing the presentation, literature coverage, and empirical positioning of the paper.

  1. Additional benchmark studies: We provide nine additional benchmark studies in our response to Reviewer 4J7o. Per your suggestion, the added benchmarks covering the following domains: interactive coding, customer-service dialogue and policy following, multi-application instruction following, and research-paper implementation.

  2. Clarification on the SWE-bench cost definition: We use character counts divided by four as a standardized token-cost proxy to ensure consistent and reproducible cost comparisons across recorded agent trajectories, without relying on model-specific tokenizers or API pricing. Importantly, since all costs are normalized by the full evaluation cost, the constant conversion factor cancels out and does not affect the reported relative cost reductions. We would like to clarify that adopting this token cost proxy does not limit the generalizability of our proposed framework. Our methodology accommodates arbitrary stage-specific execution costs Ci,sC_{i,s}, which can incorporate actual input/output token pricing, tool usage, or execution time. Therefore, the proposed design can be directly adapted to different API pricing structures without modifying the statistical framework or its theoretical guarantees.

  3. Justification of Assumption 1: The variation in task difficulty and execution length described by the reviewer is compatible with Assumption 1. We model complete task runs as independent draws from a common superpopulation that includes the full range of benchmark tasks. The i.i.d. assumption applies to sampling from this superpopulation and allows execution behavior to vary across tasks. We will clarify this distinction in Section 4.

  4. Clarification on the methodology: Regarding how to cold-start our method on an entirely new benchmark, we would like to clarify that our framework does not require historical completed trajectories. As stated in Section 3.1, historical data are used to pretrain the initial representation when available, but are not a prerequisite for the proposed methodology. In fact, ACES explicitly begins with a randomly selected pilot batch evaluated to completion (Algorithm 1), which provides initial data for learning the outcome predictor and evaluation policy. Furthermore, the statistical validity of our estimator does not depend on the availability of historical data or the accuracy of the initial predictions.

  5. Compute time study: Thank you for your suggestion. We have provided a compute time study below. The compute time table suggests that ACES fits the evaluator once before evaluation and keeps its parameters fixed throughout the held-out batch, so each trajectory step incurs zero additional training FLOPs. One-time fitting required 0.00206 CPU-hours on ALFWorld, 0.00839 on ScienceWorld, and 0.07082 on SWE-bench. All three fits used 0 GPU-hours.

Table 1: One-time compute required to fit the ACES evaluator.

BenchmarkCPU-hGPU-hDense MLP (TFLOPs)Elapsed fit time (s)
ALFWorld (semi-synthetic)0.002060—6.74
ScienceWorld0.008390—24.40
SWE-bench0.0708202.173139.01
  1. Performance of different agent models: Per your suggestion, we have provided additional results on the effect of different agent models on the RMSE and cost reduction in the table below on Intercode benchmark:

Table 2: Effect of the evaluated agent model on ACES in InterCode.

BaselineBudgetSQL, handicap / Chat-BisonSQL, handicap / Text-BisonSQL, standard / StarChatBash, fs_1 / StarChatBash, fs_1 / Text-Bison
A. RMSE reduction (%)
vs. No early stopping20%29.6841.6466.1265.2260.52
40%39.4154.1387.2196.9595.93
60%43.0248.7891.4999.0187.72
vs. Random subsampling20%33.5945.7070.3458.8858.18
40%42.8057.3489.2396.2895.67
60%46.1652.3693.2498.6485.91
vs. FAQ20%36.5447.5570.9762.1660.00
40%52.6164.2390.8397.0396.42
60%63.5067.2995.2999.1190.47
B. Evaluation cost reduction with ACES at 2 pp RMSE (%)
vs. No early stopping37.2351.4368.3968.7364.23
vs. Random subsampling39.3153.6573.9967.9263.87
vs. FAQNot reachedNot reached82.67Not reachedNot reached
Reviewer_eDgPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ACES combines randomized task selection and trajectory stopping with inverse-probability correction to estimate agent success rates. It gives conditional statistical guarantees and evaluates execution-cost savings on recorded trajectories. The central claim is an adaptive evaluator that learns allocation policies while preserving reliable inference. However, the highlighted ALFWorld result changes the available information, the experiments do not establish the cost of the complete adaptive procedure, and the statistical target differs between the theory and evaluation. These are substantial gaps in the evidence for the claimed contribution.

Critical strengths
  1. I find the no-stopping comparison useful. Exps evaluates ScienceWorld without generated covariates or accelerated information. Comparing ACES with task selection alone helps distinguish the added benefit of stopping from the overall saving relative to full evaluation. This is more informative than reporting only the latter percentage.
  2. The correction mechanism is explicit and interpretable.
Critical weaknesses
  1. The main ALFWorld saving relies on information unavailable in the original online setting. Section 5.1 generates an initial feature from the terminal outcome, and supplies at stage t a prediction computed from the original prefix. The highlighted 36% saving therefore combines artificial outcome information with observations that would not yet be available during execution. These transformations are disclosed and can be used in a controlled simulation. However, they leave the main practical claim untested under the information constraints. IMO, that missing experiment is central to evaluating the paper's contribution.
  2. The mathematical development does not establish the contribution of the implemented evaluator. Sections 3–4 formalize allocation, sequential correction, and variance. Equation (5)'s unbias follows from inverse-probability weighting of telescoping prediction increments, as the paper explains. Proposition 2 gives a potentially informative condition for an oracle stopping policy with a fixed predictor, but says little about how closely a policy learned from limited target-agent data approaches that result. I am not convinced that the paper adequately connects this analysis to the method it proposes to run.
  3. The end-to-end cost benefit remains unclear. The initial full-run pilot consumes budget in sec 3, but the main studies do not provide a reproducible accounting of pilot acquisition, predictor fitting, or online policy updates? The detailed supplementary replay also excludes pilot and prediction overhead. Costs are measured through character-based token proxies and action counts, without wall-clock measurements. In the ScienceWorld study, it requires approximately 73 of full-evaluation cost, compared with 78 without stopping and 80 for random. Much of the headline 27 saving is therefore already obtained by the controls.
  4. Presentation and writing: the paper does not clearly close the gap between its claims and demonstrated findings. It devotes substantial space to the allocation-and-estimation formalism, then moving directly to the statement parts and references. There is no conclusion section. More, the final discussion does not bring together the limits of the semi-synthetic experiment, the conditional theory, and the cost measurements. I find it difficult to identify a precise practical takeaway that the full body of evidence supports.
Questions for authors
  1. What evidence supports the ALFWorld savings when both the outcome-generated feature and the accelerated information schedule are removed?
  2. Which experiment evaluates the actual batch-adaptive procedure, including newly acquired target-agent pilot data and prediction/update costs, against the strongest matched-precision control?
  3. What is the estimand and variance formula in each coverage experiment, and which additional result establishes the claimed theoretical contribution beyond the correction construction?
1 comment

We thank the reviewer for the valuable feedback.

1. Motivation and additional results on the ALFWorld study:

  • Our goal in this experiment is to investigate under what conditions adaptive task selection and early stopping can provide substantial efficiency gains. Specifically, we introduce a generated initial covariate and modify the timing of predictive information to create a controlled setting in which initial information is informative and trajectory-level information becomes available sufficiently early. This allows us to isolate the two sources of efficiency gains and investigate how they affect the cost-efficiency of ACES. We additionally evaluate ACES on ALFWorld without the generated initial covariate or any modification to the timing of information, with results shown in the table below. We observe that the additional efficiency gain from early stopping is relatively modest in this setting.

    Table 1. ALFWorld benchmark without the synthetic covariate.

    ComparisonReduction (%)
    20% cost0.49
    40% cost0.81
    60% cost3.39
    80% cost16.99
    Cost at 1 pp RMSE3.89
  • The real ALFWorld benchmark results align with Proposition 2: early stopping helps when partial trajectories are informative and substantial execution cost remains. Original ALFWorld trajectories appear to provide limited early predictive information, explaining the modest incremental gain.

2. Contributions and learned policy:

We thank the reviewer for this comment, but respectfully disagree that our theoretical developments are disconnected from the proposed methodology. We address this comment in three points:

  • First, while IPW is well established, our theoretical contributions go beyond this standard argument. Proposition 1 establishes unbiasedness and asymptotically valid inference under adaptively learned policies. Equation (6) derives the variance decomposition that directly motivates the policy optimization in Equation (2). Proposition 2 provides a necessary and sufficient condition under which early stopping improves efficiency beyond optimal task selection.

  • Second, Proposition 2 provides practical guidance on when early stopping is beneficial, rather than finite-sample guarantees for learned policies. We have already demonstrated this insight empirically in Section 5.2 (Figure 4(a)–(b)), showing that earlier predictive information leads to greater efficiency gains.

  • Third, oracle characterization and convergence of learned policies are different questions, as is standard in statistical policy learning (Kitagawa and Tetenov, 2018; Athey and Wager, 2021). To further bridge theory and implementation, our additional experiments show differences in mean continuation probabilities below 0.05 between learned and oracle policies across evaluation budgets.

3. End-to-end cost:

We provide an end-to-end cost table and a cost breakdown table to show the savings from task selection and early stopping across benchmarks.

Table 2. ScienceWorld cost breakdown at a target RMSE.

ComponentTokens (Million)Wall-clock (s)
Full pilot7.4540.002
Pilot predictions01.828
Predictor fitting018.706
Held-out predictions05.560
Held-out replay7.3440.027
Evaluation with fitted models7.3445.587
ACES first use, including pilot14.79826.123
Full held-out evaluation9.8730.00024

Table 3. Breakdown of ACES evaluation cost savings from selection and early stopping.

Benchmark / agentTask selection (%)Early stopping (additional, %)Total savings (%)
ScienceWorld21.434.1925.62
InterCode SQL63.5424.9488.47
AppWorld5.9768.8074.77
Paper2Env13.9934.8048.78

4. Batch-adaptive experiment:

We provide additional results on adaptive batch updates. Using the first batch as the baseline, we show that further adaptive batch updates reduce RMSE and costs.

Table 4. RMSE and cost reduction on the ALFWorld benchmark through batch updates. The reductions are with respect to the first batch run.

BatchesRMSE reduction (%)Cost reduction (%)
100
22.81.0
34.01.4
44.62.6

5. Clarification of Appendix Tables 1–3:

We use a fixed evaluation bank of recorded trajectories and take the full-bank average μbank=n−1∑i=1nYi\mu_{\text{bank}}=n^{-1}\sum_{i=1}^n Y_i as the ground-truth reference, allowing us to isolate the variability from randomized task selection and early stopping. The reported confidence intervals use the score-based variance estimator in Eq. (7): Var^(μ^n)=1n(n−1)∑i=1n(ψi−μ^n)2.\widehat{\mathrm{Var}}(\hat\mu_n)=\frac1{n(n-1)}\sum_{i=1}^n(\psi_i-\hat\mu_n)^2.

References

  • Kitagawa, T., & Tetenov, A. (2018). Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice. Econometrica, 86(2), 591–616.
  • Athey, S., & Wager, S. (2021). Policy Learning With Observational Data. Econometrica, 89(1), 133–161.
Reviewer_4J7oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a framework for cutting the cost of evaluating AI agents on long horizon benchmarks. A single learned policy decides both which tasks to run and when to stop a run early. Replaying recorded agent runs, the authors reach fixed error targets at 36% (ALFWorld) and 26.7% (ScienceWorld) lower cost than full evaluation, with near-nominal coverage. On SWE-bench, early stopping adds only marginal gains.

Critical strengths

The paper supports its theory with results. The paper contains proofs that expected accuracy is preserved under their method. The paper also gives an estimator for when the policy in the paper helps. The problem is very well-motivated. Evaluations are soon becoming a bottleneck in model development.

Critical weaknesses

Early stopping doesn't seem to save that much time/cost. Improvements over random subsampling are not significant.

Questions for authors

Please report the cost to reach the error target against random subsampling as well as against full evaluation. Do tables 1-3 refer to E[Y] or average over fixed subset of recorded runs? Can the authors also share wall-clock time improvement under similar conditions?

4 comments

We thank the reviewer for the insightful comments and suggestions.

We hope to clarify that the benefits of early stopping depend on whether early run trajectories are “informative” about task outcomes, and we have demonstrated this point both through multiple benchmarks and theoretical analysis.

  • Theoretically, Proposition 2 establishes a necessary and sufficient condition and demonstrates that early stopping is helpful as long as the remaining variance of the task outcome is sufficiently small relative to the expected remaining execution cost.
  • Empirically, in our submitted version, Section 5.2 systematically investigates how the timing of predictive information affects the benefits of early stopping. For example, in Section 5 (ALFWorld), to reach the same RMSE, ACES design further reduces 18% costs compared with random subset sampling and 15% costs compared with the "without early stopping" design. These reductions can translate into substantial monetary savings. At an average inference cost of \2.07perGPT−4otask.ThesavingsobservedinALFWorldwouldtranslateto2.07 per GPT-4o task. The savings observed in ALFWorld would translate to \\18,630 relative to random subsampling and \$15,525 relative to the design without early stopping, at the same estimation accuracy.
  • During this rebuttal, we have also expanded our empirical results to 9 additional benchmarks. See results below.

Moreover, we would like to note that the cost reductions achieved by ACES arise from both adaptive task selection and early stopping. Even when the additional savings from early stopping are modest, adaptive task selection can substantially reduce evaluation costs relative to random subsampling by allocating resources to more informative tasks. By jointly optimizing these two decisions, ACES achieves greater overall cost efficiency than either strategy alone, as demonstrated in our empirical studies. For example, ACES reduces costs by 70.34% and 66.12% versus random subset sampling and without early stopping design on Intercode benchmark, respectively.

Table 1: ACES versus random subsampling and no early stopping.

A. RMSE reduction (%) at matched expected cost

Benchmark / AgentCostvs. Random (%)vs. No Early Stopping (%)
InterCode SQL / StarChat20%70.3466.12
40%89.2387.21
60%93.2491.49
80%97.1395.96
2-bench / GPT-4o-mini20%62.6013.81
40%96.4285.44
60%95.0567.22
80%98.0383.20
AppWorld (challenge) / ReAct DeepSeekCoder20%85.0170.98
40%89.5977.30
60%95.1288.64
80%88.4671.98
AppWorld (normal) / ReAct DeepSeekCoder20%25.7045.38
40%66.8478.64
60%84.8691.27
80%76.0587.23
Paper2Env / Gemini 3 Flash Preview20%8.4712.34
40%20.6323.86
60%86.5287.10
80%97.8797.99
InterCode SQL (handicap) / Chat-Bison-00120%33.5929.68
40%42.8039.41
60%46.1643.02
80%36.6433.16
InterCode SQL (handicap) / Text-Bison-00120%45.7041.64
40%57.3454.13
60%52.3648.78
80%44.2940.16
InterCode Bash / StarChat20%58.8865.22
40%96.2896.95
60%98.6499.01
80%98.8599.33
InterCode Bash / Text-Bison-00120%58.1860.52
40%95.6795.93
60%85.9187.72
80%99.0199.25

B. Cost reduction (%) to reach 2 percentage points RMSE

Benchmark / Agentvs. Random (%)vs. No Early Stopping (%)
InterCode SQL / StarChat73.9968.39
2-bench / GPT-4o-mini69.8923.57
AppWorld (challenge) / ReAct DeepSeekCoder69.0324.95
AppWorld (normal) / ReAct DeepSeekCoder66.7973.17
Paper2Env / Gemini 3 Flash Preview39.3240.45
InterCode SQL (handicap) / Chat-Bison-00139.3137.23
InterCode SQL (handicap) / Text-Bison-00153.6551.43
InterCode Bash / StarChat67.9268.73
InterCode Bash / Text-Bison-00163.8764.23

Table 2: Number of tokens to reach target RMSE of 1 percentage point, measured in millions of tokens.

BenchmarkRandom Subsampling (Millions)Full Evaluation (Millions)ACES (Millions)
ALFWorld (semi-synthetic)0.7380.8970.591
ScienceWorld0.0990.1050.094
SWE-bench9.03918.1604.304

Table 3: Wall-clock time reduction (%) to reach target RMSE.

BenchmarkACES vs. Random Subsampling (%)ACES vs. Full Evaluation (%)
ALFWorld (semi-synthetic)334.75404.65
ScienceWorld536.71559.45
SWE-bench65.50217.46

In our previous response, we overlooked the second part of the reviewer's question. We appreciate the reviewer raising this important point and would like to provide the following clarification.

  • Clarification on the coverage definition in Appendix Tables 1 to 3: In our theoretical framework, the target estimand is μ=E[Y]\mu=\mathbb E[Y], the expected agent performance over the task distribution. In Appendix Tables 1-3, we instead use the full-bank average of recorded trajectories,μbank=1n∑i=1nYi,\mu_{\mathrm{bank}}=\frac{1}{n}\sum_{i=1}^n Y_i, as the ground-truth reference. This setting holds tasks and outcomes fixed, allowing us to isolate variability from randomized task selection and early stopping. Since all terminal outcomes are available, we can directly assess estimation accuracy and compare evaluation designs against the same reference. We repeatedly simulate the evaluation process and report the empirical coverage of nominal 95% confidence intervals for μbank\mu_{\mathrm{bank}}. We will clarify this distinction in the revision.

Thank you again for your comments and questions.

Reviewer_4J7oOpenReview ↗

Response to authors

I thank the authors for their sincere effort to answer my questions, and for the detailed analysis. While the expanded results are encouraging, several concerns remain that prevent me from accepting this paper. First, the 9 newly added benchmarks lack experimental details. Also, several rows in table 1B show better cost reductions against "No Early Stopping" than against "Random," which means that adaptive task selection alone underperforms random subsampling. Second, table 3 reports wall clock reduction time over 100% which is not meaningful.

Third, the gains on the unmodified benchmarks are very modest. The reported execution costs exclude predictor training and pilot/policy-development, which is substantial for benchmarks like SWEBench.

Thank you for your response. We are somewhat puzzled by your comments and would appreciate the opportunity to provide further clarification below:

First, regarding the experimental details, the implementation follows the same setup as the benchmarking results reported in the submitted paper and supplement. For the additional benchmarking results included in the rebuttal, we did not have sufficient space to provide further details. We also believe these details are not essential, as the additional experiments follow the same evaluation protocol already described in the submission.

As for your comment that random splitting performs better than task selection, this is due to random variability arising from the limited number of tasks in those benchmarks. Furthermore, even if random splitting performs comparably to task selection, these results demonstrate that our method can still reduce computational costs through early stopping. You specifically asked for empirical evidence of cost reduction from early stopping ("Early stopping doesn't seem to save that much time/cost. Improvements over random subsampling are not significant"), which is why we provided these additional results in this rebuttal. We are thus puzzled and hope to clarify that these additional experiments were intended to address your original concern about early stopping, rather than to establish the superiority of task selection over random splitting.

We would also like to note that our method can reduce costs through two mechanisms: task selection and early stopping. Sometimes they work together, and in other cases only one component contributes to cost reduction. Our method is not restricted to either strategy and can flexibly leverage both depending on the benchmark characteristics.

Second, we would like to clarify that a percentage increase can exceed 100%. With the formula we used, values above 100% mean our method saves more time than full-run baseline. For example, the 334.75% entry for ALFWorld (row 1 in Table 3) relative to full evaluation is calculated as

= 334.75\%.$$ **Third**, we respectfully disagree with the characterization that the gains on unmodified benchmarks are "very modest," as this is not consistent with the empirical evidence provided during our rebuttal as well as in our paper. For example, on SWE-bench, ACES reduces token consumption from 9.039M to 4.304M compared with random subsampling (52.4% reduction) and from 18.160M to 4.304M compared with full evaluation (76.3% reduction), while achieving the same target RMSE of 1 percentage point. Similarly, at a target RMSE of 2 percentage points, ACES reduces evaluation costs by 66.79% on AppWorld (normal) and 39.32% on Paper2Env relative to random subsampling. While the improvements naturally vary across benchmarks, we find it difficult to reconcile these substantial reductions with the claim that the gains are "very" modest. Regarding predictor training and pilot/policy-development costs, we kindly note this concern overlooks a fundamental difference in computational scale. Our predictor and policy development rely on conventional machine learning models, rather than AI agent rollouts involving repeated LLM inference, tool calls, and environment interactions. Training these models is computationally lightweight compared with executing agents on benchmarks such as SWE-bench, where the dominant costs arise from repeated LLM calls and software execution. While pilot data collection does involve agent execution, the associated pilot costs are already accounted for in all our empirical results. Please let us know if you have any questions. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank all reviewers for the constructive comments. While reviewers view our method as novel and theoretically sound, below we summarize the main points raised in the reviews and how our responses address them:

1. Additional benchmarking data and more evidence on cost reduction. Reviewers PgUj and 4J7o request results on broader benchmarks, so we added results for 9 additional benchmarks. All show cost savings of ACES over existing baselines (random subsampling and the no-early-stopping design). Representative cost reductions at the same RMSE are:

Benchmark / agentvs. Random subsamplingvs. No early stopping
InterCode SQL / StarChat73.99%68.39%
AppWorld (normal) / ReAct DeepSeekCoder66.79%73.17%
Paper2Env / Gemini 3 Flash Preview39.32%40.45%

These results show substantial savings beyond task selection alone. Table 2 of our response to Reviewer PgUj also addresses the question about different agent models, showing savings across three models.

2. Clarification on theoretical contributions and statistical validity. Reviewer eDgP questions how our theory supports the implemented evaluator and how the learned policy compares with the oracle. We clarify these connections in Point 2 of our response to Reviewer eDgP and report absolute gaps below 5 percentage points between learned and oracle mean continuation probabilities across the tested ALFWorld budgets.

Reviewer 4J7o and Reviewer eDgP ask about the coverage estimand, while Reviewer PgUj questions whether Assumption 1 allows heterogeneous tasks. We clarify the coverage estimand and variance formula in Point 5 of our response to Reviewer eDgP and explain how Assumption 1 accommodates heterogeneous tasks in Point 3 of our response to Reviewer PgUj.

3. More detailed breakdown on implementation cost reduction and cold start. Reviewer eDgP and Reviewer PgUj ask about the cost of running and fitting ACES, and Reviewer 4J7o requests wall-clock comparisons. We provide cost breakdowns in Table 2 of our response to Reviewer eDgP and Table 1 of our response to Reviewer PgUj, with wall-clock comparisons in Table 3 of our response to Reviewer 4J7o.

Reviewer PgUj also questions the cost proxy and reliance on historical data. We clarify how the framework accommodates actual API pricing in Point 2 of our response to Reviewer PgUj and explain how ACES starts from a pilot batch without historical trajectories in Point 4.

We have provided more detailed responses to each reviewer during the rebuttal period. We sincerely look forward to hearing their feedback.

Authors of ACES

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.