- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ACES, an agentic evaluation framework designed to reduce the high computational cost of benchmarking AI agents on long-horizon tasks. ACES dynamically learns what tasks to evaluate and when to terminate trajectories early by utilizing historical traces and partial execution histories. The framework employs a closed-loop system that pairs a randomized, budget-constrained stopping policy with an inverse-probability-weighted estimator. This design aims to ensure unbiased performance estimation and valid uncertainty quantification while minimizing token expenditure. Core contributions include a theoretical characterization of conditions where early stopping reduces variance compared to task selection alone, alongside empirical evaluations on ALFWorld, ScienceWorld, and SWE-bench. Results indicate that ACES reduces evaluation costs substantially while preserving accurate performance estimates relative to uniform sampling and active querying.
Critical strengths
- The background of this study, which concerns the efficiency and cost of large model evaluation as well as the information gain obtained from the evaluations, is of great significance.
- A novel agentic evaluation framework, ACES, is proposed. Decisions about what to evaluate and when to stop are governed by a learned randomized policy under explicit evaluation-cost constraints.
- Experiments show that ACES reduces the cost required to reach a fixed estimation accuracy by 36% on ALFWorld and 26.7% on ScienceWorld, with consistent evidence across three agent benchmarks.
Critical weaknesses
- Only four benchmarks were evaluated, which limits the reliability of the results. I recommend including additional benchmarks that target diverse capability dimensions, such as coding, long-context understanding, and instruction following.
- In the SWE-bench study, the cost proxy simplifies by dividing character counts uniformly by four (Sec. C.1), an assumption that may not map accurately to the asymmetric pricing of prompt versus completion tokens in modern LLM APIs, negatively impacting real-world generalizability.
- Assumption 1 presumes that trajectories and costs are identically distributed across tasks (Sec. 4), which is technically flawed for highly heterogeneous software engineering benchmarks like SWE-bench where task difficulty and sequence lengths vary dramatically.
- The methodology heavily relies on possessing a substantial dataset of completed historical runs to train the initial representation (Sec. 3.1), meaning the framework suffers from a cold-start problem when applied to entirely novel, un-benchmarked domains.
- The analysis assumes that the "initial task information" (Sec. 3.1) is robust enough to form a meaningful starting probability, but the SWE-bench results reveal that initial selection provides the bulk of the efficiency gains (Sec. C.1), suggesting the method's efficacy degrades severely if initial task descriptors are sparse.
Questions for authors
- Please provide a quantitative ablation detailing the total compute overhead (in FLOPs or GPU-hours) required to train the models and MLPs at each batch step, proving that this overhead does not offset the trajectory execution savings.
- How do different agent models affect the results? Need to conduct ablation experiments.
We thank the reviewer for emphasizing the presentation, literature coverage, and empirical positioning of the paper.
Additional benchmark studies: We provide nine additional benchmark studies in our response to Reviewer 4J7o. Per your suggestion, the added benchmarks covering the following domains: interactive coding, customer-service dialogue and policy following, multi-application instruction following, and research-paper implementation.
Clarification on the SWE-bench cost definition: We use character counts divided by four as a standardized token-cost proxy to ensure consistent and reproducible cost comparisons across recorded agent trajectories, without relying on model-specific tokenizers or API pricing. Importantly, since all costs are normalized by the full evaluation cost, the constant conversion factor cancels out and does not affect the reported relative cost reductions. We would like to clarify that adopting this token cost proxy does not limit the generalizability of our proposed framework. Our methodology accommodates arbitrary stage-specific execution costs Ci,s, which can incorporate actual input/output token pricing, tool usage, or execution time. Therefore, the proposed design can be directly adapted to different API pricing structures without modifying the statistical framework or its theoretical guarantees.
Justification of Assumption 1: The variation in task difficulty and execution length described by the reviewer is compatible with Assumption 1. We model complete task runs as independent draws from a common superpopulation that includes the full range of benchmark tasks. The i.i.d. assumption applies to sampling from this superpopulation and allows execution behavior to vary across tasks. We will clarify this distinction in Section 4.
Clarification on the methodology: Regarding how to cold-start our method on an entirely new benchmark, we would like to clarify that our framework does not require historical completed trajectories. As stated in Section 3.1, historical data are used to pretrain the initial representation when available, but are not a prerequisite for the proposed methodology. In fact, ACES explicitly begins with a randomly selected pilot batch evaluated to completion (Algorithm 1), which provides initial data for learning the outcome predictor and evaluation policy. Furthermore, the statistical validity of our estimator does not depend on the availability of historical data or the accuracy of the initial predictions.
Compute time study: Thank you for your suggestion. We have provided a compute time study below. The compute time table suggests that ACES fits the evaluator once before evaluation and keeps its parameters fixed throughout the held-out batch, so each trajectory step incurs zero additional training FLOPs. One-time fitting required 0.00206 CPU-hours on ALFWorld, 0.00839 on ScienceWorld, and 0.07082 on SWE-bench. All three fits used 0 GPU-hours.
Table 1: One-time compute required to fit the ACES evaluator.
Table 2: Effect of the evaluated agent model on ACES in InterCode.