- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies experiment selection under a compute budget for research agents. CAES combines an online compute price with a per-candidate decision rule that chooses between skipping, completing, and partially inspecting an experiment. The price adapts to budget consumption, while inspection is valued by its potential to change the subsequent execution decision. The paper evaluates the approach on TabRepo, LCBench, and recorded AIRS-Bench agent trajectories, and introduces a Learn-then-Test procedure for certifying budget tiers relative to running the full candidate stream.
Critical strengths
None. Although efficient allocation of research compute is an important problem, I did not identify a sufficiently substantiated methodological or empirical strength that supports acceptance in the current version.
Critical weaknesses
- The evaluation does not adequately establish effectiveness in the adaptive research setting motivating the paper. On AIRS-Bench, the method selects among previously recorded experiments rather than controlling a live agent whose subsequent proposals respond to its execution decisions. This distinction is consequential: skipping an experiment changes the information and code state available for generating later experiments. The main replay retains descendants of skipped nodes and transfers their recorded improvement over the original parent to the policy’s current incumbent. This assumes that improvements remain meaningful across potentially different research paths, which is not established. The alternative conventions in Appendix A provide a useful sensitivity check, but none directly observes the counterfactual trajectory generated by the agent. And, the decision rule also depends on reliable performance predictions and informative partial executions. Appendix d describes several estimation choices, but their calibration and robustness across heterogeneous research tasks remain insufficiently demonstrated. A shared predictor controls one source of variation between methods. Taken together, the experiments support a selection policy over recorded streams more directly than an effective decision-making system for adaptive autonomous research.
- The validity and interpretation of the risk-certification results require further justification. Sec. 2.5 argues that the parameter-free quota schedule permits using the entire available task pool for certification. However, Appendix d states that failure probabilities, conditional execution success, and inspection noise are estimated using recordings from other tasks. A pricing schedule without tuned hyperparameters does not make the complete decision procedure independent of the certification data. The paper does not clearly establish how predictor fitting, policy selection, calibration, and final evaluation are separated. If calibration tasks also influence the fitted components, or if dependent leave-one-task-out losses are pooled, the standard Hoeffding–Bentkus argument does not automatically apply. A precise data-use protocol and a justification of p-value validity are needed. The certification comparison is also strongly affected by sample allocation. At the reported risk threshold of 0.2 and testing level of 0.1, the standard HB p-value with zero observed violations is approximately 0.055 for 13 calibration tasks, but approximately 0.107 for 10 tasks. Thus, a baseline left with at most 10 calibration tasks cannot pass even with zero violations. This supports a potential calibration-efficiency advantage of avoiding parameter selection, but does not by itself establish superior policy quality. An equal-calibration-sample comparison with independently fixed baseline parameters would help distinguish these effects.
Questions for authors
plese refert to Critical Weaknesses.
4 comments
Response to Reviewer cnSV (Part 2/2)
Q2: Learn Then Test (LTT) validity and equal-sample comparison
"The validity and interpretation of the risk-certification results require further justification... Appendix d states that failure probabilities, conditional execution success, and inspection noise are estimated using recordings from other tasks. A pricing schedule without tuned hyperparameters does not make the complete decision procedure independent of the certification data... An equal-calibration-sample comparison with independently fixed baseline parameters would help distinguish these effects."
A2: The cross-task estimates in Appendix D are used only for the main replay comparison. For LTT, we fix the policy’s initial settings and update rules without using the calibration tasks, then update its state from each task’s own observations. LTT tests how much compute can be saved while keeping the probability of excessive performance loss below a specified threshold. Table 6 lists the predictor, initial constants, update rule, and quota schedule used for LTT. The reported certificate applies to the gap under the replay protocol; the live closed-loop results are empirical evaluations and are not covered by that certificate.
Independence from calibration tasks. The complete policy used for LTT includes the zero-shot LLM raw-score predictor (), prior constants (, , ), within-task online updates (Equation 7), and the quota schedule (Equation 6). These components are fixed independently of the calibration tasks, separately from the leave-one-task-out (LOTO) tuned performance comparisons in Table 1. This task-local variant yields replay performance comparable to Table 1, as detailed in Appendix H.
With the update rule fixed in advance, each task's trajectory and loss depend exclusively on executions and inspections completed within task itself. Under independent task sampling, the binary calibration losses within each fold are mutually independent, satisfying the finite-sample condition of the Hoeffding–Bentkus (HB) -value (Angelopoulos et al. [1]; with 0/13 violations giving ). Each 13-task calibration fold is tested separately in the fixed descending budget sequence; overlapping LOTO folds are not pooled as independent observations, and the held-out task is used only for evaluation.
Comparison at equal calibration sample size. To separate sample-allocation efficiency ( vs. , where even with zero violations) from policy quality, we evaluated EI per cost, which selects by expected improvement per unit cost, using a pre-specified task-local threshold rule (setting each task's threshold to its own 3-candidate warm-start median EI/cost, consuming zero calibration tasks) and CArBO, a cost-aware Bayesian optimization method, using the published default decay parameters of Lee et al. [2] so that every method is tested on the exact same independent calibration tasks at . In each 13-task calibration fold, the fixed descending test sequence certifies the tightest budget tier with ; the table below reports the conservative (worst-case across the 14 calibration folds, which also equals the mode across folds) certified budget tier and corresponding compute savings:
| Method Family (Appendix H, Table 7) | Parameter Specification (Zero Calibration-Task Tuning) | Calib. Tasks | Certified Budget Tier () | Certified Compute Savings |
|---|---|---|---|---|
| CAES (strictly task-local, ours) | No calibration tuning (Eq. 6 + fixed task-local constants ) | 13 | 0.618 | 38.2% |
| EI per cost (pre-specified task-local rule) | Task-local 3-run warm-start median EI/cost | 13 | 0.842 | 15.8% |
| CArBO (literature default) | Literature default [2] | 13 | 0.865 | 13.5% |
CAES more than doubles the certified compute savings of both comparison methods (38.2% vs. 15.8% and 13.5%). With the same 13 calibration tasks and independently specified policy rules, CAES certifies a smaller budget than either comparison method. This shows that CAES saves more compute under the same risk requirement even when all methods have the same number of calibration tasks. The protocol and comparison are detailed in Section 2.5, Section 4, Appendix D, and Appendix H, Tables 6–7.
References
[1] Anastasios N. Angelopoulos et al. Learn Then Test: Calibrating Predictive Algorithms to Achieve Risk Control. The Annals of Applied Statistics, 19(2):1641–1662, 2025.
[2] Eric Hans Lee, Valerio Perrone, Cedric Archambeau, and Matthias Seeger. Cost-aware Bayesian Optimization. arXiv preprint arXiv:2003.10870, 2020.
Thank you for the detailed responses and additional experiments. I will take them into account in my final assessment.
Thank you for your feedback and updated assessment
Thank you very much for taking the time to review our responses and for updating your assessment! Your questions have strengthened both our empirical analysis and the revised manuscript.
Response to Reviewer cnSV (Part 1/2)
We thank the reviewer for the comments. We provide detailed responses below.
Q1: Live evaluation and predictor reliability
A1: The original replay experiments hold candidate sequences and execution costs fixed to test whether CAES allocates compute more effectively across the same experimental opportunities. The original results support this core claim. Your suggestion for closed-loop evaluation further tests whether this advantage holds when execution decisions change subsequent candidates. We thank you for this suggestion and added live closed-loop experiments on six tasks, with each policy generating subsequent candidates from its own executed code and results.
Consistent with the original results, CAES outperforms the comparison methods at both 10% and 20% budgets. Its mean normalized performance gaps are .154 and .118, compared with .191 and .156 for the strongest comparison method; CAES wins 5 of 6 tasks against that method at each budget. The gap measures the shortfall relative to an unbudgeted reference run; lower is better. The setup and comparisons are in the General Response's live closed-loop evaluation section and in Section 4, Appendix J, Table 10.
Regarding your concern about re-basing, replay itself cannot verify whether a recorded improvement would recur on a different code state. Appendix A tests how alternative replay conventions affect the results. The added live experiments directly execute the candidates generated by each policy and do not transfer recorded improvements between code states.
Predictor calibration and robustness. Across all 1,364 nodes on the 14 AIRS-Bench tasks, within-task online affine residual calibration (Equation 7, Table 8) achieves 93.8% ± 3.1% empirical coverage at the nominal 95% Gaussian level (range 88.2%–97.4%), 0.48 ± 0.14 within-sibling Spearman rank correlation, and 0.79 ± 0.11 Pearson correlation between 1/5-data inspection readings and full executions (positive on all 14 tasks, range 0.58–0.93). Together, these three metrics check whether the uncertainty intervals are well-calibrated, whether the predictor can rank competing sibling proposals, and whether partial runs are informative about full-run scores (Section 3.3, Appendix I, Table 8).
CAES also adjusts how often it inspects candidates based on the reliability of the prior and the inspection signal: when the prior is uninformative, a partial run helps decide whether the full run is worth paying for, so CAES inspects more often; when partial runs are noisier, they provide less information about the final score, so CAES inspects less often. At the 20% budget, flat priors (μi=bi, inspection rate rising from 17.8% to 31.4%) and tripled inspection noise (3si, inspection rate dropping to 4.2%) move the gap from .127 to .146 and .135. Under synthetic rank-inverting partial-run observations (corr(zi,Yi)=−0.5), an online correlation guard (ρ^z,y≤0.2) reduces the gap from .189 to .139 (Section 6, Appendix I, Table 9).