acceptodds
Under review as a conference paper at ICLR 2027

CAES: Cost-Aware Experiment Selection via Online Shadow Prices for Autonomous Research

Abstract

Research agents built on language models propose many experiments for each task, and running all of them consumes a large amount of compute. Recent methods study how to choose which experiments deserve compute. Cost-aware Bayesian optimization sets in advance the rate at which expected improvement is traded against compute, early stopping pays for a partial run of every candidate, and selectors built for research agents compare candidates only with one another. Each of these methods decides one experiment at a time, implicitly assuming that the decision does not affect the experiments that come after. In practice, compute given to one experiment is no longer available to later ones, whose value is still unknown when the current one is decided. To address this, we propose Cost-Aware Experiment Selection (CAES), which ties these decisions together through a single shadow price on compute. Given the shadow price, we frame the decision for each candidate as a Pandora’s box problem with a single box, whose exact solution chooses among skipping the candidate, running it in full, and first running a cheap partial version. The shadow price is updated online, rising when compute is spent faster than the search moves through the candidates and falling when it is spent more slowly. A parameter-free form of this update also admits a distribution-free certificate built with Learn-then-Test, which bounds the probability that the final result falls more than a chosen tolerance behind running everything. We evaluate CAES on TabRepo, LCBench, and AIRS-Bench. Given 5 to 40 percent of the compute needed to run everything, CAES outperforms ten baselines across the three benchmarks, ending closer to the result of running everything in 419 of 520 task-paired comparisons. These results show that cost-aware experiment selection with an online shadow price can guide research agents toward compute-efficient autonomous research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autonomous research agents, cost-aware experiment selection, shadow price, Pandora's box problem, compute efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cnSVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies experiment selection under a compute budget for research agents. CAES combines an online compute price with a per-candidate decision rule that chooses between skipping, completing, and partially inspecting an experiment. The price adapts to budget consumption, while inspection is valued by its potential to change the subsequent execution decision. The paper evaluates the approach on TabRepo, LCBench, and recorded AIRS-Bench agent trajectories, and introduces a Learn-then-Test procedure for certifying budget tiers relative to running the full candidate stream.

Critical strengths

None. Although efficient allocation of research compute is an important problem, I did not identify a sufficiently substantiated methodological or empirical strength that supports acceptance in the current version.

Critical weaknesses
  • The evaluation does not adequately establish effectiveness in the adaptive research setting motivating the paper. On AIRS-Bench, the method selects among previously recorded experiments rather than controlling a live agent whose subsequent proposals respond to its execution decisions. This distinction is consequential: skipping an experiment changes the information and code state available for generating later experiments. The main replay retains descendants of skipped nodes and transfers their recorded improvement over the original parent to the policy’s current incumbent. This assumes that improvements remain meaningful across potentially different research paths, which is not established. The alternative conventions in Appendix A provide a useful sensitivity check, but none directly observes the counterfactual trajectory generated by the agent. And, the decision rule also depends on reliable performance predictions and informative partial executions. Appendix d describes several estimation choices, but their calibration and robustness across heterogeneous research tasks remain insufficiently demonstrated. A shared predictor controls one source of variation between methods. Taken together, the experiments support a selection policy over recorded streams more directly than an effective decision-making system for adaptive autonomous research.
  • The validity and interpretation of the risk-certification results require further justification. Sec. 2.5 argues that the parameter-free quota schedule permits using the entire available task pool for certification. However, Appendix d states that failure probabilities, conditional execution success, and inspection noise are estimated using recordings from other tasks. A pricing schedule without tuned hyperparameters does not make the complete decision procedure independent of the certification data. The paper does not clearly establish how predictor fitting, policy selection, calibration, and final evaluation are separated. If calibration tasks also influence the fitted components, or if dependent leave-one-task-out losses are pooled, the standard Hoeffding–Bentkus argument does not automatically apply. A precise data-use protocol and a justification of p-value validity are needed. The certification comparison is also strongly affected by sample allocation. At the reported risk threshold of 0.2 and testing level of 0.1, the standard HB p-value with zero observed violations is approximately 0.055 for 13 calibration tasks, but approximately 0.107 for 10 tasks. Thus, a baseline left with at most 10 calibration tasks cannot pass even with zero violations. This supports a potential calibration-efficiency advantage of avoiding parameter selection, but does not by itself establish superior policy quality. An equal-calibration-sample comparison with independently fixed baseline parameters would help distinguish these effects.
Questions for authors

plese refert to Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cnSV (Part 1/2)

We thank the reviewer for the comments. We provide detailed responses below.

Q1: Live evaluation and predictor reliability

"The evaluation does not adequately establish effectiveness in the adaptive research setting motivating the paper. On AIRS-Bench, the method selects among previously recorded experiments rather than controlling a live agent whose subsequent proposals respond to its execution decisions... And, the decision rule also depends on reliable performance predictions and informative partial executions. Appendix d describes several estimation choices, but their calibration and robustness across heterogeneous research tasks remain insufficiently demonstrated."

A1: The original replay experiments hold candidate sequences and execution costs fixed to test whether CAES allocates compute more effectively across the same experimental opportunities. The original results support this core claim. Your suggestion for closed-loop evaluation further tests whether this advantage holds when execution decisions change subsequent candidates. We thank you for this suggestion and added live closed-loop experiments on six tasks, with each policy generating subsequent candidates from its own executed code and results.

Consistent with the original results, CAES outperforms the comparison methods at both 10% and 20% budgets. Its mean normalized performance gaps are .154 and .118, compared with .191 and .156 for the strongest comparison method; CAES wins 5 of 6 tasks against that method at each budget. The gap measures the shortfall relative to an unbudgeted reference run; lower is better. The setup and comparisons are in the General Response's live closed-loop evaluation section and in Section 4, Appendix J, Table 10.

Regarding your concern about re-basing, replay itself cannot verify whether a recorded improvement would recur on a different code state. Appendix A tests how alternative replay conventions affect the results. The added live experiments directly execute the candidates generated by each policy and do not transfer recorded improvements between code states.

Predictor calibration and robustness. Across all 1,364 nodes on the 14 AIRS-Bench tasks, within-task online affine residual calibration (Equation 7, Table 8) achieves 93.8% ± 3.1% empirical coverage at the nominal 95% Gaussian level (range 88.2%–97.4%), 0.48 ± 0.14 within-sibling Spearman rank correlation, and 0.79 ± 0.11 Pearson correlation between 1/5-data inspection readings and full executions (positive on all 14 tasks, range 0.58–0.93). Together, these three metrics check whether the uncertainty intervals are well-calibrated, whether the predictor can rank competing sibling proposals, and whether partial runs are informative about full-run scores (Section 3.3, Appendix I, Table 8).

CAES also adjusts how often it inspects candidates based on the reliability of the prior and the inspection signal: when the prior is uninformative, a partial run helps decide whether the full run is worth paying for, so CAES inspects more often; when partial runs are noisier, they provide less information about the final score, so CAES inspects less often. At the 20% budget, flat priors (μi=bi\mu_i = b_i, inspection rate rising from 17.8% to 31.4%) and tripled inspection noise (3si3s_i, inspection rate dropping to 4.2%) move the gap from .127 to .146 and .135. Under synthetic rank-inverting partial-run observations (corr⁡(zi,Yi)=−0.5\operatorname{corr}(z_i, Y_i) = -0.5), an online correlation guard (ρ^z,y≤0.2\hat{\rho}_{z,y} \le 0.2) reduces the gap from .189 to .139 (Section 6, Appendix I, Table 9).

AuthorsOpenReview ↗

Response to Reviewer cnSV (Part 2/2)

Q2: Learn Then Test (LTT) validity and equal-sample comparison

"The validity and interpretation of the risk-certification results require further justification... Appendix d states that failure probabilities, conditional execution success, and inspection noise are estimated using recordings from other tasks. A pricing schedule without tuned hyperparameters does not make the complete decision procedure independent of the certification data... An equal-calibration-sample comparison with independently fixed baseline parameters would help distinguish these effects."

A2: The cross-task estimates in Appendix D are used only for the main replay comparison. For LTT, we fix the policy’s initial settings and update rules without using the calibration tasks, then update its state from each task’s own observations. LTT tests how much compute can be saved while keeping the probability of excessive performance loss below a specified threshold. Table 6 lists the predictor, initial constants, update rule, and quota schedule used for LTT. The reported certificate applies to the gap under the replay protocol; the live closed-loop results are empirical evaluations and are not covered by that certificate.

Independence from calibration tasks. The complete policy used for LTT includes the zero-shot LLM raw-score predictor (y^iraw\hat{y}^{\mathrm{raw}}_i), prior constants (p0=0.10p_0 = 0.10, q0=0.90q_0 = 0.90, s0=0.10s_0 = 0.10), within-task online updates (Equation 7), and the quota schedule (Equation 6). These components are fixed independently of the calibration tasks, separately from the leave-one-task-out (LOTO) tuned performance comparisons in Table 1. This task-local variant yields replay performance comparable to Table 1, as detailed in Appendix H.

With the update rule fixed in advance, each task's trajectory and loss Lt(x)=1[gt(x)>ϵ]L_t(x) = \mathbb{1}[g_t(x) > \epsilon] depend exclusively on executions and inspections completed within task tt itself. Under independent task sampling, the n=13n = 13 binary calibration losses within each fold are mutually independent, satisfying the finite-sample condition of the Hoeffding–Bentkus (HB) pp-value (Angelopoulos et al. [1]; with 0/13 violations giving pHB=0.055≤0.10p_{\mathrm{HB}} = 0.055 \le 0.10). Each 13-task calibration fold is tested separately in the fixed descending budget sequence; overlapping LOTO folds are not pooled as independent observations, and the held-out task is used only for evaluation.

Comparison at equal calibration sample size. To separate sample-allocation efficiency (n=13n = 13 vs. n≤10n \le 10, where pHB=0.107>0.10p_{\mathrm{HB}} = 0.107 > 0.10 even with zero violations) from policy quality, we evaluated EI per cost, which selects by expected improvement per unit cost, using a pre-specified task-local threshold rule (setting each task's threshold to its own 3-candidate warm-start median EI/cost, consuming zero calibration tasks) and CArBO, a cost-aware Bayesian optimization method, using the published default decay parameters of Lee et al. [2] so that every method is tested on the exact same n=13n = 13 independent calibration tasks at (ϵ=0.10,δ=0.20,α=0.10)(\epsilon = 0.10, \delta = 0.20, \alpha = 0.10). In each 13-task calibration fold, the fixed descending test sequence certifies the tightest budget tier with pHB≤0.10p_{\mathrm{HB}} \le 0.10; the table below reports the conservative (worst-case across the 14 calibration folds, which also equals the mode across folds) certified budget tier x^\hat{x} and corresponding compute savings:

Method Family (Appendix H, Table 7)Parameter Specification (Zero Calibration-Task Tuning)Calib. Tasks nnCertified Budget Tier (x^\hat{x})Certified Compute Savings
CAES (strictly task-local, ours)No calibration tuning (Eq. 6 + fixed task-local constants p0,q0,s0p_0, q_0, s_0)130.61838.2%
EI per cost (pre-specified task-local rule)Task-local 3-run warm-start median EI/cost130.84215.8%
CArBO (literature default)Literature default [2]130.86513.5%

CAES more than doubles the certified compute savings of both comparison methods (38.2% vs. 15.8% and 13.5%). With the same 13 calibration tasks and independently specified policy rules, CAES certifies a smaller budget than either comparison method. This shows that CAES saves more compute under the same risk requirement even when all methods have the same number of calibration tasks. The protocol and comparison are detailed in Section 2.5, Section 4, Appendix D, and Appendix H, Tables 6–7.

References

[1] Anastasios N. Angelopoulos et al. Learn Then Test: Calibrating Predictive Algorithms to Achieve Risk Control. The Annals of Applied Statistics, 19(2):1641–1662, 2025.

[2] Eric Hans Lee, Valerio Perrone, Cedric Archambeau, and Matthias Seeger. Cost-aware Bayesian Optimization. arXiv preprint arXiv:2003.10870, 2020.

Thank you for the detailed responses and additional experiments. I will take them into account in my final assessment.

AuthorsOpenReview ↗

Thank you for your feedback and updated assessment

Thank you very much for taking the time to review our responses and for updating your assessment! Your questions have strengthened both our empirical analysis and the revised manuscript.

Reviewer_HdGkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to address the problem of experiment selection under compute constraints specifically in the context of machine learning engineering experiments. The authors seek to identify a mechanism that can dynamically adjust candidate selection based on the amount of compute remaining under a fixed compute regime. Existing methods for searching over the space of candidates make assumptions that don't factor in the cost of the experiment, the likelihood of success or the remaining compute budget. The authors argue that without accounting for all of these factors the agent might converge to suboptimal solutions.

Critical strengths

This paper is well scoped. The authors are clear about the constraint they want to optimize against, they come up with a solution to the problem and they demonstrate empirically that this solution is optimal in the setting that they explore.

Critical weaknesses

While this paper clearly shows that other approaches to experiment selection don't factor in all the components they need to make the right choices, it was unclear to me why this specific approach outlined in section 2 was the best way to factor that additional constraint into the selection. It is a sound way of solving the problem but the paper would benefit from explaining exactly why this method needs to be formulated in the way it is.

Questions for authors
  1. Does compute budget only count for the budget of the experiments or does it include the LLM itself?
  2. Did you consider providing the remaining compute (and some relevant context) to the LLM agent with instructions for what kinds of candidates to propose accordingly?
  3. What do the method numbers in Table 1 rever to?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HdGk (Part 1/2)

Thank you for your positive assessment and constructive suggestions. We provide detailed responses below.

Q1: Motivation for the two-layer formulation

"...it was unclear to me why this specific approach outlined in section 2 was the best way to factor that additional constraint into the selection. It is a sound way of solving the problem but the paper would benefit from explaining exactly why this method needs to be formulated in the way it is."

A1: Our two-layer formulation separates global budget pacing from single-candidate execution decisions. Spending on the current candidate leaves less compute for future candidates. Jointly solving the budgeted multi-box problem with partial inspection is NP-hard [1, 2]. We use an online compute price to pace spending across the stream. At that price, the inner rule solves the current candidate’s decision exactly: skip, inspect, or complete.

Completed executions update the best result found so far and the predictions for future candidates, while the cumulative budget constraint couples resource allocation across steps (Sections 2.2 and 2.4): ∑icost⁡(ai)≤B\sum_i \operatorname{cost}(a_i) \le B. Because the final best score telescopes into a sum of step-wise improvements over the incumbent, b1+∑i1[ai=complete](yi−bi)+b_1 + \sum_i \mathbb{1}[a_i = \mathrm{complete}](y_i - b_i)^+, introducing a Lagrange multiplier λ≥0\lambda \ge 0 on compute separates online budget pacing (Equations 5–6) from the single-candidate decision at the current state (Equations 2–4), giving each candidate ii a step-wise objective of expected improvement minus λ,cost⁡(ai)\lambda\\,\operatorname{cost}(a_i). Updating λi\lambda_i online carries the state of remaining compute into each step, raising the threshold when spending runs ahead of progress and lowering it when budget is plentiful.

At that price, a partial run is useful when its result can change whether the full run is worth its cost (Section 2.3). At a fixed shadow price, incumbent, and belief, the inner problem admits an exact single-candidate decision rule that compares the values of skipping, inspecting, and completing (Equations 2–4). In particular, Equation 4 values partial inspection by whether the reading ziz_i can change the subsequent execution decision—stopping after a poor reading while paying only fkif k_i.

The ablations in Section 4 test the contribution of each layer. For the baseline that compares expected improvement per unit cost against a fixed threshold (EI per cost), replacing that threshold with our outer price schedule improves performance in 18 vs. 3 TabRepo cells and 5 vs. 1 LCBench cells (Figure 4), while keeping the outer price and replacing Equation 4's option-valued inspection placement with uniform or random inspection at the same inspection spend loses at all four budgets (4–0 cells in Appendix C.2, Table 5). The two ablations show what each layer contributes: the outer price improves budget allocation across the stream, and the inner rule improves the choice of candidates to inspect (Sections 2.2–2.3).

Q2: Compute-budget accounting

"Does compute budget only count for the budget of the experiments or does it include the LLM itself?"

A2: In the main experiments (Table 1, Section 2.1), the compute budget BB includes the machine time of full experiment executions (kik_i) and partial inspections (fkif k_i), and excludes LLM candidate-generation and comparison overhead. We have clarified this budget accounting explicitly in Section 2.1.

AuthorsOpenReview ↗

Response to Reviewer HdGk (Part 2/2)

Q3: Budget-aware proposal generation

"Did you consider providing the remaining compute (and some relevant context) to the LLM agent with instructions for what kinds of candidates to propose accordingly?"

A3: In our original experiments, we kept the proposer fixed to isolate how different selection policies allocate compute across the same candidate stream. Giving the remaining budget to the LLM agent is a natural extension, as it lets the proposer adjust candidate complexity and runtime before selection even happens. Following your suggestion, we evaluated budget prompting both on its own and combined with CAES.

In this variant, called Budget-Prompted Agent, the candidate generator receives the remaining compute budget, elapsed candidate count, and recent execution runtimes in the aira-dojo operator prompt, with instructions to adapt model complexity and training epochs to the remaining budget. We tested both variants on the six live tasks at 10% and 20% budgets (Section 4, Appendix J, Table 10).

The results show that budget-aware generation and CAES complement each other well. For the strategy that executes candidates without preselection, adding budget information at 10% lowers the mean candidate cost from 244 s to 172 s and the normalized performance gap from .221 to .198. The gap measures the shortfall relative to an unbudgeted reference run; lower is better. At 20%, the corresponding cost falls from 241 s to 178 s and the gap from .184 to .169.

Combining budget prompting with CAES further lowers the gap from .154 to .143 at 10% and from .118 to .109 at 20%. The General Response collects the live comparison table and shared setup; Appendix J, Table 10 includes the full cost and spending results.

Q4: Interpreting Table 1

"What do the method numbers in Table 1 rever to?"

A4: The left part of Table 1 reports each method’s mean normalized performance gap to running all candidates at the given budget; lower values are better. The right part counts the pairwise wins and losses of CAES against each baseline.

  • Left twelve columns (5, 10, 20, 40 under TabRepo, LCBench, AIRS-Bench): The mean normalized optimality gap gt(x)∈[0,1]g_t(x) \in [0, 1] between the best validation score a method reaches within a budget of xx\\% (5%, 10%, 20%, 40%) of full stream execution cost and the best score achieved by running the entire unbudgeted stream (100%) on the same task (lower is better; TabRepo averages over its 117 binary, 94 multiclass, and 33 regression tasks, broken out in Appendix C.1, Table 4).

  • Right two columns (CAES wins-losses: point and boot.): Head-to-head comparisons of CAES against each row's baseline across 52 task-paired cells (36 TabRepo cells from 3 task groups ×\times 3 proposers ×\times 4 budgets, plus 12 LCBench cells from 3 proposers ×\times 4 budgets, plus 4 AIRS-Bench cells from 1 proposer ×\times 4 budgets). point counts wins and losses by mean gap: 43-9 vs. Run-all (execution without preselection) means CAES has a lower gap in 43 cells and a higher gap in 9. boot. counts statistically significant wins and losses, where the 95% task-paired bootstrap interval (2,000 resamples) excludes zero (30-1 vs. Run-all), and the bottom row (419-101 and 251-20) sums wins and losses across all 10 baselines (10×52=52010 \times 52 = 520 cells total).

We have clarified these definitions in Section 3.2 and the Table 1 caption.

References

[1] Ali Aouad, Jingwei Ji, and Yaron Shaposhnik. The Pandora’s Box Problem with Sequential Inspections. Operations Research, 74(4):2179–2195, 2026.

[2] Hu Fu, Jiawei Li, and Daogao Liu. Pandora Box Problem with Nonobligatory Inspection: Hardness and Approximation Scheme. STOC, pp. 789–802, 2023.

I would to thank the authors for their effort and for sufficiently answering my questions. I will take this into consideration as I select my final score.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for your encouraging feedback. We are glad to hear that our responses have addressed your questions and provided further clarification on the motivation and design of our approach. We sincerely appreciate your thoughtful comments and careful consideration.

Reviewer_vLu5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CAES (Cost-Aware Experiment Selection), an online decision framework for allocating constrained compute in autonomous research agents. CAES addresses the coupling of sequential decisions by introducing a dynamic shadow price on compute (via dual mirror descent or a parameter-free quota schedule). Conditioned on this price, each candidate is formulated as a single-box Pandora's box problem with optional partial inspection, yielding an exact threshold policy among skipping, full execution, and low-cost inspection. Additionally, it provides a distribution-free budget certificate via Learn-then-Test. Empirical evaluations across TabRepo, LCBench, and recorded AIRS-Bench runs show that CAES approaches unbudgeted results using 5-40% of compute, outperforming several multi-fidelity, BO, and agentic baselines.

Critical strengths
  • The paper addresses an important practical problem: allocating limited compute among candidate experiments.

  • Combining adaptive budget pricing with selective inspection is a compelling design.

  • The evaluation includes multiple benchmarks, ablations, and analyses of inspection decisions. The experimental results look promising.

Critical weaknesses
  • Key implementation details are missing, particularly the score predictor’s inputs, calibration, and uncertainty estimation. It is also unclear how ForeAgent’s pairwise LLM comparisons interact with the shared predictor, making baseline fidelity difficult to assess.

  • The replay evaluation does not fully capture adaptive research. Execution outcomes update the incumbent and predictor, but do not regenerate future candidates. The main replay also retains descendants of skipped nodes and transfers their recorded improvements to a different incumbent. This assumption limits the evidence for effectiveness in real research workflows.

Questions for authors
  • What task and dataset information does the score predictor receive? How does it account for interactions between code, data distribution, and dataset scale?

  • Could CAES support large-scale LLM training, where historical evaluations are scarce and small-scale results may not preserve full-scale rankings? How sensitive is it to inaccurate predictions or misleading inspection signals?

  • Can the authors provide a closed-loop evaluation in which each selection policy influences subsequent candidate generation? How exactly is ForeAgent implemented, including candidate pairing, confidence thresholds, and its use of the shared predictor?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vLu5 (Part 1/2)

Thank you for your positive assessment and thoughtful questions. We provide detailed responses below.

Q1: Predictor inputs and calibration

What task and dataset information does the score predictor receive? How does it account for interactions between code, data distribution, and dataset scale? Key implementation details are missing, particularly the score predictor's inputs, calibration, and uncertainty estimation.

A1: The predictor receives dataset features together with candidate code/configuration features, and updates as experiments finish on the current task. These joint inputs let it account for how a proposed change interacts with dataset scale and distribution.

Predictor inputs. On TabRepo and LCBench, each candidate is a hyperparameter configuration. We use a Gaussian process with an ARD Matérn-5/2 kernel over normalized hyperparameters and five dataset features: log sample count, log feature count, class count, missingness fraction, and class-balance entropy. It is fitted across training tasks, then updated by exact GP conditioning after each completed configuration on the test task.

On AIRS-Bench, candidates are Python scripts generated by aira-dojo. A zero-shot LLM predictor estimates validation performance before execution from:

  • The dataset profile: modality, train/validation row counts, target metric, feature schema, and baseline validation score.
  • The parent node's code diff and observed validation score.
  • The operation type (draft, improve, debug, hyperparameter, or ensemble) and static code attributes: model-family imports, cross-validation folds, epoch/iteration caps, and subsampling flags.

The baseline and parent scores anchor predictions to results already obtained on the current task. Section 3.3 and Appendix I specify these inputs.

Online calibration and uncertainty. After each experiment, we compare its actual score with the prediction made before execution and update a linear correction to subsequent predictions. Uncertainty is estimated from the remaining prediction errors, plus a term that decreases as more results become available. Calibration starts from the three warm-start runs and updates after each completed experiment (Appendix I, Equation 7).

Calibration and ranking diagnostics. We evaluated recorded nodes and sibling candidate groups from AIRS-Bench (Appendix I, Table 8):

  • Nominal 68% / 95% prediction intervals achieve coverage of 66.4% ± 4.8% / 93.8% ± 3.1%; the latter ranges from 88.2% to 97.4% across tasks.
  • Within sibling groups, predicted and actual scores have a Spearman correlation of 0.48 ± 0.14 (range 0.24–0.71).
  • Scores from trials using 1/5 of the data and full runs have a Pearson correlation of 0.79 ± 0.11 (range 0.58–0.93).

These check uncertainty calibration, ranking among competing candidates, and whether partial trials predict full-run performance. All methods within a benchmark use the same predictor architecture, initial priors, and update rule, with separate states updated from their own completed runs.

Q2: ForeAgent implementation

How exactly is ForeAgent implemented, including candidate pairing, confidence thresholds, and its use of the shared predictor?

A2: We retain ForeAgent's pairwise LLM tournament and confidence-gated pruning [1]. The shared predictor supplies information to the comparator, which makes the advancement and pruning decisions.

Each batch contains five candidates: sibling scripts from one parent on AIRS-Bench, or five consecutive arrivals on TabRepo/LCBench. They compete in arrival order: candidate 1 versus 2, the winner versus 3, and so on. The comparator receives the parent score, both code diffs/configurations, predicted costs, and predicted scores, uncertainties, and failure probabilities from ForeAgent's own current predictor.

We query each pair in both presentation orders to reduce position bias and obtain complementary win probabilities. The candidate with win probability above 0.5 advances. If confidence reaches the threshold, the loser is pruned; otherwise, it enters a fallback queue ordered by win probability and runs after the tournament winner if budget remains. The threshold is fixed at 0.70, following [1, Section 6.2], in the parameter-free arms (Figure 6; Appendix H, Table 7). For Table 1, it is selected from {0.60, 0.70, 0.80, 0.90} by leave-one-task-out selection, as for the other parameterized baselines (Section 3.2).

After each full execution, ForeAgent updates its predictor using the same rule as CAES (Algorithm 1, Line 27). ForeAgent uses predictions for pairwise selection; CAES uses them to value skipping, partial trials, and full execution at the current shadow price. Section 3.3 and Appendix I give the implementation details.

AuthorsOpenReview ↗

Response to Reviewer vLu5 (Part 2/2)

Q3: Prediction quality, inspection signals, and LLM training

"Could CAES support large-scale LLM training, where historical evaluations are scarce and small-scale results may not preserve full-scale rankings? How sensitive is it to inaccurate predictions or misleading inspection signals?"

A3: We have not evaluated CAES on large-scale LLM training. To examine the prediction and inspection concerns you raise, we tested weak priors, increased partial-run noise, and misleading partial-run observations that reverse candidate rankings on AIRS-Bench at the 20% budget.

CAES adjusts how often it inspects candidates based on the reliability of the prior and the inspection signal (Section 2.3, Equations 3–4). When the prior is uninformative, a quick partial run helps decide whether the full run is worth paying for, so CAES inspects more often. When partial runs are noisier, they provide less information about the final score, so CAES inspects less often.

We tested these regimes on AIRS-Bench at the 20% budget (Appendix I, Table 9). The gap is the normalized performance shortfall relative to an unbudgeted run; lower is better:

Stress-Test Condition on AIRS-Bench (Appendix I, Table 9)Gap to Unbudgeted Run ↓\downarrowInspection Purchase Rate
Original CAES (Table 1 baseline).12717.8%
(a) Prior degradation: Zero-information flat prior (μi=bi\mu_i = b_i, inflated σi=2τ^\sigma_i = 2\hat{\tau}).14631.4%
(b) Partial-run noise: 3× additive inspection noise (sinject=3sis_{\mathrm{inject}} = 3s_i, modeled in sis_i).1354.2%
(b) Partial-run observation model misspecification: Unmodeled rank-inverting partial-run observations (corr⁡(zi,Yi)=−0.5\operatorname{corr}(z_i, Y_i) = -0.5).18918.6%
(b) Misspecification + online correlation guard (setting si→∞s_i \to \infty when ρ^z,y≤0.2\hat{\rho}_{z,y} \le 0.2).1393.1%

As expected, inspection purchases rise from 17.8% to 31.4% under flat priors (gap .146 vs. .127 baseline) and drop to 4.2% under tripled inspection noise (gap .135). When partial-run observations reverse candidate rankings (corr⁡(zi,Yi)=−0.5\operatorname{corr}(z_i, Y_i) = -0.5, raising the gap to .189), adding an online correlation guard that disables inspections whenever empirical partial-to-full correlation falls to ρ^z,y≤0.2\hat{\rho}_{z,y} \le 0.2 reduces the gap to .139 (Section 6; Appendix I, Table 9).

Q4: Live closed-loop evaluation

"The replay evaluation does not fully capture adaptive research... Can the authors provide a closed-loop evaluation in which each selection policy influences subsequent candidate generation?"

A4: We added the live experiments summarized in the General Response. Each policy now controls its own execution history, and the research agent generates subsequent candidates from that history. At 10% and 20% budgets, CAES achieves mean normalized performance gaps of .154 and .118, compared with .191 and .156 for the strongest comparison method, and wins 5 of 6 tasks against it at each budget. A lower gap means performance is closer to the unbudgeted reference run. CAES therefore allocates compute more effectively even when execution decisions change subsequent proposals. The shared setup and comparison table are in the General Response; Section 4 and Appendix J, Table 10 provide the full results.

References

[1] Jingsheng Zheng et al. Can We Predict Before Executing Machine Learning Agents? ACL, 2026.

Thank you for your detailed response. My main concerns have been addressed. I think this is interesting work. I keep my original score.

AuthorsOpenReview ↗

Thank you for your support and constructive comments

Many thanks for reviewing our responses and for your positive support of the paper! Your feedback on predictor calibration and closed-loop evaluation helped us substantially improve the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. CAES maintains an online shadow price on compute across the search, using an inner decision rule to decide whether to skip a candidate, run a partial trial, or complete the full run. The revised manuscript marks changes in blue:

Live closed-loop evaluation (vLu5 Q4; HdGk Q3; cnSV Q1). In our original replay experiments, all methods face the same sequence of candidates and costs so we can directly compare how they allocate compute. CAES achieved lower performance gaps in these comparisons. We thank the reviewers for suggesting that we also test adaptive candidate generation. Following this suggestion, we added live closed-loop evaluations to test whether CAES’s advantage holds when each policy’s execution decisions directly shape the subsequent candidates proposed by the agent.

We added evaluations at 10% and 20% compute budgets on six AIRS-Bench tasks spanning molecular property prediction, text classification, molecular graph regression, time-series forecasting (two tasks), and coreference resolution. We chose these tasks because their shorter runtimes let us complete the live comparisons within our compute budget. Within each task and budget tier, all policies receive the same hard machine-time budget, defined as 0.10 or 0.20 of the task's unbudgeted reference execution cost. They start from the same 3-candidate warm start on identical GPU hardware.

Each policy generates its own research trajectory. At each step, the research agent, aira-dojo, generates five sibling candidates from the code and validation results actually produced by that policy; skipped or vetoed candidates do not enter the agent’s context or the sequence of code changes leading to its current best solution. We also tested budget-aware proposal generation, which gives aira-dojo the remaining budget and recent execution information to guide candidate complexity and training epochs.

The gap measures the normalized performance shortfall relative to an unbudgeted reference run; lower is better. Run-all executes candidates without preselection, ForeAgent uses pairwise LLM comparisons to select candidates, and EI per cost selects by expected improvement per unit cost. Budget-Prompted Agent gives the candidate generator information about the remaining compute budget.

Live policy10% gap ↓10% depth20% gap ↓20% depth
Run-all.2211.4.1842.3
ForeAgent.2081.9.1793.1
Budget-Prompted Agent + Run-all.1981.8.1693.0
EI per cost.1912.1.1563.6
CAES.1542.9.1184.8
Budget-Prompted Agent + CAES.1433.3.1095.4

Consistent with the replay results, CAES outperforms the comparison methods at both live budgets. Against both executing without preselection and selecting by expected improvement per unit cost, CAES wins 5 of 6 tasks at each budget and searches deeper on average. Giving the generator budget information further improves CAES; the combined policy wins all six tasks against execution without preselection at each budget. Section 4 and Appendix J, Table 10 report the setup, spending, candidate costs, and task-level win counts.

  • Certification at equal calibration sample size (cnSV Q2). We test how much compute each method can save while keeping the risk of excessive performance loss below the same threshold. With all methods using 13 calibration tasks, CAES certifies a budget fraction of 0.618, saving 38.2% of compute, compared with 15.8% and 13.5% for the two comparison methods. The policy’s initial settings and update rules are fixed without using the calibration tasks. During execution, its state updates only from observations on the current task (Sections 2.5 and 4; Appendix H, Tables 6–7).
  • Predictor and baseline diagnostics (vLu5 Q1–Q3; cnSV Q1). We specify predictor inputs, online calibration, and the ForeAgent implementation, and report calibration and robustness tests under degraded priors and noisy inspection signals (Section 3.3; Appendix I, Tables 8–9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.