- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether an LLM-based machine learning agent can learn how to allocate its search budget during execution, instead of following a fixed search policy. The authors propose Factorized Adaptive Search (FAS), which separates two decisions: which operator to apply (Draft, Debug, or Improve), and which previously evaluated solution to expand. Operator selection is handled by a fitness-rate-rank bandit with UCB-based exploration, while parent selection uses operator-specific LinUCB models that consider validation scores and features of the search history. Both policies are updated online using validation improvement as the reward, without relying on offline training or accessing held-out test feedback.
The method is implemented on top of AIRA-DOJO, keeping the underlying LLM, prompts, operators, and execution environment unchanged. The authors evaluate four configurations across 11 MLE-bench Lite tasks: the original AIRA policy, adaptive operator selection (FAS-Op), adaptive parent selection (FAS-Parent), and a combination of both. FAS-Op improves the anytime percentile-rank AUC from 0.781 to 0.820 while using approximately 26% fewer tokens. It also achieves higher candidate validity and a larger proportion of medal-worthy solutions. Interestingly, adapting the parent-selection policy has the opposite effect, substantially degrading performance both on its own and when combined with operator adaptation. The authors further examine operator usage, reward sparsity, candidate quality, and validation–test discrepancies to understand these differences.
Critical strengths
I appreciate that the authors examined operator selection and parent selection separately rather than modifying multiple components of an agent simultaneously. The 2x2 experimental design, with the underlying AIRA stack kept fixed, makes it easier to understand which search decisions influence performance. The common warm start, feasibility masks, and separation between online validation feedback and held-out evaluation are also sensible choices. Although the bandit algorithms themselves are well established, applying them to these decisions in an ML engineering agent is an interesting direction, and the controlled setup is a strength of the paper.
The empirical analysis provides useful insights into how search budget allocation affects performance. FAS-Op improves selected-incumbent AUC from 0.781 to 0.820 and oracle-coverage AUC from 0.805 to 0.833, while reducing token consumption from 15.0M to 11.1M and improving pooled candidate validity from 42.7% to 52.8%. I found the analysis in Section 7 particularly informative: FAS-Op reduces Debug operations from 55.7% to 25.4% and achieves higher per-operator medal yields. The negative findings for parent selection are also valuable. The authors show that only 5.8% and 7.3% of completed candidates receive positive rewards under the two parent-adaptive configurations, offering a plausible explanation for why LinUCB struggles to learn effective parent preferences. I appreciate that the paper examines these failures rather than focusing only on the successful configuration.
Critical weaknesses
My main concern is whether the experiments sufficiently isolate the contribution of the online learning mechanism. The authors attribute the improvement of FAS-Op to adaptive operator selection, but the results also show a substantial shift in how frequently different operators are used. It is not obvious whether the improvement comes from learning which operator is most useful at a particular point in the search, or simply from spending less time on an operator that the original baseline uses too frequently. This becomes particularly noticeable in Section 7. Under AIRA, 55.7% of completed operations are Debug, while only 7.8% are Draft and 36.5% are Improve. FAS-Op changes this distribution to 25.4% Debug, 33.7% Draft, and 40.9% Improve. This is a rather large difference. The authors also observe that many debugging attempts fail quickly and contribute little useful progress. Given this behavior, I wonder whether a simple fixed policy that performs fewer Debug operations and allocates more attempts to Draft or Improve would already achieve a substantial portion of the reported gains.
I would have liked to see a comparison against at least one non-adaptive operator policy with a more balanced allocation. For example, a fixed stochastic policy using approximately the same operator proportions as FAS-Op could serve as a useful control. Alternatively, a simple heuristic that reduces the probability of Debug after repeated failures, or changes the allocation across different stages of the search, would help establish whether the learned bandit offers an advantage beyond a better hand-designed schedule. The results in Table 3 do not fully address this concern. While FAS-Op has higher medal yields for each operator, the operations are applied to different search graphs under the two policies. The corresponding parent states and generated candidates are therefore not directly comparable. Consequently, higher per-operator success rates do not necessarily establish that the online reward updates are responsible for selecting more effective actions.
Another analysis I would have found useful is how the operator preferences evolve within individual runs. The paper primarily reports aggregate operator frequencies, but these do not tell us whether the controller is actually changing its preferences in response to previous successes and failures, or whether it quickly settles into a relatively stable distribution that happens to work better than AIRA's default. Showing the evolution of the bandit's decisions and estimated operator values could help clarify this. I think this matters for the novelty of the contribution as well. AIRA already provides the underlying decomposition of the search process, and both fitness-rate-rank bandits and contextual UCB are well-established techniques. The more interesting contribution here is demonstrating that learning operator preferences online leads to better search decisions than simpler alternatives. Without a stronger non-adaptive baseline, I am not fully convinced that the current experiments establish this distinction.
Another important concern is the strength of the experimental evidence, particularly given how stochastic LLM-based search can be. Section 5 states that the experiments cover 11 MLE-bench Lite tasks, with only one seed per task and a four-hour wall-clock budget. The authors have made a reasonable effort to keep the experimental conditions consistent, but one run per task is quite limited for evaluating an adaptive search policy. Different initial solutions, generated programs, or execution failures could change the search trajectory substantially, even when the policy itself remains the same. The primary selected-incumbent AUC improves from 0.781 to 0.820, which is encouraging, but the absolute difference is still relatively modest. Similarly, the endpoint selected percentile rank increases from 0.849 to 0.863. Without repeated runs or some measure of variability, it is difficult to determine how reliable these improvements are.
Also, the authors report that FAS-Op beats AIRA on seven tasks, ties on one, and loses on three. This suggests a positive trend, but it does not tell us whether the same ordering would hold across different random seeds. I would have been more convinced by results averaged over multiple runs, even if repetitions were limited to a representative subset of tasks because of computational constraints. Reporting per-task AUC improvements and paired uncertainty estimates would also help determine whether the overall gain is broadly consistent or driven by a few tasks.
I believe the candidate-level results require some care in their interpretation as well. Although the paper reports a large number of generated candidates, these candidates are produced within the same search trajectories and should not be treated as independent experimental repetitions. This is particularly relevant when interpreting the differences in candidate validity and medal yield.
There is also an interesting distinction in the medal results. Table 2 reports a higher proportion of medal-worthy candidates under FAS-Op. However, Appendix D shows that both FAS-Op and AIRA earn at least one medal on 54.5% of tasks. FAS-Op achieves a better distribution of medal tiers, particularly with more gold medals, which is certainly meaningful. Nevertheless, producing more medal-worthy intermediate candidates is not quite the same as improving the final task-level medal success rate. I think the paper could distinguish these outcomes more carefully.
I also found the cost-normalized reward formulation somewhat confusing. Equation 4 defines the reward as the improvement in the best validation score divided by the cost of the action. However, Section 4.2 subsequently states that one completed candidate evaluation defines the cost unit, with c_t = 1. If every completed expansion is assigned this same cost, the denominator becomes constant, and the reward effectively reduces to the improvement in validation score without accounting for differences in execution time.
This seems worth clarifying because the method is repeatedly described as using cost-normalized improvement, and the motivation emphasizes efficient allocation of a limited experimental budget. Section 7 also reports fairly substantial differences in how long different operator attempts take. If the reward does not actually account for these differences, the extent to which the bandit is learning cost-efficient behavior becomes less clear. This could simply be an issue with the explanation, but I think the authors should specify exactly how c_t is calculated in the implementation and whether it varies across actions.
TBH, I am not entirely convinced by the broader interpretation of the negative parent-selection results. The experiments clearly show that the particular LinUCB-based parent selector performs poorly, but this does not necessarily mean that learning parent selection online is generally less effective or less useful than learning operator selection. The parent policy uses a linear reward model with a fixed set of features, a particular exploration coefficient, and immediate validation improvement as the learning signal. As the authors acknowledge, the rewards are extremely sparse, and an expansion that contributes to a useful solution later may receive zero reward at the current step. Under these conditions, it is not particularly surprising that the contextual bandit struggles to outperform the simple heuristic of selecting the best-validation parent. The negative finding is still valuable, but I would be cautious about generalizing it beyond the specific parent-selection method and feedback signal evaluated in this paper.
The results suggest that operator allocation matters and that the proposed operator bandit performs better than the fixed AIRA policy in this setup. However, without stronger controls and repeated evaluations, it remains difficult to determine how much of that improvement should be attributed specifically to online learning.
Questions for authors
-
How much of FAS-Op's improvement comes from the adaptive learning mechanism, rather than simply changing the proportions of Draft, Debug, and Improve operations?
-
I would also be interested in seeing how the learned operator preferences change during individual runs. These comparisons would help establish whether the online updates provide an advantage beyond correcting the original baseline's debugging-heavy allocation.
-
The evaluation uses only one seed for each of the 11 tasks. Given the stochastic nature of LLM-generated programs and the relatively modest improvement in the primary AUC metric, could the authors provide repeated-seed results with some measure of variability? Even additional runs on a smaller, representative subset of tasks would be useful. Reporting per-task AUC differences and paired uncertainty estimates would also make it easier to assess whether the improvement is consistent across the benchmark suite.
-
Could the authors clarify how the cost-normalized reward in Equation 4 is actually implemented? Does this cost vary with the actual wall-clock duration of an expansion, or does every completed attempt receive the same cost? If it is always one, the reward appears to be equivalent to unnormalized validation improvement. I would be interested in whether normalizing by actual elapsed time changes the bandit's decisions or the resulting performance.
-
How sensitive is the negative parent-selection result to the particular LinUCB configuration and reward signal? Have the authors considered whether reducing the exploration coefficient, modifying the features, or using a less sparse reward improves parent selection? This would help clarify whether the observed failure is primarily a limitation of the particular implementation or evidence of a more general difficulty with online parent-selection learning.