- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies LLM agent configuration search under a limited budget. It constructs four tasks and six complete configuration landscapes comprising 1,458 runs in total, and uses trajectory analysis to find that some "configuration-to-behavior" relationships are more stable across tasks than "configuration-to-utility" relationships. The proposed method combines behavior-level transfer with objective-function-level transfer, using objective observations to adjust the weights of the two, and performs search via expected improvement with an adaptive cost discount. Experiments use full-landscape replay to compare search regret across tasks, within-task repetitions, and mixed sources.
Critical strengths
- Distinguishing "how a configuration changes behavior" from "whether that behavior is valuable on the target task" provides a clear rationale for transfer when task utilities change, and is more targeted than directly assuming that the source task's optimal configuration transfers.
- The full factorial grid, action trajectories, performance, time, and cost records allow different search algorithms to be compared on the same observed landscape, and also support analyses of the relationship between configuration factors and behavior.
- All methods share the initial design, use 100 search seeds, report both monetary-budget and evaluation-count budgets separately, and provide paired bootstrap intervals and fairly complete ablations, which helps clarify where the gains come from.
Critical weaknesses
- The six landscapes come from only four tasks, and the within-task and mixed-source experiments are based solely on Spooky. The 100 replay seeds mainly reflect variation in the search's initial design and cannot substitute for independent tasks or repeated agent executions, so the current confidence intervals cannot directly support broad cross-task generalization claims.
- In Table 2, under the pure cross-task setting, BEH outperforms ALT on average across all four utility directions, and on the time direction ALT's regret area is 15.4% higher. This suggests the main cross-task gains come from behavior transfer, while the adaptive combination has not yet reliably avoided negative transfer; in some within-task settings, the adaptive cost discount also clearly lags the fixed discount.
- The method uses the complete source landscape, whereas in practice one usually has only sparse, biased historical configuration records; each target configuration typically corresponds to a single execution, so searching for the optimum may be affected by chance. The current experiments mainly demonstrate search ability on a fixed, recorded landscape, and the expected utility improvement under re-execution remains unverified.
Questions for authors
- Can the authors add independent tasks and different agent systems, and re-run the final recommended configurations multiple times, to evaluate actual expected utility rather than only the best observed value in the recorded landscape?
- Why, in the pure cross-task setting, do the adaptive weights fail to sufficiently favor the better-performing BEH? Is this related to the number of source tasks, the marginal-likelihood computation, or model complexity?
- When the behavior hypothesis weight is near zero, why is the cost index still determined independently by that hypothesis? Would it be more robust to use the model weights to modulate the index, or to compute a mixed expected improvement directly over the two predictive distributions?
- When the source landscape retains only 10%, 25%, or 50% of its observations, and these observations come from historical searches rather than uniform sampling, does ALT's advantage still hold?
5 comments
Response to Reviewer 3vuN (2/3): Incomplete sources, hypothesis weights, cost exponent
Incomplete source records (W3, Q4). ALT does not need complete landscapes. Each source keeps 27, 54, or 108 records, which are 11.1%, 22.2%, and 44.4% of the grid. Three selection rules separate the amount of data from selection bias. Balanced subsets cover every factor level equally. Historical subsets are the first records of a simulated GP-EI search on each source, as in your question. Top subsets keep the highest-utility records, which is a stronger bias. All source fitting and feature selection use only the retained records, and every method receives the same records (App. D.6). Under task shift at the dollar budget, across the nine partial-source conditions and four utility tracks, 29 of 36 paired 95% intervals favor ALT, seven include zero, and none favors ScaML-EI. After averaging regret areas over the four tracks, the reductions relative to ScaML-EI at 27/54/108 records per source are 5.0%/4.0%/10.5% for factor-balanced records, 5.0%/6.2%/9.7% for historical records, and 9.1%/5.3%/9.4% for highest-utility records.
Why the weights did not favor BEH (W2, Q2). The one identified cause was the response scale. Pooled source and target utilities set the scale of the behavior prior, so it followed the scale and number of source utilities and not the target. We now standardize the behavior hypothesis with the observed target utilities and compare the two likelihoods in common utility units (Sec. 5.2). The weights in our summary are measured on identical nine-run initial designs (App. D.4). With same-task sources the weight stays at about 0.20 or below, so the change is not a general shift toward behavior. The weight still falls as sources are added, because the fitted marginal likelihoods then give more relative support to the objective hypothesis. That hypothesis fits one weight per source, while the behavior coefficients are integrated out. The comparison retains the GP marginal-likelihood complexity term but does not integrate over the fitted source weights or kernel parameters. At the cross-task dollar budget, ALT is now within about 2% of BEH on three tracks and 11.0% behind on the time track, down from 15.4%. We agree that the cross-task gains come mainly from behavior transfer. We keep the combination because BEH is clearly worse with a same-task source, where ALT reduces its regret area by 23–76% (Table 2). These results support ALT as a reasonable compromise across the evaluated source settings.
Cost exponent (W2, Q3). We adopted your first suggestion to make the cost discount reflect the same target evidence as the predictive mixture. The exponent is now , where and are the behavior and objective hypotheses’ exponents and is the predictive mixture weight (Eq. 7). A hypothesis with little predictive weight therefore also has little influence on the cost discount. In the same-task setting, the exponent at evaluation 60 is now 0.97 on the time track and 1.00 on the cost track, up from 0.86 and 0.93 (App. D.4). With a same-task source at the dollar budget, the cost-track gap to the fixed full discount falls from 26–46% to 9–14%. The full discount still leads on the time track, by 8–27%. Under task shift it is worse, and ALT reduces its area by 2.5–9.1% at the dollar budget and by 7–19% at 60 evaluations. In every setting and utility track at both budgets, ALT achieves lower mean regret area than at least one of its no-discount and full-discount ablations (Table 2, App. D.3).
Response to Reviewer 3vuN (3/3): Scale and repeated execution
Scale and intervals (W1, Q1). We agree with this assessment. The intervals in the main search experiments reflect variation in the initial design and are conditional on the recorded landscapes. We clarify this in Sec. 6 and explicitly discuss the limited task coverage and use of one agent system in the limitations paragraph (Sec. 7). Same-task and mixed results currently rely on Spooky repetitions. Four further landscapes are being collected: TGS Salt Identification, Plant Seedlings Classification, and two additional Scaling Law repetitions. The Scaling Law repetitions will provide a second task for same-task and mixed-source evaluation. Evaluating a second agent system remains beyond the scope of this study.
Repeated execution (W3, Q1). The resampling study uses 50 initial designs and 50 simulated search sequences per design, with sources from the other three tasks (App. D.8). Each evaluation samples performance, runtime, and cost jointly from one of the three recorded Spooky repetitions. Mean-utility scoring uses all three recorded outcomes of the recommended configuration, including the outcome observed during search. For the paired intervals, we average over source sets and simulated searches within each initial design, then bootstrap the 50 design means. Under mean-utility scoring at the dollar budget, ALT reduces regret area, averaged over the four tracks, by 5.2% relative to the target-only GP, 26.7% relative to random search, and 40.0% relative to retaining the best configuration from the nine initial runs. These comparisons support the benefit of search and transfer in this resampling study. Relative to ScaML-EI, ALT’s average reduction at the dollar budget is 6.6% under observed-utility scoring, with gains concentrated on the time and cost tracks (23.0% and 7.5%). Under mean-utility scoring, the average reduction is 0.9% and is not statistically significant (95% interval −0.2% to 2.1%). At a fixed budget of 60 evaluations, ALT’s mean-utility regret area is 2.1% higher than ScaML-EI’s. The ranking checks illustrate why the scoring distinction matters. On the time track, the configurations with the highest utility in Spooky repetitions 1/2/3 rank 42nd/17th/3rd of 243 when evaluated by their mean utility across repetitions. On the cost track, their ranks are 12th/22nd/1st; on balanced and performance utility, they remain in the top five. A strong individual run therefore need not identify a configuration that ranks equally well by mean utility. Evidence on execution variation is limited to three recorded repetitions of one task, Spooky. This limits the scope of our conclusions about execution variation across tasks, as stated explicitly in Sec. 7.
Response to Reviewer 3vuN (1/3): Summary of revisions
Thank you for the thorough and precise review. Your insights helped us improve the method, substantiate its claims, and make the study more comprehensive. In summary, we made the following revisions.
First, we added a study of incomplete source records with nine conditions (W3, Q4). Each source keeps 27, 54, or 108 records (11–44% of the grid), chosen by factor balance, by a simulated historical search, or by highest source utility. Under task shift at the dollar budget, ALT has lower mean regret area than ScaML-EI in 35 of 36 comparisons. Of these, 29 paired 95% intervals favor ALT and none favors ScaML-EI.
Second, we modified the weighting and the cost exponent (W2, Q2, Q3). The behavior hypothesis is now standardized with the observed target utilities, and the exponent follows both hypotheses. The mean behavior weight under task shift rises from 0.54/0.29/0.15 to 0.67/0.52/0.40 with one/two/three sources. We do not treat a higher weight as proof of a better model. The weight is still computed from the likelihood of the observed target runs, so it shows which hypothesis predicts those runs better and does not measure search performance. What has changed is the mechanism, because the scale of the behavior prior no longer depends on the source utilities.
Third, for scale and repeated execution (W1, W3, Q1), we are extending the benchmark and have added a resampling study. Collection of four further landscapes is under way. Completing them within the rebuttal window is not practical, so they are scheduled for the camera-ready version if the paper is accepted. The resampling study uses the three recorded Spooky repetitions. Each evaluation returns performance, runtime, and cost jointly from one randomly chosen repetition of the selected configuration. We use two regret definitions. Observed-utility regret compares the best utility observed during search with the best of all 3 × 243 recorded outcomes. Mean-utility regret scores the same recommended configuration by its mean across three repetitions, relative to the highest configuration-wise mean. At the dollar budget, averaging regret areas across the four tracks, ALT improves over ScaML-EI by 6.6% under observed-utility scoring. Under mean-utility scoring, the average reduction is 0.9% and is not statistically significant (95% interval −0.2% to 2.1%). A configuration with a strong observed run may rank lower when scored by its mean utility across the three recorded repetitions. We examine these ranking changes in the detailed response.
Our next two comments give the details, the numbers with their locations, and our rationale.