acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Transfer for Cost-Aware Bayesian Optimization of LLM Agent Configurations

Abstract

LLM agents can now autonomously perform long-horizon development tasks. An agent’s configuration, including its reasoning effort, tool access, and resource allocation, affects its performance, runtime, and monetary cost. An optimal configuration needs to balance these outcomes, as greater resource use does not necessarily yield higher utility. A full grid search on each new task is not practical because every configuration requires a costly, time-consuming run. Bayesian optimization can improve sample efficiency in search, and transferring knowledge from earlier tasks can further reduce search cost. The challenge is to identify what transfers when outcomes change across tasks and few target evaluations are affordable. In this work, we propose adaptive-level transfer (ALT), a cost-aware Bayesian optimization method that uses target evidence to adapt what is transferred from earlier tasks. Through systematic evaluation of every configuration of one agent across several tasks, recording agent action trajectories and outcomes, we obtain complete landscapes for exact replay of configuration search. Our analysis shows that configuration effects on several agent behavior features are relatively stable across tasks, while their effects on utility shift. ALT transfers configuration-to-behavior patterns from source trajectories and learns their utility contributions from target runs. It combines this behavior-level hypothesis with an objective-level hypothesis that transfers source utility surfaces. The target evidence decides how much each contributes to the prediction. Its acquisition weighs expected improvement against predicted cost, with the cost discount learned from target evidence on whether resource-intensive behavior raises utility. In cross-task search under monetary budgets, ALT reduces mean regret area by 4–12% relative to our main transfer baseline, ScaML-GP with expected improvement across all four evaluated utilities, with the largest reductions on the performance- and time-focused utilities. When sources include other repetitions of the target task, ALT outperforms its behavior-only ablation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimization, cost-aware optimization, transfer Bayesian optimization, Gaussian processes, LLM agent configuration, AutoML

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3vuNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies LLM agent configuration search under a limited budget. It constructs four tasks and six complete configuration landscapes comprising 1,458 runs in total, and uses trajectory analysis to find that some "configuration-to-behavior" relationships are more stable across tasks than "configuration-to-utility" relationships. The proposed method combines behavior-level transfer with objective-function-level transfer, using objective observations to adjust the weights of the two, and performs search via expected improvement with an adaptive cost discount. Experiments use full-landscape replay to compare search regret across tasks, within-task repetitions, and mixed sources.

Critical strengths
  1. Distinguishing "how a configuration changes behavior" from "whether that behavior is valuable on the target task" provides a clear rationale for transfer when task utilities change, and is more targeted than directly assuming that the source task's optimal configuration transfers.
  2. The full factorial grid, action trajectories, performance, time, and cost records allow different search algorithms to be compared on the same observed landscape, and also support analyses of the relationship between configuration factors and behavior.
  3. All methods share the initial design, use 100 search seeds, report both monetary-budget and evaluation-count budgets separately, and provide paired bootstrap intervals and fairly complete ablations, which helps clarify where the gains come from.
Critical weaknesses
  1. The six landscapes come from only four tasks, and the within-task and mixed-source experiments are based solely on Spooky. The 100 replay seeds mainly reflect variation in the search's initial design and cannot substitute for independent tasks or repeated agent executions, so the current confidence intervals cannot directly support broad cross-task generalization claims.
  2. In Table 2, under the pure cross-task setting, BEH outperforms ALT on average across all four utility directions, and on the time direction ALT's regret area is 15.4% higher. This suggests the main cross-task gains come from behavior transfer, while the adaptive combination has not yet reliably avoided negative transfer; in some within-task settings, the adaptive cost discount also clearly lags the fixed discount.
  3. The method uses the complete source landscape, whereas in practice one usually has only sparse, biased historical configuration records; each target configuration typically corresponds to a single execution, so searching for the optimum may be affected by chance. The current experiments mainly demonstrate search ability on a fixed, recorded landscape, and the expected utility improvement under re-execution remains unverified.
Questions for authors
  1. Can the authors add independent tasks and different agent systems, and re-run the final recommended configurations multiple times, to evaluate actual expected utility rather than only the best observed value in the recorded landscape?
  2. Why, in the pure cross-task setting, do the adaptive weights fail to sufficiently favor the better-performing BEH? Is this related to the number of source tasks, the marginal-likelihood computation, or model complexity?
  3. When the behavior hypothesis weight is near zero, why is the cost index still determined independently by that hypothesis? Would it be more robust to use the model weights to modulate the index, or to compute a mixed expected improvement directly over the two predictive distributions?
  4. When the source landscape retains only 10%, 25%, or 50% of its observations, and these observations come from historical searches rather than uniform sampling, does ALT's advantage still hold?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3vuN (1/3): Summary of revisions

Thank you for the thorough and precise review. Your insights helped us improve the method, substantiate its claims, and make the study more comprehensive. In summary, we made the following revisions.

First, we added a study of incomplete source records with nine conditions (W3, Q4). Each source keeps 27, 54, or 108 records (11–44% of the grid), chosen by factor balance, by a simulated historical search, or by highest source utility. Under task shift at the dollar budget, ALT has lower mean regret area than ScaML-EI in 35 of 36 comparisons. Of these, 29 paired 95% intervals favor ALT and none favors ScaML-EI.

Second, we modified the weighting and the cost exponent (W2, Q2, Q3). The behavior hypothesis is now standardized with the observed target utilities, and the exponent follows both hypotheses. The mean behavior weight under task shift rises from 0.54/0.29/0.15 to 0.67/0.52/0.40 with one/two/three sources. We do not treat a higher weight as proof of a better model. The weight is still computed from the likelihood of the observed target runs, so it shows which hypothesis predicts those runs better and does not measure search performance. What has changed is the mechanism, because the scale of the behavior prior no longer depends on the source utilities.

Third, for scale and repeated execution (W1, W3, Q1), we are extending the benchmark and have added a resampling study. Collection of four further landscapes is under way. Completing them within the rebuttal window is not practical, so they are scheduled for the camera-ready version if the paper is accepted. The resampling study uses the three recorded Spooky repetitions. Each evaluation returns performance, runtime, and cost jointly from one randomly chosen repetition of the selected configuration. We use two regret definitions. Observed-utility regret compares the best utility observed during search with the best of all 3 × 243 recorded outcomes. Mean-utility regret scores the same recommended configuration by its mean across three repetitions, relative to the highest configuration-wise mean. At the dollar budget, averaging regret areas across the four tracks, ALT improves over ScaML-EI by 6.6% under observed-utility scoring. Under mean-utility scoring, the average reduction is 0.9% and is not statistically significant (95% interval −0.2% to 2.1%). A configuration with a strong observed run may rank lower when scored by its mean utility across the three recorded repetitions. We examine these ranking changes in the detailed response.

Our next two comments give the details, the numbers with their locations, and our rationale.

AuthorsOpenReview ↗

Response to Reviewer 3vuN (2/3): Incomplete sources, hypothesis weights, cost exponent

Incomplete source records (W3, Q4). ALT does not need complete landscapes. Each source keeps 27, 54, or 108 records, which are 11.1%, 22.2%, and 44.4% of the grid. Three selection rules separate the amount of data from selection bias. Balanced subsets cover every factor level equally. Historical subsets are the first records of a simulated GP-EI search on each source, as in your question. Top subsets keep the highest-utility records, which is a stronger bias. All source fitting and feature selection use only the retained records, and every method receives the same records (App. D.6). Under task shift at the dollar budget, across the nine partial-source conditions and four utility tracks, 29 of 36 paired 95% intervals favor ALT, seven include zero, and none favors ScaML-EI. After averaging regret areas over the four tracks, the reductions relative to ScaML-EI at 27/54/108 records per source are 5.0%/4.0%/10.5% for factor-balanced records, 5.0%/6.2%/9.7% for historical records, and 9.1%/5.3%/9.4% for highest-utility records.

Why the weights did not favor BEH (W2, Q2). The one identified cause was the response scale. Pooled source and target utilities set the scale of the behavior prior, so it followed the scale and number of source utilities and not the target. We now standardize the behavior hypothesis with the observed target utilities and compare the two likelihoods in common utility units (Sec. 5.2). The weights in our summary are measured on identical nine-run initial designs (App. D.4). With same-task sources the weight stays at about 0.20 or below, so the change is not a general shift toward behavior. The weight still falls as sources are added, because the fitted marginal likelihoods then give more relative support to the objective hypothesis. That hypothesis fits one weight per source, while the behavior coefficients are integrated out. The comparison retains the GP marginal-likelihood complexity term but does not integrate over the fitted source weights or kernel parameters. At the cross-task dollar budget, ALT is now within about 2% of BEH on three tracks and 11.0% behind on the time track, down from 15.4%. We agree that the cross-task gains come mainly from behavior transfer. We keep the combination because BEH is clearly worse with a same-task source, where ALT reduces its regret area by 23–76% (Table 2). These results support ALT as a reasonable compromise across the evaluated source settings.

Cost exponent (W2, Q3). We adopted your first suggestion to make the cost discount reflect the same target evidence as the predictive mixture. The exponent is now α=πα1+(1−π)α2\alpha = \pi \alpha_{1} + (1-\pi) \alpha_2, where α1\alpha_1 and α2\alpha_2 are the behavior and objective hypotheses’ exponents and π\pi is the predictive mixture weight (Eq. 7). A hypothesis with little predictive weight therefore also has little influence on the cost discount. In the same-task setting, the exponent at evaluation 60 is now 0.97 on the time track and 1.00 on the cost track, up from 0.86 and 0.93 (App. D.4). With a same-task source at the dollar budget, the cost-track gap to the fixed full discount falls from 26–46% to 9–14%. The full discount still leads on the time track, by 8–27%. Under task shift it is worse, and ALT reduces its area by 2.5–9.1% at the dollar budget and by 7–19% at 60 evaluations. In every setting and utility track at both budgets, ALT achieves lower mean regret area than at least one of its no-discount and full-discount ablations (Table 2, App. D.3).

AuthorsOpenReview ↗

Response to Reviewer 3vuN (3/3): Scale and repeated execution

Scale and intervals (W1, Q1). We agree with this assessment. The intervals in the main search experiments reflect variation in the initial design and are conditional on the recorded landscapes. We clarify this in Sec. 6 and explicitly discuss the limited task coverage and use of one agent system in the limitations paragraph (Sec. 7). Same-task and mixed results currently rely on Spooky repetitions. Four further landscapes are being collected: TGS Salt Identification, Plant Seedlings Classification, and two additional Scaling Law repetitions. The Scaling Law repetitions will provide a second task for same-task and mixed-source evaluation. Evaluating a second agent system remains beyond the scope of this study.

Repeated execution (W3, Q1). The resampling study uses 50 initial designs and 50 simulated search sequences per design, with sources from the other three tasks (App. D.8). Each evaluation samples performance, runtime, and cost jointly from one of the three recorded Spooky repetitions. Mean-utility scoring uses all three recorded outcomes of the recommended configuration, including the outcome observed during search. For the paired intervals, we average over source sets and simulated searches within each initial design, then bootstrap the 50 design means. Under mean-utility scoring at the dollar budget, ALT reduces regret area, averaged over the four tracks, by 5.2% relative to the target-only GP, 26.7% relative to random search, and 40.0% relative to retaining the best configuration from the nine initial runs. These comparisons support the benefit of search and transfer in this resampling study. Relative to ScaML-EI, ALT’s average reduction at the dollar budget is 6.6% under observed-utility scoring, with gains concentrated on the time and cost tracks (23.0% and 7.5%). Under mean-utility scoring, the average reduction is 0.9% and is not statistically significant (95% interval −0.2% to 2.1%). At a fixed budget of 60 evaluations, ALT’s mean-utility regret area is 2.1% higher than ScaML-EI’s. The ranking checks illustrate why the scoring distinction matters. On the time track, the configurations with the highest utility in Spooky repetitions 1/2/3 rank 42nd/17th/3rd of 243 when evaluated by their mean utility across repetitions. On the cost track, their ranks are 12th/22nd/1st; on balanced and performance utility, they remain in the top five. A strong individual run therefore need not identify a configuration that ranks equally well by mean utility. Evidence on execution variation is limited to three recorded repetitions of one task, Spooky. This limits the scope of our conclusions about execution variation across tasks, as stated explicitly in Sec. 7.

Reviewer_gpAZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of efficiently searching over LLM agent configurations (model, reasoning effort, resource budget, skills, tools) for new tasks, where each evaluation requires a costly, hours-long agent run. The authors make three contributions: (1) a complete dataset of 1,458 agent runs across 243 configurations and 6 landscapes (4 tasks, with 3 repetitions of one task), recording full action trajectories and outcomes; (2) an empirical finding that configuration effects on agent behavior features are substantially more stable across tasks than their effects on utility; (3) ALT, a transfer Bayesian optimization method that combines behavior-level transfer (learning configuration-to-behavior mappings from sources, behavior-to-utility mappings on the target) with objective-level transfer (rescaled source utility surfaces), adaptively weighting the two using marginal likelihoods and selecting the next evaluation with a learned cost discount. In cross-task search under monetary budgets, ALT reduces regret area by 4–12% relative to ScaML-GP with expected improvement.

Critical strengths
  1. Valuable dataset and reproducible evaluation protocol. The full factorial evaluation of 243 configurations across 6 landscapes, with complete action trajectories, enables exact replay of any search strategy. This is a meaningful community resource analogous to tabular benchmarks in HPO/NAS, and is arguably the most lasting contribution of the paper.

  2. Well-motivated two-step transfer framework. The empirical finding that behavior features exhibit high cross-task factor-effect correlations (0.87–0.91) while utility correlations are near zero (−0.10 to 0.42) provides clear justification for transferring configuration-to-behavior mappings rather than utility surfaces. This insight is well-supported and is valuable independently of the specific method.

  3. Honest and thorough reporting. The authors transparently report cases where ALT underperforms (e.g., BEH outperforms ALT under task shift; ALT trails fixed cost discount on time/cost tracks with same-task sources). The extensive appendix with confidence intervals, multiple budget levels, and ablations enables readers to form their own assessment.

Critical weaknesses
  1. The adaptive weighting mechanism—a core contribution—does not work reliably, improvements are modest.

  2. Behavior characterization is coarse and underexploits trajectory information. Richer trajectory representations could substantially strengthen the behavior-level hypothesis.

  3. Experimental scale is narrow.

Questions for authors

How sensitive are results to the choice of exactly 3 behavior features and the linear form of h?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gpAZ

Thank you for the careful and constructive review, and for recognizing the dataset, the motivation for behavior-level transfer, and our reporting of unfavorable cases. Your first point was especially valuable. Asking why the adaptive weighting was unreliable led us to a cause we could fix. Source utilities were setting the scale of the behavior prior, and the revised method standardizes that hypothesis on target runs instead (W1). In response to your question about the three features and the linear form of h, we added sensitivity tests with richer and nonlinear behavior maps (Question, W2). On experimental scale (W3), collection of four further landscapes, listed in the general response, is under way. Completing them within the rebuttal window is not practical, so they are scheduled for the camera-ready version if the paper is accepted. Below we provide itemized details and rationale.

Reliability of the adaptive weighting (W1). We identified an issue with response scaling that affected the hypothesis weights. In the submitted version, pooled source and target utilities set the scale of the behavior prior, so it followed the sources and not the target. We now standardize the behavior hypothesis with the observed target utilities and compare the two likelihoods in common utility units. On identical nine-run initial designs, the mean behavior weight under task shift rises from 0.54/0.29/0.15 to 0.67/0.52/0.40 with one/two/three sources (App. D.4). At the cross-task dollar budget, ALT is now within about 2% of BEH on three tracks and 11.0% behind on the time track, down from 15.4%. We keep the weighting because neither level is safe alone. At the same budget, ALT reduces the regret area of behavior-only transfer (BEH) by 23–76% with a same-task source, and that of objective-only transfer (ScaML-ADAPT) by 3.4–12.1% under task shift. Each of these is the worst of the three variants in its case, and ALT is never the worst (Table 2, App. D.5). Gains over the baselines are moderate but consistent. ALT is significantly better in 118 of 144 cross-task comparisons and never significantly worse. The remaining time-track gap may reflect the difference between likelihood fit to observed runs and performance in subsequent search.

Three features and the linear form of h (Question, W2). We chose three features and an affine map because h must be learned from a few target runs, and a small model keeps that step simple and interpretable. The search starts from nine target runs, and an affine map on all seventeen features has eighteen parameters, which are harder to estimate and to interpret. We also expected added features to transfer less reliably. To test the choice, we compared it with affine maps on eight or all seventeen features and with a nonlinear Matérn map on the same three features. The tests cover full sources and balanced subsets of 27 and 108 records under balanced utility (App. D.7). In 22 of the 27 comparisons, the paired 95% interval of the change in dollar-budget regret area includes zero. In the other five it excludes zero. The largest change in dollar-budget regret area is 4.3% with full sources, 6.0% with 27 records, and 10.0% with 108 records. Seventeen features lower the cross-task area by 3.3% with full sources. All three alternatives lower the mixed-setting area by 8–10% with 108 records. The nonlinear map gives no cross-task gain and is 2.9% worse with 27 records. The main results therefore do not depend on exactly three features or on a linear h. The 8–10% gain is large, and it shows that the default is not the best choice in every case. We agree that the behavior description is coarse. These tests are also limited, since they vary only the number of features and the form of h. The gains from more features suggest that richer trajectory representations can help, and the conclusion now names this as future work.

Thanks for your response and I stay with my positive score.

Reviewer_uqEAOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of searching for LLM agent configurations (models, reasoning effort, resources, skills, and tools). The authors ran all configurations across 4 tasks, producing 6 complete result tables. Their analysis reveals that the impact of configurations on agent behavior features is relatively stable across tasks, whereas their impact on utility is unstable. Based on this, they propose adaptive-level transfer that uses Bayesian optimization to transfer from earlier tasks.

Critical strengths
  1. The comprehensively recorded configuration results and trajectory datasets are valuable and can be used to reproduce the search process.

  2. The insight that "behavioral features are easier to transfer across tasks than utility" is interesting

Critical weaknesses
  1. The method is a combination of existing Bayesian Optimization (BO) components. Target-level transfer directly uses ScaML-GP; behavioral transfer adds a linear feature term to the kernel; the two are blended via marginal likelihood model averaging; and the cost discount is a variant of EI per cost. Every component is standard practice, meaning novelty lies primarily in the dataset rather than the method.

  2. Limited practical utility. The method requires complete configuration results from multiple tasks as source data, a condition the authors themselves acknowledge is rarely met in practical deployments. The search budget equates to the cost of 60 random evaluations, with each evaluation taking several hours. The paper fails to discuss whether such a search is cost-effective compared to simply using a strong default configuration.

Questions for authors
  1. Since BEH comprehensively outperforms ALT in the cross-task setting, and fixed EI/cost outperforms ALT in the single-task setting, what is the necessity of adaptive exponent?
4 comments
AuthorsOpenReview ↗

Response to Reviewer uqEA

Thank you for your review, and for noting the value of the recorded landscapes and the behavior-transfer insight. Your summary reflects the paper’s core idea accurately. On presentation, we revised Algorithm 1, the notation, and the regret definitions. We would welcome pointers to any passage that remains hard to follow. Below we clarify how ALT follows from the Section 4 finding (W1), report new experiments with incomplete source records and comparisons against deploying without search (W2), and explain the refined cost exponent (Q1).

Novelty (W1). We agree that every component is standard, and we do not claim new BO machinery. Our methodological contribution is the transfer design: we reuse a configuration-to-behavior map, learn its utility contributions from target runs, and combine this with objective-level transfer using target evidence. This design follows from our problem framing, the recorded landscapes (Sec. 3), and the finding that several configuration-to-behavior effects are more consistent across tasks than utility effects (Sec. 4). BEH uses the behavior hypothesis alone, so it tests behavior-level transfer directly. Its cross-task results, which you note in Q1, support transferring at this level. ALT adds objective-level transfer, which helps when sources resemble the target. With a same-task source, ALT’s regret area at the dollar budget is 23–76% lower than that of BEH (Table 2). Under task shift, it is 3.4–12.1% lower than that of the objective-only variant ScaML-ADAPT (App. D.5). These experiments assess the practical value of the transfer design motivated by Sec. 4. We have revised the introduction to make this connection explicit.

Incomplete sources (W2). ALT and the transfer baselines accept incomplete source records. We collected complete landscapes to enable controlled replay and evaluate search against the best recorded configuration. We have added nine source-data conditions: 27, 54, or 108 records per source (11–44% of the grid), selected by factor balance, simulated GP-EI histories, or highest source utility. Each condition covers the original three transfer settings, four utility tracks, and 100 initialization seeds. At the cross-task dollar budget, ALT has lower mean regret area than ScaML-EI in 35 of 36 comparisons; 29 paired 95% intervals favor ALT and none favors ScaML-EI. Averaging areas over the four tracks, the reduction is 4.0–10.5% across the nine conditions (App. D.6).

Practical search cost (W2). For a one-off run, a strong default may be preferable. ALT targets repeated deployment with historical runs already available and can update between successive target runs. We compare it with a fixed Fable-high default and a source-best reference that evaluates each source’s best configuration on the target and retains the best, counting those 1–4 evaluations. With full sources under task shift, normalized regret area at the dollar budget is 6.0% for ALT, 32.5% for source-best, and 34.2% for the Fable-high default under balanced utility. Across all four utility tracks, ALT reduces regret area by 68.9–81.9% relative to source-best and 75.0–87.2% relative to Fable-high (App. D.9). These improvements in configuration quality can benefit later deployments when the selected configurations are reused.

Why an adaptive exponent (Q1). BEH leads under task shift and itself uses an adaptive exponent, so we examine the exponent through ALT’s own fixed-exponent ablations. At the dollar budget, ALT reduces the regret area of the full-discount version (α = 1) by 2.5–9.1% under task shift. With mixed or same-task sources, it reduces the area of the no-discount version (α = 0) by 64–69% on the cost track. Full discounting still works better on the time and cost tracks in those settings, where ALT’s area is 8–27% higher. At 60 evaluations, ALT reduces the full-discount version’s area by 7–19% under task shift. The useful degree of discounting therefore depends on the sources, the utility, and the budget. The adaptive exponent remains a reasonable compromise because ALT is never the worst of the three in any setting or track at either budget (Table 2, App. D.3).

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Thanks a lot for your detailed response and experimental results. It addresses some of my concerns. But my main concerns still remain, the improvement does not seems to have a good advantage. I'll keep my score.

Thank you for reading our response and for your reply. To clarify the magnitude of the improvements and the rationale for the transfer design, we briefly summarize the evidence in the revised manuscript and its supplement.

  • Magnitude. In the full-source experiments under task shift, ALT's regret area at the dollar budget is 5–14% below that of ScaML-EI on all four utility tracks (Table 2). For the practical comparison with deployment without search, it is 69–87% below that of source-best deployment or a fixed default (App. D.9).
  • Consistency across baselines and source conditions (task shift). Against the six model-based baselines, ALT is significantly better in 118 of 144 full-source comparisons across six budgets and four tracks, and significantly worse in none. App. D.6 reports the partial-source study against ScaML-EI; to show that the advantage with incomplete sources is not specific to that baseline, we add the counts for all six, taken from the released results in the supplement (outputs/partial/paired.csv). Across nine partial-source conditions and four tracks at the dollar budget, ALT is significantly better in 179 of 216 comparisons and significantly worse in 1.
  • Role of BEH. BEH is ALT's behavior hypothesis used alone, so it directly tests the behavior-transfer idea motivated by Section 4, and its cross-task performance supports that part of the contribution. The rationale for combining transfer levels is their complementary performance across source settings: at the dollar budget with full sources, ALT reduces BEH's regret area by 23–76% when a same-task source is present, and that of objective-only ScaML-ADAPT by 3.4–12.1% under task shift. Each single-level variant is the worst of the three in one of these cases and ALT never is, which makes ALT a robust choice when it is not known in advance how well the sources match the target (Table 2, App. D.5).

Results with a fifth task are summarized separately in our general update. We would appreciate clarification on whether your remaining concern is the magnitude of these improvements, the benefit of the adaptive combination, or another aspect of the contribution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response Round 1

We thank the reviewers for their careful reviews and constructive insights, which helped us improve the method and make the scope of its claims more precise. In the revised work, our central motivation and contributions remain: using complete configuration landscapes and recorded agent trajectories, we identify comparatively stable configuration-to-behavior effects, motivating us to use target observations to combine behavior-level and objective-level transfer for cost-aware search. With full source data, the refined ALT retains its advantage under task shift over the main transfer baseline, ScaML-EI, with mean regret area at the dollar budget 5–14% lower on all four utility tracks. Across six budgets, four tracks, and six model-based baselines in that setting, ALT is significantly better in 118 of 144 comparisons and significantly worse in none. We summarize our revision below (W and Q refer to each reviewer’s numbered weaknesses and questions).

  1. For comments on hypothesis weighting and cost discount (gpAZ W1, 3vuN W2/Q2/Q3, uqEA Q1), we identified a scaling issue in the weighting and revised both components. In the submitted version, source utilities influenced the behavior prior’s scale, and the cost discount depended only on the behavior hypothesis. We now standardize that hypothesis using observed target runs and average the cost exponent across hypotheses with the same evidence weights, retaining likelihood-based weighting in common utility units. On identical nine-run initial designs, the mean weight on the behavior hypothesis under task shift rises from 0.54/0.29/0.15 to 0.67/0.52/0.40 with one/two/three sources. Aggregate performance changes modestly, and the central empirical conclusions remain. We retain the submitted version as an ablation.

  2. For comments on incomplete source records and search cost (uqEA W2, 3vuN W3/Q4), we added experiments with partial sources, and ALT’s dollar-budget advantage under task shift holds. We test 27/54/108 observations per source (11.1%/22.2%/44.4% of the configuration grid). These include subsets balancing each configuration factor’s levels, records selected sequentially by simulated BO searches on the recorded source landscapes, and highest-utility subsets. These nine source-data conditions each cover the same three transfer settings, four utility tracks, and 100 initialization seeds as the full-source study. Under task shift at the dollar budget, ALT has lower mean regret area than ScaML-EI in 35 of 36 condition-by-utility comparisons; 29 paired 95% intervals favor ALT and none favors ScaML-EI. With full sources under task shift, ALT achieves 69–87% lower regret area than deploying source-best configurations or a fixed default.

  3. For comments on sensitivity of the behavior representation (gpAZ W2/Q), we added a sensitivity study (App. D.7). To assess whether three behavior features are restrictive, we compare affine utility maps using 3/8/17 features and a nonlinear map using the same three features, with full and partial sources under balanced utility. Across the other-task, same-task and mixed settings, ALT’s mean regret area at the dollar budget differs from the three-feature default by less than 5% with full sources and by up to about 10% with partial sources; the nonlinear map provides no cross-task gain in these tests. The three-feature affine map remains our interpretable default.

  4. For comments on execution variation (3vuN W1/W3/Q1), we added a resampling study on the three Spooky repetitions (App. D.8). It uses 50 initial designs and 50 simulated search sequences per design to incorporate variation in both search initialization and run outcomes. At each search step, the selected configuration receives its performance, runtime, and cost together from one randomly chosen recorded repetition. We score each recommendation by its mean utility across all three repetitions. At the dollar budget, ALT reduces regret area by 5.2% relative to the target-only GP and 26.7% relative to random search, and performs similarly to ScaML-EI (0.9% lower mean regret area on average across four tracks, 95% interval −0.2% to 2.1%).

  5. For comments on scope and presentation (gpAZ W3, 3vuN W1/Q1, uqEA W1), we clarify that the study covers four tasks and one agent system. The same-task and mixed-source results rely on the three Spooky repetitions. We are collecting four further landscapes (scheduled for the camera-ready version), one each for TGS Salt Identification and Plant Seedlings Classification and two additional repetitions of Scaling Law. The revised introduction explains how the observed stability of configuration-to-behavior effects motivated BEH and its integration into ALT. We also revised Algorithm 1, the notation, and the regret definitions for better presentation.

Changes are marked in blue in the revised PDF. The reviewer-specific responses give the details, rationale, remaining limitations, and manuscript locations.

AuthorsOpenReview ↗

Additional Data Collection Status Update

We have completed the TGS Salt Identification landscape as an extra independent task, the first of the four further landscapes announced in our response, extending the benchmark to five tasks and seven landscapes. We ran the full-source and nine partial-source conditions for 56 additional cross-task scenarios (14 with TGS Salt as the target, 42 with it as an additional source), on top of the 57 scenarios of the submitted and revised versions, with the same 14 methods of Table 2, utility tracks, 100 initialization seeds, and evaluation protocol. All comparisons below concern cross-task transfer. Values show revised manuscript → expanded study; significance is determined by paired 95% bootstrap intervals.

ComparisonALT significantly betterNot significantALT significantly worse
Full sources, dollar budget: 28 comparisons24 → 264 → 10 → 1
Full sources, 60 evaluations: 28 comparisons22 → 276 → 10 → 0
Full sources, six budgets: 144 comparisons118 → 13326 → 70 → 4
Partial sources, dollar budget: 36 comparisons against ScaML-EI29 → 337 → 30 → 0

In the table above, the 28-comparison summaries include six model-based baselines and random search across four utility tracks; the 144-comparison summary excludes random search and covers all six budgets. At the dollar budget, ALT's regret-area reduction against ScaML-EI changes from 5–14% to 7–14% across the four tracks. With partial sources at the dollar budget, all 36 mean regret areas favor ALT. These results support the cross-task advantage over the main baseline with full and incomplete source records.

The supplement is unchanged and reproduces the revised manuscript. The manuscript remains the four-task, six-landscape version; the same-task, mixed-source, representation-sensitivity, objective-only ScaML-ADAPT, and execution-resampling studies retain their original scope. Full integration, with the remaining three landscapes, is planned for the camera-ready version if the paper is accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.