acceptodds
Under review as a conference paper at ICLR 2027

ADAPT for Science: A Multi-Agent System for Self-Evolving Scientific Optimization

Abstract

Scientific progress often depends on finding a small number of useful solutions in vast and constrained design spaces. Scientific automation and embodied intelligence have made experiments and simulations easier to execute. Yet deciding what to try next remains difficult when evaluations are costly, feasible solutions are sparse, and parameters are strongly coupled. Recent AI for AI methods can automatically improve algorithms and optimization procedures. However, general purpose approaches are not designed around scientific priors, physical constraints, and limited evaluation budgets. We introduce ADAPT for Science, a multi-agent framework that treats scientific optimization as two coupled decisions: where to search and how to search. ADAPT uses historical designs to identify promising and diverse initial conditions, progressively concentrates computation on favorable search directions, and evolves the optimization procedure from execution feedback. We evaluate ADAPT on the design of gravitational wave detectors using a differentiable physics simulator and eight unseen detector topologies. Under matched scientific evaluation budgets, ADAPT achieves a mean loss of , reducing loss by relative to the best classical optimizer and by relative to the best general purpose agentic evolution baseline. ADAPT achieves the lowest loss on all unseen topologies. ADAPT provides a step toward scientific optimization systems that continuously learn from prior experience and their own search outcomes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Design Optimization, Autonomous Agents, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wTmeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper treats scientific optimization as a search over the full search policy: selecting the starting design from a library of historical designs and evolving the optimizer program. They are searched by LLM agents, and the final policy is tested on eight unseen gravitational-wave detector topologies under a matched time budget.

Critical strengths

The where/how decomposition is clean. The limitations section shows the dependence on the historical experience library.

Critical weaknesses
  1. The claim that the search policy is "left to evolution" is not supported. The supplement's prompts already contain the shape of the final policy, its constants, and related hints, so it is unclear what evolution actually discovered.

  2. The baselines have no detail settings or initialization, and standard baselines for expensive black-box optimization such as Bayesian optimization and CMA-ES are missing.

  3. The title's scope is much broader than the evidence. All experiments are on a synthetic domain and a problem family, and there is no second task to support the general claim.

Questions for authors
  1. How many runs per method? Please report seed-level variance, especially on T2 where the margin over GEPA is 0.5% and over Sophia is 1.0%.

  2. Please give the full baseline configurations: learning rates, initialization, hardware, budget, and tuning protocol, and add a Bayesian-optimization or CMA-ES baseline.

  3. The introduction lists four factors but the method keeps two, both stages allocate budget, and the pipeline is sequential rather than jointly optimized. Why exactly where and how, and how does this relate to multi-start and algorithm-selection practice?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wTme

Q1. Evolutionary Discovery and Prompt Constraints

Thank you for raising this important concern. We have clarified what is prescribed by the scientific constraints and what is learned through evolution.

Scientific optimization requires explicit constraints to ensure valid and efficient exploration. Our prompts define the scientific objective, executable interfaces, evaluation budgets, and permissible search operations. They also suggest possible directions, including candidate screening, staged probing, and numerical safeguards, without prescribing the final algorithms. Within this search space, LLMs propose and refine executable policies based on measured scientific feedback.

The Selection Evolver determines the combination of four high-quality and four diversity-preserving candidates, the progressive 64-to-8-to-2-to-1 selection strategy, and the associated probing and budget-allocation rules. The Optimizer Evolver identifies a shorter momentum memory with β₁ = 0.7, a quadratic time-dependent update schedule, and explicit numerical state protection. These specific rules, parameter values, and their final implementations are selected through evolution rather than fixed by the prompts.

Figures 2 and 5 trace these successive changes from the reference implementations to the final algorithms, while Figure 4 presents the resulting executable optimizer. We have further added error bars from 10 random seeds to assess the stability of the evolution trajectories. These results clarify that ADAPT performs constrained, feedback-driven algorithm evolution, with scientific guidance defining the search boundaries and measured performance determining the resulting policies.

Q2. Baseline Configurations and Additional Comparisons

Thank you for this valuable suggestion. We have added Bayesian Optimization and CMA-ES as additional baselines and repeated all experiments across the same 10 random seeds with identical random starting points. All methods use matched scientific objectives and evaluation budgets of 4 hours per topology on eight NVIDIA H100 GPUs.

ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 4.765 ± 0.080 for Bayesian Optimization and 4.534 ± 0.094 for CMA-ES, corresponding to reductions of 95.6% and 95.4%, respectively. ADAPT also outperforms the other classical and agentic baselines under matched evaluation conditions.

We have updated Tables 1–3 in the main manuscript with the repeated-run results and added complete baseline configurations to the supplementary material, including initialization procedures, learning rates, hyperparameters, hardware, evaluation budgets, and tuning protocols.

AuthorsOpenReview ↗

Additional Responses to Reviewer wTme

Q3. Cross-Domain Generalizability

Thank you for highlighting this limitation. We have clarified the generalizable components of ADAPT and introduced a second scientific optimization task to evaluate its transferability.

ADAPT provides a reusable framework for historical experience construction, initialization selection, and optimizer evolution. Its core workflow is domain-independent. Applying ADAPT to a new scientific problem requires adapting the domain-specific data, retrieval and optimization interfaces, and prompts while preserving the overall evolution framework.

To validate its transferability, we introduced a second case study on aerodynamic airfoil optimization with 21 continuous variables, comprising 20 free-form deformation parameters and one angle of attack. The task minimizes aerodynamic drag under lift, area, and thickness constraints using a physics-based simulator. We constructed a library of 1,000 historical designs and evaluated ADAPT on eight held-out targets using 32 retrieved initializations. All methods were evaluated across the same 10 random seeds, with identical random initializations for their random components and matched evaluation budgets. ADAPT achieves a mean penalized objective of 0.171 ± 0.091, reducing the penalized objective by 66.2% relative to randomly initialized trust-constr and by 53.8% relative to the strongest agentic evolution baseline. This result extends our evaluation beyond gravitational-wave detector design and provides further evidence of the framework's applicability across scientific domains.

We have incorporated this experiment into Section 4.6 of the revised manuscript and clarified the distinction between reusable framework components and domain-specific adaptations.

Q4. Repeated-Run Reliability

Thank you for highlighting this concern. We have repeated all methods across the same 10 random seeds with identical random starting points and matched evaluation budgets. The revised tables now report seed-level variability for each topology.

On T2, ADAPT achieves a mean loss of 0.408 ± 0.025, compared with 0.422 ± 0.012 for GEPA and 0.480 ± 0.068 for Sophia. Although individual methods may perform better on specific runs or topologies, ADAPT achieves the lowest overall mean loss of 0.208 ± 0.098, compared with 0.568 ± 0.232 for GEPA and 0.739 ± 0.302 for Sophia. These results demonstrate consistent overall performance across repeated runs, without assuming uniform superiority on every topology.

We have updated Tables 1–3 in the main manuscript with the complete 10-seed results. The additional experiments provide stronger evidence of the effectiveness and robustness of ADAPT.

Q5. Policy Decomposition and Methodological Contributions

Thank you for this insightful comment. We have clarified the rationale for the policy decomposition and highlighted the methodological contributions of ADAPT beyond established optimization approaches.

The four factors identified in the Introduction are organized into two complementary decisions. The selection policy determines where to search by combining historical retrieval, initialization selection, and resource allocation across candidate trajectories. The optimizer determines how to search through update rules, numerical safeguards, and the use of the remaining computational budget. Both stages share a single evaluation budget but operate at different decision levels.

ADAPT adopts a sequential evolution strategy. Simultaneous joint optimization would introduce a large, highly coupled search space, substantially increasing computational cost and optimization complexity, particularly for expensive scientific evaluations. The Selection Evolver first identifies a promising initialization policy. The Optimizer Evolver then develops an optimization procedure conditioned on the selected starts. This decomposition reduces search complexity while retaining the complete search procedure as the optimization target. We have clarified this design choice in the revised manuscript.

ADAPT advances multi-start optimization and algorithm selection through feedback-driven policy evolution. It unifies historical scientific experience, adaptive initialization, resource allocation, and executable optimizer evolution within a single framework. This integration offers a new direction for AI-driven optimization and, to our knowledge, represents its first application as a unified approach to scientific optimization under physical constraints and limited computational budgets.

I appreciate the authors' efforts in addressing the comments. I have some more questions:

  1. What are the values of the baseline constants injected in the Selection Evolver prompt, and how does ADAPT compare against a same-budget, one-shot implementation of the prompt-suggested shortlist-and-race configuration without any evolution?
  2. Please recompute the aggregate Mean±Std column under a single stated convention (Table 1 baseline Std values equal the across-topology std of the per-topology means, whereas ADAPT's 0.013 does not), and reconcile the "lowest loss on all eight topologies" claim with OpenEvolve's lower T2 loss (0.321 vs. 0.408).
Reviewer_tA9fOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how to improve scientific optimization under a limited budget by adapting initialization selection and optimization procedures. ADAPT combines historical-design retrieval with evolution of selection and optimization programs. It outperforms the evaluated baselines on eight held-out gravitational-wave detector topologies and examines its components through ablations.

Critical strengths
  1. Studying where to start and how to optimize under a limited budget is interesting and practically relevant.
  2. The paper is clearly written, and the connection between the problem, method, and experiments is easy to follow.
Critical weaknesses
  1. The experiment cover only eight topologies in gravitational-wave detector design, making this closer to a case study. Topology retrieval and parameter transfer are also specific to this setting. Although the limitation is acknowledged, it remains unclear which designs would transfer, so support for the broader science framing is limited.
  2. The paper emphasizes initialization, but does not systematically examine sensitivity to the candidate count, topology distance and related settings. Component ablations and evolution traces do not establish their robustness. Comparisons are needed to assess how much the gains depend on these choices.
Questions for authors

Please see the questions raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tA9f

Q1. Cross-Domain Generalizability

Thank you for highlighting this limitation. We have clarified the generalizable components of ADAPT and introduced a second scientific optimization task to evaluate its transferability.

ADAPT provides a reusable framework for historical experience construction, initialization selection, and optimizer evolution. Its core workflow is domain-independent. Applying ADAPT to a new scientific problem requires adapting the domain-specific data, retrieval and optimization interfaces, and prompts while preserving the overall evolution framework.

To validate its transferability, we introduced a second case study on aerodynamic airfoil optimization with 21 continuous variables, comprising 20 free-form deformation parameters and one angle of attack. The task minimizes aerodynamic drag under lift, area, and thickness constraints using a physics-based simulator. We constructed a library of 1,000 historical designs and evaluated ADAPT on eight held-out targets using 32 retrieved initializations. All methods were evaluated across the same 10 random seeds, with identical random initializations for their random components and matched evaluation budgets. ADAPT achieves a mean penalized objective of 0.171 ± 0.091, reducing the penalized objective by 66.2% relative to randomly initialized trust-constr and by 53.8% relative to the strongest agentic evolution baseline. These results extend our evaluation beyond gravitational-wave detector design and provide further evidence of the framework's applicability across scientific domains.

We have incorporated this experiment into Section 4.6 of the revised manuscript and clarified the distinction between reusable framework components and domain-specific adaptations.

Q2. Initialization Sensitivity and Robustness

Thank you for this insightful suggestion. Your comment prompted us to further examine the relationship between optimization dimensionality and initialization pool size. We have expanded our experiments to assess initialization sensitivity and robustness.

We evaluated candidate counts of 32, 64, and 128 using the same ADAPT pipeline, changing only the candidate count. Detailed per-topology results are provided in Supplementary Appendix C.3. All three configurations were repeated across 10 random seeds with identical random starting points and matched evaluation budgets. The corresponding mean losses are 0.245 ± 0.109, 0.208 ± 0.098, and 0.270 ± 0.103, respectively. The 64-candidate setting achieves the lowest mean loss. Inspired by hierarchical selection, we further propose a dimension-dependent heuristic, K(d)=2⌈log⁡2d⌉K(d) = 2^{\lceil\log_2 d\rceil}, which facilitates progressive binary screening. We applied this rule to our second case study on aerodynamic airfoil optimization with 21 continuous variables, using 32 initial candidates. ADAPT achieves a mean penalized objective of 0.171 ± 0.091, reducing the penalized objective by 66.2% relative to randomly initialized trust-constr and by 53.8% relative to the strongest agentic evolution baseline.

For topology retrieval, we use a fixed, physically informed structural distance based on optical component types, orientations, and boundary configurations. This rule is interpretable and requires no learned topology representation. Its definition and parameter-transfer procedure are detailed in Section 3.3 and Supplementary Appendix B.

To further assess robustness, we repeated all classical and agentic baselines and component ablations across 10 random seeds under matched experimental conditions. We also introduced Bayesian Optimization and CMA-ES as additional baselines. ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 0.432 ± 0.236 without evolved initialization selection and 0.259 ± 0.102 with fixed Adam. We also added error bars from the 10-seed experiments to the evolution trajectories in Figures 2 and 5 to assess their stability. These results support the robustness and complementary contributions of the proposed components. We have updated Tables 1–3 and Figures 2 and 5 with repeated-run results.

Thank you for your rebuttal and experiments. While these efforts address some of my concerns, I still think that the framework relies on several predefined thresholds and domain-specific components, and I therefore remain concerned about its generalizability and transferability. Of course, this is partly a matter of personal preference. I maintain my original score.

AuthorsOpenReview ↗

Additional Clarification on Generalizability and Evolutionary Discovery

Thank you for your thoughtful follow-up and for acknowledging our additional experiments. We respect your perspective and agree that generalizability and transferability are important criteria for evaluating scientific optimization frameworks.

First, we believe that our new cross-domain experiment provides meaningful evidence of transferability. Gravitational-wave detector design and aerodynamic shape optimization involve different physical systems, design representations, simulators, and constraints. ADAPT achieves substantial improvements in both domains while preserving the same core framework for historical experience, initialization selection, and optimizer evolution. We acknowledge that two scientific domains do not establish universal generalizability, but we believe these results demonstrate that ADAPT extends beyond a single domain-specific case study.

Second, we would like to clarify the role of predefined components and evolutionary discovery. Scientific optimization necessarily requires domain-specific objectives, physical constraints, and representations. These components define the scientific problem. The selection and optimization policies are developed through execution-based evolution.

As illustrated in Figures 2 and 3, the Selection Evolver begins with direct 64-to-1 selection. Successive iterations discover quality-diversity shortlisting, staged optimization probes, progressive budget allocation, and protected exploration. Mean loss decreases from 4.3215 to 3.0054 across these iterations. The prompts specify scientific constraints and suggest possible directions, but do not prescribe the final executable policy. Similarly, the Optimizer Evolver discovers shorter momentum memory, time-dependent update scaling, and numerical safeguards through execution feedback.

We fully respect your high standards for generalizability and appreciate that this is partly a matter of research perspective. We hope these additional experiments and clarifications help distinguish the transferable framework from its necessary domain-specific interfaces. We would be grateful if you could reconsider your assessment in light of this additional evidence.

Thank you again for your time and constructive feedback.

Thank you for the further clarification. I was genuinely torn between giving you a 2 or a 3, as part of my assessment reflects my own research taste, and I understand that such taste varies among researchers. To encourage the additional clarification and experiments you have made, I have decided to raise my score to 3 while lowering my confidence to 1. I would suggest that the AC base the final decision primarily on the review comments of the other two reviewers.

Reviewer_DJC5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ADAPT for Science combines historical design retrieval with evolution of an initialization-selection policy and an optimization program. For gravitational-wave detector design, it retrieves 64 compatible initializations from 15,409 historical topology–design records, evolves a staged search policy, and evolves an Adam-based optimizer with shorter momentum memory, time-dependent update scaling, and numerical safeguards. The final policies are evaluated on eight held-out detector topologies under a shared wall-clock budget that includes candidate screening and warmup. The reported mean loss is 0.204, compared with 0.452 for the strongest classical baseline and 0.540 for the strongest agentic-evolution baseline. Ablations examine start selection, optimizer evolution, and scientific guidance.

Critical strengths
  1. The application is a meaningful test of optimization with reusable scientific experience. The retrieval and parameter-transfer rules are concrete, and the paper excludes historical records with the exact target topology. Counting initialization screening and staged probing within the final run budget is also an important experimental choice. The reported gains occur across all eight test topologies rather than being driven solely by one favorable example.
  2. The discovered procedures are interpretable, and the component ablations provide useful evidence about their contributions. With the selected start retained, fixed Adam reaches a mean loss of 0.256 versus 0.204 for the full method, while removing evolved start selection gives 0.423. These comparisons indicate that both initialization and subsequent optimization matter, with initialization accounting for a substantial part of the gain. Showing the resulting executable Adam modifications makes the output of the evolution process more tangible.
Critical weaknesses
  1. The boundary between policy development and held-out evaluation is ambiguous in the supplied manuscript. Sections 3.4–3.5 describe target-indexed policies whose evolution can use target evaluations, whereas Sections 3.2 and 3.6 state that held-out targets and outcomes are excluded from strategy development and never returned to the evolvers. A policy can legitimately use objective calls during deployment, but those calls must be distinguished from feedback that changes the policy code or selects among workflow–model configurations. Without an explicit split and chronology, it is difficult to tell whether the eight-topology result tests transfer of frozen policies, target-specific policy search, or a combination. The PDF supplied for this review has 12 pages and does not contain the supplementary details or Appendix E referenced in the text; this concern is therefore conditional on what those materials provide.
  2. The comparisons do not establish how much benefit requires the proposed evolution system beyond strong initialization and ordinary optimizer tuning. The final optimizer is close to Adam, with beta1=0.7, a quadratic time schedule, and finite-value guards. Fixed+Adam is helpful, but does not compare a conventionally tuned Adam/NAAdam using the same selected starts, safeguards, and development allowance. It is also unclear whether the main baselines share the historical candidate pool and comparable tuning resources. Finally, the reported standard deviations summarize variation across eight topologies, not repeated-run uncertainty. These gaps limit both the causal attribution and the reliability of claims about the evolution framework.
Questions for authors
  1. Please specify the topology split and the exact chronology of retrieval, selection-policy evolution, optimizer evolution, configuration selection, and final evaluation. On a held-out topology, what is frozen, what can change, and which target feedback reaches an LLM?
  2. What are the numerical development and deployment budgets, including the 16 workflow–model/reference configurations? Please separate offline evolution cost from the final run budget and explain which historical data and tuning resources each baseline receives.
  3. Can you compare the full method with a tuned Adam or NAAdam that uses the same retrieved/selected starts, finite-value safeguards, and development budget? This would test whether policy evolution adds value beyond conventional tuning of the discovered procedure.
3 comments
AuthorsOpenReview ↗

Response to Reviewer DJC5

Q1. Development and Evaluation Separation

Thank you for highlighting this ambiguity. We have clarified the distinction between policy development and held-out evaluation in the manuscript.

Our protocol strictly separates development and evaluation targets. Eight detector topologies were reserved exclusively for final evaluation. A separate panel of eight development topologies was used to provide objective feedback during policy evolution. The historical library contains 18,024 transferable designs and excludes the held-out targets.

During development, the Selection Evolver and Optimizer Evolver use feedback from the development panel to refine their policies. The 16 workflow configurations are compared at this stage. Both the selection policy and the optimization program are frozen before final evaluation.

On the eight held-out topologies, the frozen policies use target-specific objective evaluations only to select initializations and optimize detector parameters. These evaluations do not modify the policy code or trigger further configuration selection. No held-out feedback is returned to either evolver or used for hyperparameter tuning. Thus, the reported results evaluate the transfer of frozen policies to unseen topologies, rather than target-specific policy evolution.

We have revised Sections 3.2–3.6 to make the experimental split and chronology explicit.

We also confirm that the supplementary material was submitted through OpenReview. It provides the development and evaluation protocols in Appendices B–F, including the complete evolution prompts. The additional held-out examples previously referenced as Appendix E are now provided in Appendix H. We regret that these materials were not accessible during the review and hope they can be made available during the rebuttal period.

AuthorsOpenReview ↗

Additional Responses to Reviewer DJC5

Q2. Component Contributions and Reliability

Thank you for this valuable comment. We have conducted additional experiments across ten random seeds, introduced two classical baselines, and updated the main results and ablation tables.

Our ablations isolate the contributions of initialization selection and optimizer evolution. The evolved optimizer retains the Adam structure but introduces shorter momentum memory, a quadratic time-dependent update schedule, and numerical safeguards. With identical selected starts, ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 0.259 ± 0.102 for Fixed+Adam, a 19.7% reduction. Removing evolved initialization selection or scientific guidance increases mean loss to 0.432 ± 0.236 and 0.318 ± 0.153, respectively. These results demonstrate the complementary contributions of the individual components.

The historical library and evolution pipeline are integral contributions of ADAPT. We constructed a library of 18,024 historical scientific designs to enable experience-guided optimization. Classical baselines follow their standard initialization procedures, while agentic baselines, including GEPA and OpenEvolve, have access to the same library but do not use ADAPT's evolved 64-to-1 selection policy. All methods share identical random starting points, matched random seeds, hardware, scientific objectives, and final evaluation budgets. The historical library is a component developed specifically for ADAPT, rather than an external resource available only to our method. Thus, the main comparisons assess the benefits of the complete framework under matched evaluation conditions, while the ablations isolate the contributions of individual components.

We further introduced Bayesian Optimization and CMA-ES and repeated all experiments across ten seeds. ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 0.518 ± 0.104 for NAAdam, 0.870 ± 0.443 for Adam, 4.765 ± 0.080 for Bayesian Optimization, and 4.534 ± 0.094 for CMA-ES. The updated standard deviations reflect repeated-run variability rather than variation across topologies alone. These results strengthen the evidence for the effectiveness and reproducibility of ADAPT.

Q3. Experimental Chronology and Feedback

Thank you for this question. We have clarified the experimental split, development sequence, and feedback boundaries in the revised manuscript.

We reserve eight detector topologies exclusively for final evaluation. A separate panel of eight development topologies provides feedback for policy evolution. The historical library contains 18,024 scientific designs and excludes the held-out targets.

The procedure follows a fixed sequence. We first construct the historical library and establish the target-aware retrieval mechanism. The Selection Evolver then develops initialization policies using feedback from the development topologies. The Optimizer Evolver subsequently evolves optimization programs using the selected initializations. Policies are compared across 16 workflow configurations using development feedback. During evolution, LLMs receive execution summaries and composite scores reflecting solution quality, stability, improvement, and computational efficiency. The selected policies, optimizer program, and configuration are frozen before final evaluation.

On each held-out topology, the frozen retrieval mechanism identifies 64 compatible historical designs. The selection policy uses target-specific evaluations to select an initialization, followed by numerical optimization using the frozen optimizer. Only the selected initialization, optimization trajectory, and numerical states can change. Policy code, optimizer rules, and configurations remain fixed. No held-out feedback reaches an LLM or informs further policy evolution, configuration selection, or hyperparameter tuning.

The results therefore measure the generalization of frozen policies rather than additional evolution on test targets. The complete protocol and feedback boundaries are detailed in Appendices B–F.

AuthorsOpenReview ↗

Additional Responses to Reviewer DJC5

Q4. Computational Budgets and Baseline Fairness

Thank you for raising this question. We have clarified the computational budgets, experimental protocols, and resources available to each baseline.

During offline development, we evaluate 16 configurations comprising four evolution frameworks, two language models, and two reference optimizers. Selection evolution uses eight RTX 4090 GPUs with a fixed 4-hour budget per development topology, including up to 7200 seconds for initialization selection and the remaining time for optimization. Each iteration requires 32 RTX 4090 GPU-hours per configuration, totaling 512 across all configurations. After freezing the selection policy, optimizer evolution uses the same eight GPUs with a fixed 2-hour budget per topology, requiring 16 GPU-hours per configuration and 256 per iteration. Both stages undergo 3 to 10 iterations, yielding a total offline development budget of 2,304–7,680 RTX 4090 GPU-hours.

After development, the selected policies and optimizers are frozen. Final evaluation uses eight NVIDIA H100 GPUs, with a fixed budget of 4 hours per held-out topology, totaling 32 H100 GPU-hours. This budget includes initialization screening, staged probing, and subsequent optimization. Offline evolution costs are accounted for separately and excluded from the final evaluation budget. These protocols and computational settings are detailed in Appendices C and F.

The historical experience library and evolution pipeline are central contributions of ADAPT. We constructed a library of 18,024 historical scientific designs to support experience-guided optimization. Classical baselines, including Adam and NAAdam, use standard initialization without this library. Agentic evolution baselines, including GEPA and OpenEvolve, have access to the same historical library but use random initialization without ADAPT's evolved 64-to-1 selection policy.

The main comparisons assess the complete ADAPT framework under matched evaluation budgets. We further isolate the contributions of individual components through ablation experiments. Removing evolved initialization selection increases mean loss from 0.208 to 0.432. Replacing the evolved optimizer with fixed Adam increases mean loss to 0.259. Removing scientific guidance increases mean loss to 0.318. These results demonstrate the complementary contributions of experience-guided initialization, optimizer evolution, and scientific guidance. Further details are provided in Appendix G.

Q5. Comparison with Conventional Optimizers

Thank you for this suggestion. We have strengthened the comparison by repeating all experiments across ten random seeds with identical random starting points and matched evaluation budgets. We also introduced Bayesian Optimization and CMA-ES as additional classical baselines and updated the results in the main manuscript.

ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 0.518 ± 0.104 for NAAdam, 0.870 ± 0.443 for Adam, 4.765 ± 0.080 for Bayesian Optimization, and 4.534 ± 0.094 for CMA-ES. Furthermore, our Fixed+Adam ablation uses the same selected initializations as ADAPT and achieves 0.259 ± 0.102. The 19.7% lower mean loss of ADAPT provides evidence that optimizer evolution contributes beyond initialization selection alone.

These results strengthen the evidence for the effectiveness of ADAPT under controlled random initialization. We have incorporated the additional comparisons and repeated-run statistics into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence for the Area Chair

We thank the Area Chair and reviewers for their constructive feedback. ADAPT for Science introduces a unified framework that evolves both where to search and how to optimize, combining historical scientific experience, adaptive initialization selection, and executable optimizer evolution under limited computational budgets. We evaluate this framework on gravitational-wave detector design and aerodynamic shape optimization. In response to the reviews, we conducted extensive additional experiments, introduced a second scientific case study, strengthened baseline comparisons and robustness analyses, and comprehensively revised both the main manuscript and appendices.

1. Cross-domain validation. We introduced a second scientific optimization task on aerodynamic airfoil design with 21 continuous variables, a library of 1,000 historical designs, and eight held-out targets. ADAPT achieves a mean penalized objective of 0.171 ± 0.091, improving upon randomly initialized trust-constr by 66.2% and the strongest agentic baseline by 53.8%. This experiment extends our evaluation beyond gravitational-wave detector design and provides evidence of transferability to a distinct physics-based optimization problem.

2. Extensive repeated experiments and stronger baselines. We repeated all methods and component ablations across 10 matched random seeds, with identical random starting points and evaluation budgets. We additionally introduced Bayesian Optimization and CMA-ES. On gravitational-wave detector design, ADAPT achieves a mean loss of 0.208 ± 0.098, compared with 0.518 ± 0.104 for NAAdam, 0.870 ± 0.443 for Adam, 4.765 ± 0.080 for Bayesian Optimization, and 4.534 ± 0.094 for CMA-ES. We updated the main tables and evolution figures to report repeated-run variability.

3. Component attribution and initialization robustness. We expanded the ablation experiments to isolate the contributions of initialization selection, optimizer evolution, and scientific guidance. With identical selected initializations, ADAPT reduces mean loss by 19.7% relative to Fixed+Adam. We further evaluated initialization pools of 32, 64, and 128 candidates under matched conditions, identifying 64 as the best-performing configuration in the gravitational-wave task. These experiments provide additional evidence for the contributions and robustness of the proposed components.

4. Comprehensive revisions to the main manuscript and appendices. We clarified the complete experimental chronology, the construction of the 18,024-design historical library, baseline access to historical experience, and the separation between policy development and held-out evaluation. We detailed the 16 development configurations, computational budgets, complete evolution prompts, and executable optimization procedures. Both evolved policies are frozen before final evaluation, and no held-out feedback is returned to the LLMs. We also expanded the appendices with experimental protocols, candidate-count sensitivity results, and detailed specifications of the second scientific task.

Together, these additional experiments and comprehensive revisions directly address the reviewers' principal concerns regarding generalizability, experimental reliability, baseline fairness, and methodological contributions. We sincerely appreciate the reviewers' constructive suggestions and hope the strengthened scientific evidence will support a favorable assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.