acceptodds
Under review as a conference paper at ICLR 2027

WingBench: Benchmarking How AI Agents Search for Better Designs

Abstract

Design search often uses cheap learned predictions to choose which candidates deserve expensive tests, but these tests can fail, especially on the unusual designs that optimization favors. We introduce WingBench, a multifidelity benchmark in which optimizers and AI agents design transonic wings with computational fluid dynamics (CFD) inside the search. They can query a learned aerodynamic surrogate and spend a fixed budget of sequential CFD evaluations, each returning a score or a failure type. WingBench grades the design each method recommends, reporting separately whether it receives a valid score and how good that score is. Using WingBench, we contribute a study of five baseline optimizers and a controlled comparison of three AI agent methods, which produced 6,629 CFD evaluation records. Feedback helps baseline optimizers deliver valid designs when it changes their later proposals, but the differential-evolution variant does so with lower aerodynamic quality. An open-ended research agent is competitive with the strongest baseline optimizers over 32 CFD queries, and in early search, our reflection and planning implementations do not consistently improve on it. We release the data, code, agent prompts, and analysis scripts for reproducibility.

Primary area:
datasets and benchmarks
Keywords:
optimization agents, large language model agents, benchmarking, multifidelity optimization, black-box optimization, scientific machine learning, computational fluid dynamics, aerodynamic design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jYTEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WingBench, a multifidelity benchmarkthat includes both optimizers and AI agents designing transonic wings with CFD tools. The authors use WingBench to study five baseline optimizers and a controlled comparison of three AI agent methods.

Critical strengths
  1. Each method proceeds through a defined sequence of design attempts and must submit a final wing. This is great for comparison purposes. The paper reports how often the method succeeds in submitting a valid wing and how good that wing is.
  2. The 6k recorded evaluations, along with reasons for failure and reply capabilities, makes the results inspectable and reviewable.
Critical weaknesses

I worry that methods are not compared on an apples-to-apples basis. For instance, the four standard optimizers receive 60 CFD results, while the agent does not (unless I'm mistaken). Likewise, some methods can make intermediate surrogate model calls while others do not.

Questions for authors

How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?

3 comments
AuthorsOpenReview ↗

Initial CFD information and evaluation allowances

Thank you for recognizing the value of the evaluation protocol and records, and for raising the question of comparability.

I worry that methods are not compared on an apples-to-apples basis. For instance, the four standard optimizers receive 60 CFD results, while the agent does not (unless I'm mistaken). Likewise, some methods can make intermediate surrogate model calls while others do not.

The four surrogate-assisted baselines receive 62 prior CFD records that the agents do not receive, and the cross-family comparison does not match total computation or wall-clock limits. The revised manuscript makes these conditions explicit and reports resource usage from the existing records.

Section 5 and its resource-conditions table distinguish the experimental settings. The four surrogate-assisted baselines share the same 62-record context, an allowance of 32 online CFD queries, and 4,096 surrogate evaluations per episode. Their allocation of surrogate evaluations and use of CFD feedback differ. SMAC3 is a separate CFD-only reference with no warm start or surrogate access.

The three agent implementations share the SuperWing source data, language model, tools, and evaluation allowances, and are compared at the common four-query checkpoint. The four-query comparison evaluates an intermediate checkpoint within the original 32-query task; the agents were not instructed to optimize specifically for a four-query horizon. They do not receive the baselines' 62-record context. Planning and reflection introduce additional model calls, so these shared conditions do not imply equal computational expenditure. Appendix D describes the executed implementations, including bounded recoveries.

Appendix B distinguishes surrogate access and allowances from actual expenditure. Mean charged surrogate evaluations per episode at the common four-query checkpoint are:

MethodMean charged surrogate evaluations
Each archive-only CMA-ES variant4,096
Each feedback-guided CMA-ES/DE method512
SMAC30
Open-ended research912.8
Planning704.1
Reflection828.3

These are actual expenditures up to query 4; the 4,096-surrogate allowance applies to the full episode. Appendix B specifies the counting conventions, including uncharged agent cache hits.

The open-ended extension continues the existing 30 episodes to query 32 within the same total surrogate allowance. At query 32, open-ended research averages 1,988.9 charged surrogate evaluations, compared with 4,096 for each surrogate-assisted baseline.

The within-baseline comparisons nevertheless answer a controlled question about how the tested strategies use feedback. The CMA-ES variants share the same initial CFD information and evaluation allowances. In the exploratory comparison pooled across tasks, feedback-guided CMA-ES improves recommendation delivery over either archive-only variant by 57 percentage points, with a descriptive 95% bootstrap interval of [37, 77] percentage points. This supports the comparison of those search strategies under the stated protocol.

The agent comparisons likewise evaluate the implemented approaches under shared source data, model, tools, and evaluation allowances, with their differing model-call and surrogate expenditures reported explicitly. The revised abstract, results, and conclusion interpret the findings according to these experimental conditions.

AuthorsOpenReview ↗

Matched-resource rankings and additional measurements

We appreciate this specific question about rankings under matched resource limits.

How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?

The reported rankings compare the tested configurations under their stated information and evaluation budgets. Determining how those rankings change under matched initial information and common compute or wall-clock limits requires a separate controlled comparison.

Appendix B reports online and private CFD requests, charged surrogate evaluations, model calls and tokens, reference model charges, recorded session durations, and available CFD timing. At the common four-query checkpoint, total recorded session duration divided by the number of online CFD requests is:

Agent methodSummed recorded session duration / online CFD requests (minutes per request)
Open-ended research30.97
Planning25.01
Reflection31.40

The numerator sums the durations of recorded sessions, with overlapping sessions contributing separately. It is therefore not a single elapsed episode duration. These measurements include tool execution and local code; they are not pure model inference or end-to-end wall time.

The resource accounting explicitly identifies quantities that were not consistently measured across the full study, including optimizer CPU/GPU usage, model-server compute, and total elapsed wall time.

We are conducting a focused set of additional runs for selected baseline and agent configurations under the original search settings and evaluation budgets. These runs will record end-to-end wall-clock time, CFD and decision-stage time, optimizer CPU/GPU usage, surrogate evaluations, and model calls and tokens.

We will report these measurements alongside recommendation delivery and conditional quality from the same runs, specifying the methods, tasks, query checkpoints, number of repetitions, and execution conditions. Server-side compute will be reported separately where directly measurable.

They will clarify resource costs rather than establish fully matched rankings. We will report the measurements in a follow-up response once the runs are complete.

AuthorsOpenReview ↗

Measured costs and results

How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?

Thank you for suggesting this analysis. The additional runs are completed and support competitive open-ended agent performance under the original search budgets. At four CFD queries, open-ended research delivered valid recommendations in 9/9 runs across the three tasks, with high-load mean L/D 30.97 versus 24.14 for feedback-guided CMA-ES. We have added this study to the revised manuscript, distinguishing CMA-ES comparisons with shared initial information and evaluation allowances from cross-family results under the original search budgets.

We tested each method and load below with three seeds, for 24 initiated runs in total. Each method kept its 32-query budget and 4,096-surrogate limit. We recorded recommendations at 0, 1, 2 and 4 CFD queries. The agents were instructed to optimize over the usual 32-query budget; we stopped the runs after query 4.

Results after four CFD queries. Success counts include all three runs in each group. Mean L/D includes only valid recommendations. Each time column is a separate median for runs that reached query 4: two for planning, three for every other row.

Method / loadValid/3Mean L/DSearch CFD hGrading CFD hLocal CPU h
Feedback-guided CMA-ES / low3/317.174.120.070.47
Open-ended / low3/317.3510.892.621.73
Feedback-guided CMA-ES / nominal3/330.326.114.540.49
Selection-only CMA-ES / nominal1/332.870.180.183.25
Open-ended / nominal3/330.579.212.632.22
Planning / nominal2/330.8710.842.322.08
Feedback-guided CMA-ES / high3/324.1413.192.820.55
Open-ended / high3/330.9712.742.943.06

The open-ended agent and feedback-guided CMA-ES both returned valid wings in 9/9 runs. Their mean L/D was close at low and nominal load. At high load, the agent reached 30.97 versus 24.14, while using more local CPU time. Selection-only CMA-ES had the highest nominal mean L/D but returned a valid wing in only 1/3 runs. Its time medians include the two runs without valid recommendations.

Compute use. We used 94 search CFD evaluations (23 x 4 + 2) and 43 additional evaluations to grade recommendations. Private grading does not count toward the search budget. If a recommendation already had a search result, we reused it.

By query 4, feedback-guided CMA-ES used 512 surrogate evaluations per run; selection-only used all 4,096. Open-ended runs used 643-2,306, and the two completed planning runs used 1,068-1,471. Across all 12 agent runs we recorded 2,653 model calls, 397.61 million input tokens (including 396.02 million cached) and 2.75 million output tokens.

Search decisions took 25-32 minutes for feedback-guided CMA-ES and 156-248 minutes for selection-only. Agent research and review took 76-252 minutes per run.

CFD hours sum each evaluator's start-to-finish execution time, separately for search and recommendation grading; they are not processor-core hours. Local CPU hours sum user and system CPU time across the baseline controller process tree, or agent research/review and episode-service process trees, including surrogates. This is accumulated CPU time, not elapsed time, and excludes remote CFD and model-provider computation. No local GPUs were used. Model-provider usage is reported as calls and tokens.

CFD and baselines ran on AWS m7a.4xlarge machines (16 vCPUs); agents and surrogates shared a CPU-only r7a.12xlarge machine. Each batch allowed up to 12 concurrent CFD jobs.

Reviewer_Z8EbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a benchmarking framework and set of baseline comparisons for the task of Wing Optimization. It's key claim contributions are the benchmarking framework, which includes trained surrogates plus a pass through to ADFlow and associated data, a comparison study across different gradient-free optimizers (mostly evolutionary and BayesOpt) alongside those that leverage agents (DeepSeek v4.1 Flash in this case) as the optimizer. The paper's overall goal is to improve benchmarking and evaluation for Wing Design.

Critical strengths

I appreciated the efforts of the authors to address an important and interesting area that brings additional emphasis to physical problems and design to the machine learning community. These kinds of papers are generally the ones that I want to support and grow within the wider community, and I want to encourage the authors to continue work in this area.

Critical weaknesses

As much as I, in general, like these kinds of papers, unfortunately, I don't think that this particular study has been executed at sufficient rigor such that I could find it acceptable. Per the ICLR template, I am limiting my review to only critical and broad reaching weaknesses, and thus didn't have the space to go into every particular.

On the Benchmarking framework itself (before the optimizers/agents), there are several issues that would make me doubt the results or not want additional papers to build their analyses on this benchmark. First, on the high fidelity evaluator, something is very clearly wrong with the implementation. Over 4000 of the 6629 evaluations "delivered" no score, which the paper says can result from unmeshable geometry, incomplete meshing, or an unconverged flow solver. Having worked on very similar problems and solvers, the overall reported failure rate (close to 2/3rds of the runs) should be nowhere near this high if the entire pipeline was implemented correctly and allowed to properly converge (typically rates would be <5-10% max). This makes me very skeptical of the reported results, to be honest, and would also make me unlikely to recommend the framework as a "ground truth" evaluator for ML tasks, where this paper/repo to be published. I understand that the paper might using an available pipeline from the previously published SuperWing paper, but whether they used it off the shelf or modified it in some way, the currently reported results significantly undermine my confidence that the foundational library/evaluator implementation is correct. I would almost argue that accepting the current implementation and letting folks build new papers upon it might be more harmful to progress in the field than not having it.

On the second contribution of the optimizer baselines: these baselines are simply not at all competitive baselines within the community that would study these problems. You would not take the 38 dimensional representation in ADFlow, and then run evolutionary algorithms BayesOpt on that type of space, in part because optimization convergence will be extremely slow (as can already be seen in Figure 4), and the actual people who do wing optimization use either (a) adjoint methods (which would be appropriate for the flow regimes studied here) or (b) gradient free methods but only in reduced dimension spaces. So the existing optimizer baseline study doesn't tell us much in terms of advancing the field forward.

The third contribution asks Agent's to perform optimization, and attempt to reflect upon feedback from tool calls and failure cases. This actually had the potential to be very interesting, if more powerful optimizers or different kind of data were fed to the agent or it was trying to learn to use existing, more state of the art optimizers effectively. This is probably one area of future work that could be quite interesting and that I would encourage the authors to pursue. However, in the current manuscript, both the tooling available to the agent (the AeroTransformer surrogate plus the above mentioned simulator) and how the agent is directed to analyze the results are very surface level. The paper doesn't provide concrete examples of agent based corrections that would allow us to assess to what extent the agent prompts or instructions were effective in eliciting unique or interesting behavior. Essentially, I was not able to learn much from the quantitative results of the agent analysis that gave me much insight into how best to use agents in concert with existing optimizers. It's a shame really, since that could have been quite interesting and I hope the authors follow up in a different venue with a deeper dive on this topic, executed more rigorously.

Separate from the contributions, the literature review was very lightweight on the relevant areas to this application domain. I already addressed above the lack of coverage of the most common aerodynamic optimizers. In addition to this, there is a rich history of multifidelity surrogate modeling and optimization, even just in wing design, that the paper doesn't cover at all. There have also been more in depth studies on evaluation frameworks, datasets, comparison methods for parameterizations, and so on, that are directly relevant to the claimed contributions, but not covered at all in the paper. This was very disappointing.

Technical issues aside, I also found the writing in the paper to be particularly convoluted. It introduced additional, confusing, terminology and metrics where it could have (and should have) used existing conventions and definitions in the field to make it more readily comparable to past work. Many of the paragraphs lacked tightness and logical flow, and much of the material in the appendix was bloated and unnecessary, while somehow at the same time missing critical elements (e.g., on important elements of the meshing and solver sides, examples of agent corrections, etc.). I think the paper (and appendices) could have been reduced in length by about one half length and, in so doing, make the paper far more effective.

Questions for authors

To be honest, no additional clarifications or answers would materially affect my assessment. This would be a very substantial revision beyond what is practical within the ICLR review process.

10 comments
AuthorsOpenReview ↗

Failure rates

We appreciate your detailed review and the opportunity to clarify the evaluator checks, baseline selection, and feedback-strategy comparison.

First, the failure-rate comparison alone does not establish an implementation error. The review compares our 60.5% no-score rate with an uncited 5-10% reference without establishing comparable failure definitions, counting units, or evaluation conditions.

Second, the review's statement that gradient-free wing optimization is used "only in reduced dimension spaces" is too broad. Reducing the number of variables is one way to manage search; concentrating evaluations in promising regions while retaining the variables of the chosen formulation is another. Trust-region Bayesian optimization uses the latter approach, as demonstrated by Kakkar et al. [Kakkar26] in their 116-variable, mid-fidelity aerostructural study. Consequently, the variable count and slow convergence in Figure 4 do not, by themselves, establish that gradient-free methods are inappropriate baselines.

Third, optimizer competitiveness and benchmark contribution require separate assessment. WingBench links simulator feedback and search decisions to separately graded recommendations, revealing reliability differences that successful-design quality alone would obscure. Its replicated CMA-ES comparisons examine feedback-use strategies under shared initial information and evaluation allowances.

We address these points below through evaluator checks, replicated comparisons, and recorded agent behavior.

Failure definitions and direct evaluator checks

Over 4000 of the 6629 evaluations "delivered" no score, which the paper says can result from unmeshable geometry, incomplete meshing, or an unconverged flow solver. Having worked on very similar problems and solvers, the overall reported failure rate (close to 2/3rds of the runs) should be nowhere near this high if the entire pipeline was implemented correctly and allowed to properly converge (typically rates would be <5-10% max).

The review's uncited 5-10% reference does not establish an implementation error without comparable failure definitions, counting units, and evaluation conditions. Our 4,009/6,629 (60.5%) statistic counts graded requests, including repeated requests, and combines meshing failures, flow timeouts, and execution errors. A complete score requires acceptance at all three Mach conditions. This is neither a failure rate over independent optimization episodes nor a single-condition numerical-nonconvergence rate.

We tested meshing consistency directly. Of 1,000 randomly selected, previously successful SuperWing geometries, 999 produced accepted meshes with WingBench's spacing and all 1,000 with the original spacing. We then reran all 1,913 distinct failed geometries recoverable from WingBench's records, covering the entire available failed-geometry set, with the original, unmodified meshing script released with SuperWing in the AeroTransformer repository (gen-mesh.superwing.py). The script was byte-for-byte identical to the pinned published version and was run directly, without WingBench's case-preparation or grading wrappers, under matched inputs and execution settings. 1,912 of the 1,913 remained unsuccessful; only one produced an accepted mesh. These checks show closely matching meshing outcomes in both cohorts and establish meshing consistency under the tested configuration.

Separately, among 3,845 audited requests with accepted meshes, the pooled individual-condition failure rate was 20.8%; requiring all three conditions to succeed gave a request-level failure rate of 31.9%. These outcomes include timeouts and execution errors, so they do not measure numerical nonconvergence alone.

Published aerodynamic optimization studies also illustrate the importance of context. Rasines and Verstraete [Rasines24, Table 2] report mean final-population failure rates of 49.3-57.6% for SU2, defining failure through undefined objective or constraint values. Their population-based statistic differs from our request-based statistic, and the failure event, denominator, design space, and execution conditions must be specified before drawing direct numerical comparisons.

The revised appendices report the direct meshing checks, the stage-specific audit, and mesh, solver, and acceptance settings. These distinguish where the configured pipeline fails to return a score; the aggregate no-score fraction alone cannot diagnose an implementation defect.

AuthorsOpenReview ↗

Purpose of the baselines and scope of the benchmark

We thank the reviewer for raising the questions of baseline relevance, dimensionality, and the role of adjoint methods.

these baselines are simply not at all competitive baselines within the community that would study these problems. You would not take the 38 dimensional representation in ADFlow, and then run evolutionary algorithms BayesOpt on that type of space, in part because optimization convergence will be extremely slow (as can already be seen in Figure 4), and the actual people who do wing optimization use either (a) adjoint methods (which would be appropriate for the flow regimes studied here) or (b) gradient free methods but only in reduced dimension spaces.

The criticism conflates optimizer competitiveness with the scientific value of the feedback-strategy comparison. The review discounts the baseline study's contribution because it considers the selected optimizers uncompetitive. Our experiments address a distinct question: how recommendation reliability and aerodynamic quality differ across strategies that use simulator feedback in different ways.

The three CMA-ES variants deliberately compare a fixed surrogate-generated archive, feedback-based archive reordering, and feedback-guided candidate generation. They share 62 prior CFD records and allowances of 4,096 surrogate evaluations and 32 online CFD queries. Differential evolution and CFD-only SMAC3 broaden the comparison.

In the exploratory comparison pooled across tasks, feedback-guided CMA-ES improves recommendation delivery over either archive-only variant by 57 percentage points, with a descriptive 95% bootstrap interval of [37, 77] percentage points. Archive reordering changes final delivery little; feedback-guided candidate generation achieves substantially higher reliability. These are replicated differences between complete search strategies, including their surrogate allocations, and identify a concrete direction for agents to implement and test.

Reducing the number of variables and concentrating evaluations within promising regions are distinct ways to manage search. Adaptive trust-region Bayesian optimization uses local surrogate models and adjusts the search region while retaining the variables of the chosen formulation. Kakkar et al. [Kakkar26] apply this approach within a 116-variable, mid-fidelity aerostructural wing-design formulation. Ji et al. [Ji23] use 50 parameters for integrated wing/nacelle optimization. These studies demonstrate applications of Bayesian optimization with substantially more than 38 variables. The variable count and slow convergence of the tested configurations in Figure 4 do not, by themselves, establish that this method class is inappropriate for the benchmark.

Our initial emphasis on methods that could use the existing evaluator directly also reflected the additional work needed to integrate and validate an adjoint baseline. Although ADflow provides adjoint capabilities, a reliable optimization loop requires consistent sensitivities through geometry and mesh deformation, together with converged flow and adjoint solutions. Mesh quality and deformation robustness are documented challenges [Secco21], while ill-conditioned adjoint systems can require strong preconditioning [Kenway19]; an accepted CFD score alone does not guarantee a usable gradient. Sensitivity calculations also add work beyond the forward evaluation: Kenway et al. report an adjoint-to-flow runtime ratio of approximately 0.8 for drag sensitivities with the Jacobian-free adjoint in a transonic CRM wing case [Kenway19, Table 5]. This is a case-specific illustration, not a WingBench cost estimate. Comparability also informed our initial selection: sensitivity calculations can make computation between design evaluations non-negligible, making CFD query counts alone a less reliable proxy for total computational cost. Our existing timing records already sum to approximately 200,000 core-hours. Adding and validating another replicated baseline therefore requires a separate allocation of implementation and computational resources.

An adjoint baseline may help assess optimizer competitiveness. It is not a prerequisite for WingBench's replicated comparison of recommendation reliability and aerodynamic quality across feedback strategies.

AuthorsOpenReview ↗

Concrete agent corrections and their interpretation

Thank you for requesting concrete examples of how agents work with numerical optimizers.

However, in the current manuscript, both the tooling available to the agent (the AeroTransformer surrogate plus the above mentioned simulator) and how the agent is directed to analyze the results are very surface level. The paper doesn't provide concrete examples of agent based corrections that would allow us to assess to what extent the agent prompts or instructions were effective in eliciting unique or interesting behavior. Essentially, I was not able to learn much from the quantitative results of the agent analysis that gave me much insight into how best to use agents in concert with existing optimizers.

The traces show how agents modify numerical search: revising CMA-ES candidate selection, screening geometry after mesh failures, isolating design changes after timeouts, and using CFD to test predictor disagreement. Section 6.4 and Appendix D provide concrete examples:

Example 1: Correcting CMA-ES selection (A080, open-ended). The first two surrogate predictions were 32.76 and 35.70, while CFD returned 32.06 and 33.15. The agent then introduced a source-distance-based penalty into its selection objective; the next CFD score was 33.40. This records a change to how the numerical optimizer selects candidates.

Example 2: Screening geometry after mesh failures (A006, planning). After two local refinements failed meshing, the executor added a surface-triangle quality screen. Its fourth wing scored 28.55, compared with 28.52 initially. The handoffs also report replacing placeholder plans with executable procedures, so this demonstrates executor adaptation rather than planning effectiveness in isolation.

Example 3: Isolating a change after timeouts (A076, open-ended extension). Changing both sweep and root thickness led to a timeout, as did a near-repeat. At query 6, the agent returned to its verified query-2 wing and changed only sweep; CFD improved from 16.04 to 16.35. Subsequent smaller changes reached 16.64 and 16.77. These later steps belong to the extension beyond the common four-query comparison.

Example 4: Testing predictor disagreement (A035, reflection). At query 4, the agent explicitly tested a candidate for which AeroTransformer predicted 24.85 but its local ensemble predicted 22.91. CFD returned 24.89. The agent used an experiment to challenge the supplementary predictor's pessimistic assessment.

The expanded analysis covers all 360 common-horizon handoffs, including 64 transitions following unsuccessful evaluations, and documents unsuccessful adaptations, placeholder-plan substitutions, and bounded recoveries. The traces establish which changes occurred and what followed; they do not isolate the causal effect of each heuristic or prompt.

These examples motivate using failure feedback to change screening or candidate generation, isolating changes after failed tests, and checking supplementary predictions against CFD. At the common four-query checkpoint, neither planning nor reflection consistently improved both delivery and conditional quality across tasks. The quantitative comparisons assess complete implementations; the traces explain their decisions and subsequent outcomes.

Reviewer_bxiNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about benchmarking design search agents. The context is a CFD task, agents designing wings.

Critical strengths

I think it's valuable to document and understand which optimizers are performing better or worse than others on a task like this. There are all kinds of design tasks in CFD and physics, so I think there's good practical motivation here.

I think the variety of solvers and examples chosen feels appropriate.

Critical weaknesses

It is not clear to me from reading the paper how easy it is to extend the benchmark suite to add new agentic methods, for instance. It is also not clear how maintainable or well-maintained the benchmark repository will be - what is the risk of things going stale here? Speaking to extensibility and sustainability more would be appealing.

The authors choose around 6k for their data set size. Why is this enough or a good size to pick? Why not fewer, why not more?

I didn't really come away from this paper with a better idea of how to design good agents for design tasks, which is unfortunate. In a perfect world a benchmark paper like this would give more intuition and prescription of why some algorithms do better or worse than others, and how that might lead to the discovery of more better algorithsm in the future. This is not a huge slight against the present paper, as I think that's a very tall order potentially, but it is missed.

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Extensibility and sustainability

We are grateful for your constructive review and your questions about extending WingBench to new methods and maintaining its long-term usability.

It is not clear to me from reading the paper how easy it is to extend the benchmark suite to add new agentic methods, for instance.

New methods implement three operations: propose(ctx), observe(result), and recommend(). The context provides the task, evaluation history, remaining budgets, and surrogate interface. The common runner handles CFD feedback and private recommendation grading. An agent can maintain its own state, invoke a language model, and execute analysis or optimization code within this interface.

The release includes a worked optimizer example and a recorded-result backend for checking integrations before running CFD. Researchers can introduce new search methods without rebuilding the evaluator or recommendation-grading procedure.

It is also not clear how maintainable or well-maintained the benchmark repository will be - what is the risk of things going stale here? Speaking to extensibility and sustainability more would be appealing.

To preserve reproducibility as the software develops, the release retains the original records and content-pinned protocol, together with container recipes, source and model fingerprints, integration checks, and analysis scripts. This preserves a reproducible reference for published comparisons while allowing new methods and evaluator versions to be introduced. We have made the integration procedure and versioning policy more prominent in the manuscript, with a clearer pointer in Section 2.3 to the detailed guidance in Appendix J.

AuthorsOpenReview ↗

Evaluation count and study size

We welcome this question and appreciate the opportunity to explain the study size.

The authors choose around 6k for their data set size. Why is this enough or a good size to pick? Why not fewer, why not more?

The 6,629 evaluations follow from a replicated study design: 150 baseline episodes and 90 agent episodes across three tasks, with ten runs per method-task combination.

Evaluation componentEpisode coverageRecords
Baseline online evaluations150 baseline episodes4,800
Agent online evaluations at the common checkpoint90 agent episodes360
Additional evaluations in the open-ended extensionThe existing 30 open-ended episodes840
Private recommendation evaluationsAdditional grading records629
Total6,629

This scale provides repeated comparisons across methods and operating conditions at substantial computational cost. The available CFD timing aggregates sum to approximately 200,000 processor-core hours across 6,576 timed records. The reported intervals distinguish large reliability differences from smaller performance differences that remain uncertain. Fewer episodes would reduce precision, while more episodes would help resolve the smaller differences, but at the cost of more compute. We have clarified in Section 5 how the evaluation count follows from the study design and how the number of independent episodes limits the precision of the reported comparisons. We do not claim that 6,629 evaluations is a proven optimal count; this study design reflects a practical balance between computational cost and the ability to detect differences between methods.

AuthorsOpenReview ↗

Lessons for designing better agents

Many thanks for this constructive suggestion on making the lessons for agent design more explicit.

give more intuition and prescription of why some algorithms do better or worse than others, and how that might lead to the discovery of more better algorithsm in the future

The experiments identify three practical directions for agent design: revise candidate generation using feedback, check supplementary predictions against CFD, and maintain an explicit recommendation throughout exploration.

The baseline study compares three CMA-ES strategies: evaluating a fixed surrogate-generated archive, using CFD feedback to reorder it, and using feedback to generate new candidates. Reordering candidates within a fixed archive produced little change in final delivery, whereas feedback-guided CMA-ES, which generates subsequent candidates using returned feedback, achieved much higher reliability. In the exploratory comparison pooled across tasks, feedback-guided CMA-ES improves recommendation delivery over either archive-only variant by 57 percentage points, with a descriptive 95% bootstrap interval of [37, 77] percentage points. These strategy-level results support developing agents that use feedback to revise their search procedures and generate new candidates.

The agent traces show how such revisions can be implemented. After mesh failures, agents changed extreme geometry or introduced a surface-quality screen. In another episode, an agent used observed surrogate errors to modify its CMA-ES selection objective. These records provide concrete examples of how an agent can translate feedback into changes to an executable search procedure.

The traces also reveal pitfalls in adding more modeling or reasoning. Some supplementary predictors became overly optimistic or discouraged testing productive candidates. At the four-query checkpoint, neither the planning nor reflection implementation consistently improved both delivery and conditional quality across tasks. These findings motivate checking supplementary predictions against CFD evidence and evaluating additional reasoning through its effect on subsequent experiments.

A further practical lesson is to maintain an explicit recommendation throughout exploration. At nominal load after 32 online CFD queries, the methods show the following outcomes (ten runs per strategy):

MethodValid recommendationsMean L/D among valid recommendations
Each archive-only CMA-ES variant4/10 runsApproximately 33.2
Feedback-guided CMA-ES10/10 runs33.04

Comparing conditional quality alone would slightly favor the archive-only methods while overlooking their six unsuccessful runs. A practical design implication is to track CFD-validated candidates separately from exploratory proposals, retain their evaluation records, and check which design is nominated. This makes recommendation selection an explicit part of the agent's procedure.

We have revised the manuscript's discussion of the baseline results and agent behavior in Sections 6.1 and 6.4 to make these lessons clearer, linking each lesson to the relevant quantitative comparison or recorded example. Appendix D provides the supporting decision sequences and distinguishes observed adaptations from demonstrated improvements.

Reviewer_bxiNOpenReview ↗

Response to authors

I would like to thank the authors for their revisions and the thoughtful response. I think the authors have given me answers to most of my concerns. Having read the other reviewers' comments and the responses to them, I do not feel comfortable raising my score at this time, but I still think I tend to lean positively on the paper, and plan to maintain my score. Thank you again.

Comments on the paper

3 comments
AuthorsOpenReview ↗

References

Reference key for author responses

We use the shorthand citations below throughout our reviewer responses.

AuthorsOpenReview ↗

Follow-up reference key

Follow-up reference key

These keys apply to our follow-up responses. They supplement the first-round reference key.

AuthorsOpenReview ↗

Author summary of the discussion and revisions

To the Area Chair and reviewers,

Thank you for your time and consideration throughout the discussion. The main changes are summarized below.

1. Direct evidence on the evaluator. Reviewer Z8Eb questioned whether the evaluator was implemented correctly, citing its high overall failure rate. We remeshed successful source geometries, reran all 1,913 recoverable failed geometries with the original, unmodified upstream script under matched settings, and audited flow outcomes:

Evaluator checkResult
Successful source geometries, WingBench spacing999/1,000 accepted
Same source geometries, original source spacing1,000/1,000 accepted
All recoverable failed geometries, unchanged upstream script1,912/1,913 remained unsuccessful
Accepted-mesh audit: individual conditions, pooled20.8% failure
Accepted-mesh audit: all three Mach conditions required31.9% failure

The flow audit covers 3,845 requests with accepted meshes. Its failures include timeouts and execution errors.

These checks support implementation consistency and clarify why the aggregate no-score rate is not a single-condition numerical-nonconvergence rate. Regarding Reviewer Z8Eb's point that some failed wings in Figure 1 look meshable, we do not claim that they cannot be meshed under any procedure; a tuned or more manually involved procedure may succeed. WingBench reports outcomes under one specified, fully automatic procedure, because every proposed wing must be evaluated without case-by-case human intervention (Appendix C).

2. A baseline finding beyond “feedback helps.” The nominal-load trajectories show how the comparison changes with the CFD budget. Each strategy has ten runs; mean L/D includes only valid recommendations:

CMA-ES strategyQuery 4: valid/10Query 4: mean L/DQuery 32: valid/10Query 32: mean L/D
No feedback: fixed archive3/1033.224/1033.22
Selection only: archive reordering4/1033.184/1033.22
Feedback-guided generation10/1028.7310/1033.04

The archive methods start with higher quality among valid recommendations. Feedback-guided CMA-ES largely catches up in quality while maintaining delivery in all ten runs. The early quality gap largely closes while the reliability gap persists. The baseline study thus reveals a trade-off between reliability and quality that is strongest early and narrows as the CFD budget grows (Section 6.1; Table 22).

3. More inspectable agent behavior and clearer scope. We expanded the analysis to all 360 common-horizon decision records and added examples of changing CMA-ES selection, screening geometry after mesh failures, and testing supplementary predictions against CFD (Section 6.4; Appendix D). We also clarified that the agent study's main contribution is comparing open-ended research with the executed reflection and planning configurations.

4. Completed resource measurements. In response to Reviewer jYTE, we added a focused study of 24 initiated runs, with three seeds per method-load group. Appendix Q reports search and private-grading CFD time separately, local CPU usage, surrogate evaluations, and model calls and tokens. Results after four CFD queries:

Method / loadValid/3Mean L/DSearch CFD hGrading CFD hLocal CPU h
Feedback-guided CMA-ES / low3/317.174.120.070.47
Open-ended / low3/317.3510.892.621.73
Feedback-guided CMA-ES / nominal3/330.326.114.540.49
Selection-only CMA-ES / nominal1/332.870.180.183.25
Open-ended / nominal3/330.579.212.632.22
Planning / nominal2/330.8710.842.322.08
Feedback-guided CMA-ES / high3/324.1413.192.820.55
Open-ended / high3/330.9712.742.943.06

These are descriptive results: initial information and total computation remain unmatched across families. The new measurements clarify costs without claiming a fully resource-matched ranking.

5. References added during the discussion. Our two reference-key comments list the published work cited in our responses to Reviewer Z8Eb. Notable additions:

Reference addedPoint supported
SuperWing in Scientific Data, with a mesh-convergence study; used in an AIAA Journal study and AeroJEPAValue of the source dataset
Rasines and Verstraete (ICAS 2024): 49.3-57.6% mean final-population failure rates with SU2Failure rates depend on definitions
Chan et al. and Chuen et al. (AIAA AVIATION 2022, 2023): mesh validity can change with the procedureMeshability versus a fixed automatic recipe
Kakkar et al. (2026) and Ji et al. (2023): wing design with 116 and 50 variablesBayesian optimization in higher dimensions

Thank you for considering the revised manuscript and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.