- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WingBench, a multifidelity benchmarkthat includes both optimizers and AI agents designing transonic wings with CFD tools. The authors use WingBench to study five baseline optimizers and a controlled comparison of three AI agent methods.
Critical strengths
- Each method proceeds through a defined sequence of design attempts and must submit a final wing. This is great for comparison purposes. The paper reports how often the method succeeds in submitting a valid wing and how good that wing is.
- The 6k recorded evaluations, along with reasons for failure and reply capabilities, makes the results inspectable and reviewable.
Critical weaknesses
I worry that methods are not compared on an apples-to-apples basis. For instance, the four standard optimizers receive 60 CFD results, while the agent does not (unless I'm mistaken). Likewise, some methods can make intermediate surrogate model calls while others do not.
Questions for authors
How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?
3 comments
Matched-resource rankings and additional measurements
We appreciate this specific question about rankings under matched resource limits.
How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?
The reported rankings compare the tested configurations under their stated information and evaluation budgets. Determining how those rankings change under matched initial information and common compute or wall-clock limits requires a separate controlled comparison.
Appendix B reports online and private CFD requests, charged surrogate evaluations, model calls and tokens, reference model charges, recorded session durations, and available CFD timing. At the common four-query checkpoint, total recorded session duration divided by the number of online CFD requests is:
| Agent method | Summed recorded session duration / online CFD requests (minutes per request) |
|---|---|
| Open-ended research | 30.97 |
| Planning | 25.01 |
| Reflection | 31.40 |
The numerator sums the durations of recorded sessions, with overlapping sessions contributing separately. It is therefore not a single elapsed episode duration. These measurements include tool execution and local code; they are not pure model inference or end-to-end wall time.
The resource accounting explicitly identifies quantities that were not consistently measured across the full study, including optimizer CPU/GPU usage, model-server compute, and total elapsed wall time.
We are conducting a focused set of additional runs for selected baseline and agent configurations under the original search settings and evaluation budgets. These runs will record end-to-end wall-clock time, CFD and decision-stage time, optimizer CPU/GPU usage, surrogate evaluations, and model calls and tokens.
We will report these measurements alongside recommendation delivery and conditional quality from the same runs, specifying the methods, tasks, query checkpoints, number of repetitions, and execution conditions. Server-side compute will be reported separately where directly measurable.
They will clarify resource costs rather than establish fully matched rankings. We will report the measurements in a follow-up response once the runs are complete.
Measured costs and results
How do the rankings change when methods start with the same CFD information total limits on CFD calls, surrogate calls, model computing time, and wall-clock time?
Thank you for suggesting this analysis. The additional runs are completed and support competitive open-ended agent performance under the original search budgets. At four CFD queries, open-ended research delivered valid recommendations in 9/9 runs across the three tasks, with high-load mean L/D 30.97 versus 24.14 for feedback-guided CMA-ES. We have added this study to the revised manuscript, distinguishing CMA-ES comparisons with shared initial information and evaluation allowances from cross-family results under the original search budgets.
We tested each method and load below with three seeds, for 24 initiated runs in total. Each method kept its 32-query budget and 4,096-surrogate limit. We recorded recommendations at 0, 1, 2 and 4 CFD queries. The agents were instructed to optimize over the usual 32-query budget; we stopped the runs after query 4.
Results after four CFD queries. Success counts include all three runs in each group. Mean L/D includes only valid recommendations. Each time column is a separate median for runs that reached query 4: two for planning, three for every other row.
| Method / load | Valid/3 | Mean L/D | Search CFD h | Grading CFD h | Local CPU h |
|---|---|---|---|---|---|
| Feedback-guided CMA-ES / low | 3/3 | 17.17 | 4.12 | 0.07 | 0.47 |
| Open-ended / low | 3/3 | 17.35 | 10.89 | 2.62 | 1.73 |
| Feedback-guided CMA-ES / nominal | 3/3 | 30.32 | 6.11 | 4.54 | 0.49 |
| Selection-only CMA-ES / nominal | 1/3 | 32.87 | 0.18 | 0.18 | 3.25 |
| Open-ended / nominal | 3/3 | 30.57 | 9.21 | 2.63 | 2.22 |
| Planning / nominal | 2/3 | 30.87 | 10.84 | 2.32 | 2.08 |
| Feedback-guided CMA-ES / high | 3/3 | 24.14 | 13.19 | 2.82 | 0.55 |
| Open-ended / high | 3/3 | 30.97 | 12.74 | 2.94 | 3.06 |
The open-ended agent and feedback-guided CMA-ES both returned valid wings in 9/9 runs. Their mean L/D was close at low and nominal load. At high load, the agent reached 30.97 versus 24.14, while using more local CPU time. Selection-only CMA-ES had the highest nominal mean L/D but returned a valid wing in only 1/3 runs. Its time medians include the two runs without valid recommendations.
Compute use. We used 94 search CFD evaluations (23 x 4 + 2) and 43 additional evaluations to grade recommendations. Private grading does not count toward the search budget. If a recommendation already had a search result, we reused it.
By query 4, feedback-guided CMA-ES used 512 surrogate evaluations per run; selection-only used all 4,096. Open-ended runs used 643-2,306, and the two completed planning runs used 1,068-1,471. Across all 12 agent runs we recorded 2,653 model calls, 397.61 million input tokens (including 396.02 million cached) and 2.75 million output tokens.
Search decisions took 25-32 minutes for feedback-guided CMA-ES and 156-248 minutes for selection-only. Agent research and review took 76-252 minutes per run.
CFD hours sum each evaluator's start-to-finish execution time, separately for search and recommendation grading; they are not processor-core hours. Local CPU hours sum user and system CPU time across the baseline controller process tree, or agent research/review and episode-service process trees, including surrogates. This is accumulated CPU time, not elapsed time, and excludes remote CFD and model-provider computation. No local GPUs were used. Model-provider usage is reported as calls and tokens.
CFD and baselines ran on AWS m7a.4xlarge machines (16 vCPUs); agents and surrogates shared a CPU-only r7a.12xlarge machine. Each batch allowed up to 12 concurrent CFD jobs.
Initial CFD information and evaluation allowances
Thank you for recognizing the value of the evaluation protocol and records, and for raising the question of comparability.
The four surrogate-assisted baselines receive 62 prior CFD records that the agents do not receive, and the cross-family comparison does not match total computation or wall-clock limits. The revised manuscript makes these conditions explicit and reports resource usage from the existing records.
Section 5 and its resource-conditions table distinguish the experimental settings. The four surrogate-assisted baselines share the same 62-record context, an allowance of 32 online CFD queries, and 4,096 surrogate evaluations per episode. Their allocation of surrogate evaluations and use of CFD feedback differ. SMAC3 is a separate CFD-only reference with no warm start or surrogate access.
The three agent implementations share the SuperWing source data, language model, tools, and evaluation allowances, and are compared at the common four-query checkpoint. The four-query comparison evaluates an intermediate checkpoint within the original 32-query task; the agents were not instructed to optimize specifically for a four-query horizon. They do not receive the baselines' 62-record context. Planning and reflection introduce additional model calls, so these shared conditions do not imply equal computational expenditure. Appendix D describes the executed implementations, including bounded recoveries.
Appendix B distinguishes surrogate access and allowances from actual expenditure. Mean charged surrogate evaluations per episode at the common four-query checkpoint are:
These are actual expenditures up to query 4; the 4,096-surrogate allowance applies to the full episode. Appendix B specifies the counting conventions, including uncharged agent cache hits.
The open-ended extension continues the existing 30 episodes to query 32 within the same total surrogate allowance. At query 32, open-ended research averages 1,988.9 charged surrogate evaluations, compared with 4,096 for each surrogate-assisted baseline.
The within-baseline comparisons nevertheless answer a controlled question about how the tested strategies use feedback. The CMA-ES variants share the same initial CFD information and evaluation allowances. In the exploratory comparison pooled across tasks, feedback-guided CMA-ES improves recommendation delivery over either archive-only variant by 57 percentage points, with a descriptive 95% bootstrap interval of [37, 77] percentage points. This supports the comparison of those search strategies under the stated protocol.
The agent comparisons likewise evaluate the implemented approaches under shared source data, model, tools, and evaluation allowances, with their differing model-call and surrogate expenditures reported explicitly. The revised abstract, results, and conclusion interpret the findings according to these experimental conditions.