acceptodds
Under review as a conference paper at ICLR 2027

Do Candidate-Pool Benchmarks Faithfully Evaluate LLM Optimizers?

Abstract

A benchmark for a generative optimizer should evaluate the designs that the optimizer proposes. In scientific optimization, however, new measurements are often unavailable, and an optimizer is instead scored by Selecting from a pool of measured candidates or by Mapping its proposals to the nearest stored design. We ask whether these candidate-pool protocols preserve conclusions about prospective optimization. Across four materials benchmarks, larger search steps by large language model (LLM) optimizers are associated with fewer evaluations to reach the target region. In a controlled prompt experiment, a one-line instruction changes this reaching time without changing the underlying model. These observations motivate the trajectory-imprinting hypothesis: a pool can inherit geometric structure from the optimization runs that produced it. To test fidelity beyond the pool, we directly evaluate optimizer-proposed designs using physics-based simulations of planar MOSFETs and GaAs deposition. In GaAs deposition, Gaussian-process (GP) optimizers reach lower nonuniformity under Generation, whereas LLM optimizers reach the pool optimum first under Selection. Direct evaluation of mapped proposals reveals changes in measured-and-feasible status in 38.4% of MOSFET and 19.0% of deposition pairs. Among pairs feasible on both sides, mean projection credit is larger for GP optimizers than for LLM optimizers. We construct core-set pools from designs collected during previous optimization runs, keeping the initial design and the best candidate fixed. These pools have different measured geometric properties, but optimizer orderings under Selection still differ from those under Generation. Proxy fidelity must therefore be assessed for the optimizer set, pool, protocol, and metric rather than assumed from pool geometry alone.

Primary area:
optimization
Keywords:
Large Language Models, Bayesian Optimization, Materials Optimization, Physics-Based Simulation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1MmiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether candidate-pool benchmarks faithfully evaluate LLM optimizers by comparing selection from stored candidates and nearest-neighbor mapping with direct evaluation of proposed designs. Experiments on materials datasets and two physics-based simulators show that optimizer rankings can differ across evaluation protocols and that mapping introduces optimizer-dependent distortions in feasibility and objective values.

Critical strengths
  • The paper presents an approach for analyzing how candidate pools and evaluation protocols affect the assessment of LLM optimizers.
  • The analysis uses physics-based simulators to directly evaluate optimizer-proposed designs.
Critical weaknesses
  • The importance of the motivation and research question is not sufficiently established.
  • The paper does not provide a practical solution alternative to evaluation based on previously measured data.
  • Trajectory imprinting remains a hypothesis, and the evidence is insufficient to establish its presence and causal influence on optimization behavior.
  • Some aspects of the experimental setup remain unclear, and the experiments do not sufficiently isolate the effects of different candidate pools.
Questions for authors
  • Section 3: Could the observed relationship simply reflect overly small search steps by some LLMs, with models taking larger steps reaching the target region sooner, independently of trajectory imprinting?
  • Section 4: Are the LLMs informed about how each candidate pool was collected? If so, could they use this information to guide their choices under Selection?
  • Section 5.2, Figure 4: Could the larger projection credit observed for GP optimizers be driven primarily by LLM-generated pools, while projection errors are smaller on GP-generated pools? Reporting the results separately for each pool would help distinguish these possibilities.
  • Section 6: How was the initial design z_0 chosen? Although its parameter values are provided in Appendix A, the rationale for selecting this design is unclear, and the geometric diagnostics may be sensitive to this choice.
3 comments

We thank the reviewer for the careful reading and the specific questions. each pointed to something the paper had left unclear. We have uploaded a revised manuscript addressing the reviewers' concerns. All section, figure, and table references below refer to this revised version.


Q1. Step size versus trajectory imprinting

Section 3: Could the observed relationship simply reflect overly small search steps by some LLMs, with models taking larger steps reaching the target region sooner, independently of trajectory imprinting?

Yes, it could. The step-size association cannot separate "some LLMs take steps that are too small" from an effect of pool geometry, and step size and reaching time are computed from the same trajectories. The revised paper states this and no longer uses the association as evidence of trajectory imprinting [Section 3, last paragraph; Appendix E].

The association is also not universal: in thin-film deposition it is -0.82 for the LLM runs, while GP-UCB shows +0.71 on the source pools (the 50-candidate pools recorded along optimization trajectories), and similar negative values appear on some pools with no optimization provenance [Section 5.1].

Why the question still matters (Weakness 1).

Figure 2 is the observation that raised the question, not the evidence for the answer. In a finite materials pool (BN, 12 LLMs), LLMs that take larger steps reach the target sooner, and a prompt instruction to search farther or nearer changes the number of evaluations to the target [Section 3, Figure 2].

So a pool score may reflect how an optimizer moves among stored candidates, not the quality of the designs it would propose. We test this directly: on the 14 source pools, Generation resolves 92 optimizer comparisons (Bonferroni-adjusted 95% bootstrap interval excludes zero), and Selection or Mapping preserves 30, leaves 40 unresolved, and reverses 22 [Section 5.2, Table 2].

Status of trajectory imprinting (Weakness 3).

It is now a measured property of a pool, not a hypothesis about cause. We measure how closely the candidates lie to the segment from the initial design to the best recorded design. The LLM-collected source pools have relative radius R10 of at most 0.01 and R90 between 0.02 and 0.55, against about 1 for a Sobol pool [Section 5.1, Table 1].

We make no claim that this concentration causes the step-size association.


Q2. Pool provenance disclosed to LLMs

Section 4: Are the LLMs informed about how each candidate pool was collected? If so, could they use this information to guide their choices under Selection?

No. Under Selection the LLMs receive the candidate input values with neutral identifiers that are reassigned so that they carry no information. They do not receive the collector's identity, the order in which candidates were acquired, or any outcome that has not been observed in the run [Section 4.1; Appendix A; Appendix K].

The distribution of the candidate inputs is of course visible. In a controlled experiment, showing or hiding the candidate inputs produced no resolved effect for either LLM [Section 6.1] (Weakness 4: setup).

Q3. Projection credit by pool source

Section 5.2, Figure 4: Could the larger projection credit observed for GP optimizers be driven primarily by LLM-generated pools, while projection errors are smaller on GP-generated pools? Reporting the results separately for each pool would help distinguish these possibilities.

Largely yes for two of the three tasks, and we now report the credit separately for every source pool [Appendix I, Table 16; the credit figure is Figure 3 in the revision] (Weakness 4: isolating pool effects). GP-minus-LLM credit by pool source:

Task (unit)LLM-collected pools (Luna, Sonnet)GP-collected pools (GP-UCB, GP-EI)Human-collected poolConsistent with the suggestion?
Thin-film deposition (percentage points)+5.94, +5.16+2.78, -2.15+1.52Yes
Plasma etching (loss units)+31.4, +28.0+18.2, +9.7no poolYes, although still positive on GP-collected pools
Transistor fabrication (µA)+109.9, -4.0+73.8, -25.4+85.5No

Each value is the mean credit of the two GPs minus the mean credit of the two LLMs on that pool; a positive value means Mapping credits the GPs more. Table 16 gives the credit of each optimizer on each pool.

  • In thin-film deposition and plasma etching the gap is largest on the two LLM-collected pools and smallest on the GP-EI-collected pool, as the reviewer suggests.

  • It is not confined to LLM-collected pools. The gap is positive in 11 of 14 task-pool cells, including both human-collected pools and all three GP-UCB-collected pools. In transistor fabrication the pattern does not hold: the gap is -4.0 µA on the Sonnet-collected pool and +85.5 and +73.8 µA on the human- and GP-UCB-collected pools.

What a benchmark user can do about it (Weakness 2: practical alternative).

Section 7 gives three steps; the first two need no new evaluation.

(i) Enlarge the pool: with candidates drawn from optimization trajectories, the GP-minus-LLM credit gap falls from 49.8 to 0.4 µA, from 12.4 to 2.6 percentage points, and from 20.0 to 3.3 loss units between 256 candidates and the largest pool, and the score error falls in all nine tested sequences; with Sobol-only pools the credit gap did not narrow consistently [Section 7.2, Table 3; Appendix J].

(ii) Report the proposal-projection distance of each optimizer; it needs only inputs and proposals [Section 7.1, Figure 4].

(iii) If a small budget exists, evaluate a sample of original proposals to measure the credit directly [Section 7.3].

A larger pool reduced score distortion but improved rank correlation in only five of eight sequences, so these steps lower the risk and do not replace Generation.


Q4. Choice of initial design and sensitivity

Section 6: How was the initial design z_0 chosen? Although its parameter values are provided in Appendix A, the rationale for selecting this design is unclear, and the geometric diagnostics may be sensitive to this choice.

How it was chosen.

Each task has one base initial design, and the 50-candidate source pools were collected starting from it.

  • Transistor fabrication and thin-film deposition: the base designs are poor starting points, as at the start of process development, where the first recipe is usually far from the optimum. The transistor-fabrication design is feasible with an on-current of 1.65 x 10^(-6) µA, against the 300 µA target given in the prompt. The thin-film deposition design violates the growth-rate constraint (0.41 µm/h against the required 3 µm/h) and has a nonuniformity of 93.4% [Appendix A].

  • Plasma etching: the base design is the first draw of a uniform random sample of the input box (loss 98.5) [Appendix A].

Sensitivity (Weakness 4).

The other five initial designs of each task were drawn at random from an evaluated Sobol set of 4,096 points with a fixed seed, with no selection on objective value. For the two tasks with a constraint, the draw was stratified so that three of the six designs are feasible and three infeasible [Appendix A].

Generation is reported for all six designs: it resolves six of six optimizer pairs for transistor fabrication, three for thin-film deposition, and five for plasma etching [Appendix F, Table 8].

Because the ordering depends on the initial design, each pool is compared with Generation from the same initial design [Section 5.2], and the pool-size and geometry studies use one of the randomly drawn designs, not the base design; that design was chosen for each task after inspecting how well Generation separates the optimizers there [Appendix J].

Geometric diagnostics.

The trajectory-imprinting radii in Table 1 are defined about the segment from a pool's own initial design to its best recorded design, so they describe that pool as it was collected. We do not claim the same values for a different starting point [Section 5.1; Appendix D].

Dear Reviewer 1Mmi,

We sincerely thank you for your time, careful review, and constructive feedback. Your comments have been invaluable in helping us improve our manuscript.

We have uploaded a revised manuscript and provided detailed responses to your concerns, including additional experiments and clarifications.

With approximately two hours remaining in the discussion period, we would be very grateful if you could take a final look at our responses and let us know whether they have adequately addressed your main concerns. If any questions or concerns remain, we would be more than happy to provide further clarification before the discussion closes.

We sincerely appreciate your time and thoughtful consideration, especially during this busy review period.

Best regards,

The Authors

Reviewer_Mn2MOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explore a question on how we measure AI agents in scientific discovery, which is whether a pre-defined dataset based benchmark accurately reflect an AI's ability in invent new designs. The paper shows that evaluating LLM optimizers by having them pick from a pre-definied dataset ("selection") or by mapping their new prediction back to the closest data point ("mapping") produces completely different performance rankings compared to actually test out their new designs in a true simulator.

Critical strengths
  • This paper provides some evidence that we shouldn't evaluate AI agents just based a static dataset.
  • Those evidences also suggest that using actual simulators can be a robust way to evaluate LLM optimizers.
Critical weaknesses
  • The paper's presentation is very dense and hard to parse.
  • I think the paper's core conclusion is somewhat already known to the practioners in the field, that where possible, researcher would already use direct generation to evaluate AI agents, and the reason searchers use a static pools is just that the real experiments can be expensive or unavailable.
  • The experiments well supports the paper, but the experiments are only based on two tasks. It would be stronger if more data points are gathered.
Questions for authors
  • How can a research build a better unbiased static benchmark if real-world simulator are simply too hard or expensive or unavailable for some specific domains?
2 comments

We thank the reviewer for the candid assessment and for the practical question, which led us to write a new Section 7. We have uploaded a revised manuscript addressing the reviewers' concerns. All section, figure, and table references below refer to this revised version.


Q1. Building a static benchmark when direct evaluation is unavailable

How can a research build a better unbiased static benchmark if real-world simulator are simply too hard or expensive or unavailable for some specific domains?

What the paper adds to what practitioners know (Weakness 2).

We agree that Generation is preferred when it is affordable. What was not measured before is how far, and in which direction, a pool result departs from it. On 14 source pools, Generation resolves 92 optimizer comparisons (Bonferroni-adjusted 95% bootstrap interval excludes zero); Selection or Mapping preserves 30, leaves 40 unresolved, and reverses 22 [Section 5.2, Table 2].

The shift is not the same for all optimizers: in a controlled plasma-etching experiment, restriction to the pool raises GP scores by 8.55 and 8.44 loss units and leaves LLM scores unresolved [Section 6.1], and Mapping credits GPs more than LLMs by +52.7 µA, +2.65 percentage points, and +21.8 loss units in the three tasks [Section 6.2, Figure 3].

Evidence base (Weakness 3) and organization (Weakness 1).

The revision has three simulator tasks, six initial designs per task, 38 pools, 1,120 Generation and 1,920 pool trajectories [Section 4.3].

It is reorganized around one question per section: how often pools disagree with Generation (Section 5), two ways in which the protocols shift optimizers unequally (Section 6), and what reduces the disagreement (Section 7).

Answer to the question.

Without any new measurement, no static pool can be certified as unbiased. What can be done with recorded data only is the following, each with the result that motivates it.

  1. State which protocol is used and report every pairwise comparison with uncertainty.

    • Reason: Selection scores the choice among recorded candidates and Mapping scores a proposal with another design's outcome; neither is proposal quality, and only 30 of 92 resolved comparisons are preserved [Section 7.3; Table 2].
  2. Make the pool as large as the data allow.

    • Reason: from the smallest to the largest pool, the between-optimizer shift D and the score error E fell in all nine tested size sequences. For transistor fabrication the median D falls from 78.9 to 37.1 µA and the median E from 148.3 to 26.3 µA between 256 and about 4,096 candidates, and in the sequence built from optimization trajectories the rate of feasibility-status changes falls from 47% to 23% [Section 7.2, Table 3; Appendix J].
    • Rank correlation improved in only five of eight sequences, so size reduces distortion but does not guarantee the ranking [Sections 7.2, 7.3].
  3. Check the pool geometry against the optimizers being compared, and report it.

    • Reason: LLM proposals are generally farther from diffuse pools and GP proposals from concentrated pools, so a given pool is not equally close to both classes [Section 7.1, Figure 4; Appendix J, Figure 19].
    • With feasibility counts matched, pools with a concentrated core and a broad remainder had a mean rank correlation of 0.84 with Generation, against 0.62 (Narrow) and 0.69 (Diffuse), and an inverted ranking in 1.4% of resamples against 10.4% and 10.5% [Section 7.2, Table 4].
    • Pools recorded from LLM optimization runs are at the narrow extreme (relative radius R10 of at most 0.01, against about 1 for a Sobol pool) [Section 5.1, Table 1].
    • These radii are computed from recorded data. The differences between geometries are not statistically resolved, so geometry is a risk indicator to report, not a rule for a preferred pool [Section 7.3].
  4. Under Mapping, report the proposal-projection distance of each optimizer and treat a large or unequal distance as a warning.

    • Reason: the distance is how far the credited design is from the proposed one; in our audit the two differ in feasibility status in 15.5% to 45.7% of cases [Section 6.2].
    • The distance needs only inputs and proposals [Section 7.3].
  5. If even a small evaluation budget exists, evaluate a sample of original proposals. This measures the substitution rate and the projection credit directly [Section 7.3].

These steps reduce and expose the risk; they do not prove that a static benchmark is unbiased.

Dear Reviewer Mn2M,

We sincerely thank you for your time, careful review, and constructive feedback. Your comments have been invaluable in helping us improve our manuscript.

We have uploaded a revised manuscript and provided detailed responses to your concerns, including additional experiments and clarifications.

With approximately two hours remaining in the discussion period, we would be very grateful if you could take a final look at our responses and let us know whether they have adequately addressed your main concerns. If any questions or concerns remain, we would be more than happy to provide further clarification before the discussion closes.

We sincerely appreciate your time and thoughtful consideration, especially during this busy review period.

Best regards,

The Authors

Reviewer_ntGjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether candidate-pool evaluations reflect optimizers’ ability to generate useful new designs. It compares selecting existing candidates and mapping proposals to their nearest stored neighbors with directly evaluating proposals through physics-based simulations. Experiments investigate search-step behavior, protocol-dependent performance differences, proposal–projection discrepancies, and alternative pool constructions. The results show that the tested protocols can produce different optimizer rankings and that nearest-neighbor mapping can change feasibility status and objective values.

Critical strengths

The paper investigates an interesting evaluation question. Its direct simulation of original proposals provides useful evidence that nearest-neighbor mapping can substantially alter credited outcomes, with different effects across optimizers.

Critical weaknesses
  1. The presentation obscures the contribution and the relationships between experiments. The abstract and introduction introduce numerous terms alongside frequent qualifications before clearly establishing the main findings. Step-size observations, trajectory imprinting, protocol comparisons, mapping audits, and core-set experiments are presented without sufficiently explaining how their evidence connects. In particular, the step-size results do not establish the proposed geometric mechanism, and the mapping audit does not demonstrate that projection credit explains the ranking differences. The strongest finding concerns evaluation bias across optimizers generally, while the specifically LLM-related insight remains less clear.
  2. Limited insight beyond protocol-dependent differences. Changing the admissible search space or replacing the evaluated proposal can reasonably be expected to change optimization outcomes. The paper documents such discrepancies, but does not sufficiently establish their causes, robustness, or implications beyond the studied cases. Direct evaluation uses only two simulation tasks and one initialization per task. The core-set experiments change geometry, feasibility composition, and outcome distribution simultaneously, making their results difficult to interpret. Consequently, the evidence provides limited guidance about when candidate-pool evaluations are reliable or how their fidelity can be improved.
Questions for authors
  1. Beyond demonstrating that different evaluation protocols can produce different outcomes, what specific new conclusion should change how researchers design or interpret optimizer benchmarks? Which experiment most directly supports it?

  2. How stable are the performance differences and rankings across additional initializations and independently constructed pools? Can uncertainty in the ranking comparisons be reported, particularly where methods have nearly identical AUC values?

  3. Can the effects of pool geometry be isolated from feasibility composition and objective-value distribution? Without this separation, what can the core-set experiments establish about trajectory imprinting or benchmark fidelity?

5 comments

We thank the reviewer for the careful reading and the constructive comments, which directly shaped the revision. We have uploaded a revised manuscript addressing the reviewers' concerns. All section, figure, and table references below refer to this revised version.


Q1. Practical contribution beyond protocol differences

Beyond demonstrating that different evaluation protocols can produce different outcomes, what specific new conclusion should change how researchers design or interpret optimizer benchmarks? Which experiment most directly supports it?

The conclusion.

Selection and Mapping do not shift all optimizers by the same amount, and the difference is large enough to flip comparisons that Generation settles. We call a pair of optimizers resolved when the Bonferroni-adjusted 95% bootstrap interval for the difference in mean AUC50 excludes zero, and reversed only when Generation and the pool protocol both resolve it, in opposite directions [Section 4.3].

On the 14 source pools (50-candidate pools recorded along one optimization trajectory), Generation resolves 92 of 168 pool-protocol-pair comparisons. The pool protocol preserves 30 of them, leaves 40 unresolved, and reverses 22 (9 under Selection, 13 under Mapping) [Section 5.2, Table 2].

This is the experiment that most directly supports the conclusion.

Two measured sources of the unequal shift (Weakness 2: causes).

  • Selection: in a controlled experiment on plasma etching (one input-random pool), restricting choices to the pool raises the score of GP-UCB by 8.55 and of GP-EI by 8.44 loss units (both intervals exclude zero), while the changes for the two LLMs (+0.65 and -1.32) are not resolved. With unrestricted proposals each LLM is resolved as better than both GPs; with restricted selection no LLM-GP comparison is resolved [Section 6.1; Appendix H, Figure 17].

  • Mapping: we evaluated 13,500 original proposals directly. The proposal and the pool member that replaces it differ in feasibility or validity status in 45.7%, 18.3%, and 15.5% of cases (transistor fabrication, thin-film deposition, plasma etching). The stored outcome is better than what the proposal earns by more for GPs than for LLMs: +52.7 µA [25.8, 79.3], +2.65 percentage points [1.75, 3.56], and +21.8 loss units [14.1, 29.8] [Section 6.2, Figure 3].

  • The audit measures unequal substitution. It does not show that this alone produces each reversal, because mapped feedback also changes later proposals, and the revised paper says so [Section 6.2] (Weakness 1).

What should change in benchmark design and interpretation, and why (Weakness 2: guidance).

  1. Do not read a pool ranking as a ranking of proposal quality; report the protocol and the uncertainty of every pairwise comparison.

    • Reason: only 30 of 92 resolved comparisons are preserved [Table 2].
  2. Use a larger pool.

    • Reason: from the smallest to the largest pool, the unequal shift D and the score error E fell in all nine tested Mapping size sequences. For transistor fabrication, the median D falls from 78.9 to 37.1 µA and the median E from 148.3 to 26.3 µA between 256 and about 4,096 candidates [Section 7.2, Table 3].
    • The rate of feasibility-status changes also fell in all nine (for example from 47% to 23%) [Appendix J].
    • Rank correlation improved in only five of eight sequences, so a larger pool reduces score distortion but does not certify the ranking [Sections 7.2, 7.3].
  3. Check the pool geometry against the optimizers being compared.

    • Reason: a pool is not equally close to every optimizer. As pools go from concentrated to diffuse, the proposal-projection distance of the LLMs generally grows while that of the GPs shrinks (the LLM trend is weaker in thin-film deposition) [Section 7.1, Figure 4; Appendix J, Figure 19].
    • The LLM-collected source pools are highly concentrated (relative radius R10 of at most 0.01, against about 1 for a Sobol pool), while the radii of GP-collected pools vary by task [Section 5.1, Table 1].
    • With feasible, infeasible, and failed counts matched, pools with a concentrated core and a broad remainder (Mixed) had a mean rank correlation with Generation of 0.84, against 0.62 (Narrow) and 0.69 (Diffuse), and an inverted ranking in 1.4% of bootstrap resamples, against 10.4% and 10.5% [Section 7.2, Table 4].
    • The differences between geometries are not statistically resolved, so geometry is a risk indicator to report, not a rule for a preferred pool [Section 7.3].
  4. Under Mapping, report the proposal-projection distance of each optimizer, and treat a large or unequal distance as a warning.

    • Reason: this distance is how far the credited design is from the proposed one. It falls by about one-third to one-half between 256 and about 4,096 candidates [Section 7.1, Figure 4d], the same range over which D and E fall [Table 3].
    • It needs only candidate inputs and proposals, no new evaluation [Section 7.3].

For LLM optimizers specifically (Weakness 1).

Both mechanisms credit the GP baselines more than the LLMs: restriction raises GP scores and not LLM scores, and Mapping gives GPs more projection credit (in plasma etching, mean credit is negative for both LLMs and positive for both GPs) [Sections 6.1, 6.2].

LLM proposals are also generally farther from diffuse pools, the opposite of the GPs [Figure 19].

On the Luna-collected plasma-etching pool, all four LLM-GP comparisons are reversed under both protocols [Section 5.2].

In these experiments, the comparisons that both mechanisms shift are therefore the LLM-versus-GP comparisons.

How the experiments connect (Weakness 1).

The revised paper follows one line: Section 3 states the observation that raised the question, Section 5 measures how often pool protocols disagree with Generation, Section 6 measures two ways in which the protocols shift optimizers unequally, and Section 7 tests what reduces the disagreement.

Q2. Stability across initializations and pools

How stable are the performance differences and rankings across additional initializations and independently constructed pools? Can uncertainty in the ranking comparisons be reported, particularly where methods have nearly identical AUC values?

Initializations (Weakness 2: one initialization, two tasks).

The revision has three simulator tasks and six initial designs per task, with 1,120 Generation trajectories [Section 4.3].

Over the six designs, Generation resolves six of six optimizer pairs for transistor fabrication, three for thin-film deposition, and five for plasma etching [Appendix F, Table 8, Figure 5].

The ordering does depend on the initial design, so every pool result is compared with Generation from the same initial design, not with the six-design average [Section 5.2].

Independently constructed pools.

We added 24 pools with no optimization provenance: 9 input-random, 9 Sobol, and 6 with a fixed number of feasible candidates (10 or 25 of 50), 38 pools in total [Appendix G].

Counting the comparisons that Generation resolves [Table 2; Appendix G, Tables 10 to 13]:

Pool typeResolved by GenerationPreservedReversed
Source (optimization trajectory)923022
Input-random60118
Sobol60172
Fixed feasible count60256

Reversals are less frequent on independently constructed pools (16 of 180, against 22 of 92) but do not disappear, and most comparisons become unresolved, not preserved. The three plasma-etching Sobol pools reverse none, and Mapping preserves 10 of 12 there [Appendix G, Table 10].

Uncertainty and nearly identical AUC values.

Every pairwise comparison has an adjusted bootstrap interval over whole trajectories [Section 4.3].

  • A pair whose interval includes zero is unresolved. It is counted neither as preserved nor as reversed, so a near tie cannot produce a reversal. Of the 168 source-pool comparisons, 76 are of this kind under Generation and are left out of the 92; at the base thin-film deposition design, for example, Generation resolves only one of six pairs [Section 5.2].

  • A pair that Generation resolves but the pool protocol does not is counted as unresolved (40 of 92), not as a disagreement [Table 2].

  • In five transistor-fabrication reference pools no candidate improves on the initial design, so all four optimizers receive exactly the same score. These pools are reported with zero resolved pairs and an undefined rank correlation, not a correlation of zero [Appendix G, Table 11].

  • We also report the interval of each optimizer's mean AUC50 [Table 8], the rank correlation with its 95% interval for each pool [Tables 9, 11 to 13], and a ceiling from resampled Generation runs: 94%, 64%, and 87% mean-order agreement by task, against observed 53% and 57%, 57%, and 50% and 42% [Table 2; Appendix G].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.