- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether candidate-pool benchmarks faithfully evaluate LLM optimizers by comparing selection from stored candidates and nearest-neighbor mapping with direct evaluation of proposed designs. Experiments on materials datasets and two physics-based simulators show that optimizer rankings can differ across evaluation protocols and that mapping introduces optimizer-dependent distortions in feasibility and objective values.
Critical strengths
- The paper presents an approach for analyzing how candidate pools and evaluation protocols affect the assessment of LLM optimizers.
- The analysis uses physics-based simulators to directly evaluate optimizer-proposed designs.
Critical weaknesses
- The importance of the motivation and research question is not sufficiently established.
- The paper does not provide a practical solution alternative to evaluation based on previously measured data.
- Trajectory imprinting remains a hypothesis, and the evidence is insufficient to establish its presence and causal influence on optimization behavior.
- Some aspects of the experimental setup remain unclear, and the experiments do not sufficiently isolate the effects of different candidate pools.
Questions for authors
- Section 3: Could the observed relationship simply reflect overly small search steps by some LLMs, with models taking larger steps reaching the target region sooner, independently of trajectory imprinting?
- Section 4: Are the LLMs informed about how each candidate pool was collected? If so, could they use this information to guide their choices under Selection?
- Section 5.2, Figure 4: Could the larger projection credit observed for GP optimizers be driven primarily by LLM-generated pools, while projection errors are smaller on GP-generated pools? Reporting the results separately for each pool would help distinguish these possibilities.
- Section 6: How was the initial design z_0 chosen? Although its parameter values are provided in Appendix A, the rationale for selecting this design is unclear, and the geometric diagnostics may be sensitive to this choice.
3 comments
Q3. Projection credit by pool source
Section 5.2, Figure 4: Could the larger projection credit observed for GP optimizers be driven primarily by LLM-generated pools, while projection errors are smaller on GP-generated pools? Reporting the results separately for each pool would help distinguish these possibilities.
Largely yes for two of the three tasks, and we now report the credit separately for every source pool [Appendix I, Table 16; the credit figure is Figure 3 in the revision] (Weakness 4: isolating pool effects). GP-minus-LLM credit by pool source:
| Task (unit) | LLM-collected pools (Luna, Sonnet) | GP-collected pools (GP-UCB, GP-EI) | Human-collected pool | Consistent with the suggestion? |
|---|---|---|---|---|
| Thin-film deposition (percentage points) | +5.94, +5.16 | +2.78, -2.15 | +1.52 | Yes |
| Plasma etching (loss units) | +31.4, +28.0 | +18.2, +9.7 | no pool | Yes, although still positive on GP-collected pools |
| Transistor fabrication (µA) | +109.9, -4.0 | +73.8, -25.4 | +85.5 | No |
Each value is the mean credit of the two GPs minus the mean credit of the two LLMs on that pool; a positive value means Mapping credits the GPs more. Table 16 gives the credit of each optimizer on each pool.
-
In thin-film deposition and plasma etching the gap is largest on the two LLM-collected pools and smallest on the GP-EI-collected pool, as the reviewer suggests.
-
It is not confined to LLM-collected pools. The gap is positive in 11 of 14 task-pool cells, including both human-collected pools and all three GP-UCB-collected pools. In transistor fabrication the pattern does not hold: the gap is -4.0 µA on the Sonnet-collected pool and +85.5 and +73.8 µA on the human- and GP-UCB-collected pools.
What a benchmark user can do about it (Weakness 2: practical alternative).
Section 7 gives three steps; the first two need no new evaluation.
(i) Enlarge the pool: with candidates drawn from optimization trajectories, the GP-minus-LLM credit gap falls from 49.8 to 0.4 µA, from 12.4 to 2.6 percentage points, and from 20.0 to 3.3 loss units between 256 candidates and the largest pool, and the score error falls in all nine tested sequences; with Sobol-only pools the credit gap did not narrow consistently [Section 7.2, Table 3; Appendix J].
(ii) Report the proposal-projection distance of each optimizer; it needs only inputs and proposals [Section 7.1, Figure 4].
(iii) If a small budget exists, evaluate a sample of original proposals to measure the credit directly [Section 7.3].
A larger pool reduced score distortion but improved rank correlation in only five of eight sequences, so these steps lower the risk and do not replace Generation.
Q4. Choice of initial design and sensitivity
Section 6: How was the initial design z_0 chosen? Although its parameter values are provided in Appendix A, the rationale for selecting this design is unclear, and the geometric diagnostics may be sensitive to this choice.
How it was chosen.
Each task has one base initial design, and the 50-candidate source pools were collected starting from it.
-
Transistor fabrication and thin-film deposition: the base designs are poor starting points, as at the start of process development, where the first recipe is usually far from the optimum. The transistor-fabrication design is feasible with an on-current of 1.65 x 10^(-6) µA, against the 300 µA target given in the prompt. The thin-film deposition design violates the growth-rate constraint (0.41 µm/h against the required 3 µm/h) and has a nonuniformity of 93.4% [Appendix A].
-
Plasma etching: the base design is the first draw of a uniform random sample of the input box (loss 98.5) [Appendix A].
Sensitivity (Weakness 4).
The other five initial designs of each task were drawn at random from an evaluated Sobol set of 4,096 points with a fixed seed, with no selection on objective value. For the two tasks with a constraint, the draw was stratified so that three of the six designs are feasible and three infeasible [Appendix A].
Generation is reported for all six designs: it resolves six of six optimizer pairs for transistor fabrication, three for thin-film deposition, and five for plasma etching [Appendix F, Table 8].
Because the ordering depends on the initial design, each pool is compared with Generation from the same initial design [Section 5.2], and the pool-size and geometry studies use one of the randomly drawn designs, not the base design; that design was chosen for each task after inspecting how well Generation separates the optimizers there [Appendix J].
Geometric diagnostics.
The trajectory-imprinting radii in Table 1 are defined about the segment from a pool's own initial design to its best recorded design, so they describe that pool as it was collected. We do not claim the same values for a different starting point [Section 5.1; Appendix D].
Dear Reviewer 1Mmi,
We sincerely thank you for your time, careful review, and constructive feedback. Your comments have been invaluable in helping us improve our manuscript.
We have uploaded a revised manuscript and provided detailed responses to your concerns, including additional experiments and clarifications.
With approximately two hours remaining in the discussion period, we would be very grateful if you could take a final look at our responses and let us know whether they have adequately addressed your main concerns. If any questions or concerns remain, we would be more than happy to provide further clarification before the discussion closes.
We sincerely appreciate your time and thoughtful consideration, especially during this busy review period.
Best regards,
The Authors
We thank the reviewer for the careful reading and the specific questions. each pointed to something the paper had left unclear. We have uploaded a revised manuscript addressing the reviewers' concerns. All section, figure, and table references below refer to this revised version.
Q1. Step size versus trajectory imprinting
Yes, it could. The step-size association cannot separate "some LLMs take steps that are too small" from an effect of pool geometry, and step size and reaching time are computed from the same trajectories. The revised paper states this and no longer uses the association as evidence of trajectory imprinting [Section 3, last paragraph; Appendix E].
The association is also not universal: in thin-film deposition it is -0.82 for the LLM runs, while GP-UCB shows +0.71 on the source pools (the 50-candidate pools recorded along optimization trajectories), and similar negative values appear on some pools with no optimization provenance [Section 5.1].
Why the question still matters (Weakness 1).
Figure 2 is the observation that raised the question, not the evidence for the answer. In a finite materials pool (BN, 12 LLMs), LLMs that take larger steps reach the target sooner, and a prompt instruction to search farther or nearer changes the number of evaluations to the target [Section 3, Figure 2].
So a pool score may reflect how an optimizer moves among stored candidates, not the quality of the designs it would propose. We test this directly: on the 14 source pools, Generation resolves 92 optimizer comparisons (Bonferroni-adjusted 95% bootstrap interval excludes zero), and Selection or Mapping preserves 30, leaves 40 unresolved, and reverses 22 [Section 5.2, Table 2].
Status of trajectory imprinting (Weakness 3).
It is now a measured property of a pool, not a hypothesis about cause. We measure how closely the candidates lie to the segment from the initial design to the best recorded design. The LLM-collected source pools have relative radius R10 of at most 0.01 and R90 between 0.02 and 0.55, against about 1 for a Sobol pool [Section 5.1, Table 1].
We make no claim that this concentration causes the step-size association.
Q2. Pool provenance disclosed to LLMs
No. Under Selection the LLMs receive the candidate input values with neutral identifiers that are reassigned so that they carry no information. They do not receive the collector's identity, the order in which candidates were acquired, or any outcome that has not been observed in the run [Section 4.1; Appendix A; Appendix K].
The distribution of the candidate inputs is of course visible. In a controlled experiment, showing or hiding the candidate inputs produced no resolved effect for either LLM [Section 6.1] (Weakness 4: setup).