- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces an LLM-based automatic problem formulation method for expensive simulation-driven design problems. The authors revealed findings that rarely observed but promising search samples contribute more to optimization problems, designed an anchor-state-based formulation evaluation criterion to guide formulation search based on the findings, and developed an evolutionary formulation discovering pipline with evolutionary operators. The proposed method is validated on both multi-objective calibration problems and expensive antenna design problems, demonstrating its applicability in real-world tasks. Overall, the topic is interesting, the findings are inspiring, and the method is well-presented.
Critical strengths
- The problem is of great practical significance, since formulation is a key step for many real-world problems that lack standard formulations.
- It is novel and well-motivated to incorporate search hardness in automatic problem formulation, bringing the community new insight about what we should consider in problem formulation.
- The method was validated in two real-world tasks, and the promising results demonstrate significant advantage in expensive simulation-based design, indicating great potential for real-world applications.
- Both API cost and time are provided. This is commendable as it is very useful for potential applicants.
Critical weaknesses
- The applicability boundaries of the proposed method remain unclear.
- The relationship between sample rarity and search hardness is not clearly analyzed.
- Since the simulation responses are numerical, the necessity of introducing an LLM as the judge should be further discussed.
Questions for authors
-
The scope of application should be narrowed. Many optimization problems in real-world applications have one or several well-adopted formulations, which may be exclusive or derived from classical physical or mechanical models through rigorous mathematical methods.
-
For a given formulation, is the hardness related to the optimization algorithm and initial sample strategy used? The authors claim that they use the initially evaluated data to detect early partial paths toward the target solution, but different algorithms may prefer different paths or search directions.
-
Is there any risk that the hardness decrease is at the sacrifice of accuracy?
-
Given the initial expensive evaluations, shaping better formulations or leveraging this information to design landscape-aware optimization algorithms, which one is more practical?
-
In the Response-State Judge Prompt, the input is limited to curve data, which may be inherited from the used antenna design tasks. The authors should clarify the applicability to other data
4 comments
W3: Necessity of the LLM Judge for Numerical Responses
“Since the simulation responses are numerical, the necessity of introducing an LLM as the judge should be further discussed.”
We thank the reviewer for the comment. We agree that design requirements with explicit mathematical rules can be evaluated directly through numerical computation without an LLM. However, SHA-PF targets natural language-driven simulation design problems. Although simulation responses are numerical, design requirements may involve multiple performance metrics, specific response regions, and combinations of conditions.
Therefore, the role of the LLM Judge is not to perform numerical calculations, but to map natural language requirements to simulation responses and determine whether each requirement is satisfied, thereby constructing Satisfaction States. For tasks with explicit evaluation rules, the LLM Judge can be directly replaced with deterministic numerical evaluation without affecting the core Formulation Search mechanism of SHA-PF.
Q2: Dependence of Search Hardness on Optimizers and Initial Sampling
“For a given formulation, is the hardness related to the optimization algorithm and initial sample strategy used? The authors claim that they use the initially evaluated data to detect early partial paths toward the target solution, but different algorithms may prefer different paths or search directions.”
We thank the reviewer for the question. We agree that actual search hardness is related to both the optimization algorithm and the initial sampling strategy. Different optimization algorithms use different search mechanisms. Therefore, even with the same formulation, their search paths and efficiency may differ.
SHA-PF does not attempt to predict the actual search paths of specific optimizers. Instead, it uses initial samples to identify rare states with high potential for search progress and assigns higher search priority to these states through the formulation. The cross-optimizer experiments in Table 1 also show that the same formulation can perform differently across optimizers, while the formulations identified by SHA-PF achieve effective results with multiple optimizers.
Regarding initial sampling, we use Latin Hypercube Sampling (LHS), which is widely adopted in expensive black-box optimization, rather than a sampling method specifically designed for SHA-PF. SHA-PF only uses these initial samples to estimate the state distribution and thus does not require a specific initial sampling strategy. However, different sampling strategies may affect the observed state distribution and the resulting Anchor Selection.
Q3: Trade-off Between Search Efficiency and Solution Accuracy
“Is there any risk that the hardness decrease is at the sacrifice of accuracy?”
We thank the reviewer for the question. If we understand correctly, the reviewer is concerned about whether improved search efficiency comes at the cost of final solution quality or satisfaction of the design requirements.
SHA-PF does not reduce search hardness by relaxing the original design requirements. Instead, it searches for formulations that can guide optimization more efficiently while keeping the design requirements unchanged. Specifically, different formulations may change how candidate solutions are evaluated and ranked through different objective functions and constraint expressions, but they do not change the original design requirements.
In our experiments, all methods use exactly the same original design requirements as the final success criterion. For the five antenna design tasks, a solution is considered successful only if its simulation responses satisfy all the original design requirements. Furthermore, the HBV experiment (Figure 3(b)) shows that SHA-PF not only improves search efficiency but also achieves better NSE and ROCE values for the best feasible solutions than all baselines.
Q4: Formulation Optimization vs. Landscape-Aware Optimization
“Given the initial expensive evaluations, shaping better formulations or leveraging this information to design landscape-aware optimization algorithms, which one is more practical?”
We thank the reviewer for the insightful question. We believe that improving formulations using initial evaluation data and designing landscape-aware optimization algorithms are not conflicting approaches, but rather complementary ones. The former improves how candidate solutions are evaluated and ranked by adjusting objectives and constraints, while the latter improves the search strategy by using landscape information.
From a practical perspective, researchers often already have existing optimization algorithms and their code implementations. Rather than redesigning or modifying these algorithms, SHA-PF can be directly combined with existing optimizers without changing their implementations, improving search efficiency through better formulations.
However, we do not believe that one approach is necessarily better than the other. Instead, using the same initial evaluation data to improve both the formulation and the optimization algorithm may further improve the efficiency of expensive simulation-driven optimization, which is a promising direction for future research.
Q5: Applicability Beyond Curve-Based Simulation Responses
“In the Response-State Judge Prompt, the input is limited to curve data, which may be inherited from the used antenna design tasks. The authors should clarify the applicability to other data”
We thank the reviewer for the question. We would like to clarify that the Response-State Judge Prompt in Appendix B.1 is a specific implementation example for antenna design tasks. It does not mean that SHA-PF is limited to curve data.
The core of SHA-PF is to map simulation responses into binary Satisfaction States based on design requirements, regardless of the specific data format. For different types of simulation responses, only the input descriptions and evaluation rules in the Judge Prompt need to be adjusted accordingly.
For example, in addition to the antenna design tasks based on frequency response curves, we also evaluate SHA-PF on the HBV hydrological model calibration task. In this task, the simulation responses consist of multiple numerical performance metrics rather than curve data.
Reassessment After Rebuttal
Thanks for the detailed responses.
We thank the reviewer for the constructive comments and insightful questions. We address each point below.
W1 & Q1: Applicability and Scope of SHA-PF
We thank the reviewer for the valuable suggestion regarding the applicability of our method. We agree that many real-world optimization problems already have well-established formulations derived from rigorous mathematical models. For such problems, if an existing formulation can accurately represent the optimization objectives and support efficient search, automated problem formulation may not be necessary.
We would like to clarify that SHA-PF mainly targets expensive simulation-driven design problems where simulation responses need to be translated into optimization objectives and constraints based on specific design requirements. In these problems, simulators typically produce performance responses for given design parameters, while design requirements may involve multiple performance metrics and thresholds. Since requirements can vary across tasks, existing formulations may not directly apply to new design requirements. Therefore, formulations need to be constructed accordingly.
More importantly, even when multiple formulations correctly represent the same design requirements, they may lead to different search landscapes and thus affect optimization efficiency. The correctness or widespread use of an existing formulation does not necessarily mean that it provides the best search efficiency under a limited evaluation budget. Therefore, SHA-PF is not intended to replace established physical models or mathematical modeling methods. Instead, it aims to identify more search-efficient formulations for design problems where multiple valid formulations exist, while keeping the original design requirements unchanged. We will further clarify this scope in the revised manuscript.
W2: Relationship Between Sample Rarity and Search Hardness
We thank the reviewer for the comment. We would like to further clarify the relationship between sample rarity and search hardness. In SHA-PF, initial samples are obtained through space-filling sampling to approximate the distribution of requirement satisfaction states in the design space. If a state appears infrequently among the initial samples, its corresponding region may be difficult to reach through global sampling. Therefore, we use sample rarity as an approximate indicator of search hardness.
However, rarity alone does not mean that a state is worth exploring further. We therefore introduce Dominance Support to measure the advantage of a state over other observed states in terms of requirement satisfaction. By combining these two factors, SHA-PF prioritizes rare states with high potential for search progress, thereby guiding the subsequent Formulation Search.
To further investigate the contributions of these two factors, we conducted additional ablation experiments with two variants: w/o Rarity, which selects the Anchor based solely on Dominance Support, and w/o Progress Potential, which selects the Anchor based solely on state frequency. The results are shown in Table R2.
Table R2. Search cost of ablation variants on HSE.
The results show that combining Rarity and Progress Potential achieves better search efficiency than using either factor alone, requiring only 63 evaluations compared with 144 for Rarity-only and 77 for Progress-only. This supports the complementary roles of the two factors in Anchor Selection.