acceptodds
Under review as a conference paper at ICLR 2027

Search Hardness-Aware LLM-Based Problem Formulation for Expensive Simulation-Driven Design

Abstract

Expensive simulation-driven design is widely used in engineering to identify requirement-satisfying designs with as few high-fidelity simulations as possible. Most existing efforts address this challenge by improving optimization algorithms under fixed formulations, yet the formulation itself shapes the search landscape by defining the objectives and constraints optimized by the solver. Recent LLM-based automatic problem formulation methods generate formulations from natural-language requirements, but they mainly focus on design-intent alignment and overlook whether the formulation induces an efficient search process. To address this limitation, we propose SHA-PF, a search hardness-aware LLM-based problem formulation framework. We find that a formulation is more likely to guide efficient search when it prioritizes rare samples with greater progress potential. Based on this finding, SHA-PF defines a formulation search objective guided by search hardness, scoring each candidate formulation according to the priority. SHA-PF then searches the formulation space under this objective through LLM-based generation, repair, and evolutionary refinement. Experiments on the real-world multi-objective benchmark and five expensive antenna design benchmarks show that the formulations discovered by SHA-PF require significantly fewer evaluations to reach the design requirements than other baselines.

Primary area:
optimization
Keywords:
Automated Problem Formulation, Large Language Models, Expensive Simulation-Driven Design

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qdvSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an LLM-based automatic problem formulation method for expensive simulation-driven design problems. The authors revealed findings that rarely observed but promising search samples contribute more to optimization problems, designed an anchor-state-based formulation evaluation criterion to guide formulation search based on the findings, and developed an evolutionary formulation discovering pipline with evolutionary operators. The proposed method is validated on both multi-objective calibration problems and expensive antenna design problems, demonstrating its applicability in real-world tasks. Overall, the topic is interesting, the findings are inspiring, and the method is well-presented.

Critical strengths
  1. The problem is of great practical significance, since formulation is a key step for many real-world problems that lack standard formulations.
  2. It is novel and well-motivated to incorporate search hardness in automatic problem formulation, bringing the community new insight about what we should consider in problem formulation.
  3. The method was validated in two real-world tasks, and the promising results demonstrate significant advantage in expensive simulation-based design, indicating great potential for real-world applications.
  4. Both API cost and time are provided. This is commendable as it is very useful for potential applicants.
Critical weaknesses
  1. The applicability boundaries of the proposed method remain unclear.
  2. The relationship between sample rarity and search hardness is not clearly analyzed.
  3. Since the simulation responses are numerical, the necessity of introducing an LLM as the judge should be further discussed.
Questions for authors
  1. The scope of application should be narrowed. Many optimization problems in real-world applications have one or several well-adopted formulations, which may be exclusive or derived from classical physical or mechanical models through rigorous mathematical methods.

  2. For a given formulation, is the hardness related to the optimization algorithm and initial sample strategy used? The authors claim that they use the initially evaluated data to detect early partial paths toward the target solution, but different algorithms may prefer different paths or search directions.

  3. Is there any risk that the hardness decrease is at the sacrifice of accuracy?

  4. Given the initial expensive evaluations, shaping better formulations or leveraging this information to design landscape-aware optimization algorithms, which one is more practical?

  5. In the Response-State Judge Prompt, the input is limited to curve data, which may be inherited from the used antenna design tasks. The authors should clarify the applicability to other data

4 comments

We thank the reviewer for the constructive comments and insightful questions. We address each point below.

W1 & Q1: Applicability and Scope of SHA-PF

“The applicability boundaries of the proposed method remain unclear. The scope of application should be narrowed. Many optimization problems in real-world applications have one or several well-adopted formulations, which may be exclusive or derived from classical physical or mechanical models through rigorous mathematical methods.”

We thank the reviewer for the valuable suggestion regarding the applicability of our method. We agree that many real-world optimization problems already have well-established formulations derived from rigorous mathematical models. For such problems, if an existing formulation can accurately represent the optimization objectives and support efficient search, automated problem formulation may not be necessary.

We would like to clarify that SHA-PF mainly targets expensive simulation-driven design problems where simulation responses need to be translated into optimization objectives and constraints based on specific design requirements. In these problems, simulators typically produce performance responses for given design parameters, while design requirements may involve multiple performance metrics and thresholds. Since requirements can vary across tasks, existing formulations may not directly apply to new design requirements. Therefore, formulations need to be constructed accordingly.

More importantly, even when multiple formulations correctly represent the same design requirements, they may lead to different search landscapes and thus affect optimization efficiency. The correctness or widespread use of an existing formulation does not necessarily mean that it provides the best search efficiency under a limited evaluation budget. Therefore, SHA-PF is not intended to replace established physical models or mathematical modeling methods. Instead, it aims to identify more search-efficient formulations for design problems where multiple valid formulations exist, while keeping the original design requirements unchanged. We will further clarify this scope in the revised manuscript.


W2: Relationship Between Sample Rarity and Search Hardness

“The relationship between sample rarity and search hardness is not clearly analyzed.”

We thank the reviewer for the comment. We would like to further clarify the relationship between sample rarity and search hardness. In SHA-PF, initial samples are obtained through space-filling sampling to approximate the distribution of requirement satisfaction states in the design space. If a state appears infrequently among the initial samples, its corresponding region may be difficult to reach through global sampling. Therefore, we use sample rarity as an approximate indicator of search hardness.

However, rarity alone does not mean that a state is worth exploring further. We therefore introduce Dominance Support to measure the advantage of a state over other observed states in terms of requirement satisfaction. By combining these two factors, SHA-PF prioritizes rare states with high potential for search progress, thereby guiding the subsequent Formulation Search.

To further investigate the contributions of these two factors, we conducted additional ablation experiments with two variants: w/o Rarity, which selects the Anchor based solely on Dominance Support, and w/o Progress Potential, which selects the Anchor based solely on state frequency. The results are shown in Table R2.

Table R2. Search cost of ablation variants on HSE.

Method / TaskHSE (evaluations / hours)
w/o Anchor Selection– / 11.3
w/o Formulation Repair– / 12.1
w/o Progress Potential (Rarity-only)144 / 8.7
w/o Rarity (Progress-only)77 / 4.7
SHA-PF63 / 3.9

The results show that combining Rarity and Progress Potential achieves better search efficiency than using either factor alone, requiring only 63 evaluations compared with 144 for Rarity-only and 77 for Progress-only. This supports the complementary roles of the two factors in Anchor Selection.

W3: Necessity of the LLM Judge for Numerical Responses

“Since the simulation responses are numerical, the necessity of introducing an LLM as the judge should be further discussed.”

We thank the reviewer for the comment. We agree that design requirements with explicit mathematical rules can be evaluated directly through numerical computation without an LLM. However, SHA-PF targets natural language-driven simulation design problems. Although simulation responses are numerical, design requirements may involve multiple performance metrics, specific response regions, and combinations of conditions.

Therefore, the role of the LLM Judge is not to perform numerical calculations, but to map natural language requirements to simulation responses and determine whether each requirement is satisfied, thereby constructing Satisfaction States. For tasks with explicit evaluation rules, the LLM Judge can be directly replaced with deterministic numerical evaluation without affecting the core Formulation Search mechanism of SHA-PF.


Q2: Dependence of Search Hardness on Optimizers and Initial Sampling

“For a given formulation, is the hardness related to the optimization algorithm and initial sample strategy used? The authors claim that they use the initially evaluated data to detect early partial paths toward the target solution, but different algorithms may prefer different paths or search directions.”

We thank the reviewer for the question. We agree that actual search hardness is related to both the optimization algorithm and the initial sampling strategy. Different optimization algorithms use different search mechanisms. Therefore, even with the same formulation, their search paths and efficiency may differ.

SHA-PF does not attempt to predict the actual search paths of specific optimizers. Instead, it uses initial samples to identify rare states with high potential for search progress and assigns higher search priority to these states through the formulation. The cross-optimizer experiments in Table 1 also show that the same formulation can perform differently across optimizers, while the formulations identified by SHA-PF achieve effective results with multiple optimizers.

Regarding initial sampling, we use Latin Hypercube Sampling (LHS), which is widely adopted in expensive black-box optimization, rather than a sampling method specifically designed for SHA-PF. SHA-PF only uses these initial samples to estimate the state distribution and thus does not require a specific initial sampling strategy. However, different sampling strategies may affect the observed state distribution and the resulting Anchor Selection.


Q3: Trade-off Between Search Efficiency and Solution Accuracy

“Is there any risk that the hardness decrease is at the sacrifice of accuracy?”

We thank the reviewer for the question. If we understand correctly, the reviewer is concerned about whether improved search efficiency comes at the cost of final solution quality or satisfaction of the design requirements.

SHA-PF does not reduce search hardness by relaxing the original design requirements. Instead, it searches for formulations that can guide optimization more efficiently while keeping the design requirements unchanged. Specifically, different formulations may change how candidate solutions are evaluated and ranked through different objective functions and constraint expressions, but they do not change the original design requirements.

In our experiments, all methods use exactly the same original design requirements as the final success criterion. For the five antenna design tasks, a solution is considered successful only if its simulation responses satisfy all the original design requirements. Furthermore, the HBV experiment (Figure 3(b)) shows that SHA-PF not only improves search efficiency but also achieves better NSE and ROCE values for the best feasible solutions than all baselines.

Q4: Formulation Optimization vs. Landscape-Aware Optimization

“Given the initial expensive evaluations, shaping better formulations or leveraging this information to design landscape-aware optimization algorithms, which one is more practical?”

We thank the reviewer for the insightful question. We believe that improving formulations using initial evaluation data and designing landscape-aware optimization algorithms are not conflicting approaches, but rather complementary ones. The former improves how candidate solutions are evaluated and ranked by adjusting objectives and constraints, while the latter improves the search strategy by using landscape information.

From a practical perspective, researchers often already have existing optimization algorithms and their code implementations. Rather than redesigning or modifying these algorithms, SHA-PF can be directly combined with existing optimizers without changing their implementations, improving search efficiency through better formulations.

However, we do not believe that one approach is necessarily better than the other. Instead, using the same initial evaluation data to improve both the formulation and the optimization algorithm may further improve the efficiency of expensive simulation-driven optimization, which is a promising direction for future research.


Q5: Applicability Beyond Curve-Based Simulation Responses

“In the Response-State Judge Prompt, the input is limited to curve data, which may be inherited from the used antenna design tasks. The authors should clarify the applicability to other data”

We thank the reviewer for the question. We would like to clarify that the Response-State Judge Prompt in Appendix B.1 is a specific implementation example for antenna design tasks. It does not mean that SHA-PF is limited to curve data.

The core of SHA-PF is to map simulation responses into binary Satisfaction States based on design requirements, regardless of the specific data format. For different types of simulation responses, only the input descriptions and evaluation rules in the Judge Prompt need to be adjusted accordingly.

For example, in addition to the antenna design tasks based on frequency response curves, we also evaluate SHA-PF on the HBV hydrological model calibration task. In this task, the simulation responses consist of multiple numerical performance metrics rather than curve data.

Reviewer_qdvSOpenReview ↗

Reassessment After Rebuttal

Thanks for the detailed responses.

Reviewer_eveXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel way to solve real-world black-box optimisation problems more effectively -- by optimising the problem formulation with LLMs from natural language. The authors describe their approach and evaluate it empirically, demonstrating performance improvements on real-world benchmarks.

Critical strengths

The paper takes a new and interesting approach to an important problem. It is well written and, as far as I can tell, the empirical evaluation is well done. The experimental results show that the proposed approach is much better than the baselines the authors compare to.

Critical weaknesses

I found myself wondering whether the LLMs were able to come up with a problem formulation that was amenable to more efficient solving because the problems and solutions were part of the training data. A brief discussion of this aspect would be appreciated, as it potentially invalidates the experimental results.

It was also unclear to me whether similarly improved result could be achieved for a "standard" problem formulation with a better initial design. This is not a criticism of the method, but a brief discussion to what extent the different formulations enable fundamentally different forms of solving would be useful for the reader to understand how the proposed method complements existing ones.

Similarly, it would be useful to have a "standard" formulation with Bayesian Optimisation as a baseline in the experiments, or a discussion of why this is not possible or does not make sense here.

Questions for authors

See above.

3 comments

We thank the reviewer for the constructive comments and valuable suggestions. We address each concern below.

W1: Potential Training Data Contamination

“I found myself wondering whether the LLMs were able to come up with a problem formulation that was amenable to more efficient solving because the problems and solutions were part of the training data. A brief discussion of this aspect would be appreciated, as it potentially invalidates the experimental results.”

We thank the reviewer for raising the concern about potential training data contamination. First, all LLMs used in this work are general-purpose pretrained models, without any additional training or fine-tuning on the benchmarks used in our experiments. Although these benchmarks are based on published engineering studies, to the best of our knowledge, there are no publicly available formulation datasets or reference solutions specifically designed to improve optimization search efficiency for these tasks. However, since the pretraining data of general-purpose LLMs are not fully disclosed, we cannot completely rule out the possibility that these models have been exposed to related studies.

Second, SHA-PF does not simply rely on the existing knowledge of LLMs to directly generate formulations. Instead, it uses initial simulation data from the current task to construct the Formulation Search Objective and iteratively improves formulations through search. As shown in Table 1, when using the same LLM, directly generated formulations (LLM-A/B) succeed on only some tasks, while SHA-PF succeeds on all five antenna design tasks. These results suggest that the task-data-driven Formulation Search mechanism plays an important role in improving performance, rather than relying solely on the direct generation ability of LLMs. We will further clarify this point and discuss the potential risk of training data contamination in the revised manuscript.


W2: Complementarity Between Initial Design and Formulation Optimization

“It was also unclear to me whether similarly improved result could be achieved for a "standard" problem formulation with a better initial design. This is not a criticism of the method, but a brief discussion to what extent the different formulations enable fundamentally different forms of solving would be useful for the reader to understand how the proposed method complements existing ones.”

We thank the reviewer for the suggestion. We agree that better initial sampling designs may also improve optimization efficiency. However, identifying an effective initial sampling strategy for different tasks is itself challenging. SHA-PF uses the widely adopted Latin Hypercube Sampling (LHS) and does not rely on a specially optimized initial design. Instead, it uses the available initial evaluation data to further improve the formulation.

We would like to further clarify the difference between these two approaches. Improving the initial design mainly affects the starting point of optimization, while improving the formulation changes how the optimizer evaluates and ranks candidate solutions throughout the search process by modifying the objectives and constraints. Therefore, even with the same initial samples and optimization algorithm, different formulations may still lead to different search directions.

In our experiments, all formulations use the same initial samples and optimizer. The results in Table 1 show that SHA-PF can improve search efficiency through better formulations. This demonstrates that Formulation Optimization can provide additional benefits without changing the initial design, and it can complement more advanced initial sampling methods.

W3: Standard Formulations with Bayesian Optimization

“Similarly, it would be useful to have a "standard" formulation with Bayesian Optimisation as a baseline in the experiments, or a discussion of why this is not possible or does not make sense here.”

We thank the reviewer for the suggestion. First, we would like to clarify that, for the expensive simulation-driven design problems considered in this work, there is no single standard formulation that applies to all design requirements. Researchers typically need to manually construct objectives and constraints based on specific performance metrics and design requirements. Following this traditional modeling approach, we include two baseline formulations, Expert-A and Expert-B. Both were manually developed by domain experts based on the design requirements and have been previously applied and validated in real-world industrial collaboration projects.

Furthermore, SCBO (Scalable Constrained Bayesian Optimization) is itself a constrained Bayesian optimization algorithm. Therefore, Expert-A (SCBO) and Expert-B (SCBO) in Table 1 already represent the experimental setting of combining traditional expert-designed formulations with Bayesian optimization. All methods use the same initial samples, optimizer, and evaluation budget to ensure a fair comparison.

Thank you for the clarifications.

Reviewer_1XHhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SHA-PF discovers problem formulations (objective plus constraints) that let expensive simulation-driven optimization reach requirement-satisfying designs in fewer evaluations. Unlike prior LLM formulation methods that only check design-intent alignment, SHA-PF scores candidate formulations by how well they rank "anchor" states that are rare in the initial evaluated data yet Pareto-dominate many other states, then searches the formulation space with LLM generation, repair, and evolutionary refinement. Evaluated on the HBV rainfall-runoff calibration problem and five HFSS antenna design tasks against two expert and two direct-LLM formulations.

Critical strengths
  • Addresses a real gap. Most LLM formulation work checks whether a formulation is correct and aligned, not whether it induces efficient search. Making search efficiency the formulation-search objective is well-motivated, and the anchor-state criterion gives a concrete handle on "hard but promising".
  • Evaluated on genuinely expensive real simulators (HFSS, roughly 500 to 960 seconds per evaluation) with cross-optimizer, LLM-backbone, and ablation studies that each show the design is not brittle.
Critical weaknesses
  • No repeated runs, seeds, or variance are reported. Table 1 gives single search-cost numbers, and the abstract claims "significantly fewer evaluations" without any statistical test. Search cost is stochastic, so single numbers cannot support that claim.
  • The reported metric excludes the cost of searching for the formulation. SHA-PF runs 25 generations of a population of 5 with a repair step per candidate, while the expert and direct-LLM baselines produce one formulation each. The total-cost comparison is therefore incomplete.
Questions for authors
  1. How many independent runs produced the numbers in Table 1, and how stable is the "fewer evaluations" claim across runs and initial-sample draws given that no variance or seeds are reported?
  2. Section 4.1 states the antenna budget as 300 HFSS simulations, but Table 3's note states 200 and the "saving 110–137 simulations" calculation only matches 200. Which budget was used?
  3. The hardness score h(z) combines rarity and progress potential, but the w/o Anchor Selection ablation replaces the whole criterion with a random anchor. Does the evidence isolate whether both terms matter, or only that non-random anchor selection helps?
  4. The satisfaction state is binary, so a response just below a threshold and one far below it are treated identically. How much does this coarseness affect anchor selection and the resulting formulation?
4 comments

We thank the reviewer for the constructive comments and insightful questions. We address each point below.

W1 & Q1: Repeated Runs and Statistical Reliability

“No repeated runs, seeds, or variance are reported. Table 1 gives single search-cost numbers, and the abstract claims 'significantly fewer evaluations' without any statistical test. Search cost is stochastic, so single numbers cannot support that claim. How many independent runs produced the numbers in Table 1, and how stable is the 'fewer evaluations' claim across runs and initial-sample draws given that no variance or seeds are reported?”

We thank the reviewer for pointing out the lack of independent runs and statistical analysis in our experiments. In the original experiments, each configuration was evaluated in one independent run. Since all five antenna design tasks rely on time-consuming HFSS electromagnetic simulations, a single optimization run takes approximately 16 hours on average to exhaust the optimization budget, making large-scale repeated experiments computationally expensive. We have conducted two additional runs with different random seeds and independent LHS initial samples, resulting in three independent runs for each configuration. The additional results are presented in Table R1.

Table R1. Search costs and success rates of different formulation methods over three independent runs on five antenna design tasks.

Formulation (SCBO)HAEHSLHSECSLWHG
Expert-A181.3 ± 32.3 / 10.1 ± 3.4 (1/3)200.0 ± 0.0 / 8.8 ± 0.3 (0/3)178.7 ± 37.0 / 14.1 ± 2.5 (1/3)200.0 ± 0.0 / 52.9 ± 21.2 (0/3)200.0 ± 0.0 / 15.6 ± 2.5 (0/3)
Expert-B81.3 ± 102.8 / 3.5 ± 3.7 (2/3)200.0 ± 0.0 / 18.1 ± 3.6 (0/3)200.0 ± 0.0 / 13.6 ± 2.2 (0/3)200.0 ± 0.0 / 52.7 ± 26.0 (0/3)146.7 ± 92.4 / 14.7 ± 8.0 (1/3)
LLM-A157.0 ± 74.5 / 6.8 ± 3.4 (1/3)200.0 ± 0.0 / 10.2 ± 1.3 (0/3)200.0 ± 0.0 / 15.3 ± 3.7 (0/3)132.3 ± 85.7 / 15.9 ± 9.1 (2/3)200.0 ± 0.0 / 15.5 ± 3.0 (0/3)
LLM-B200.0 ± 0.0 / 12.8 ± 0.9 (0/3)200.0 ± 0.0 / 13.3 ± 0.3 (0/3)200.0 ± 0.0 / 15.1 ± 2.8 (0/3)200.0 ± 0.0 / 23.9 ± 7.5 (0/3)200.0 ± 0.0 / 18.1 ± 4.3 (0/3)
SHA-PF53.0 ± 22.5 / 3.3 ± 2.3 (3/3)125.7 ± 64.5 / 8.7 ± 3.8 (2/3)91.7 ± 24.9 / 5.5 ± 1.4 (3/3)16.3 ± 10.1 / 2.2 ± 0.5 (3/3)101.7 ± 35.0 / 11.2 ± 2.3 (3/3)

Note: Entries are mean ± standard deviation of evaluations / hours over three independent runs, followed by the number of runs that found a requirement-satisfying antenna design (out of three). The evaluation budget is 200 for the subsequent optimization stage, excluding the 100 shared initial evaluations. Runs that fail to find a requirement-satisfying design within the budget are recorded as 200 evaluations.

The results show that SHA-PF achieves the lowest mean number of evaluations across all five tasks, with 14 successful runs out of 15, including three successful runs on four tasks and two on HSL. These results indicate that SHA-PF maintains its advantage in average search efficiency and achieves a higher observed success rate than the baselines across different random seeds. We will include the random seeds, repeated experimental results, and corresponding statistics in the revised manuscript. We will also revise our claims regarding search efficiency and stability to ensure that they are supported by the experimental evidence.

W2: Total Computational Cost of Formulation Search

“The reported metric excludes the cost of searching for the formulation. SHA-PF runs 25 generations of a population of 5 with a repair step per candidate, while the expert and direct-LLM baselines produce one formulation each. The total-cost comparison is therefore incomplete.”

We thank the reviewer for the suggestion regarding the overall computational cost. First, expert-designed formulations are not developed in a single step. Instead, domain experts typically need to revise and test them multiple times based on the design requirements. The Expert Formulations used in this work also required substantial manual effort and time during previous real-world industrial collaboration projects.

Compared with Direct LLM Formulations, SHA-PF introduces additional generation, repair, and search steps to improve formulation quality. The corresponding LLM API costs and time overhead are reported in Figure 5(b). Considering that a single HFSS simulation typically takes tens of minutes, reducing the number of subsequent simulation evaluations may offset the additional cost of Formulation Search. Furthermore, for the same design requirements, the generated formulation can be reused in future optimization runs without repeating Formulation Search, further reducing the average upfront cost per run.


Q2: Clarification of the Evaluation Budget

“Section 4.1 states the antenna budget as 300 HFSS simulations, but Table 3's note states 200 and the 'saving 110–137 simulations' calculation only matches 200. Which budget was used?”

We thank the reviewer for pointing out this potentially confusing description. We would like to clarify that the total budget for all antenna design tasks is 300 HFSS simulation evaluations, including 100 initial evaluations and 200 subsequent optimization evaluations.

Since all methods share the same 100 initial evaluations, we report only the number of evaluations during the subsequent optimization stage to provide a clearer and fairer comparison of how different formulations affect search efficiency. The shared 100 initial evaluations are therefore excluded from the reported results, as indicated by "excluding the initial evaluations" in the note of Table 3.

Thus, both the budget of 200 evaluations in Table 3 and the reported savings of 110–137 evaluations refer to the subsequent optimization stage. These are fully consistent with the total budget of 300 evaluations described in Section 4.1. We will clarify this distinction in the revised manuscript.


Q3: Contributions of Rarity and Progress Potential

“The hardness score h(z) combines rarity and progress potential, but the w/o Anchor Selection ablation replaces the whole criterion with a random anchor. Does the evidence isolate whether both terms matter, or only that non-random anchor selection helps?”

We thank the reviewer for pointing out this issue. To address it, we have designed two additional ablation variants: w/o Rarity, which selects the Anchor based only on Dominance Support, and w/o Progress Potential, which selects the Anchor based only on state frequency. The additional experimental results are presented in Table R2.

Table R2. Search cost of ablation variants on HSE.

Method / TaskHSE (evaluations / hours)
w/o Anchor Selection– / 11.3
w/o Formulation Repair– / 12.1
w/o Progress Potential (Rarity-only)144 / 8.7
w/o Rarity (Progress-only)77 / 4.7
SHA-PF63 / 3.9

The results show that combining Rarity and Progress Potential achieves better search efficiency than using either factor alone, requiring only 63 evaluations compared with 144 for Rarity-only and 77 for Progress-only. This supports the complementary roles of the two factors in Anchor Selection.

Q4: Impact of Binary State Granularity

“The satisfaction state is binary, so a response just below a threshold and one far below it are treated identically. How much does this coarseness affect anchor selection and the resulting formulation?”

We thank the reviewer for the insightful question. We agree that binary states cannot distinguish how far different responses within the same state are from the design requirement thresholds, while finer-grained state representations can preserve more information.

However, the benefit of finer-grained representations depends on the number of initial samples and their distribution across states. When a state contains many samples, further dividing it may help distinguish different levels of requirement satisfaction. In contrast, when a state contains only a few samples, further division may provide limited additional information while making the state distribution even sparser.

This issue is particularly relevant to our experiments. We further examined the state distributions and found that, among the 100 initial samples in each task, the selected Anchor States contain only 4 samples for CSL and HAE, and only 1 sample for HSE, HSL, and WHG. These results show that the selected Anchor States are already very sparse, suggesting that further dividing them may offer limited benefits.

Thanks to the authors for their thorough and detailed rebuttal. The authors have addressed my concerns with more work & new results, which I believe have improved the paper, so I am maintaining my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.