- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to evaluate LLM-generated solver configurations under limited search budgets and distinguish the quality of a selected configuration from the contribution of the method that generated it. The authors propose PACER, which fixes the scoring rule, validation-based selection, held-out testing, and solver-call accounting, and compares five conventional configuration search methods against a remeasured historical LLM-generated configuration.
On GISP-easy, this reference improves PDI by 34.3% relative to the stored default measurements. As the budget increases from 25 to 60 trials along the same search trajectories, the number of results exceeding the reference rises from 5/15 to 11/15. In addition, replays of six historical configurations across three problem families yield improvements in both PDI and runtime in all point estimates. An ablation on CA-easy shows that removing the learned MIP start preserves nearly all of the PDI gain. The paper offers a valuable evaluation perspective and useful empirical evidence, but remains limited in its methodological contributions to LLM-based solver configuration search, direct comparisons of search mechanisms, and experimental coverage.
Critical strengths
-
Instead of merely reporting that the LLM outperforms the solver’s default setting, this work further examines the search budget required by conventional methods to achieve equivalent quality, demonstrating strong methodological value.
-
The evaluation pipeline is relatively comprehensive, and key comparison conditions are publicly available, standardizing the feedback loop, test selection procedure and invocation accounting across five classical search methods. These designs improve the interpretability of the results.
Critical weaknesses
-
Only one set of evaluation criteria and protocols is designed, and no new LLM-based solver configuration method is proposed. Its novelty and contributions hardly meet the publication bar for ICLR.
-
The core controlled comparison experiments are conducted solely on the GISP-easy task. Although replay experiments across six tasks verify the effectiveness of retained configurations, they do not compare the performance of search methods versus LLM-based configuration on these tasks, leading to insufficient cross-task evidence.
-
The baseline comparisons only include fundamental methods such as GP, RANDOM and TPE. More efficient search-based methods (e.g., HEBO) are absent. It is therefore difficult to demonstrate the advantages of the LLM solver configuration approach over general search methods. Additional experiments comparing against SOTA baselines are recommended.
Questions for authors
-
The solver-related knowledge and parameter biases acquired by the LLM during pretraining may incline it to suggest certain common settings or make only minor modifications to default configurations. Could the authors clarify to what extent such prior biases affect the final configurations and their performance? Have the authors examined result stability across different models or prompts, so as to disentangle the contributions from pretrained priors and feedback-driven search?
-
What exactly constitutes the “solver feedback” in the historical LLM workflow? Does it only consist of performance metrics such as PDI and runtime, or does it also include per-instance outcomes and complete CPLEX solver logs? If logs are utilized, does the LLM analyze the solving behaviors within them to adjust configurations accordingly?
3 comments
Response to Reviewer 68Kp (2/2)
Q1. Pretrained priors and model/prompt stability. The controlled GISP-easy reference changes two global CPLEX controls, governing branching-variable selection and backtracking. Both lie in the 31-axis categorical search space available to every conventional proposer (Sec. 4.1). This makes the reference directly comparable as a configuration, with no privileged parameter settings. Determining why the LLM selected those controls, however, requires model identities, prompt variants, sampling settings, and full feedback histories. The revised Appendix C (blue) distinguishes preserved evaluator outputs from historical prompt-visible feedback, whose exact contents were not retained. Disentangling pretrained knowledge from feedback-driven search is an important causal question and a natural next step once those records can be prospectively controlled. PACER supplies the common evaluator and cost accounting necessary for such an experiment.
Q2. Historical solver feedback. The retained records document the actual evaluation history: for GISP-easy, 1,164 sandbox CPLEX calls produced 12 distinct candidate configurations, followed by nine platform-validation rounds of 100 calls each (App. C). The archive preserves candidate configurations and round-level outcomes, while App. K reports historical PDI and runtime measurements. The exact information embedded in each historical prompt—including whether detailed per-instance traces or complete solver logs were shown to the LLM—is not recorded. This distinction between archived evaluator output and prompt-visible feedback is essential for valid causal attribution, and is now explicit in revised Appendix C (blue). The current study uses the verifiable artifact and execution records. A prospective feedback-ablation study would build directly on PACER to isolate how solver metrics, logs, and semantic information influence proposal quality.
We appreciate your recognition of PACER's methodological value and hope our responses further establish the significance of its theoretical and empirical contributions, and we respectfully ask you to reconsider your rating in light of the revised manuscript.
I appreciate the authors' responses and respect their opinion. I will maintain my rating.
Response to Reviewer 68Kp (1/2)
We thank the reviewer for recognizing the value of examining conventional-search budgets rather than relying only on improvement over solver defaults. We respectfully disagree with the conclusion that a new LLM tuning algorithm or a demonstration of LLM superiority is required for PACER to make a substantial contribution. The paper addresses a different fundamental question: how to attribute and interpret the performance of LLM-generated solver configurations when proposal mechanisms have different search opportunities. Its theory, controlled experiments, and component-level analyses make meaningful progress on that question.
W1. Novelty and contribution. The key insight is that artifact performance and proposer effectiveness are distinct quantities. Proposition 1 makes the dependence on proposal quality and budget explicit for adaptive searches, and PACER connects this insight to solver-call resources and held-out performance. The empirical results are substantive: 34.3% held-out PDI improvement for the historical reference on GISP-easy, improvement on 95/100 held-out instances, and a shift from 5/15 to 11/15 conventional-search endpoints exceeding that reference as the nested search budget grows (Table 2). The CA-easy component ablation further isolates the source of nearly all its PDI gain (Table 17). This combination of theory, budget-sensitive evaluation, and performance attribution is the contribution, rather than a new optimizer that would itself need this kind of evaluation. Modular reuse of established methods strengthens this contribution by making the evaluation accessible to a broad range of existing and future proposers, as clarified in the revised Introduction, C1, and Section 3.3 (blue).
W2. Cross-task evidence. We evaluate performance on three distinct binary MILP families, each with easy and medium variants. All six retained artifacts improve both PDI and runtime in the reported held-out point estimates (Table 3), demonstrating that the performance observations extend beyond a single benchmark. The controlled proposer comparison on GISP-easy provides a complementary, deeper analysis of search budgets. The revised Section 4.3 (blue) clarifies the generator regimes, and the existing Sections 4.2–4.3 explicitly separate cross-task artifact performance from controlled budget-dependent proposer comparisons. Extending the latter across all problem families is a valuable next stage, building on the reproducible evaluator established here.
W3. Baselines and HEBO. The study includes five methods—GP, TPE, Random, SMAC3, and RF-EI—rather than only the three identified in the review (Table 4). These cover density-based search, Gaussian-process search, random forests, and direct random sampling, all under the same categorical search domain and billed solver-call budgets. We respectfully disagree that HEBO is necessary to substantiate the central finding: among these independently implemented conventional-search methods, the number of endpoints surpassing the historical LLM reference increases from 5/15 to 11/15 as budget increases (Table 2). HEBO and further strong baselines would extend the comparative landscape and may change the detailed attainment curves. The existing five-method study already establishes the budget dependence that motivates PACER. The reusable
ask/tell/failinterface in revised Section 3.3 (blue) is designed precisely to let future baselines such as HEBO enter the same evaluation without changing its scoring or accounting.