acceptodds
Under review as a conference paper at ICLR 2027

Evaluating LLM-Generated Solver Configurations under Controlled Search Budgets

Abstract

Large language model (LLM) workflows can generate solver configurations that improve optimization performance. A strong configuration establishes artifact quality, while attribution to the proposal mechanism requires comparison with alternative search under a common evaluation procedure and budget. We introduce PACER (Proposer Ablation for Configuration Evaluation under Matched Resources), a protocol that fixes scoring, validation selection, held-out testing, and solver-call accounting while varying the configuration-search method and validation budget. On the easy generalized independent set problem (GISP-easy), a remeasured historical LLM-generated IBM ILOG CPLEX configuration improves the held-out primal-dual integral (PDI) by . An independent eight-trial snapshot yields 2 of 15 conventional-search endpoints above this reference. Along the nested searches, the count increases from 5 of 15 after 25 trials to 11 of 15 after 60 trials. Replays of six configurations retained from the historical workflow across three problem families improve both PDI and wall-clock time in every point estimate. An ablation of the easy combinatorial-auction artifact shows that fixed CPLEX settings account for nearly all of its measured PDI gain. These results show how search opportunity changes the interpretation of an LLM-generated solver configuration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Optimization Solver, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_68KpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to evaluate LLM-generated solver configurations under limited search budgets and distinguish the quality of a selected configuration from the contribution of the method that generated it. The authors propose PACER, which fixes the scoring rule, validation-based selection, held-out testing, and solver-call accounting, and compares five conventional configuration search methods against a remeasured historical LLM-generated configuration.

On GISP-easy, this reference improves PDI by 34.3% relative to the stored default measurements. As the budget increases from 25 to 60 trials along the same search trajectories, the number of results exceeding the reference rises from 5/15 to 11/15. In addition, replays of six historical configurations across three problem families yield improvements in both PDI and runtime in all point estimates. An ablation on CA-easy shows that removing the learned MIP start preserves nearly all of the PDI gain. The paper offers a valuable evaluation perspective and useful empirical evidence, but remains limited in its methodological contributions to LLM-based solver configuration search, direct comparisons of search mechanisms, and experimental coverage.

Critical strengths
  1. Instead of merely reporting that the LLM outperforms the solver’s default setting, this work further examines the search budget required by conventional methods to achieve equivalent quality, demonstrating strong methodological value.

  2. The evaluation pipeline is relatively comprehensive, and key comparison conditions are publicly available, standardizing the feedback loop, test selection procedure and invocation accounting across five classical search methods. These designs improve the interpretability of the results.

Critical weaknesses
  1. Only one set of evaluation criteria and protocols is designed, and no new LLM-based solver configuration method is proposed. Its novelty and contributions hardly meet the publication bar for ICLR.

  2. The core controlled comparison experiments are conducted solely on the GISP-easy task. Although replay experiments across six tasks verify the effectiveness of retained configurations, they do not compare the performance of search methods versus LLM-based configuration on these tasks, leading to insufficient cross-task evidence.

  3. The baseline comparisons only include fundamental methods such as GP, RANDOM and TPE. More efficient search-based methods (e.g., HEBO) are absent. It is therefore difficult to demonstrate the advantages of the LLM solver configuration approach over general search methods. Additional experiments comparing against SOTA baselines are recommended.

Questions for authors
  1. The solver-related knowledge and parameter biases acquired by the LLM during pretraining may incline it to suggest certain common settings or make only minor modifications to default configurations. Could the authors clarify to what extent such prior biases affect the final configurations and their performance? Have the authors examined result stability across different models or prompts, so as to disentangle the contributions from pretrained priors and feedback-driven search?

  2. What exactly constitutes the “solver feedback” in the historical LLM workflow? Does it only consist of performance metrics such as PDI and runtime, or does it also include per-instance outcomes and complete CPLEX solver logs? If logs are utilized, does the LLM analyze the solving behaviors within them to adjust configurations accordingly?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 68Kp (1/2)

We thank the reviewer for recognizing the value of examining conventional-search budgets rather than relying only on improvement over solver defaults. We respectfully disagree with the conclusion that a new LLM tuning algorithm or a demonstration of LLM superiority is required for PACER to make a substantial contribution. The paper addresses a different fundamental question: how to attribute and interpret the performance of LLM-generated solver configurations when proposal mechanisms have different search opportunities. Its theory, controlled experiments, and component-level analyses make meaningful progress on that question.

W1. Novelty and contribution. The key insight is that artifact performance and proposer effectiveness are distinct quantities. Proposition 1 makes the dependence on proposal quality and budget explicit for adaptive searches, and PACER connects this insight to solver-call resources and held-out performance. The empirical results are substantive: 34.3% held-out PDI improvement for the historical reference on GISP-easy, improvement on 95/100 held-out instances, and a shift from 5/15 to 11/15 conventional-search endpoints exceeding that reference as the nested search budget grows (Table 2). The CA-easy component ablation further isolates the source of nearly all its PDI gain (Table 17). This combination of theory, budget-sensitive evaluation, and performance attribution is the contribution, rather than a new optimizer that would itself need this kind of evaluation. Modular reuse of established methods strengthens this contribution by making the evaluation accessible to a broad range of existing and future proposers, as clarified in the revised Introduction, C1, and Section 3.3 (blue).

W2. Cross-task evidence. We evaluate performance on three distinct binary MILP families, each with easy and medium variants. All six retained artifacts improve both PDI and runtime in the reported held-out point estimates (Table 3), demonstrating that the performance observations extend beyond a single benchmark. The controlled proposer comparison on GISP-easy provides a complementary, deeper analysis of search budgets. The revised Section 4.3 (blue) clarifies the generator regimes, and the existing Sections 4.2–4.3 explicitly separate cross-task artifact performance from controlled budget-dependent proposer comparisons. Extending the latter across all problem families is a valuable next stage, building on the reproducible evaluator established here.

W3. Baselines and HEBO. The study includes five methods—GP, TPE, Random, SMAC3, and RF-EI—rather than only the three identified in the review (Table 4). These cover density-based search, Gaussian-process search, random forests, and direct random sampling, all under the same categorical search domain and billed solver-call budgets. We respectfully disagree that HEBO is necessary to substantiate the central finding: among these independently implemented conventional-search methods, the number of endpoints surpassing the historical LLM reference increases from 5/15 to 11/15 as budget increases (Table 2). HEBO and further strong baselines would extend the comparative landscape and may change the detailed attainment curves. The existing five-method study already establishes the budget dependence that motivates PACER. The reusable ask/tell/fail interface in revised Section 3.3 (blue) is designed precisely to let future baselines such as HEBO enter the same evaluation without changing its scoring or accounting.

AuthorsOpenReview ↗

Response to Reviewer 68Kp (2/2)

Q1. Pretrained priors and model/prompt stability. The controlled GISP-easy reference changes two global CPLEX controls, governing branching-variable selection and backtracking. Both lie in the 31-axis categorical search space available to every conventional proposer (Sec. 4.1). This makes the reference directly comparable as a configuration, with no privileged parameter settings. Determining why the LLM selected those controls, however, requires model identities, prompt variants, sampling settings, and full feedback histories. The revised Appendix C (blue) distinguishes preserved evaluator outputs from historical prompt-visible feedback, whose exact contents were not retained. Disentangling pretrained knowledge from feedback-driven search is an important causal question and a natural next step once those records can be prospectively controlled. PACER supplies the common evaluator and cost accounting necessary for such an experiment.

Q2. Historical solver feedback. The retained records document the actual evaluation history: for GISP-easy, 1,164 sandbox CPLEX calls produced 12 distinct candidate configurations, followed by nine platform-validation rounds of 100 calls each (App. C). The archive preserves candidate configurations and round-level outcomes, while App. K reports historical PDI and runtime measurements. The exact information embedded in each historical prompt—including whether detailed per-instance traces or complete solver logs were shown to the LLM—is not recorded. This distinction between archived evaluator output and prompt-visible feedback is essential for valid causal attribution, and is now explicit in revised Appendix C (blue). The current study uses the verifiable artifact and execution records. A prospective feedback-ablation study would build directly on PACER to isolate how solver metrics, logs, and semantic information influence proposal quality.

We appreciate your recognition of PACER's methodological value and hope our responses further establish the significance of its theoretical and empirical contributions, and we respectfully ask you to reconsider your rating in light of the revised manuscript.

I appreciate the authors' responses and respect their opinion. I will maintain my rating.

Reviewer_KTJoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PACER, a protocol for distinguishing the held-out quality of an LLM-generated solver configuration from the effectiveness of the proposal mechanism that produced it. PACER fixes the evaluator, validation/test split, PDI-based score, and solver-call budget, then compares validation-selected configurations from conventional search methods against a historical LLM-produced CPLEX configuration.

Critical strengths

The paper addresses an important evaluation issue in LLM-based optimization by distinguishing the quality of generated solver configurations from the effectiveness of the underlying search methods. The budget-controlled comparisons with conventional search methods provide a useful perspective on assessing LLM-generated optimization artifacts.

Critical weaknesses
  1. Limited methodological novelty. PACER does not propose a new algorithm for solver parameter optimization. Instead, it mainly integrates existing techniques, including automated algorithm configuration, PDI-based evaluation, budget control, and held-out testing, into an evaluation protocol for LLM-generated solver configurations.
  2. Lack of direct LLM proposer evaluation. The experiments do not evaluate an LLM proposer under PACER. Instead, they compare a fixed historical LLM-generated configuration against five conventional search methods, limiting conclusions about their relative search efficiency.
  3. Limited generalizability and analysis of LLM capabilities. The controlled proposer comparison is restricted to GISP-easy and CPLEX. Moreover, potential advantages of LLM-based search, such as semantic guidance and cross-problem generalization, are not investigated.
Questions for authors
  1. Could the authors include an LLM proposer in PACER and conduct additional experiments comparing its search performance with conventional methods under matched evaluation budgets?
  2. Could the authors extend the controlled proposer comparison to more MILP benchmarks and using other MILP solvers, such as Gurobi or SCIP, to assess the generalizability of PACER and its experimental conclusions?
  3. To what extent does PACER capture the potential advantages of LLM-based search, such as semantic guidance, knowledge transfer, and cross-problem generalization? A more detailed discussion of these capabilities and the limitations of the current evaluation framework would be valuable.
2 comments
AuthorsOpenReview ↗

Response to Reviewer KTJo

We thank the reviewer for recognizing the importance of distinguishing selected-artifact quality from proposer effectiveness and the usefulness of budget-controlled comparisons.

W1. Methodological and theoretical contribution. PACER makes three mutually reinforcing contributions: a finite-budget guarantee connecting the proposer's target-hit probability to its evaluation opportunities (Prop. 1); a reproducible, solver-aware protocol for measuring artifact attainability using PDI, held-out testing, and solver-call accounting (Sec. 3); and substantial empirical evidence that budget and component attribution change the interpretation of solver-configuration gains (Tables 2–3, 17–18). In particular, a historical LLM-generated configuration achieves 34.3% held-out PDI improvement and improves 95/100 test instances, while alternative-search attainability changes sharply with evaluation budget. The paper's scientific contribution is to establish what such performance shows and how to evaluate its generating mechanism under controlled search opportunities. This synthesis is now more explicit in the revised Abstract, Introduction (C1 and C3), Sections 3.2–3.3, and Conclusion (blue). The fact that PACER can encapsulate existing optimizers is an engineering strength: it makes rigorous comparison more reproducible and easier to extend.

W2 / Q1. Direct LLM proposer evaluation. The historical LLM workflow already supplies a strong, concrete optimization artifact. Re-evaluating it independently is scientifically valuable: the reference achieves a test PDI ratio of 0.6572, and conventional-search methods are then evaluated against exactly this target using the same solver and scoring procedure. At 25 trials, 5/15 endpoints exceed the reference. At 60 trials, 11/15 do so (Table 2). This establishes an empirical budget-to-attainability curve for the selected LLM artifact, rather than simply comparing it with default CPLEX settings.

A fully prospective, matched end-to-end proposer comparison is a harder problem: it requires controlling the LLM's access to prior information and feedback while measuring model calls, generation time, solver evaluations, and possible human intervention. Those historical provenance records are incomplete (App. C). PACER is a rigorous first step toward that comparison: it fixes the otherwise confounded evaluation side and measures the performance attainable by five alternative search methods under explicit budgets. The paper's present claims concern this observable and reproducible comparison, which is already informative. Revised Section 4.4 and the Conclusion (blue) identify the additional records needed for a prospective end-to-end study and position PACER as its practical first step.

W3 / Q2. Additional benchmarks and solvers. The empirical study is broader than the controlled GISP-easy experiment alone. Table 3 re-evaluates six retained configurations across MIS, CA, and GISP, with improvements in both PDI and runtime in all six reported point estimates. The controlled GISP-easy study, in turn, holds the task, solver, and evaluator fixed to identify budget effects across five search methods and three seeds. The two studies answer complementary questions: whether retained artifacts perform well across problem families, and how alternative search opportunity changes the standing of a particular strong artifact. Cross-solver tests with Gurobi or SCIP and full search-budget curves for every problem family are worthwhile extensions, but would constitute a substantially larger experimental program. PACER establishes a common evaluation foundation for such studies. Revised Section 4.3 (blue) clarifies the easy/medium generator regimes and reinforces this two-level experimental design.

Q3. Semantic guidance and transfer. PACER's theoretical framework is proposer-agnostic. Semantic priors, problem descriptions, transfer, and solver feedback can all affect a proposer's conditional probability of reaching a target-quality configuration. The protocol can therefore evaluate such mechanisms on the same footing once their information access and resource use are specified. The current study measures the quality of retained LLM-generated configurations and their attainability by conventional search. Historical model and prompt records are not sufficiently complete to isolate the causal contribution of each LLM capability (App. C). The revised Sections 3.2–3.3 and 4.4, together with Appendix C (blue) make this separation between adaptable proposer logic, common evaluation, and historical feedback provenance explicit.

We hope our clarifications of the theoretical insights, empirical findings, and broader applicability of PACER address your concerns, and we would sincerely appreciate your reconsideration of the rating.

Thank you for the response. I understand that the authors position PACER as a unified evaluation foundation rather than an exhaustive study across multiple problems and solvers. However, I still believe that not directly incorporating LLMs as proposers, but instead relying on artifacts from a historical LLM workflow, substantially limits PACER's applicability, particularly for evaluating different and emerging frontier models. Meanwhile, the lack of broader cross-problem and cross-solver experiments leaves concerns about its generalizability. As PACER mainly integrates existing techniques, I remain unconvinced that its novelty and contributions meet the publication bar for ICLR. Therefore, I maintain my original score.

Reviewer_szv3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

For recurring MILP workloads, solver hyperparameters must be chosen. This can be done with conventional hyperparameter search methods (e.g., Bayesian optimization or random search) or by asking an LLM to propose the best configuration. This paper provides a protocol for comparing LLM-generated hyperparameters to those found by conventional search. They first run the default parameters and record their PDI. They then run five search methods across different budgets, compute the held-out PDI of the best configuration for each method and seed, and evaluate how well each method performs. On GISP-easy, the LLM-generated configuration improves the primal-dual integral by 34.3% over the default CPLEX parameters.

Critical strengths

The appendix provides detailed documentation of the search space, method implementations, and seed-level results, making it easy for others to utilize the results of this study. The LLM-generated configuration is also remeasured under the same evaluator as the baselines.

Critical weaknesses

Overall, I found that this work has limited novelty:

  1. Not specific to MILP: Apart from the use of PDI as a metric and the CPLEX parameter space, nothing in the protocol is specific to combinatorial optimization. It is a general comparison of hyperparameter search methods against an LLM-generated configuration and would transfer directly to, e.g., tuning an ML model. Q1: Can the authors clarify what specifically about PACER is uniquely suited to combinatorial optimization?
  2. Not specific to LLMs: The protocol applies equally to any search method and does not engage with properties of LLM-based proposers and it is unclear why a dedicated protocol for LLM-generated configurations is needed. Q2: Can the authors clarify what specifically about PACER makes it uniquely suited to evaluating LLM proposers?
  3. Standard Evaluation Setup: From my understanding, PACER amounts to selecting a single benchmark, running hyperparameter tuning with several methods across seeds, choosing the best configuration on validation, and evaluating it on a held-out set. The paper emphasizes that it fixes the evaluator, but this corresponds to a fixed experimental setup on a single benchmark, and holding the setup constant across compared methods is standard practice rather than a contribution in itself.
  4. Prop. 1: It is unclear why Proposition 1 is there as the relevant probability pπ(τ)p_\pi(\tau) is never estimated for any method. From my understanding, this proposition essentially states that more trials increase the probability of reaching a target score, which is intuitive and is not necessary to state formally.
Questions for authors

Q1-Q2 - See above. Answers to both of these would significantly help frame the novelty of this work better.

Q3: What are the problem sizes for each workload (number of decision variables and constraints, and the generator parameters, e.g., graph size, number of bids/items), and how do the "easy" and "medium" regimes differ?

8 comments
AuthorsOpenReview ↗

Response to Reviewer szv3 (1/2)

We thank the reviewer for recognizing the detailed experimental documentation and the independent remeasurement of the LLM-generated reference. We respectfully disagree with the assessment that PACER offers limited novelty simply because it builds on established evaluation practices. The substantive contribution is the theoretical and empirical treatment of a specific attribution problem: a strong final configuration does not, by itself, reveal how effectively its generating process searches under finite resources. Moreover, building on existing methods is a strength rather than a weakness of PACER. Its modular, proposer-agnostic architecture encapsulates established search methods behind a common evaluator, making them readily reusable and new methods straightforward to integrate. This practical extensibility is a deliberate engineering advantage, now emphasized in the revised Introduction, C1, and Section 3.3 (blue).

W1–W3 / Q1–Q2. Contribution and relevance to MILP and LLM proposers. PACER advances this problem in three ways.

First, Proposition 1 formalizes the two distinct factors determining target-quality attainment: the conditional probability assigned to satisfactory configurations and the number of opportunities to sample them. It applies to adaptive, history-dependent proposers. This is particularly relevant to LLM-assisted optimization, where a final artifact may be visible even when the full proposal history and its cost are only partially recorded. PACER gives an operational way to distinguish that artifact's quality from the quality that alternative search methods can attain at specified budgets.

Second, MILP solver configuration requires solver-aware measurement. A CPLEX execution produces time-varying primal incumbents and dual proof bounds, and many instances terminate at a fixed time limit. PACER uses the primal-dual integral (PDI) to measure these trajectories consistently, rather than equating optimization progress with a final objective, node count, or runtime alone. It also separates global solver settings from instance-dependent inputs. This distinction matters empirically: removing the learned MIP start from the CA-easy artifact changes its PDI ratio only from 0.787882 to 0.788030 (Table 17). Conversely, earlier MIS-easy and CA-easy artifacts reduce node counts to 0.0985 and 0.3478 of default, yet worsen PDI and runtime (Table 18). These findings expose nontrivial attribution and measurement issues specific to solver behavior.

Third, the controlled experiments reveal a budget effect that cannot be inferred from default-relative performance. The remeasured LLM configuration achieves 34.3% held-out PDI improvement, yet conventional-search reference exceedance increases from 5/15 to 11/15 between 25 and 60 trials along the same search trajectories (Table 2). Thus, PACER delivers a concrete answer to how the interpretation of an LLM-generated configuration changes with the search opportunities available to alternatives. The fact that its underlying evaluation principles can extend to other optimization domains broadens the value of this contribution. The revised Abstract, Introduction (C1 and C3), and Sections 3.2–3.3 (blue) now foreground the attribution question, the adaptive finite-budget guarantee, and the reusable proposer interface. The revised Section 4.2 (blue) connects these messages directly to the observed 5/15-to-11/15 shift.

AuthorsOpenReview ↗

Response to Reviewer szv3 (2/2)

W4. The role of Proposition 1. Proposition 1 provides a finite-budget success guarantee for adaptive proposal processes. For the target set AτA_\tau and a conditional success-probability lower bound pπ(τ)p_\pi(\tau), it yields

\qquad \mathbb E[Q_{\pi,\tau}]\le \frac{1}{p_\pi(\tau)}.$$ Because this guarantee conditions on unsuccessful search histories, it covers feedback-dependent proposals **without an independence assumption**. Equation (9), $S_\pi(k)=k|I_{\mathrm{val}}|$, then translates proposal opportunities into billed solver calls. **The theoretical insight is the explicit connection among target-quality attainability, adaptive proposal behavior, and evaluation resources**, which motivates the budget-controlled design and interpretation of Table 2. We now state this interpretation explicitly **immediately after Proposition 1 in revised Section 3.2 (blue)**. The result is a general analytical guide to the interaction between proposal probability and evaluation opportunities, while the empirical comparison measures actual outcomes without assuming a value for $p_\pi(\tau)$. **Q3. Workload sizes and easy/medium regimes.** Section 4.1 and App. K provide the formulations and the **800/100/100** data splits. The model dimensions follow directly from those formulations: GISP contains $|V|+|E_1|$ binary variables and $|E_0|+|E_1|$ structural constraints. CA contains one binary variable per bid and item-capacity constraints, including dummy items. MIS contains one binary variable per vertex, with the realized constraint count depending on the clique reformulation (App. K). The easy and medium regimes use the same mathematical formulation within each family but different instance-generator settings, as now emphasized in **revised Section 4.3 (blue)** and detailed in **Appendix K**. Realized instance-size statistics and numerical generator parameters would further characterize the regimes. These require the underlying benchmark instances or generator metadata and are not inferred from the formulation alone. We hope our responses and manuscript revisions better demonstrate PACER's `theoretical contributions, methodological novelty, and practical significance`, and we respectfully invite you to reconsider your assessment and rating in light of these clarifications.
Reviewer_szv3OpenReview ↗

Response

Thank you for the detailed response and clarifications. Here are my comments: Proposition 1: I appreciate the clarification that Proposition 1 both here and in your below comment.

Nevertheless, my primary concerns regarding the novelty of PACER remain largely unaddressed.

Artifact quality vs. proposer effectiveness: I agree that this distinction is important, but I remain unconvinced that PACER provides generalizable or actionable insights into proposer effectiveness. PACER evaluates workloads independently, without characterizing when or why particular proposers outperform others. Consequently, its findings offer limited guidance to practitioners choosing a configuration method for a new workload.

Specificity to MILP solvers: I appreciate the clarification regarding solver-aware evaluation, but this does not change my assessment. PDI is a well-established metric for evaluating MILP solver performance (e.g., [1]), and using it for hyperparameter tuning is not a novel methodological contribution. Beyond PDI and the CPLEX-specific search space, I do not see how PACER differs methodologically from standard hyperparameter optimization benchmarking.

Specificity to LLMs: I still remain unsure about what makes PACER specifically suited to evaluating LLM-based proposers. The protocol does not incorporate any LLM-specific evaluation mechanisms or methodological modifications and treats LLMs almost the same as any other HPO method. Additionally, the prompts, the models used, and other relevant information regarding the LLM setup is not provided. This is important as the baseline number could very well have come from a relatively weaker model and these results could change given a better/more frontier model. Furthermore, while I understand the motivation for evaluating an LLM-generated configuration, it remains unclear what PACER offers over established hyperparameter optimization evaluation protocols. Regarding the observed increase in reference exceedance from 5/15 to 11/15; I agree that this illustrates the importance of accounting for search budgets. However, this is an expected consequence of providing hyperparameter search methods with additional evaluation opportunities, and is the type of comparison already captured by standard budget-controlled hyperparameter optimization benchmarking protocols (e.g., [2]). This is also only evaluating one LLM generated configuration from an unknown model, and the results might change if the LLM used is switched.

I appreciate the authors' efforts to clarify PACER's motivation and practical value. However, my concerns regarding methodological novelty remain largely unaddressed. PACER still appears to apply established hyperparameter optimization evaluation practices to an LLM-generated configuration, without introducing a substantive contribution specific to MILP solvers or LLM-based proposers. Therefore, my assessment remains unchanged.

[1] Gasse, Maxime, et al. "The Machine Learning for Combinatorial Optimization Competition (ML4CO): Results and Insights." NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022. [2] Eggensperger, Katharina, Marius Lindauer, and Frank Hutter. "Pitfalls and best practices in algorithm configuration." Journal of Artificial Intelligence Research 64 (2019): 861-893.

Reviewer_1iGpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PACER to compare LLM-generated solver config with traditional optimization methods, with a focus on evaluating proposer quality under controlled search budgets.

Critical strengths

I like the motivation of this paper. The diff between artifact and proposer quality is important, and I think comparing LLM-based search with traditional optimization methods is a good direction.

Critical weaknesses

W1. Cost Reporting and Budget Fairness

The paper mainly uses solver evaluations to measure search budgets, but I couldn't find details about the LLM model/version, token usage, or total wall-clock time. It would be useful to compare LLM and BO under both equal evaluator calls and equal wall-clock time. More importantly, since the LLM configuration comes from a historical experiment, I'm not sure the current comparison fully supports conclusions about proposer effectiveness under matched budgets.

W2. Generalizability of PACER

Since PACER is presented as an evaluation protocol, I would have liked to see broader validation. The main controlled comparison is mostly limited to GISP-easy, so it's unclear how well the findings generalize to other optimization tasks.

W3. Robustness of the Evaluation Protocol

I'm curious whether the conclusions would remain stable with different validation sets or budget settings. Although the paper reports multiple seeds and validation-test correlations, some additional sensitivity analysis would make the results more convincing.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1iGp

We appreciate the reviewer's recognition that distinguishing artifact quality from proposer quality is important. We address the concerns below.

W1. Cost reporting and budget fairness. PACER uses billed solver calls as its primary controlled resource, because every candidate configuration must be assessed through computationally expensive CPLEX executions. On GISP-easy, one validation trial evaluates 100 instances. Hence k=8,25,60k=8,25,60 corresponds to 800, 2,500, and 6,000 solver calls per search, respectively (Sec. 3.3). All five conventional proposers use the same evaluator, score, validation set, and call-accounting rule. Table 9 additionally reports 1.8–4.0 hours of elapsed time for the 60-trial searches. This gives both an exact evaluator-budget comparison and observed search runtimes.

The LLM-generated configuration serves as a fixed, independently remeasured reference, while the conventional proposers are compared under controlled search budgets. Section 4.4 further places the historical totals of 2,064 and 2,109 solver calls alongside PACER's budget scale (approximately 21 full-validation trials). The important challenge is that the retained historical workflow does not include the model, prompt, token, and full runtime records needed to reconstruct its end-to-end generation cost. PACER therefore establishes the verifiable common ground first: artifact quality under one evaluator and the budget needed for alternative searches to attain it. The revised Section 4.4 (blue) states why full end-to-end attribution requires additional generation-side records, while Section 3.3 already specifies the exact billed solver-call budgets. Standardizing the evaluator is the necessary first step toward a fully matched comparison, not a substitute for one.

W2. Generalizability. Our evidence spans two complementary evaluation levels. First, GISP-easy provides a controlled five-method, three-seed study across multiple budgets, revealing how the relative standing of a strong LLM-generated reference changes with search opportunity (Table 2). Second, six retained-artifact replays cover easy and medium variants of MIS, CA, and GISP, and all six improve PDI and runtime (Table 3). Their PDI improvements range from 6.1% to 34.4%. These results demonstrate meaningful artifact performance beyond a single formulation, while the controlled study provides the more detailed proposer- and budget-level analysis. We clarify the complementary roles of the two studies and the easy/medium generator regimes in revised Section 4.3 (blue). Expanding matched proposer studies across additional solvers and problem families is a natural next step requiring substantially more solver evaluations.

W3. Robustness. The submission already includes several independent checks. The k=25k=25 and k=60k=60 results use nested prefixes of the same search trajectories (Table 2), isolating the role of additional evaluations. Among the four methods initialized at the default configuration, reference exceedance still rises from 5/12 to 8/12. Geometric-mean and aggregate PDI scoring give the same reference-exceedance classification for all 45 endpoints (App. H, Table 11). Validation and held-out rankings also agree closely, with Spearman correlations of 0.89, 0.90, and 0.96 across the three budgets (Fig. 2). Together, these analyses support the budget effect, scoring stability, and validation-to-test consistency observed in the study. These checks are documented in Sections 4.2–4.4, Figure 2, and Appendix H. The revised Section 4.2 (blue) further emphasizes why the nested 25/60-trial result changes the interpretation of the unchanged LLM reference.

We hope our clarifications on budget fairness, generalizability, and robustness, together with the revised manuscript, address your concerns, and we would greatly appreciate your reconsideration of the rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Contributions and Manuscript Revisions

We thank the reviewers for their careful evaluation and for recognizing the importance of assessing LLM-generated solver configurations beyond comparisons with default settings. PACER establishes a fundamental distinction between the quality of a selected optimization artifact and the effectiveness of the process that generates it. We develop this perspective through a finite-budget theoretical result, a controlled evaluation framework, and substantial empirical evidence. To further clarify these contributions and address the reviewers' concerns, we have revised the manuscript, with all changes highlighted in blue.

Theoretical insight. Proposition 1 characterizes the interaction between proposal quality and search opportunity. If an adaptive proposer has conditional probability at least pπ(τ)p_\pi(\tau) of reaching a target-quality set after any unsuccessful search history, then its probability of success within qq proposals is at least 1−(1−pπ(τ))q1-(1-p_\pi(\tau))^q, with expected hitting time at most 1/pπ(τ)1/p_\pi(\tau). The result does not require independent proposals. Coupled with PACER's solver-call accounting, this connects proposal behavior, evaluation budget, and the probability of attaining a desired solver quality.

Evaluation and performance. PACER implements this distinction with a frozen, trajectory-aware PDI evaluator, common validation selection, held-out testing, and explicitly billed CPLEX calls. The historical LLM-generated GISP-easy configuration achieves 34.3% held-out PDI improvement, outperforming the default on 95/100 test instances (Sec. 4.2). Six retained configurations spanning MIS, CA, and GISP improve both PDI and runtime in every reported point estimate (Table 3). Meanwhile, conventional-search endpoints exceeding the historical reference increase from 5/15 after 25 trials to 11/15 after 60 trials along the same trajectories (Table 2). The CA-easy ablation identifies fixed solver settings as the source of nearly all measured PDI gain (Table 17), while the node-reduction analysis demonstrates why solution quality and runtime require direct evaluation (Table 18).

These results establish both strong artifact performance and a theory-guided way to interpret it under finite search budgets. A fully matched, end-to-end comparison of live LLM and conventional proposers additionally requires synchronized model, prompt, feedback, and cost records. PACER takes a rigorous first step toward that broader problem by establishing a reproducible evaluator, controlled alternative-search curves, and fine-grained artifact attribution. The revised manuscript now makes this theoretical–empirical narrative explicit in the Abstract, Introduction (C1–C3), Sections 3.2–3.3 and 4.2–4.4, and Conclusion. The new passages are marked in blue. An important design advantage is modularity: PACER encapsulates established and emerging proposers behind a common evaluator, enabling practical, extensible comparisons without redesigning their internal algorithms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.