- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ConfigAgent, an LLM-based framework for algorithm configuration in combinatorial optimization. The framework uses an agentic propose-execute-reflect loop to explore solver parameter configurations and accumulate numerical trial records and textual insights in a shared knowledge base. During deployment, previously collected experience is reused through taxonomy-based retrieval. Zero-shot deployment selects configurations through deterministic ranking of historical trials, while few-shot deployment allows additional LLM-guided adaptation. The authors evaluate ConfigAgent across five combinatorial optimization families and multiple exact, heuristic, and evolutionary solvers. Experiments compare the framework against conventional algorithm configuration methods, including SMAC3 and irace, and investigate transfer across objectives, instance scales, and problem variants. The reported results suggest competitive configuration quality with reduced target-side configuration time, supported by component ablations and additional transfer experiments.
Critical strengths
1. A practically relevant problem and a reusable configuration framework.
The paper addresses the practical challenge of transferring algorithm configuration experience across related optimization contexts. Combining historical configuration records with contextual information is a useful direction for reducing repeated tuning costs, particularly when solver evaluations are expensive.
2. Broad experimental coverage and useful efficiency analysis.
The evaluation covers multiple combinatorial optimization families and several solver classes, including exact solvers, heuristic search methods, and multi-objective evolutionary algorithms. The paper also investigates transfer across instance scales, objectives, and structural variants, and provides component ablations and some transfer-aware numerical baselines. These experiments offer useful evidence of the potential benefits of configuration reuse.
Critical weaknesses
1. The contribution of LLM-based reasoning to the reported transfer performance is insufficiently isolated.
The central claim is that ConfigAgent enables knowledge-driven algorithm configuration through accumulated experience and LLM-generated insights. However, the zero-shot deployment procedure does not invoke the LLM or use textual insights for final configuration selection. Instead, it retrieves previously evaluated configurations and ranks them using deterministic statistics and literature-based parameter constraints. Consequently, the strong zero-shot results do not directly establish that the proposed reasoning and reflection mechanisms are responsible for the observed transfer performance.
Although the paper includes component ablations and transfer-aware numerical baselines, the latter are primarily evaluated on a restricted set of TSP variants sharing the NSGA-II parameter space. A broader comparison against numerical experience reuse under matched source information and deployment budgets is needed to substantiate the claimed contribution of the agentic framework.
2. Important inconsistencies in the reported methodology and experimental evidence.
There are inconsistencies between the stated training protocol and the provided execution example. Section 3.2 and Algorithm 1 explicitly disable cross-context retrieval during training. However, the training example in Appendix A.10 (trial 20/100) includes cross-problem evidence retrieved from a TSP context for an FJSP configuration task. Moreover, this source appears to match only one of the three structural dimensions, whereas Eq. (6) requires at least two matching dimensions.
There are also discrepancies between the main experimental results and their corresponding statistical analysis. For example, Table 2 reports a hypervolume of 2.583 x for ConfigAgent on TSP with MOEA/D, whereas Table 16 reports 2.4910 x for the nominally corresponding two-objective setting. The relative ranking against irace is also reversed.
These inconsistencies may have explanations involving different evaluation runs or protocols, but they are not adequately clarified. Since they concern the core transfer mechanism and the empirical evidence supporting the main claims, they reduce confidence in the reported conclusions.
Questions for authors
Q1. Contribution of LLM-generated knowledge.
Since zero-shot deployment selects configurations using numerical trial records without invoking the LLM or consulting textual insights, what evidence demonstrates that the LLM-generated knowledge contributes independently to the reported transfer performance? Could the authors provide a comparison against a numerical-only knowledge base with the same source records, parameter priors, and deployment budget?
Q2. Consistency of the training protocol.
Section 3.2 states that cross-context retrieval is disabled during training, yet the trial-20 example in Appendix A.10 includes evidence from another problem context. Could the authors explain how this evidence became available? Furthermore, how does this retrieval satisfy the structural overlap criterion in Eq. (6)?
Q3. Discrepancies between main results and statistical tests.
Could the authors explain why the results in Table 16 differ from the corresponding entries in Table 2? For example, the reported ConfigAgent hypervolume on TSP with MOEA/D changes from 2.583 x 10^8 to 2.4910 x 10^8, and the ranking relative to irace is reversed. Were different test instances, random seeds, or evaluation protocols used? Which results support the main statistical conclusions?
Q4. Comparability of cross-context configuration records.
The zero-shot selection procedure aggregates retrieved trial records and ranks configuration buckets by their measured quality. How are performance values normalized when source contexts differ in problem family, instance scale, or objective count? How does the implementation ensure compatibility between source configurations and the target solver's parameter schema?
Q5. Fairness of the efficiency comparison.
The reported deployment speedups largely compare frozen-KB zero-shot selection against classical baselines that retune from scratch. Could the authors clarify the efficiency gains under matched access to source experience, including the cost of constructing the knowledge base? It would also be useful to distinguish improvements in configuration quality from savings obtained simply by avoiding target-side optimization.
2 comments
I appreciate the authors' response. However, I still have some reservations about the novelty of the proposed approach and the experimental evidence supporting its contributions. I will maintain my original rating.
W1/Q1 Contribution of reasoning and textual guidance
Thank you for asking us to isolate these contributions. Explicit reasoning and textual guidance improve adaptation in the reported five-objective Routing comparison. Numerical records support reuse, while the ablations test the additional contribution of textual insights and reflection. Matched ablations keep numerical source records, priors and the same target adaptation budget fixed while removing source insights, target reflection, both, or explicit proposal reasoning. On this target, full adaptation improves over proposals without explicit reasoning by 3.29%, 5.53% and 2.82% for ConfigAgent (7B), ConfigAgent (14B) and ConfigAgent (32B), respectively. The controls span four MOEA/D targets, extending the paper’s NSGA-II transfer comparisons.
On five-objective Routing, retaining source insights improves ConfigAgent (32B)'s HV by 0.69%, with a 95% interval of [0.10%, 1.43%]. Compared with using only numerical configuration results, the full method improves HV by 0.66% for ConfigAgent (14B) and 0.37% for ConfigAgent (32B). Beyond these quality gains, textual explanations make tuning decisions inspectable, and the accumulated experience supports reuse across target settings without repeating source construction. These are central to our claim of interpretable, generalizable and efficient configuration.
W2/Q2/Q3 Source training example and quality comparisons
We would like to clarify that source training and cross-context deployment are separate stages. As specified in Section 3.2 and Algorithm 1, source training combines the current task's description, solver background knowledge and local trial feedback, with cross-context retrieval disabled. Eq. (6) governs retrieval at deployment. The cross-context illustration in Appendix A.10 should therefore be read separately from the independent training protocol. In the supplementary training runs, the recorded prompts follow this separation. The LLM builds experience independently on source tasks and subsequently uses retrieved experience to guide few-shot adaptation on related targets.
For the quality comparison, Table 16 tests a fixed selected configuration, while independent tuning repeats measure variability across runs. Its TSP–MOEA/D configuration pc=0.9, pm=0.01, T=35, Tchebycheff gives 2.4910 ×10⁸ over 100 per-instance observations. We repeated tuning five times using 500 training solver calls per run, 100 held-out instances × ten common solver seeds, and 2,500 fitness evaluations per solve. ConfigAgent (7B) and ConfigAgent (32B) achieve 2.5637 ±0.0485 and 2.5547 ±0.0479, within 0.75% and 1.10% of the main-table 2.583 ×10⁸, while SMAC achieves 2.5567 ±0.0651. These repeats remain close to the original result and support comparable quality. Configurations and per-run observations are in the anonymous repository.
Q4 Cross context scoring and parameter compatibility
We would like to clarify the distinction between retrieval and scoring. Eq. (6) determines source eligibility, while Eq. (7) ranks configuration buckets using their recorded quality, support and variability. Eq. (7) does not itself define normalization across contexts with different metric scales. In the new controlled experiment, configurations are ranked within each designated source context and then evaluated under the target's objectives and reference point. Thus, target comparisons use the same metric and evaluation setting. Transferred configurations must also satisfy the target solver's parameter schema, including the complete crossover, mutation, neighborhood and decomposition settings for MOEA/D.
Q5 Efficiency with matched experience
We would like to clarify that the efficiency gain comes from avoiding repeated target-side configuration. Table 5's 80× compares 100 s of deployment with 8,000 s of fresh tuning, while Appendix B separately includes source construction in the lifecycle cost. With direct reuse available to all methods, Table 37 reports ConfigAgent HV above the better classical incumbent by 2.36%, 0.79% and 0.22% on ATSP, mixed-scale TSP and clustered TSP, with zero target tuning trials for each method.
The new ConfigAgent/Warm-SMAC comparison gives both methods the same frozen source records and the same tuning budget to assess adaptation quality. ConfigAgent (32B) records 357–390 s of solver-plus-LLM adaptation time across four target means. Source experience is reused across these targets, consistent with our claim of competitive quality and reduced repeated configuration effort.