acceptodds
Under review as a conference paper at ICLR 2027

ConfigAgent: Agentic Algorithm Configuration for Combinatorial Optimization

Abstract

Algorithm configuration (AC) is essential in combinatorial optimization (CO), yet existing methods such as SMAC3 and irace remain context-specific, and need to repeatedly tune for each solver-problem pair from scratch, limiting transferability and efficiency. We propose ConfigAgent, an agentic framework that reformulates AC as knowledge-driven reasoning over accumulated configuration experience across multiple contexts. It maintains a structured knowledge base containing historical trials and distilled insights across different solvers, problem families, objectives, and instance scales. For a new configuration context at deployment, ConfigAgent retrieves relevant prior experience from the knowledge base, reasons over contextual similarities, and generates evidence-grounded configurations. We evaluate ConfigAgent on five CO families with a wide range of algorithms including exact solvers, heuristic search engines, and multi-objective evolutionary algorithms. Results show that ConfigAgent consistently matches or outperforms strong AC baselines while requiring substantially fewer configuration trials. Moreover, the learned knowledge base enables efficient zero-shot or few-shot transfer to unseen scales, objectives, and structural variants, significantly reducing deployment cost by up to two orders of magnitude.

Primary area:
optimization
Keywords:
Combinatorial Optimization, Algorithm Configuration, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gR23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ConfigAgent, an LLM-based framework for algorithm configuration in combinatorial optimization. The framework uses an agentic propose-execute-reflect loop to explore solver parameter configurations and accumulate numerical trial records and textual insights in a shared knowledge base. During deployment, previously collected experience is reused through taxonomy-based retrieval. Zero-shot deployment selects configurations through deterministic ranking of historical trials, while few-shot deployment allows additional LLM-guided adaptation. The authors evaluate ConfigAgent across five combinatorial optimization families and multiple exact, heuristic, and evolutionary solvers. Experiments compare the framework against conventional algorithm configuration methods, including SMAC3 and irace, and investigate transfer across objectives, instance scales, and problem variants. The reported results suggest competitive configuration quality with reduced target-side configuration time, supported by component ablations and additional transfer experiments.

Critical strengths

1. A practically relevant problem and a reusable configuration framework.

The paper addresses the practical challenge of transferring algorithm configuration experience across related optimization contexts. Combining historical configuration records with contextual information is a useful direction for reducing repeated tuning costs, particularly when solver evaluations are expensive.

2. Broad experimental coverage and useful efficiency analysis.

The evaluation covers multiple combinatorial optimization families and several solver classes, including exact solvers, heuristic search methods, and multi-objective evolutionary algorithms. The paper also investigates transfer across instance scales, objectives, and structural variants, and provides component ablations and some transfer-aware numerical baselines. These experiments offer useful evidence of the potential benefits of configuration reuse.

Critical weaknesses

1. The contribution of LLM-based reasoning to the reported transfer performance is insufficiently isolated.

The central claim is that ConfigAgent enables knowledge-driven algorithm configuration through accumulated experience and LLM-generated insights. However, the zero-shot deployment procedure does not invoke the LLM or use textual insights for final configuration selection. Instead, it retrieves previously evaluated configurations and ranks them using deterministic statistics and literature-based parameter constraints. Consequently, the strong zero-shot results do not directly establish that the proposed reasoning and reflection mechanisms are responsible for the observed transfer performance.

Although the paper includes component ablations and transfer-aware numerical baselines, the latter are primarily evaluated on a restricted set of TSP variants sharing the NSGA-II parameter space. A broader comparison against numerical experience reuse under matched source information and deployment budgets is needed to substantiate the claimed contribution of the agentic framework.

2. Important inconsistencies in the reported methodology and experimental evidence.

There are inconsistencies between the stated training protocol and the provided execution example. Section 3.2 and Algorithm 1 explicitly disable cross-context retrieval during training. However, the training example in Appendix A.10 (trial 20/100) includes cross-problem evidence retrieved from a TSP context for an FJSP configuration task. Moreover, this source appears to match only one of the three structural dimensions, whereas Eq. (6) requires at least two matching dimensions.

There are also discrepancies between the main experimental results and their corresponding statistical analysis. For example, Table 2 reports a hypervolume of 2.583 x 10810^8 for ConfigAgent on TSP with MOEA/D, whereas Table 16 reports 2.4910 x 10810^8 for the nominally corresponding two-objective setting. The relative ranking against irace is also reversed.

These inconsistencies may have explanations involving different evaluation runs or protocols, but they are not adequately clarified. Since they concern the core transfer mechanism and the empirical evidence supporting the main claims, they reduce confidence in the reported conclusions.

Questions for authors

Q1. Contribution of LLM-generated knowledge.

Since zero-shot deployment selects configurations using numerical trial records without invoking the LLM or consulting textual insights, what evidence demonstrates that the LLM-generated knowledge contributes independently to the reported transfer performance? Could the authors provide a comparison against a numerical-only knowledge base with the same source records, parameter priors, and deployment budget?

Q2. Consistency of the training protocol.

Section 3.2 states that cross-context retrieval is disabled during training, yet the trial-20 example in Appendix A.10 includes evidence from another problem context. Could the authors explain how this evidence became available? Furthermore, how does this retrieval satisfy the structural overlap criterion in Eq. (6)?

Q3. Discrepancies between main results and statistical tests.

Could the authors explain why the results in Table 16 differ from the corresponding entries in Table 2? For example, the reported ConfigAgent hypervolume on TSP with MOEA/D changes from 2.583 x 10^8 to 2.4910 x 10^8, and the ranking relative to irace is reversed. Were different test instances, random seeds, or evaluation protocols used? Which results support the main statistical conclusions?

Q4. Comparability of cross-context configuration records.

The zero-shot selection procedure aggregates retrieved trial records and ranks configuration buckets by their measured quality. How are performance values normalized when source contexts differ in problem family, instance scale, or objective count? How does the implementation ensure compatibility between source configurations and the target solver's parameter schema?

Q5. Fairness of the efficiency comparison.

The reported deployment speedups largely compare frozen-KB zero-shot selection against classical baselines that retune from scratch. Could the authors clarify the efficiency gains under matched access to source experience, including the cost of constructing the knowledge base? It would also be useful to distinguish improvements in configuration quality from savings obtained simply by avoiding target-side optimization.

2 comments

W1/Q1 Contribution of reasoning and textual guidance

Thank you for asking us to isolate these contributions. Explicit reasoning and textual guidance improve adaptation in the reported five-objective Routing comparison. Numerical records support reuse, while the ablations test the additional contribution of textual insights and reflection. Matched ablations keep numerical source records, priors and the same target adaptation budget fixed while removing source insights, target reflection, both, or explicit proposal reasoning. On this target, full adaptation improves over proposals without explicit reasoning by 3.29%, 5.53% and 2.82% for ConfigAgent (7B), ConfigAgent (14B) and ConfigAgent (32B), respectively. The controls span four MOEA/D targets, extending the paper’s NSGA-II transfer comparisons.

On five-objective Routing, retaining source insights improves ConfigAgent (32B)'s HV by 0.69%, with a 95% interval of [0.10%, 1.43%]. Compared with using only numerical configuration results, the full method improves HV by 0.66% for ConfigAgent (14B) and 0.37% for ConfigAgent (32B). Beyond these quality gains, textual explanations make tuning decisions inspectable, and the accumulated experience supports reuse across target settings without repeating source construction. These are central to our claim of interpretable, generalizable and efficient configuration.

W2/Q2/Q3 Source training example and quality comparisons

We would like to clarify that source training and cross-context deployment are separate stages. As specified in Section 3.2 and Algorithm 1, source training combines the current task's description, solver background knowledge and local trial feedback, with cross-context retrieval disabled. Eq. (6) governs retrieval at deployment. The cross-context illustration in Appendix A.10 should therefore be read separately from the independent training protocol. In the supplementary training runs, the recorded prompts follow this separation. The LLM builds experience independently on source tasks and subsequently uses retrieved experience to guide few-shot adaptation on related targets.

For the quality comparison, Table 16 tests a fixed selected configuration, while independent tuning repeats measure variability across runs. Its TSP–MOEA/D configuration pc=0.9, pm=0.01, T=35, Tchebycheff gives 2.4910 ×10⁸ over 100 per-instance observations. We repeated tuning five times using 500 training solver calls per run, 100 held-out instances × ten common solver seeds, and 2,500 fitness evaluations per solve. ConfigAgent (7B) and ConfigAgent (32B) achieve 2.5637 ±0.0485 and 2.5547 ±0.0479, within 0.75% and 1.10% of the main-table 2.583 ×10⁸, while SMAC achieves 2.5567 ±0.0651. These repeats remain close to the original result and support comparable quality. Configurations and per-run observations are in the anonymous repository.

Q4 Cross context scoring and parameter compatibility

We would like to clarify the distinction between retrieval and scoring. Eq. (6) determines source eligibility, while Eq. (7) ranks configuration buckets using their recorded quality, support and variability. Eq. (7) does not itself define normalization across contexts with different metric scales. In the new controlled experiment, configurations are ranked within each designated source context and then evaluated under the target's objectives and reference point. Thus, target comparisons use the same metric and evaluation setting. Transferred configurations must also satisfy the target solver's parameter schema, including the complete crossover, mutation, neighborhood and decomposition settings for MOEA/D.

Q5 Efficiency with matched experience

We would like to clarify that the efficiency gain comes from avoiding repeated target-side configuration. Table 5's 80× compares 100 s of deployment with 8,000 s of fresh tuning, while Appendix B separately includes source construction in the lifecycle cost. With direct reuse available to all methods, Table 37 reports ConfigAgent HV above the better classical incumbent by 2.36%, 0.79% and 0.22% on ATSP, mixed-scale TSP and clustered TSP, with zero target tuning trials for each method.

The new ConfigAgent/Warm-SMAC comparison gives both methods the same frozen source records and the same tuning budget to assess adaptation quality. ConfigAgent (32B) records 357–390 s of solver-plus-LLM adaptation time across four target means. Source experience is reused across these targets, consistent with our claim of competitive quality and reduced repeated configuration effort.

I appreciate the authors' response. However, I still have some reservations about the novelty of the proposed approach and the experimental evidence supporting its contributions. I will maintain my original rating.

Reviewer_2tREOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ConfigAgent uses an LLM to tune solver parameters and stores trials and insights in a shared knowledge base (KB). For a new context, it either picks a configuration from matching entries in the KB (zero-shot, no LLM call) or runs 10 additional LLM-guided trials (few-shot). The paper claims quality on par with SMAC3/irace with far fewer trials, and strong transfer to new scales, objectives, and variants.

Critical strengths
  • Reusing configuration experience across contexts is a useful goal. The benchmark is broad (57 contexts), and the paper includes transfer-aware AC baselines that many papers skip (Warm-SMAC, RGPE, MI-SMBO).
Critical weaknesses
  • The appendix contradicts the main results: Table 2 bolds ConfigAgent in every row but the paired analysis on the same contexts in Table 16 shows irace with a higher mean HV than ConfigAgent in 9 of 11 rows, and only 3 of 22 tests are significant. Table 19 has irace winning all five NSGA-II rows. The same cells also take different values in different tables (e.g., CVRP MOEA/D SMAC3: 4.602 vs. 4.0632·10^4). Each tuner is run only once, and the median margin (0.116%) is smaller than the run-to-run variation of the Default configuration (about 0.7%).
  • The cost and transfer claims are not convincing. Without cross-context KB data, ConfigAgent falls below Default on BPP and CVRP (in Table 6), yet the reported cost counts only the 100 trials on the target. The 80× speedup leaves out the 3,800 s it took to build the KB. The OOD tests leak: mixed-scale TSP already has its own training context in the KB, and the ATSP results cite an ATSP training trace. Leave-one-family-out results are at or below Default.
  • It is unclear what the LLM contributes compared with classical warm-starting. Zero-shot mode, which drives the two orders of magnitude claim, makes no LLM call: it is an argmax over stored trials filtered by taxonomy and literature bands. This is close to configuration transfer / meta-learned initialization (Feurer et al., cited). LLAMBO and OPRO are compared only at T=100 without retrieval, and the transfer-aware baselines (Warm-SMAC, RGPE) appear only in the appendix (Table 36), not in the main comparison. The key missing ablation: build the KB with SMAC3/irace instead of the LLM, then use the same zero-shot picker. If that performs similarly, the contribution reduces to the KB/retrieval layer and the hand-written literature bands (Band-SMAC3 already matches ConfigAgent on 3/5 rows in Table 19).
Questions for authors
  1. Which table is correct, Table 2 or Tables 16/19? Can you report mean ± std over at least 5 tuner runs?
  2. How many cross-context KB trials does each Table 2 row use, and what happens if SMAC3 is warm-started with the same data?
  3. The paper says the code will be released, but no supplementary material was submitted. Could you share an anonymous repo? Given the inconsistencies between tables, this is important for the assessment.
  4. If the KB is built from SMAC3/irace trials instead of LLM trials, with the same zero-shot picker, how do the results change?
2 comments

W1/Q1 Quality comparisons and independent tuning repeats

Thank you for suggesting independent tuning repeats. The repeated comparison supports comparable configuration quality. Table 2 reports the main comparison, Table 16 evaluates selected configurations with paired observations, and Table 19 tests numerical-prior bands. The selections can differ: for example, CVRP–MOEA/D uses SMAC configurations T=30/weighted-sum and T=14/PBI in Tables 2/16. LLM proposals and feedback can also produce different tuning trajectories.

We repeated the complete tuning procedure five times under a common TSP–MOEA/D setting: 500 training solver calls, 100 held-out instances × ten common solver seeds, and 2,500 fitness evaluations per solve. Selection uses training performance without the parameter-band filter. HV ×10⁸ (mean ± SD across tuning runs) is 2.5637 ±0.0485 for 7B, 2.4691 ±0.1399 for 14B, 2.5547 ±0.0479 for 32B, and 2.5567 ±0.0651 for SMAC. The 7B/32B differences of +0.28%/−0.08% support comparable quality. Per-run results and the separate band-filtered comparison are in the anonymous supplement.

W2 Cost accounting and transfer separation

We would like to clarify that source-construction cost is included: Table 5 reports 3,800 s, and Appendix B accounts for B × 3,800 + N × deployment cost. The 80× figure compares deployment with fresh tuning, 8,000 s / 100 s. Our efficiency claim concerns reusing source experience across targets without repeating its construction.

The cited examples do not establish test leakage. The original mixed-scale setting tests reuse on new instances from the source scale range. The new experiment tests transfer from a frozen TSP n=500 source to mixed-scale targets. The ATSP tuning illustration is separate from the transfer protocol, which specifies symmetric TSP source records. The supplementary records document the controlled source–target separation.

Table 6 removes deployment retrieval, so it measures that component's contribution rather than the full method's quality. Table 32 tests whether experience from other families can retain quality close to the in-family ConfigAgent reference without target-family source trials. For example, TSP few-shot HV is 2.3368 ×10⁸ versus 2.339 ×10⁸ in-family, a difference of about 0.09%. This experiment examines cross-family reuse rather than requiring every row to exceed Default.

In the new experiments, ConfigAgent (32B) reuses frozen source experience without target tuning in zero-shot mode. Few-shot adaptation records 357–390 s of solver-plus-LLM request time across four target means. Table 37 separately compares direct reuse by all methods: ConfigAgent improves HV over the better classical incumbent by 2.36%, 0.79% and 0.22% on ATSP, mixed-scale TSP and clustered TSP. These results support competitive quality through efficient reuse.

W3/Q4 Contribution and alternative knowledge base builders

We would like to clarify that our contribution is efficient, interpretable configuration and reuse at competitive quality. ConfigAgent (32B) combines problem properties, solver and algorithm background knowledge, and measured feedback to explain configuration choices and guide adaptation. Comparable quality with a classical builder is consistent with this claim, which also concerns the cost of obtaining and reusing that experience.

The original manuscript reports the cost of obtaining reusable experience in Section 4.4, Table 5. Source-context tuning takes 3,800 s with ConfigAgent versus 9,200 s with SMAC3, a 2.42× speedup that saves 5,400 s (90 minutes) per context under the paper's protocol. Table 4 also gives a concrete deployment example on Solomon VRPTW with OR-Tools: ConfigAgent takes 75 s versus 3,980 s for irace, a 53.1× speedup that saves 3,905 s, while attaining objective 751.18 versus 761.55 (lower is better). Source construction is accounted for once in Appendix B and is not repeated for each deployment.

The new builder comparison reuses previously collected and frozen source records from ConfigAgent and SMAC3. It applies the same picker to their stored configurations and evaluates transfer to four targets without rebuilding either knowledge base or tuning on the targets. Each source run used 500 solver calls. Across five repeats, ConfigAgent (32B) has HV differences of +0.01%, −0.64%, −3.47% and −1.67% relative to SMAC3 on mixed-scale TSP, clustered Routing, five-objective TSP and five-objective Routing. These results support comparable transfer quality from reusable experience. Together with the original timing results, they support our claim of matching classical tuning quality while reducing repeated configuration effort.

Adding reasoning improves the quality of adaptation. On five-objective Routing, ConfigAgent (32B) achieves 2.82% higher HV with explicit proposal reasoning than without it, with a 95% confidence interval of [1.22%, 4.43%]. This ablation demonstrates a benefit from reasoning when adapting reused experience, while the recorded proposals and solver feedback make the tuning process inspectable.

Q2 Matched source experience

Thank you for requesting a comparison with equal source experience. Source training uses zero cross-context KB trials, as specified in Algorithm 1, and the completed records are then shared at deployment. For the supplementary comparison, ConfigAgent and Warm-SMAC receive the same frozen source records and the same target tuning budget. Across ConfigAgent (7B), ConfigAgent (14B) and ConfigAgent (32B) on four targets, examples include +0.53% relative to Warm-SMAC for 7B on mixed-scale TSP and +0.16% for 14B on five-objective Routing. The full comparisons and uncertainty estimates are in the anonymous results.

Q3 Anonymous code

Thank you for requesting the code. We provide the core pipeline, supplementary experimental results and reproduction instructions at https://anonymous.4open.science/r/configagent-review-7DF3/.

Reviewer_zrd8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ConfigAgent uses an LLM to configure fixed combinatorial optimization solvers and stores the resulting trials and short, scoped observations in a shared knowledge base. Source contexts are trained independently. At deployment, a deterministic selector retrieves and ranks measured configurations for zero-shot reuse; a few-shot mode adds a small number of LLM-guided target trials. The experiments span five problem families, exact and heuristic solvers, and several distribution shifts. The central claim is that this reuse retains competitive solution quality while reducing the cost of configuring related targets.

Critical strengths
  1. The separation between measured configuration records and textual insights is sensible. In particular, zero-shot deployment selects an evaluated configuration rather than treating an LLM's explanation as a performance measurement. This makes the reuse mechanism relatively easy to inspect.
  2. The evaluation goes beyond comparisons with from-scratch tuners. Appendix B.1 includes a control using the same numerical prior bands, and Appendix C.3 includes direct incumbent reuse and matched-budget numerical transfer methods. Together with the range of solvers, these are useful steps toward testing where the framework adds value.
Critical weaknesses
  1. The tables supporting the quality claim do not give a consistent picture. For TSP with MOEA/D, Table 2 reports ConfigAgent at 2.583∗1082.583 * 10^8, slightly above irace at 2.580∗1082.580 * 10^8. Table 16, presented as the paired analysis of the primary two-objective results, instead reports 2.4910∗1082.4910 * 10^8 versus 2.5823∗1082.5823 * 10^8, reversing that comparison. The zero-shot transfer results also change materially. These differences are much larger than rounding and alter whether zero-shot reuse matches retuning. The paper needs to identify which runs, configurations, seeds, and evaluation protocols each table represents before the claimed quality–cost tradeoff can be assessed.
  2. Some transfer experiments are not explained by the stated method and source data. Equation 6 requires two matches among solver class, encoding, and family. When Knapsack is held out in Table 32, none of the other four families has its binary encoding or family label, so there appears to be no eligible cross-family trial to select; Scheduling's unique dual-layer encoding raises the same issue. These missing distinctions matter both for what is held out and for the source cost being amortized.
Questions for authors
  1. Can you reconcile Tables 2 and 16, and Tables 26 and 37, using a common set of run identifiers and evaluation settings? Which paired results support the final headline results?
  2. What exact source records and fallback rule produce the zero-shot Knapsack and Scheduling results in Table 32 when Equation 6 has no eligible cross-family bucket? If a fallback is used, how much of those results comes from it?
  3. Can you list the source contexts used for each transfer table, including the single-objective and mixed-scale sources, and clarify which source-construction costs enter the lifecycle analysis?
1 comment

W1/Q1 Quality comparisons and table settings

Thank you for examining these comparisons closely. We would like to clarify that the quality claim is to match strong classical tuners, and the transfer claim is to reuse that quality with less target-side tuning. Table 2 reports configuration quality, Table 16 supplies paired tests for selected configurations, and Tables 26/37 examine different transfer conditions.

For CVRP–MOEA/D, SMAC uses T=30/weighted-sum in Table 2 and T=14/PBI in Table 16, while Scheduling uses T=16 and T=20, respectively. Table 16’s TSP ConfigAgent configuration is pc=0.9, pm=0.01, T=35, Tchebycheff, with mean HV 2.4910 ×10⁸ over 100 per-instance observations. Table 26 evaluates ATSP across three algorithms and 2/3/5 objectives, whereas Table 37 fixes NSGA-II/two objectives and distinguishes incumbent reuse, target retuning, ConfigAgent zero-shot and ten-trial few-shot adaptation.

To assess tuning variability directly, we repeated TSP–MOEA/D tuning five times with 500 training solver calls per run, followed by 100 held-out instances × ten common solver seeds, at 2,500 fitness evaluations per solve. ConfigAgent (7B), ConfigAgent (32B) and SMAC achieve 2.5637 ±0.0485, 2.5547 ±0.0479 and 2.5567 ±0.0651 HV ×10⁸. The ConfigAgent means are within 1.1% of the original 2.583 ×10⁸, and differ from SMAC by +0.28% and −0.08%. LLM proposals and feedback can lead separate tuning runs to select different configurations, while these repeated results remain consistent with comparable quality.

W2/Q2 Held out family transfer and selected records

We would like to clarify that Table 32 tests cross-family generalization: how much configuration quality is retained when the target family contributes no source trials. Its reference is in-family ConfigAgent. For example, TSP few-shot HV is 2.3368 ×10⁸ versus 2.339 ×10⁸ in-family, a difference of about 0.09%. When no structurally eligible source exists for Knapsack or Scheduling, the selector falls back to the remaining NSGA-II records. It transfers crossover and mutation parameters, not solutions with another problem's encoding. In the archived selection, Knapsack receives pc=0.9, pm=0.001 from TSP trials, while Scheduling receives pc=0.9, pm=0.01 from BPP/Knapsack trials. These selections use the fallback rather than Eq. (6)'s usual eligibility rule.

Q3 Source contexts and lifecycle cost

We would like to clarify how source experience supports transfer. The multi-objective study uses primary two-objective sources across the five problem families, while the single-objective variants use their corresponding base-family solver sources. The mixed-scale subsection additionally describes a mixed-scale source, and Table 36 uses TSP sources at different scales. In the new controlled study, the frozen sources are two-objective TSP n=500 and Routing n=200. ConfigAgent (32B) combines problem descriptions, solver and algorithm background knowledge, and measured feedback to build configuration experience. Few-shot transfer uses this experience to guide LLM proposals on the target, while zero-shot directly selects an evaluated configuration. Source construction is counted once in the lifecycle analysis and shared across subsequent deployments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.