- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RULER, an LLM-guided framework for online algorithm configuration. The main idea is to let LLMs summarize historical configuration trials into reusable edit–response rules that specify the conditions under which parameter modifications may be effective, their expected solver responses, and the corresponding supporting or contradicting evidence.
The framework has three major components: (1) edit-response rule induction which extracts reusable knowledge from historical experiments; (2) contextual hypothesis assessment, which lets an LLM decide whether retrieved rules are applicable to the current optimization state; and (3) hypothesis-guided experiment planning, which decides one of the following subsequent experiments: memory validation, guided improvement, mechanism contrast, and coverage exploration.
The authors evaluate RULER on TSP and CVRP. RULER achieves an average 10.7% relative reduction in the best-known-solution gap compared with the strongest baselines.
Critical strengths
-
Interesting idea to summarize optimization knowledge. Instead of simply conditioning an LLM on historical results, RULER explicitly introduces edit–response hypotheses with supporting and contradicting evidence. This provides a structured mechanism for accumulating and reusing knowledge across optimization instances.
-
Comprehensive experiment results. The paper evaluates four problem–solver settings and compares RULER against varieties of baselines, including some classical methods, and LLM-based methods such as AutoEP. RULER achieves the lowest average reference gap across all four settings.
-
Useful component ablations and analyses. The results shown in the ablation studies generally demonstrate that removing introduced components degrades performance. The authors also provide other related results.
Critical weaknesses
-
Do the authors run everything for multiple time and get the average? Reporting variance would be important to assess the performance of RULER compared to baselines.
-
Analysis of experimental roles. It would be useful to understand how frequently each role is selected during optimization. Could the authors provide a breakdown of the selection frequency?
Questions for authors
- AutoEP implementation: The paper mentions that AutoEP does not provide a complete runnable Python evaluation pipeline or full prompt set, motivating the re-implementation, i.e., AutoEP-Rep. However, AutoEP has an official public GitHub repo containing source code. Could the authors clarify which components of the original implementation were unavailable or could not be reused, and explain any important differences between AutoEP-Rep and the released AutoEP implementation?
5 comments
Response to Weakness 2 – Analysis of experimental roles
Thanks for your question and suggestion, we report detail experiment result of how frequently each role is selected during optimization in Response to Weakness 2 and Question 2-- Contribution of the Experimental Role Decomposition for Reviewer FJxY In summary, all four roles are utilized in each of the four experimental settings, although their allocation frequencies vary substantially across solvers. Guided Improvement is the most frequently selected role in both TSP settings (45.70% and 45.56%), while Coverage Exploration is dominant in CVRP PSO+ILS (44.89%), and Mechanism Contrast is most frequently selected in CVRP SA+ILS (39.77%). These results illustrate that role allocation varies with the optimization setting rather than following a fixed allocation pattern. Please refer to Table 5 in our response to Reviewer FJxY for the complete results.
Response to Question 1 – Autoep Implementation details
We sincerely thank the reviewer for this question and AutoEP's authors for providing open-source code. The original AutoEP's official repo includes Python solver code, a TSP FastGPT workflow export with embedded prompts, and Flask/MySQL components for data exchange. However, we were unable to run the released implementation using the public artifacts in an end-to-end manner. In particular, the released Python drivers delegate LLM-based parameter updates to a separately configured FastGPT workflow and exchange state through Flask and MySQL. Therefore, we reused the released TSP GA and CVRP PSO solver implementations and adapted AutoEP's documented configuration loop to run directly within our shared Python evaluation harness. This integration, which we call AutoEP-Rep., runs end-to-end on our evaluation benchmarks under the same evaluation protocol as RULER. We will revise the manuscript to state precisely what AutoEP releases and which parts we reused or adapted. We also provide a solver-aligned PSO-2opt comparison in our response to Reviewer FJxY's Q4.
Response to Weakness 1 – Robustness of experiment
We sincerely thank the reviewer for carefully reading our manuscript and providing valuable suggestions. The main experiments use global seed 0 and greedy decoding (temperature 0). So we conducted three independent robustness runs for each of the four main settings, using different global random seeds and an LLM temperature of 0.2. The corresponding results, including the mean, sample standard deviation, and variance, are reported in Table 3 of our response to Reviewer FJxY, Weakness 2 ("Component Ablation"). Appendix E.1 further clarifies the randomness protocol.