- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies LLM-based automatic heuristic design (AHD) by adding two components to MCTS-AHD. Search Experience Memory (SEM) records code lineage, edit differences, performance changes, and failure attributions, and retrieves historical cases from other search paths during mutation. Meta-Reflection uses search statistics, including stagnation, branch distribution, operator performance, and failure categories, to adjust exploration parameters, the novelty weight, and operator allocation online. The framework is evaluated with constructive and ant-colony solvers, covering five combinatorial optimization problems, six problem–solver combinations, and nineteen size configurations.
According to the means in Tables 1–3, the full method outperforms the four evaluated AHD baselines on sixteen configurations and performs slightly worse than MCTS-AHD on the remaining three. However, its central design substantially overlaps with existing experience-driven and adaptive evolutionary methods. The experiments do not establish an advantage over the most closely related work or contemporary program-evolution frameworks. The implementation contains differences, but their methodological significance and the paper's broad superiority claims are not adequately supported.
Critical strengths
- Concrete implementation details. The appendices specify the retrieval function, prompts, memory capacity, triggering conditions, parameter constraints, and token usage, making the system's operation reasonably clear.
- Improvements on some configurations. Several TSP and CVRP configurations improve over the selected AHD baselines, suggesting that the implementation warrants further investigation.
Critical weaknesses
W1. Limited novelty relative to closely related work
The central combination of experience memory and adaptive search control has substantial precedents: HiFo-Prompt couples an insight pool with search-state guidance [1], EvoPH co-evolves prompts and heuristics using execution experience [2], and RefineEvo combines parent–offspring reflection, positive and negative experience retrieval, and adaptive operator management [3]. ShinkaEvolve, EvoX, and AdaEvolve further demonstrate experience reuse or adaptation of the search process [4–6]. The remaining distinctions—identifier-based retrieval, lineage exclusion, failure categories, and bounded parameter updates—are plausible implementation choices, but the paper does not establish that they yield a substantive methodological advance, a new capability, or a generalizable insight.
W2. Missing comparisons with the most relevant recent methods
Tables 1–3 compare only against ReEvo, HSEvo, PathWise, and MCTS-AHD, omitting HiFo-Prompt and EvoPH despite citing them, as well as the directly relevant RefineEvo [1–3]. Comparisons with these methods are essential to establish the value of the proposed memory and control mechanisms. The evaluation should also include ShinkaEvolve [4] and a compatible adaptive-search baseline, such as EvoX or AdaEvolve [5,6], under a common model, evaluator, and resource budget. Without these comparisons, the reported gains do not establish an advance over the current field.
W3. The contributions of the proposed components are not isolated
Despite the claim that the host search's selection logic is unchanged, Equation (9) adds a novelty bonus to UCT, while Appendix C.3 introduces difference-aware mutation guidance. The ablations lack a control that retains both modifications while disabling SEM and Meta-Reflection, preventing clean attribution of improvements over MCTS-AHD. The additional use of deepseek-v4-pro for reflection introduces a further confound: matching the code-generation model and evaluation count does not match model capability or total inference expenditure. A common execution-layer control and matched-resource comparisons are needed to isolate the contributions of the two proposed components.
W4. The reported gains lack sufficient statistical support
Recomputing improvements against the best AHD baseline in each configuration yields a median relative gain of approximately 0.279%; 11 of 19 configurations improve by less than 1%, including three regressions. These calculations summarize reported means and do not establish statistical significance. Although some CVRP gains are appreciable, differences such as 202.698 versus 202.695 on BPP-ACO at N=500 are difficult to interpret without uncertainty estimates. Reporting only three-run means, without standard deviations, confidence intervals, or individual results, is insufficient to establish the robustness of the claimed advantage.
W5. Generalization and the reliability of reflection remain unverified
The manuscript does not clearly specify the sizes and separation of search, validation, and test sets, leaving test independence unresolved. Separate searches at different sizes do not establish cross-size generalization, and transfer beyond MCTS-AHD is untested. Failure attribution is also insufficiently validated: parent–child code and objective values alone provide limited evidence for diagnoses such as overfitting, while substring-based category parsing may propagate misclassification into global control. Success-only memory, unattributed failure feedback, and random-retrieval controls, together with comparisons against simple rule-based or bandit controllers, are needed to establish whether the proposed reflection mechanisms are reliable and necessary.
W6. Cost accounting and several central claims require correction
The reported 9.03% cost premium compares the two-model setup with pricing the same calls entirely using flash; it does not measure overhead relative to the baselines. Unequal evaluation budgets and modified population settings further complicate fairness. The claim of matching or outperforming baselines on all 19 configurations contradicts three lower-performing means, while component indispensability is undermined by CVRP-ACO at N=100, where removing Meta improves the objective from 14.679 to 14.055. Appendix E.2 also acknowledges prompts requesting lower objectives for maximization tasks. Complete resource accounting, corrected-prompt controls, and claims consistent with the reported evidence are required.
Questions for authors
- Novelty: Relative to HiFo-Prompt, EvoPH, and RefineEvo [1–3], which central mechanism is not already covered, and why is the distinction more than an implementation choice?
- Essential comparisons: Under the same model, evaluator, and total cost, does the method still outperform the directly related experience-driven methods [1–3], ShinkaEvolve [4], and EvoX or AdaEvolve [5,6]?
- Attribution: How much robust improvement does each component contribute when the common control retains novelty-enhanced UCT and difference-aware prompting but disables both SEM and Meta-Reflection?
- Controller necessity: Does LLM meta-reflection outperform simple rules or bandit control based on stagnation and operator rewards? Does categorical failure attribution improve over unclassified execution feedback?
- Reliability and generalization: Can the authors specify the search, validation, and test splits and provide individual run results, uncertainty estimates, search curves, and cross-size or out-of-distribution evaluations?
- Interpretation: What explains the deterioration caused by Meta on some CVRP configurations and the very small SEM gains on some TSP-ACO configurations? Do the conclusions hold after correcting the maximization-task prompts?
- Cost and claims: Can the authors report complete costs and runtimes relative to each baseline and revise the claims of 19/19 non-inferiority, component indispensability, and approximately 9% additional cost?
References
[1] Chentong Chen, Mengyuan Zhong, Ye Fan, Jialong Shi, and Jianyong Sun. HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design. ICLR, 2026. arXiv:2508.13333.
[2] Yihong Liu, Junyi Li, Hongyu Lu, Xin Zhao, and Ji-Rong Wen. Experience-Driven Reflective Co-Evolution of Prompts and Heuristics for Autonomous Algorithm Design. Findings of ACL, pp. 4981–4995, 2026. DOI: 10.18653/v1/2026.findings-acl.245. (EvoPH.)
[3] Yang Wu, Junran Pan, Yifan Zhang, Ning Xu, Fanshuo Zeng, and Jian Cheng. RefineEvo: Planning-Guided Heuristic Evolution with Bidirectional Experience. ICML, PMLR 306:136582–136624, 2026.
[4] Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution. ICLR, 2026. arXiv:2509.19349.
[5] Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, et al. EvoX: Meta-Evolution for Automated Discovery. arXiv preprint arXiv:2602.23413, 2026.
[6] Mert Cemri, Shubham Agrawal, Akshat Gupta, et al. AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization. arXiv preprint arXiv:2602.20133, 2026.
I thank the authors for the clarifications and additional experiments. However, the response does not sufficiently resolve my concerns about novelty and attribution of the reported gains. I therefore maintain my original score.