- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SAMR, a state-aware reflection framework for LLM-based automatic heuristic design. It tracks the momentum, stagnation, and age of individual islands to determine their search states. Based on these states, it dynamically selects short-term, long-term, and cross-island reflection to guide program generation. The method is evaluated on 36 CO-Bench tasks using Claude Sonnet 4.5 and DeepSeek V3. Results show improvements over several existing baselines, while CORAL achieves slightly better performance at a substantially higher token cost.
Critical strengths
- The idea of adapting reflection based on the search state of individual islands is reasonable and intuitive.
- The paper includes comparisons across two LLM backbones and ablation studies to examine the contributions of different components.
Critical weaknesses
-
Stagnation may reflect task difficulty rather than ineffective search. Score improvements may naturally become smaller or less frequent as solution quality increases. Thus, SAMR's state controller may misinterpret normal optimization plateaus as stagnation, leading to premature exploration or resets.
-
Unfair budget comparison with CORAL. The authors match methods by the number of heuristic evaluations, but CORAL consumes 26.6× more tokens than SAMR. Without reporting CORAL's performance under the same token budget, the claimed quality–cost advantage is not sufficiently supported.
-
Limited benchmark coverage in the CORAL comparison. CORAL was originally evaluated on diverse mathematical and systems optimization tasks, yet none of its original benchmarks appear to be included in SAMR's evaluation. This makes the comparison less convincing, as it remains unclear whether SAMR can achieve comparable performance on tasks where CORAL has demonstrated strong results.
Questions for authors
See my weakness