- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SA-DREAM, a candidate selection framework for LLM-based AHD that prioritizes candidates worth evaluating. Candidate edits are expressed in a typed grammar and applied deterministically to a parent algorithm. A frozen Qwen2.5-Coder-7B model with a CodeReasoner adapter predicts each edit’s consequence, conditioned on past transitions. Candidates are scored by predicted improvement over the current incumbent, discounted by model–retrieval agreement, and only top-scoring candidates receive real evaluation. Experiments cover five combinatorial problem families at four scales. They compare against surrogate selectors under a shared edit interface and against native AHD systems including EoH, LLaMEA, ReEvo, PartEvo, and MCTS-AHD. The paper reports average ranks of 1.15 and 1.05, respectively.
Critical strengths
- The motivation is strong, and the problem is important. Prioritizing which candidates to evaluate is a meaningful, practical problem for AHD.
- The incumbent-relative utility is well motivated and precisely stated
- Separating controlled and native comparisons is a sound design choice.
Critical weaknesses
Rank-based reporting and inconsistent results.
The main claims rest on average ranks, which hide effect size. For example, SA-DREAM’s 11.6% loss to MCTS-AHD on native BPP S1 appears only as “rank 2.”
The magnitude data in the appendix conflict with the main results:
- Table 1(a) cannot be derived from Table 6, contrary to its caption. KNN is last in all 20 settings of Table 6, yet has overall rank 7.00 among 8 methods.
- Table 6 contradicts Table 9 at every budget. On BPP, Table 9 places all methods within 0.032 canonical quality, while Table 6 shows KNN using 43% more bins than SA-DREAM at S1. On CVRP, KNN is worst at every scale in Table 6 but never worst in Table 9.
- TSP magnitudes in Table 6 are implausible. SA-DREAM’s S4 value (25,236) is about 11× the expected optimal tour for 1,000 uniform points.
Dated backbone
The predictor is Qwen2.5-Coder-7B, a model released in 2024. Stronger models of similar cost now exist. CodeReasoner is not cited or explained, and its reasoning is suppressed by the JSON-only, 256-token output format. Figure 4 tests adaptation only at 7B and scale only without adaptation, so the two effects are confounded. “7B+Weave” is undefined.
Unspecified LLMs in the end-to-end comparison
The paper does not name the LLM used by the native baselines, by SA-DREAM’s generator, or by the Direct LLM Ranker. Native AHD systems typically run on stronger API models. Such models would generate better candidates and use evaluation feedback more effectively, which may reduce the need for pre-screening. Cheap evaluators and budgets of 8–32 evaluations make evaluation artificially scarce. Table 10 also lists fewer LLM calls than evaluated candidates for every native method, which needs explanation.
Missing details and baselines
There is no random-selection baseline, even though at B=8 the selector discards only about one of roughly nine rankable candidates per root. The label-to-score readout, tie rules, and parent-selection rule are deferred to the artifact. “World-model planning” overstates a one-step predictor.
Questions for authors
- Why are the main results reported as ranks? Please provide per-setting relative improvements over the best baseline in raw objective units, with paired confidence intervals for both panels. Please also reconcile Tables 1, 6, 8 and 9, state which budget Tables 6 and 8 correspond to, and explain why SA-DREAM’s values differ between Tables 6 and 8 for the same settings.
- Why Qwen2.5-Coder-7B rather than a current model? Which CodeReasoner checkpoint and training data were used? Do the results hold with a newer predictor, or with CodeReasoner-14B?
- Which LLM did each native baseline, the generator, and the Direct LLM Ranker use? If they differ from each other or from current strong models, why? Does the advantage persist when all systems share the same strong LLM and are matched on total compute as well as evaluations?