- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RADAR, a framework for LLM-based automated solver design that separates candidate ranking from code revision. It combines runtime-aware diagnosis, racing-based evaluation, and branch-adaptive reflection to improve solver generation. Experiments across seven combinatorial optimization benchmarks demonstrate competitive performance against existing automated heuristic design methods.
Critical strengths
- The paper identifies an important limitation of score-based feedback and provides a clear motivation for runtime-aware solver revision.
- The proposed framework effectively integrates execution-trace diagnosis, racing, and adaptive reflection into automated solver design.
- Experiments across seven optimization benchmarks and multiple LLM backbones demonstrate promising performance.
- The ablation studies, trace-swapping controls, and racing analysis provide useful evidence supporting the proposed mechanisms.
Critical weaknesses
- The computational efficiency of the complete framework remains unclear, as the main comparisons do not match LLM inference costs.
- The main experiments use only four independent searches per configuration, limiting the statistical reliability of some reported improvements.
- The racing mechanism relies on a fixed anchor sample and heuristic threshold, raising concerns about robustness under distribution shifts.
- The individual components build on existing execution-guided revision and racing techniques, making the methodological novelty somewhat incremental.
Questions for authors
-
How does the complete RADAR framework compare with baselines under matched total computational budgets, including LLM inference costs?
-
Can the authors provide stronger statistical evidence for improvements with relatively small performance margins?
-
How sensitive is racing to the anchor sample and promotion threshold under heterogeneous or shifted instance distributions?
5 comments
Response to reviewer Mg3P (2/2)
W3, Q3: Racing sensitivity to the anchor sample and keep fraction
Racing is largely robust to both, including on mixed instances, and it helps under distribution shift.
Keeping the top 40% is lenient, since the gate only needs to catch clear losers. In the replay of 104 searches (Appendix G), using any of the five training samples as the anchor loses at most one champion per task, and keep fractions from 30% to 50% keep at least 96% of the top-10% programs.
On mixed instances (uniform, clustered and anisotropic; complete RADAR, Qwen3-32B; Appendix G, p. 24), racing stops 59% (CVRP) and 39% (OP) of programs and still keeps every top-10% program and every champion. This holds for any anchor geometry and for keep fractions from 25% to 60%, because a program that is weak on one geometry is weak on the others (rank correlation between geometries 0.87 to 0.96). In live searches on these instances, racing is ahead of no racing on both tasks: 20.56 vs. 21.69 on CVRP-200 and 86.79 vs. 86.56 on OP-200 (mixed test set, so not comparable with Table 2 in the paper).
A single anchor weakens only when instances differ so much that the rank on one says little about the others. We built such a case: CVRP with and under the same 5 s limit, replaying racing on held-back samples for ~9,000 programs from 92 no-racing searches. There a single anchor keeps 82% / 83% of the top-10% programs, and one anchor sample per instance size brings this back to 97% / 93%:
| Training mix | Anchor | Top-10% kept (Qwen / GPT) | Champions kept (Qwen / GPT) |
|---|---|---|---|
| Single size () | one anchor | 98% / 89% | 48/48 / 39/44 |
| Moderate (50 + 200) | one anchor () | 94% / 89% | 47/48 / 41/44 |
| Moderate (50 + 200) | one anchor per size | 98% / 94% | 48/48 / 43/44 |
| Extreme (50 + 1000) | one anchor () | 82% / 83% | 40/48 / 38/44 |
| Extreme (50 + 1000) | one anchor per size | 97% / 93% | 46/48 / 44/44 |
We have added this to the paper as the recommended setting for mixed instance types (Appendix G, p. 24).
Racing also helps under distribution shift. We trained on synthetic CVRP-200 and tested the champions, with no further search, on the 100 real CVRPLIB Set X instances ( = 100 to 1000, 60 s each; Appendix P, p. 32). The gap of a champion on an instance is , where BKS is the best-known solution value provided in the CVRPLIB dataset. Racing lowers the gap in all three comparisons:
| Training set | Comparison | Gap without racing | Gap with racing |
|---|---|---|---|
| Uniform (Table 3a) | diagnosis only racing + diagnosis | 10.00% | 7.34% |
| Uniform (Table 3a) | neither racing only | 15.16% | 13.12% |
| Mixed (uniform, clustered, anisotropic) | no racing racing (top 40%) | 12.72% | 7.08% |
W4: Novelty of the components
Our main novelty is the view that ranking programs and revising them need different signals, and each mechanism beats its direct alternative. The mechanisms are deliberately simple ways to act on that view.
Current AHD methods use the final score, or text derived from it, for both: to choose which programs to keep and to tell the LLM what to edit. We find that the score works for ranking but not for revision, because programs with the same score can need different edits (behavioral aliasing), and score-derived reflections add no detectable value (Section 4.2). An execution trace does help: a diagnoser with the parent's trace beats the same diagnoser without it (Table 3b; CVRP-200: 24.23 vs. 25.14, OP-200: 58.09 vs. 57.61). To our knowledge earlier work does not make this separation: LaGO's feedback describes the final solutions, and DASH uses the convergence curve mainly to choose between solvers.
We also compared each of the two main mechanisms with a direct alternative. A single writer call given the same trace and search history does worse than the separate diagnoser (24.52 vs. 24.23 on CVRP-200 and 57.77 vs. 58.09 on OP-200; a new experiment for Reviewer izpJ), and standard successive halving at the same budget is behind our gate on both tasks (25.19 vs. 23.98 on CVRP-200 and 58.73 vs. 58.91 on OP-200). Racing and diagnosis also improve MCTS-AHD and ReEvo when added unchanged (Table 3c).
Follow up to reviewer Mg3P
Thank you again for taking the time to review our paper. We have posted our responses along with the updated paper, and we would be grateful to hear whether they address your concerns. We are happy to discuss further or answer any additional questions.
Response to reviewer Mg3P (1/2)
We thank the reviewer for the detailed review.
W1, Q1: Comparison under matched total budget, including LLM inference
The complete RADAR stays ahead when LLM cost is matched. We stopped every method, including the complete RADAR (diagnosis, racing and BAR), at 1M tokens per search with Qwen3-32B. The budget counts the input and output tokens of every LLM call, so for RADAR it covers the diagnoser as well as the writer. The other methods are run unchanged, without racing.
Table G2. Comparison at the same LLM budget with Qwen3-32B (Appendix H, pp. 25-26); held-out score, mean ± SE.
RADAR is best on both Qwen3-32B tasks, uses fewer solver runs, and leads from a quarter of the budget onward.
W2, Q2: Statistical evidence for small margins
We doubled the searches to 8 per method on all seven tasks (Qwen3-32B), and the evidence is stronger.
Table G1. Main comparison of Table 2a with 8 searches per method, double the 4 in the paper (Qwen3-32B; full table in Appendix M, p. 29); mean ± SE. ↓ means lower is better and ↑ higher is better.
RADAR has the best mean on all seven tasks against every baseline (sign test p=0.016 per baseline). It is now significantly better than the strongest other method on four tasks (OP-200, TSP-200, CVRPTW-100 and PFSP-500; Welch t-test, p<0.05).
The margins that remain small are on tasks near saturation. On MKP and UFLP RADAR is within 0.8% and 0.3% of the reference, and with GPT-5.4-mini nearly every method is that close on four tasks, so we claim no gain there. Where there is room, the lead on OP and CVRP also holds in two further settings: at matched LLM cost with Qwen3-32B (W1, Q1 above) and, for CVRP, on real CVRPLIB Set X instances (6.48% vs. 8.30% gap for the second best).