acceptodds
Under review as a conference paper at ICLR 2027

RADAR: RUNTIME-AWARE DIAGNOSIS AND RACING FOR AGENTIC SOLVER DESIGN

Abstract

LLM-based automated algorithm design often uses a solver's final scalar score both to rank candidates and to guide code revisions. Yet similar scores can conceal different runtime behavior that calls for different edits, a problem we term behavioral aliasing. In controlled studies of MCTS-AHD and ReEvo, we find no detectable performance gain from exposing exact score magnitudes or using run-specific reflections. We introduce RADAR (Runtime-Aware Diagnosis and Racing), an agentic framework that separates the two roles: execution traces guide revisions, and scores allocate evaluation effort. A trusted evaluator records when valid solutions improve, how long the solver runs without further improvement, and why outputs are rejected; a diagnostic agent turns this trace and the search history into one measured bottleneck and one code edit for a writer agent. Racing first runs each new program on a single training sample and stops weak programs before they are tested on the remaining samples, and Branch-Adaptive Reflection changes the edit strategy when a search branch stalls. Across seven benchmarks, RADAR with Qwen3-32B achieves the best mean score among the evaluated search baselines on every task; on CVRP-200 its gap to the reference is 7.20% versus 11.76% for the strongest baseline. With GPT-5.4-mini, RADAR leads or ties on five of the seven tasks. Attaching our racing and diagnosis mechanisms to MCTS-AHD and ReEvo improves both controllers on all three tested tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Automated Heuristic Design, Automated Algorithm Design, LLM-generated solvers, combinatorial optimization, heuristic search, self feedback, racing, tree search

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Mg3POpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RADAR, a framework for LLM-based automated solver design that separates candidate ranking from code revision. It combines runtime-aware diagnosis, racing-based evaluation, and branch-adaptive reflection to improve solver generation. Experiments across seven combinatorial optimization benchmarks demonstrate competitive performance against existing automated heuristic design methods.

Critical strengths
  1. The paper identifies an important limitation of score-based feedback and provides a clear motivation for runtime-aware solver revision.
  2. The proposed framework effectively integrates execution-trace diagnosis, racing, and adaptive reflection into automated solver design.
  3. Experiments across seven optimization benchmarks and multiple LLM backbones demonstrate promising performance.
  4. The ablation studies, trace-swapping controls, and racing analysis provide useful evidence supporting the proposed mechanisms.
Critical weaknesses
  1. The computational efficiency of the complete framework remains unclear, as the main comparisons do not match LLM inference costs.
  2. The main experiments use only four independent searches per configuration, limiting the statistical reliability of some reported improvements.
  3. The racing mechanism relies on a fixed anchor sample and heuristic threshold, raising concerns about robustness under distribution shifts.
  4. The individual components build on existing execution-guided revision and racing techniques, making the methodological novelty somewhat incremental.
Questions for authors
  1. How does the complete RADAR framework compare with baselines under matched total computational budgets, including LLM inference costs?

  2. Can the authors provide stronger statistical evidence for improvements with relatively small performance margins?

  3. How sensitive is racing to the anchor sample and promotion threshold under heterogeneous or shifted instance distributions?

5 comments
AuthorsOpenReview ↗

Response to reviewer Mg3P (1/2)

We thank the reviewer for the detailed review.

W1, Q1: Comparison under matched total budget, including LLM inference

The complete RADAR stays ahead when LLM cost is matched. We stopped every method, including the complete RADAR (diagnosis, racing and BAR), at 1M tokens per search with Qwen3-32B. The budget counts the input and output tokens of every LLM call, so for RADAR it covers the diagnoser as well as the writer. The other methods are run unchanged, without racing.

Table G2. Comparison at the same LLM budget with Qwen3-32B (Appendix H, pp. 25-26); held-out score, mean ±\pm SE.

Model, budget per searchTaskRADARBest other methodSolver runs (RADAR / other)
Qwen3-32B, 1M tokensCVRP-200 ↓\downarrow24.03 ±\pm 0.2224.47 ±\pm 0.22 (EoH)371 / 1154
Qwen3-32B, 1M tokensOP-200 ↑\uparrow59.19 ±\pm 0.2058.52 ±\pm 0.43 (MCTS-AHD)434 / 1284

RADAR is best on both Qwen3-32B tasks, uses fewer solver runs, and leads from a quarter of the budget onward.

W2, Q2: Statistical evidence for small margins

We doubled the searches to 8 per method on all seven tasks (Qwen3-32B), and the evidence is stronger.

Table G1. Main comparison of Table 2a with 8 searches per method, double the 4 in the paper (Qwen3-32B; full table in Appendix M, p. 29); mean ±\pm SE. ↓\downarrow means lower is better and ↑\uparrow higher is better.

TaskRADARStrongest other method
TSP-200 ↓\downarrow11.19 ±\pm 0.0511.35 ±\pm 0.04 (Best-of-N)
CVRPTW-100 ↓\downarrow15.35 ±\pm 0.4017.39 ±\pm 0.58 (MCTS-AHD)
PFSP-500 ↓\downarrow25893 ±\pm 725953 ±\pm 15 (EoH)
CVRP-200 ↓\downarrow23.78 ±\pm 0.1224.39 ±\pm 0.31 (Best-of-N)
OP-200 ↑\uparrow58.88 ±\pm 0.2857.63 ±\pm 0.32 (MCTS-AHD)
MKP-500 ↑\uparrow104.78 ±\pm 0.18104.61 ±\pm 0.19 (MCTS-AHD)
UFLP-1000 ↓\downarrow185.22 ±\pm 0.08185.37 ±\pm 0.17 (EoH)

RADAR has the best mean on all seven tasks against every baseline (sign test p=0.016p = 0.016 per baseline). It is now significantly better than the strongest other method on four tasks (OP-200, TSP-200, CVRPTW-100 and PFSP-500; Welch t-test, p<0.05p < 0.05).

The margins that remain small are on tasks near saturation. On MKP and UFLP RADAR is within 0.8% and 0.3% of the reference, and with GPT-5.4-mini nearly every method is that close on four tasks, so we claim no gain there. Where there is room, the lead on OP and CVRP also holds in two further settings: at matched LLM cost with Qwen3-32B (W1, Q1 above) and, for CVRP, on real CVRPLIB Set X instances (6.48% vs. 8.30% gap for the second best).

AuthorsOpenReview ↗

Response to reviewer Mg3P (2/2)

W3, Q3: Racing sensitivity to the anchor sample and keep fraction

Racing is largely robust to both, including on mixed instances, and it helps under distribution shift.

Keeping the top 40% is lenient, since the gate only needs to catch clear losers. In the replay of 104 searches (Appendix G), using any of the five training samples as the anchor loses at most one champion per task, and keep fractions from 30% to 50% keep at least 96% of the top-10% programs.

On mixed instances (uniform, clustered and anisotropic; complete RADAR, Qwen3-32B; Appendix G, p. 24), racing stops 59% (CVRP) and 39% (OP) of programs and still keeps every top-10% program and every champion. This holds for any anchor geometry and for keep fractions from 25% to 60%, because a program that is weak on one geometry is weak on the others (rank correlation between geometries 0.87 to 0.96). In live searches on these instances, racing is ahead of no racing on both tasks: 20.56 vs. 21.69 on CVRP-200 and 86.79 vs. 86.56 on OP-200 (mixed test set, so not comparable with Table 2 in the paper).

A single anchor weakens only when instances differ so much that the rank on one says little about the others. We built such a case: CVRP with n=50n = 50 and n=1000n = 1000 under the same 5 s limit, replaying racing on held-back samples for ~9,000 programs from 92 no-racing searches. There a single anchor keeps 82% / 83% of the top-10% programs, and one anchor sample per instance size brings this back to 97% / 93%:

Training mixAnchorTop-10% kept (Qwen / GPT)Champions kept (Qwen / GPT)
Single size (n=200n = 200)one anchor98% / 89%48/48 / 39/44
Moderate (50 + 200)one anchor (n=50n = 50)94% / 89%47/48 / 41/44
Moderate (50 + 200)one anchor per size98% / 94%48/48 / 43/44
Extreme (50 + 1000)one anchor (n=50n = 50)82% / 83%40/48 / 38/44
Extreme (50 + 1000)one anchor per size97% / 93%46/48 / 44/44

We have added this to the paper as the recommended setting for mixed instance types (Appendix G, p. 24).

Racing also helps under distribution shift. We trained on synthetic CVRP-200 and tested the champions, with no further search, on the 100 real CVRPLIB Set X instances (nn = 100 to 1000, 60 s each; Appendix P, p. 32). The gap of a champion on an instance is (cost−BKS)/BKS(\text{cost} - \text{BKS}) / \text{BKS}, where BKS is the best-known solution value provided in the CVRPLIB dataset. Racing lowers the gap in all three comparisons:

Training setComparisonGap without racingGap with racing
Uniform (Table 3a)diagnosis only →\to racing + diagnosis10.00%7.34%
Uniform (Table 3a)neither →\to racing only15.16%13.12%
Mixed (uniform, clustered, anisotropic)no racing →\to racing (top 40%)12.72%7.08%

W4: Novelty of the components

Our main novelty is the view that ranking programs and revising them need different signals, and each mechanism beats its direct alternative. The mechanisms are deliberately simple ways to act on that view.

Current AHD methods use the final score, or text derived from it, for both: to choose which programs to keep and to tell the LLM what to edit. We find that the score works for ranking but not for revision, because programs with the same score can need different edits (behavioral aliasing), and score-derived reflections add no detectable value (Section 4.2). An execution trace does help: a diagnoser with the parent's trace beats the same diagnoser without it (Table 3b; CVRP-200: 24.23 vs. 25.14, OP-200: 58.09 vs. 57.61). To our knowledge earlier work does not make this separation: LaGO's feedback describes the final solutions, and DASH uses the convergence curve mainly to choose between solvers.

We also compared each of the two main mechanisms with a direct alternative. A single writer call given the same trace and search history does worse than the separate diagnoser (24.52 vs. 24.23 on CVRP-200 and 57.77 vs. 58.09 on OP-200; a new experiment for Reviewer izpJ), and standard successive halving at the same budget is behind our gate on both tasks (25.19 vs. 23.98 on CVRP-200 and 58.73 vs. 58.91 on OP-200). Racing and diagnosis also improve MCTS-AHD and ReEvo when added unchanged (Table 3c).

AuthorsOpenReview ↗

Follow up to reviewer Mg3P

Thank you again for taking the time to review our paper. We have posted our responses along with the updated paper, and we would be grateful to hear whether they address your concerns. We are happy to discuss further or answer any additional questions.

Reviewer_NccpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RADAR separates two roles of evaluation in LLM-based solver design: ranking candidate programs and guiding their revision. It uses runtime evidence to diagnose specific improvements, racing to allocate evaluation effort, and adaptive reflection when search stalls. Experiments across seven optimization tasks show promising results, supported by ablations and integrations into existing search methods.

Critical strengths
  • The paper identifies a useful distinction: a final score indicates how well a program performs, while its execution behavior can help explain how to improve it. This provides a clear motivation for more informative feedback in automated solver design.

  • The experimental controls strengthen the contribution. Comparisons using correct, withheld, and swapped traces help establish that relevant execution evidence contributes beyond simply adding another LLM call. Improvements when integrated into existing controllers also suggest practical usefulness.

Critical weaknesses
  • The overall cost–benefit remains unclear. Main comparisons match solver runs, while RADAR uses more LLM calls. It remains uncertain when the complete pipeline justifies its additional cost and complexity.

  • The breadth of tasks is encouraging, but the robustness of the advantage is less established. Main comparisons use four searches per setting, and several gains with the stronger model are small or tied. The racing audit also identifies occasional losses from pruning useful ancestors. These limitations leave uncertainty about how consistently the method helps beyond the evaluated synthetic settings.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to reviewer Nccp

We thank the reviewer for the detailed review.

Weakness 1: cost-benefit of the complete RADAR

The complete RADAR stays ahead when LLM cost is counted. RADAR does use more LLM calls at equal solver runs: it adds one diagnosis call, of at most 80 words, for about 78 of every 100 programs, and racing lets it write about twice as many programs. We therefore stopped every method, including the complete RADAR (diagnosis, racing and BAR), at 1M tokens per search with Qwen3-32B. The budget counts the input and output tokens of every LLM call, so for RADAR it covers the diagnoser as well as the writer. The other methods are run unchanged, without racing.

Table G2. Comparison at the same LLM budget with Qwen3-32B (Appendix H, pp. 25-26); held-out score, mean ±\pm SE.

Model, budget per searchTaskRADARBest other methodSolver runs (RADAR / other)
Qwen3-32B, 1M tokensCVRP-200 ↓\downarrow24.03 ±\pm 0.2224.47 ±\pm 0.22 (EoH)371 / 1154
Qwen3-32B, 1M tokensOP-200 ↑\uparrow59.19 ±\pm 0.2058.52 ±\pm 0.43 (MCTS-AHD)434 / 1284

RADAR is best on both Qwen3-32B tasks, uses fewer solver runs, and leads from a quarter of the budget onward.

On complexity, the two main parts are modular. Added unchanged to MCTS-AHD and ReEvo, racing and diagnosis improve both (Table 3c), so they can be adopted without the rest of RADAR.

Weakness 2: four searches; small or tied gains with the stronger model

With 8 searches the advantage is clearer. We doubled the searches to 8 per method on all seven tasks.

Table G1. Main comparison of Table 2a with 8 searches per method, double the 4 in the paper (Qwen3-32B; full table in Appendix M, p. 29); mean ±\pm SE. ↓\downarrow means lower is better and ↑\uparrow higher is better.

TaskRADARStrongest other method
TSP-200 ↓\downarrow11.19 ±\pm 0.0511.35 ±\pm 0.04 (Best-of-N)
CVRPTW-100 ↓\downarrow15.35 ±\pm 0.4017.39 ±\pm 0.58 (MCTS-AHD)
PFSP-500 ↓\downarrow25893 ±\pm 725953 ±\pm 15 (EoH)
CVRP-200 ↓\downarrow23.78 ±\pm 0.1224.39 ±\pm 0.31 (Best-of-N)
OP-200 ↑\uparrow58.88 ±\pm 0.2857.63 ±\pm 0.32 (MCTS-AHD)
MKP-500 ↑\uparrow104.78 ±\pm 0.18104.61 ±\pm 0.19 (MCTS-AHD)
UFLP-1000 ↓\downarrow185.22 ±\pm 0.08185.37 ±\pm 0.17 (EoH)

RADAR has the best mean on all seven tasks. It is now significantly better than the strongest other method on four tasks (OP-200, TSP-200, CVRPTW-100 and PFSP-500; Welch t-test, p<0.05p < 0.05). The small or tied gains with GPT-5.4-mini come from saturation. On TSP, MKP, UFLP and PFSP every method except MOTIF is within about 1% (TSP) or 0.3% (MKP, UFLP, PFSP) of the reference (a proven optimum or best-known solution), so no method has room to pull ahead. On the tasks that do leave room (OP, CVRP and CVRPTW), RADAR has the lowest gap to the reference, for example 3.61% vs. 6.48% for the next best method on CVRPTW (Table 2b).

Weakness 2 (continued): pruning losses, and results beyond the synthetic settings

We see no sign that racing costs quality, and RADAR's lead holds on real instances the search never saw. The best program of each CVRP-200 search in Table 2 (its champion) was run, with no further search, on the 100 real CVRPLIB Set X instances (nn = 100 to 1000, 60 s each; Appendix P, p. 32). The gap of a champion on an instance is (cost−BKS)/BKS(\text{cost} - \text{BKS}) / \text{BKS}, where BKS is the best-known solution value provided in the CVRPLIB dataset. RADAR has the lowest gap among the LLM-based methods, and the smallest spread across searches (mean ±\pm SE over the 8 searches of each method):

MethodGap (%)
RADAR6.48 ±\pm 0.74
Best-of-N8.30 ±\pm 1.36
ReEvo11.31 ±\pm 1.29
EoH12.76 ±\pm 2.15
MOTIF16.30 ±\pm 1.84
MCTS-AHD16.97 ±\pm 1.97

We see no sign on the same benchmark that pruning costs quality: champions found with racing transfer better than champions found without it. Adding racing to diagnosis lowers the gap on Set X from 10.00% to 7.34%, and adding it to the baseline with neither mechanism lowers it from 15.16% to 13.12%. The pruning losses are small. In the replay of Appendix G, the gate keeps 103 of 104 champions and 99.2% of the top-10% programs. Counting programs lost because an ancestor was stopped, the best remaining program is on average only 0.36% worse than the champion. This is a pessimistic estimate, because the replay does not count what racing gives back: the saved solver runs let the search test about twice as many programs for the same budget, which more than makes up for the loss.

AuthorsOpenReview ↗

Follow up to Reviewer Nccp

Thank you again for taking the time to review our paper. We have posted our responses along with the updated paper, and we would be grateful to hear whether they address your concerns. We are happy to discuss further or answer any additional questions.

Reviewer_izpJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RADAR, an agentic framework for automated solver design that uses runtime execution traces to guide program revisions and racing to allocate evaluation budgets. RADAR consists of three main components. First, a diagnostic agent analyzes execution traces and search history to identify a bottleneck and recommend a concrete code edit. Second, a racing mechanism evaluates candidate programs on an anchor instance and selectively promotes promising candidates for further evaluation. Third, Branch-Adaptive Reflection (BAR) adjusts the revision strategy when a search branch stagnates. The authors evaluate RADAR on seven combinatorial optimization benchmarks using Qwen3-32B and GPT-5.4-mini. The results show competitive or improved performance relative to several automated heuristic design baselines. Additional studies investigate the value of score magnitudes and reflection, isolate the contributions of racing and diagnosis, and examine the transferability of these mechanisms to existing search frameworks.

Critical strengths

1. A well-motivated and practically meaningful approach to improving revision feedback.

The paper identifies an important limitation of score-driven automated solver design: final objective scores are useful for ranking candidates but provide limited information about which code modifications should be made next. The proposed use of execution traces to identify specific runtime bottlenecks is well motivated. In particular, the distinction between ranking and revision provides a useful perspective on how execution feedback should be incorporated into LLM-based algorithm design.

2. Targeted empirical evaluation of the proposed mechanisms.

The paper provides more than an overall performance comparison. The factorial ablation of racing and diagnosis, the trace-withholding and trace-swapping controls, and the retrospective analysis of the racing gate are useful attempts to examine why the method works. The experiments also cover seven optimization tasks and multiple LLM backbones. Notably, some improvements are practically meaningful, such as the CVRP-200 results with Qwen3-32B. These findings provide encouraging evidence that execution-guided revision can improve automated solver design.

Critical weaknesses

1. Limited methodological novelty beyond existing execution-guided frameworks.

The main concern is the limited methodological novelty of the proposed framework. RADAR uses execution traces, including improvement history, stagnation duration, and invalid-output statistics, to guide code revisions, while using final scores for candidate ranking and evaluation allocation. However, the general idea of using execution feedback to direct program refinement has already been explored in prior work.

In particular, LaGO (Yu et al., 2026) explicitly separates heuristic optimization into forward evaluation, backward analysis, and program refinement, where an Analyst processes execution traces to generate structured feedback for a code-generating model. DASH (Wang et al., 2026) has also demonstrated the value of solver convergence trajectories beyond final objective scores. Although RADAR applies runtime information to more specific code-edit recommendations, its diagnostic agent mainly translates additional execution statistics into a bottleneck description and an edit instruction.

The other components also build on established techniques. Racing adapts early-stopping and resource-allocation strategies from algorithm configuration, while Branch-Adaptive Reflection relies on manually designed stagnation thresholds and predefined revision modes. Consequently, the main contribution appears to lie primarily in the integration and task-specific adaptation of existing ideas, rather than in a substantially new algorithmic or methodological mechanism. The paper does not sufficiently establish what fundamentally new capability is introduced by this particular combination.

2. Insufficient evidence for the necessity and independent contribution of the proposed mechanisms.

The experiments demonstrate that runtime feedback can be useful, but do not adequately establish the necessity of the dedicated diagnostic agent. In particular, the trace-grounding ablations compare correct, missing, and mismatched traces, but do not include a direct baseline where the same execution traces are provided to the code-writing agent without an intermediate diagnosis step. Therefore, the observed improvements may primarily result from providing richer runtime information rather than from the proposed diagnostic mechanism itself.

Moreover, Racing enables RADAR to evaluate substantially more candidate programs under the same solver-run budget, while diagnosis introduces additional LLM calls. Although Appendix H includes an approximate LLM-call-matched comparison, it does not evaluate the complete framework and does not account for token-level inference costs. This makes it difficult to separate improvements due to better revision guidance from those due to additional exploration and computational resources.

The empirical evidence for individual components is also not uniformly convincing. For example, Racing alone slightly outperforms the combined Racing-and-Diagnosis configuration on OP-200 in Table 3(a), and the main comparisons rely on only four independent searches per condition. These limitations weaken the claim that each proposed mechanism provides a substantial and reliable contribution beyond existing execution-guided search methods.

Questions for authors

Q1. What is the independent contribution of the diagnostic agent?

What additional benefit does the dedicated diagnostic agent provide beyond directly incorporating the same execution traces into the writer's prompt or an existing reflection mechanism? In particular, could the authors compare RADAR against a trace-augmented writer that receives identical runtime information but does not use a separate diagnoser? This seems essential for establishing whether the proposed diagnostic mechanism is necessary.

Q2. How does RADAR differ methodologically from existing execution-guided frameworks?

LaGO already introduces a structured evaluation-analysis-update framework in which execution traces are processed by an Analyst to guide program refinement, while DASH incorporates solver convergence dynamics into automated heuristic design. Beyond the specific runtime statistics and prompt formats employed, what is the substantive methodological distinction of RADAR? Could the same behavioral aliasing problem be addressed through a straightforward extension of these existing frameworks?

Q3. What is the algorithmic novelty of the Racing component?

Racing follows the established principle of allocating fewer evaluations to unpromising candidates. Beyond applying an anchor-based score gate to a stream of LLM-generated programs, what distinguishes the proposed mechanism from existing racing or successive-halving approaches? Could the authors explain whether its advantages persist against a straightforward adaptation of these methods under the same evaluation budget?

Q4. Do the reported improvements remain when the additional computational budget is controlled?

The main comparison matches solver runs but allows RADAR to generate more candidate programs and perform additional LLM diagnosis calls. Appendix H partially addresses this issue, but does not evaluate the complete RADAR framework under matched LLM inference costs. Could the authors provide stronger evidence that the gains arise from the proposed revision mechanism rather than additional candidate generation or inference resources?

5 comments
AuthorsOpenReview ↗

Response to reviewer izpJ (1/3)

We thank the reviewer for the detailed review.

W1, Q2: Contribution and novelty relative to LaGO and DASH

Our main novelty is the view that ranking programs and revising them need different signals, and the new results show the gain comes from the trace, not from adding a diagnosis step. Current AHD methods use the final score, or text derived from it, for both: to choose which programs to keep and to tell the LLM what to edit. We find that the score works for ranking but not for revision, because programs with the same score can need different edits. The three solvers in Fig. 1 reach the same score, but one stalls early, one is still improving at the time limit, and one produces many invalid outputs (behavioral aliasing), and score-derived reflections add no detectable value (Section 4.2). For revision, RADAR records an execution trace, and a diagnoser turns it into one edit. We also bring racing, a classic approach in algorithm configuration for spending little budget on weak candidates, to AHD with a single gate that needs no per-task threshold: a new program is tested on one training sample first, and on the others only if it scores well (Section 4.5).

Neither LaGO nor DASH takes this position. LaGO looks only at the final solutions, not at how the run went. Its Analyst module computes statistics of the final solutions and passes them to the LLM that writes the next program (LaGO, Section 3.4). So a solver that stalls early and one still improving at the time limit get the same feedback when their final solutions are alike, although they need different edits. RADAR's trace records when each improvement happened and how long the solver ran without one, so its diagnoser can tell the two apart.

DASH records how the best solution improves over time (the convergence curve), but uses it for ranking, where we find the final score is already enough: the curve is reduced to one number that breaks ties between programs with similar final scores. The paper describes showing the curve to the LLM only to tune how long each phase of a fixed solver runs (DASH, Sections 3.3 and 3.4). RADAR's diagnoser instead uses the trace to name one edit to the program. The curve also keeps only the best solution so far, so a solver that keeps producing invalid outputs looks like a stalled one. RADAR's trace records which outputs were rejected and why, so the diagnoser can tell the two apart.

The reviewer asks whether a straightforward extension of these frameworks could address behavioral aliasing. Such an extension would have to read an execution trace and return one edit to the program, which is the separation we propose. Table 3b tests whether the trace matters. Its "No trace" row is a diagnoser that reads the program and its final results but not the trace. This is similar to LaGO, whose feedback also describes the final solutions and not how the run went (LaGO's code is not public, so we could not run LaGO itself). The rows of Table 3b that matter here (Qwen3-32B, racing and BAR off, held-out score):

What the diagnoser readsOP-200 ↑\uparrowCVRP-200 ↓\downarrowUFLP-1000 ↓\downarrow
Score-only (no diagnoser)56.9626.24187.71
No trace (program and final results)57.6125.14187.82
Trace (program and the parent's execution trace)58.0924.23186.32

With the trace the diagnoser is best on all three tasks. Without the trace it recovers only part of the gain on OP and CVRP and none on UFLP. So most of the benefit depends on what the diagnoser reads, not only on adding a diagnosis step. For DASH, we tried our best but could not find a public implementation, and our reimplementation could not reproduce its results.

W2, Q1: Does the separate diagnoser matter?

Yes: given the same information, the separate diagnoser beats a single writer call on all three tasks. We ran the comparison the reviewer suggested (Qwen3-32B, racing and BAR off; Appendix F, p. 21). The direct trace writer removes the diagnoser: a single LLM call receives the trace, and also the search history the diagnoser would have read, and writes the program. Held-out scores, mean ±\pm SE over four searches:

TaskSeparate diagnoser (RADAR)Direct trace writer
CVRP-200 ↓\downarrow24.23 ±\pm 0.3924.52 ±\pm 0.49
OP-200 ↑\uparrow58.09 ±\pm 0.9857.77 ±\pm 0.29
UFLP-1000 ↓\downarrow186.32 ±\pm 0.44188.17 ±\pm 1.54

The separate diagnoser and the direct trace writer receive the same information, and the separate diagnoser is ahead on all three tasks. We think this is a division of labor: the diagnoser only has to read the trace and commit to one change, and the writer only has to implement it, while a single call must do both.

AuthorsOpenReview ↗

Response to reviewer izpJ (2/3)

W2: Only four searches per condition

We doubled the searches to 8 per method on all seven tasks, and the conclusions hold.

Table G1. Main comparison of Table 2a with 8 searches per method, double the 4 in the paper (Qwen3-32B; full table in Appendix M, p. 29); mean ±\pm SE. ↓\downarrow means lower is better and ↑\uparrow higher is better.

TaskRADARStrongest other method
TSP-200 ↓\downarrow11.19 ±\pm 0.0511.35 ±\pm 0.04 (Best-of-N)
CVRPTW-100 ↓\downarrow15.35 ±\pm 0.4017.39 ±\pm 0.58 (MCTS-AHD)
PFSP-500 ↓\downarrow25893 ±\pm 725953 ±\pm 15 (EoH)
CVRP-200 ↓\downarrow23.78 ±\pm 0.1224.39 ±\pm 0.31 (Best-of-N)
OP-200 ↑\uparrow58.88 ±\pm 0.2857.63 ±\pm 0.32 (MCTS-AHD)
MKP-500 ↑\uparrow104.78 ±\pm 0.18104.61 ±\pm 0.19 (MCTS-AHD)
UFLP-1000 ↓\downarrow185.22 ±\pm 0.08185.37 ±\pm 0.17 (EoH)

RADAR has the best mean on all seven tasks. It is now significantly better than the strongest other method on four tasks (OP-200, TSP-200, CVRPTW-100 and PFSP-500; Welch t-test, p<0.05p < 0.05).

Q3: Algorithmic novelty of racing

Our gate needs no per-task threshold and beats standard successive halving on both tested tasks. Our contribution here is the observation from our answer to W1, Q2: the final score is a reliable signal for ranking, and one run is enough to obtain it (Section 4.5). Current AHD methods do not act on this. The AHD baselines we evaluate run every program on all training samples. AlphaEvolve uses an evaluation cascade whose thresholds the user sets by hand for each task, and predictor-based methods (Hercules-P, NeRM) skip runs by predicting performance, which their own ablations show costs solution quality. RADAR instead uses one online empirical-quantile gate: each new program is compared with the programs before it, so no per-task score threshold is needed and the same setting is shared by all seven tasks. The solver runs this saves go to testing more programs.

The gate itself belongs to the racing and successive-halving family (Section 2), adapted to AHD, where programs arrive one at a time: each new program runs once on the anchor sample and is tested on the other four training samples only if its score is in the top 40% of the programs before it.

We thank the reviewer for suggesting successive halving. We ran it (1, 2, then 5 samples, keeping the top half at each stage) with the complete RADAR at the same 500 solver runs, on the same instances as Table 2 in the paper (four searches each):

TaskOur gateSuccessive halvingNo racing
CVRP-200 ↓\downarrow23.9825.1924.83
OP-200 ↑\uparrow58.9158.7358.01

Our gate has the best mean on both tasks. Successive halving beats no racing on OP-200 but not on CVRP-200. A likely reason is that its middle stage costs solver runs without adding much, because one anchor sample already ranks programs reliably (Section 4.5): 20 to 27 programs per search are stopped only after two runs. It fully tests fewer programs than our gate (62 vs. 77 per search on CVRP-200, 67 vs. 79 on OP-200), and only fully tested programs can become parents. This is now in Appendix G (pp. 23-24).

Q4, W2: Gains when LLM inference cost is controlled

Yes: at the same LLM budget the complete RADAR is best on both Qwen3-32B tasks. We reran the comparison with the complete RADAR and stopped every method at 1M tokens per search with Qwen3-32B. The budget counts the input and output tokens of every LLM call, so for RADAR it covers the diagnoser as well as the writer. The other methods are run unchanged, without racing.

Table G2. Comparison at the same LLM budget with Qwen3-32B (Appendix H, pp. 25-26); held-out score, mean ±\pm SE.

Model, budget per searchTaskRADARBest other methodSolver runs (RADAR / other)
Qwen3-32B, 1M tokensCVRP-200 ↓\downarrow24.03 ±\pm 0.2224.47 ±\pm 0.22 (EoH)371 / 1154
Qwen3-32B, 1M tokensOP-200 ↑\uparrow59.19 ±\pm 0.2058.52 ±\pm 0.43 (MCTS-AHD)434 / 1284

RADAR is best on both Qwen3-32B tasks and is ahead from a quarter of the budget onward. It also needs fewer solver runs than the best other method, largely because racing stops weak programs early. So extra resources do not explain the gain.

AuthorsOpenReview ↗

Response to reviewer izpJ (3/3)

W2: Racing only vs. racing + diagnosis on OP-200

The two rows of Table 3a are tied on OP-200, and the combination is best on the other two tasks. Against the baseline with neither mechanism (56.96 on OP-200), racing only adds 1.77 and racing with diagnosis adds 1.61, and the 0.16 between them is smaller than the standard error of either row (0.25 and 0.36). On CVRP-200 and UFLP-1000 the combination is the best of the four rows in Table 3a. Racing does especially well on OP because the gate stops almost no good program there (it keeps 239 of the 240 top-10% OP programs; Appendix G), so diagnosis has less left to add on this task.

W1: BAR relies on manually designed thresholds

BAR's thresholds were set once and shared by all seven tasks, and BAR beats threshold-free periodic and random switching on all three tested tasks. BAR comes from a pattern we saw repeatedly in our early searches: a branch's best program stops improving, yet the next edits keep refining the same design (example in Appendix I).

Existing AHD feedback does not see this. ReEvo's reflection compares two programs and which one scored better, and MCTS-AHD and EoH apply their edit types by fixed rules, so none of them checks whether the branch being edited has gone many evaluations without a gain. The search history does record it, and BAR reads it. When a branch stalls, the diagnoser is asked for a different kind of edit, a parameter change (Repair) or one replaced component (Rethink), in place of another refinement. This is the paper's main idea applied to the search itself: the next edit is decided from a record of how the process ran, and not from the score alone.

The rule is a simple heuristic. Its thresholds were set once from those early searches and are shared by all seven tasks, and BAR improves the mean on six of the seven (Appendix I). We also tested switching to Repair or Rethink at the same rate, but on a fixed period or at random, which needs no thresholds. BAR has the best mean on all three tasks. Held-out scores (BAR is the Table 2 result in the paper; Appendix I, p. 26):

TaskBARPeriodicRandom
CVRP-200 ↓\downarrow23.9824.3924.79
CVRPTW-100 ↓\downarrow15.3916.2115.76
OP-200 ↑\uparrow58.9158.7258.33

The three modes (Refine, Repair, Rethink) are few and general on purpose: they only set how large an edit the diagnoser may request, and the edit itself still comes from the trace.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their detailed reviews. They found the split between ranking and revision useful and well motivated (izpJ, Nccp, Mg3P), the targeted ablations informative (izpJ), and the mechanisms useful when added to existing methods (Nccp). The shared concerns were the number of searches, matched LLM cost, and novelty relative to LaGO and DASH. We ran 8 new sets of experiments and have added them to the updated paper (new text in blue). Key findings:

  1. The lead holds with 8 searches. RADAR has the best mean on 7/7 tasks and is significantly better on 4. (Table G1; Appendix M, p. 29)
  2. The lead holds at matched LLM cost, with fewer solver runs. With Qwen3-32B, RADAR is best on both tasks while using about one third of the solver runs. (Table G2; Appendix H, pp. 25-26)
  3. The trace matters: the same diagnoser does better with the trace than without it on 3/3 tasks. A diagnoser without the trace (LaGO-style) recovers only part of the gain, and one writer call given the same trace is worse than a separate diagnoser on 3/3 tasks. (izpJ W1, W2; Appendix F, p. 21)
  4. Each mechanism beats its standard alternative. Our racing gate beats successive halving (2/2 tasks), and BAR beats periodic and random switching (3/3). (izpJ Q3, W1; Appendix G, pp. 23-24, and Appendix I, p. 26)
  5. Transfer to real instances. On 100 CVRPLIB Set X instances RADAR's champions have the lowest gap (6.48% vs. 8.30% for the second best), and racing lowers the gap in all 3 comparisons. (Nccp, Mg3P; Appendix P, p. 32)
  6. Racing is largely robust to the anchor and keep fraction, including on mixed instance types. (Mg3P W3; Appendix G, p. 24)

Below we detail findings 1 and 2, which address concerns raised by all three reviewers; the others are in the individual replies.

General Response 1: only four searches per method (Table G1)

Comments addressed:

  • Reviewer izpJ, W2: "the main comparisons rely on only four independent searches per condition"
  • Reviewer Nccp, Weakness 2: "Main comparisons use four searches per setting"
  • Reviewer Mg3P, W2 and Q2: "The main experiments use only four independent searches per configuration"

Table G1: main comparison with 8 searches per method, double the 4 in the paper. This repeats Table 2a of the paper (Qwen3-32B; full table in Appendix M, p. 29); mean ±\pm SE. ↓\downarrow means lower is better and ↑\uparrow higher is better.

TaskRADARStrongest other method
TSP-200 ↓\downarrow11.19 ±\pm 0.0511.35 ±\pm 0.04 (Best-of-N)
CVRPTW-100 ↓\downarrow15.35 ±\pm 0.4017.39 ±\pm 0.58 (MCTS-AHD)
PFSP-500 ↓\downarrow25893 ±\pm 725953 ±\pm 15 (EoH)
CVRP-200 ↓\downarrow23.78 ±\pm 0.1224.39 ±\pm 0.31 (Best-of-N)
OP-200 ↑\uparrow58.88 ±\pm 0.2857.63 ±\pm 0.32 (MCTS-AHD)
MKP-500 ↑\uparrow104.78 ±\pm 0.18104.61 ±\pm 0.19 (MCTS-AHD)
UFLP-1000 ↓\downarrow185.22 ±\pm 0.08185.37 ±\pm 0.17 (EoH)

Conclusion: The result of Table 2a holds with twice as many searches: RADAR still has the best mean on all seven tasks, and it is now significantly better than the strongest other method on four tasks (OP-200, TSP-200, CVRPTW-100 and PFSP-500; Welch t-test, p<0.05p < 0.05).

General Response 2: LLM cost is not matched (Table G2)

Comments addressed:

  • Reviewer izpJ, W2 and Q4: Appendix H "does not evaluate the complete framework and does not account for token-level inference costs"
  • Reviewer Nccp, Weakness 1: "Main comparisons match solver runs, while RADAR uses more LLM calls."
  • Reviewer Mg3P, W1 and Q1: "the main comparisons do not match LLM inference costs"

Table G2: comparison at the same LLM budget with Qwen3-32B (Appendix H, pp. 25-26). Every method, including the complete RADAR (diagnosis, racing and BAR), was stopped at 1M tokens per search. The budget counts the input and output tokens of every LLM call, so for RADAR it covers the diagnoser as well as the writer. The other methods are run unchanged, without racing. Held-out score, mean ±\pm SE:

Model, budget per searchTaskRADARBest other methodSolver runs (RADAR / other)
Qwen3-32B, 1M tokensCVRP-200 ↓\downarrow24.03 ±\pm 0.2224.47 ±\pm 0.22 (EoH)371 / 1154
Qwen3-32B, 1M tokensOP-200 ↑\uparrow59.19 ±\pm 0.2058.52 ±\pm 0.43 (MCTS-AHD)434 / 1284

Conclusion: RADAR is best on both Qwen3-32B tasks. It also needs fewer solver runs than the best other method, largely because racing stops weak programs early. So extra LLM calls or extra resources do not explain the gain.

Full tables, with every method at intermediate budgets, are in Appendix H (pp. 25-26).

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

AuthorsOpenReview ↗

Final Author Response

Dear AC and Reviewers,

We thank the AC and the reviewers for their time and careful reviews, which have made the paper better.

Our paper proposes RADAR, an Automated Heuristic Design (AHD) method built on one observation: the final score is a good signal for choosing which programs to keep, but a poor signal for deciding how to edit them, because programs with the same score can fail in different ways. RADAR keeps the score for ranking and uses an execution trace for revision. A simple racing gate stops weak programs early to save solver runs.

Two of the three reviewers recommended acceptance from the start, and after the discussion all three now do (3/3/3). All three reviewers found the split between ranking and revision well motivated and useful (izpJ, Nccp, Mg3P). Reviewers izpJ and Mg3P also found the ablations and racing analysis informative, and Reviewer Nccp noted that our mechanisms improve existing search controllers when added to them.

The reviewers asked for more experiments, and we added them to the updated paper (marked in blue):

  1. All three reviewers asked for more searches per method, so we doubled them to 8 on all seven tasks. RADAR has the best mean on all seven and is significantly better than the strongest other method on four (Appendix M).
  2. All three reviewers also asked for a comparison at matched LLM cost. We stopped every method, including the complete RADAR, at the same budget of 1M tokens per search with Qwen3-32B, and RADAR is best on both tested tasks while using about a third of the solver runs (Appendix H).
  3. Reviewers izpJ and Mg3P asked how RADAR differs from LaGO and DASH. LaGO's feedback describes only the final solutions, and DASH uses the convergence curve mainly to rank programs. Neither turns a record of how the run went into a concrete edit to the program. In our experiments, the same diagnoser does better with the trace than without it on all three tasks (Appendix F).
  4. Reviewer izpJ asked whether the separate diagnoser is needed. We compared it with the direct trace writer, which receives the same trace and search history but writes the program in a single LLM call. The separate diagnoser is ahead of the direct trace writer on all three tasks (Appendix F).
  5. Reviewer izpJ also asked how racing and BAR compare with standard alternatives. Our racing gate is ahead of successive halving on both tested tasks, and BAR is ahead of periodic and random switching on all three (Appendices G and I).
  6. Reviewers Nccp and Mg3P asked about results beyond the synthetic settings. On 100 real CVRPLIB Set X instances, RADAR's champions have the lowest gap among the LLM-based methods (6.48% vs. 8.30% for the next best), and racing lowers the gap in all three comparisons (Appendix P). On the pruning losses Reviewer Nccp raised, the gate keeps 103 of 104 champions when we replay it on 104 searches without racing (Appendix G).
  7. Reviewer Mg3P asked how sensitive racing is to the anchor sample and keep fraction. Racing is largely robust to both: on mixed instance types it keeps every top-10% program and every champion for any anchor geometry and for keep fractions from 25% to 60%, and we added a recommended setting for instance sets with very different sizes (Appendix G).

After the discussion, Reviewer izpJ raised their score from 2 to 3, noting that the new experiments addressed their main empirical concerns and made the overall contribution more convincing. Reviewer Mg3P confirmed that the new results address their main concerns and kept their acceptance rating. We thank Reviewers izpJ and Mg3P for engaging with the rebuttal.

We have not yet heard back from Reviewer Nccp. Reviewer Nccp's main points, the number of searches and matched LLM cost, are the same ones that Reviewers izpJ and Mg3P now consider addressed. We would be grateful if Reviewer Nccp could take a look at our response when they have time, and we are happy to clarify anything.

We thank the AC and the reviewers again for their time and insightful comments.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.