- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies the bottleneck part, iterative refinement of solvers, of LLM-based AHD. The authors observe the "feasibility oscillation," referring to the feasibility drop in the solver's performance after revision. To address the issue, the paper proposes DiSH, a harness to adaptively control the revision scope and direction based on the selected failure cases. By evaluating on nine problems, the paper reports higher QYI than HeuriGym and five population-based AHD methods, with fewer token consumptions
Critical strengths
The motivation is clear, and feasibility oscillation is a sharp, well-illustrated observation
The method is simple and interpretable, and the prompts are fully documented. The ablation study is effective.
Critical weaknesses
1. The problem needs further verification
The problem is observed based on the collected trajectories using mid-tier and older models. Given progress in LLMs, especially their coding ability, is the problem already partially or fully solved? We see the same trend in Table 2, where the gap between baselines and DiSH shrinks as models get more powerful. The author should verify the problem with at least one of the latest models, like GPT-luna-6 or DeepSeekv4.
2. Baseline adaptation narrows the effectiveness.
Only HeuriGym is a comparable baseline; DiSH is the HeuriGym loop with a different prompt. Unlike generating the full solving code, other baselines are trying to find better heuristics within a given skeleton (usually large neighborhood search). Adapting them to the full solver refinement setting rather than applying DiSH to them limits the paper's scope.
3. The claimed contribution is not directly measured against the baselines.
The central claim is reduced feasibility oscillation. However, the paper reports only QYI, which mixes feasibility with the objective. Moreover, “feasibility” here includes crashes and timeouts. Timeouts are the largest failure category (about 35% of all records), and Stage II failures are negligible. The largest gain is on TSP (+34.9 over HeuriGym), where producing a valid tour is trivial, so baseline TSP Yield of 40–52% likely reflects timeouts on test instances. That is a different phenomenon from the coupled-constraint failures the paper uses as motivation.
4. Gains are concentrated in a few settings.
TSP, Crew, and E-Graph account for about 70% of DiSH-CRS’s gain over HeuriGym. On the other six problems, the average gain over the per-problem best baseline is about +2 QYI. Figure 15 shows further concentration. On Operator Scheduling, all three variants are below HeuriGym on 3 of 4 backbones. DiSH-GRS’s E-Graph gain comes from a single backbone, and the CVRP gains appear only on Qwen and DeepSeek. Gains are smallest on the strongest backbone (Gemini-2.5-Flash: DiSH-GRS +1.3 QYI over HeuriGym). Quality there is below EoH for all three variants. No significance tests are reported.
Questions for authors
- How was each AHD baseline adapted to the complete-solver setting, and what do their prompts look like?
- Which solver is reported for each method? DiSH reports the best solver found during refinement. Do HeuriGym and the other baselines also report their best solver, or their last one? Also, do HeuriGym and the ablation variants use the same system prompt as DiSH?
- How often does a previously feasible instance become infeasible after a revision, for every method and every backbone? Table 3 currently shows this kind of result only against HeuriGym.
- What kind of failures does DiSH fix? For each method, how many test failures are timeouts, crashes, invalid outputs, or constraint violations? This matters most for TSP, where producing a valid tour should be easy.
- Does the method still help with a current frontier model? Could you compare DiSH with HeuriGym using one recent frontier model, even on a few problems? In the current results, the gains are smallest on the strongest backbone.
13 comments
All six baselines show feasibility losses on every benchmark with both models. In all six task–model settings, every DiSH variant has a lower observed oscillation rate than every baseline. No losses were observed for DiSH with GPT-6-Luna under this metric (Table 3). These zero rates describe the evaluated runs and do not establish that future revisions will always preserve feasibility.
The results support the continued value of DiSH’s guidance on what to change and how extensively to revise the solver, even with the two recent models tested here.
Weakness 2
Only HeuriGym is a comparable baseline; DiSH is the HeuriGym loop with a different prompt. Unlike generating the full solving code, other baselines are trying to find better heuristics within a given skeleton (usually large neighborhood search). Adapting them to the full solver refinement setting rather than applying DiSH to them limits the paper's scope.
Response: We agree that HeuriGym is the closest baseline. Our focus is how execution feedback should guide successive revisions of a complete solver. DiSH addresses this problem through explicit rules for selecting feedback and adjusting the requested extent of code changes.
1. The role of the baseline comparisons.
Generating a heuristic component within a fixed algorithm and refining a complete solver are different settings. We study the latter, where a revision must address unresolved failures while preserving behavior that already works. How to use execution and verification feedback to manage these competing demands is an important research question within automatic heuristic design.
HeuriGym provides the direct comparison in this setting. For EoH, ReEvo, HSEvo, HiFo-Prompt, and RefineEvo, we adapt the generation target to complete solvers while preserving their core search procedures, as described in Appendix E.2, Table 7 of the original paper and Table 11 of this response. These comparisons assess alternative search strategies under a shared task and evaluation protocol. Their conclusions apply to the adapted complete-solver setting.
2. What DiSH adds to HeuriGym.
DiSH makes two decisions after each evaluation (Figure 2 of the original paper):
- What to revise: It selects representative feedback from failed and successful instances to identify problems to address and behavior to preserve.
- How extensively to revise: It adjusts the requested scope of changes according to the current demonstration feasibility rate. It allows broader changes when feasibility is low and progressively narrows them as feasibility improves.
These decisions are computed by the harness and communicated through prompts. They are updated as the solver’s behavior changes. The contribution is therefore the feedback selection and revision control process implemented through those prompts.
This design addresses a specific difficulty in complete-solver refinement: correcting a failure on one instance can disrupt previously successful behavior on another. Across nine benchmarks and four LLMs, the experiments show improvements in overall performance and refinement stability (Table 3 and Figure 8 of the original paper). For example, DiSH-CRS achieves an average QYI of 80.3, compared with 67.4 for HeuriGym (Table 2 of the original paper).
3. Extensions to other search frameworks.
DiSH’s feedback selection and revision guidance could also be incorporated into candidate generation in EoH or ReEvo, while retaining their population search, evolutionary operations, and reflection mechanisms. Such integration would test whether these controls complement broader search procedures.
We have not evaluated that integration in this work. The current experiments support DiSH’s effectiveness for iterative refinement of complete solvers; extending and testing its controls within other AHD frameworks is a further research direction.
Weakness 3
The central claim is reduced feasibility oscillation. However, the paper reports only QYI, which mixes feasibility with the objective. Moreover, “feasibility” here includes crashes and timeouts. Timeouts are the largest failure category (about 35%of all records), and Stage II failures are negligible. The largest gain is on TSP (+34.9 over HeuriGym), where producing a valid tour is trivial, so baseline TSP Yield of 40–52% likely reflects timeouts on test instances. That is a different phenomenon from the coupled-constraint failures the paper uses as motivation.
Response: We agree that the large TSP improvement mainly reflects fewer timeouts and should be distinguished from better constraint satisfaction (Table 14). In our benchmark, feasibility requires successful execution within the time limit and a valid solution that satisfies the constraints. We assess refinement stability separately from QYI and report failure types to clarify the sources of improvement.
1. Refinement stability is measured separately from QYI.
We follow HeuriGym in using QYI to assess Quality and Yield. Appendix D.3 of the original paper also defines two stability metrics:
- Convergence Rate: the proportion of runs that solve every demonstration instance at the final iteration.
- Survival Rate: among runs that solve every demonstration instance at least once, the proportion that maintain full feasibility from its first attainment through the end of refinement.
Both metrics appear in the original ablation study (Table 3 of the original paper). Table 4 extends the comparison.
Table 4. Feasibility stability metrics (%, higher is better).
| Method | Convergence Rate ↑ | Survival Rate ↑ |
|---|---|---|
| HeuriGym | 55.2 | 40.0 |
| DiSH without scope control | 55.6 | 46.6 |
| DiSH without direction control | 91.4 | 92.3 |
| DiSH-GWRS | 88.3 | 93.0 |
| DiSH-CRS | 92.9 | 94.2 |
| DiSH-GRS | 91.4 | 91.9 |
All three DiSH variants exceed HeuriGym on both measures. DiSH-CRS increases Convergence Rate from 55.2% to 92.9% and Survival Rate from 40.0% to 94.2%, gains of 37.7 and 54.2 percentage points. The configuration without scope control records substantially lower rates. These results show improved attainment and preservation of full demonstration feasibility, measured without the solution-quality component of QYI.
2. Timeouts count as unsuccessful evaluations under a shared time budget.
A timeout can reflect high computational cost or a failure to terminate, such as an infinite loop. It is recorded separately from a constraint violation. This distinction matters on TSP, where returning a valid tour may be straightforward, but the generated solver must still finish within the time limit.
Each benchmark has its own execution limit, which is identical across methods. For example, Intra includes instances with tens of thousands of nodes and uses a 60-second limit. TSP likewise uses the same benchmark-specific limit for every method.
HeuriGym evaluates complete solvers under time and memory limits and records timeouts as execution failures. The other baseline implementations used in our comparison also account for timeouts. EoH and HiFo-Prompt assign no valid objective value to timed-out candidates and exclude them from population selection. ReEvo and HSEvo mark them as execution failures, assign an objective value of , and exclude them from parent selection. RefineEvo assigns no valid objective value and records timeouts separately.
3. Improvements also occur in crashes and constraint violations.
Table 5 provides examples where timeout reduction does not explain the main gains.
On Operator Scheduling, HeuriGym’s timeout rate is 1.02%, while constraint violations account for 5.85% of evaluations. DiSH-GWRS and DiSH-CRS both achieve 100% observed success, with no observed constraint violations.
On E-Graph, DiSH-CRS reduces crashes from 29.41% to 8.33% and constraint violations from 12.04% to 8.60%. Its success rate rises from 53.50% to 78.17%, while the timeout rate remains nearly unchanged (5.04% versus 4.89%). These comparisons show improvements in execution reliability and constraint satisfaction beyond timeout reduction.
Table 5. Reported test outcomes on two held-out benchmarks. Values are percentages of all test evaluations. Best values in each column are bolded.
(a) Operator Scheduling
| Method | Success ↑ | Timeout ↓ | Crash ↓ | Constraint violation ↓ |
|---|---|---|---|---|
| HeuriGym | 93.13 | 1.02 | 0.00 | 5.85 |
| EoH | 84.80 | 3.95 | 0.00 | 11.26 |
| ReEvo | 66.08 | 2.63 | 2.78 | 28.51 |
| HSEvo | 64.33 | 0.88 | 2.78 | 32.02 |
| HiFo-Prompt | 78.51 | 4.82 | 0.15 | 16.52 |
| RefineEvo | 64.77 | 6.29 | 3.51 | 25.44 |
| DiSH-GWRS | 100.00 | 0.00 | 0.00 | 0.00 |
| DiSH-CRS | 100.00 | 0.00 | 0.00 | 0.00 |
| DiSH-GRS | 97.22 | 0.00 | 0.00 | 2.78 |
(b) E-Graph
| Method | Success ↑ | Timeout ↓ | Crash ↓ | Constraint violation ↓ |
|---|---|---|---|---|
| HeuriGym | 53.50 | 5.04 | 29.41 | 12.04 |
| EoH | 42.04 | 5.03 | 0.00 | 52.93 |
| ReEvo | 26.80 | 1.63 | 8.57 | 62.99 |
| HSEvo | 40.00 | 3.27 | 2.86 | 53.88 |
| HiFo-Prompt | 26.94 | 8.16 | 0.41 | 64.49 |
| RefineEvo | 43.67 | 2.31 | 0.00 | 54.01 |
| DiSH-GWRS | 57.69 | 3.67 | 30.75 | 7.89 |
| DiSH-CRS | 78.17 | 4.89 | 8.33 | 8.60 |
| DiSH-GRS | 66.53 | 1.63 | 14.29 | 17.55 |
These analyses address different aspects of reliability. The trajectory metrics measure whether full demonstration feasibility is attained and preserved. The test breakdown identifies the types of failures observed on unseen instances.
Weakness 1
Response: We added experiments with Gemini 3.1 Flash-Lite and GPT-6-Luna on Protein Design, TSP, and CVRP. We also reanalyzed the original HeuriGym and DiSH runs on these tasks with Qwen3.5-122B-A10B, DeepSeek-V3, and GPT-4.1-mini. This gives a comparison across five models using the same measure of feasibility loss. The results show that feasibility losses persist with both recent models, while DiSH maintains lower observed rates.
1. Feasibility oscillation persists with newer LLM backbones..
We use the same rule to measure feasibility oscillation for every method and model. For each newly evaluated candidate, the reference is the previously evaluated solver with the highest demonstration QYI. We count one event if the candidate fails on at least one demonstration instance solved by the reference, even if the total number of solved instances remains unchanged.
The rate is the proportion of such events among evaluations whose reference solves at least one instance. The reference need not be the candidate’s parent, so this rule applies to both sequential refinement and population-based search.
Table 1. Comparison across five models. Each cell reports mean feasibility oscillation rate / mean Test QYI (%), averaged across Protein Design, TSP, and CVRP.
HeuriGym’s mean Test QYI is higher with Gemini 3.1 Flash-Lite and GPT-6-Luna than with GPT-4.1-mini. However, its mean oscillation rate is also higher: 37.36% and 36.11%, respectively, compared with 21.27%. Thus, better final solver performance does not necessarily imply more stable refinement. Across all five models, every DiSH variant achieves a lower mean oscillation rate and higher mean Test QYI than HeuriGym (Table 1).
2. DiSH retains lower feasibility loss rates in the broader baseline comparison.
We also compared all six baselines and three DiSH variants using the two recent models. Table 2 and Table 3 report the results for each benchmark.
Table 2. Test QYI and feasibility oscillation rates (%) with Gemini 3.1 Flash-Lite.
Table 3. Test QYI and feasibility oscillation rates (%) with GPT-6-Luna.