acceptodds
Under review as a conference paper at ICLR 2027

Harnessing LLMs with Direction-Scope Control for Combinatorial Solver Refinement

Abstract

Harnesses are increasingly unlocking the potential of large language model (LLM) agents by making their actions more controllable. In automatic heuristic design (AHD) for combinatorial optimization (CO), LLMs iteratively revise solvers using execution and verification feedback. However, these revisions often break previously validated solver behavior, making refinement unstable and costly. We characterize this pattern as *feasibility oscillation* and document it across CO benchmarks, AHD methods, and backbone LLMs. Tracing this oscillation, our analysis reveals that feasibility degradation often follows large solver revisions, while substantial feedback redundancy makes it harder to identify which failures deserve attention. Motivated by these findings, we introduce DiSH, a harness that jointly controls revision direction and scope. It selects representative failure evidence to guide each update and progressively narrows the revision scope as feasible behavior accumulates. Across nine CO benchmarks, DiSH consistently improves QYI scores and reduces feasibility oscillation, while roughly halving feedback token consumption relative to prior methods that use full feedback. These results establish DiSH as a strong baseline for solver refinement under harness control.

Primary area:
optimization
Keywords:
Combinatorial Optimization, Automatic Heuristic Design, Harness, Feasibility Oscillation, Revision Direction and Scope, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3AMnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the bottleneck part, iterative refinement of solvers, of LLM-based AHD. The authors observe the "feasibility oscillation," referring to the feasibility drop in the solver's performance after revision. To address the issue, the paper proposes DiSH, a harness to adaptively control the revision scope and direction based on the selected failure cases. By evaluating on nine problems, the paper reports higher QYI than HeuriGym and five population-based AHD methods, with fewer token consumptions

Critical strengths

The motivation is clear, and feasibility oscillation is a sharp, well-illustrated observation

The method is simple and interpretable, and the prompts are fully documented. The ablation study is effective.

Critical weaknesses

1. The problem needs further verification

The problem is observed based on the collected trajectories using mid-tier and older models. Given progress in LLMs, especially their coding ability, is the problem already partially or fully solved? We see the same trend in Table 2, where the gap between baselines and DiSH shrinks as models get more powerful. The author should verify the problem with at least one of the latest models, like GPT-luna-6 or DeepSeekv4.

2. Baseline adaptation narrows the effectiveness.

Only HeuriGym is a comparable baseline; DiSH is the HeuriGym loop with a different prompt. Unlike generating the full solving code, other baselines are trying to find better heuristics within a given skeleton (usually large neighborhood search). Adapting them to the full solver refinement setting rather than applying DiSH to them limits the paper's scope.

3. The claimed contribution is not directly measured against the baselines.

The central claim is reduced feasibility oscillation. However, the paper reports only QYI, which mixes feasibility with the objective. Moreover, “feasibility” here includes crashes and timeouts. Timeouts are the largest failure category (about 35% of all records), and Stage II failures are negligible. The largest gain is on TSP (+34.9 over HeuriGym), where producing a valid tour is trivial, so baseline TSP Yield of 40–52% likely reflects timeouts on test instances. That is a different phenomenon from the coupled-constraint failures the paper uses as motivation.

4. Gains are concentrated in a few settings.

TSP, Crew, and E-Graph account for about 70% of DiSH-CRS’s gain over HeuriGym. On the other six problems, the average gain over the per-problem best baseline is about +2 QYI. Figure 15 shows further concentration. On Operator Scheduling, all three variants are below HeuriGym on 3 of 4 backbones. DiSH-GRS’s E-Graph gain comes from a single backbone, and the CVRP gains appear only on Qwen and DeepSeek. Gains are smallest on the strongest backbone (Gemini-2.5-Flash: DiSH-GRS +1.3 QYI over HeuriGym). Quality there is below EoH for all three variants. No significance tests are reported.

Questions for authors
  1. How was each AHD baseline adapted to the complete-solver setting, and what do their prompts look like?
  2. Which solver is reported for each method? DiSH reports the best solver found during refinement. Do HeuriGym and the other baselines also report their best solver, or their last one? Also, do HeuriGym and the ablation variants use the same system prompt as DiSH?
  3. How often does a previously feasible instance become infeasible after a revision, for every method and every backbone? Table 3 currently shows this kind of result only against HeuriGym.
  4. What kind of failures does DiSH fix? For each method, how many test failures are timeouts, crashes, invalid outputs, or constraint violations? This matters most for TSP, where producing a valid tour should be easy.
  5. Does the method still help with a current frontier model? Could you compare DiSH with HeuriGym using one recent frontier model, even on a few problems? In the current results, the gains are smallest on the strongest backbone.
13 comments

Weakness 1

The problem is observed based on the collected trajectories using mid-tier and older models. Given progress in LLMs, especially their coding ability, is the problem already partially or fully solved? We see the same trend in Table 2, where the gap between baselines and DiSH shrinks as models get more powerful. The author should verify the problem with at least one of the latest models, like GPT-luna-6 or DeepSeekv4.

Response: We added experiments with Gemini 3.1 Flash-Lite and GPT-6-Luna on Protein Design, TSP, and CVRP. We also reanalyzed the original HeuriGym and DiSH runs on these tasks with Qwen3.5-122B-A10B, DeepSeek-V3, and GPT-4.1-mini. This gives a comparison across five models using the same measure of feasibility loss. The results show that feasibility losses persist with both recent models, while DiSH maintains lower observed rates.

1. Feasibility oscillation persists with newer LLM backbones..

We use the same rule to measure feasibility oscillation for every method and model. For each newly evaluated candidate, the reference is the previously evaluated solver with the highest demonstration QYI. We count one event if the candidate fails on at least one demonstration instance solved by the reference, even if the total number of solved instances remains unchanged.

The rate is the proportion of such events among evaluations whose reference solves at least one instance. The reference need not be the candidate’s parent, so this rule applies to both sequential refinement and population-based search.

Table 1. Comparison across five models. Each cell reports mean feasibility oscillation rate / mean Test QYI (%), averaged across Protein Design, TSP, and CVRP.

MethodQwen3.5-122B-A10BDeepSeek-V3GPT-4.1-miniGemini 3.1 Flash-LiteGPT-6-Luna
HeuriGym56.71 / 58.2530.38 / 65.2521.27 / 72.0437.36 / 75.1036.11 / 76.61
DiSH-GWRS11.67 / 84.240.00 / 83.721.19 / 87.995.37 / 89.430.00 / 90.97
DiSH-CRS9.29 / 82.122.39 / 87.860.00 / 88.351.52 / 87.510.00 / 90.40
DiSH-GRS13.60 / 84.975.73 / 82.680.00 / 84.865.03 / 89.420.00 / 91.44

HeuriGym’s mean Test QYI is higher with Gemini 3.1 Flash-Lite and GPT-6-Luna than with GPT-4.1-mini. However, its mean oscillation rate is also higher: 37.36% and 36.11%, respectively, compared with 21.27%. Thus, better final solver performance does not necessarily imply more stable refinement. Across all five models, every DiSH variant achieves a lower mean oscillation rate and higher mean Test QYI than HeuriGym (Table 1).

2. DiSH retains lower feasibility loss rates in the broader baseline comparison.

We also compared all six baselines and three DiSH variants using the two recent models. Table 2 and Table 3 report the results for each benchmark.

Table 2. Test QYI and feasibility oscillation rates (%) with Gemini 3.1 Flash-Lite.

MethodProtein QYI ↑Protein Oscillation ↓TSP QYI ↑TSP Oscillation ↓CVRP QYI ↑CVRP Oscillation ↓
HeuriGym78.0627.6060.5940.7086.6443.80
EoH72.5366.7041.5958.8054.6561.10
ReEvo66.6753.3013.7891.7084.7285.70
HSEvo43.7233.3016.3791.7077.1773.30
HiFo-Prompt67.0950.0019.8461.1073.6558.10
RefineEvo66.0835.0042.51100.0078.4473.30
DiSH-GWRS80.595.0095.7211.1092.000.00
DiSH-CRS78.530.0092.004.5092.000.00
DiSH-GRS84.205.3091.363.6092.696.30

Table 3. Test QYI and feasibility oscillation rates (%) with GPT-6-Luna.

MethodProtein QYI ↑Protein Oscillation ↓TSP QYI ↑TSP Oscillation ↓CVRP QYI ↑CVRP Oscillation ↓
HeuriGym80.4746.6757.4241.6791.9320.00
EoH55.1255.5687.2927.2792.526.25
ReEvo65.4615.3866.8177.7897.2016.67
HSEvo59.5516.6763.23100.0073.3433.33
HiFo-Prompt65.2250.0083.6066.6794.6612.50
RefineEvo49.5450.0085.5150.0096.5518.75
DiSH-GWRS80.400.0094.860.0097.650.00
DiSH-CRS80.870.0092.690.0097.640.00
DiSH-GRS81.500.0095.200.0097.610.00

All six baselines show feasibility losses on every benchmark with both models. In all six task–model settings, every DiSH variant has a lower observed oscillation rate than every baseline. No losses were observed for DiSH with GPT-6-Luna under this metric (Table 3). These zero rates describe the evaluated runs and do not establish that future revisions will always preserve feasibility.

The results support the continued value of DiSH’s guidance on what to change and how extensively to revise the solver, even with the two recent models tested here.


Weakness 2

Only HeuriGym is a comparable baseline; DiSH is the HeuriGym loop with a different prompt. Unlike generating the full solving code, other baselines are trying to find better heuristics within a given skeleton (usually large neighborhood search). Adapting them to the full solver refinement setting rather than applying DiSH to them limits the paper's scope.

Response: We agree that HeuriGym is the closest baseline. Our focus is how execution feedback should guide successive revisions of a complete solver. DiSH addresses this problem through explicit rules for selecting feedback and adjusting the requested extent of code changes.

1. The role of the baseline comparisons.

Generating a heuristic component within a fixed algorithm and refining a complete solver are different settings. We study the latter, where a revision must address unresolved failures while preserving behavior that already works. How to use execution and verification feedback to manage these competing demands is an important research question within automatic heuristic design.

HeuriGym provides the direct comparison in this setting. For EoH, ReEvo, HSEvo, HiFo-Prompt, and RefineEvo, we adapt the generation target to complete solvers while preserving their core search procedures, as described in Appendix E.2, Table 7 of the original paper and Table 11 of this response. These comparisons assess alternative search strategies under a shared task and evaluation protocol. Their conclusions apply to the adapted complete-solver setting.

2. What DiSH adds to HeuriGym.

DiSH makes two decisions after each evaluation (Figure 2 of the original paper):

  • What to revise: It selects representative feedback from failed and successful instances to identify problems to address and behavior to preserve.
  • How extensively to revise: It adjusts the requested scope of changes according to the current demonstration feasibility rate. It allows broader changes when feasibility is low and progressively narrows them as feasibility improves.

These decisions are computed by the harness and communicated through prompts. They are updated as the solver’s behavior changes. The contribution is therefore the feedback selection and revision control process implemented through those prompts.

This design addresses a specific difficulty in complete-solver refinement: correcting a failure on one instance can disrupt previously successful behavior on another. Across nine benchmarks and four LLMs, the experiments show improvements in overall performance and refinement stability (Table 3 and Figure 8 of the original paper). For example, DiSH-CRS achieves an average QYI of 80.3, compared with 67.4 for HeuriGym (Table 2 of the original paper).

3. Extensions to other search frameworks.

DiSH’s feedback selection and revision guidance could also be incorporated into candidate generation in EoH or ReEvo, while retaining their population search, evolutionary operations, and reflection mechanisms. Such integration would test whether these controls complement broader search procedures.

We have not evaluated that integration in this work. The current experiments support DiSH’s effectiveness for iterative refinement of complete solvers; extending and testing its controls within other AHD frameworks is a further research direction.


Weakness 3

The central claim is reduced feasibility oscillation. However, the paper reports only QYI, which mixes feasibility with the objective. Moreover, “feasibility” here includes crashes and timeouts. Timeouts are the largest failure category (about 35%of all records), and Stage II failures are negligible. The largest gain is on TSP (+34.9 over HeuriGym), where producing a valid tour is trivial, so baseline TSP Yield of 40–52% likely reflects timeouts on test instances. That is a different phenomenon from the coupled-constraint failures the paper uses as motivation.

Response: We agree that the large TSP improvement mainly reflects fewer timeouts and should be distinguished from better constraint satisfaction (Table 14). In our benchmark, feasibility requires successful execution within the time limit and a valid solution that satisfies the constraints. We assess refinement stability separately from QYI and report failure types to clarify the sources of improvement.

1. Refinement stability is measured separately from QYI.

We follow HeuriGym in using QYI to assess Quality and Yield. Appendix D.3 of the original paper also defines two stability metrics:

  • Convergence Rate: the proportion of runs that solve every demonstration instance at the final iteration.
  • Survival Rate: among runs that solve every demonstration instance at least once, the proportion that maintain full feasibility from its first attainment through the end of refinement.

Both metrics appear in the original ablation study (Table 3 of the original paper). Table 4 extends the comparison.

Table 4. Feasibility stability metrics (%, higher is better).

MethodConvergence Rate ↑Survival Rate ↑
HeuriGym55.240.0
DiSH without scope control55.646.6
DiSH without direction control91.492.3
DiSH-GWRS88.393.0
DiSH-CRS92.994.2
DiSH-GRS91.491.9

All three DiSH variants exceed HeuriGym on both measures. DiSH-CRS increases Convergence Rate from 55.2% to 92.9% and Survival Rate from 40.0% to 94.2%, gains of 37.7 and 54.2 percentage points. The configuration without scope control records substantially lower rates. These results show improved attainment and preservation of full demonstration feasibility, measured without the solution-quality component of QYI.

2. Timeouts count as unsuccessful evaluations under a shared time budget.

A timeout can reflect high computational cost or a failure to terminate, such as an infinite loop. It is recorded separately from a constraint violation. This distinction matters on TSP, where returning a valid tour may be straightforward, but the generated solver must still finish within the time limit.

Each benchmark has its own execution limit, which is identical across methods. For example, Intra includes instances with tens of thousands of nodes and uses a 60-second limit. TSP likewise uses the same benchmark-specific limit for every method.

HeuriGym evaluates complete solvers under time and memory limits and records timeouts as execution failures. The other baseline implementations used in our comparison also account for timeouts. EoH and HiFo-Prompt assign no valid objective value to timed-out candidates and exclude them from population selection. ReEvo and HSEvo mark them as execution failures, assign an objective value of +∞+\infty, and exclude them from parent selection. RefineEvo assigns no valid objective value and records timeouts separately.

3. Improvements also occur in crashes and constraint violations.

Table 5 provides examples where timeout reduction does not explain the main gains.

On Operator Scheduling, HeuriGym’s timeout rate is 1.02%, while constraint violations account for 5.85% of evaluations. DiSH-GWRS and DiSH-CRS both achieve 100% observed success, with no observed constraint violations.

On E-Graph, DiSH-CRS reduces crashes from 29.41% to 8.33% and constraint violations from 12.04% to 8.60%. Its success rate rises from 53.50% to 78.17%, while the timeout rate remains nearly unchanged (5.04% versus 4.89%). These comparisons show improvements in execution reliability and constraint satisfaction beyond timeout reduction.

Table 5. Reported test outcomes on two held-out benchmarks. Values are percentages of all test evaluations. Best values in each column are bolded.

(a) Operator Scheduling

MethodSuccess ↑Timeout ↓Crash ↓Constraint violation ↓
HeuriGym93.131.020.005.85
EoH84.803.950.0011.26
ReEvo66.082.632.7828.51
HSEvo64.330.882.7832.02
HiFo-Prompt78.514.820.1516.52
RefineEvo64.776.293.5125.44
DiSH-GWRS100.000.000.000.00
DiSH-CRS100.000.000.000.00
DiSH-GRS97.220.000.002.78

(b) E-Graph

MethodSuccess ↑Timeout ↓Crash ↓Constraint violation ↓
HeuriGym53.505.0429.4112.04
EoH42.045.030.0052.93
ReEvo26.801.638.5762.99
HSEvo40.003.272.8653.88
HiFo-Prompt26.948.160.4164.49
RefineEvo43.672.310.0054.01
DiSH-GWRS57.693.6730.757.89
DiSH-CRS78.174.898.338.60
DiSH-GRS66.531.6314.2917.55

These analyses address different aspects of reliability. The trajectory metrics measure whether full demonstration feasibility is attained and preserved. The test breakdown identifies the types of failures observed on unseen instances.

Reviewer_YUnLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important limitation of LLM-based automatic heuristic design: iterative solver modifications frequently introduce new constraint violations while repairing existing errors, a phenomenon the authors identify as feasibility oscillation. To address this issue, the proposed DiSH framework explicitly controls both the direction and scope of solver revisions. Direction control identifies representative failures and successful cases to prioritize the next update, while scope control progressively restricts modifications as the solver accumulates feasible behavior. The framework introduces three alternative direction strategies: Group-Wise Representative Sampling (GWRS), Correction Representative Sampling (CRS), and Scope-Aware Group Representative Sampling (GRS). Experiments across nine combinatorial optimization benchmarks and four LLM backbones compare DiSH with EoH, ReEvo, HSEvo, HiFo-Prompt, RefineEvo, and HeuriGym. The strongest DiSH variant, CRS, achieves an average Quality-Yield Index (QYI) of 80.3%, compared with 67.4% for the strongest baseline.

Critical strengths
  • The paper identifies a failure mode specific to iterative solver development and proposes explicit mechanisms to control it. The distinction between revision direction and revision scope gives the framework methodological clarity and makes it adaptable to different LLM backbones.
  • The evaluation covers nine optimization problems, four backbone models, multiple competitive AHD methods, and component ablations. In particular, removing direction control yields 76.4% QYI, while removing scope control reduces it to 69.0%, supporting the conclusion that scope control provides a substantial portion of the improvement.
Critical weaknesses
  • Selecting representative errors and progressively restricting changes can be interpreted as established ideas from local search, software repair, and trust-region methods adapted to LLM prompts. The paper needs a sharper explanation of what is technically new beyond these principles.
  • The association between large code changes and reduced feasibility does not establish causation. In particular, harder problems may require both larger modifications and exhibit greater failure rates.
  • Although token efficiency is emphasized, the paper should establish whether all relevant verification, execution, orchestration, and wall-clock costs are consistently included. Generalization from demonstration-instance feasibility to truly unseen solver requirements also requires further scrutiny.
Questions for authors

The principal technical concern is that DiSH controls solver changes through LLM-facing instructions rather than enforcing a formal constraint on program transformations. Consequently, the realized update may deviate from the requested scope, and preservation of previously verified behavior is empirical rather than guaranteed. Restricting revision magnitude may also discourage necessary structural redesign when the current solver is fundamentally incorrect. The experiments demonstrate convincing improvements within the evaluated refinement protocol, but stronger evidence would require matched-budget comparisons with generic minimal-patch prompting, rollback-based repair, trust-region-inspired editing, and failure-prioritization baselines. Additional analyses should isolate the effects of revision magnitude, failure diversity, initialization quality, and problem hardness. The benchmark results also exhibit heterogeneity: DiSH does not win on every problem, indicating that the proposed control mechanism is not universally superior.

Here are the key questions to be addressed. Q1: How does DiSH fundamentally differ from trust-region optimization, minimal-edit program repair, regression-aware debugging, and iterative repair with rollback? Q2: Can the authors demonstrate that restricting revision magnitude causally reduces feasibility oscillation, independently of problem difficulty and solver complexity? Q3: How often do generated solver revisions violate the prescribed modification scope, and what happens when they do? Q4: What are the results for generic minimal-change prompting, best-so-far checkpointing, rollback-based revision, and random failure subsampling under identical budgets? Q5: Are total input/output tokens, LLM calls, solver execution time, verification overhead, and wall-clock latency included in the efficiency comparison? Q6: How does performance change when validation instances differ substantially from the demonstration distribution or contain previously unseen constraint interactions? Q7: Why does DiSH underperform HiFo-Prompt on Pedigree, and under what conditions does progressively narrowing the revision scope prevent necessary solver redesign?

14 comments

Weakness 1

Selecting representative errors and progressively restricting changes can be interpreted as established ideas from local search, software repair, and trust-region methods adapted to LLM prompts. The paper needs a sharper explanation of what is technically new beyond these principles.

Response: We acknowledge that representative error selection and restricted edits draw on established ideas in optimization and software repair. DiSH’s contribution is a concrete revision policy for LLM-generated combinatorial solvers. At each iteration, it uses execution feedback to coordinate two decisions: which cases should guide the next revision and how broadly the solver should be revised.

The challenge is that an instruction to revise code does not fully determine the resulting change or its effect on feasibility. A revision that fixes some instances may introduce failures on previously feasible ones (Figure 3 of the original paper). DiSH addresses this through two mechanisms:

  • Direction control: DiSH selects representative failures and successful instances under a limited feedback budget. The failures identify what needs repair, while the successful instances identify behavior that should be preserved. This gives the LLM focused guidance on both objectives.
  • Scope control: DiSH uses the current demo feasibility rate to select among Broad, Moderate, Conservative, and Protected revision instructions (Table 5 of the original paper). It requests broader changes when feasibility is low and more limited changes when feasibility is high. The selected mode is updated according to the solver’s current state.

These decisions are combined in the revision prompt. Feedback selection defines the repair targets and preservation goals, while the selected mode specifies the requested extent of change. The harness implements this control through feedback and instructions, without modifying the underlying LLM.

Our experiments assess the value of this design. The ablations examine the contributions of direction and scope control to solver refinement (Table 3 of the original paper). Additional comparisons with HeuriGym + Revert and HeuriGym + Diff-only show that DiSH achieves higher QYI than these safeguards in the evaluated settings (Table 4 of this response, where the safeguards are listed as HeuriGym-Rollback and HeuriGym-MiniPatch). Together with the observed improvements in feasibility stability (Table 3 and Figure 8 of the original paper) and feedback token use (Figure 7 of the original paper), these results support the effectiveness of the combined design.

We will clarify these mechanisms and their relationship to existing principles in the revised paper.


Weakness 2

The association between large code changes and reduced feasibility does not establish causation. In particular, harder problems may require both larger modifications and exhibit greater failure rates.

Response: We agree that problem difficulty may influence both revision size and feasibility regression. We therefore added analyses that account for observed differences in experimental settings and solver performance.

Association across runs. We analyzed over 3,000 revisions from HeuriGym and the three DiSH variants, excluding revisions in Protected mode. We measured code changes between consecutive solvers using Jaccard distance. We counted a feasibility oscillation event when at least one demo instance that the parent solver passed failed after the revision. Both Pearson and Spearman correlations indicate a modest positive association. Their 95% confidence intervals, estimated using 20,000 bootstrap resamples at the run level, exclude zero (Table 1).

Table 1. Correlation between revision distance and feasibility oscillation.

StatisticEstimate95% CI
Pearson correlation0.205[0.170, 0.238]
Spearman correlation0.202[0.168, 0.234]

Adjusting for observed differences. Harder problems may require larger code changes and may also be more prone to feasibility loss. To examine whether the association persists after adjustment, we fitted a logistic regression controlling for benchmark, method, backbone, demo feasibility before the revision, and refinement iteration. We used robust standard errors clustered by run, with the CR1 correction, to account for dependence among revisions within each run.

We report three models in Table 2. The first row uses the baseline controls. The second row adds the initial solver’s demo QYI to account for starting performance and the log of the parent solver’s Python token count to account for code size. The third row uses the same controls as the second and includes only revisions for which both the parent and revised code compiled successfully. All three analyses exclude revisions in Protected mode.

Table 2. Adjusted association between revision distance and feasibility oscillation. Odds ratios are reported per 0.1 increase in Jaccard distance.

AnalysisRevisionsAdjusted odds ratio95% CI
Baseline controls3,3711.221[1.170, 1.274]
Added initial demo QYI and parent code length3,3711.255[1.200, 1.312]
Same controls as row 2, both versions compiled3,3321.265[1.208, 1.325]

In the first model, a 0.1 increase in Jaccard distance was associated with 22.1% higher odds of losing at least one demo instance that the parent solver passed. The adjusted odds ratio was 1.255 after adding the two controls and 1.265 when restricting that model to revisions where both versions compiled. All three confidence intervals lie above 1, so the positive association persists across these checks. Unmeasured differences in problem difficulty could still contribute to the association.


Weakness 3

Although token efficiency is emphasized, the paper should establish whether all relevant verification, execution, orchestration, and wall-clock costs are consistently included. Generalization from demonstration-instance feasibility to truly unseen solver requirements also requires further scrutiny.

Response: On Protein Design with Gemini-2.5-Flash, demo solver execution and verification took 77.6–83.5 s for the three DiSH variants, compared with 98.2 s for HeuriGym. After normalization to one final test evaluation, workflow time was 159.0–161.5 s for DiSH and 211.3 s for HeuriGym (Table 9 of this response).

For held-out generalization, Crew Pairing instances contain 37–44 flight legs in the demo set and 206–6,799 in the test set (Appendix E.1, Table 6 of the original paper). Among the delivered solvers that passed all demo instances in the matched groups, the three DiSH variants achieved held-out Test QYIs of 63.7%–74.2%, compared with 50.1% for the best baseline (Table 10 of this response).


Opening paragraph of the Questions section

The principal technical concern is that DiSH controls solver changes through LLM-facing instructions rather than enforcing a formal constraint on program transformations. Consequently, the realized update may deviate from the requested scope, and preservation of previously verified behavior is empirical rather than guaranteed. Restricting revision magnitude may also discourage necessary structural redesign when the current solver is fundamentally incorrect. The experiments demonstrate convincing improvements within the evaluated refinement protocol, but stronger evidence would require matched-budget comparisons with generic minimal-patch prompting, rollback-based repair, trust-region-inspired editing, and failure-prioritization baselines. Additional analyses should isolate the effects of revision magnitude, failure diversity, initialization quality, and problem hardness. The benchmark results also exhibit heterogeneity: DiSH does not win on every problem, indicating that the proposed control mechanism is not universally superior.

1. Scope compliance and preservation of verified behavior

We agree that DiSH’s instructions do not formally enforce revision scope or guarantee preservation of previously verified behavior. Our additional analysis examines the observed outcomes of scope violations.

Table 3 summarizes changes in the number of feasible demo instances among 221 confirmed violations.

Table 3. Changes in the number of feasible demo instances among confirmed scope violations.

Change in feasible instance countRevisionsPercentage
Unchanged17277.83%
Increased135.88%
Decreased3616.29%
Total221100%

These results show that scope violations do not always reduce demo feasibility. Rejecting every violation would also reject the 13 revisions that increased the feasible instance count. However, an unchanged count does not guarantee that the same instances remain feasible. This analysis also does not establish whether strict enforcement would improve overall performance.

DiSH provides guidance before code generation, while its candidate acceptance and rejection policies are modular. This allows stricter scope checks, selective rejection, and rollback to be combined with adaptive guidance and evaluated under the same protocol.

2. Allowing structural redesign

DiSH selects the requested revision scope according to the current demo feasibility rate. When feasibility is low, Broad mode allows substantial changes, including structural redesign. At higher feasibility rates, it requests more limited changes. Feedback from failed and successful instances helps identify what to repair and what to preserve.

Figure 12(a) of the original paper provides a concrete example: under Broad mode, DiSH replaces the core algorithm on Pedigree and increases demo feasibility from 1/5 to 5/5.

The controlled comparisons also support adaptive scope selection. In Figure 9 of the original paper, the adaptive policy achieves a mean QYI of 72.8%, compared with 65.4%–69.1% for the fixed policies; Table 6 below provides the full results. With GWRS feedback held fixed, Table 3 of the original paper shows that adaptive scope increases Convergence Rate from 55.6% to 88.3% and Survival Rate from 46.6% to 93.0%.

These results show that DiSH can accommodate structural redesign and benefit from adaptive scope selection. They do not rule out restricting changes too early when demo feasibility is high, a limitation discussed in Point 5 below.

3. Additional baseline comparisons

Editing and rollback baselines. We implemented three safeguards within HeuriGym and evaluated them on Protein Design, TSP, and Pedigree:

  • HeuriGym-MiniPatch requests minimal code patches.
  • HeuriGym-Rollback retains the previously accepted solver when demo feasibility decreases.
  • HeuriGym-TrustRegion adaptively adjusts and enforces a Jaccard-distance limit on code changes.

For Protein Design, we used a variant of the TrustRegion acceptance rule. Table 4 reports the results.

Table 4. Test QYI (%) in the additional safeguard comparisons.

MethodProtein Design ↑TSP ↑Pedigree ↑Mean ↑
HeuriGym73.0559.5445.4159.33
HeuriGym-Rollback74.4941.8040.0752.12
HeuriGym-MiniPatch49.2740.4234.2741.32
HeuriGym-TrustRegion50.8260.3448.3853.18
DiSH-GWRS77.4987.0864.7876.45
DiSH-CRS78.0485.9372.8978.95
DiSH-GRS78.0876.4170.2174.90

All three DiSH variants achieve higher Test QYI than the tested safeguards on each benchmark. Their mean QYIs range from 74.90 to 78.95, compared with 41.32–53.18 for the safeguards and 59.33 for HeuriGym.

Feedback selection baselines. On Protein Design, we compared structured selection with random sampling and three simple priorities: failure stage, failure-type frequency, and consecutive failures. We matched the failure-case budgets and disabled scope control to compare the feedback selection strategies (Table 5 of this response).

Table 5. Test QYI on Protein Design with scope control disabled and matched failure-case budgets.

MethodTest QYI (%) ↑
HeuriGym-RandomFeedback34.24
HeuriGym-StageFirst30.91
HeuriGym-FrequencyFeedback33.40
HeuriGym-HardFeedback57.80
HeuriGym-GWRS63.73
HeuriGym-CRS62.04
HeuriGym-GRS64.64

GWRS, CRS, and GRS each outperform the tested alternatives. These results support the value of the structured feedback selection strategies under the matched failure-case budgets.

Reviewer_KdACOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

When an LLM repeatedly revises a solver, an edit often breaks instances that used to pass. The paper calls this as feasibility oscillation. DiSH addresses it with two mechanisms: grouping failures and showing the LLM only representative ones (direction), and restricting how much of the code the LLM may change as the pass rate rises (scope). On 9 CO problems with 4 backbones, DiSH improves QYI over HeuriGym (about 78–80 vs. 67.4) and uses roughly half the feedback tokens.

Critical strengths
  • The problem is practical, and anyone who has done LLM refinement has seen it. The experimental grid is fairly broad with 9 problems and 4 backbones and repeats. All prompts are released, and there are some decent controls (random selection at the same evidence budget, fixed vs. adaptive scope).
Critical weaknesses
  • The comparison with AHD baselines is not fair: EoH, ReEvo, and HSEvo get only about 8 solvers with pop_size=2, and HiFo-Prompt and RefineEvo only 6 (Table 7). That is far below what these population methods are designed for. They also see only scalar fitness, while DiSH sees per-instance diagnostics. DiSH returns its best checkpoint (App. B.1), and the paper does not say what the baselines return. Only HeuriGym is really a like-for-like comparison.
  • The analysis is correlational, and the gain comes mostly from Yield. "Large edits cause regressions" is supported only by a few representative trajectories, not by statistics over all runs. Protected mode tells the LLM to return the code unchanged, so a high survival rate follows by construction. Most of the improvement is in Yield while Quality barely moves and Quality is computed on a different subset of instances for each method. There are no significance tests, and on Gemini the margin is within noise.
  • Limited novelty, and baselines are missing. Grouping failures by error signature and restricting edit size as tests pass are common practice in LLM code repair and agentic coding (e.g., diff-based editing, SWE-agent-style harnesses). The scope thresholds are also hand-set. The most important missing baseline is a simple revert when the pass rate drops (keep the best solver and discard regressing edits), which directly removes oscillation at no cost. A diff-only editing baseline and MCTS-AHD (cited but not compared) are also missing. Without these, it is unclear whether DiSH beats trivial safeguards.
Questions for authors
  1. What does each baseline deliver at the end? If every method returns its best checkpoint, do the results still hold?
  2. How does DiSH compare with EoH and ReEvo at a more realistic budget (about 50–100 evaluations), with the same diagnostic feedback?
  3. What is Quality on the instances that all methods solve?
  4. The paper has no code. Could you release an anonymous repo, including the baseline configurations?
  5. How does DiSH compare with simple "keep-best / revert-on-regression" and diff-only editing baselines on HeuriGym?
6 comments

Weaknesses 1

The comparison with AHD baselines is not fair: EoH, ReEvo, and HSEvo get only about 8 solvers with pop_size=2, and HiFo-Prompt and RefineEvo only 6 (Table 7). That is far below what these population methods are designed for. They also see only scalar fitness, while DiSH sees per-instance diagnostics. DiSH returns its best checkpoint (App. B.1), and the paper does not say what the baselines return. Only HeuriGym is really a like-for-like comparison.

Response: We clarify the search budgets, feedback, and checkpoint selection below and provide additional experiments to assess their effects.

1. Search budgets and population sizes

Our main experiments compare all methods under the same total token budget. The population sizes and solver counts in Table 7 of the paper reflect this budget. Under these settings, each baseline generates more candidate solvers than DiSH. We acknowledge that population-based methods may benefit from larger search budgets.

To examine this possibility, we conducted additional experiments on Protein Design. We allowed EoH and ReEvo 20, 30, or 50 new candidate attempts while keeping DiSH at 10 refinement iterations. Table 1 reports the Protein Design results.

Table 1. Performance and token consumption on Protein Design.

MethodSearch budgetTest QYI (%) ↑Tokens ↓
EoH20 candidate attempts56.2195,424
EoH30 candidate attempts55.20134,896
EoH50 candidate attempts58.39234,479
ReEvo20 candidate attempts63.99176,902
ReEvo30 candidate attempts68.65227,397
ReEvo50 candidate attempts73.96456,803
DiSH-GWRS10 refinement iterations78.3357,695
DiSH-CRS10 refinement iterations77.8546,629
DiSH-GRS10 refinement iterations78.5947,343

ReEvo improves as its search budget increases, reaching a QYI of 73.96 at 50 attempts. EoH reaches 58.39 at the same budget. Both remain below all three DiSH variants, which achieve 77.85–78.59 with 10 iterations. At 50 attempts, EoH and ReEvo also consume approximately 4–5 and 8–10 times as many tokens as DiSH, respectively. Thus, DiSH retains its performance advantage on Protein Design even when these baselines receive substantially larger search budgets.

2. Feedback available to each method

We preserve each baseline’s original feedback mechanism. EoH, ReEvo, HSEvo, HiFo-Prompt, and RefineEvo use scalar fitness scores, while HeuriGym uses per-instance diagnostics. These comparisons therefore evaluate the methods with their original feedback designs. We agree that HeuriGym provides the most direct comparison for assessing the additional benefit of DiSH’s harness.

DiSH uses the same underlying diagnostics as HeuriGym but changes how they guide revisions. It selects informative feedback to guide what to change and uses the current feasibility rate on the demo set to guide how much to change. This makes revisions more targeted and reduces feedback token use (Figure 7 of the paper).

To examine the contribution of per-instance feedback within DiSH, we evaluated DiSH-Scalar. This variant retains DiSH’s control mechanisms but gives the LLM only a scalar fitness score. Table 2 reports the results.

Table 2. Effect of feedback type on Protein Design with Gemini-2.5-Flash.

MethodFeedback typeTest QYI (%) ↑
DiSH-ScalarScalar fitness54.26
DiSH-GWRSPer-instance diagnostics77.89
DiSH-CRSPer-instance diagnostics78.77
DiSH-GRSPer-instance diagnostics79.44

DiSH-Scalar achieves a QYI of 54.26, compared with 77.89–79.44 for the full variants. This gap supports the importance of per-instance diagnostics within DiSH. The comparison with HeuriGym assesses the added value of the harness when both methods have access to the same underlying diagnostic information.

3. Checkpoint selection

We follow each baseline’s original checkpoint selection rule:

  • EoH, HiFo-Prompt, and RefineEvo select the best solver from the final population using their original fitness criteria.
  • ReEvo and HSEvo select the best solver found throughout the search using their original fitness criteria.
  • HeuriGym and DiSH select the solver with the highest demo feasibility rate across the entire search, breaking ties using the lowest average demo cost.

DiSH therefore uses the same selection rule as HeuriGym. In our experiments, DiSH’s selected checkpoint also coincides with its final checkpoint.

To assess whether differences in selection rules affect the results, we applied a common rule across all methods: select the checkpoint with the highest demo QYI from the entire search history and evaluate its test QYI. Across 1,467 runs, 98% selected the same solver as under the original rules. Table 3 reports the independent retest on Protein Design with Gemini-2.5-Flash from the 20-attempt experiment, which included EoH, ReEvo, and the three DiSH variants.

Table 3. Test QYI (%) under the original and common checkpoint selection rules on Protein Design with Gemini-2.5-Flash.

MethodTest QYI (%) ↑ (original rule)Test QYI (%) ↑ (common rule)
EoH56.2156.21
ReEvo64.0164.01
DiSH-GWRS76.9676.96
DiSH-CRS78.6378.63
DiSH-GRS78.7278.72

For the remaining runs, the change in average QYI was no more than 1.02 points for any baseline. These results indicate that our conclusions remain unchanged under a common checkpoint selection rule.


Weaknesses 2

The analysis is correlational, and the gain comes mostly from Yield. "Large edits cause regressions" is supported only by a few representative trajectories, not by statistics over all runs. Protected mode tells the LLM to return the code unchanged, so a high survival rate follows by construction. Most of the improvement is in Yield while Quality barely moves and Quality is computed on a different subset of instances for each method. There are no significance tests, and on Gemini the margin is within noise.

Response: We have expanded the statistical analysis and added a Quality comparison on the same set of feasible instances. We also clarify the role of Protected mode and the limits of the conclusions supported by these results.

1. Statistical evidence and the contribution of Yield

Statistical evidence across runs. We extended the analysis beyond the representative trajectories in Figure 4 of the paper to more than 3,000 revisions outside Protected mode. These revisions were drawn from 1,296 experimental runs covering nine benchmarks, four backbones, and four methods: HeuriGym and the three DiSH variants (Table 4 of this response).

Table 4. Coverage of the revision–regression analysis.

StatisticCount
Benchmark problems9
Backbone LLMs4
Methods4
Experimental runs1,296
Potential revision events5,184
Analyzed revisions outside Protected mode>3,000

Pearson and Spearman correlations indicate a modest positive association between revision distance and feasibility regression. We estimated their 95% confidence intervals using 20,000 bootstrap resamples at the run level.

We also fitted a logistic regression controlling for benchmark, method, backbone, feasibility before the revision, and refinement iteration. We used robust standard errors clustered by run to account for dependence among revisions within the same run. After these adjustments, each 0.1 increase in revision distance was associated with 22.1% higher odds of feasibility regression. Table 5 reports the estimates.

Table 5. Association between revision distance and feasibility regression. All revisions in Protected mode are excluded.

StatisticEstimate95% confidence interval
Pearson correlation0.205[0.170, 0.238]
Spearman correlation0.202[0.168, 0.234]
Adjusted odds ratio per 0.1 increase in revision distance1.221[1.170, 1.274]

These results provide evidence of an association across thousands of revisions. They support our motivation for controlling revision size, but do not establish causality.

Yield and solution quality. We agree that DiSH’s larger improvement is in Yield. Following HeuriGym’s protocol, Yield measures the proportion of test instances for which a solver produces feasible solutions. These are held-out instances, some substantially larger than the demo instances used during refinement (Appendix E.1, Table 6 of the paper).

Compared with HeuriGym, DiSH-CRS increases average Yield from 68.6% to 84.5%, a gain of 15.9 percentage points. The corresponding average failure rate falls from 31.4% to 15.5%, a relative reduction of approximately 50.6% (Appendix G.1, Table 10 of the paper). This represents a substantial improvement in feasible-solution coverage on unseen instances.

Quality is computed only on instances that each method solves feasibly, so the evaluation subsets can differ across methods. To address this limitation, we also compared Quality on instances solved feasibly by all methods. On the shared feasible subset, the three DiSH variants achieve Quality scores of 89.97%–91.07%, compared with 90.65% for HeuriGym (Table 6 of this response). The results therefore support broader feasible-solution coverage on unseen instances (Appendix G.1, Table 10 of the paper), with competitive solution quality on the shared feasible subset.

2. The role of Protected mode

DiSH selects among Broad, Moderate, Conservative, and Protected modes according to the current feasibility rate on the demo set, as described in Table 5 and Figure 5 of the original paper. It allows broader changes at lower feasibility rates and limits changes as feasibility improves.

Protected mode is activated only after demo feasibility reaches 100%, when all demo instances already have feasible solutions. At this stage, permitted edits are restricted to minor parameter adjustments intended to preserve the feasibility already achieved.

We agree that restricting changes in Protected mode can contribute to a high survival rate. However, this mode does not explain how full demo feasibility is reached, because it becomes active only after that point. We therefore distinguish reaching full feasibility from preserving it, and do not treat survival in Protected mode alone as evidence of stable improvement. The statistical analysis in Tables 4 and 5 of this response excludes all revisions in Protected mode and measures the revision–regression association outside this mode.

3. Quality, Yield, and comparison on the same instances

We agree that the larger improvement is in Yield. Under the original evaluation, DiSH-CRS improves average Quality from 74.6% to 80.7% (+6.1 percentage points) and Yield from 68.6% to 84.5% (+15.9 percentage points) relative to HeuriGym. Table 10 of the paper (Appendix G.1) reports these metrics separately across the four LLM backbones.

We follow HeuriGym’s definition of the Quality-Yield Index (QYI). Yield measures the proportion of instances with feasible solutions, while Quality evaluates objective values relative to the reference optimum on feasible instances. Consequently, each method’s Quality is computed on its own feasible subset. We agree that these different subsets limit direct comparisons of Quality.

To address this issue, we also evaluated Quality on instances for which every method produced a feasible solution. This ensures that all methods are compared on the same instances. Table 6 reports both the original Quality scores and the scores on this common feasible subset.

Table 6. Quality (%) on each method’s feasible instances and on the common feasible subset.

MethodQuality ↑Quality on common feasible instances ↑
HeuriGym74.690.65
EoH76.992.10
ReEvo63.191.86
HSEvo64.289.67
HiFo-Prompt66.990.20
RefineEvo67.888.40
DiSH-GWRS80.091.07
DiSH-CRS80.790.26
DiSH-GRS79.289.97

On the common feasible subset, DiSH achieves Quality scores of 89.97%–91.07%, compared with 90.65% for HeuriGym and 88.40%–92.10% for the five other AHD baselines. These results show competitive solution quality on the same instances, alongside higher Yield in the full evaluation. They support a more precise interpretation of DiSH’s main benefit: broader feasible coverage with competitive solution quality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely appreciate the reviewers' thoughtful reviews and constructive feedback. We are highly encouraged by their positive comments on our work:

  • Research problem and motivation. Feasibility oscillation is a practical challenge in iterative solver refinement (Reviewer KdAC, Reviewer YUnL). The paper is clearly motivated and illustrates this phenomenon through concrete refinement trajectories (Reviewer 3AMn).
  • Method design. Distinguishing revision direction from revision scope gives DiSH a clear methodological structure and facilitates adaptation to different LLM backbones (Reviewer YUnL). The method is simple and interpretable (Reviewer 3AMn).
  • Experiments and ablations. The evaluation covers nine problems and four LLM backbones (Reviewer KdAC, Reviewer YUnL), with random-selection controls under matched feedback budgets and comparisons of revision-scope strategies (Reviewer KdAC). Component ablations support the effectiveness of the design (Reviewer YUnL, Reviewer 3AMn).
  • Presentation and prompt availability. The paper is clear and easy to follow, and all prompts are provided (Reviewer KdAC, Reviewer 3AMn).

Building on the reviewers' constructive suggestions, we conducted additional experiments and analyses that further strengthen our understanding of the research problem, method design, and empirical findings:

  1. Research problem: Stronger LLM capabilities do not automatically lead to more stable iterative refinement. Experiments with two additional LLM backbones show that feasibility oscillation persists despite higher final QYI, highlighting the continued importance of stable, cumulative solver refinement (Reviewer 3AMn, Table 1).

  2. Method design: Coordinating revision direction and scope improves solver refinement. Feedback-selection comparisons under matched failure-case budgets and revision-scope ablations support the effectiveness of both mechanisms (Reviewer YUnL, Tables 5–6; Original Paper, Table 3). DiSH's gains over Diff-only and Revert further highlight the value of guiding revisions before candidate generation using failure and success feedback, rather than relying solely on simple safeguards (Reviewer KdAC, Table 8; Reviewer YUnL, Table 7).

  3. Empirical findings: Additional experiments further validate DiSH's effectiveness and generalization performance. Statistical tests confirm significant overall Test QYI and Yield gains over HeuriGym (Reviewer KdAC, Table 7; Reviewer 3AMn, Table 6). Even in matched groups where all methods have fully demo-feasible delivered solvers, DiSH outperforms all baselines in held-out Test QYI on four of five tasks (Reviewer YUnL, Table 10). A separate analysis shows competitive solution quality on the common feasible subset (Reviewer KdAC, Table 6).

We hope the following responses address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.