acceptodds
Under review as a conference paper at ICLR 2027

Improving LLM-driven program evolution with dynamic scope

Abstract

LLM-driven program evolution typically searches over candidate programs while treating the extent that each mutation may change as a fixed generation interface. This creates a structural tension: narrow editing boundaries preserve locality and attribution but can make dependency-spanning improvements unreachable, whereas broad boundaries admit coordinated changes at the cost of a larger proposal space and weaker attribution. Because the appropriate side of this trade-off can change over a lineage, a fixed boundary can become either a reachability ceiling or an unnecessarily broad search interface. We introduce DyScope, which turns the permitted editing boundary into explicit, executable, and verifiable search state. A structural scope graph defines available regions, and each boundary action jointly specifies a target, dependency width, mutation regime, and transition guard. Trajectory evidence can retain productive local scope, widen to dependencies, redirect the target, contract after a broad transition, or invoke recovery. Before generation, the selected boundary changes the context exposed to the proposer and the set of reachable transformations. After generation, an executable transition contract compares the requested boundary with the realized parent–child diff, protects invariant regions, and records whether the transition is admissible. DyScope therefore adapts and verifies program transitions rather than only proposing programs. Feedback supplies evidence for updating boundary state; specific thresholds and rankings instantiate the selection policy rather than define the abstraction. DyScope attains 96.67%, 99.39%, and 98.83% on AIME 2026, HumanEval, and MBPP, respectively. In the matched boundary-policy comparison, dynamic scope reaches 90.18% versus 82.15% for fixed scope. These results identify the editing boundary as an adaptive dimension of program search rather than a static implementation choice. By making changes to an evolving system selectable, enforceable, and traceable, DyScope provides a concrete control interface toward auditable recursive self-improvement.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM-driven program evolution, dynamic scope, evolutionary search, program optimization, structural code editing, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WftsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DyScope, a framework for LLM-driven program evolution that dynamically adjusts the editing scope based on search feedback. It uses a structural dependency graph to select scopes to edit and a transition contract to check whether generated modifications respect the selected boundaries. The authors evaluate DyScope on reasoning and coding benchmarks and introduce ScopeBench to compare dynamic scope control against baselines and the ablated versions, and show the superior performance of DyScope.

Critical strengths
  1. The motivation is reasonable: restricting edits to one region can prevent useful changes that require modifying multiple functions, while allowing unrestricted edits can make search less efficient. This scope selection trade-off is worth exploring.

  2. The method checks the actual code changes against the selected scope rather than relying solely on prompt instructions. This helps ensure that the generated edits follow the controller’s decisions.

Critical weaknesses
  1. The core method is not described clear enough to reproduce. Equations 6 and 7 introduce scores for selecting operators and editing boundaries, but the paper does not explain how several terms are computed or how the gating rules are implemented. Moreover, the paper does not motivate why the formula is designed this way and where do the parameters come from. These decisions are central to the method, so it is difficult to assess what actually contributes to the reported improvements.

  2. The main experimental results are inconsistent across the text, tables, and figures. Table 2 reports 90.18% for DyScope and 82.15% for fixed scope, a difference of 8.03 percentage points, but lines 356–357 report a 3.66% improvement. Figure 6(d) shows gains over initialization of 0.077 and 0.046, whose difference is only 0.031.

Questions for authors
  1. How are the terms in Equation 6 and the boundary score JtJ_t in Equation 7 computed? Please specify the gating rules and how the weights and thresholds were chosen. What evidence supports these design choices?

  2. Can you explain the inconsistencies in weaknesses?

  3. Does the fixed-scope baseline permit all edits needed to solve each ScopeBench task? Have you compared DyScope with a baseline that allows whole-program editing throughout search? This distinction matters for determining whether the benefit comes from adapting scope or simply removing restrictions imposed on the baseline.

  4. How is the scope graph constructed in the experiments? Please specify the node and edge definitions, the programming languages and analysis tools used, whether annotations are manually provided, and whether the graph is updated after each program modification. Dependency analysis methods could vary across different programming languages and package management environments, and could be imprecise.

  5. Is there a public repository for the code and artifacts? Since some implementation details are not specified in the paper, the code could help future researchers understand better how this is implemented.

  6. The reproducibility statement refers to a supplement containing the full protocol, policy settings, and additional results, but I could not find it on the submission page or in the paper PDF. Could the authors clarify where this material is available?

5 comments

We thank the reviewer for the detailed concerns about reproducibility, numerical consistency, and the scope-graph construction. We address each question directly.

Q1.

In Equation 6, score = empirical rank + lineage credit + coverage − risk − cost. Rank is the historical fraction ≤ the adjusted operator prior (0.5 without history); coverage is 0.10[1 − n_o/max(1, max_j n_j)], using lineage counts. Risk reflects search/edit failures; cost is operator-specific and zero in quality-first mode. Selection takes the eligible argmax.

In Equation 7, the action component adds the prior to mean value, exploration, validity, breakout and novelty, weighted 0.45/0.20/0.10/0.20/0.10, followed by capability adjustments. Exploration is sqrt(log(N+1)/(n_o+1)), using visit counts. Width is cross-scope if breakout ≥ 0.6; otherwise dependency-cone if uncertainty ≥ 0.55 or progress ≤ 0; otherwise single-region.

Before ranking, gating priority is recovery → rescue → refinement → cycling → stagnation → early risk; only stage-compatible operators qualify. Rescue requires jump-gain ≥ 0.55 and diff-failure/interface-break rates both < 0.35.

Weights and thresholds are manually set heuristics balancing improvement, exploration and reliability. Routing, recovery and enforcement ablations assess these component choices; recorded transitions verify rule execution. They do not establish coefficient optimality.

Q2.

Table 2 reports the following scores and differences from Fixed-scope (differences calculated from the displayed means):

MethodScoreDifference vs. Fixed-scope
Fixed-scope82.15%—
DyScope90.18%+8.03 pp
DyScope w/o structural routing88.69%+6.54 pp
DyScope w/o boundary enforcement85.15%+3.00 pp
DyScope w/o recovery89.41%+7.26 pp

The +3.66% figure is the relative improvement of the boundary-enforcement ablation over Fixed-scope, calculated before rounding, not the full DyScope improvement of 8.03 percentage points.

Figure 6 compares Dynamic scope and Fixed scope on 18 tasks, three seeds and 20 proposals. Panels (a–b) show terminal scores; (c–d) show gains over initialization; (e) shows gains at common cumulative time budgets. The +0.077/+0.046 labels are gains over initialization, not absolute scores or relative percentages. Panel (f) follows knapsack-L2, seed 47: Dynamic scope rises from 0.7280 to 0.9999, while Fixed scope ends at 0.7280.

Q3.

Fixed-scope allows edits only within the region selected at initialization. If a proposed repair also changes a helper outside that region, it is rejected. DyScope can expand the boundary to allow both changes; Fixed-wide permits all legal regions throughout search.

At the same 20-proposal budget, DyScope achieves comparable quality to Fixed-wide (90.70% vs. 90.57%, +0.13 pp, 95% CI [−0.48, +1.02]) with 12.81% less end-to-end time (89.42 vs. 102.56 s/run) and 21.65% fewer output tokens (14,866 vs. 18,974/run). This isolated comparison covers 18 tasks × three seeds with identical model and patch representation. DyScope's advantage is therefore comparable quality at lower time and output-token costs under the same proposal budget.

Q4.

Nodes are source regions with locations, nesting and mutability information. Our experiments generate region labels automatically, without manual annotation, using Python function/class boundaries and configured naming heuristics. Existing labels are retained.

Edges combine declared dependencies with AST name-use relations: if A defines a name used in B, an edge B → A records the dependency. The repository adapter additionally resolves cross-file imports, including relative imports. Dependency inference uses static analysis, not an LLM.

The graph is rebuilt from the selected parent's current source before each proposal, so subsequent proposals reflect its edits. Parsing failures omit inferred edges but preserve declared relations; untagged non-Python inputs fall back to a whole-block region. Tests verify dependency removal and cross-file import edges.

Q5–Q6.

The code and supplementary material are available in our anonymous repository: https://anonymous.4open.science/r/dyscope-FC8B/

It contains the DyScope controller, benchmark adapters and protocol configurations, and the complete ScopeBench package: all 18 tasks, initial programs, evaluators and dataset documentation. The README explains the file structure, installation, executable examples and experiment reproduction, including feedback and budget rules. All 18 ScopeBench initial-program checks and 20 offline tests pass. The paper and supplement are available under papers/dyscope.pdf and papers/supplement.pdf; ScopeBench is also included as a standalone ZIP.

Thank you for the detailed response. I will take this rebuttal into account.

W1. Controller definition, parameter choices and reproducibility.

DyScope first gates eligible actions, then ranks them, chooses an editable boundary, and verifies the actual patch before evaluation. Code, configurations and reproduction instructions are available in our anonymous repository.

Equation 6: the score combines empirical rank, lineage credit and coverage, minus risk and cost. Rank is the historical fraction at or below the adjusted operator prior, defaulting to 0.5 for empty history. Lineage credit attributes subsequent gains to preceding actions. Coverage is 0.10[1−no/max⁡(1,max⁡jnj)]0.10[1-n_o/\max(1,\max_j n_j)]. Risk reflects collapse, cycling and edit/evaluation failures; cost is an operator-specific heuristic, set to zero in quality-first mode. Selection takes the eligible argmax.

Equation 7: the action score adds the prior to mean value, visit-count exploration, validity, breakout and novelty, weighted 0.45/0.20/0.10/0.20/0.10, followed by capability adjustments. Exploration is log⁡(N+1)/(no+1)\sqrt{\log(N+1)/(n_o+1)}. Targets are ranked separately using runtime, improvement, novelty and edit-attributed success/failure signals. Width becomes cross-scope when breakout ≥ 0.6; otherwise dependency-cone when uncertainty ≥ 0.55 or progress ≤ 0; otherwise single-region.

Gating: priority is failure recovery → rescue → refinement → cycling → stagnation → early risk/stability. Recovery enables repair/refinement; stagnation enables expansion and disables strict refinement. Rescue requires jump-gain ≥ 0.55 and diff-failure/interface-break rates below 0.35. Only compatible actions enter ranking.

The weights and thresholds are fixed design heuristics. Rank normalization reduces scale sensitivity; lineage credit rewards enabling edits; coverage favors less-used actions; risk discourages repeated failures. Routing, recovery and enforcement ablations evaluate the components, while execution checks verify the implemented rules. These experiments do not individually optimize the coefficients.

Source regions and dependencies are generated automatically in our experiments. AST name-use relations connect definitions to dependent regions; the repository adapter additionally resolves imports. The graph is rebuilt from the selected parent before proposal generation, and the transition guard enforces the resulting editing permissions.

W2. Inconsistent effects across text, table and figure.

The Table 2 comparison is 90.18% − 82.15% = 8.03 percentage points: Method Score Difference vs. Fixed-scope Fixed-scope 82.15% — DyScope 90.18% +8.03 pp w/o structural routing 88.69% +6.54 pp w/o boundary enforcement 85.15% +3.00 pp w/o recovery 89.41% +7.26 pp

The +3.66% statement describes the ablation-versus-Fixed-scope relative improvement calculated before rounding. Figure 6 uses a separate 18-task, three-seed trajectory comparison: panel (d)’s 0.077/0.046 are gains over initialization, differing by approximately 3.10 pp. These are distinct from the full-method final-score difference in Table 2.

The supplement and ScopeBench package are accessible through the anonymous repository, including papers/supplement.pdf. The README describes installation and experiment reproduction.

Reviewer_ffzfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DyScope treats the editable boundary as an explicit state variable in LLM-driven program evolution. A controller chooses a mutation operator, target region, dependency width, and transition guard. Before generation, the scope constrains the proposed edit; afterward, a contract checks the realized changes, frozen regions, and interfaces before semantic evaluation. The intended benefit is to expand the editable region when progress requires coordinated changes and contract it when local editing is sufficient. The evaluation reports benchmark results and a paired ScopeBench study across 18 tasks and three seeds, intended to isolate the effect of dynamic boundary control while sharing the verifier.

Critical strengths
  1. The paper makes edit scope an explicit state variable and separates the requested local edit from the verified realized change. Recording transitions and recovery provenance provides a useful interface for analyzing program-evolution trajectories.

  2. The ScopeBench design is promising: paired tasks and seeds, a shared verifier, and task-level bootstrap uncertainty can isolate boundary-policy effects from verification. If the reported quantities reconcile, it would provide informative evidence.

Critical weaknesses
  1. The main quantitative evidence is not reconciled. Table 2 reports a difference of 8.03 percentage points, the text reports +3.66% with interval [0.00185, 0.07414], and Figure 6 reports 3.10 points; these cannot all describe the same estimand. Table 1 also claims to reproduce SPARK Table 1, but the public v1/v3 values for OpenEvolve/FunSearch DFS are 32.54/74.50 and averages 68.30/77.61, not 20.17/17.25 and 52.52/41.74. This prevents a reliable assessment of the main effect.

  2. The controller and comparison are underspecified. Equations 6–7 lack the operational detail needed to reconstruct the controller instance, and per-benchmark objects, feedback, splits, and budgets are unclear. Figure 5 does not establish matched search cost, and fixed and dynamically expanding regions have different reachable sets, so adaptation is not separated from simply permitting wider edits. A cost-matched fixed-wide control is needed.

  3. HumanEval and MBPP are near-saturated and were introduced in 2021. DyScope's 99.39% on HumanEval ties CodeEvolve and exceeds OpenEvolve by one solved problem out of 164; these benchmarks do not test coordination across dependent regions. Harder coding tasks are needed.

Questions for authors
  1. What runs, weights, and estimands underlie Table 2, the +3.66% effect, and Figure 6? Can you provide the 54 paired scores and the exact source or corrected provenance of the CLRS baselines?

  2. Can you provide the complete reproduction protocol, including the executable controller instance and each benchmark's feedback, split, and budget rules, and a cost-matched fixed-wide or predetermined-expansion control?

  3. Given the near-saturated HumanEval and MBPP results, what improvement does dynamic scope provide over the same proposer under a matched search budget, and how does it perform on harder coding tasks with cross-function or cross-file dependencies?

4 comments

We thank the reviewer for the focused critique of numerical consistency, the missing cost-matched control, and the saturation issue.

Q1.

After checking Table 2, the reported DyScope result is 90.18%, with the following task-family scores:

ScopeBench task familyReported DyScope score
Cache71.27%
Graph routing93.88%
Knapsack99.85%
Scheduling99.57%
Overall (task-weighted mean)90.18%

The overall score weights tasks equally, not the four families equally. The +3.66% figure refers to the relative improvement of an ablation configuration over Fixed-scope, not the full DyScope result. The difference between the reported DyScope and Fixed-scope means is 90.18% − 82.15% = 8.03 percentage points.

Figure 6 reports the Dynamic-scope/Fixed-scope comparison on 18 tasks, three seeds and 20 proposals (54 runs per policy). Panels (a–b) show terminal scores by task family and difficulty; (c) shows terminal-score gain over initialization; (d) shows mean best-so-far gain across proposals, with interquartile bands; (e) shows gain at common cumulative wall-time budgets. The +0.077/+0.046 labels in (d) are gains over initialization, not absolute scores or relative percentages. Panel (f) follows knapsack-L2, seed 47: Dynamic scope rises from 0.7280 to 0.9999, while Fixed scope ends at 0.7280.

The CLRS results are taken from those reported in the SPARK paper. To ensure a fair comparison, we used the same experimental protocol and model for our CLRS experiments.

Q2.

At each search step, the controller selects a parent program and an editable region, then asks the proposer for a SEARCH/REPLACE patch. It checks the actual changes against that region before evaluation; edits outside the boundary are rejected. Evaluation scores and rejection feedback inform the next decision.

The benchmark-specific feedback, evaluation and budget rules are:

  • ScopeBench: the executable two-policy comparison covers 18 tasks, seed 45, 20 proposals and at most 21 evaluations per run, including initialization. Search receives scores and guard feedback, not evaluator source; reported scores use this evaluator without a separate held-out split.
  • CLRS: following SPARK, DeepSeek-R1-0528 makes 100 DFS architecture-edit attempts using training/evaluation feedback. The selected architecture is retrained on the other nine algorithms and scored on official OOD tests.
  • AIME: 30 problems/year (2024–2026); search revises answers and arguments using independent derivation and certificate checks. Frozen answers are evaluated by numerical exact match.
  • HumanEval/MBPP: 164/257 test items; generated Python implementations are evaluated by passing all task tests. The single-attempt controls allow one request per item, no feedback, and freeze candidates before testing.

The paired runs share initialization, proposer, SEARCH/REPLACE representation, parent selection, guard and evaluator. Fixed-scope (the Fixed-local setting) keeps the initial editable region; DyScope adjusts the boundary using search feedback. Both receive 20 proposals.

We observed that max tokens limits could truncate code or patches, interrupting search through parsing/execution failures. We therefore impose no completion-token or cumulative token cap, retaining provider limits and the proposal budget. Recorded usage for 18 tasks, seed 45, 20 proposals per run:

MetricDyScopeFixed-scope
Mean final score90.66%84.02%
Total requests360360
Input tokens3,203,6692,877,853
Output tokens274,037175,694
Total tokens3,477,7063,053,547

DyScope gains 6.63 pp using 13.89% more tokens: equal proposal counts, not equal token consumption.

Q3.

On SWE-bench Pro, representative cases illustrating DyScope's advantages are:

  • OpenLibrary import validation: DyScope changes importapi/code.py and import_validator.py (+37/−13 lines), aligning import handling with revised validation. Fixed-local only changes code.py, leaving validation unchanged.
  • Ansible AIX: DyScope corrects processor fields in aix.py; Fixed-local is locked to linux.py and submits no changes.
  • Ansible BIG-IP: DyScope creates the 526-line bigip_message_routing_route.py module; Fixed-local permits only bigip_device_info.py, excluding that creation, and submits no changes.
  • qutebrowser changelog: DyScope adds VersionChange and version classification in configfiles.py; Fixed-local is locked to configtypes.py and submits no changes.
  • qutebrowser CLI: DyScope integrates argument validation into qutebrowser.py's startup path. Fixed-local adds a helper in browser/commands.py without wiring it into startup parsing.

DyScope's HOLISTIC scopes authorize the cross-file import repair, with accepted edits checked by the transition contract. Its successful patches extend beyond the fixed initial file.

Reviewer_ffzfOpenReview ↗

Follow-up on the Authors' Response

Thank you for your response!

Regarding Q1, we understand your clarification regarding the discrepancies between Table 2 and Figure 6. However, we still find the CLRS baseline results puzzling. After consulting the SPARK paper you mentioned, we found that the DFS OOD accuracy of FunSearch is reported as 74.50, which differs substantially from the value of 17.25 reported in your paper. Therefore, your response does not fully address our concern.

Regarding Q2, we understand and acknowledge your explanation that DyScope achieves better results while using approximately 10% more tokens.

Regarding Q3, we appreciate the five representative cases you provided. However, these examples alone cannot capture the overall performance of the method. We still encourage you to evaluate DyScope on more recent benchmarks.

Thank you again for your response. If a score of 2.5 were available, we would consider increasing our rating to 2.5. However, since the rating can only be increased by one full point at a time, we will maintain our current score of 2.

Thank you for the follow-up. We clarify the CLRS discrepancy and report the completed repository-level evaluations below.

Q1. CLRS baseline results and experimental configuration

The FunSearch DFS accuracy of 74.50% is the value reported in SPARK. Our 17.25% came from local runs with a different experimental configuration. Our earlier description of these entries as SPARK’s reported results was incorrect.

Our local comparison used 20 proposals per task and three seeds (45–47). SPARK uses 100 proposal attempts on DFS, then transfers the selected architecture to the other tasks for training from scratch. Our smaller search budget and different search-and-transfer setup therefore did not reproduce SPARK’s protocol.

The retained local FunSearch DFS results are:

SeedDFS OOD accuracy
459.96%
4620.80%
4721.00%
Mean17.25%

The 11.04-percentage-point spread shows substantial variation across these seeded runs. Reduced search budget and limited replication are plausible contributors to the discrepancy, but these records do not separately identify their effects from training and initialization differences.

For clarity, we provide OpenEvolve and FunSearch’s values directly from SPARK Table 1, alongside the DyScope values in our submission. All entries are OOD accuracy (%), retaining their respective experimental sources.

TaskOpenEvolve (SPARK)FunSearch (SPARK)DyScope (our Table 1)
DFS32.5474.5098.14
AP61.3294.0778.82
QS*3.711.4979.59
SCC45.2162.0884.01
AS95.4395.6199.53
FMS77.0577.7590.76
Dij95.8596.8481.42
LCS87.5382.9388.28
Min98.1097.9185.28
TS86.3192.8888.37
Reported average68.3077.6187.42

SPARK defines QS as Quickselect, whereas our local task is quicksort. This table preserves the reported values and averages; it is not a task- and protocol-matched comparison.

Source: SPARK, Section 4.2 and Table 1.

Q3. Completed evaluations on harder repository tasks

We have now completed all 100 RefactorBench tasks and the 30 SWE-bench Pro V2 tasks fixed before evaluation. These evaluations include every task in the predefined sets, rather than only DyScope’s successful cases.

Within each benchmark, DyScope and Fixed-scope use the same proposer, seed 45, and 20 proposals per task. Final candidates are frozen before official evaluation. All 260 runs are complete.

BenchmarkTasksDyScope resolvedFixed-scope resolvedImprovement
RefactorBench10037/100 (37%)18/100 (18%)+19 pp
SWE-bench Pro V2309/30 (30%)3/30 (10%)+20 pp

RefactorBench uses its official AST-based refactoring checks. SWE-bench Pro uses its official final tests in isolated containers. Fixed-scope freezes the initial one-hop dependency region in RefactorBench and the initially selected file in SWE-bench Pro; DyScope adjusts the editable region during search. The SWE-bench Pro result covers the complete predefined 30-task subset.

The full repository breakdown is:

Benchmark / repositoryTasksDyScope resolvedFixed-scope resolved
RefactorBench / Ansible1154
RefactorBench / Celery1270
RefactorBench / Django1841
RefactorBench / FastAPI631
RefactorBench / Flask631
RefactorBench / Requests1055
RefactorBench / Salt1553
RefactorBench / Scrapy1320
RefactorBench / Tornado933
SWE-bench Pro / Ansible1020
SWE-bench Pro / OpenLibrary1032
SWE-bench Pro / qutebrowser1041

On RefactorBench, 26 tasks are solved only by DyScope, compared with 7 solved only by Fixed-scope. On SWE-bench Pro, the corresponding counts are 7 versus 1. These completed comparisons extend the earlier illustrative cases with evidence of higher repository-level success under the same proposal allowance.

Thank you for identifying these concerns. We address numerical consistency, reproducibility and harder coding tasks below.

W1. Numerical consistency and CLRS provenance.

Table 2 reports 90.18% for DyScope and 82.15% for Fixed-scope: +8.03 percentage points, with equal task weighting. The +3.66% statement instead describes an ablation-versus-Fixed-scope relative improvement calculated before rounding.

Figure 6 presents a separate 18-task, three-seed trajectory comparison. Its 0.077/0.046 values are gains over initialization, differing by approximately 3.10 percentage points, rather than final scores.

For CLRS, SPARK reports FunSearch DFS accuracy of 74.50%, while our 17.25% entry comes from local 20-proposal runs with three seeds: 9.96%, 20.80% and 21.00%. SPARK uses 100 DFS architecture-edit attempts followed by transfer evaluation. Our local results should therefore be identified as local runs, not values reproduced from SPARK’s table. The recovered results show seed variability, but do not isolate how much of the gap comes from search budget versus other configuration differences.

W2. Reproducibility, controller specification and search costs.

The executable controller, configurations, evaluators and reproduction instructions are available in our anonymous repository.

Each step selects a parent and boundary, generates a patch, checks its actual changes against the authorized region, evaluates admissible candidates, and updates the search state. Gates determine eligible actions before ranking; the released implementation specifies the scoring terms, weights, thresholds and update rules.

Benchmark protocols distinguish the following interfaces:

  • ScopeBench: search receives evaluator scores and boundary-check feedback, but not evaluator source. Reporting uses the search evaluator, without a separate held-out split.
  • AIME: search revises answers and arguments; the final answer is frozen before reference matching.
  • HumanEval/MBPP: candidates implement public task interfaces and are evaluated against benchmark tests.
  • CLRS: candidates are reasoning architectures evaluated through training and out-of-distribution testing. Local and published results retain their respective protocols.

The reported ScopeBench results are:

MethodScoreDifference vs. Fixed-scope
Fixed-scope82.15%—
DyScope90.18%+8.03 pp
w/o structural routing88.69%+6.54 pp
w/o boundary enforcement85.15%+3.00 pp
w/o recovery89.41%+7.26 pp

Removing routing, enforcement or recovery reduces DyScope’s score by 1.49, 5.03 and 0.77 pp, respectively.

We omit a user-defined completion-token cap to avoid truncated patches, while retaining the proposal limit and recording actual costs. An additional 18-task, seed-45 comparison completed 20 proposals per method per task:

MetricDyScopeFixed-scope
Mean final score90.66%84.02%
Total requests360360
Input tokens3,203,6692,877,853
Output tokens274,037175,694
Total tokens3,477,7063,053,547

DyScope uses 13.89% more total tokens. This reports actual costs under equal proposal counts; it is not an equal-token-budget comparison. The component ablations assess their contributions but do not alone separate adaptation from always permitting wider edits.

W3. Saturated benchmarks and harder dependent edits.

We completed comparisons on RefactorBench and a predefined 30-task SWE-bench Pro V2 subset, using the same proposer within each benchmark, seed 45 and 20 proposals per method per task.

BenchmarkTasksDyScopeFixed-scope
RefactorBench10037/100 (37.00%)18/100 (18.00%)
SWE-bench Pro V2309/30 (30.00%)3/30 (10.00%)

Under the same 20-proposal budget, DyScope resolves 19 more tasks on RefactorBench and six more tasks on SWE-bench Pro, achieving 2.06× and 3× as many successful repairs, respectively. Dynamic boundaries allow the search to revise its initial editing scope as additional dependencies emerge. These repository-level results support the practical value of adjusting editing permissions during search rather than keeping the initial scope fixed.

These results include every task in the predefined sets, not only successful examples. Final candidates are frozen before official evaluation. The gains of 19 pp on RefactorBench and 20 pp on the SWE-bench Pro subset extend the evidence to repository-level refactoring and repair.

Reviewer_R6dcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DyScope, a method for LLM-driven program evolution in which the region an LLM is allowed to edit is adapted during the search rather than fixed. The method is evaluated against five reference methods on a range of benchmarks, and against a fixed-scope control.

Critical strengths
  • (S1) A large number of benchmark datasets are used to evaluate the method on many tasks and the authors’ method is compared against 5 reference methods, giving a good overview. As reported, DyScope attains the highest average score in each block of Table 1.

  • (S2) The authors also show that the method improves over a fixed-scope control in a matched comparison on ScopeBench, which isolates the effect of the boundary policy.

Critical weaknesses
  • (W1) The experiment design is not explicit enough. It is not clearly explained how the benchmark datasets were evaluated (considering their different tasks) and how the reference methods were configured. This seems especially relevant for the AIME datasets, since it would be important to understand how the problems of these datasets were evaluated within the context of this paper. For reference method configuration, the language model used for them and if it was the same as the one used for DyScope is not clear.

  • (W2) Another issue is that there is no zero-shot/single attempt comparison to judge how well the model performs on these tasks by itself and how saturated it is on them.

Questions for authors
  • (Q1) How were the AIME datasets evaluated and how is the method mapped onto these datasets? What was the starting point and how did the mutations work for these?

  • (Q2) Do all methods use the same language model? It says in the paper "For CLRS-10, the OpenEvolve, FunSearch, and SPARK entries reproduce the values reported in SPARK Table 1", does this mean for CLRS-10 a different LLM was used than in the other experiments and that it differs from DyScope here? Which LLM was used for each method for the other tasks besides CLRS-10?

  • (Q3) Table 1, several methods report the same CLRS-10 accuracy. CodeEvolve, ShinkaEvolve and FunSearch all have 6.41% on AP while CodeEvolve and ShinkaEvolve both have 8.50% on QS. What is the explanation for them having the exact same scores on these tasks?

  • (Q4) What is the explanation for the much lower runtime when adding DyScope to Codex + DeepSeek V4.1 Flash? How many requests and tokens did each approach use?

4 comments

We thank the reviewer for the specific questions about evaluation interfaces, model configuration, and the runtime anomaly. We address each in turn.

Q1.

DyScope revises each problem's answer and mathematical justification, not a shared solver program. Starting only from the public problem statement, the first request generates an initial solution.

The controller can independently re-solve the problem without seeing the first solution, reconcile derivations through evidence checks, or locally repair an incomplete argument. Scope identifies the argument to revise; contract checks examine its structure, arithmetic and answer consistency. Reference answers are used only for final evaluation.

We freeze the final submission and compare its extracted numerical answer with the reference. Correct answers score 1; wrong or unparseable answers score 0. Accuracy is averaged over all 30 problems per year.

Q2.

To fairly compare DyScope with SPARK and the other methods reported in the SPARK paper on CLRS, we used the same model and experimental protocol. Specifically, SPARK, Sections 4–4.2 uses DeepSeek-R1-0528, starts from the official CLRS reference architecture, and performs 100 proposal attempts on DFS. Candidates passing feasibility checks are trained and scored; rejected proposals still consume attempts. The best DFS architecture is then trained from scratch on nine additional CLRS tasks, without task-specific evolution. The reported metric is per-task out-of-distribution accuracy, not direct LLM answer accuracy.

For other benchmarks, we use DeepSeek-V4-Flash:

  • AIME 2024–2026: 30 problems per year; candidate answers and arguments are generated and checked as described in Q1. Final numerical answers are evaluated by exact matching.
  • HumanEval / MBPP: 164 HumanEval test problems and 257 sanitized MBPP test problems. The initial input is the public function prompt or task description and interface. The native adapter generates and revises Python implementations using task-derived checks; final submissions are executed against the benchmark tests. We report the fraction passing all tests.

For these native tasks, generation/checking temperatures are 0.2/0.0. Each problem yields one final submission after verification. No user-defined completion-token cap is imposed, avoiding artificial truncation of solutions.

Q3.

Identical test accuracy does not imply identical programs: the CLRS metric summarizes predictive performance on a fixed evaluation set, rather than uniquely identifying an implementation. Different programs can therefore receive exactly the same score. We observed this directly in recovered FunSearch quicksort evaluation records: 10 distinct candidate hashes at seed 46 all score 4.541016%, and six at seed 47 all score 19.726563%, with successful official evaluations.

The reported AP and QS means also reproduce from the saved per-seed scores, so their repetition is not a transcription or rounding artifact. These records demonstrate how different programs can tie; score equality alone is not evidence of a numerical error. They do not establish which final programs were selected by every method in the table.

Q4.

The displayed pair (91.23%, 269.28 s) mixed pools: quality came from a plain proposer, time from DyScope runs averaging 89.36%, including three quota-limited runs with only 2–3 completed iterations. They should not have been combined; the historical 17.4× attribution is unsupported.

We ran an isolated, serial timing comparison: 18 tasks × 3 seeds per arm (108 runs), identical model, decoding and SEARCH/REPLACE representation. The recorded API usage is:

MetricCodex + DyScopeCodex
Input tokens/run (mean)184,663193,627
Output tokens/run (mean)14,86618,974
Total input tokens9,971,81310,455,844
Total output tokens802,7551,024,588

Means are rounded; totals sum all recorded requests, not rounded means. Input/output counts are API-reported prompt/completion tokens, respectively. DyScope uses 21.65% fewer output tokens without reducing request count.

Mean API time is 82.33 vs. 95.14 s/run: saving 12.81 s (95% task-bootstrap CI [5.99, 19.68]), or 13.46%. End-to-end time is 89.42 vs. 102.56 s/run: saving 13.14 s [6.14, 20.20], or 12.81%. Quality is 90.70% vs. 90.57% (+0.13 pp, CI [−0.48, +1.02]; neither superiority nor equivalence established).

With identical patch representation, boundary policy changes accompany shorter outputs and lower API time; output reduction is a plausible contributor, not an independently identified mediator. This measures the API-proposer harness, not a complete Codex tool loop. Paired records and hashes are in the offline artifact.

I thank the authors for the detailed clarifications and additional experiments. The explanations of the benchmark evaluation procedures, model configurations, and identical CLRS scores address some of my concerns.

However, the AIME evaluation appears to revise mathematical arguments rather than executable programs, making its relation to the proposed program-evolution mechanism less clear.

Also, I appreciate the investigation of the runtime discrepancy, although the combined different experimental pools is concerning and should be corrected in the paper. Additionally, the missing zero-shot/single-attempt comparison (W2) remains unaddressed.

Overall, the responses are helpful, but some concerns about the empirical evaluation remain.

Thank you for identifying the remaining concerns. We address the relationship between AIME and our mechanism, the runtime correction, and the single-attempt baseline directly.

  1. How AIME relates to the proposed mechanism

The connection is through controlled revision: the controller selects which part of a solution to reconsider, generates a revision, and checks whether it satisfies the specified constraints. In AIME, the editable object is an answer and its mathematical argument; in program evolution, it is executable source code.

We agree that these are different applications. AIME evaluates the extension to mathematical reasoning, while evidence for source-region selection, dependency-aware editing, and actual-diff enforcement comes from the program-editing experiments. We will make this distinction explicit in the paper.

We have also completed repository-level comparisons beyond the original coding benchmarks. With the same proposer and 20 proposals per task, DyScope resolves 37/100 RefactorBench tasks versus 18/100 for Fixed-scope, and 9/30 SWE-bench Pro tasks versus 3/30 on the predefined task subset. These evaluations directly test executable repository modifications.

  1. Correcting the runtime comparison

The original quality–time pair combined different experimental pools and should not have appeared as one result. We will correct the corresponding table entry and remove the associated 17.4× speedup claim in the revised paper.

The completed controlled comparison uses 18 tasks × 3 seeds per method, identical patch representation, and isolated serial execution. DyScope requires 89.42 versus 102.56 seconds end-to-end per run, a 12.81% reduction, and 14,866 versus 18,974 output tokens, a 21.65% reduction. Both methods make 20 requests per run. Quality is 90.70% versus 90.57%, with a paired difference of +0.13 pp and a 95% task-bootstrap interval of [−0.48, +1.02] pp. These measurements concern the API-proposer harness rather than a complete Codex agent tool loop.

  1. Single-attempt baseline (W2)

We have completed genuine single-attempt evaluations using DeepSeek-Flash: one generation request per problem, no revision or search feedback, with outputs frozen before final scoring. Benchmark Correct / total Accuracy HumanEval 156/164 95.12% MBPP 194/257 75.49% AIME 2024 23/30 76.67% AIME 2025 21/30 70.00% AIME 2026 22/30 73.33%

For AIME, these results use a uniform post-hoc parsing correction applied to all frozen outputs before reference-answer scoring; no answers were regenerated. The original parser rejected 79/90 outputs, largely because it did not recognize boxed answers. The corrected parser accepts 82/90 outputs; the remaining eight count as incorrect.

These baselines directly quantify the model’s performance without iterative refinement. HumanEval is already near saturation, whereas MBPP and AIME retain substantial headroom. For CLRS, the corresponding no-evolution baseline evaluates the unchanged initial architecture after training and averages 42.55% across ten tasks.

W1. Evaluation interfaces and reference-method settings.

On AIME, DyScope revises a numerical answer and its mathematical justification. Each problem starts from its public statement; the first request generates an initial solution. Subsequent actions independently re-solve the problem, reconcile derivations through evidence checks, or repair an incomplete argument. The independent derivation does not receive the first solution. The final answer is frozen before reference matching; wrong or unparseable answers count as incorrect, with all 30 problems per year in the denominator.

Here, scope controls which part of the mathematical argument is reconsidered. This evaluates the extension of controlled revision to reasoning tasks. Evidence for controlling executable program edits comes from ScopeBench and the repository-editing experiments.

On HumanEval/MBPP, the object is a Python implementation of the public task interface. Final evaluation measures the fraction passing all benchmark tests, over 164/257 items.

For CLRS, the search object is a neural algorithmic reasoning architecture, evaluated by training and measuring official out-of-distribution accuracy. Our provenance audit found that the displayed local OpenEvolve/FunSearch results used 20 proposals and three seeds, whereas SPARK reports 100 DFS architecture-edit attempts followed by transfer to the other algorithms. The local values therefore should not have been described as direct reproductions of SPARK’s published table. For example, SPARK reports FunSearch DFS accuracy of 74.50%; our local three-seed mean is 17.25%. These are different experimental configurations.

On ScopeBench, candidate patches are checked against the authorized source boundary before evaluation. Search receives evaluator scores and guard feedback; the reported score uses the search evaluator, without a separate held-out split. Controlled runs use 20 proposals, with initialization evaluated separately. We record requests and token usage without imposing a user-defined completion-token cap that could truncate patches.

The executable controller, benchmark protocols and reproduction instructions are available in our anonymous repository.

W2. Missing single-attempt baseline.

We completed genuine one-request baselines: each public problem receives one model response, with no search feedback, repair or resampling. Outputs are frozen before evaluation. Generation uses deepseek-flash, temperature 0.2 and thinking disabled. Benchmark Single-attempt result HumanEval 156/164 (95.12%) MBPP 194/257 (75.49%) AIME 2024 23/30 (76.67%) AIME 2025 21/30 (70.00%) AIME 2026 22/30 (73.33%)

For AIME, these scores include a disclosed post-hoc parsing correction applied uniformly to all 90 frozen outputs, without regeneration or reference-guided answer selection. The original parser accepted 11/90 outputs; the corrected extractor accepts 82/90. All eight unparsed outputs count as incorrect, yielding 66/90 overall accuracy (73.33%).

For CLRS, the corresponding no-evolution baseline trains the unchanged initial architecture. All ten algorithms completed, achieving 42.55% mean accuracy under the 300-training-step baseline protocol.

These results directly characterize performance without iterative search and show that substantial headroom remains on MBPP and AIME.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response(1)

Overall Response, Part I: Contribution, Mechanism and Reproducibility We thank the reviewers for recognizing the motivation for adaptive editing boundaries and the importance of verifying actual code changes. The discussion has helped us clarify DyScope’s contribution and strengthen the evidence supporting it.

  1. What DyScope contributes to LLM-driven program evolution

Program evolution requires deciding both what to change and where changes should be permitted. A fixed local boundary can exclude coordinated repairs; opening every region throughout search increases the space of possible modifications. DyScope makes this boundary an explicit search decision that can change with feedback.

Its contribution connects three operations:

  • Before generation: select an operator, target region and dependency width, defining the authorized editing scope.
  • After generation: check the actual patch against that authorization, including protected regions and interfaces.
  • After evaluation: update the search state so that subsequent decisions reflect progress, failures and dependencies. This connection matters because requesting a local edit does not guarantee that the generated code remains local. DyScope makes the boundary operational through candidate admission. It also records the relationship between the selected boundary, realized changes and evaluation outcome, enabling inspection of how editing decisions affect search.

The resulting research contribution is a mechanism for feedback-driven control of the editable search space, combining structural information with enforceable permissions.

  1. How the controller implements this decision

The controller first gates eligible actions, then ranks them and selects a boundary. Equation 6 combines empirical rank, lineage credit and coverage, minus risk and cost. These terms respectively normalize historical signals, reward actions that enable later gains, encourage less-used actions and discourage repeated failures or costly operations.

Equation 7’s action component combines the prior with value, exploration, validity, breakout and novelty. Target selection and dependency width then determine the editable region. Recovery favors repair actions; stagnation enables expansion; refinement favors focused changes. The weights and thresholds are fixed heuristic settings, with their exact values and update rules specified in the released implementation.

This design separates two questions: which actions are appropriate in the current state, and which eligible action should be selected? The transition contract subsequently verifies whether the proposed patch respects that decision.

  1. ScopeBench supports the component contributions

The reported task-weighted results are: Method Score Difference vs. Fixed-scope Fixed-scope 82.15% — DyScope 90.18% +8.03 pp w/o structural routing 88.69% +6.54 pp w/o boundary enforcement 85.15% +3.00 pp w/o recovery 89.41% +7.26 pp

Removing routing, enforcement or recovery reduces the reported score by 1.49, 5.03 and 0.77 percentage points, respectively. These comparisons support the combination of identifying relevant regions, checking actual modifications and recovering from unsuccessful transitions.

The largest reduction among these ablations occurs when enforcement is removed. This supports the practical value of coupling boundary selection with execution-time checks rather than relying entirely on generation instructions.

  1. Automatic construction and reproducibility

Our experiments generate source-region labels automatically, without manual annotation. Python function/class boundaries and segmentation rules identify regions; AST name-use analysis constructs dependencies. The repository adapter additionally resolves imports. The graph is rebuilt from the selected parent before proposal generation, allowing subsequent decisions to reflect accepted changes.

The implementation makes the method inspectable at three levels: controller decisions, actual patch authorization and evaluator outcomes. The released materials provide the controller, configurations, benchmark adapters, ScopeBench package and reproduction instructions.

The anonymous repository also contains the supplementary PDF at papers/supplement.pdf. These materials connect the mathematical description to executable behavior and benchmark-specific protocols.

DyScope therefore contributes both an adaptive editing mechanism and an implementation through which its decisions can be reproduced and examined.

AuthorsOpenReview ↗

General Response(2)

Overall Response, Part II: Additional Evidence and Manuscript Corrections Following the reviewers’ requests, we extended the evaluation beyond function-level generation and measured efficiency against unrestricted legal-region editing. These experiments strengthen the evidence for DyScope’s practical contribution.

  1. Complete comparisons on harder repository tasks

We evaluated all 100 RefactorBench tasks and a predefined 30-task SWE-bench Pro V2 subset. Each comparison uses the same proposer within the benchmark, seed 45 and 20 proposals per method per task. Benchmark DyScope Fixed-scope Improvement RefactorBench 37/100 (37.00%) 18/100 (18.00%) +19 pp SWE-bench Pro V2 9/30 (30.00%) 3/30 (10.00%) +20 pp

DyScope resolves 19 additional RefactorBench tasks and six additional SWE-bench Pro tasks, achieving 2.06× and 3× as many successful repairs, respectively. These comparisons cover every task in the predefined evaluation sets, with final candidates frozen before official evaluation.

Fixed-scope retains the initial one-hop dependency region on RefactorBench and the initially selected file on SWE-bench Pro. DyScope can revise these permissions during search. The results support the value of allowing editing decisions to evolve as additional dependencies become relevant.

A concrete example is the OpenLibrary Wikisource repair, which coordinates changes across importapi/code.py and import_validator.py. The aggregate results extend this case-level evidence to complete task-set comparisons.

  1. Efficiency when wider edits are already permitted

To examine whether adaptive boundaries offer value beyond removing local restrictions, we conducted an isolated serial comparison against Fixed-wide, which permits all legal source regions throughout search.

The comparison covers 18 tasks × three seeds × 20 proposals, with identical model settings and SEARCH/REPLACE patch representation: Metric DyScope Fixed-wide Mean quality 90.70% 90.57% End-to-end time/run 89.42 s 102.56 s Output tokens/run 14,866 18,974 Input tokens/run 184,663 193,627 Requests/run 20 20

The quality difference is +0.13 pp (95% task-bootstrap CI [−0.48, +1.02]). DyScope uses 12.81% less time and 21.65% fewer output tokens. The paired time saving is 13.14 seconds, with a 95% CI of [6.14, 20.20].

Both methods generate patches and make the same number of requests. Thus, the measured advantage concerns resource consumption under the same proposal budget, with similar observed quality. Shorter generated edits are a plausible contributor to this efficiency.

  1. Performance without iterative search

We also completed genuine one-request baselines, without feedback, repair or resampling. Results are 95.12% on HumanEval, 75.49% on MBPP, and 23/30, 21/30 and 22/30 on AIME 2024–2026.

The AIME results use a disclosed uniform parsing correction on frozen responses; all remaining unparseable answers count as incorrect. The unchanged CLRS architecture achieves 42.55% mean accuracy under the 300-training-step baseline protocol. These measurements directly address the request to characterize performance without evolution.

  1. Corrections committed for the revised manuscript

We will make the numerical definitions and experimental provenance explicit:

  • ScopeBench: distinguish the full-method difference of 8.03 pp, the ablation’s 3.66% relative improvement, and Figure 6’s approximately 3.10 pp difference in gains over initialization.
  • CLRS: separate local results from SPARK’s published values. FunSearch’s 17.25% local mean comes from 20-proposal, three-seed runs; SPARK reports 74.50% under a different search-and-transfer protocol. We will correct the attribution and document the verified model, search and training configurations.
  • Runtime: correct the historical row that combined quality and time from different pools, replacing its acceleration interpretation with the controlled measurements above, explicitly identified as an API-proposer experiment.
  • Reporting: distinguish published comparisons, local runs and controlled experiments, with their budgets, seeds, evaluation rules and statistical definitions. The strengthened evidence shows two practical benefits: more successful repository edits than fixed initial boundaries, and lower measured generation cost than continuously opening all legal regions in the isolated comparison. The revised manuscript will present these contributions with consistent numerical definitions and clearly attributed experimental protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.