- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes DyScope, a framework for LLM-driven program evolution that dynamically adjusts the editing scope based on search feedback. It uses a structural dependency graph to select scopes to edit and a transition contract to check whether generated modifications respect the selected boundaries. The authors evaluate DyScope on reasoning and coding benchmarks and introduce ScopeBench to compare dynamic scope control against baselines and the ablated versions, and show the superior performance of DyScope.
Critical strengths
-
The motivation is reasonable: restricting edits to one region can prevent useful changes that require modifying multiple functions, while allowing unrestricted edits can make search less efficient. This scope selection trade-off is worth exploring.
-
The method checks the actual code changes against the selected scope rather than relying solely on prompt instructions. This helps ensure that the generated edits follow the controller’s decisions.
Critical weaknesses
-
The core method is not described clear enough to reproduce. Equations 6 and 7 introduce scores for selecting operators and editing boundaries, but the paper does not explain how several terms are computed or how the gating rules are implemented. Moreover, the paper does not motivate why the formula is designed this way and where do the parameters come from. These decisions are central to the method, so it is difficult to assess what actually contributes to the reported improvements.
-
The main experimental results are inconsistent across the text, tables, and figures. Table 2 reports 90.18% for DyScope and 82.15% for fixed scope, a difference of 8.03 percentage points, but lines 356–357 report a 3.66% improvement. Figure 6(d) shows gains over initialization of 0.077 and 0.046, whose difference is only 0.031.
Questions for authors
-
How are the terms in Equation 6 and the boundary score in Equation 7 computed? Please specify the gating rules and how the weights and thresholds were chosen. What evidence supports these design choices?
-
Can you explain the inconsistencies in weaknesses?
-
Does the fixed-scope baseline permit all edits needed to solve each ScopeBench task? Have you compared DyScope with a baseline that allows whole-program editing throughout search? This distinction matters for determining whether the benefit comes from adapting scope or simply removing restrictions imposed on the baseline.
-
How is the scope graph constructed in the experiments? Please specify the node and edge definitions, the programming languages and analysis tools used, whether annotations are manually provided, and whether the graph is updated after each program modification. Dependency analysis methods could vary across different programming languages and package management environments, and could be imprecise.
-
Is there a public repository for the code and artifacts? Since some implementation details are not specified in the paper, the code could help future researchers understand better how this is implemented.
-
The reproducibility statement refers to a supplement containing the full protocol, policy settings, and additional results, but I could not find it on the submission page or in the paper PDF. Could the authors clarify where this material is available?
5 comments
Thank you for the detailed response. I will take this rebuttal into account.
W1. Controller definition, parameter choices and reproducibility.
DyScope first gates eligible actions, then ranks them, chooses an editable boundary, and verifies the actual patch before evaluation. Code, configurations and reproduction instructions are available in our anonymous repository.
Equation 6: the score combines empirical rank, lineage credit and coverage, minus risk and cost. Rank is the historical fraction at or below the adjusted operator prior, defaulting to 0.5 for empty history. Lineage credit attributes subsequent gains to preceding actions. Coverage is . Risk reflects collapse, cycling and edit/evaluation failures; cost is an operator-specific heuristic, set to zero in quality-first mode. Selection takes the eligible argmax.
Equation 7: the action score adds the prior to mean value, visit-count exploration, validity, breakout and novelty, weighted 0.45/0.20/0.10/0.20/0.10, followed by capability adjustments. Exploration is . Targets are ranked separately using runtime, improvement, novelty and edit-attributed success/failure signals. Width becomes cross-scope when breakout ≥ 0.6; otherwise dependency-cone when uncertainty ≥ 0.55 or progress ≤ 0; otherwise single-region.
Gating: priority is failure recovery → rescue → refinement → cycling → stagnation → early risk/stability. Recovery enables repair/refinement; stagnation enables expansion and disables strict refinement. Rescue requires jump-gain ≥ 0.55 and diff-failure/interface-break rates below 0.35. Only compatible actions enter ranking.
The weights and thresholds are fixed design heuristics. Rank normalization reduces scale sensitivity; lineage credit rewards enabling edits; coverage favors less-used actions; risk discourages repeated failures. Routing, recovery and enforcement ablations evaluate the components, while execution checks verify the implemented rules. These experiments do not individually optimize the coefficients.
Source regions and dependencies are generated automatically in our experiments. AST name-use relations connect definitions to dependent regions; the repository adapter additionally resolves imports. The graph is rebuilt from the selected parent before proposal generation, and the transition guard enforces the resulting editing permissions.
W2. Inconsistent effects across text, table and figure.
The Table 2 comparison is 90.18% − 82.15% = 8.03 percentage points: Method Score Difference vs. Fixed-scope Fixed-scope 82.15% — DyScope 90.18% +8.03 pp w/o structural routing 88.69% +6.54 pp w/o boundary enforcement 85.15% +3.00 pp w/o recovery 89.41% +7.26 pp
The +3.66% statement describes the ablation-versus-Fixed-scope relative improvement calculated before rounding. Figure 6 uses a separate 18-task, three-seed trajectory comparison: panel (d)’s 0.077/0.046 are gains over initialization, differing by approximately 3.10 pp. These are distinct from the full-method final-score difference in Table 2.
The supplement and ScopeBench package are accessible through the anonymous repository, including papers/supplement.pdf. The README describes installation and experiment reproduction.
We thank the reviewer for the detailed concerns about reproducibility, numerical consistency, and the scope-graph construction. We address each question directly.
Q1.
In Equation 6, score = empirical rank + lineage credit + coverage − risk − cost. Rank is the historical fraction ≤ the adjusted operator prior (0.5 without history); coverage is 0.10[1 − n_o/max(1, max_j n_j)], using lineage counts. Risk reflects search/edit failures; cost is operator-specific and zero in quality-first mode. Selection takes the eligible argmax.
In Equation 7, the action component adds the prior to mean value, exploration, validity, breakout and novelty, weighted 0.45/0.20/0.10/0.20/0.10, followed by capability adjustments. Exploration is sqrt(log(N+1)/(n_o+1)), using visit counts. Width is cross-scope if breakout ≥ 0.6; otherwise dependency-cone if uncertainty ≥ 0.55 or progress ≤ 0; otherwise single-region.
Before ranking, gating priority is recovery → rescue → refinement → cycling → stagnation → early risk; only stage-compatible operators qualify. Rescue requires jump-gain ≥ 0.55 and diff-failure/interface-break rates both < 0.35.
Weights and thresholds are manually set heuristics balancing improvement, exploration and reliability. Routing, recovery and enforcement ablations assess these component choices; recorded transitions verify rule execution. They do not establish coefficient optimality.
Q2.
Table 2 reports the following scores and differences from Fixed-scope (differences calculated from the displayed means):
The +3.66% figure is the relative improvement of the boundary-enforcement ablation over Fixed-scope, calculated before rounding, not the full DyScope improvement of 8.03 percentage points.
Figure 6 compares Dynamic scope and Fixed scope on 18 tasks, three seeds and 20 proposals. Panels (a–b) show terminal scores; (c–d) show gains over initialization; (e) shows gains at common cumulative time budgets. The +0.077/+0.046 labels are gains over initialization, not absolute scores or relative percentages. Panel (f) follows knapsack-L2, seed 47: Dynamic scope rises from 0.7280 to 0.9999, while Fixed scope ends at 0.7280.
Q3.
Fixed-scope allows edits only within the region selected at initialization. If a proposed repair also changes a helper outside that region, it is rejected. DyScope can expand the boundary to allow both changes; Fixed-wide permits all legal regions throughout search.
At the same 20-proposal budget, DyScope achieves comparable quality to Fixed-wide (90.70% vs. 90.57%, +0.13 pp, 95% CI [−0.48, +1.02]) with 12.81% less end-to-end time (89.42 vs. 102.56 s/run) and 21.65% fewer output tokens (14,866 vs. 18,974/run). This isolated comparison covers 18 tasks × three seeds with identical model and patch representation. DyScope's advantage is therefore comparable quality at lower time and output-token costs under the same proposal budget.
Q4.
Nodes are source regions with locations, nesting and mutability information. Our experiments generate region labels automatically, without manual annotation, using Python function/class boundaries and configured naming heuristics. Existing labels are retained.
Edges combine declared dependencies with AST name-use relations: if A defines a name used in B, an edge B → A records the dependency. The repository adapter additionally resolves cross-file imports, including relative imports. Dependency inference uses static analysis, not an LLM.
The graph is rebuilt from the selected parent's current source before each proposal, so subsequent proposals reflect its edits. Parsing failures omit inferred edges but preserve declared relations; untagged non-Python inputs fall back to a whole-block region. Tests verify dependency removal and cross-file import edges.
Q5–Q6.
The code and supplementary material are available in our anonymous repository: https://anonymous.4open.science/r/dyscope-FC8B/
It contains the DyScope controller, benchmark adapters and protocol configurations, and the complete ScopeBench package: all 18 tasks, initial programs, evaluators and dataset documentation. The README explains the file structure, installation, executable examples and experiment reproduction, including feedback and budget rules. All 18 ScopeBench initial-program checks and 20 offline tests pass. The paper and supplement are available under
papers/dyscope.pdfandpapers/supplement.pdf; ScopeBench is also included as a standalone ZIP.