- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a method for software agents that code iteratively under adapting requirements and proposes a training method specifically for adapting, changing requirements. The authors show on a benchmark that evaluates adaptation of the code to the new requirements (SlopCodeBench) that their method beats previous standard methods.
Critical strengths
- The quality of the code and extensibility is an important and underexplored area both for research and for applications.
- the method is simple and aligned well with the motivation.
- The paper is well written and easy to follow.
Critical weaknesses
- The results improvement over baselines are not that high.
- The ablations are pretty minimal.
- It is hard to isolate the effect of each method due to lack of analysis.
Questions for authors
- Can you create a comparison to a baseline trained on your dataset with "greedy" reward to understand how much of the performance gain is from the fine-tuning itself and how much is from the method design for a long-horizon task?
4 comments
Response to W(2-3), Q1 1/3
Ablation Experiments
Full Ablation Matrix
We designed 7 ablation arms. All experiments use identical experimental configurations, differing only in the training data type and reward signal design. Each arm is independently evaluated three times on both benchmarks, and the mean results are reported.
The training data comprise two types: isolated single-issue tasks and Consequence Chains involving multi-step sequential development. The reward signals are organized into three levels according to the amount of downstream development consequence information they contain: (1) correctness-only rewards, including whole-chain multiplicative correctness and step-wise average correctness as two credit assignment schemes; (2) correctness rewards combined with an uncalibrated judge; and (3) correctness rewards combined with a calibrated judge, further distinguishing between a Step-local Judge that uses only current-step evidence and a Hindsight Judge that uses complete-chain execution evidence.
We distinguish between the judge and the rubric: the judge is the model that assigns scores, while the rubric consists of the scoring instructions provided to it. All judge-based arms use the same judge model and the same reward formulation, , differing only in the rubric and the evidence window.
Evaluation Metrics
Current-step correctness (the ability to solve the immediate task)
- SCB strict d1–8 (/196 checkpoints): A checkpoint is counted as solved only if all tests pass, measuring the ability to complete the current step with all tests passing. For the 35B model, all strict solves occur at depths 1–3, while all seven arms consistently achieve zero strict solves across the 88 checkpoints at depths 4–8. Therefore, this metric reflects only current-step capability.
- ChainSWE per-bug (/304 steps): The per-bug resolution rate, measuring average single-step correctness.
Long-horizon capability (whether improvements transfer downstream)
- SCB core d4–8 (/88 checkpoints): Evaluates only deeper requirements (steps 4–8). These requirements build on code previously written by the agent itself and extend beyond the three-step training horizon.
- ChainSWE complete-chain (/100 chains): The fraction of chains successfully completed in their entirety in a single sequential run.
Overall performance
- SCB core d1–8 (/196 checkpoints): Core-test success across all depths 1–8.
Full Ablation Results
Qwen3.6-35B. Each cell reports the mean over three evaluations. Differences are computed as arm − base.
| Arm | Data | Judge | Strict d1–8 (current-step) | Per-bug (current-step) | Core d4–8 (long-horizon) | Complete-chain (long-horizon) | Core d1–8 (overall) |
|---|---|---|---|---|---|---|---|
| Base | — | — | 3.40 | 40.02 | 22.35 | 13.00 | 30.44 |
| Whole-chain multiplicative correctness | Chain | None | +0.17 (0.93) | +0.66 (0.64) | +0.76 (0.79) | −0.67 (0.63) | +1.19 (0.56) |
| Chain-level step-wise correctness | Chain | None | +0.51 (0.086) | −0.33 (0.82) | +0.38 (0.77) | +0.67 (0.72) | +1.36 (0.43) |
| Single-step SWE | Single issue | None | +1.70 (0.009) | +0.33 (0.74) | +0.76 (0.68) | −0.33 (0.84) | +1.36 (0.47) |
| A1 | Chain | Uncalibrated rubric + full-chain evidence | +2.04 (p<0.001) | −0.66 (0.60) | +0.38 (0.82) | +2.33 (0.16) | +1.87 (0.38) |
| A2 | Chain | Calibrated rubric + step-local evidence | +0.68 (0.38) | +1.64 (0.21) | +3.79 (0.057) | −0.00 (0.97) | +2.89 (0.17) |
| Ours | Chain | Calibrated rubric + full-chain evidence | +2.38 (p<0.001) | +1.64 (0.17) | +5.68 (0.008) | +3.67 (0.033) | +4.59 (0.011) |
Ours is the only arm that significantly outperforms the base model on all four metrics and the only arm that achieves significant improvements on both long-horizon metrics. The 95% confidence intervals are provided in the supplementary table at the end.
Response to W(2-3), Q1 2/3
Direct Paired Comparisons Between Ours and Each Control Arm
We assess statistical distinguishability using direct paired comparisons between Ours and each control arm, rather than inferring differences from whether each arm is individually significant relative to the base model.
| Control Arm | Strict d1–8 | Per-bug | Core d4–8 | Complete-chain | Chain-wise exact test | Core d1–8 |
|---|---|---|---|---|---|---|
| Whole-chain multiplicative correctness | +2.21 (p<0.001) | +0.99 (0.51) | +4.92 (0.15) | +4.33 (0.038) | 15:4 (0.019) | +3.40 (0.076) |
| Chain-level step-wise correctness | +1.87 (0.0033) | +1.97 (0.11) | +5.30 (0.029) | +3.00 (0.077) | 11:5 (0.210) | +3.23 (0.088) |
| Single-step SWE | +0.68 (0.24) | +1.32 (0.15) | +4.92 (0.14) | +4.00 (0.025) | 13:5 (0.096) | +3.23 (0.13) |
| A1 (uncalibrated rubric) | +0.34 (0.63) | +2.30 (0.015) | +5.30 (0.047) | +1.33 (0.25) | 7:3 (0.344) | +2.72 (0.15) |
| A2 (calibrated rubric + step-local evidence) | +1.70 (0.063) | −0.00 (0.96) | +1.89 (0.53) | +3.67 (0.062) | 15:4 (0.019) | +1.70 (0.45) |
Two observations emerge. First, Ours is statistically distinguishable from each of the five control arms on at least one metric. Second, the metrics on which the control arms fall short differ: although several arms match Ours on individual metrics, no single control arm matches Ours across all metrics.
Correctness-Only Pool
The three correctness-only approaches—whole-chain multiplicative correctness, chain-level step-wise correctness, and single-step SWE—correspond to the same treatment condition: the reward contains only correctness information from executable tests. These three approaches are pairwise statistically indistinguishable on the following four metrics (all p ≥ 0.38). We therefore combine them into a control pool comprising nine evaluations.
| Metric | Correctness-only pool vs Base | A1+A2 pool vs Base | Ours vs Base | Ours vs Correctness-only pool |
|---|---|---|---|---|
| Per-bug (current-step) | +0.22 (0.84) | +0.49 (0.63) | +1.64 (0.17) | +1.43 (0.18) |
| SCB core d4–8 (long-horizon) | +0.63 (0.68) | +2.08 (0.20) | +5.68 (0.008) | +5.05 [+0.33, +10.50] (0.038) |
| ChainSWE complete-chain (long-horizon) | −0.11 (0.96) | +1.17 (0.44) | +3.67 (0.033) | +3.78 [+0.67, +7.22] (0.020) |
| SCB core d1–8 (overall) | +1.30 (0.39) | +2.38 (0.19) | +4.59 (0.011) | +3.29 [+0.22, +6.49] (0.037) |
Correctness-only RL shows no statistically significant difference from the base model on any of the four metrics, with essentially zero improvements on the two long-horizon metrics (d4–8: +0.63; complete-chain: −0.11). In contrast, Ours is statistically distinguishable from the correctness-only pool on three metrics.
Ablation Conclusions
1. The gains arise from the J term in the reward, and J requires chain-based calibration to transfer improvements across requirement boundaries
Ours and A1 differ in only one aspect: the scoring rubric. Ours uses the calibrated rubric, whereas A1 uses the rubric before calibration. The judge model, reward formulation , training data, judge evidence window (full-chain), computational budget, and evaluation protocol are all identical.
The two arms are statistically distinguishable on the long-horizon core d4–8 metric (+5.30 [+0.33, +11.24], p=0.047) and on current-step per-bug correctness (+2.30 [+0.44, +4.33], p=0.015). A gain of the same magnitude appears in another comparison: Ours also outperforms the chain-level step-wise correctness arm by +5.30 [+0.70, +10.59] on core d4–8 (p=0.029).
Thus, what calibration provides is not simply stronger overall performance, but the ability to carry improvements across requirement boundaries. Completing the immediate step with all tests passing does not inherently depend on rubric quality. On strict solved, Ours (5.78), A1 (5.44), and even single-step SWE without any judge (5.10) perform comparably (all pairwise p ≥ 0.23). On this metric, any correctness-based RL approach can achieve gains: both A1 (+2.04) and single-step SWE (+1.70) significantly outperform the base model.
The effect of calibration is not apparent on strict solved; it becomes visible only when crossing requirement boundaries. With an uncalibrated rubric, A1 achieves only +0.38 on core d4–8 (p=0.82), essentially remaining at the base-model level, whereas Ours achieves +5.68.
Response to W(2-3), Q1 2/3
2. Full-chain evidence during training corresponds to successfully completing the entire chain
Ours and A2 also differ in only one aspect: the length of the evidence window available to the judge. Both use the same calibrated rubric, but A2 restricts the judge's evidence to the current step.
A2 statistically matches Ours on the deeper long-horizon metric (core d4–8: +1.89, p=0.53) and completely matches Ours on current-step per-bug correctness (−0.00, p=0.96). However, its complete-chain improvement disappears: A2 achieves a complete-chain success rate of 13.00, identical to the base model. In contrast, the chain-wise exact test comparing Ours against A2 gives 15:4, p=0.019 (bootstrap: +3.67, p=0.062).
3. The two roles of Consequence Chains
The first role is to provide the raw material for calibrating long-horizon reward criteria. The calibration procedure in Section 3 uses the signal that two implementations of the same chain, both passing CP1, diverge in their subsequent development outcomes. The frequency of such events is directly determined by the structural coupling within the chains.
The second role is to provide full-chain execution evidence during training.
These two roles are identified by Conclusions 1 and 2, respectively. Their contributions are distinct: the calibrated rubric drives generalization beyond the training horizon and improves per-step correctness, while full-chain evidence during training drives successful completion of entire chains in a single run. Both components are needed to achieve improvements across all four metrics simultaneously.
4. Standard SWE correctness-based RL cannot explain the long-horizon gains
The three compute-matched correctness-only approaches—chain data with whole-chain multiplicative correctness gating, chain data with step-wise average correctness, and single-issue data with binary correctness—show no statistically significant differences from the base model on either long-horizon metric.
For core d4–8, their improvements are +0.76 / +0.38 / +0.76 (all p ≥ 0.68), while for complete-chain success, they are −0.67 / +0.67 / −0.33 (all p ≥ 0.63). In contrast, Ours achieves improvements of +5.68 and +3.67, respectively.
The three correctness-only approaches are also statistically indistinguishable from one another. Despite using three different correctness credit assignment schemes, their core d4–8 improvements fall within a narrow 0.4-percentage-point range (all pairwise p ≥ 0.84). Whole-chain multiplicative correctness and the single-step arm achieve exactly the same core d4–8 improvement (+0.76), while whole-chain multiplicative correctness and chain-level step-wise correctness differ by only +0.17 on core d1–8 (p=0.95).
Changing the correctness signal from a whole-chain product to a step-wise average does not alter the outcome. What is missing is not denser correctness credit assignment, but the J term.
Supplementary Table: 95% Confidence Intervals for the Full Ablation Results
| Arm | Strict d1–8 | Per-bug | Core d4–8 | Complete-chain | Core d1–8 |
|---|---|---|---|---|---|
| Whole-chain multiplicative correctness | [−0.51, +1.00] | [−1.98, +3.61] | [−5.43, +6.87] | [−4.00, +3.00] | [−2.98, +5.47] |
| Chain-level step-wise correctness | [+0.00, +1.12] | [−2.99, +2.21] | [−2.47, +2.93] | [−2.67, +4.00] | [−1.99, +4.76] |
| Single-step SWE | [+0.34, +3.44] | [−1.66, +2.39] | [−3.70, +5.30] | [−3.33, +2.67] | [−2.31, +5.15] |
| A1 | [+0.83, +3.45] | [−3.08, +1.66] | [−4.17, +4.44] | [−0.67, +5.33] | [−2.36, +6.15] |
| A2 | [−0.50, +2.13] | [−0.87, +4.47] | [−0.34, +7.97] | [−3.67, +3.67] | [−1.34, +7.09] |
| Ours | [+1.05, +3.84] | [−0.75, +4.02] | [+1.35, +10.42] | [+0.33, +7.00] | [+1.02, +8.38] |
Statistical protocol: We use paired bootstrap with clustering by program (SCB, 36 units) or by chain (ChainSWE, 100 chains), with B=200,000. The reported p-value is the median across five random seeds, while the confidence interval is taken from the first seed.
Response to W1
We thank the reviewer for this suggestion and have adopted it. Every number in the main table is now the mean over three independent repeated evaluations; we additionally report the standard deviation, the 95% confidence interval, and a paired test for each cell. The complete results will be included in the revised manuscript; a summary is given below.
Main-table gains with 95% confidence intervals and paired-test p-values, each computed over three independent repeated evaluations.
All nine main cells move in the positive direction, and seven of them are significant at the two-sided 0.05 level. The two non-significant cells are both floor effects on SCB strict for the 2B model: averaged over the three repeats, the base solves 0.67 out of 196 instances and ours solves 2.33 out of 196, so the absolute counts are too small for the difference to reach significance even though the direction is consistent.