acceptodds
Under review as a conference paper at ICLR 2027

Training Under an Unknown Future: Long-Horizon Task Agents That Leave Room for What Comes Next

Abstract

Agents are increasingly asked to implement successive requirements over long horizons, and a change focused solely on the current requirement may make later ones infeasible or substantially more expensive to implement. Existing training does not address this failure: rewards are assigned to the current task before downstream consequences become observable. On SlopCodeBench, which evaluates three to eight sequential requirements while requiring all earlier tests to keep passing, none of the fifteen evaluated frontier agents solves a complete problem. We propose a training method built on consequence chains mined from real development histories. Each chain links an early structural decision to later requirements that expose its downstream effects. We train on three-requirement chains, a practical length for exposing these effects. The objective first requires whole-chain test correctness, then refines the reward with a model-based judge's assessment of downstream impact, paying the model for writing code future steps can build on. On ChainSWE, our method improves the complete-chain success rate on the agent's own accumulated patches by at least relative to the corresponding base models, across scales from B to B. Our B model achieves a sequential bug-resolution rate of on ChainSWE, compared with for Claude Opus 4.6 under the same harness. On SlopCodeBench, checkpoint-level improvements extend beyond the three-step training horizon: on sequences of up to eight steps, the trained B model improves core solve rate by percentage points at steps four through eight, compared with points over the first three.

Primary area:
reinforcement learning
Keywords:
Agentic RL, Multi Turn Long Horizon Task, Self Evolving Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7n6kOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a method for software agents that code iteratively under adapting requirements and proposes a training method specifically for adapting, changing requirements. The authors show on a benchmark that evaluates adaptation of the code to the new requirements (SlopCodeBench) that their method beats previous standard methods.

Critical strengths
  • The quality of the code and extensibility is an important and underexplored area both for research and for applications.
  • the method is simple and aligned well with the motivation.
  • The paper is well written and easy to follow.
Critical weaknesses
  • The results improvement over baselines are not that high.
  • The ablations are pretty minimal.
  • It is hard to isolate the effect of each method due to lack of analysis.
Questions for authors
  • Can you create a comparison to a baseline trained on your dataset with "greedy" reward to understand how much of the performance gain is from the fine-tuning itself and how much is from the method design for a long-horizon task?
4 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for this suggestion and have adopted it. Every number in the main table is now the mean over three independent repeated evaluations; we additionally report the standard deviation, the 95% confidence interval, and a paired test for each cell. The complete results will be included in the revised manuscript; a summary is given below.

Main-table gains with 95% confidence intervals and paired-test p-values, each computed over three independent repeated evaluations.

ModelMetricbase → oursdiff [95% CI]two-sided p
MiniCPM5-2BChainSWE (whole chain)3.67 → 9.00+5.33 [+2.00, +9.00]<0.001
MiniCPM5-2BChainSWE (per-bug)27.52 → 31.14+3.62 [+1.43, +5.90]<0.001
MiniCPM5-2BSCB strict0.34 → 1.19+0.85 [+0.00, +1.94]0.087
Qwen3.6-35B-A3BChainSWE (whole chain)13.00 → 16.67+3.67 [+0.33, +7.00]0.033
Qwen3.6-35B-A3BChainSWE (per-bug)40.02 → 41.67+1.64 [−0.75, +4.02]0.174
Qwen3.6-35B-A3BSCB strict3.40 → 5.78+2.38 [+1.05, +3.84]<0.001
Qwen3.8-27BChainSWE (whole chain)14.00 → 18.33+4.33 [+1.00, +8.33]0.0062
Qwen3.8-27BChainSWE (per-bug)43.75 → 47.04+3.29 [+1.30, +5.54]<0.001
Qwen3.8-27BSCB strict9.86 → 14.12+4.25 [+1.37, +7.50]0.0024

All nine main cells move in the positive direction, and seven of them are significant at the two-sided 0.05 level. The two non-significant cells are both floor effects on SCB strict for the 2B model: averaged over the three repeats, the base solves 0.67 out of 196 instances and ours solves 2.33 out of 196, so the absolute counts are too small for the difference to reach significance even though the direction is consistent.

AuthorsOpenReview ↗

Response to W(2-3), Q1 1/3

Ablation Experiments

Full Ablation Matrix

We designed 7 ablation arms. All experiments use identical experimental configurations, differing only in the training data type and reward signal design. Each arm is independently evaluated three times on both benchmarks, and the mean results are reported.

The training data comprise two types: isolated single-issue tasks and Consequence Chains involving multi-step sequential development. The reward signals are organized into three levels according to the amount of downstream development consequence information they contain: (1) correctness-only rewards, including whole-chain multiplicative correctness and step-wise average correctness as two credit assignment schemes; (2) correctness rewards combined with an uncalibrated judge; and (3) correctness rewards combined with a calibrated judge, further distinguishing between a Step-local Judge that uses only current-step evidence and a Hindsight Judge that uses complete-chain execution evidence.

We distinguish between the judge and the rubric: the judge is the model that assigns scores, while the rubric consists of the scoring instructions provided to it. All judge-based arms use the same judge model and the same reward formulation, R=A+0.4 AJR = A + 0.4\,AJ, differing only in the rubric and the evidence window.

Evaluation Metrics

Current-step correctness (the ability to solve the immediate task)

  • SCB strict d1–8 (/196 checkpoints): A checkpoint is counted as solved only if all tests pass, measuring the ability to complete the current step with all tests passing. For the 35B model, all strict solves occur at depths 1–3, while all seven arms consistently achieve zero strict solves across the 88 checkpoints at depths 4–8. Therefore, this metric reflects only current-step capability.
  • ChainSWE per-bug (/304 steps): The per-bug resolution rate, measuring average single-step correctness.

Long-horizon capability (whether improvements transfer downstream)

  • SCB core d4–8 (/88 checkpoints): Evaluates only deeper requirements (steps 4–8). These requirements build on code previously written by the agent itself and extend beyond the three-step training horizon.
  • ChainSWE complete-chain (/100 chains): The fraction of chains successfully completed in their entirety in a single sequential run.

Overall performance

  • SCB core d1–8 (/196 checkpoints): Core-test success across all depths 1–8.

Full Ablation Results

Qwen3.6-35B. Each cell reports the mean over three evaluations. Differences are computed as arm − base.

ArmDataJudgeStrict d1–8 (current-step)Per-bug (current-step)Core d4–8 (long-horizon)Complete-chain (long-horizon)Core d1–8 (overall)
Base——3.4040.0222.3513.0030.44
Whole-chain multiplicative correctness a1a2a3a_1a_2a_3ChainNone+0.17 (0.93)+0.66 (0.64)+0.76 (0.79)−0.67 (0.63)+1.19 (0.56)
Chain-level step-wise correctness (a1+a2+a3)/3(a_1+a_2+a_3)/3ChainNone+0.51 (0.086)−0.33 (0.82)+0.38 (0.77)+0.67 (0.72)+1.36 (0.43)
Single-step SWESingle issueNone+1.70 (0.009)+0.33 (0.74)+0.76 (0.68)−0.33 (0.84)+1.36 (0.47)
A1ChainUncalibrated rubric + full-chain evidence+2.04 (p<0.001)−0.66 (0.60)+0.38 (0.82)+2.33 (0.16)+1.87 (0.38)
A2ChainCalibrated rubric + step-local evidence+0.68 (0.38)+1.64 (0.21)+3.79 (0.057)−0.00 (0.97)+2.89 (0.17)
OursChainCalibrated rubric + full-chain evidence+2.38 (p<0.001)+1.64 (0.17)+5.68 (0.008)+3.67 (0.033)+4.59 (0.011)

Ours is the only arm that significantly outperforms the base model on all four metrics and the only arm that achieves significant improvements on both long-horizon metrics. The 95% confidence intervals are provided in the supplementary table at the end.

AuthorsOpenReview ↗

Response to W(2-3), Q1 2/3

Direct Paired Comparisons Between Ours and Each Control Arm

We assess statistical distinguishability using direct paired comparisons between Ours and each control arm, rather than inferring differences from whether each arm is individually significant relative to the base model.

Control ArmStrict d1–8Per-bugCore d4–8Complete-chainChain-wise exact testCore d1–8
Whole-chain multiplicative correctness+2.21 (p<0.001)+0.99 (0.51)+4.92 (0.15)+4.33 (0.038)15:4 (0.019)+3.40 (0.076)
Chain-level step-wise correctness+1.87 (0.0033)+1.97 (0.11)+5.30 (0.029)+3.00 (0.077)11:5 (0.210)+3.23 (0.088)
Single-step SWE+0.68 (0.24)+1.32 (0.15)+4.92 (0.14)+4.00 (0.025)13:5 (0.096)+3.23 (0.13)
A1 (uncalibrated rubric)+0.34 (0.63)+2.30 (0.015)+5.30 (0.047)+1.33 (0.25)7:3 (0.344)+2.72 (0.15)
A2 (calibrated rubric + step-local evidence)+1.70 (0.063)−0.00 (0.96)+1.89 (0.53)+3.67 (0.062)15:4 (0.019)+1.70 (0.45)

Two observations emerge. First, Ours is statistically distinguishable from each of the five control arms on at least one metric. Second, the metrics on which the control arms fall short differ: although several arms match Ours on individual metrics, no single control arm matches Ours across all metrics.

Correctness-Only Pool

The three correctness-only approaches—whole-chain multiplicative correctness, chain-level step-wise correctness, and single-step SWE—correspond to the same treatment condition: the reward contains only correctness information from executable tests. These three approaches are pairwise statistically indistinguishable on the following four metrics (all p ≥ 0.38). We therefore combine them into a control pool comprising nine evaluations.

MetricCorrectness-only pool vs BaseA1+A2 pool vs BaseOurs vs BaseOurs vs Correctness-only pool
Per-bug (current-step)+0.22 (0.84)+0.49 (0.63)+1.64 (0.17)+1.43 (0.18)
SCB core d4–8 (long-horizon)+0.63 (0.68)+2.08 (0.20)+5.68 (0.008)+5.05 [+0.33, +10.50] (0.038)
ChainSWE complete-chain (long-horizon)−0.11 (0.96)+1.17 (0.44)+3.67 (0.033)+3.78 [+0.67, +7.22] (0.020)
SCB core d1–8 (overall)+1.30 (0.39)+2.38 (0.19)+4.59 (0.011)+3.29 [+0.22, +6.49] (0.037)

Correctness-only RL shows no statistically significant difference from the base model on any of the four metrics, with essentially zero improvements on the two long-horizon metrics (d4–8: +0.63; complete-chain: −0.11). In contrast, Ours is statistically distinguishable from the correctness-only pool on three metrics.

Ablation Conclusions

1. The gains arise from the J term in the reward, and J requires chain-based calibration to transfer improvements across requirement boundaries

Ours and A1 differ in only one aspect: the scoring rubric. Ours uses the calibrated rubric, whereas A1 uses the rubric before calibration. The judge model, reward formulation R=A+0.4 AJR = A + 0.4\,AJ, training data, judge evidence window (full-chain), computational budget, and evaluation protocol are all identical.

The two arms are statistically distinguishable on the long-horizon core d4–8 metric (+5.30 [+0.33, +11.24], p=0.047) and on current-step per-bug correctness (+2.30 [+0.44, +4.33], p=0.015). A gain of the same magnitude appears in another comparison: Ours also outperforms the chain-level step-wise correctness arm by +5.30 [+0.70, +10.59] on core d4–8 (p=0.029).

Thus, what calibration provides is not simply stronger overall performance, but the ability to carry improvements across requirement boundaries. Completing the immediate step with all tests passing does not inherently depend on rubric quality. On strict solved, Ours (5.78), A1 (5.44), and even single-step SWE without any judge (5.10) perform comparably (all pairwise p ≥ 0.23). On this metric, any correctness-based RL approach can achieve gains: both A1 (+2.04) and single-step SWE (+1.70) significantly outperform the base model.

The effect of calibration is not apparent on strict solved; it becomes visible only when crossing requirement boundaries. With an uncalibrated rubric, A1 achieves only +0.38 on core d4–8 (p=0.82), essentially remaining at the base-model level, whereas Ours achieves +5.68.

AuthorsOpenReview ↗

Response to W(2-3), Q1 2/3

2. Full-chain evidence during training corresponds to successfully completing the entire chain

Ours and A2 also differ in only one aspect: the length of the evidence window available to the judge. Both use the same calibrated rubric, but A2 restricts the judge's evidence to the current step.

A2 statistically matches Ours on the deeper long-horizon metric (core d4–8: +1.89, p=0.53) and completely matches Ours on current-step per-bug correctness (−0.00, p=0.96). However, its complete-chain improvement disappears: A2 achieves a complete-chain success rate of 13.00, identical to the base model. In contrast, the chain-wise exact test comparing Ours against A2 gives 15:4, p=0.019 (bootstrap: +3.67, p=0.062).

3. The two roles of Consequence Chains

The first role is to provide the raw material for calibrating long-horizon reward criteria. The calibration procedure in Section 3 uses the signal that two implementations of the same chain, both passing CP1, diverge in their subsequent development outcomes. The frequency of such events is directly determined by the structural coupling within the chains.

The second role is to provide full-chain execution evidence during training.

These two roles are identified by Conclusions 1 and 2, respectively. Their contributions are distinct: the calibrated rubric drives generalization beyond the training horizon and improves per-step correctness, while full-chain evidence during training drives successful completion of entire chains in a single run. Both components are needed to achieve improvements across all four metrics simultaneously.

4. Standard SWE correctness-based RL cannot explain the long-horizon gains

The three compute-matched correctness-only approaches—chain data with whole-chain multiplicative correctness gating, chain data with step-wise average correctness, and single-issue data with binary correctness—show no statistically significant differences from the base model on either long-horizon metric.

For core d4–8, their improvements are +0.76 / +0.38 / +0.76 (all p ≥ 0.68), while for complete-chain success, they are −0.67 / +0.67 / −0.33 (all p ≥ 0.63). In contrast, Ours achieves improvements of +5.68 and +3.67, respectively.

The three correctness-only approaches are also statistically indistinguishable from one another. Despite using three different correctness credit assignment schemes, their core d4–8 improvements fall within a narrow 0.4-percentage-point range (all pairwise p ≥ 0.84). Whole-chain multiplicative correctness and the single-step arm achieve exactly the same core d4–8 improvement (+0.76), while whole-chain multiplicative correctness and chain-level step-wise correctness differ by only +0.17 on core d1–8 (p=0.95).

Changing the correctness signal from a whole-chain product to a step-wise average does not alter the outcome. What is missing is not denser correctness credit assignment, but the J term.

Supplementary Table: 95% Confidence Intervals for the Full Ablation Results

ArmStrict d1–8Per-bugCore d4–8Complete-chainCore d1–8
Whole-chain multiplicative correctness[−0.51, +1.00][−1.98, +3.61][−5.43, +6.87][−4.00, +3.00][−2.98, +5.47]
Chain-level step-wise correctness[+0.00, +1.12][−2.99, +2.21][−2.47, +2.93][−2.67, +4.00][−1.99, +4.76]
Single-step SWE[+0.34, +3.44][−1.66, +2.39][−3.70, +5.30][−3.33, +2.67][−2.31, +5.15]
A1[+0.83, +3.45][−3.08, +1.66][−4.17, +4.44][−0.67, +5.33][−2.36, +6.15]
A2[−0.50, +2.13][−0.87, +4.47][−0.34, +7.97][−3.67, +3.67][−1.34, +7.09]
Ours[+1.05, +3.84][−0.75, +4.02][+1.35, +10.42][+0.33, +7.00][+1.02, +8.38]

Statistical protocol: We use paired bootstrap with clustering by program (SCB, 36 units) or by chain (ChainSWE, 100 chains), with B=200,000. The reported p-value is the median across five random seeds, while the confidence interval is taken from the first seed.

Reviewer_GCCPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets sequential software development, where requirements are revealed one at a time and the agent must build each one on its own earlier changes. The authors mine 142 three-step "consequence chains" from 17 Python repositories, run the agent on a workspace that is never reset, and train with asynchronous PPO on a chain-level reward R = A + λAJ. Here A = a1·a2·a3 is whole-chain test correctness and J is a hindsight score from an LLM judge, whose rubric is calibrated offline by comparing step-local and continuation-informed assessments. Trained MiniCPM5-2B, Qwen3.6-35B-A3B and Qwen3.8-27B improve ChainSWE complete-chain success (3 to 9, 13 to 17, and 14 to 18 out of 100) and SlopCodeBench solve rates. For the 35B model, SlopCodeBench core solve rate at checkpoint depths 4 to 8, beyond the three-step training horizon, rises from 25.0% to 33.0%. A λ=0 ablation and a single-step SWE training baseline are reported for the 35B model.

Critical strengths
  1. The problem is well chosen and the reward is designed carefully. Several recent benchmarks show agents degrade when building on their own code, and current post-training does not assign credit across requirement boundaries. The gate in R = A + λAJ means a failed chain can never score above a correct one, and a miscalibrated judge can move the reward by at most λ.

  2. Many comparisons are set up properly: each trained model is compared with its own base under the same harness, the λ=0 arm uses the same chains and budget, training repositories are disjoint from both benchmarks, and the conditional-continuation analysis (Tables 3, 8, 9) handles the selection effect, including a version on the intersection of eligible chains.

Critical weaknesses
  1. The contribution is not separated from simpler alternatives. Each component of the recipe is standard on its own (asynchronous PPO, a test-based correctness reward, an LLM-judge score), so the paper needs to show that its two non-standard choices, the consequence-chain data and the hindsight judge, produce the gains. It does not show either, and it compares against no other training method. (a) Reward. Table 4 shows that chain data with a correctness-only reward (λ=0) does not help: ChainSWE chain success goes 13.0 to 12.0, SlopCodeBench core solve 32.7 to 31.6, and depths 4 to 8 drop from 25.0 to 21.6. The single-step correctness baseline does better than λ=0 on SlopCodeBench (35.2 vs 31.6). So the gains come from adding J. Two arms are needed: single-step SWE data trained with the same judge, and chain data with a step-local judge that does not see the continuation (Appendix D already has a per-step scoring mode). If the first works about as well, the result becomes "an LLM code-quality judge is a useful reward", which is the line of work Sec. 2.2 positions against. The offline calibration in Sec. 3.3, which is what makes J a hindsight judgment, comes with no numbers: how many rollouts, which model, what discrepancies were found, and how the rubric changed. The final rubric in Appendix C reads like general clean-code guidance (root cause, YAGNI, single source of truth, small diffs), which is hard to square with Sec. 2.1's argument that general engineering advice does not help. (b) Data. The only data comparison is the single-step arm in Table 4, which changes the data and the reward at the same time and does not say where its data comes from. The paper selects chains whose later patches build on structure introduced earlier (Sec. 3.1) but never tests whether this selection matters. The natural control is consecutive commit sequences from the same 17 repositories that do not pass the coupling filter, trained with the same reward and budget. A comparison with training on an existing public SWE training set of similar size would also help place the data contribution. (c) Other RL methods. No other training recipe is run on the same base models, for example a per-step correctness reward (a1 + a2 + a3 instead of the product), a patch-similarity reward as in SWE-RL (Wei et al., 2025a), a static code-quality reward as in Robeyns & Aitchison (2025), or the cross-task credit assignment of SAGE, SkillRise or CoD that Sec. 2.2 discusses. Without at least one of these at matched compute, it is unclear whether the gains come from the paper's design or from RL on SWE tasks in general.

  2. The statistics do not support the headline claims. Every number comes from one training run and one evaluation at temperature 1.0, with no confidence intervals. For the two larger models the ChainSWE chain-success gain is 4 chains out of 100 (standard error of the difference about 5 pp); "at least 28% relative" is 14 to 18. The beyond-horizon result is 22 to 29 of 88 checkpoints nested in 36 programs, and the highlighted depth-8 gain is 0 to 1 checkpoint. Some per-position results also go against the narrative: for the 35B model, Seq+Mem step-3 resolution goes 30 to 29 and Oracle chain success drops 18 to 17 (Tables 11, 12), so part of the narrowing of its Oracle minus Seq+Mem gap comes from Oracle getting worse. The comparison with Claude Opus 4.6 in the abstract mostly reflects the base model: base Qwen3.8-27B already has 43.4% per-bug resolution against 42.4% for Opus 4.6 and ties it on chain success (14 vs 14), and training adds 2.3 pp.

Questions for authors
  1. (Could change my rating) Results for (a) single-step SWE data with the same judge reward and (b) chain data with a step-local judge.

  2. (Could change my rating) Results for (a) consecutive-commit chains from the same 17 repositories that do not pass the coupling filter of Sec. 3.1, trained with the same reward and budget, and (b) at least one other RL recipe on the same base model at matched compute (e.g., a per-step correctness reward, or a SWE-RL-style patch-similarity reward). What is the source and size of the single-step SWE data in Table 4?

  3. (Could change my rating) Details of the offline calibration: number of rollouts, judge model, the discrepancies found, the rubric before and after, and an ablation with the uncalibrated rubric.

  4. (Could change my rating) Repeated runs or repeated evaluations with confidence intervals, and paired tests on ChainSWE (e.g., McNemar).

  5. (Clarification) Please add the SlopCodeBench depth breakdown for the 27B and 2B models, and report the fraction of training chains with A = 1 per model. For the 2B model, if A = 1 is rare during training, J rarely enters the reward; what then drives the 3 to 9 improvement?

  6. (Clarification) Appendix D argues that a uniform shift in J is removed by advantage normalization. Because J enters as λAJ, a shift changes the reward only for trajectories with A = 1, so it is not uniform across the batch. The effect is likely small, but the argument should be made by magnitude. The comparison that matters for training is judge agreement among A = 1 trajectories of the same chain; can you report that, and name the frontier judge used for the 2B and 35B runs?

12 comments
AuthorsOpenReview ↗

Response to W2: Oracle-Mode Fluctuations and Repeated Significance Tests

Thank you for the reviewer's careful observation. We agree with this point: in the original experimental results, some per-position metrics for the 35B model show slight decreases, and the narrowing of the Oracle–Seq+Mem gap cannot be entirely attributed to improvements in Seq+Mem. Our original wording was not sufficiently precise. We will revise the relevant statements in the manuscript to clearly distinguish between directly observed performance improvements under Seq+Mem and changes in the Oracle–Seq+Mem gap. To further examine whether the performance changes under Oracle mode are statistically significant, we conducted three independent evaluations each for Base and Ours using Qwen3.6-35B-A3B and performed two-sided significance tests. The additional experimental results are as follows:

  • Complete-chain success rate: Base achieves 17.05% ± 2.93%, while Ours achieves 19.38% ± 0.67%, with a two-sided p-value of 0.21.
  • Per-bug success rate: Base achieves 52.04% ± 1.17%, while Ours achieves 51.78% ± 0.22%, with a two-sided p-value of 0.882. These results indicate that the slight performance decreases observed under Oracle mode in the original single evaluation are not supported by statistically significant differences in the repeated experiments. At the same time, we acknowledge that fluctuations in Oracle performance may indeed have contributed to the magnitude of the gap reduction in the original results. Therefore, the entire change in the gap should not be attributed to improvements in Seq+Mem.
AuthorsOpenReview ↗

Source, Size, and Compute Budget of the Single-Step SWE Data in Table 4

Thank you for the reviewer's question. We will clarify the data source, dataset size, and training configuration of the Single-step + correctness baseline in Table 4 in the revised manuscript. This baseline uses 1,296 R2E-Gym-style single-issue software engineering task instances from 10 open-source Python repositories, which overlap with the source repositories of our consequence-chain data. Thus, the two training settings share part of their code-domain distribution. We also confirm that there is no data contamination between these 10 training repositories and the SlopCodeBench or ChainSWE evaluation benchmarks. Therefore, the single-step baseline in Table 4 represents a standard correctness-based RL comparison using a larger-scale single-issue dataset with matched training compute, rather than a baseline weakened by a reduced training scale.

AuthorsOpenReview ↗

Response to Q5 : SlopCodeBench Depth Breakdown for All Three Models, Per-Model A = 1 Rates, and What Drives the 2B Gain

SlopCodeBench Depth Breakdown for All Three Models, Per-Model A = 1 Rates, and What Drives the 2B Gain

We thank the reviewer for the suggestion. We have added the checkpoint-depth breakdown of SlopCodeBench for all three models (split into depths 1--3 and depths 4--8) and computed the fraction of complete-chain successes (A = 1) during training for each model.

1. SlopCodeBench depth breakdown.

ModelMetricd1–3: base → ours (diff, p)d4–8: base → ours (diff, p)
Qwen3.8-27Bcore solved61.11 → 62.04 (+0.93, p=0.74)52.27 → 56.82 (+4.55, p=0.051)
MiniCPM5-2Bcore solved4.32 → 6.79 (+2.47, p=0.055)0.76 → 2.27 (+1.52, p=0.028)
Qwen3.6-35B-A3Bcore solved37.04 → 40.74 (+3.70, p=0.179)22.35 → 28.03 (+5.68, p=0.008)
Qwen3.8-27Bstrict solved17.59 → 22.53 (+4.94, p=0.017)0.38 → 3.79 (+3.41, p=0.010)

For the 27B model, gains in core solved are concentrated at depths 4--8, while strict solved improves significantly at both depth ranges. For the 2B model, although absolute success rates remain low, core solved still rises from 0.76% to 2.27% at depths 4--8, indicating that improvements beyond the three-step training horizon are not confined to larger models. In deeper checkpoint ranges, strict solved has limited power to separate models, since both the 2B and the 35B-A3B models score near the floor on it; we therefore use core solved as the primary metric for the depth analysis and report the strict-solved breakdown only for the 27B model.

2. Fraction of A = 1 during training.

ModelFraction of training trajectories with A = 1
Qwen3.8-27B61.00%
Qwen3.6-35B-A3B18.00%
MiniCPM5-2B15.10%

These fractions measure how often a single rollout completes the entire three-step chain during training.

3. What drives the 2B improvement.

The reviewer is right that the reward is sparser for the 2B model than for the larger ones, but sparse is not absent. At the checkpoint used in the paper, 15.1% of 2B training trajectories complete the full three-step chain, and this fraction grows over training: 10.6% over the first 20 rollout groups versus 19.4% over the last 20. The event a1 = a2 = a3 = 1 is therefore neither rare at the start nor static.

One point is worth clarifying, as it may be the source of the reviewer's concern. We use asynchronous PPO with a learned critic, so the advantage baseline is a function of the chain state rather than a within-group mean. Between two A = 1 trajectories of the same chain, the advantage difference is lambda * (J_i - J_j); for lambda * A * J to produce a non-zero gradient it is not necessary for a single batch to contain many successful trajectories.

How much does J actually spread? In the 2B judge logs, solved trajectories receive a mean judge score of 0.738 with a range of 0.35--0.95, and across successful trajectories within the same chain the mean spread of J is 0.267, i.e. 0.4 * 0.267 = 0.11 in reward units. The judge is thus not a constant offset on successful trajectories, but a quantity that discriminates how well a requirement was implemented.

As for the 3 → 9 improvement itself, we attribute it primarily to the correctness gate A: the training-chain success rate rises from 10.6% to 19.4%, matching both the direction and the rough magnitude of the 3% → 9% change on ChainSWE. The judge term acts as a quality correction bounded by lambda = 0.4; it never reorders a success against a failure, and only separates already-correct implementations by quality, which reduces rework in later steps.

Reviewer_uxb6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sequential software-development agents that must implement requirements on their own accumulated code without knowing later requirements. It constructs three-step consequence chains from repository histories and trains agents with a whole-chain correctness reward augmented by a hindsight judge of whether the resulting code supports later development. Experiments on ChainSWE and SlopCodeBench compare trained and base models, including reward ablations and performance at horizons exceeding training length.

Critical strengths
  1. The paper identifies a concrete failure mode: a patch can pass the current task’s tests while making later development harder. The gated reward, R=A+λAJ, keeps functional correctness primary while distinguishing among correct trajectories.
  2. The experiments test the proposed mechanism. ChainSWE complete-chain success improves at all three model scales.
Critical weaknesses
  1. The hindsight mechanism is not isolated. The 35B ablation shows that the judge term helps, but the rubric also rewards general code-quality properties. There is no comparison with a judge that cannot see later execution, and the offline calibration procedure is described only qualitatively.
  2. The reliability of the gains is unclear. The 35B and 27B ChainSWE gains each amount to four additional successful chains out of 100. The paper reports neither uncertainty estimates nor multiple training seeds. The longer-horizon analysis is primarily shown for one model.
  3. The methodological novelty is limited. At a high level, the training method appears to add an LLM judge’s assessment of code quality or future extensibility to an existing correctness-based reward.
Questions for authors
  1. Can you describe the offline calibration procedure in enough detail to reproduce it, and compare the proposed judge with a current-step-only judge?
  2. Can you report paired uncertainty estimates or repeated runs for the main gains?
  3. Can you hold the continuation model fixed and compare its success on matched, passing prefixes produced by the base and trained models? This would clarify whether the generated code itself is easier to extend.
6 comments
AuthorsOpenReview ↗

Response to Reviewer <uxb6> — Q1(a): Offline Rubric Calibration (reproducible procedure)

We thank the reviewer for this suggestion. The original manuscript described the offline calibration procedure too briefly; the revision will include the full procedure.

1. Calibration dataset. From the 142 three-stage Consequence Chains we build A/B pairs out of existing rollout records. Each pair satisfies: (i) A and B come from the same chain and both pass the CP1 acceptance tests; (ii) A then completes CP2 and CP3 and passes all tests; (iii) B passes CP1 but requires substantial rework later and ultimately fails. Pairs are drawn from the same model under identical execution settings; when several candidates exist, we take as A the successful implementation with the lowest downstream development cost recorded in the original trajectory, and as B the failed one with the highest, giving a clear cost gap. This yields 112 candidate pairs. A manual causal-relevance review then removes pairs whose later failure is clearly unrelated to the CP1 structure (e.g. caused only by mistakes in code newly written at a later stage), leaving 82 pairs: 52 train / 30 test.

2. Prospective vs. hindsight stages. The judge is GLM-5.3, run under two strictly separated information regimes. Prospective: the judge sees only CP1-time information (the requirement, repository context, the CP1 implementation, its test results) and scores A and B independently — this tests whether the rubric can flag maintenance risk from the current implementation alone. Hindsight: the judge additionally receives the full CP1-CP3 trajectories (successive diffs, test results, rework), compares them with its earlier scores, and reports which early structural judgements contradicted the realized outcome. Calibration therefore uses consequences revealed by actual execution to expose gaps in the criteria, and converts them into principles applicable to current code without any knowledge of future requirements.

3. One-change-at-a-time rubric iteration. Each version adds, removes or replaces exactly one criterion, so an improvement is attributable to a single change. V1: fully satisfy the explicit semantics of the requirement. V2: fix root causes, not symptoms. V3: no speculative design unsupported by the current requirement. V4: later modifications must be implementable locally. V5: one authoritative implementation per piece of knowledge (no duplicated logic requiring synchronized edits). V6: reshape structure as requirements evolve (reasonable extension vs. patch accumulation). V7: no dead code, unused options or leftover scaffolding. V5-V7 make "supports later development" concrete, replacing the vague initial wording ("well-organized", "easy to extend"). For instance, in the Tortoise chain, B kept similar query logic in several places and lost a duplicate-field check in a later rewrite — exactly the V5 no-duplication constraint, which corrects the ranking.

4. Calibration effectiveness. We score each rubric version by chain-equal-weighted ranking accuracy: A/B ranking accuracy is computed within each chain, then averaged over chains (%).

RubricTrain (52 pairs)Test (30 pairs)V056.3155.70V160.3559.21V263.3865.35V364.3967.98V466.9269.30V569.4473.25V673.9977.19V778.2882.02\begin{array}{l|cc} \textbf{Rubric} & \textbf{Train (52 pairs)} & \textbf{Test (30 pairs)} \\\\ \hline \text{V0} & 56.31 & 55.70 \\\\ \text{V1} & 60.35 & 59.21 \\\\ \text{V2} & 63.38 & 65.35 \\\\ \text{V3} & 64.39 & 67.98 \\\\ \text{V4} & 66.92 & 69.30 \\\\ \text{V5} & 69.44 & 73.25 \\\\ \text{V6} & 73.99 & 77.19 \\\\ \text{V7} & \mathbf{78.28} & \mathbf{82.02} \end{array}

5. Traceable example. We keep per-revision A/B cases (project, pre/post scores, diffs, later acceptance results). In the Paramiko connect-and-authenticate chain, both implementations pass the CP1 authentication tests. A separates key loading from server authentication, so the next two requirements need only local edits (1 CP1 line rewritten/deleted; CP3 touches 1 file). B merges the two into one exception-handling block, swallowing the real authentication-failure cause; later development rewrites/deletes 44 CP1 lines, CP3 touches 4 files, and all tests still fail. Under the same scoring seed, V0 gives A/B = 0.65/0.70 (wrongly preferring B) while V7 gives 0.60/0.55 (correct order). The final rubric thus tracks structural maintenance risk rather than current pass status or surface-level organization.

AuthorsOpenReview ↗

Ablation: Step-Local vs. Hindsight Judge Evidence

Setup. We thank the reviewer for this suggestion. We have run the step-local judge ablation (A2) and report it below; it will be included in the revised manuscript. A2 and Ours share the same base model (Qwen3.6-35B-A3B), the same asynchronous PPO framework, the same compute budget, the same judge model, and the same reward form R=A+0.4AJR = A + 0.4AJ with the identical calibrated rubric. The only difference is the evidence window given to the judge during training: A2 sees the current step alone, whereas Ours sees the execution trace of the whole chain. Each arm was evaluated three times independently.

Levels below are means over three independent evaluation runs (± s.d. across runs, %); parentheses give the difference from Base with its two-sided pp. Significance is from an item-level paired bootstrap (SCB: per depth cell; ChainSWE: per step, clustered by chain; B=200000B = 200000); the s.d. is reported as a dispersion summary only.

MetricBaseStep-local (A2)Ours (hindsight)Ours − A2
SCB core d1–8 (/196)30.44 ±2.0633.33 ±1.79 (+2.89, 0.17)35.03 ±2.41 (+4.59, 0.011)+1.70 (0.45)
SCB core d1–3 (/108)37.04 ±1.6039.20 ±3.74 (+2.16, 0.47)40.74 ±0.93 (+3.70, 0.18)+1.54 (0.59)
SCB core d4–8 (/88)22.35 ±2.8626.14 ±4.10 (+3.79, 0.057)28.03 ±4.59 (+5.68, 0.008)+1.89 (0.53)
SCB strict d1–8 (/196)3.40 ±0.784.08 ±1.77 (+0.68, 0.38)5.78 ±1.28 (+2.38, p<0.001p<0.001)+1.70 (0.063)
ChainSWE per-bug (/304)40.02 ±0.5041.67 ±0.83 (+1.64, 0.21)41.67 ±0.19 (+1.64, 0.17)−0.00 (0.96)
ChainSWE full-chain (/100)13.00 ±0.0013.00 ±1.00 (−0.00, 0.97)16.67 ±0.58 (+3.67, 0.033)+3.67 (0.062)

Exact per-chain test over the 100 chains: Ours vs. Base 12:3 (p=0.035p = 0.035); Ours vs. A2 15:4 (p=0.019p = 0.019); A2 vs. Base 7:6 (p=1.00p = 1.00).

Results. The comparison is sharper than a uniform ranking. On the per-step and per-depth metrics, A2 is statistically indistinguishable from Ours: per-bug is exactly tied (−0.00, p=0.96p = 0.96) and core d4–8 differs by +1.89 (p=0.53p = 0.53). A calibrated rubric therefore recovers most of the per-step quality signal even from current-step evidence alone. What A2 loses is the chain-level gain, and it loses all of it: A2 reaches 13.00 on full-chain success, exactly the base model's score (p=0.97p = 0.97), while Ours reaches 16.67 (+3.67 over Base, p=0.033p = 0.033); the direct Ours-vs-A2 comparison is significant under the exact per-chain test (15:4, p=0.019p = 0.019; bootstrap p=0.062p = 0.062).

This is the behaviour the design predicts. A judge restricted to the current step can tell whether that step was implemented cleanly, but it cannot see whether the implementation it rewarded is the one that later steps could build on. Whole-trajectory evidence is what makes "the entire chain succeeds on a single attempt" a rewardable property, and it is on exactly that metric that removing the evidence window removes the gain.

AuthorsOpenReview ↗

Variance and statistical significance of the main-table gains

We thank the reviewer for this suggestion and have adopted it. Every number in the main table is now the mean over three independent repeated evaluations; we additionally report the standard deviation, the 95% confidence interval, and a paired test for each cell. The complete results will be included in the revised manuscript; a summary is given below.

Main-table gains with 95% confidence intervals and paired-test p-values, each computed over three independent repeated evaluations.

ModelMetricbase → oursdiff [95% CI]two-sided p
MiniCPM5-2BChainSWE (whole chain)3.67 → 9.00+5.33 [+2.00, +9.00]<0.001
MiniCPM5-2BChainSWE (per-bug)27.52 → 31.14+3.62 [+1.43, +5.90]<0.001
MiniCPM5-2BSCB strict0.34 → 1.19+0.85 [+0.00, +1.94]0.087
Qwen3.6-35B-A3BChainSWE (whole chain)13.00 → 16.67+3.67 [+0.33, +7.00]0.033
Qwen3.6-35B-A3BChainSWE (per-bug)40.02 → 41.67+1.64 [−0.75, +4.02]0.174
Qwen3.6-35B-A3BSCB strict3.40 → 5.78+2.38 [+1.05, +3.84]<0.001
Qwen3.8-27BChainSWE (whole chain)14.00 → 18.33+4.33 [+1.00, +8.33]0.0062
Qwen3.8-27BChainSWE (per-bug)43.75 → 47.04+3.29 [+1.30, +5.54]<0.001
Qwen3.8-27BSCB strict9.86 → 14.12+4.25 [+1.37, +7.50]0.0024

All nine main cells move in the positive direction, and seven of them are significant at the two-sided 0.05 level. The two non-significant cells are both floor effects on SCB strict for the 2B model: averaged over the three repeats, the base solves 0.67 out of 196 instances and ours solves 2.33 out of 196, so the absolute counts are too small for the difference to reach significance even though the direction is consistent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

What we claim. We train coding agents on consequence chains — sequences of real requirements mined from commit histories, where the agent repeatedly modifies its own earlier code without knowing the future requirements. The reward adds a hindsight judge term R=A+0.4 AJR = A + 0.4\,AJ whose rubric is calibrated on observed downstream outcomes (cases where two implementations that both pass the current checkpoint diverge in how the rest of the chain unfolds), and the judge conditions on the actual execution results of the later requirements. The claim is not "a judge helps," but that a signal grounded in realized downstream consequences buys what correctness-only RL cannot: gains that cross the requirement boundary.

What we ran. The submitted version had only one ablation contrast. During the rebuttal we ran a seven-arm matrix — {data: single issue ↔ consequence chain} × {reward: correctness-only ↔ + uncalibrated rubric ↔ + calibrated rubric (current-step ↔ full-chain evidence)} — all arms sharing one base model, one asynchronous PPO framework and matched compute, each evaluated three times per benchmark. Three arms answer the reviewers directly:

Reviewer's concernArm we ran, and the result
"It's just correctness + a generic LLM quality score."A1 uses our pre-calibration judge prompt — "is this code well-designed and easy to extend later" — with no access to later execution results; everything else is identical to Ours (judge model, reward form, data, full-chain evidence, compute, protocol). A1 matches Ours on the current step (strict, Ours − A1 = +0.34, p = 0.63) but falls back to base downstream (steps 4–8: +0.38, p = 0.82; Ours − A1 = +5.30, p = 0.047). The same +5.30 recurs against the step-wise-average correctness arm (p = 0.029), so this is the effect of calibration, not a quirk of one arm.
"Maybe any correctness signal, or denser credit, would do."The three compute-matched correctness-only paths (a1a2a3a_1a_2a_3; (a1+a2+a3)/3(a_1{+}a_2{+}a_3)/3; single-issue binary) are mutually indistinguishable (pairwise p ≥ 0.38), so we pool them into nine evaluations. The pool is null everywhere (steps 4–8 +0.63; complete chain −0.11; core 1–8 +1.30; all p ≥ 0.39), while Ours − pool = +5.05 (p = 0.038) and +3.78 (p = 0.020). What is missing is not credit density but JJ.
"The hindsight mechanism is not isolated — there is no comparison with a judge that cannot see later execution."A2 uses the same calibrated rubric but narrows the evidence window to the current step. On steps 4–8 it is indistinguishable from Ours (+1.89, p = 0.53), indicating that calibration has already absorbed most of the hindsight information into the rubric; but the whole-chain gain disappears entirely (13.00 = base; per-chain exact test 15:4, p = 0.019), indicating that seeing the later execution at training time makes a separate contribution, and that it lands exactly on "getting an entire chain right in one pass."

Ours is the only arm significantly above base on all four reported views (strict +2.38, p < 0.001; steps 4–8 +5.68, p = 0.008; complete chain +3.67, p = 0.033; core 1–8 +4.59, p = 0.011), and no single control arm matches it across all of them — each arm falls short precisely on the view corresponding to the component removed from it. All comparisons are paired bootstrap clustered by program (36 units) or chain (100 chains), B = 200,000, p-values the median over five seeds, with 95% CIs and per-chain exact tests reported in full in our responses to the individual reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.