acceptodds
Under review as a conference paper at ICLR 2027

Beyond Rejection Sampling: Learning from Guided Rollouts to Improve LLMs for Software Engineering

Abstract

Rejection-sampling supervised fine-tuning (SFT) for coding agents retains only trajectories that pass executable tests. On difficult repository tasks, repeated unguided rollouts often produce no successful trajectory. Consequently, **many available tasks and execution environments yield no retained SFT examples**, wasting executable software-engineering data. We use answer-derived guidance to recover training trajectories. Based on when guidance enters an interactive rollout, we design three mechanisms: **Pre guidance** derives a hint from the reference solution before initial rollout; **Post guidance** translates an unsuccessful attempt and the reference solution into a hint for fresh retry; and **Intra guidance** provides answer-derived assistance at eligible interaction steps. Our method, *SFT with Guided-Success Augmentation* (GS-SFT), augments rejection-sampling SFT with verifier-accepted guided trajectories and uses ordinary supervised fine-tuning. We distinguish **Supply**, retained SFT examples normalized by the number of executable tasks, from **Utility**, the task resolved rate after SFT, measured on SWE-bench Verified and SWE-bench Multilingual. Across the ordered guidance constructions, Supply increases steadily, whereas **Utility peaks at an earlier or intermediate non-oracle condition and then decreases under more explicit guidance**. Direct reference patches produce the most data but not the highest resolved rate. For offline guidance, our results favor **retaining hints in the SFT input rather than removing them**. The additional guided data produces the largest resolved-rate gains on issues with no actionable solution or reproduction lead, while also improving issues that already identify a likely solution or code location. Overall, moderately informative answer-derived guidance provides a simple way to use executable software-engineering data more efficiently during SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Software Engineering, Coding Agent, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_dcgcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses reference patches to provide guidance of varying strength before rollouts, after a failed attempt, or during interaction, then trains a software-engineering student on successful trajectories. Recovering useful training data from tasks that unguided sampling repeatedly fails to solve is a worthwhile objective. The experiments also show that producing more successful trajectories need not yield a better student. However, matched collection costs and training budgets are still needed to establish the contribution of guidance itself.

Critical strengths
  1. Sections 3-4 organize the experiments around when guidance is introduced and how much information it provides. This offers a clear, interpretable framework for studying the use of privileged reference information.

  2. Table 5 reports both data yield and student performance, showing that directly revealing a reference patch can produce more successful trajectories while reducing their learning value.

  3. Section 7 examines whether guidance should remain in the training input and how gains vary across categories of issue information. This broadens the empirical analysis of how the choice of training inputs affects learning from trajectories.

Critical weaknesses
  1. The central claim about the value of moderate guidance is confounded by collection and training budgets. Sections 5-6 compare guided sampling against only three unguided attempts per task, without additional unguided retries at the same call or token cost. Section 5.2 and Appendix A.3 also use two SFT epochs, so guidance strength changes task coverage, data mixture, and the number of training updates simultaneously. These comparisons do not isolate the benefit of guidance information from additional computation and training.

  2. Table 2 draws training tasks from multiple software-engineering sources, but the paper does not describe deduplication against SWE-bench Verified and Multilingual at the issue, commit, or near-duplicate-patch level. Independence between training and evaluation therefore remains unconfirmed when reference patches are used.

  3. The Pre/Post Keep-Remove comparisons in Table 3 explicitly use different experimental batches, and the historical Intra results differ from the defaults in the current visibility controls. The general conclusion that offline guidance should be retained therefore requires validation under fully matched conditions.

Questions for authors
  1. At the same total collection-token cost, how does moderate guidance compare with additional unguided retries and direct patch distillation?

  2. Does the nonmonotonic Utility trend persist when the number of added examples, task set, and training-token or update budget are matched across guidance levels?

  3. Can you provide the deduplication rules and overlap statistics for training and evaluation, together with results after excluding any overlapping tasks?

  4. Can you repeat the Pre/Post Keep-Remove comparisons with the same trajectory source, initialization, and evaluation batch to distinguish training variability from the effect of guidance visibility?

2 comments
AuthorsOpenReview ↗

Thanks for your review!

Thank you for the clear review and for highlighting our framework.

  • Weakness 1 and Question 1 (Additional unguided vs. guided rollout pass and token cost; Section 7.1, Table 4 & General Response #1): Across the 2,527 twice-failed tasks (35.45% Supply, 34.21% Utility), total token consumption across one additional unguided pass and guided Pre/Post L3 and L4 passes varies by less than 5% (5.07B to 5.32B tokens, including <0.8% hint overhead). Even so, a third unguided pass lifts Supply only to 37.00% and Utility to 35.09%, whereas moderate guidance lifts Supply to 47.18% to 49.48% and Utility to 40.48% (+ Pre L3) and 43.61% (+ Post L3); more explicit + Post L4 reaches 57.63% Supply yet drops to 39.97%, matching the L6 patch-distillation drop in our main sweep (34.71% in Post and 30.58% in Intra). Within a <5% token spread, moderate guidance clearly outperforms both extra unguided retries and overly explicit supervision.

  • Weakness 1 and Question 2 (Nonmonotonic Utility at matched trajectories and SFT updates; Appendix B.2, Table 8 & General Response #2): Training each unique trajectory for 2 epochs evaluates the balance between extra Supply and trajectory artificiality, whereas repeating the 1,480 baseline trajectories equates to 2.56 and 2.68 epochs. When we hold added trajectories (414 and 506), prioritized task overlap, and update steps (238 and 250) fixed, downsampled L5 drops to 37.09% in Pre and 32.96% in Post, clearly below Pre L3 (39.22%) and Post L3 (37.72%), while upsampled Baseline (38.97% / 36.84%) also stays below 2-epoch L3. This separates guidance quality from update count and confirms that the nonmonotonic Utility trend persists under matched SFT budgets.

  • Weakness 2 and Question 3 (Deduplication check; General Response #3): We audited all 4,000 training tasks against SWE-bench Verified (500) and Multilingual (298) using normalized repository names, PR/issue numbers, and normalized reference patches, confirming zero overlap between our training tasks and the evaluation benchmarks.

  • Weakness 3 and Question 4 (Keep/Remove matching and Table 5 Intra fix): Both points arose from draft mistakes that we have now corrected. First, "cross-batch" in the caption and "historical/current" labels were editing errors: all 10 Pre/Post Keep/Remove pairs shared identical trajectories, Qwen3.5-9B-Base initialization, SFT settings, and evaluation setup (Keep wins 10/10 pairs, +1.88 to +6.77 pp, sign test p = 0.002). Second, Table 3 (now Table 5) had inadvertently listed an older deprecated run in the Intra Removed (default) column; updating L1 to L4 to our main Intra checkpoints (39.72%, 38.47%, 36.97%, 39.72%) aligns all tables. Our conclusion that task-specific offline hints should remain in the SFT input is therefore supported under matched conditions, distinct from level-dependent online reminder insertion.

We hope these controlled experiments and manuscript corrections address all the confounding factors you raised, and we would be very grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Thanks again for your feedback

Dear Reviewer dcgc,

As the discussion phase wraps up, we wanted to express our sincere thanks again for your careful reading and helpful comments. We hope that our response above and the General Response have addressed the points you raised for your final evaluation.

Reviewer_KZc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GS-SFT to collect more successful training trajectories for coding agents using answer-derived guidance. It studies three guidance strategies, Pre, Post, and Intra, across six guidance levels. The results show that stronger guidance increases Supply but does not always improve Utility. The paper also examines whether guidance should be retained in the SFT input.

Critical strengths
  1. GS-SFT increases the number of verified training trajectories from tasks that the agent initially failed to solve. Some settings also improve the task resolution rate over the no-guidance baseline, showing that guided trajectory collection can make better use of existing executable tasks.
  2. The paper provides a useful empirical analysis of how different guidance strategies and levels affect Supply and Utility. It also examines guidance visibility during SFT and which types of issues benefit most from guided training data, providing insights into how such data should be collected and used.
Critical weaknesses
  1. The paper lacks key controlled comparisons. Different guidance settings use different amounts of SFT data and numbers of training steps. The paper also lacks comparisons under the same computational cost for trajectory collection. In addition, the paper only uses Seed 2.0 Pro for trajectory collection. It is unclear whether a stronger model could produce higher-quality training trajectories and further improve the task resolution rate.
  2. The paper does not report statistical significance tests or confidence intervals for the improvements in task resolution rate. Moreover, several settings show smaller or no improvements on SWE-bench Multilingual despite improving on SWE-bench Verified (Table 6). This makes it difficult to assess the reliability and consistency of the reported improvements.
Questions for authors
  1. What are the main differences between GS-SFT and FailForge (Lv et al., 2026), especially in terms of their empirical findings?
  2. Would the conclusions about Utility still hold if different settings used the same amount of SFT training data and computation?
  3. Have the authors checked whether the issue-PR pairs used for trajectory collection overlap with the evaluation tasks in SWE-bench Verified and SWE-bench Multilingual? If so, how was this checked?
  4. How does GS-SFT compare with existing guided trajectory collection approaches, such as FailForge (Lv et al., 2026), in terms of Supply and Utility under comparable settings?
  5. Would using a stronger rollout model than Seed 2.0 Pro produce better training trajectories and improve Utility?
2 comments
AuthorsOpenReview ↗

Thanks for your review!

Thank you for recognizing our trajectory recovery from unsolved tasks and our empirical analyses of Supply, Utility, visibility, and issue categories.

  • Weakness 1 and Question 2 (Controlled collection-cost and SFT-budget comparisons):

    1. Additional unguided vs. guided rollout pass (Section 7.1, Table 4 & General Response #1): Across the 2,527 twice-failed tasks (35.45% Supply, 34.21% Utility), total token consumption for one additional unguided pass and guided Pre/Post L3 and L4 passes varies by less than 5% (5.07B to 5.32B tokens, including <0.8% hint overhead). However, a third unguided pass reaches only 37.00% Supply and 35.09% Utility, whereas guided passes reach 47.18% to 57.63% Supply and 39.97% to 43.61% Utility (+ Post L3: 43.61% vs. higher-Supply + Post L4: 39.97%). This shows that under comparable collection cost, introducing guidance is far more effective than continuing unguided sampling.
    2. Fixed SFT trajectory and step budget (Appendix B.2, Table 8 & General Response #2): Equal 2-epoch training captures the trade-off between extra Supply and trajectory artificiality (repeating Baseline = 2.56 and 2.68 epochs). When we fix trajectory counts and optimization steps (238 in Pre and 250 in Post), downsampled L5 (37.09% / 32.96%) stays well below Pre L3 / Post L3 (39.22% / 37.72%), and upsampled Baseline (38.97% / 36.84%) also remains below L3. Our conclusions on nonmonotonic Utility and the advantage of moderate guidance therefore still hold under identical SFT data size and computation.
  • Weakness 2 (Statistical significance, confidence intervals, and Multilingual): Training and evaluating each 131k-context 9B checkpoint once across 798 containerized tasks follows standard SWE post-training practice (SWE-Gym, Pan et al., 2025; SWE-smith, Yang et al., 2025; R2E-Gym, Jain et al., 2025; FailForge, Lv et al., 2026). On the shared 798 tasks, paired 95% CIs and exact two-sided McNemar tests confirm significant gains over Baseline (Post L1: +5.01 pp [+1.42, +8.61], p = 0.008; Pre L3: +4.14 pp [+0.72, +7.55], p = 0.022; Intra L1/L4: +4.64 pp, p < 0.02) as well as over L6 in Post (+5.39 pp [+1.91, +8.87], p = 0.003) and Intra (+9.15 pp [+5.74, +12.55], p < 0.0001). Multilingual shows more fluctuation because it contains only 298 tasks across multiple languages and non-Python data accounts for just 14.70% (588/4,000) of our training pool. Adopting the combined 798-task Overall Utility thus gives a reliable basis for our conclusions.

  • Question 1 and Question 4 (Comparison with FailForge, Lv et al., 2026; Sections 2 & 7.2): FailForge extracts a reusable task-agnostic procedural skill for post-failure rerollouts and strips it before SFT. In contrast, GS-SFT studies a 6-level task-specific answer hierarchy across Pre, Post, and Intra timings, and finds that stripping task-specific hints lowers Utility in 10/10 pairs (+1.88 to +6.77 pp) because removing a task-specific clue breaks consistency between the prompt and subsequent edits. Because FailForge uses a different base model, scaffold, and task set without released aligned trajectories on our pool, raw numbers cannot be compared directly across papers; however, our findings point to a clear takeaway: unlike generic procedural prompts, task-specific hints should be moderately informative and kept visible in offline SFT.

  • Question 3 (Overlap check; General Response #3): We checked all 4,000 training tasks against SWE-bench Verified (500) and Multilingual (298) using normalized repository names, PR/issue numbers, and normalized reference patches, finding zero overlap and ruling out test-set leakage.

  • Weakness 1 and Question 5 (Stronger rollout model): Due to the limited rebuttal window and the heavy cost of running multi-family interactive rollouts across 4,000 containerized tasks, we were unable to switch to another model API and re-run the full pipeline. That said, the patterns observed in our experiments show nothing specific to Seed 2.0 Pro, and we expect the same trade-off between recovering unsolved tasks and avoiding overly explicit hints to hold for other capable generators.

We hope these new controlled experiments, statistical tests, and clarifications address your soundness concerns, and we sincerely hope you will consider re-evaluating our score.

References

  • Jain, N., et al. (2025). R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights software engineering agents. COLM.
  • Lv, D., et al. (2026). FailForge: Distilling procedural competence from persistent failures into code agents. arXiv:2608.08570.
  • Pan, J., et al. (2025). Training software engineering agents and verifiers with SWE-Gym. ICML.
  • Yang, J., et al. (2025). SWE-smith: Scaling data for software engineering agents. arXiv:2504.21798.
AuthorsOpenReview ↗

Thank you for your time and feedback

Dear Reviewer KZc1,

With the discussion window closing soon, we wanted to thank you again for your time and constructive feedback on our work. We hope that our reply above and the General Response have resolved your concerns and will be useful as you finalize your assessment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their thoughtful feedback and for recognizing our practical motivation, the distinction between Supply and Utility, and our downstream gains without test-time hints. Below we address the three questions shared across all reviews and have updated the manuscript accordingly.

1. Guided vs. Additional Unguided Rollouts on Unsolved Tasks (Added to Section 7.1, Table 4)

Once initial rejection sampling leaves a pool of unsolved tasks, how should we spend the next rollout pass: run one more unguided pass, or introduce guidance? Starting from two unguided trials across all 4,000 tasks (2-trial baseline: 1,418 trajectories, 35.45% Supply, 34.21% Utility), we take the 2,527 tasks that failed both attempts and compare running one additional rollout per task without guidance against running Pre or Post L3 and L4 guidance:

Strategy (on 2,527 unsolved tasks)Rollout In / Out (M)Guidance In / Out (M)Supply (%)Utility (/798)
2-trial baseline0.0 / 0.00.0 / 0.035.45%34.21% (273)
+ 3rd unguided rollout5,021.7 / 47.20.0 / 0.037.00%35.09% (280)
+ Pre L35,196.1 / 51.111.9 / 1.447.18%40.48% (323)
+ Pre L45,255.4 / 50.811.2 / 1.752.45%41.10% (328)
+ Post L35,182.7 / 49.636.9 / 1.649.48%43.61% (348)
+ Post L45,075.5 / 48.536.3 / 1.957.63%39.97% (319)

Results: Total token usage across all five single-pass strategies (including guidance generation) fluctuates by less than 5% (from 5.07B to 5.32B tokens), yet leads to very different downstream gains. A third unguided pass lifts Supply only from 35.45% to 37.00% and Utility to 35.09%, whereas one guided pass lifts Supply to 47.18% to 57.63% and Utility to 39.97% to 43.61% (+ Post L3 reaches 43.61%). Meanwhile, + Post L4 yields higher Supply (57.63%) than + Post L3 (49.48%) but lower Utility (39.97% vs. 43.61%), again showing that maximizing Supply does not maximize Utility.

2. Equal-Epoch Rationale and Fixed-Update SFT Controls (Added to Appendix B.2, Table 8)

In our main experiments, training every unique trajectory for 2 epochs is a deliberate choice: stronger guidance brings more SFT Supply while also introducing greater trajectory artificiality from relying on privileged hints absent at test time. Repeating the 1,480 baseline trajectories (186 steps) to match L3's dataset size instead trains those easy tasks for 2.56 and 2.68 epochs. To separate Supply, trajectory artificiality, and step count, we anchor to Pre L3 (1,894 trajectories, 238 steps) and Post L3 (1,986 trajectories, 250 steps):

ConditionTraj.Unique Traj.Opt. StepsUtility (/798)
Shared baseline (2.00 epochs)1,4801,48018635.09% (280)
Pre L3 (2.00 epochs)1,8941,89423839.22% (313)
Pre L5 (downsampled; full: 36.72%)1,8941,89423837.09% (296)
Baseline (upsampled to Pre L3; 2.56 epochs)1,8941,48023838.97% (311)
Post L3 (2.00 epochs)1,9861,98625037.72% (301)
Post L5 (downsampled; full: 37.34%)1,9861,98625032.96% (263)
Baseline (upsampled to Post L3; 2.68 epochs)1,9861,48025036.84% (294)

Results: Neither trajectory count nor update steps explains our main trend. First, once L5's Supply advantage is removed by downsampling, Utility reaches to 37.09% in Pre and drops to 32.96% in Post, clearly below matched-step Pre L3 (39.22%) and Post L3 (37.72%); overly explicit guidance lowers per-trajectory training value. Second, repeating Baseline for more updates (38.97% and 36.84%) helps to some extent, but still falls short of 2-epoch guided SFT.

3. Training and Evaluation Overlap Check

We checked all 4,000 collection tasks from the four training sources against SWE-bench Verified (500) and SWE-bench Multilingual (298) using normalized repository names, PR/issue numbers, and normalized reference patches, confirming zero overlap between the training pool and evaluation benchmarks.

We hope these additional experiments and revisions address the shared questions, and we would appreciate it if the reviewers would take them into account in their final evaluation.

AuthorsOpenReview ↗

Thanks for your review!

Thank you for the constructive review and for noting the practical fit of GS-SFT as well as the strong Post L1 improvement.

  • Weakness 1 and Question 1 (Additional attempts, teacher tokens, and unguided continuation; Section 7.1, Table 4 & General Response #1): Each guided condition runs one guided rollout per unsolved task, and generating the teacher hints adds less than 0.8% input-token overhead (11.2M to 36.9M input and 1.4M to 1.9M output tokens across the 2,527 twice-failed tasks). Across all five single-pass strategies in Table 4, total token usage varies by less than 5% (5.07B to 5.32B tokens), yet a third unguided pass reaches only 37.00% Supply and 35.09% Utility whereas guided passes reach 47.18% to 57.63% Supply and 39.97% to 43.61% Utility (+ Post L3: 43.61%). Once initial unguided sampling stalls, spending the next pass on guidance is much more effective than continuing unguided rollouts at nearly the same token cost.

  • Weakness 1 and Question 1 (Preferred guidance level at fixed SFT budget; Appendix B.2, Table 8 & General Response #2): Training each unique trajectory for 2 epochs is intended to reflect the trade-off between extra guided Supply and trajectory artificiality, whereas repeating the 1,480 baseline trajectories to match L3 trains easy tasks for 2.56 and 2.68 epochs. When we hold trajectory counts and update steps fixed (238 in Pre and 250 in Post), downsampled L5 (37.09% / 32.96%) stays clearly below Pre L3 / Post L3 (39.22% / 37.72%), and upsampled Baseline (38.97% / 36.84%) also remains below L3. Our preference for moderate over overly explicit guidance therefore holds even under a fixed trajectory count and SFT step budget.

  • Weakness 2 (Evaluation variability, small gaps, and Multilingual): Training and evaluating each 131k-context 9B model once across 798 containerized tasks follows standard practice in SWE post-training (SWE-Gym, Pan et al., 2025; SWE-smith, Yang et al., 2025; R2E-Gym, Jain et al., 2025). On the 798 shared tasks, paired two-sided McNemar tests show significant gains for Post L1 (+5.01 pp, 95% CI [+1.42, +8.61], p = 0.008) and Pre L3 (+4.14 pp, 95% CI [+0.72, +7.55], p = 0.022) over Baseline and over Post/Intra L6 (p < 0.01), while we agree that small differences like Pre L3 (39.22%) vs. Pre L6 (38.22%, despite 4.1x more added data) should not be over-claimed. For Multilingual, the test split is smaller (298 tasks across multiple languages) and non-Python tasks make up only 14.70% (588/4,000) of our training pool, leading to larger subset fluctuation. We therefore base our main conclusions on the combined 798-task Overall Utility rather than small subset or 1-point differences.

  • Weakness 2 and Question 2 (Keep/Remove setup, draft labels, and Post L5 denominator): First, "cross-batch" in the caption and "historical/current" in the text were our editing mistakes from earlier drafts: all 10 Pre/Post Keep/Remove pairs shared the exact same trajectories, Qwen3.5-9B-Base initialization, SFT settings, and evaluation setup (Keep wins 10/10 pairs, +1.88 to +6.77 pp, sign test p = 0.002; we also corrected a mis-entered Intra L1 to L4 default column in new Table 5). Second, your calculation on Post L5 is completely right: Overall previously excluded 11 transient container or timeout infrastructure errors (298/787 = 37.87%) while the subset columns used fixed 500 and 298 denominators. We have now unified the entire paper to fixed denominators of 798, 500, and 298 without infra exclusions (Post L5 is 298/798 = 37.34%) and added exact resolved counts for all 19 conditions in Appendix Table 7. These updates confirm that offline hint retention is validated under matched conditions and all numbers are arithmetically consistent.

  • Question 3 (Training and evaluation overlap): We checked all 4,000 training tasks against the 798 evaluation tasks by normalized repository names, PR/issue IDs, and normalized reference patches, finding zero overlap and confirming that our gains do not come from benchmark leakage.

We hope these new controlled experiments, exact counts, and manuscript fixes resolve your concerns, and we kindly invite you to reconsider your assessment of our paper.

References

  • Jain, N., et al. (2025). R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights software engineering agents. COLM.
  • Pan, J., et al. (2025). Training software engineering agents and verifiers with SWE-Gym. ICML.
  • Yang, J., et al. (2025). SWE-smith: Scaling data for software engineering agents. arXiv:2504.21798.
AuthorsOpenReview ↗

Thank you and brief follow-up

Dear Reviewer EKTm,

Thank you again for your time and thoughtful review of our paper. As the discussion period comes to a close soon, we wanted to gently follow up to see if our response above and the General Response have addressed your questions. Please feel free to let us know if any further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.