acceptodds
Under review as a conference paper at ICLR 2027

FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

Abstract

Repository-level software engineering (SWE) is a challenging long-horizon setting: agents must reason over extended interactions, use tools across multiple turns, and adapt to observations from stateful environments. Recent work trains SWE agents with reinforcement learning methods such as Group Relative Policy Optimization (GRPO), which independently sample multiple trajectories per issue, test the resulting patches, and compare terminal rewards within a fixed group. However, this training setup does not reuse verifier feedback from failed patches as context for subsequent attempts, even though this feedback contains valuable diagnostic information about what went wrong. Training on recovery trajectories is challenging because the preceding outcome determines whether the next trajectory is generated, while the failed execution determines its conditioning context. We introduce FC-SWE, a failure-conditioned RL framework that incorporates recovery attempts into policy training. After a patch fails verification, FC-SWE restores the repository to its original task state and uses the failed patch and verifier feedback as context for a recovery trajectory. Each attempt is a complete, multi-turn tool-use trajectory with its own verifier reward; a later success leaves the reward assigned to an earlier failed attempt unchanged. FC-SWE adapts GRPO to estimate advantages from the trajectories actually executed for each issue. Failed attempts remain part of training, while unexecuted recovery attempts are excluded from the comparison group. On all 500 SWE-bench Verified tasks under a verifier-assisted protocol, FC-SWE with Qwen3.5-4B and SWE-agent achieves 41.7% Resolved@1 and 52.8% Resolved@2, compared with 38.9% and 48.5% for GRPO. Although trained with at most two attempts per chain, FC-SWE reaches 70.7% Resolved@11 under an eleven-attempt test-time budget.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, software engineering agents, large language model agents, verifier feedback, group-relative policy optimization, long-horizon agents, failure-conditioned recovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tRnMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper trains repository-level SWE agents to recover after unsuccessful patches given tests feedback. In case initial verification fails, the repository is reset and the agent receives the previous patch and "verifier" diagnostics. FC-SWE modifies GRPO in two ways: each attempt receives its own success/failure reward, so a successful retry does not change the earlier failed attempt’s reward. For each attempt, the advantage baseline is the average reward of all other executed attempts on the same issue, including both initial attempts and retries.

Authors use hidden tests output for the agent to improve the score during evaluation.

Critical strengths

The paper studies a relevant question: can we train the agent to exploit some external verifier feedback. The baselines are evaluated under the same feedback protocol as FC-SWE, so the within-paper comparison is internally consistent. The training-time failure-context ablation also provides evidence that additional retries alone do not explain the entire improvement.

Critical weaknesses
  1. The multi-attempt evaluation results rely on hidden-test feedback. The test-time “verifier” is the official SWE-bench harness, including FAIL_TO_PASS tests introduced by the golden PR that are not present in the repository at task time. Up to 6,000 characters of verifier output, including test names, tracebacks, and assertion messages, are provided to the policy before the second attempt.

Numbers such as 70.7% therefore measure performance with access to privileged test feedback. One possible explanation for the small gains: a few percentage points over the baseline - is that this feedback already provides substantial leakage about the solution and it might be quite easy for a model to just extract it.

If the agent can access these tests at inference time, why not make them available from the first attempt? It is difficult to see which practical setting motivates this access pattern. Moreover, SWE-bench Verified test output can reveal the exact location of an issue or specific expected behaviour. Appendix C.1 illustrates this concern: the FC-SWE recovery reasons that the test does not expect the warning and removes it. This passes the hidden tests but, as the authors themselves acknowledge, does not establish full compliance with the issue’s requested deprecation behaviour.

Therefore, the results do not establish that the trained policy can use verifier feedback effectively beyond this benchmark-specific setting. They demonstrate its ability to use leaked information supplied by the benchmark’s evaluation procedure. The paper acknowledges this limitation in Appendix G but provides no experiment using a deployable feedback source, such as checking whether the tests that were already in the repository still pass or submitting agent-generated tests output.

  1. The novelty is questionable. Multi-turn RL with execution feedback is already central to RLEF (Gehring et al., 2024), here adapted to a repository-level agent setting. In my view, the two proposed mechanisms are relatively small engineering choices within this broader approach.
Questions for authors
  1. The main results are reported without ±SEM, making it difficult to assess the variability relative to the claimed improvements. Readers must consult the appendix E and calculate this themselves, although it is important for interpreting the relatively small gains. Please report mean ±SEM across decoding seeds directly in the main tables for comparisons evaluated with multiple seeds.

  2. Checkpoint selection overlaps with the reported evaluation set. Appendix B.1 states that the FC-SWE checkpoint was selected using multiple validation sets, including 100 SWE-bench Verified tasks, while the headline results cover all 500 tasks. Consequently, the reported evaluation is not wholly held out from model selection. What is the score on the 400 SWE-bench tasks not used for selection?

2 comments
AuthorsOpenReview ↗

Comment 1 of 2: hidden-test feedback (Weakness 1)

Answer in brief. Leaked test information does not produce the gap between FC-SWE and GRPO. The gap is concentrated in failures whose retry context contains no test output, and it is statistically indistinguishable from zero where FAIL_TO_PASS diagnostics are shown. We ran this stratified analysis in response to your request for a deployable feedback source.

Recovery by feedback content. We split every first-attempt failure (all 500 tasks, three decoding seeds) by what its retry context contained; cells give second-attempt recovery, with task-clustered bootstrap CIs.

Retry context after the first failureBaseGRPOFC-SWEFC-SWE − GRPO [95% CI]
No verifier output (no applicable patch)12.0% (61/507)18.6% (64/344)26.9% (57/212)+8.3 [+1.4, +15.2]
Output with PASS_TO_PASS failures only15.0% (6/40)16.3% (7/43)22.7% (15/66)+6.4 [−7.2, +19.2]
Output with FAIL_TO_PASS failures10.9% (58/534)13.8% (73/530)15.8% (94/596)+2.0 [−1.7, +5.7]

A deployable feedback source. The first row is a deployable setting, because no test is consulted: the harness observes that the attempt left no applicable patch, and the retry context holds 0 characters of verifier output. Here FC-SWE recovers 26.9% of failures against 18.6% for GRPO. The second row corresponds to your suggestion of checking whether existing tests still pass; FC-SWE reaches 22.7% against 16.3%, although with 66 and 43 failures the interval remains wide. Resolved@1, which involves no feedback of any kind, improves by 2.9 points [+0.3, +5.4].

"The feedback leaks the solution and is easy to extract." The data contradict this explanation twice. If the 6,000-character excerpt made the fix easy to extract, every policy would profit from it; yet Base and GRPO recover fewer failures with this feedback than with blind resampling at the same budget (11.6% vs. 13.7%, and 15.7% vs. 18.6%; Figure 4a). Extracting a repair from test output is evidently hard for a 4B policy, and it becomes possible only after failure-conditioned training (19.0% vs. 17.0%). Moreover, all three policies receive identical feedback, so leakage could separate FC-SWE from GRPO only if FC-SWE exploited it better, and the third row shows that on exactly these failures the two policies differ by 2.0 points with an interval that includes zero.

"Why not provide the tests from the first attempt?" The feedback is the execution result of a submitted patch, and therefore cannot exist before a patch does. The access pattern is that of continuous integration and code review, where a submission is tested and the log is returned to its author. The agent receives at most 6,000 characters of output for its own patch and never the test files.

Appendix C.1. C.1 is a single selected case from the FAIL_TO_PASS stratum, and the paper says explicitly that it does not establish full compliance with the issue. It illustrates one behaviour: FC-SWE acts on the feedback where GRPO resubmits the same patch. No aggregate claim rests on it; the aggregate advantage rests on the first two rows above and on Resolved@1.

The 70.7% figure. The paper reports this number as verifier-assisted in the abstract and never as a standard score. The claim attached to it is a comparison at identical access: +3.4 points over GRPO at K = 11 [+1.6, +5.3], reached by a policy that was trained with at most two attempts.

AuthorsOpenReview ↗

Comment 2 of 2: novelty relative to RLEF, ±SEM, held-out 400 tasks

Answer in brief. FC-SWE is not RLEF moved to repositories. RLEF trains a single episode of at most three code solutions, in which feedback arrives between turns and one end-of-episode reward is credited to all turns; FC-SWE trains a chain of complete, separately verified trajectories, and our ablation shows that carrying RLEF-style shared credit over to such chains costs 3.0 points of Resolved@2 and can collapse training. The ±SEM values and the held-out-400 scores you asked for are given below.

Novelty relative to RLEF. The two settings pose different RL problems, and the difference is substantive for three reasons.

  • The unit of feedback changes the estimator. In RLEF, a turn is one code solution to a competitive-programming task, public-test results are appended to the same conversation, and a learned value function serves as the baseline. In FC-SWE, each attempt is a tool-use trajectory of up to 100 interactions that ends in its own verification, and the next attempt restarts from a clean repository with a bounded failure context. A recovery trajectory therefore exists only if its predecessor failed, so the comparison group for critic-free advantage estimation has an outcome-dependent size. This problem does not exist in RLEF, whose episodes are scored by a critic instead of by a comparison group, and it is precisely the one our two mechanisms solve.
  • The choice of credit assignment is consequential. Sharing the chain outcome across attempts, which is the direct transfer of episode-level credit, lowers Resolved@2 from 54.8% to 51.8% and conditional recovery from 20.4% to 17.7% (Table 2B); once the runtime penalties are also removed, training collapses from 76.6% to 2.3% online success within three steps. A choice with this effect is not a minor engineering detail.
  • The setting yields a finding that function-level work could not. For repository-level agents, feedback is not automatically useful: Base and GRPO recover fewer failures with feedback than with blind resampling (−2.1 and −2.9 points), and only failure-conditioned training reverses the sign (+2.0; Figure 4a). Removing failure context from training alone reduces recovery from 19.0% to 12.6% (Table 2A).

Q1 — mean ± SEM in the main table. Following your request, Table 1 reports mean ± SEM across the three decoding seeds.

All 500 tasksResolved@1Resolved@2Resolved@11
Base27.9 ± 1.636.3 ± 1.561.1 ± 1.5
GRPO38.9 ± 1.648.5 ± 1.767.3 ± 1.2
FC-SWE41.7 ± 0.852.8 ± 1.070.7 ± 1.3
FC-SWE − GRPO+2.9 [+0.3, +5.4]+4.3 [+2.1, +6.5]+3.4 [+1.6, +5.3]

Because the three seeds share the same tasks, the last row gives the paired, task-clustered bootstrap 95% CI of the difference. All three intervals exclude zero, so the gains are not within seed variability.

Q2 — the 400 tasks not used for selection. The lead is larger on these tasks than on all 500, and again every interval excludes zero.

Held-out 400 tasksResolved@1Resolved@2Resolved@11
Base27.8 ± 1.636.2 ± 1.161.2 ± 1.3
GRPO39.0 ± 2.148.8 ± 2.167.8 ± 1.1
FC-SWE42.0 ± 0.853.5 ± 0.971.6 ± 1.3
FC-SWE − GRPO+3.0 [+0.1, +5.9]+4.7 [+2.1, +7.2]+3.8 [+1.8, +5.9]

GRPO's checkpoint (step 65) was selected with the same procedure and on the same fixed 100-task subset as FC-SWE's (step 67). Since the lead grows on the held-out tasks, selection did not favour FC-SWE.

Reviewer_i5pQOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FC-SWE, a failure-conditioned reinforcement learning framework for improving the recovery capabilities of long-horizon software engineering agents. Unlike conventional GRPO, which independently samples trajectories, FC-SWE generates recovery attempts conditioned on previously failed patches and diagnostic verifier feedback, while resetting the repository to its original state. The framework introduces trajectory-local reward assignment and active-set advantage estimation to support policy optimization over initial and recovery trajectories.

The authors train Qwen3.5-4B on R2E-Gym and evaluate the resulting policy on SWE-bench Verified. FC-SWE improves Resolved@1 from 38.9% to 41.7% and Resolved@2 from 48.5% to 52.8% compared with GRPO, and achieves 70.7% Resolved@11 under a multi-attempt evaluation protocol.

The paper addresses a meaningful research problem and presents a reasonably clear methodology with encouraging empirical results. However, I have significant concerns about the evaluation protocol, particularly the use of official benchmark verifier feedback during multi-attempt evaluation, which potentially undermines the intended separation between agent-visible information and hidden evaluation tests. Furthermore, the methodological novelty and experimental evidence are not yet sufficiently convincing to establish the independent contribution of the proposed approach.

Critical strengths
  1. The paper addresses an important problem: enabling software engineering agents to learn from failed attempts and improve their subsequent solutions.

  2. The proposed framework is clearly described, and the experiments provide evidence that failure-conditioned training can improve recovery performance within the proposed evaluation protocol.

Critical weaknesses
  1. Fundamental concerns about the use of ground-truth verifier feedback during training and evaluation.

In standard SWE settings, agents can typically access existing tests and write their own tests to iteratively improve their solutions. However, the final evaluation tests, including both their contents and outcomes, are intentionally hidden from agents. An important capability of SWE agents is to independently design tests, identify shortcomings in their solutions, and make corrections without relying on feedback from the final evaluator.

My primary concern is that FC-SWE departs from this principle during both training and evaluation.

During training, FC-SWE directly provides agents with diagnostic feedback from task-specific ground-truth verifiers, allowing them to revise solutions based on the tests used to assess correctness. This raises concerns about whether the learned recovery capability genuinely generalizes to settings without such privileged feedback.

During downstream evaluation, the same mechanism is applied to SWE-bench Verified, where agents repeatedly receive feedback from the official benchmark verifier. This effectively exposes information derived from hidden evaluation tests, potentially enabling test-directed refinement rather than measuring genuine problem-solving capabilities.

Although the authors acknowledge this protocol, I believe its motivation and validity remain insufficiently justified. Evaluation with independently hidden final tests is necessary to establish the practical significance of the reported improvements.

  1. Insufficient experimental evidence to support the methodological contribution.

The improvements over GRPO are relatively modest, and the experimental comparisons leave several questions unresolved. FC-SWE uses additional recovery trajectories during training without a matched total training-compute budget. The proposed active-set advantage estimation is also not independently ablated. Furthermore, Appendix B.1 indicates that a subset of SWE-bench Verified was used for checkpoint selection, potentially affecting the independence of the final evaluation.

These limitations make it difficult to isolate the effectiveness of the proposed training mechanisms.

Questions for authors
  1. What is the motivation for providing agents with ground-truth verifier feedback during both training and evaluation? How do the authors justify this design, given that final evaluation tests and their outcomes are typically hidden in SWE tasks? If the authors can provide a convincing justification for this protocol and address my concerns, I would be willing to reconsider my rating.

  2. Could the authors evaluate FC-SWE using only tests or verification tools that would realistically be available to the agent, while reserving the official SWE-bench tests exclusively for final scoring? This comparison would be important for establishing the practical validity of the approach.

  3. Appendix B.1 mentions using 100 SWE-bench Verified tasks for checkpoint selection. Could the authors report results on the remaining 400 tasks and clarify how checkpoint selection was performed for all compared methods?

  4. Could the authors provide compute-matched comparisons with GRPO and an independent ablation of active-set advantage estimation to better establish the contribution of the proposed RL mechanisms?

2 comments
AuthorsOpenReview ↗

Comment 1 of 2: verifier feedback (Weakness 1, Q1, Q2)

Answer in brief. FC-SWE's advantage over GRPO does not come from the hidden tests. Following your Q2, we separated second-attempt recovery by what the retry context actually contained: the advantage is largest when the context contains no test output at all, and it vanishes statistically where hidden-test diagnostics are shown.

Q2 — recovery without official-test feedback. We split every first-attempt failure (all 500 tasks, three decoding seeds) by the content of its retry context. In the first group the attempt produced no applicable patch, so the retry context holds 0 characters of verifier output, and the failure is visible to any agent harness without running a single test. In the third group the context contains diagnostics from failing FAIL_TO_PASS tests, which is the hidden-test feedback you describe.

Retry context after the first failureBaseGRPOFC-SWEFC-SWE − GRPO [95% CI]
No verifier output (no applicable patch)12.0% (61/507)18.6% (64/344)26.9% (57/212)+8.3 [+1.4, +15.2]
Output with PASS_TO_PASS failures only15.0% (6/40)16.3% (7/43)22.7% (15/66)+6.4 [−7.2, +19.2]
Output with FAIL_TO_PASS failures10.9% (58/534)13.8% (73/530)15.8% (94/596)+2.0 [−1.7, +5.7]
All first-attempt failures11.6%15.7%19.0%+3.3 [+0.3, +6.3]

Cells give second-attempt recovery; intervals are task-clustered bootstrap CIs.

Three facts settle the question. First, with no test output in context, FC-SWE recovers 26.9% of failures against 18.6% for GRPO, and the gap is even larger on the 400 tasks that played no part in checkpoint selection (29.7% vs. 19.3%; +10.4 [+2.5, +18.2]). Second, where hidden-test diagnostics are shown, the two policies differ by only 2.0 points with an interval that includes zero, so test-directed refinement cannot be what separates them. Third, Resolved@1 involves no verifier feedback and is scored once by the official tests, exactly as in standard SWE-bench, and on this metric FC-SWE improves on GRPO by 2.9 points [+0.3, +5.4].

Q1 — why verifier feedback, in training and in evaluation. The paper asks one question: can a policy be trained to turn evidence of failure into a better next attempt? Answering it requires a feedback source that is reliable and identical for every compared policy, and the benchmark verifier is the only source that meets both conditions without introducing a second, confounding variable (the quality of a learned critic or of generated tests). In training, the verifier runs on R2E-Gym tasks, whose tests already define the reward of the GRPO baseline; FC-SWE additionally shows the policy the diagnostic text of those training tasks, and no SWE-bench test enters any parameter update. In evaluation, the result is a controlled comparison at identical access, and it carries a finding that test access alone cannot produce: Base and GRPO recover fewer failures with this feedback than with blind resampling (11.6% vs. 13.7%; 15.7% vs. 18.6%), whereas only the policy trained with FC-SWE reverses the sign (19.0% vs. 17.0%; Figure 4a). Access to the tests is therefore not what yields recovery; training is.

What Resolved@K measures. For K ≥ 2, Resolved@K is verifier-assisted recovery. The paper declares this in the abstract, in Section 4.1 and in Appendix A, and never presents these numbers as leaderboard scores. The stratified table above joins Figure 4 in Section 4.4, so that the feedback-free and feedback-dependent parts of the result can be read separately.

AuthorsOpenReview ↗

Comment 2 of 2: held-out 400 tasks, compute, active set (Weakness 2, Q3, Q4)

Answer in brief. On the 400 tasks never used for checkpoint selection, FC-SWE leads GRPO by 3.0, 4.7 and 3.8 points at K = 1, 2 and 11, which is more than on all 500 tasks, and every interval excludes zero. The lead is not bought with extra training trajectories: a control that executes the same recovery trajectories without failure context falls below GRPO in Resolved@2 and in recovery.

Q3 — the remaining 400 tasks. As you requested, we recomputed every headline metric on the 400 tasks outside the selection subset (mean ± SEM over three decoding seeds; differences with paired, task-clustered bootstrap 95% CIs).

Held-out 400 tasksResolved@1Resolved@2Resolved@11Cond. recovery
Base27.8 ± 1.636.2 ± 1.161.2 ± 1.311.7%
GRPO39.0 ± 2.148.8 ± 2.167.8 ± 1.116.1%
FC-SWE42.0 ± 0.853.5 ± 0.971.6 ± 1.319.8%
FC-SWE − GRPO+3.0 [+0.1, +5.9]+4.7 [+2.1, +7.2]+3.8 [+1.8, +5.9]+3.7 [+0.3, +7.1]

The corresponding differences on all 500 tasks are +2.9 [+0.3, +5.4], +4.3 [+2.1, +6.5] and +3.4 [+1.6, +5.3]. Checkpoint selection followed the same procedure for both trained policies: the checkpoints of each run were evaluated on the same fixed 100-task subset, which gave step 65 for GRPO and step 67 for FC-SWE, whereas Base involves no selection. Since the lead grows on the held-out tasks, selection did not favour FC-SWE.

On the size of the gains. A 4.3-point gain at Resolved@2 lifts a 4B model above an untrained model 2.25 times its size: FC-SWE exceeds Qwen3.5-9B at both budgets under the same protocol (41.7% vs. 41.2%; 52.8% vs. 50.4%), even though the 9B runs had the longer agent timeout, whereas GRPO stays below it (38.9%; 48.5%). The lead also persists to K = 11, far beyond the two-attempt training horizon.

Q4 — training compute. Through step 67, FC-SWE executed 12,839 training trajectories (8,576 initial and 4,263 recovery), compared with 8,320 for GRPO at step 65. Two results show that this difference is not the source of the gain. (i) The w/o failure context variant in Table 2A executes the same failure-triggered recovery attempts with the same rewards and the same advantage estimator, and withholds only the failed patch and verifier output. It reaches 47.9% Resolved@2 and 12.6% recovery, which is below GRPO (48.5%, 15.7%) and far below FC-SWE (52.8%, 19.0%); the same extra compute, spent without failure conditioning, buys no gain in Resolved@2 or in recovery. (ii) GRPO was trained on to step 100, i.e. 12,800 trajectories, which matches FC-SWE's 12,839. Its online success fell from 58.7% (steps 56–65) to 48.4% (steps 91–100), so a matched trajectory budget lowers GRPO's performance instead of closing the gap.

Q4 — active-set advantage estimation. The active set is a correctness condition of the estimator, and the alternatives to it are mis-specified estimators, which is why we did not run them as an ablation. Consider an issue on which five of eight chains succeed at once and one of the three retries succeeds: eleven trajectories were executed, with six successes. A fixed 16-slot group counts the five never-run retries as failures, so the baseline of a failed trajectory drops from 6/10 to 6/15; the penalty on real failures thus shrinks precisely as the policy improves. Grouping by attempt index avoids this but compares recovery trajectories only with each other, and the group degenerates to a single trajectory with no baseline whenever seven chains succeed at first. The active set is the one grouping that uses every executed trajectory and nothing else. The measured gains reported in the paper belong to the two choices that are genuine alternatives: failure-context conditioning (Table 2A: +4.9 Resolved@2, +6.4 recovery) and trajectory-local rewards (Table 2B: +3.0 Resolved@2, +2.7 recovery).

Reviewer_xMFCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper extends GRPO to train SWE agents to be able to recover from failed patches. The key design is that during training, when a submitted patch failed verification, a new attempt will be launched and is conditioned on the failed patch, a summarized verifier diagnostics. The paper shows improved performance with Qwen3.5-4B on SWE-bench Verified.

Critical strengths

This is a well-formulated and practical problem. The idea to teach the model to recover from failure experiences is clear and neat.

The proposed solution is intuitive and easy to implement.

Great design of the controled experiments to analyze the effectiveness and role of the feedbacks.

Critical weaknesses

The paper seems to require oracle assistance at inference. Specifically, the controller gives away the diagnostics between attempts that involve official test results. In this sense, the Resolved@K seems to be an oracle-assisted metric, as compared to other standard SWE-bench metrics.

The proposed FC-SWE seems to use more traces when compared with standard GRPO.

I'd also like the authors to comment on the model focusing on failed and reattempted cases and how this would affect the genererlization to the general SWE tasks.

The evaluation focuses on a single model on a single benchmark, therefore a bit limited.

Questions for authors

Pease refer to the weakness section.

2 comments
AuthorsOpenReview ↗

Comment 1 of 2: oracle assistance and training traces (Weaknesses 1 and 2)

Answer in brief. FC-SWE's gains do not rest on oracle assistance. On Resolved@1, the standard feedback-free SWE-bench metric, FC-SWE improves over GRPO by 2.9 points, and its recovery advantage is largest precisely where the retry context contains no test output. The extra training traces do not explain the gains either, because a control with the same traces falls below GRPO in Resolved@2 and in recovery.

W1 — which results involve the oracle. Resolved@1 uses a single attempt that is scored once by the official tests, as in standard SWE-bench: FC-SWE reaches 41.7 ± 0.8 against 38.9 ± 1.6 for GRPO (mean ± SEM over three decoding seeds; paired difference +2.9, 95% CI [+0.3, +5.4]). Resolved@K for K ≥ 2 is verifier-assisted, and the paper reports it under that name instead of as a standard score. To show how much of the second-attempt result depends on the oracle, we added an analysis that splits every first-attempt failure by what its retry context contained; cells give second-attempt recovery.

Retry context after the first failureBaseGRPOFC-SWEFC-SWE − GRPO [95% CI]
No verifier output (no applicable patch)12.0% (61/507)18.6% (64/344)26.9% (57/212)+8.3 [+1.4, +15.2]
Output with PASS_TO_PASS failures only15.0% (6/40)16.3% (7/43)22.7% (15/66)+6.4 [−7.2, +19.2]
Output with FAIL_TO_PASS failures10.9% (58/534)13.8% (73/530)15.8% (94/596)+2.0 [−1.7, +5.7]

In the first row no test is consulted at all: the attempt left no applicable patch, which any agent harness can observe, and the retry context holds 0 characters of verifier output. This is where FC-SWE gains most over GRPO (26.9% vs. 18.6%). In the third row, where diagnostics from the hidden FAIL_TO_PASS tests are shown, the two policies differ by 2.0 points with an interval that includes zero. The oracle diagnostics therefore raise the absolute level of Resolved@K for every policy, but they are not what separates FC-SWE from GRPO.

W2 — more training traces than GRPO. Through step 67, FC-SWE executed 12,839 training trajectories (8,576 initial and 4,263 recovery), compared with 8,320 for GRPO at step 65. Two results show that this difference is not the source of the gain. (i) The w/o failure context variant in Table 2A executes the same failure-triggered recovery attempts with the same rewards and the same advantage estimator, and withholds only the failed patch and verifier output. It reaches 47.9% Resolved@2 and 12.6% recovery, which is below GRPO (48.5%, 15.7%) and far below FC-SWE (52.8%, 19.0%); the same extra traces, generated without failure conditioning, buy no gain in Resolved@2 or in recovery. (ii) GRPO was trained on to step 100, i.e. 12,800 trajectories, which matches FC-SWE's 12,839. Its online success fell from 58.7% (steps 56–65) to 48.4% (steps 91–100), so a matched trajectory budget lowers GRPO's performance instead of closing the gap.

AuthorsOpenReview ↗

Comment 2 of 2: generalization to general SWE tasks; one model, one benchmark (Weaknesses 3 and 4)

Answer in brief. Training on failed and reattempted cases makes the policy better at general SWE tasks, not narrower. Ordinary single-attempt solving improves from 38.9% to 41.7%, on tasks, repositories and an agent scaffold that were all unseen during training, and the resulting 4B policy outperforms an untrained 9B model.

W3 — effect on general SWE tasks. The evidence is Resolved@1, in which the policy receives only the issue and makes one attempt without any feedback. FC-SWE reaches 41.7 ± 0.8 against 38.9 ± 1.6 for GRPO (paired difference +2.9, 95% CI [+0.3, +5.4]), and 42.0 ± 0.8 against 39.0 ± 2.1 on the 400 tasks that played no part in checkpoint selection. Three properties of the training explain why first attempts benefit instead of being traded away.

  • Recovery is added to ordinary training and replaces none of it. Every update still contains all 128 initial trajectories, so 8,576 of the 12,839 trajectories executed through step 67 are plain first attempts.
  • A recovery attempt is itself a full SWE task. It restarts from the clean repository and must resolve the whole issue end to end, which makes it additional full-task practice concentrated on the issues the policy has just failed. Consistent with this reading, the variant trained with retries but without failure context already reaches 40.4% Resolved@1, and failure conditioning raises it further to 41.7% (Table 2A).
  • First attempts become more reliable. Over the 1,500 evaluation runs, first attempts that end without an applicable patch fall from 344 for GRPO to 212 for FC-SWE, and the share of training attempts truncated by the token budget falls from 10.5% to 5.2% (Appendix F).

W4 — a single model and a single benchmark. The evaluation is a transfer test along three axes at once: the policy is trained on 4,518 R2E-Gym tasks from ten repositories with OpenHands, and it is evaluated on all 500 SWE-bench Verified tasks, which share no repository with the training set, under a different scaffold (SWE-agent). The gains hold across three decoding seeds, on the 400 tasks outside checkpoint selection, and at every attempt budget from 1 to 11. Table 1 further shows, under the same controller, that weak recovery is a property of models without recovery training and not of one particular model: conditional recovery is 11.6% for Qwen3.5-4B, 10.6% for Nemotron-3-Nano-4B and 15.6% for Qwen3.5-9B, all below the 19.0% of FC-SWE. A model 2.25 times larger thus recovers less often than the FC-SWE 4B policy, which also exceeds it at both budgets (41.7% vs. 41.2% Resolved@1; 52.8% vs. 50.4% Resolved@2), even though the 9B runs had the longer agent timeout.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Shared concerns: hidden-test feedback, checkpoint selection, training trajectories

Answer in brief. We thank the reviewers for their careful reading. Three concerns recur across the reviews, and on each the evidence points the same way: FC-SWE's advantage over GRPO is largest where the retry context contains no test output, it grows on the 400 tasks outside checkpoint selection, and it is absent in a control that executes the same extra trajectories without failure context. The analyses below were added in response to the reviews; they are computed from the per-task outputs of the evaluation runs reported in the paper (500 tasks, three decoding seeds).

1. Hidden-test feedback (xMFC W1; i5pQ W1, Q1, Q2; tRnM W1). The hidden tests are not what separates FC-SWE from GRPO. We split every first-attempt failure by what its retry context contained; cells give second-attempt recovery, with task-clustered bootstrap CIs.

Retry context after the first failureBaseGRPOFC-SWEFC-SWE − GRPO [95% CI]
No verifier output (no applicable patch)12.0% (61/507)18.6% (64/344)26.9% (57/212)+8.3 [+1.4, +15.2]
Output with PASS_TO_PASS failures only15.0% (6/40)16.3% (7/43)22.7% (15/66)+6.4 [−7.2, +19.2]
Output with FAIL_TO_PASS failures10.9% (58/534)13.8% (73/530)15.8% (94/596)+2.0 [−1.7, +5.7]

In the first row no test is consulted: the attempt left no applicable patch, which any agent harness can observe, and the retry context holds 0 characters of verifier output. FC-SWE gains most here (26.9% vs. 18.6%), and more still on the 400 held-out tasks (29.7% vs. 19.3%; +10.4 [+2.5, +18.2]). In the third row, where diagnostics from FAIL_TO_PASS tests are shown, the two policies differ by 2.0 points with an interval that includes zero. Two further results agree with this reading. Resolved@1 involves no feedback and is scored once by the official tests, as in standard SWE-bench, and FC-SWE improves on GRPO by 2.9 points. In addition, test access by itself does not yield recovery: Base and GRPO recover fewer failures with this feedback than with blind resampling (11.6% vs. 13.7%; 15.7% vs. 18.6%), and only the policy trained with FC-SWE reverses the sign (19.0% vs. 17.0%; Figure 4a). Resolved@K for K ≥ 2 is verifier-assisted recovery, declared as such in the abstract, Section 4.1 and Appendix A; every claim attached to it is a comparison of policies at identical access.

2. Checkpoint selection and seed variability (i5pQ Q3; tRnM Q1, Q2). The lead is larger on the 400 tasks never used for selection than on all 500, and every interval excludes zero. Entries are mean ± SEM over three decoding seeds; differences carry paired, task-clustered bootstrap 95% CIs.

Resolved@1Resolved@2Resolved@11
All 500: GRPO38.9 ± 1.648.5 ± 1.767.3 ± 1.2
All 500: FC-SWE41.7 ± 0.852.8 ± 1.070.7 ± 1.3
All 500: FC-SWE − GRPO+2.9 [+0.3, +5.4]+4.3 [+2.1, +6.5]+3.4 [+1.6, +5.3]
Held-out 400: GRPO39.0 ± 2.148.8 ± 2.167.8 ± 1.1
Held-out 400: FC-SWE42.0 ± 0.853.5 ± 0.971.6 ± 1.3
Held-out 400: FC-SWE − GRPO+3.0 [+0.1, +5.9]+4.7 [+2.1, +7.2]+3.8 [+1.8, +5.9]

Both trained policies were selected with the same procedure on the same fixed 100-task subset (step 65 for GRPO, step 67 for FC-SWE). Since the lead grows on the held-out tasks, selection did not favour FC-SWE.

3. Training trajectories (xMFC W2; i5pQ Q4). The extra trajectories are not the source of the gain. Through step 67, FC-SWE executed 12,839 training trajectories (8,576 initial and 4,263 recovery), compared with 8,320 for GRPO at step 65. (i) The w/o failure context variant in Table 2A executes the same failure-triggered recovery attempts with the same rewards and the same advantage estimator, and withholds only the failed patch and verifier output. It reaches 47.9% Resolved@2 and 12.6% recovery, which is below GRPO (48.5%, 15.7%) and far below FC-SWE (52.8%, 19.0%). (ii) GRPO was trained on to step 100, i.e. 12,800 trajectories, which matches FC-SWE's 12,839. Its online success fell from 58.7% (steps 56–65) to 48.4% (steps 91–100), so a matched trajectory budget lowers GRPO's performance instead of closing the gap.

Changes in the revision. Table 1 reports mean ± SEM across decoding seeds; the stratified recovery table joins Figure 4 in Section 4.4; the held-out-400 results accompany the description of checkpoint selection in Appendix B.1. The points raised by individual reviewers (active-set estimation, the relation to RLEF, generalization beyond one model and benchmark) are answered in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.