- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper trains repository-level SWE agents to recover after unsuccessful patches given tests feedback. In case initial verification fails, the repository is reset and the agent receives the previous patch and "verifier" diagnostics. FC-SWE modifies GRPO in two ways: each attempt receives its own success/failure reward, so a successful retry does not change the earlier failed attempt’s reward. For each attempt, the advantage baseline is the average reward of all other executed attempts on the same issue, including both initial attempts and retries.
Authors use hidden tests output for the agent to improve the score during evaluation.
Critical strengths
The paper studies a relevant question: can we train the agent to exploit some external verifier feedback. The baselines are evaluated under the same feedback protocol as FC-SWE, so the within-paper comparison is internally consistent. The training-time failure-context ablation also provides evidence that additional retries alone do not explain the entire improvement.
Critical weaknesses
- The multi-attempt evaluation results rely on hidden-test feedback. The test-time “verifier” is the official SWE-bench harness, including FAIL_TO_PASS tests introduced by the golden PR that are not present in the repository at task time. Up to 6,000 characters of verifier output, including test names, tracebacks, and assertion messages, are provided to the policy before the second attempt.
Numbers such as 70.7% therefore measure performance with access to privileged test feedback. One possible explanation for the small gains: a few percentage points over the baseline - is that this feedback already provides substantial leakage about the solution and it might be quite easy for a model to just extract it.
If the agent can access these tests at inference time, why not make them available from the first attempt? It is difficult to see which practical setting motivates this access pattern. Moreover, SWE-bench Verified test output can reveal the exact location of an issue or specific expected behaviour. Appendix C.1 illustrates this concern: the FC-SWE recovery reasons that the test does not expect the warning and removes it. This passes the hidden tests but, as the authors themselves acknowledge, does not establish full compliance with the issue’s requested deprecation behaviour.
Therefore, the results do not establish that the trained policy can use verifier feedback effectively beyond this benchmark-specific setting. They demonstrate its ability to use leaked information supplied by the benchmark’s evaluation procedure. The paper acknowledges this limitation in Appendix G but provides no experiment using a deployable feedback source, such as checking whether the tests that were already in the repository still pass or submitting agent-generated tests output.
- The novelty is questionable. Multi-turn RL with execution feedback is already central to RLEF (Gehring et al., 2024), here adapted to a repository-level agent setting. In my view, the two proposed mechanisms are relatively small engineering choices within this broader approach.
Questions for authors
-
The main results are reported without ±SEM, making it difficult to assess the variability relative to the claimed improvements. Readers must consult the appendix E and calculate this themselves, although it is important for interpreting the relatively small gains. Please report mean ±SEM across decoding seeds directly in the main tables for comparisons evaluated with multiple seeds.
-
Checkpoint selection overlaps with the reported evaluation set. Appendix B.1 states that the FC-SWE checkpoint was selected using multiple validation sets, including 100 SWE-bench Verified tasks, while the headline results cover all 500 tasks. Consequently, the reported evaluation is not wholly held out from model selection. What is the score on the 400 SWE-bench tasks not used for selection?
2 comments
Comment 2 of 2: novelty relative to RLEF, ±SEM, held-out 400 tasks
Answer in brief. FC-SWE is not RLEF moved to repositories. RLEF trains a single episode of at most three code solutions, in which feedback arrives between turns and one end-of-episode reward is credited to all turns; FC-SWE trains a chain of complete, separately verified trajectories, and our ablation shows that carrying RLEF-style shared credit over to such chains costs 3.0 points of Resolved@2 and can collapse training. The ±SEM values and the held-out-400 scores you asked for are given below.
Novelty relative to RLEF. The two settings pose different RL problems, and the difference is substantive for three reasons.
- The unit of feedback changes the estimator. In RLEF, a turn is one code solution to a competitive-programming task, public-test results are appended to the same conversation, and a learned value function serves as the baseline. In FC-SWE, each attempt is a tool-use trajectory of up to 100 interactions that ends in its own verification, and the next attempt restarts from a clean repository with a bounded failure context. A recovery trajectory therefore exists only if its predecessor failed, so the comparison group for critic-free advantage estimation has an outcome-dependent size. This problem does not exist in RLEF, whose episodes are scored by a critic instead of by a comparison group, and it is precisely the one our two mechanisms solve.
- The choice of credit assignment is consequential. Sharing the chain outcome across attempts, which is the direct transfer of episode-level credit, lowers Resolved@2 from 54.8% to 51.8% and conditional recovery from 20.4% to 17.7% (Table 2B); once the runtime penalties are also removed, training collapses from 76.6% to 2.3% online success within three steps. A choice with this effect is not a minor engineering detail.
- The setting yields a finding that function-level work could not. For repository-level agents, feedback is not automatically useful: Base and GRPO recover fewer failures with feedback than with blind resampling (−2.1 and −2.9 points), and only failure-conditioned training reverses the sign (+2.0; Figure 4a). Removing failure context from training alone reduces recovery from 19.0% to 12.6% (Table 2A).
Q1 — mean ± SEM in the main table. Following your request, Table 1 reports mean ± SEM across the three decoding seeds.
| All 500 tasks | Resolved@1 | Resolved@2 | Resolved@11 |
|---|---|---|---|
| Base | 27.9 ± 1.6 | 36.3 ± 1.5 | 61.1 ± 1.5 |
| GRPO | 38.9 ± 1.6 | 48.5 ± 1.7 | 67.3 ± 1.2 |
| FC-SWE | 41.7 ± 0.8 | 52.8 ± 1.0 | 70.7 ± 1.3 |
| FC-SWE − GRPO | +2.9 [+0.3, +5.4] | +4.3 [+2.1, +6.5] | +3.4 [+1.6, +5.3] |
Because the three seeds share the same tasks, the last row gives the paired, task-clustered bootstrap 95% CI of the difference. All three intervals exclude zero, so the gains are not within seed variability.
Q2 — the 400 tasks not used for selection. The lead is larger on these tasks than on all 500, and again every interval excludes zero.
| Held-out 400 tasks | Resolved@1 | Resolved@2 | Resolved@11 |
|---|---|---|---|
| Base | 27.8 ± 1.6 | 36.2 ± 1.1 | 61.2 ± 1.3 |
| GRPO | 39.0 ± 2.1 | 48.8 ± 2.1 | 67.8 ± 1.1 |
| FC-SWE | 42.0 ± 0.8 | 53.5 ± 0.9 | 71.6 ± 1.3 |
| FC-SWE − GRPO | +3.0 [+0.1, +5.9] | +4.7 [+2.1, +7.2] | +3.8 [+1.8, +5.9] |
GRPO's checkpoint (step 65) was selected with the same procedure and on the same fixed 100-task subset as FC-SWE's (step 67). Since the lead grows on the held-out tasks, selection did not favour FC-SWE.
Comment 1 of 2: hidden-test feedback (Weakness 1)
Answer in brief. Leaked test information does not produce the gap between FC-SWE and GRPO. The gap is concentrated in failures whose retry context contains no test output, and it is statistically indistinguishable from zero where FAIL_TO_PASS diagnostics are shown. We ran this stratified analysis in response to your request for a deployable feedback source.
Recovery by feedback content. We split every first-attempt failure (all 500 tasks, three decoding seeds) by what its retry context contained; cells give second-attempt recovery, with task-clustered bootstrap CIs.
A deployable feedback source. The first row is a deployable setting, because no test is consulted: the harness observes that the attempt left no applicable patch, and the retry context holds 0 characters of verifier output. Here FC-SWE recovers 26.9% of failures against 18.6% for GRPO. The second row corresponds to your suggestion of checking whether existing tests still pass; FC-SWE reaches 22.7% against 16.3%, although with 66 and 43 failures the interval remains wide. Resolved@1, which involves no feedback of any kind, improves by 2.9 points [+0.3, +5.4].
"The feedback leaks the solution and is easy to extract." The data contradict this explanation twice. If the 6,000-character excerpt made the fix easy to extract, every policy would profit from it; yet Base and GRPO recover fewer failures with this feedback than with blind resampling at the same budget (11.6% vs. 13.7%, and 15.7% vs. 18.6%; Figure 4a). Extracting a repair from test output is evidently hard for a 4B policy, and it becomes possible only after failure-conditioned training (19.0% vs. 17.0%). Moreover, all three policies receive identical feedback, so leakage could separate FC-SWE from GRPO only if FC-SWE exploited it better, and the third row shows that on exactly these failures the two policies differ by 2.0 points with an interval that includes zero.
"Why not provide the tests from the first attempt?" The feedback is the execution result of a submitted patch, and therefore cannot exist before a patch does. The access pattern is that of continuous integration and code review, where a submission is tested and the log is returned to its author. The agent receives at most 6,000 characters of output for its own patch and never the test files.
Appendix C.1. C.1 is a single selected case from the FAIL_TO_PASS stratum, and the paper says explicitly that it does not establish full compliance with the issue. It illustrates one behaviour: FC-SWE acts on the feedback where GRPO resubmits the same patch. No aggregate claim rests on it; the aggregate advantage rests on the first two rows above and on Resolved@1.
The 70.7% figure. The paper reports this number as verifier-assisted in the abstract and never as a standard score. The claim attached to it is a comparison at identical access: +3.4 points over GRPO at K = 11 [+1.6, +5.3], reached by a policy that was trained with at most two attempts.