- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Retrospection-Only Fine-Tuning (ROFT). After attempting a task, the agent generates retrospections from its experience and available feedback, then trains only on the retrospection tokens. Software-engineering experiments show improvements in subsequent task performance without supplying retrospections at inference.
Critical strengths
- Learning from unsuccessful attempts without requiring successful action targets is a useful direction.
- The procedure is simple, and the experiments examine several training settings and aspects of the retrospection learning signal.
- The behavioral results are interesting, particularly the effect of retrospection instructions on subsequent rollout length and the combination with GRPO.
Critical weaknesses
-
The main missing comparison is SFT on successful trajectories from the same collected attempts. ROFT requires additional inference to generate retrospections after collecting those attempts. A comparison using the same source data, with both collection and training costs accounted for, is needed to establish whether this extra step is worthwhile. ERL and SCFT change other parts of the pipeline and do not isolate this choice of training target.
-
The specific value of retrospection remains unclear. ROFT is self-generated SFT conditioned on a retrospection instruction and the recorded experience. Most ablations examine choices within this setup, rather than distinguish retrospection training from ordinary self-SFT or training on other experience-derived text. The behavioral analyses are suggestive but do not establish a retrospection-specific transfer mechanism.
-
The benefit of continued training is not established. ROFT makes early progress, but its training performance later plateaus and declines. GRPO’s rising training reward does not establish better generalization either. The current results support early gains, but leave unclear whether ROFT can keep extracting useful supervision as training continues.
Questions for authors
- How does ROFT compare with SFT on successful trajectories from the same attempts under matched collection and total compute budgets? Does combining the two provide further gains?
- Would self-SFT on trajectory summaries or other experience-derived text produce similar improvements?
- What causes the later plateau and decline, and can refreshing the retrospection data sustain improvements on held-out tasks?
2 comments
Response to Reviewer FYUd (2/2): Plateau and decline; refreshing the retrospection data
GRPO's test reward plateaus then declines. What causes the plateau and then decline for ROFT? Can refreshing the retrospection data sustain improvements and avoid decline?
We thank the reviewer for this question. ROFT plateaus and then declines for the same reason GRPO does: overfitting to the training tasks. A ROFT variant that refreshes the retrospection data by rotating through retrospection prompts holds its held-out score much longer: 48.2% on SWE-bench Verified at update 70, whereas standard ROFT has fallen to 41.6% by update 40.
Cause of the plateau and decline. Held-out accuracy peaks and then falls for both methods: GRPO's goes from 48.0% at update 40 to 46.0% at update 90, and ROFT's from 49.2% at update 20 to 41.6% at update 40. For ROFT, the training set is fixed, so later updates revisit tasks the model has already reflected on, and it writes similar retrospections about them given the same retrospection prompt. Thus further training fits the training tasks more closely instead of generalizing (Sec. 3.2; the SFT loss and entropy of the retrospections fall throughout training, App. E.3). Accordingly, our claim is that ROFT gives early, inexpensive gains at competitive accuracy, not that it improves monotonically with training length.
Refreshing the retrospection data. We tested this with a variant that rotates through retrospection prompts instead of using a single one. The new training-curve comparison in the revised paper (new App. E.4) shows that the variant lasts longer: standard ROFT's training reward peaks at 0.563 near update 18 and falls to 0.483 by update 40, whereas the variant's stays near 0.60 (between 0.56 and 0.64) through update 80. On SWE-bench Verified, the variant still scores 48.2% at update 70.
We again thank the reviewer, and hope that these responses and the new results address their concerns. We would be glad to answer any further questions.
Response to Reviewer FYUd (1/2): SFT baselines and the retrospection transfer mechanism
We thank the reviewer for their thoughtful review and constructive questions. In response, we ran additional experiments and analysis and address each point below:
We thank the reviewer for these questions. Yes to both. Success-filtered SFT scores 44.0% on SWE-bench Verified at update 10 and degrades to 40.8% at update 20. Combined with ROFT, the same run reaches 45.2% and 47.6%.
New experiment: success-filtered SFT, alone and combined with ROFT. Both runs keep ROFT's training setup and change only the training target. Each update collects the same 64 attempts as ROFT. Success-filtered SFT keeps the attempts that pass the tests (about 36 of the 64 per update) and fine-tunes on the model's own reasoning and actions in them, with no retrospections. The combined run adds ROFT's retrospection loss on the same attempts. Both train for 20 updates and are evaluated on SWE-bench Verified (500 tasks).
Success-filtered SFT degrades between update 10 and update 20, likely due to overfitting to its own actions. Consistent with this, its evaluation rollouts get longer: 277 of the 500 run into the 100-step limit at update 20, up from 212 at update 10. With retrospection targets added, the same run keeps improving and ends 6.8 points higher at update 20. This shows a clear advantage of training on retrospections over training directly on actions, which tends to overfit and harden the model's behavior. The two also differ structurally: on tasks where none of the 64 attempts succeeds (Sec. 3.1), success-filtered SFT has no training data, while ROFT still learns from the failed attempts. We thank the reviewer for suggesting this baseline, which has helped strengthen our claims.
We thank the reviewer for these questions. Yes to both. Self-SFT on the trajectory summaries of the same attempts, with no retrospection ("Trajectory-only SFT" below), scores lower than ROFT. Over full training runs, three baselines that fine-tune on other experience-derived text score 45.6% to 46.8% on SWE-bench Verified, below ROFT's 49.2%. For the mechanism, a new attention-weighting experiment gives supporting evidence (+4.1 points over uniform weighting).
New experiment: self-SFT on the trajectory summary. We use the one-update setup of App. G.3: Qwen3.5-4B receives one update built from 64 attempts (one per training task) and is then evaluated on those 64 tasks with ten paired seeds (640 attempts per arm). Uniform ROFT trains on the model's retrospections of those attempts, as in the paper. Attention-up is described under "Mechanism" below.
After one update, Trajectory-only SFT scores 0.9 points below Uniform ROFT.
Other experience-derived text. The paper compares ROFT with three methods that fine-tune on other text derived from the model's attempts, all with Qwen3.5-4B on SWE-bench Verified (Fig. 10, App. C).
All three score below ROFT, by 2.4 to 3.6 points.
Mechanism: Attention-up. We hypothesize that predicting retrospection tokens sends gradients through attention into the representations of the recorded trajectory, which the model also relies on when it acts. If so, retrospection tokens that attend more to the trajectory should matter more. Attention-up weights each retrospection token by one plus the base model's attention to the prompt content that holds the trajectory, with weights normalized to mean one. It scores 54.7%, +4.1 points over uniform weighting (table above). We offer this as supporting evidence for the mechanism.
Attention-up is in the revised Fig. 7c and App. G.3.