acceptodds
Under review as a conference paper at ICLR 2027

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

Abstract

An unsuccessful attempt can reveal useful lessons even when it earns no reward. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT learns to solve very difficult tasks where the pass@64 is 0 and there is no reward signal. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 29.0% solve rates after 20 updates, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. Behavioral analyses finds that ROFT indirectly conducts credit assignment, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, language model agents, retrospective supervision, software engineering, reinforcement learning from verifiable rewards, sparse rewards, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FYUdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Retrospection-Only Fine-Tuning (ROFT). After attempting a task, the agent generates retrospections from its experience and available feedback, then trains only on the retrospection tokens. Software-engineering experiments show improvements in subsequent task performance without supplying retrospections at inference.

Critical strengths
  • Learning from unsuccessful attempts without requiring successful action targets is a useful direction.
  • The procedure is simple, and the experiments examine several training settings and aspects of the retrospection learning signal.
  • The behavioral results are interesting, particularly the effect of retrospection instructions on subsequent rollout length and the combination with GRPO.
Critical weaknesses
  • The main missing comparison is SFT on successful trajectories from the same collected attempts. ROFT requires additional inference to generate retrospections after collecting those attempts. A comparison using the same source data, with both collection and training costs accounted for, is needed to establish whether this extra step is worthwhile. ERL and SCFT change other parts of the pipeline and do not isolate this choice of training target.

  • The specific value of retrospection remains unclear. ROFT is self-generated SFT conditioned on a retrospection instruction and the recorded experience. Most ablations examine choices within this setup, rather than distinguish retrospection training from ordinary self-SFT or training on other experience-derived text. The behavioral analyses are suggestive but do not establish a retrospection-specific transfer mechanism.

  • The benefit of continued training is not established. ROFT makes early progress, but its training performance later plateaus and declines. GRPO’s rising training reward does not establish better generalization either. The current results support early gains, but leave unclear whether ROFT can keep extracting useful supervision as training continues.

Questions for authors
  • How does ROFT compare with SFT on successful trajectories from the same attempts under matched collection and total compute budgets? Does combining the two provide further gains?
  • Would self-SFT on trajectory summaries or other experience-derived text produce similar improvements?
  • What causes the later plateau and decline, and can refreshing the retrospection data sustain improvements on held-out tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer FYUd (1/2): SFT baselines and the retrospection transfer mechanism

We thank the reviewer for their thoughtful review and constructive questions. In response, we ran additional experiments and analysis and address each point below:


Is there an comparison against a success-trajectory SFT baseline, where we SFT on successful trajectories only? It is possible to combine this with ROFT?

We thank the reviewer for these questions. Yes to both. Success-filtered SFT scores 44.0% on SWE-bench Verified at update 10 and degrades to 40.8% at update 20. Combined with ROFT, the same run reaches 45.2% and 47.6%.

New experiment: success-filtered SFT, alone and combined with ROFT. Both runs keep ROFT's training setup and change only the training target. Each update collects the same 64 attempts as ROFT. Success-filtered SFT keeps the attempts that pass the tests (about 36 of the 64 per update) and fine-tunes on the model's own reasoning and actions in them, with no retrospections. The combined run adds ROFT's retrospection loss on the same attempts. Both train for 20 updates and are evaluated on SWE-bench Verified (500 tasks).

ModelUpdate 10Update 20
Success-filtered SFT44.0%40.8%
Success-filtered SFT + ROFT45.2%47.6%

Success-filtered SFT degrades between update 10 and update 20, likely due to overfitting to its own actions. Consistent with this, its evaluation rollouts get longer: 277 of the 500 run into the 100-step limit at update 20, up from 212 at update 10. With retrospection targets added, the same run keeps improving and ends 6.8 points higher at update 20. This shows a clear advantage of training on retrospections over training directly on actions, which tends to overfit and harden the model's behavior. The two also differ structurally: on tasks where none of the 64 attempts succeeds (Sec. 3.1), success-filtered SFT has no training data, while ROFT still learns from the failed attempts. We thank the reviewer for suggesting this baseline, which has helped strengthen our claims.

Are there comparisons against SFTing on the trajectory summaries alone (without the retrospection) or other experience-derived text? Is there more analysis to support the retrospection transfer mechanism?

We thank the reviewer for these questions. Yes to both. Self-SFT on the trajectory summaries of the same attempts, with no retrospection ("Trajectory-only SFT" below), scores lower than ROFT. Over full training runs, three baselines that fine-tune on other experience-derived text score 45.6% to 46.8% on SWE-bench Verified, below ROFT's 49.2%. For the mechanism, a new attention-weighting experiment gives supporting evidence (+4.1 points over uniform weighting).

New experiment: self-SFT on the trajectory summary. We use the one-update setup of App. G.3: Qwen3.5-4B receives one update built from 64 attempts (one per training task) and is then evaluated on those 64 tasks with ten paired seeds (640 attempts per arm). Uniform ROFT trains on the model's retrospections of those attempts, as in the paper. Attention-up is described under "Mechanism" below.

ArmPass ratevs. Uniform ROFT, points
Uniform ROFT50.6%—
Trajectory-only SFT49.7%−0.9
Attention-up54.7%+4.1

After one update, Trajectory-only SFT scores 0.9 points below Uniform ROFT.

Other experience-derived text. The paper compares ROFT with three methods that fine-tune on other text derived from the model's attempts, all with Qwen3.5-4B on SWE-bench Verified (Fig. 10, App. C).

MethodTrained onSWE-bench Verified
ERLReflections and the retries they guide, when the retry succeeds45.6%
CFTCritiques of the model's attempts, written by a teacher model46.0%
SCFTThe model's own critiques and revisions of its attempts, when the revision succeeds46.8%
ROFTThe model's own retrospections49.2%

All three score below ROFT, by 2.4 to 3.6 points.

Mechanism: Attention-up. We hypothesize that predicting retrospection tokens sends gradients through attention into the representations of the recorded trajectory, which the model also relies on when it acts. If so, retrospection tokens that attend more to the trajectory should matter more. Attention-up weights each retrospection token by one plus the base model's attention to the prompt content that holds the trajectory, with weights normalized to mean one. It scores 54.7%, +4.1 points over uniform weighting (table above). We offer this as supporting evidence for the mechanism.

Attention-up is in the revised Fig. 7c and App. G.3.

AuthorsOpenReview ↗

Response to Reviewer FYUd (2/2): Plateau and decline; refreshing the retrospection data

GRPO's test reward plateaus then declines. What causes the plateau and then decline for ROFT? Can refreshing the retrospection data sustain improvements and avoid decline?

We thank the reviewer for this question. ROFT plateaus and then declines for the same reason GRPO does: overfitting to the training tasks. A ROFT variant that refreshes the retrospection data by rotating through retrospection prompts holds its held-out score much longer: 48.2% on SWE-bench Verified at update 70, whereas standard ROFT has fallen to 41.6% by update 40.

Cause of the plateau and decline. Held-out accuracy peaks and then falls for both methods: GRPO's goes from 48.0% at update 40 to 46.0% at update 90, and ROFT's from 49.2% at update 20 to 41.6% at update 40. For ROFT, the training set is fixed, so later updates revisit tasks the model has already reflected on, and it writes similar retrospections about them given the same retrospection prompt. Thus further training fits the training tasks more closely instead of generalizing (Sec. 3.2; the SFT loss and entropy of the retrospections fall throughout training, App. E.3). Accordingly, our claim is that ROFT gives early, inexpensive gains at competitive accuracy, not that it improves monotonically with training length.

Refreshing the retrospection data. We tested this with a variant that rotates through retrospection prompts instead of using a single one. The new training-curve comparison in the revised paper (new App. E.4) shows that the variant lasts longer: standard ROFT's training reward peaks at 0.563 near update 18 and falls to 0.483 by update 40, whereas the variant's stays near 0.60 (between 0.56 and 0.64) through update 80. On SWE-bench Verified, the variant still scores 48.2% at update 70.


We again thank the reviewer, and hope that these responses and the new results address their concerns. We would be glad to answer any further questions.

Reviewer_bVB6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ROFT trains an agent only on self-generated retrospections of its own attempts. Actions and observations provide context but receive no target loss. Later attempts use updated weights without stored reflections. Coding experiments show improved task performance and faster early progress than the evaluated GRPO baseline, with additional behavioral and ablation studies.

Critical strengths
  • The method isolates transfer through weights without direct action-target supervision.
  • Two test sets, a 9B extension, and a no-final-verdict ablation support the central claim.
  • Detailed prompts and cost accounting make the procedure understandable.
Critical weaknesses
  • ROFT’s smoothed training reward falls from 0.563 near update 18 to 0.483 at update 40, but the corresponding 40-update held-out results are not reported (Appendix E.3). This leaves the durability of its early gains and the sensitivity to checkpoint selection unclear.
  • On SWE-bench Verified, the main 4B comparison improves from 48.0% to 49.2%, a difference of six tasks, without reported uncertainty across training seeds. Furthermore, all reported 95% intervals for token-weighting improvements over Uniform include zero (Table 5). These results leave uncertainty about both reliable superiority and the claimed benefits of specific retrospection content.
  • ROFT has no independent semantic-quality filter. The experiments do not establish how incorrect explanations affect learning or whether accurate diagnostic content is necessary for improvement. The no-verdict ablation also retains observations from the agent’s own tests, so robustness under less informative feedback remains unclear (Appendix H.1).
Questions for authors
  • How were the reported checkpoints selected? Can you report held-out performance at update 40 and explain whether the decline in training reward also affects generalization?
  • Can you provide repeated training and evaluation runs, with paired confidence intervals for the main benchmark differences? Which conclusions about token weighting remain supported after accounting for uncertainty?
  • Can you audit retrospection accuracy and compare original retrospections with token-matched factual summaries and deliberately corrupted explanations? Does ROFT still improve when detailed test feedback is unavailable?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bVB6 (1/2): Checkpoint selection, training seeds, and Table 5

We thank the reviewer for their thoughtful review and constructive questions. In response, we ran additional experiments and analysis and address each point below:


What is ROFT test performance at update 40? What is the policy for checkpoint selection?

We thank the reviewer for these questions. ROFT scores 41.6% on SWE-bench Verified at update 40: its held-out score rises to 49.2% at update 20 and then declines. Both methods follow the same checkpoint-selection rule: we report the best evaluated checkpoint within each method's training budget, together with both methods at matched updates.

ROFT scores 49.0% at update 10 and 49.2% at update 20 (Fig. 5a), then declines to 41.6% at update 40 (an evaluation not in the submission). GRPO's held-out score has the same rise-then-decline shape over a longer horizon: it peaks at at update 40 and falls by update 90.

Checkpoint selection. We evaluate saved checkpoints on SWE-bench Verified at multiples of ten updates. For each method we report the best evaluated checkpoint within its training budget (ROFT at update 20 and GRPO at update 40, Fig. 5b), and we report both methods at the matched updates 10 and 20 (Fig. 5a). Selecting by training reward instead would be less favorable to the baseline: GRPO's training reward is highest late in training, where its Verified score is 46.0% (update 90), whereas ROFT's training reward peaks near update 18 (App. E.3), close to the checkpoint we report. The comparison also does not depend on which early ROFT checkpoint is chosen. ROFT at update 10 (49.0%) and at update 20 (49.2%) both score above GRPO's best evaluated checkpoint (48.0%), and above GRPO at the same updates (46.4% and 47.8%).

Would like to see more ROFT training seeds

We thank the reviewer for this request. We now report 9 evaluation runs of ROFT on SWE-bench Verified across three training seeds. The scores fall between 48.0% and 49.4%, with a mean of 48.9% and a standard deviation of 0.5 points.

ROFT on SWE-bench Verified, 9 evaluation runs across three training seeds.

Statistic (n = 9)Score
Mean48.87%
Median49.0%
Std. dev. (sample)0.53 pts
Std. error of mean0.18 pts

All nine runs lie within 1.4 points (seven tasks) of one another.

Under 95% CI, results in Table 5 of the Appendix do not appear to be signficant

We thank the reviewer for their careful examination of our appendix. We agree that no weighting variant is individually significant at the 95% level; we present this study as an exploratory analysis and report it with standard errors (SE). Each variant needs its own training run and 640 agentic rollouts sampled at temperature 1 (64 problems, ten seeds each), so the study was sized to show the direction and approximate size of each effect, not to test each variant at 95%. For that purpose we believe ±1 SE is the better measure of uncertainty: it states the precision of each estimate directly, and the reader can scale it to any confidence level (about ±2 SE for 95%).

We would kindly direct the reviewer to Fig. 7c in the main text, which shows these results with ±1 SE bars. Doubling the weight of evidence, correction, and lesson tokens changes the solve rate by +3.59, +3.75, and +3.13 percentage points relative to uniform weighting, each with an SE of 2.1 to 2.3 points, so each gain is about 1.5 to 1.8 SE above zero.

This experiment is an analysis of which retrospection content matters most, and none of the paper's main claims depend on it. We have qualified it accordingly in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer bVB6 (2/2): Retrospection quality; ROFT without test feedback

How big an impact does retrospection quality have on training?

We thank the reviewer for raising this question. We have now measured it: tasks with better retrospections gain more from ROFT, and the effect is modest (correlations of 0.19 to 0.29 across 64 training tasks).

New analysis: does retrospection quality track per-task gain? For each of the 64 training tasks in App. G.3, a separate judge model (GPT-6 Astra) rated every retrospection used as a training target (255 in total), giving a 1-5 quality score and an accurate / inaccurate / unverifiable label. We then correlated each task's ratings with its gain from one ROFT update, defined as its pass rate after the update minus its pass rate before, over ten paired evaluation seeds.

Retrospection measure, per taskPearson r [90% CI]Spearman rho [90% CI]
Mean quality score (1-5)0.24 [0.04, 0.41]0.19 [-0.02, 0.38]
Fraction labeled accurate0.29 [0.07, 0.47]0.23 [0.01, 0.42]

Tasks with higher-rated retrospections tended to gain more from the update, both by the quality score and by the fraction labeled accurate. All four correlations are positive, and three of the four 90% intervals exclude zero. The effect is modest: retrospection quality explains about 6 to 8% of the variance in per-task gain (Pearson r^2). Intervals are percentile bootstrap with 10,000 resamples of the 64 tasks. We have added this analysis to the revised paper (new App. G.5).

The no-verdict run still contains tests, so it is unclear whether ROFT works when detailed test feedback is unavailable.

We thank the reviewer for raising this. Yes, ROFT works without test feedback: in a new experiment on text-adventure games, where no tests exist at all, ROFT improves all three models we tested. Scores rise by 3.2 to 13.3 points. We first clarify what the no-verdict run contains.

What the no-verdict run contains. Environment output remains in this run. It comes only from code that the agent itself chose to run while working on the task. This is ordinary interaction with the environment, available in any deployment, and involves no privileged verifier. Everything produced by the grader is removed from the retrospection prompt: the ground-truth test results, the grader's logs, the list of tests the grader selects, the reward, and the pass/fail label (App. H.1).

New experiment: an environment with no tests (TextWorld). TextWorld generates text-based games: the model reads a description of its surroundings and types commands such as "open fridge" or "take knife" to reach a goal, here cooking a recipe (CookingWorld). There is no code to run and no test suite, so test feedback does not exist at any point, during the attempt or in the retrospection. We applied ROFT to Qwen3.5-4B (the paper's main model), OLMo-3-7B-Think, and Llama-3.1-8B, and scored each on the CookingWorld evaluation games of the TALES benchmark.

ModelBaseROFTGain
Qwen3.5-4B48.6%61.9%+13.3 points
OLMo-3-7B-Think14.3%21.9%+7.6 points
Llama-3.1-8B32.0%35.2%+3.2 points

ROFT improves all three models with no test feedback of any kind, so it does not depend on executable tests. The gain is largest for Qwen3.5-4B and modest for Llama.


We again thank the reviewer, and hope that these responses and the new results address their concerns. We would be glad to answer any further questions.

Reviewer_TriUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an algorithm called ROFT (which is primarily based on SFT). Their core hypothesis is that if we train the model only on the reflective texts it generates itself, can we improve its subsequent actions through weight transfer? To verify this hypothesis, the authors conducted a very simple loop: collect experience, write reflections, apply SFT to the reflections, and then perform the task again using the new weights. After going through this loop, the model successfully learned to solve problems it previously could not solve at all, and there was no significant drop in performance on the test set. Furthermore, ROFT converges faster, and the resulting model exhibits stronger generalization capabilities.

Critical strengths

Setting aside the concern of which data qwen3.5 used for pretraining, ROFT achieves something that other RL methods such as GRPO, PPO, and DPPO struggle to accomplish: it successfully solves problems where pass@64=0, while offering faster convergence and better generalization. The ablation study is also very thorough.

In theory, ROFT is completely free of the many drawbacks associated with off-policy learning, and experiments have shown that it can be trained entirely using the base model’s rollout.

Critical weaknesses

My concerns in this paper stem primarily from two points.

First, the quality of ROFT depends to a large extent on the quality of reflection, or, to put it more generally, on the model’s ability to reflect. For models with very poor reflection capabilities, or for tasks that lie completely outside the training distribution, I believe it would be difficult to achieve improvements using this method (for example, you could try the llama or olmo series, or have them perform tasks they’ve never encountered before, such as manipulating a robotic arm or constructing 3D models in Blender). If these models cannot achieve improvements, then the contribution of “being able to solve problems that were previously unsolvable” would be somewhat diminished.

Second, this method does not actually differ significantly from other RL-based methods. Although I believe this can be accounted for by the contribution of “being able to solve problems that were previously unsolvable,” as mentioned in the previous point, there are actually many doubts surrounding this contribution.

Questions for authors

I'm curious about the scale of OOD for this unsolvable task. For example, what is the upper limit on the ppl for which a task can be solved by ROFT, and beyond which it becomes very difficult to continue solving it? Also, to what extent must reflection capabilities be developed for ROFT to yield significant gains?

In theory, ROFT can operate fully off-policy, since it is an SFT-based method. However, I’m curious: if the gap between it and an on-policy model is too large, might the model end up explaining problems that will no longer occur, or repeatedly explain similar, superficial patterns, thereby causing truly critical issues to be overlooked, which could lead to the reward plateauing and possibly even overfitting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TriU (1/2): Retrospection quality; out-of-distribution tasks and weaker models

We thank the reviewer for their thoughtful review and constructive questions. In response, we ran additional experiments and analysis and address each point below:


The claim that ROFT can solve beyond frontier problems depends on how well the model can reflect. How much does reflection quality impact ROFT performance?

We thank the reviewer for this question. We agree that retrospection quality matters, and we have now measured how much. ROFT learns from the model's own retrospections, so it depends on how well the model can reflect, just as GRPO depends on the quality of the trajectories the model samples. Our new analysis below finds that tasks with better retrospections gain more from ROFT, and that the effect is modest (correlations of 0.19 to 0.29).

New analysis: does retrospection quality track per-task gain? For each of the 64 training tasks in App. G.3, a separate judge model (GPT-6 Astra) rated every retrospection used as a training target (255 in total), giving a 1–5 quality score and an accurate / inaccurate / unverifiable label. We then correlated each task's ratings with its gain from one ROFT update, defined as its pass rate after the update minus its pass rate before, over ten paired evaluation seeds.

Retrospection measure, per taskPearson r [90% CI]Spearman ρ [90% CI]
Mean quality score (1–5)0.24 [0.04, 0.41]0.19 [−0.02, 0.38]
Fraction labeled accurate0.29 [0.07, 0.47]0.23 [0.01, 0.42]

Tasks with higher-rated retrospections tended to gain more from the update. All four correlations are positive, and three of the four 90% intervals exclude zero. The effect is modest: retrospection quality explains about 6 to 8% of the variance in per-task gain (Pearson r²). Intervals are percentile bootstrap with 10,000 resamples of the 64 tasks. We have added this analysis to the revised paper (new App. G.5).

The reviewer's related question about weaker models (Llama, OLMo) and out-of-distribution tasks is addressed in our next response.

Tasks far outside the distribution the model was originally trained on may not improve with ROFT. Can ROFT handle out-of-distribution tasks? Can weaker models like Llama or OLMo where the task is more OOD also benefit from ROFT?

We thank the reviewer for raising this. Yes on both counts: in a new experiment on text-adventure games, a domain far from software engineering, ROFT improves all three model families we tested, including Llama and OLMo. Scores rise by 3.2 to 13.3 points.

New experiment: text-adventure games (TextWorld). TextWorld generates text-based games: the model reads a description of its surroundings and types commands such as "open fridge" or "take knife" to reach a goal, here cooking a recipe (CookingWorld). This is far from the paper's software-engineering tasks: there is no code and there are no tests, and success depends on exploring and planning over many steps from text alone. We applied ROFT to Qwen3.5-4B (the paper's main model), OLMo-3-7B-Think, and Llama-3.1-8B, and scored each on the CookingWorld evaluation games of the TALES benchmark.

ModelBaseROFTGain
Qwen3.5-4B48.6%61.9%+13.3 points
OLMo-3-7B-Think14.3%21.9%+7.6 points
Llama-3.1-8B32.0%35.2%+3.2 points

ROFT improves every model. OLMo and Llama both start below Qwen3.5-4B on these games, so the task is harder for them, and both still improve. The gain is largest for Qwen3.5-4B and modest for Llama.

Difficult tasks in the submitted paper. The beyond-frontier experiments in the paper (Sec. 3.1, Fig. 3) point the same way within software engineering: ROFT is trained on individual tasks that the base model failed in all 64 sampled attempts, and it goes on to produce successful solutions.

We thank the reviewer for suggesting these experiments on out-of-distribution tasks and other model families; they have strengthened our claims.

AuthorsOpenReview ↗

Response to Reviewer TriU (2/2): Difference from RL-based methods; off-policy retrospections

ROFT does not differ much from RL-based methods.

We thank the reviewer for raising this point. ROFT differs from RL-based methods in what is trained, not only in how it is optimized: there is no reward and no loss on actions, and the model is fine-tuned only on its own written retrospection of each attempt. This leads to three practical differences from GRPO, each measured in the paper:

  1. No verifier is needed. ROFT's objective contains no reward, and it performs the same when the pass/fail verdict is withheld from the retrospection (Fig. 17).
  2. It learns where RL has no signal. When every attempt on a task fails, GRPO's within-group advantage is zero. ROFT starts from 0 of 64 successful attempts on a task and goes on to solve it (Sec. 3.1).
  3. It is competitive in less than half the training time, on problems selected to favor GRPO. We kept only training problems that the base model both passes and fails, so GRPO always has a reward signal (Sec. 3.2). ROFT's update-20 checkpoint, reached in 3.05 hours, is on par with GRPO's update-40 checkpoint, reached in 8.30 hours.

The table below gives the numbers behind these points and two further differences.

GRPOROFT
Training targetActions, weighted by reward advantageThe retrospection only; no loss on actions
Needs a verifierYesNo: 49.2% on SWE-bench Verified with and without the verdict (Fig. 17)
Tasks where all 64 base-model attempts failNo learning signal (zero advantage)Solve rate reaches 1.75% and 3.29% on two such tasks after 40 updates (Fig. 3)
Held-out accuracy, SWE-bench Verified / Pro48.0% / 25.3% at update 4049.2% / 29.0% at update 20 (Fig. 5)
Training cost for 40 updates8.30 h, 11,576 solution attempts4.32 h, 6,035 solution attempts (Fig. 4)
Credit assignment: share of correct turns made more likely, minus share of incorrect turns−1.2 points+11.2 points (Fig. 6a)

The last row is the percentage of correct turns whose likelihood rises after ten updates, minus the same percentage for incorrect turns: ROFT favors correct turns, while GRPO shows no separation. The two methods are also complementary: adding a retrospection step to each GRPO rollout gives 50.4% on SWE-bench Verified, above GRPO (47.8%) and ROFT (49.2%) alone at the same update (Fig. 6c). Table 1 compares ROFT with other prior methods by training target and required supervision.

If retrospections come from a policy far from the acting policy, the model may explain problems that no longer occur or repeat superficial patterns, leading to a plateau or overfitting. Are there any experiments on this?

We thank the reviewer for raising this. Yes, we ran this experiment (Sec. 3.4, Fig. 8a), and it supports the reviewer's intuition: ROFT works best when the retrospections come from the acting policy itself (49.2% on SWE-bench Verified, against 47.4% and 48.0% when they come from the frozen base model).

Fig. 8a trains the same base model (Qwen3.5-4B) under three regimes, which differ in which policy produces the attempts and which produces the retrospections:

Training regimeAttempts fromRetrospections fromSWE-bench Verified
Online (ROFT)Current learnerCurrent learner49.2% (246/500)
Off-policy reflectionsCurrent learnerFrozen base model47.4% (237/500)
OfflineFrozen base modelFrozen base model48.0% (240/500)

The two frozen arms separate the two failure modes the reviewer describes. In the off-policy-reflections arm, the frozen base model still reads the learner's current attempt, so it cannot explain problems that no longer occur. In the offline arm it can: the corpus is fixed before training, so its retrospections describe base-model attempts regardless of how far the learner has moved. Both arms score below online training, by 1.8 and 1.2 points. The offline arm scores slightly higher than the off-policy-reflections arm, which is consistent with a benefit from keeping the attempt and retrospection generators aligned, rather than refreshing attempts alone.

We also compared retrospections from the three arms qualitatively. Those written by the frozen base model rather than the current learner show signs of the second pattern the reviewer describes: they are more repetitive and tend to repeat general, superficial observations. The three regimes are described in full in App. H.2.


We again thank the reviewer, and hope that these responses and the new results address their concerns. We would be glad to answer any further questions.

Thank you for your response. I appreciate you sharing this very compelling approach; unfortunately, I am unable to try it out myself at the moment, so I cannot give it a higher rating right now. However, if I find it effective after testing it, I will certainly recommend it to others.

Thank you for maintaining your positive assessment of the paper and for your thoughtful engagement with our work, we are glad you found it compelling. We appreciate your interest in trying ROFT and would be glad to hear your feedback whenever you have the opportunity to explore it!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the review discussion

We thank the Area Chair for handling our submission and the reviewers for their thoughtful reviews. We summarize the discussion below for convenience.


Where the reviewers stand. All three reviewers now have positive assessments of the paper.

ReviewerRatingAfter the rebuttal
FYUd2 → 3Raised their score. The new experiments and clarifications "address most of my concerns. Therefore, I will raise my score."
TriU3Kept their positive rating and called the approach "very compelling." They raised no further concerns, and noted they could not go higher without trying the method themselves.
bVB63No reply yet at the time of writing. We have addressed each of their questions and concerns, most with new results.

Strengths identified by the reviewers.

  • It learns where RL has no signal. ROFT "achieves something that other RL methods such as GRPO, PPO, and DPPO struggle to accomplish: it successfully solves problems where pass@64=0, while offering faster convergence and better generalization" (TriU).
  • The method is simple and isolates one question. It "isolates transfer through weights without direct action-target supervision" (bVB6). "The procedure is simple," and "learning from unsuccessful attempts without requiring successful action targets is a useful direction" (FYUd).
  • The evidence is thorough. "The ablation study is also very thorough" (TriU). "Two test sets, a 9B extension, and a no-final-verdict ablation support the central claim" (bVB6).
  • The behavioral findings are interesting, "particularly the effect of retrospection instructions on subsequent rollout length and the combination with GRPO" (FYUd).

Concerns raised and how we addressed them. Scores are solve rates on SWE-bench Verified unless stated otherwise.

Concern (reviewer)What we didResult
Does ROFT depend on how well the model can reflect? (TriU, bVB6)New analysis: an independent judge model rated all 255 training retrospections across 64 tasks.Tasks with better retrospections gain more from ROFT. The effect is modest (correlations of 0.19 to 0.29).
Does ROFT work outside software engineering, on weaker models, and without test feedback? (TriU, bVB6)New experiment: text-adventure games (TextWorld), which have no code and no tests, with three model families.ROFT improves all three models: +13.3 points (Qwen3.5-4B), +7.6 (OLMo-3-7B-Think), and +3.2 (Llama-3.1-8B).
A baseline is missing: SFT on successful trajectories. (FYUd)New experiment: success-filtered SFT, alone and combined with ROFT, in ROFT's training setup.SFT alone degrades from 44.0% (update 10) to 40.8% (update 20). With ROFT's retrospection loss added, it keeps improving, from 45.2% to 47.6%.
Is the gain specific to retrospections, or would other self-generated text do? (FYUd)New experiments: SFT on trajectory summaries, and an attention-weighting test of the proposed mechanism. The paper also compares three related methods.Trajectory summaries score 0.9 points below ROFT after one update (on the 64 training tasks), and attention weighting adds 4.1 points. The three related methods reach 45.6% to 46.8%, against ROFT's 49.2%.
Do the early gains last, and how were checkpoints selected? (bVB6, FYUd)Reported the held-out score at update 40 and the selection rule, which is the same for both methods. New variant that rotates retrospection prompts.Standard ROFT peaks at 49.2% (update 20) and declines to 41.6% (update 40); GRPO shows the same shape later. The rotating-prompt variant still scores 48.2% at update 70. ROFT at updates 10 and 20 is above GRPO's best checkpoint (48.0%).
No uncertainty across training seeds. (bVB6)Nine evaluation runs across three training seeds.Mean 48.9%, standard deviation 0.5 points (range 48.0% to 49.4%).
ROFT does not differ much from RL-based methods. (TriU)Clarified with a side-by-side comparison with GRPO, using results from the paper.ROFT uses no reward and no loss on actions. It needs no verifier, learns on tasks where all 64 base-model attempts fail, and is on par with GRPO in less than half the training time.
Could off-policy retrospections cause a plateau? (TriU)Pointed to the existing experiment (Sec. 3.4, Fig. 8a).It supports the reviewer's intuition: on-policy ROFT scores highest (49.2%, against 47.4% and 48.0% with retrospections from the frozen base model).

We thank the reviewers, whose suggestions strengthened the paper, and the Area Chair for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.