acceptodds
Under review as a conference paper at ICLR 2027

RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

Abstract

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task → insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14–31% with insights in context during training and, crucially, 11–14% when no insight is in context at eval time. We identify that the key is the task → insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form ”on this sort of task, keep this sort of thing in mind”, which we hope to inspire future research on.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Self-improvement, Feedback, Hint, Agentic

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ufXgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

RLVR will fail when task is difficult that the agent has a low or even no chance of success. This paper enables effective RLVR by continuously generating feedback during iterative self-improvement and incorporating it into subsequent training to enhance the agent's trial success rate. Experimental results illustrate that training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts.

Critical strengths
  1. This paper enables effective RLVR by continuously generating feedback during iterative self-improvement and incorporating it into subsequent training to enhance the agent's trial success rate.
Critical weaknesses
  1. The paper only used Qwen 3.5 9B for experiments, and the reliability of the experimental conclusions is insufficient
  2. The paper only used three datasets for the experiment and the Synthetic API is a pure synthetic environment. Moreover, the method proposed in this article has significantly poor performance on Leetcode as shown in Table 1. The effectiveness of our method is questionable.
  3. The paper does not have open source code. This makes me doubt this article.
Questions for authors
  1. Can you provide more model experimental results besides Qwen3.5-9b?
  2. Can you increase the experimental dataset to observe whether this method is truly effective in real-world scenarios?
2 comments

We would like to thank the reviewer for their time. We’re happy to provide clarifications below.

W1: The paper only used Qwen 3.5 9B for experiments, and the reliability of the experimental conclusions is insufficient.

Thanks for the suggestion! We have now replicated our experiments on the Granite 4.2 8B model. We observe the same effect: GRPO fails to learn, whereas our insight internalization breaks through this learning barrier. This worked out of the box, without any editing to the insight generation or internalization logic or hyperparameters. This is unsurprising because our method is simple and generic, agnostic of the model. The numbers below are Pass@1 on rollouts without insights in context on the SAPI frontier-difficult split, just like in the main paper.

Train progress (env interactions)50k100k150k200k250k300k
GRPO0.010.010.010.010.020.04
RLTLDR0.060.080.110.090.100.12

W2: The paper only used three datasets for the experiment and the Synthetic API is a pure synthetic environment. Moreover, the method proposed in this article has significantly poor performance on Leetcode as shown in Table 1. The effectiveness of our method is questionable.

Synthetic API is not "a pure synthetic environment" in the sense of some toy task. It is the largest collection of realistic on-device tasks to our knowledge, and training on it gives SOTA performance on other benchmarks, (e.g., the human-written Appworld benchmark with real-world on-device tasks) [1].

We do not see which results would permit to conclude that the method is ineffective. It learns on all three datasets on tasks where the agent has Pass@128=0 and GRPO flatlines.

The only result that is negative is the generalization in coding tasks, in which all baselines we tested do not generalize to the test set either. We believe it's important to sketch limitations, and we communicated this openly in the main text, conclusion, and discussion in the originally submitted manuscript. We can highlight it a bit more in the revised version if you deem it important.

W3: The paper does not have open source code. This makes me doubt this article.

We unfortunately cannot provide full source code for legal reasons. But we do provide a full step-by-step reproduction guide in Appendix B, disclose all training hyperparameters and prompts in Appendix A, and provide a guide on how to evaluate correctly by deconfounding evaluation in Appendix D (which we hope is going to be helpful even outside our own experiments). We invite to also have a look at the experiments requested by the other reviewers. All of them confirmed the effects we have reported in the paper.

After the ICLR deadline, two other papers have independently confirmed our paper's claims. [2, 3] So if your argument is that you doubt the article because there is no code (which in the era of AI slop papers might be a good prior), we think it's apparent our article shouldn't fall into this category.

Q1: Can you provide more model experimental results besides Qwen3.5-9b?

See our response to W1.

Q2: Can you increase the experimental dataset to observe whether this method is truly effective in real-world scenarios?

We provide results on three datasets which all have the same conclusion. All three are "real world scenarios". We'd be happy to cater for you if you provided more details on what experiment you want to see.

Let us know what concrete evidence you would like to see to move from a clear rejection score.


[1] Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis, Szot et al., 2026, https://arxiv.org/abs/2610.37648

[2] Shockingly Simple Self-retrospection Improves Agentic Models Without RL, Jonathan Light et al., 2026, https://openreview.net/forum?id=fcQaMnBdzB

[3] Teaching NVIDIA's Nemotron 3.5 Super-VL ML Engineering, Bespoke Labs, 2026, https://bespokelabs.ai/blog/teaching-nvidias-nemotron-ml-engineering

Reviewer_ufXgOpenReview ↗

Confirm

I have reviewed the rebuttal and maintained my score

Reviewer_AG3QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RLTL;DR generates short insights from failed attempts and verifier feedback, uses them to guide subsequent attempts, and trains the model to predict these insights from the task alone. The paper also shows that training only on task–insight pairs can improve performance without insights at inference.

Critical strengths
  • The experiments are well structured, with clear questions and relevant ablations.

  • The paper builds a clear, coherent argument and carefully distinguishes gains with insights from improvements without them.

  • The evaluation covers several aspects of the method and yields interesting findings, particularly on learning from short insights alone.

Critical weaknesses
  • My main concern is novelty. Feedback-conditioned self-distillation and learning directly from critiques have already been explored in SDPO [1], RLTF [2], and CFT [3]. Predicting short insights directly from the task is a concrete difference, but I am not convinced the paper establishes enough new understanding or a consistent advantage over these approaches. Full-trajectory SFT also performs better in Table 2.

  • The method relies heavily on informative verifier feedback. Removing failed-test information substantially reduces performance. More analysis is needed to distinguish transferable lessons from task-specific hints or answer-related information extracted from the verifier.

  • Sequential attempts use different, dependent contexts but share a GRPO reward baseline. Appendix F.2 provides a useful split-group ablation, but the optimization objective and possible bias remain insufficiently explained.

  • An insight generated for a particular failed attempt may not remain appropriate for other solutions or a stronger policy. Online regeneration partly addresses this, but changes in wording do not establish continued usefulness. The mixed generalization results leave this concern open.

[1] Jonas Hübotter et al. Reinforcement Learning via Self-Distillation. 2026.

[2] Yuda Song et al. Expanding the Capabilities of Reinforcement Learning via Text Feedback. ICML, 2026.

[3] Yubo Wang et al. Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate. 2025.

Questions for authors
  • What new finding beyond existing feedback modeling and distillation does the task-to-insight objective establish under matched feedback and compute?

  • How much task-specific information do the insights retain, and do they remain useful as the policy improves?

  • What objective does the shared advantage estimator optimize when attempts have sequentially dependent contexts?

2 comments

We would like to thank the reviewer for their time. We’ll slightly reorder the rebuttal to start with the requested new experiments, and then go into clarifications.

Q3: Sequential attempts use different, dependent contexts but share a GRPO reward baseline. Appendix F.2 provides a useful split-group ablation, but the optimization objective and possible bias remain insufficiently explained.

We don't find this too grave for two reasons. First, we favor simplicity and performance. Since our method works as good with mixed-context groups as when taking them apart (see Appendix F.2), we favor not complicating the GRPO objective, see also our response to Q3.

Second, for us GRPO is just one of many methods that we apply our method to. Insight internalization works when adding it to GRPO (with mixed groups as in the main paper or split groups as in Table 9), when adding it to Self-Distillation (Appendix E.9), and even on its own (SFTL;DR). We have now added another ablation where we add our method to a standard PPO with +1/-1 rewards, and with our insight internalization it still learns (though not without). See the table below, which we will include in the revised version, it reports Pass@1 without hints in context on SAPI (so same as in the main paper). Some more notes on GRPO theory are in the question response below.

We comment on the optimization objective in our response to Q3.

Train progress (env interactions)50k100k150k200k250k300k
GRPO0.000.000.000.000.000.00
GRPO + Ours0.010.030.070.110.120.13
Self-Distillation0.020.040.070.070.080.06
Self-Distillation + Ours0.010.050.100.190.170.21
+1/-1 rewards0.000.000.00still runningstill runningstill running
+1/-1 rewards + Ours0.010.060.110.110.130.15

W4: An insight generated for a particular failed attempt may not remain appropriate for other solutions or a stronger policy. Online regeneration partly addresses this, but changes in wording do not establish continued usefulness. The mixed generalization results leave this concern open.

We have added an ablation. Similar to the existing plots in the appendix, we track the Pass@4 if the first attempt is done without insight in context, the second with one insight from the previous failed attempt, the 3rd with 2 and the 4th with 3. This measures how useful insights are. In the new ablation, we now also do this for insights from random other tasks of the same app, and from random insights from any task. The table below shows a clear ordering that insights are better the closer to the current task they are (as one would expect), but even insights from random other tasks improve the success rate 3x over not giving any insights. This confirms that insights generalize. We will run this experiment up to Pass@16 and include it in the revised version.

We describe a second way to look at insight generalization in Appendix D.2. It checks whether the internalized hints generalize. By the fact that the train curve (on rollouts without insights) rises in within the first epoch, when every task is new, we can follow that the internalized hints from old tasks indeed generalize to new tasks.

Insights in contextPass@4
no insights2.6%
insight from random task7.8%
insight from same app18.0%
insight from same task32.5%

W1.1: My main concern is novelty.

There are three main novelties. 1) We outperform the prior works (both standalone and when adding our insight internalization to, e.g., SDPO, see Figure 13), 2) we simplify it. Because we train task → insight (which already exists autoregressively in the task → insight → rollout contexts) rather than (task, rollout, judge prompt) → insight, we don't need to manage two different rollout buffers, but instead can do it all in the same rollout, with just a flip of the backprop mask. And, maybe most importantly, 3) we show that LLMs can internalize P(insight | task) directly and apply this knowledge in code when producing P(rollout | task). Previous work just showed that backpropagating P(insight | task, rollout, in a judge prompt) makes the model better on this very task. This opens quite some new fields: We can learn from any form of compressed knowledge, whether grounded in a rollout or not, such that future works might consider internalizing skills or human feedback. We also show that one can learn fully without the rollout, which is important for privacy-preserving learning. We discuss these findings in the conclusion in the submitted paper and now, with the added page, bring a dedicated discussion section into the main paper, discussing both the previous works and the trajectory we see form it and ours.

(continuing in the next comment)

W1.1: Full trajectory SFT also performs better in Table 2.

There is a catch in this experiment, which is that the data we SFT on is the data collected from our method's RL run (see the caption of Table 2). Without our method, the Pass@128 is zero and data collection, for example by GRPO, flatlines (see Figure 2 and Table 2, row "RLTL;DR, lambda=0"). SFT on its own would score 0%.

The method relies heavily on informative verifier feedback. Removing failed-test information substantially reduces performance. More analysis is needed to distinguish transferable lessons from task-specific hints or answer-related information extracted from the verifier.

This is correct, but this follows standard practice in related works. We’d argue that returning some feedback rather than just a binary flag mirrors real world usage (a terminal printout, debugging logs, a web API giving an error code, or even just a user that is upset). Note also that we hide any right-side values of assert statements in the verifier (i.e., the insight generator is told that money was sent to the wrong person, but who is correct person is is masked out). For the transferability analysis, see our responses to W4 and Q2.

Q1: What new finding beyond existing feedback modeling and distillation does the task-to-insight objective establish under matched feedback and compute?

See the response to W1, 1) we outperform those methods, 2) we simplify them and save a buffer, 3) our findings open new applications, and enhance the understanding of what LLMs can learn from. Note also that all our experiments against the reimplemented baselines are already under matched feedback and compute.

Q2: How much task-specific information do the insights retain, and do they remain useful as the policy improves?

See the response to W4 for how much insights generalize across tasks. As for whether insights remain useful, we track that via the insight advantage metric described in Section D.2, i.e., the difference in Pass@1 of a rollout with insights minus without insights. Figure 3 in the appendix shows that the insight advantage shrinks throughout training from 0.26 to 0.19 as the model internalizes more and more insights, however it stays far from 0. We have seen this consistently throughout other runs: The gap between the line that tracks overall Pass@1 in all rollouts compared to the one with only non-insight conditioned rollouts shrinks, but stays positive. We will include plots of this for all three datasets in the revised version.

Q3: What objective does the shared advantage estimator optimize when attempts have sequentially dependent contexts?

In short, the expected return, irrespective of whether insights are in context or not. It is true that we build our GRPO groups across both the original task as input and the adapted task (i.e., with insights added after the task description). We found in Table 9 that splitting the reward groups does not increase performance (if anything slightly decreases it). Purely mathematically speaking, GRPO (and how it builds its groups) is just one estimator for expected return, and neither its mean centering, nor its definition of a group are unchangable. One can in theory substract any mean that helps reduce variance (or not substract anything, as in the +1/-1 rewards shared above). In our work, you can see the substraction of the average reward per task (albeit with different amounts of insights in context) as a simple normalization by task difficulty. This, in turn, leads to balanced positive and negative advantages, to prevent entropy implosion and explosion.

Some previous works have aimed to avoid this, either by chunking the task rollouts into subgroups with the same fixed insights in context, or by revisiting the task a second time with all insights from the first attempt in context. Our insight internalization is fully compatible with either. However, this adds complexity to the task scheduling (while we strive for simplicity) and did not add any performance benefits (in fact, it increases the time until first solve). Compared to that, our method strives for simplicity and performance.

Reviewer_MR9VOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets RLVR on tasks where the policy almost never succeeds, so group-relative advantages are zero. In RLTL;DR, after each failure the policy reads the verifier output and writes a one-sentence insight, and later attempts on the same task see these insights in context. It adds an SFT loss on the insight tokens conditioned on the task (L = L_GRPO + λ·L_SFT). On Pass@128=0 subsets of a synthetic API dataset (SAPI), AppWorld and LeetCode with Qwen3.5-9B, GRPO stays at 0–1% while RLTL;DR reaches 11–14% no-insight Pass@1 (Fig. 2) and improves held-out splits (Table 1). "SFTL;DR", training only on (task, insight) pairs, is reported to be close to SFT on full rollouts (Table 2).

Critical strengths
  • The method is simple: sequential retries with self-written insights, plus an SFT loss on the insight tokens that only changes the loss mask.
  • The SFTL;DR result is interesting: training on about 68k backward tokens of (task, insight) pairs raises eval Pass@1 from 4% to 16.9% (Table 2).
Critical weaknesses
  1. The gap to RLTF (Song et al., ICML 2026) is small. RLTF already studies the same problem with the same two approaches. RLTF-SD distills feedback-conditioned attempts into the first-turn policy, and RLTF-FM adds a loss on the feedback tokens to the RL objective, in the same form as L_GRPO + λ·L_SFT. What is new here is mainly that the loss is conditioned on the task π(f|g), rather than on the attempt π(c|x,y), and that the policy writes its own feedback. The conditioning difference is tested only in Table 8 (one run, training tasks, first 80 updates), and RLTF-FM is absent from the main comparison.

  2. The method does not beat offline SFT. In Table 2, SFT on just 100 successful rollouts (20.8%) beats RLTL;DR (18.9%), which beats SFTL;DR (16.9%), and those rollouts come from the RLTL;DR run itself. Sequential insight GRPO without the insight loss (λ=0) already reaches 13.2%. So the claims that internalization is "the key" (l. 024) and that SFTL;DR "recovers almost the full performance" (l. 026–028) are not supported.

  3. The headline numbers are mostly training-set numbers, and the benefit is shown only when training on filtered subsets. The "11–14% Pass@1 at eval time" (l. 022–024) comes from Fig. 2, which is on training tasks. AppWorld (34 tasks) and LeetCode (123 tasks) are revisited many times during training which leads to potential overfitting. On held-out LeetCode, RLTL;DR is below GRPO (49.1% vs. 55.1%, Table 1). All main results train only on Pass@128=0 subsets, which is not how one would train in practice. When training on the full data, RLTL;DR only matches GRPO (l. 311, App. E.7), and only overall Pass@1 is reported. So it remains unclear whether the advantage on hard tasks survives under standard full data training.

Minor: Table 1 has no base-model row. Figs. 12 and 14 look like duplicates.

Questions for authors
  1. How does L_SFT differ from RLTF-FM beyond conditioning on the task instead of the attempt? Can you report RLTF-FM in the main comparison, and the (g, τ) vs. g conditioning ablation on held-out tasks?
  2. Can you compare against SFT then RL (SFT on successes or on (task, insight) pairs collected once with the frozen base model, followed by GRPO, at matched total compute)? A compute-matched GRPO would also help.
  3. Table 5 suggests the most frequent insights are environment conventions (e.g., enum capitalization, 15.3%). What drives the 57.8%→91.1% SAPI held-out gain? Does SFT on only these few convention insights reproduce it?
  4. When training on the full datasets (App. E.7), what is Pass@1 on the held-out tasks that the base model cannot solve (Pass@128=0)? Does RLTL;DR still beat GRPO on this set?
6 comments

We would like to thank the reviewer for their time. We’ll slightly reorder the rebuttal to start with the requested new experiments, and then go into clarifications.

Q4: When training on the full datasets (App. E.7), what is Pass@1 on the held-out tasks that the base model cannot solve (Pass@128=0)? Does RLTL;DR still beat GRPO on this set?

Thanks for the suggestion! We’ve now evaluated the checkpoint that was GRPO trained (without ours) on the full dataset, at train iteration 300k. We then tested it on the Pass@128=0 tasks. Its Pass@1 is 0.67%. The RLTL;DR trained on the full dataset (same iteration and trained on the same tasks, because we fixed task order in App E.7) is at 4.7%. The RLTL;DR run that only trained on Pass@128=0 tasks is even at 13.4% at train iteration 300k (though that last checkpoint is of course a bit unfair because it has been trained on those tasks. So we also evaluated the RLTLDR checkpoint trained on the Pass@128 tasks but at 150k iterations, and evaled on the remaining unseen Pass@128=0 tasks, where it scores 11.2%). We believe this strengthens our conclusion that RLTLDR indeed teaches new skills that would otherwise not be picked up by GRPO, even when allowed to learn from related, simpler tasks.

SetupPass@1
GRPO trained on full dataset0.67%
RLTLDR trained on full dataset4.7%
RLTLDR trained on frontier-hard tasks13.4%
RLTLDR trained on frontier hard, excl. a heldout set11.2%

Q2: Can you compare against SFT then RL (SFT on successes or on (task, insight) pairs collected once with the frozen base model, followed by GRPO, at matched total compute)? A compute-matched GRPO would also help.

There are no successes in the base model on these frontier-hard tasks within 128 attempts, hence SFT then RL would flatline and score at 0% (the same as the GRPO baseline we report throughout the paper). All comparisons are already token-matched, and even at double the compute GRPO still flatlines.

W1: The gap to RLTF

We outperform RLTF-FM by such a large margin that we decided to only compare against the RLTF-SD method that the same paper proposed in our main text and show RLTF-FM in Appendix F.1. We have now tried RLTF-FM on the other two datasets as well, and do not see any Pass@1 > 1% (at eval time without hints, on the frontier hard task, same setup as in the main paper). We will keep those runs running and inform you if they take off throughout the rebuttal period.

Besides that, we'd argue the seemingly small difference to RLTF-FM is rather big in practice. First, backpropagating P(insight|task) simplifies both the approach and the implementation. Because we train task → insight (which already exists autoregressively in the task → insight → rollout contexts) rather than (task, rollout, judge prompt) → insight, we don't need to manage two different rollout buffers, but instead can do it all in the same rollout, with just a flip of the backprop mask. Second, and maybe most importantly, we show that LLMs can internalize P(insight | task) directly and apply this knowledge when producing P(rollout | task) (compared to backpropagating P(insight | task, rollout, in a judge prompt), where the knowledge is learned in the same way it is produced). This jump opens quite some new fields: We can learn from any form of compressed knowledge, whether grounded in a rollout or not, such that future works might consider internalizing skills or human feedback. We also show that one can learn fully without the rollout, which is important for privacy-preserving learning. We discuss these findings in the conclusion in the submitted paper and now, with the added page, bring a dedicated discussion section into the main paper, discussing both the previous works and the trajectory we see form it and ours.

W2: The method does not beat offline SFT

There is a catch in this experiment, which is that the data we SFT on is the data collected from our method's RL run (see the caption of Table 2). Without our method, the Pass@128 is zero and data collection, for example by GRPO, flatlines (see Figure 2 and Table 2, row "RLTL;DR, lambda=0", the latter already having the benefit of sampling with hints enabled but still not performing well). SFT would thus score 0%.

W3: The headline numbers are mostly training-set numbers, and the benefit is shown only when training on filtered subsets.

This is intended. We want to study how to get a training signal on tasks that are so hard GRPO does not find solutions and flatlines. That's a problem frontier models face, and we rebuild it via the Pass@128=0 split. Studying such hard problems, without a supporting dataset of easier ones, is common in related works [1, 2]. See also the response to Q4 that shows that the advantage on hard tasks survives under standard full data training.

We openly state in the main experiment text (and in the caption of the headline result in Figure 2) that AppWorld and LeetCode are potentially overfitting, hence the large-scale SAPI experiments (which is one of the few datasets to our knowledge that has enough Pass@128=0 tasks to train on them).

Minor Weaknesses

Thanks, fixing this in the revised version!

Q1: How does L_SFT differ from RLTF-FM beyond conditioning on the task instead of the attempt? Can you report RLTF-FM in the main comparison, and the (g, τ) vs. g conditioning ablation on held-out tasks?

See the response to W1. The results reported there are already on heldout tasks.

Q3: Table 5 suggests the most frequent insights are environment conventions (e.g., enum capitalization, 15.3%). What drives the 57.8%→91.1% SAPI held-out gain? Does SFT on only these few convention insights reproduce it?

Table 5 also reports that the remaining 84.7% of hints are about a variety of things, such as saving progresses, improving search behavior, double-checking shopping carts, etc. Capitalization is the largest cluster we could make out as one cluster, but far from the only insight the agent internalizes. For the SFT on the insights, we refer to Table 2. We have also done an ablation for reviewer AG3Q’s W4 that confirms that insights generalize to other tasks and apps. Let us know if you have some other experiment in mind there.


[1] Shockingly Simple Self-retrospection Improves Agentic Models Without RL, Jonathan Light et al., 2026, https://openreview.net/forum?id=fcQaMnBdzB

[2] Scaling Discovery through Test-Time Communication, Jongho Park et al., 2026, https://arxiv.org/abs/2609.21032

Thank you for the response. The new Q4 result is convincing. My concern regarding Q2 is still not resolved. I was wondering whether the insights can be collected from the frozen base model. Insights are generated from failed attempts, so they do not require any successes, and Fig. 11 shows that the frozen model with sequential insights already solves ~57% of the hard tasks. I'm interested to see how SFT on such data compares with RLTL;DR.

Comments on the paper

1 comment

Summary at the end of the rebuttal

As a quick summary of the rebuttal:

  • We ran all requested experiments. Our method worked out-of-the-box on any setting (ours on top of a different model, ours on top of Self-Distillation and +1/-1 rewards instead of GRPO)
  • We ran the requested analyses, all showing that our method works as intended (improvements on datasets that also have easy examples, generalization of insights to other tasks)
  • We clarified some misunderstandings about the SFT experiments (which is ran on data collected by ours, normal SFT would reach 0%)

As it stands, we see two remaining critiques:

  • Reviewers MR9V and AG3Q criticize the novelty over RLTF-FM, which also internalizes feedback, using an auxiliary task that predicts (task, rollout, in a judge prompt) -> feedback. Our argument is that we internalize feedback via predicting task -> feedback, which is already part of the task -> feedback -> 2nd attempt rollouts. This makes the training architecture much simpler (just a flip in the backprop mask of existing rollouts rather than requiring a second rollout buffer), is a first proof that models can generalize better than previously assumed, and also we just outperform RLTF-FM in all datasets by a large margin. Reviewer MR9V raised their score, reviewer AG3Q did not respond.
  • Reviewer ufXg critiques that we do not publish code. We indeed cannot publish the code for legal reasons, but describe in 3 appendices how to reproduce ours step-by-step. Two groups [1, 2] reproduce the effect, one concurrently, one citing ours. We believe this shows the effect is robust, reproducible, and also just that there is interest in the field, going back to the novelty argument. Reviewer ufXg responded in a one-liner without changing score.

We hope this is useful for the AC in their decision, or any reader that might stumble upon this.


[1] Shockingly Simple Self-retrospection Improves Agentic Models Without RL, Jonathan Light et al., 2026, https://openreview.net/forum?id=fcQaMnBdzB

[2] Teaching NVIDIA's Nemotron 3.5 Super-VL ML Engineering, Bespoke Labs, 2026, https://bespokelabs.ai/blog/teaching-nvidias-nemotron-ml-engineering

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.