acceptodds
Under review as a conference paper at ICLR 2027

Retrospection Makes a Better Learner: Retrospective On-Policy Self-Distillation for Reasoning Models

Abstract

On-policy post-training methods for reasoning models share a common learning loop where the model attempts a problem, observes corrective information such as a verifier's reward or a reference solution, and updates itself where the attempt and the observation diverge. These methods differ in what is observed and how the update is made, but all produce the attempt the same way, where the model simply responds to the question with no earlier attempt to build on. A human, by contrast, re-examines a first attempt and tries again before consulting a solution, and learns more from the solution as a result. Bringing this behavior to a model means training the attempt stage itself, under two requirements set by its position in the loop: 1) no external information, such as a ground-truth answer or reference solution, since attempting precedes observing, and 2) a single generation pass, since we need a policy that attempts in one pass, in the learning that follows and at inference. We introduce Retrospective On-Policy Self-Distillation (RSD), which meets both requirements with retrospection. The model conditioned on its own prior attempt serves as the teacher, and its behavior is distilled into a single generation pass. We show that retrospection preserves single-attempt accuracy while expanding pass@ coverage. Our central finding is that an RSD-trained policy is a better learner, so that on-policy learning schemes initialized from it achieve consistently higher performance across benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training, On-Policy Self-Distillation, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HsquOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Retrospective On-Policy Self-Distillation (RSD). A teacher conditioned on the model's own previous attempt supervises a student that sees only the question. Before distillation, a brief role-reversed warmup is used to address the teacher's shortened reasoning. RSD uses no answers or rewards during these stages and aims to improve the model's initialization for later training. Experiments on Qwen3-4B and Qwen3-8B report increased pass@k coverage and improved downstream GRPO and RLRT results on mathematical reasoning benchmarks.

Critical strengths
  1. The method improves downstream training in several settings without using answers or rewards during RSD. The average pass@1 gain reaches 9.5 percentage points for Qwen3-8B with RLRT. A FLOP-matched GRPO comparison also supports some benefit beyond additional training compute.

  2. The paper is well organized and easy to read. The figures clearly explain the training procedure and reported results.

Critical weaknesses
  1. The novelty and mechanism are not sufficiently established. RSD adapts existing on-policy self-distillation through prior-attempt conditioning and teacher warmup. Stronger comparisons with related methods and simpler prompting alternatives are needed. Shorter reasoning and lower performance occur together, but this does not establish causality. Likewise, higher pass@512 does not directly explain the benefit to GRPO, which samples eight attempts per prompt.

  2. The evidence for broad and stable benefits is limited. Experiments use only Qwen3-4B and Qwen3-8B, mathematical reasoning tasks, and one training seed. Gains vary across settings: 4B with GRPO improves by only 1.6 percentage points, while larger gains occur elsewhere. External-teacher distillation shows almost no benefit. These results leave the method's broader significance and reliability uncertain.

Questions for authors
  1. What advantage does role-reversed warmup offer over simply prompting the teacher to produce complete reasoning? How sensitive are the results to stopping warmup at step 20?

  2. Can length-controlled or shuffled-prior-attempt comparisons support the proposed mechanism? Does RSD increase training-set pass@8 and the fraction of rollout groups with both correct and incorrect answers?

  3. Do the gains persist across training seeds or other model families? Such evidence would help assess whether the benefits are reliable beyond the current settings.

  4. Section 6.3 and Table 3 use the 8B baseline at step 180, while Appendix B.4 states step 200. Which checkpoint produced the reported FLOP-matched results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Hsqu (Part 1/4)

We thank the reviewer for the constructive feedback. We respond to each question in order below, and the revised parts of the manuscript are marked in orange.

Q1. What advantage does role-reversed warmup offer over simply prompting the teacher to produce complete reasoning? How sensitive are the results to stopping warmup at step 20?

(1) Prompting versus the role-reversed warmup

Following the reviewer's suggestion, we tested prompting on Qwen3-4B. The teacher without warmup receives one of the following instructions in the system turn of its prompt, with one prior attempt in its context as before:

  • Reason completely: "The assistant writes out its own complete reasoning from start to finish, step by step, as fully as it would if no samples had been shared; it does not shorten its reasoning or skip steps because the samples already contain them."
  • Reason completely and verify: The sentence above, followed by "The samples may contain errors, so the assistant works through every step itself and does not adopt a sample's final answer unless its own reasoning confirms it."
Teacher (one prior attempt in context)Reasoning length (% of base)
No warmup, original prompt36.6
No warmup, instructed to reason completely45.4
No warmup, instructed to reason completely and to verify47.1
Warmup, step 20109.8

(We have added these results in Appendix D.9, Table 13, with a pointer from Section 4.1.)

Both instructions lengthen the reasoning only slightly, and the teacher stays below half the length of the base policy, whereas the warmup restores the full length within 20 steps. This confirms that the length collapse is not removed by an instruction, which is the advantage of the role-reversed warmup. The warmup also requires no labels, since its target is the model's own unconditioned policy.

(2) Sensitivity to the stopping step

To assess the sensitivity, we evaluated the teacher at successive warmup checkpoints (Qwen3-4B, AIME 2024–2026).

Warmup stepReasoning length (% of base)pass@256 (%)
1571.371.5
20 (selected)109.871.3
3098.470.4
5096.770.1
20098.067.0
Base policy10066.5

(We have added these results in Appendix D.8, Figure 7, with a pointer from Section 4.1.)

Step 20 is the first checkpoint whose length has recovered. From step 20 to step 50, the teacher keeps both the full length and its coverage gain over the base policy, so any checkpoint in this range is a suitable teacher. Stopping earlier leaves the reasoning short, and continuing to step 200 removes the coverage gain.

The stopping criterion uses only the reasoning length, requiring neither labels nor a verifier.

AuthorsOpenReview ↗

Response to Reviewer Hsqu (Part 2/4)

Q2. Can length-controlled or shuffled-prior-attempt comparisons support the proposed mechanism? Does RSD increase training-set pass@8 and the fraction of rollout groups with both correct and incorrect answers? (Also addresses W1.)

The revision contains each of these comparisons, and we summarize them below.

(1) Shuffled prior attempt

We replaced the prior attempt in the teacher's context with the model's own rollout on a different problem. This keeps the source and the format of the context and removes only its relevance.

With the shuffled attempt, the teacher shows no coverage gain over the base policy, whereas with the relevant attempt it does. The same holds after distillation and RL, when the full RSD procedure is run with the shuffled attempt and followed by GRPO.

Average pass@1 on AIME 2024–2026 after GRPO (%)Qwen3-4BQwen3-8B
Base25.233.0
RSD with a shuffled prior attempt25.634.8
RSD (Ours)29.743.5

(We have added these results in Section 3, Figure 2, Section 6.2, Figure 3, and Appendix D.5, Table 11.)

This confirms that the gain requires the model's own attempt at the same problem.

(2) Length-controlled comparison

Along the teacher warmup, the reasoning length of the teacher and its coverage change independently (Qwen3-4B, AIME 2024–2026).

Warmup stepReasoning length (% of base)pass@256 (%)
0 (teacher of naive RSD)36.670.1
20 (teacher of RSD)109.871.3
20098.067.0
Base policy10066.5

(We have added these results in Appendix D.8, Figure 7.)

From step 0 to step 20, the length is restored while the coverage stays the same, so the teachers of naive RSD and of RSD differ in their reasoning length and not in their coverage. At step 20, the teacher is as long as the base policy and still 4.8 points above it, so this gain is not an effect of length.

(3) Training-set pass@8

We measured the outcome of the eight rollouts per training prompt during GRPO, at step 0 (the two initializations themselves) and at step 60.

Training prompts, Base / RSD (%)pass@8All-wrong groups (↓)All-correct groups (↑)
Qwen3-4B, step 070.8 / 73.829.2 / 26.214.9 / 14.9
Qwen3-4B, step 6076.7 / 80.423.3 / 19.624.0 / 30.5
Qwen3-8B, step 073.4 / 75.226.6 / 24.816.1 / 15.2
Qwen3-8B, step 6082.8 / 83.817.2 / 16.229.8 / 35.6

(We have added these results in Appendix D.2, Table 8.)

RSD raises pass@8 at step 0, so fewer groups have no correct rollout, and these are the groups from which GRPO cannot learn. The fraction of groups whose rollouts are all correct is similar for the two initializations at step 0, and it grows faster from the RSD initialization during training. The higher pass@8 and the lower fraction of all-wrong groups persist at every step.

AuthorsOpenReview ↗

Response to Reviewer Hsqu (Part 3/4)

Q3. Do the gains persist across training seeds or other model families? Such evidence would help assess whether the benefits are reliable beyond the current settings. (Also addresses W2.)

(1) Other model families

We added Olmo-3-7B-Instruct-DPO, a model outside the Qwen family that has not yet undergone RL post-training. With all settings identical to the Qwen3 experiments, the RSD initialization again improves over the base initialization under both GRPO and RLRT.

(We have added these results in Section 6.2, Table 2.)

(2) Training seeds

Due to the compute cost of each configuration, which consists of the two RSD stages followed by an RL run, we were unable to repeat the runs over several seeds, and each configuration is trained once. However, two properties of the results support their reliability.

First, the evaluation has little sampling noise. We do not estimate pass@k from k generations. Instead, every pass@1 and pass@4 is computed from 64 samples per problem with the unbiased estimator (as described in Appendix B.5). Even in the worst case, the standard error that sampling contributes to the five-benchmark average pass@1 is below 0.5 points, against gains of 2.6 to 9.5 points.

Second, the gain is consistent across independent pairs of runs, where the two runs of each pair share the configuration and differ only in the initialization.

Average pass@1 over five benchmarks (%)BaseRSDGain
Qwen3-4B, GRPO31.234.2+3.0
Qwen3-4B, RLRT43.452.0+8.6
Qwen3-8B, GRPO36.945.2+8.3
Qwen3-8B, RLRT43.753.2+9.5
Olmo-3-7B-Instruct-DPO, GRPO34.837.7+2.9
Olmo-3-7B-Instruct-DPO, RLRT35.237.8+2.6

Across three models and two on-policy algorithms, the RSD initialization is ahead in every pair. The same holds for the two GRPO runs that are trained on logical reasoning instead of mathematics (Section 6.2, Table 3). If the differences came from the variation between training runs, we would expect the sign of the gain to vary between pairs.

Note on the Qwen3-4B GRPO baseline. While running the additional experiments, we found that the Qwen3-4B Base + GRPO run of the submission was run on 8 GPUs, whereas the RSD + GRPO run it is compared with was run on 4. Both use 128 prompts per optimizer step, but in our trainer the scale of the gradient depends on the number of GPUs, so the two runs were not trained under the same conditions. We reran the baseline with the matched configuration and report this run in the revision (Section 6.2, Table 2). With it, the gain of the RSD initialization in this setting is 3.0 / 3.5 points in average pass@1 / pass@4, rather than the 1.6 / 1.2 points of the submission that the reviewer notes in W2.

Reviewer_To2rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes retrospective on-policy self-distillation (RSD) as an initialization for reasoning post-training. A teacher conditions on the model's prior attempt. A short role-reversed warm-up precedes distillation into a student that receives only the question. The initialization uses neither rewards nor reference answers, and inference requires a single generation pass. The experiments report improved large-k coverage and better downstream GRPO and RLRT performance across two Qwen3 model sizes.

Critical strengths
  1. RSD enables training an initialization model using self-generated attempts—free of external teachers or correctness feedback—while preserving standard single-pass inference. As outlined in Algorithm 1, the two stages integrate seamlessly.

  2. As shown in Table 2, the five-benchmark average pass@1 for Qwen3-8B improves from 36.9 to 45.2 under GRPO and from 43.7 to 53.2 under RLRT, with consistent gains also observed at the 4B scale.

  3. The paper directly addresses the additional initialization overhead: Section 6.3 and Table 3 benchmark RSD against extended base-policy GRPO training.

Critical weaknesses
  1. The claim that coverage gains stem primarily from previously unsolved problems needs more rigorous validation. Observing zero successes out of 1,024 samples indicates a low pass rate, but does not definitively prove impossibility. Crucially, the manuscript does not clarify whether the difficulty bins were defined independently of the baseline evaluation. If the same sample draws are used both to define a "zero-success" bin and to estimate its baseline pass rate, subsequent fresh samples drawn from the same base policy will naturally exhibit regression-to-the-mean effects, appearing to yield artificial improvements within that bin. To conclusively localize the gains reported in Figure 2(b) (distinct from the aggregate pass@kk gains in Figure 2(a)), the authors should employ an independent sample pool for binning or evaluate the base policy on a held-out set.

  2. Because teacher warmup alters both response length and the output token distribution simultaneously, the naive-vs-RSD comparison alone cannot establish length collapse as the primary cause of failure in direct distillation. Furthermore, the method contains an internal tension: Equation 4 can theoretically converge to a teacher policy that ignores the prior attempt. Although Section 5 selects step 20, the manuscript lacks direct evidence that this specific checkpoint successfully restores response length while maintaining meaningful conditioning on the prior context. To substantiate the causal claims in Sections 3 and 8, the authors could provide: 1) an ablation that isolates the effect of sequence length from broader distributional shifts; 2) an explicit measure that verifies context dependence on prior attempts.

Questions for authors
  1. Were the difficulty bins in Figure 2(b) defined using a sample pool independent of the reported baseline solve rates? If not, can you evaluate fresh base-policy samples in the fixed bins, or use independent samples to define the bins, and report whether the claimed concentration of gains remains?

  2. What evidence separates the effect of restoring response length from other changes made by warmup? At the selected step-20 teacher, can a controlled evaluation show that the prior attempt still affects useful predictions rather than only output length?

3 comments
AuthorsOpenReview ↗

Response to Reviewer To2r (Part 1/2)

We thank the reviewer for the constructive feedback. We respond to each question in order below, and the revised parts of the manuscript are marked in orange.

Q1. Were the difficulty bins in Figure 2(b) defined using a sample pool independent of the reported baseline solve rates? If not, can you evaluate fresh base-policy samples in the fixed bins, or use independent samples to define the bins, and report whether the claimed concentration of gains remains? (Also addresses W1.)

In the submission, the bins and the base policy's solve rates were computed from the same samples, so they were not independent.

Following the reviewer's suggestion, we redid the analysis with bins that are defined on base-policy samples independent of the samples used for the reported solve rates.

(We have updated Section 3, Figure 2-(b) and Appendix D.1, Figure 4-(b) with these results.)

With independent samples, we observe the same concentration of the gain. The gain of retrospection still comes from the problems that the base policy finds hardest, confirming that the concentration is not an artifact of defining the bins and the solve rates on the same samples.

AuthorsOpenReview ↗

Response to Reviewer To2r (Part 2/2)

Q2. What evidence separates the effect of restoring response length from other changes made by warmup? At the selected step-20 teacher, can a controlled evaluation show that the prior attempt still affects useful predictions rather than only output length? (Also addresses W2.)

(1) The step-20 teacher reasons at full length and keeps the coverage gain

To separate the effect of the response length from the effect of the prior attempt, we evaluated the conditioned policy at successive warmup checkpoints. At each checkpoint, we measured its reasoning length and its pass@256 against the base policy, which has no prior attempt in its context (Qwen3-4B, AIME 2024–2026).

Warmup stepReasoning length (% of base)pass@256 (%)Gain over base
0 (no warmup)36.670.1+3.6
1061.971.2+4.7
20 (selected)109.871.3+4.8
5096.770.1+3.6
20098.067.0+0.5
Base policy10066.5–

(We have added these results in Appendix D.8, Figure 7.)

From step 0 to step 20, the length is restored from 36.6% to the level of the base policy, while pass@256 stays at about the same value. From step 20 to step 200, the length stays at that level, while pass@256 falls to that of the base policy.

In particular, the step-20 teacher reasons as long as the base policy, so its gain of 4.8 points in pass@256 cannot come from the output length. The gain disappears only when the warmup continues until the context is ignored (step 200). This is the convergence to a context-ignoring teacher that the reviewer describes, and the early stop avoids it.

Furthermore, the selection of step 20 requires neither labels nor a verifier, since the checkpoint is chosen as the first one whose length has recovered. The choice of step is also robust, as the checkpoints from step 20 to step 50 all keep both the length and the gain.

(2) An unrelated attempt in the teacher's context does not reproduce the gain

As a controlled evaluation of the dependence on the prior attempt, we reran the full RSD procedure with a single change. The prior attempt in the teacher's context is replaced by the model's own rollout on a different problem, in both the warmup and the distillation stage. This keeps the source and the format of the context and removes only its relevance. We then ran GRPO from the resulting policy.

Average pass@1 on AIME 2024–2026 after GRPO (%)Qwen3-4BQwen3-8B
Base25.233.0
RSD with an attempt at a different problem in the teacher's context25.634.8
RSD (Ours)29.743.5

(We have added these results in Appendix D.5, Table 11.)

GRPO from the resulting policy stays close to GRPO from the base policy and far below RSD at both scales. The gain therefore depends on the prior attempt, and not just on the presence of any additional context.

Dear Reviewer To2r,

Thank you again for your careful review. As the discussion period closes in a few hours, we wanted to check whether our response has addressed your concerns. If anything remains unclear, we would be glad to clarify it in the time that is left. We sincerely appreciate your time and effort.

Reviewer_NdZcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Retrospective On-Policy Self-Distillation (RSD), a method for improving the reasoning ability of LLMs through distillation on its own prior attempts at problems. My understanding of the paper is as follows:

  • The authors argue that humans can improve their reasoning by learning from their own past attempts at solving a problem, in addition to being shown an answer: "Research in cognitive science, for instance, finds that learners who generate and revise their own solutions before being shown one learn more than those given the solution directly". They call this idea "retrospection".
  • They then investigate how LLMs utilize retrospection. They condition model generation on past attempts (the teacher), and compare this against sampling answers directly from the problem only: p(⋅∣x,τ)p(\cdot | x, \tau) vs. p(⋅∣x)p(\cdot | x). The authors argue that the former yields higher coverage, as evidenced by higher pass@k on AIME. The authors explore using this retrospective distribution to teach the student policy, but find that this fails, as the output lengths fall drastically.
  • To fix this, the authors propose training a teacher model using a role reversed objective: train the teacher using distillation, with the student as the teacher. After a warmup phase using this objective, roles are re-reversed, and the student is trained against the frozen warmed-up teacher on its own trajectories. They call this method RSD.
  • After initializing the model using RSD training, the authors conduct RL (GRPO, RLRT) on the DAPO math dataset. On Qwen3-4B and 8B with reasoning disabled, they demonstrate gains across various math benchmarks vs. pure RL with no RSD training. They show that gains hold in a FLOPS matched setting.
Critical strengths
  • I appreciate the FLOPS matched comparisons.
  • Writing is clear, presentation is good
Critical weaknesses
  • I am unconvinced by the core motivation behind this method. Why is p(⋅∣x,τ)p(\cdot | x, \tau) a good distillation target? Why do we want the policy to generate outputs when conditioned on only the problem that look like outputs generated when conditioned on a past attempt? Why, conceptually, is this a useful distribution to learn? I think models ought to behave very differently when given examples of past reasoning than when they have to reason from scratch. Furthermore, this trajectory τ\tau is not any better than what the policy can sample in expectation, so unlike in OPSD, we cannot even claim it provides useful context or privileged information. The author's indeed show this in the naive RSD setting, where they find (expectedly) a large collapse in output lengths when distilled this way. While this is superficially fixed in RSD via the warmup, I remain unconvinced (and I do not think there is substantial work done to establish) that retrospection done in this way is valuable, beyond showing some gains on math benchmarks.
  • On the broader motivation re. human learning, I agree that retrospection is a useful way to learn for humans and perhaps for LLMs too. But the natural idea would be to teach LLMs to retrospect their own solutions via reasoning, and to propose self-corrections, improvement etc. as is done within the reasoning traces of modern reasoning models. The authors write: "(ii) Single generation pass. Both the learning that follows and inference with the trained model should draw attempts in one generation pass from the query x, so the improvement must be internalized into pθ(· | x) rather than implemented as an inference-time procedure.", which presumably distinguishes their setting from the CoT exploration setting, but I am unconvinced why this requirement is a good one. What is the advantage of internalizing this vs. learning better test time search? Even then, it is unclear why this is the right thing to internalize (see my point above).
  • It is unclear what the resulting trained policy does any differently behavior-wise, as the authors do not attempt to analyze how the resulting RSD-trained model differs from the normal RL trained model. I suspect that learning to mimic p(⋅∣x,τ)p(\cdot | x, \tau) has substantial behavioral effects.
  • The experimental settings are fairly weak. (1) The Qwen models here were used with thinking disabled. It is thus unclear if RSD would help when thinking is enabled (which is generally the setting used when doing any kind of reasoning work in practice), especially since enabling thinking generally encourages retrospective behavior in the reasoning traces themselves (via test time scaling). Furthermore, distillation done in this way may curb exploration when thinking is enabled, as the model is trained to rely on past exploration (via τ\tau, due to the update towards p(⋅∣x,τ)p(\cdot | x, \tau) via distillation) instead of exploring on its own. (2) The authors train on math (DAPO) and evaluate on math, so we do not know how RSD affects OOD generalization, or whether it works in other reasoning intensive domains (science, agentic coding etc.).
  • There are very few ablations, so it is hard to ascribe credit properly. At minimum, I would like to see (1) distillation against a teacher conditioned on irrelevant context (to isolate that retrospective distillation against past attempts is driving gains) (2) using the warmed-up only model as the policy (or doing RL directly on the warmed up model) (3) learning from past attempts in a naive way e.g. with SFT or rejection sampling, followed by RL.
  • One of the main motivations connecting Section 3 (analysis) and the method is that "In Figure 2-(a), the conditioned policy’s pass@k pulls away from the base policy’s as k grows, reaching +4.6 points at k=512." In Fig 2(a), we indeed see that RSD/teacher conditioning provides no, or very small gains over the base model at low k. The RL setting chosen here uses N=8 parallel rollouts. Why do we expect coverage to transfer to improved RL if pass@k is not meaningfully different at low k?
Questions for authors

I may raise my score if the following can be answered:

  • Does this work when thinking is enabled and test time scaling via long CoT is allowed?
  • How does this affect OOD generalization?
  • How does this affect the reasoning behaviour of models vs. regular RL?
  • Which parts of the algorithm are doing the "heavy lifting"? (aka. ablations).
  • Why is p(⋅∣x,τ)p(\cdot | x, \tau) a good distillation target? Conceptually. Why do I want p(⋅∣x)p(\cdot | x) to look like p(⋅∣x,τ)p(\cdot | x, \tau)?
8 comments
AuthorsOpenReview ↗

Response to Reviewer NdZc (Part 1/6)

We thank the reviewer for the constructive feedback. We respond to each question in order below, and the revised parts of the manuscript are marked in orange.

Q1. Does this work when thinking is enabled and test time scaling via long CoT is allowed? (Also addresses W2 and W4-(1).)

(1) Thinking mode

We would like to clarify that disabling thinking does not disable reasoning. Non-thinking mode only switches off Qwen3's built-in <think> channel, and our prompt asks for step-by-step reasoning inside a <reason> block (Appendix C). The policies reason at length in a single pass, with about 2.6k tokens for the base policy and 3.2k for the RSD-trained policy (Section 3, Figure 2-(c)), under a budget of 32,768 tokens at evaluation. Long chain-of-thought reasoning is therefore already part of our setting.

This setting also follows the closest prior works, which all generate their training rollouts with thinking disabled on the same model family:

Following the reviewer's question, we also ran GRPO on Qwen3-4B with thinking enabled on the same training data.

GRPO step (Qwen3-4B, thinking enabled)0204060
pass@1 on AIME 2024–2026 (%)66.465.163.865.7
pass@4 (%)78.177.778.879.3

(We have added this discussion in Appendix B.6.)

With thinking enabled, the performance stays flat, and the training reward moves only from 0.76 to 0.78. The thinking mode of Qwen3 is itself the product of a dedicated reasoning post-training pipeline (Yang et al., 2025), so further RL on our training data has little left to improve. Since our claim concerns how much the learning loop gains from a better attempt, it cannot be measured in a regime where the loop itself does not learn, for RSD or for any other method.

To examine the learning loop on a model without a thinking switch, we added Olmo-3-7B-Instruct-DPO (Team Olmo et al., 2025). Olmo 3 releases its Instruct and Think variants as separate models, and this checkpoint precedes the RLVR stage of its post-training, so it has not yet undergone reinforcement learning. With all settings identical to the Qwen3 experiments, the RSD initialization again improves over the base initialization under both GRPO and RLRT.

Olmo-3-7B-Instruct-DPO (average over the five benchmarks of Table 2)pass@1 (%)pass@4 (%)
Base + GRPO34.852.2
RSD + GRPO (Ours)37.755.1
Base + RLRT35.252.7
RSD + RLRT (Ours)37.854.6

(We have added these results in Section 6.2, Table 2.)

(2) Test-time scaling and the single-pass requirement

We agree that test-time search is valuable, and RSD is not meant as an alternative to it. The single-pass requirement comes from the learning loop that RSD is designed to serve, in which every rollout is one sample from the policy given the query alone.

If retrospection remained an inference-time procedure, every rollout would need a first attempt followed by a second, conditioned pass. This would double the generation cost of every training step (16 generations per prompt instead of 8 in our setting) and of every later inference call. RSD pays this cost once, in a short label-free stage, after which every rollout draws from the improved distribution at the original cost.

Furthermore, a better single-pass policy is also a better starting point for test-time scaling. To verify this, we applied majority voting over k samples to the models after on-policy learning, on AIME 2024–2026.

Majority-vote accuracy (%)k=1k=4k=16k=64
Qwen3-4B, Base + GRPO25.030.235.838.6
Qwen3-4B, RSD + GRPO (Ours)29.837.342.544.8
Qwen3-8B, Base + GRPO33.343.651.655.7
Qwen3-8B, RSD + GRPO (Ours)44.453.260.462.2
Qwen3-4B, Base + RLRT41.952.560.561.1
Qwen3-4B, RSD + RLRT (Ours)53.965.972.073.3
Qwen3-8B, Base + RLRT41.659.571.476.0
Qwen3-8B, RSD + RLRT (Ours)53.970.676.878.9

(We have added these results in Appendix D.7, Table 12.)

Voting improves both initializations, and the RSD-initialized model stays ahead at every k under both GRPO and RLRT. This confirms that the benefit of RSD is complementary to test-time scaling.

AuthorsOpenReview ↗

Response to Reviewer NdZc (Part 2/6)

Q2. How does this affect OOD generalization? (Also addresses W4-(2).)

In the submission, both the RSD stage and the downstream on-policy learning use mathematical data (DAPO-Math-17k), and all evaluations are on mathematics, so out-of-domain generalization is not tested.

To address this, we added two experiments on logical reasoning. Both are evaluated on Knights and Knaves puzzles (K&K, with 4–5 and with 7–8 people) and on Web of Lies:

  • Setting 1 (trained on math, evaluated on logic): The Base + GRPO and RSD + GRPO models of Table 2, which see no logical-reasoning data at any stage, are evaluated directly on the three benchmarks.
  • Setting 2 (trained on logic from a math-distilled initialization): GRPO is run on K&K from the base policy and from the RSD initialization, which is distilled from mathematical data only.
Accuracy (%)K&K 4–5K&K 7–8Web of Lies
Setting 1: GRPO trained on math
Qwen3-4B, Base + GRPO87.367.290.3
Qwen3-4B, RSD + GRPO (Ours)94.981.995.5
Qwen3-8B, Base + GRPO93.877.288.3
Qwen3-8B, RSD + GRPO (Ours)97.887.196.7
Setting 2: GRPO trained on K&K
Qwen3-4B, Base + GRPO95.181.089.9
Qwen3-4B, RSD + GRPO (Ours)96.388.295.9
Qwen3-8B, Base + GRPO96.886.288.5
Qwen3-8B, RSD + GRPO (Ours)98.188.997.2

(We have added these results in Section 6.2, Table 3.)

Across both settings and both scales, the RSD-initialized model is better on every benchmark. In Setting 1, the gain reaches +14.7 points on K&K 7–8 at 4B and +9.9 at 8B, although neither stage was trained on logical reasoning. In Setting 2, the RSD-initialized run also stays ahead on Web of Lies, which is not part of the logical training data (89.9% → 95.9% at 4B and 88.5% → 97.2% at 8B).

This confirms that the benefit of the RSD initialization is not confined to the mathematical domain. It transfers to out-of-domain evaluation, and it also improves on-policy learning in a domain other than the one RSD was distilled on. We leave the other domains that the reviewer mentions, such as science and agentic coding, to future work.

AuthorsOpenReview ↗

Response to Reviewer NdZc (Part 3/6)

Q3. How does this affect the reasoning behaviour of models vs. regular RL? (Also addresses W3.)

The main difference is that the RSD-trained policy reaches a broader set of final answers. Because it is distilled from the warmup teacher, it does not write as if an attempt had been provided, and its reasoning is not shortened. We measured each of these on AIME 2024–2026. To separate the effect of retrospection from the effect of the additional training, we also included the irrelevant-context control that the reviewer suggests, in which RSD is run with an unrelated attempt in the teacher's context (see our response to Q4).

(1) A broader set of final answers

We sampled 256 responses per problem from each policy before RL and counted the distinct final answers. We also measured the accuracy on the hardest problems, which are those that the base policy solves less than 0.5% of the time (as in Section 3, Figure 2-(b)).

Before RL4B Base4B Control4B RSD (Ours)8B Base8B Control8B RSD (Ours)
Avg. distinct final answers / problem45.346.749.743.243.848.4
Accuracy, hardest problems (%)0.130.170.450.110.130.26

(We have added these results in Appendix D.10, Table 14.)

The RSD policy reaches more distinct final answers than the base policy at both scales. The additional answers include correct ones, as its accuracy on the hardest problems is 3.5 times that of the base policy at 4B and 2.4 times at 8B. The control stays close to the base policy on both measures, so the effect comes from the relevant attempt rather than from the additional training.

W4-(1) raises the concern that distilling from a teacher that sees an earlier attempt may curb the policy's own exploration. We observe the opposite. This broader coverage is also what RL then learns from, since it leaves fewer training prompts without a correct rollout (see our response to W6).

(2) No explicit mention of a provided attempt

W3 raises the concern that imitating pθw(⋅∣x,τ1)p^{\theta_w}(\cdot \mid x, \tau^1) has substantial behavioral effects. The most direct such effect would be a policy that writes as if an earlier attempt had been provided, with phrases such as "Sample 1 makes an error". We therefore counted such responses among 64 samples per problem, using a pattern match followed by GPT-4.1 as a judge.

Qwen3-4B policyAttempt in contextResponses that explicitly mention a provided attempt (%)
Teacher without the warmupGiven7.29
Naive RSD (distilled without the warmup)None4.86
Teacher after the warmupGiven0.07
RSD (Ours), before RLNone0.00
RSD (Ours), after GRPONone0.00

(We have added these results in Appendix D.10, Table 15.)

Without the warmup, this behavior does transfer. The naive student receives no attempt, yet it mentions one in 4.9% of its responses. After the warmup, the teacher almost never mentions the attempt, while it still makes use of it, since its coverage gain remains (Appendix D.8, Figure 7). The RSD policy contains no such mention, before or after GRPO.

(3) Reasoning length

Before RL, the RSD policy reasons for 3.2k tokens, against 2.5k for the base policy at 4B and 2.6k at 8B. After GRPO, the RSD-initialized policy also reasons longer than the base-initialized one (5.4k against 4.2k tokens at 4B and 7.4k against 5.6k at 8B).

(We have added these results in Appendix D.10, Table 16.)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback. Below is a short summary of the main concerns and our responses, with the place of each result in the revision. Additions to the paper are marked in orange.

1. Source of the gain: retrospection or additional context

We reran the full RSD procedure with the prior attempt replaced by the model's own rollout on a different problem. GRPO from the resulting policy stays close to GRPO from the base policy (25.6 vs. 25.2 average pass@1 on AIME 2024–2026 at 4B and 34.8 vs. 33.0 at 8B) and far below RSD (29.7 and 43.5). The gain therefore requires the model's attempt at the same problem. Replacing the on-policy distillation with SFT keeps part of the gain (39.1 at 8B, against 33.0 for the base policy and 43.5 for RSD).

(See Section 6.2, Figure 3, Appendix D.5, Table 11, and Appendix D.6, Figure 6.)

2. Teacher warmup

On Qwen3-4B, the selected step-20 teacher reasons as long as the base policy (109.8% of its length) and keeps its coverage gain (71.3% pass@256, against 66.5% for the base policy). The gain disappears only when the warmup continues to step 200, so the early stop restores the length without erasing the effect of the prior attempt. Prompting the teacher to reason completely lengthens its reasoning only to 45–47% of the base length. The stopping criterion uses the reasoning length alone and requires no labels.

(See Appendix D.8, Figure 7 and Appendix D.9, Table 13.)

3. From coverage to RL

With difficulty bins defined on independent samples, the gain still comes from the problems that the base policy rarely solves. GRPO draws eight rollouts per prompt. On the training prompts, RSD raises pass@8 before any RL step (70.8 → 73.8 at 4B and 73.4 → 75.2 at 8B), so fewer prompts are left without a correct rollout, and this advantage persists during training.

(See Section 3, Figure 2-(b) and Appendix D.2, Table 8.)

4. Generality and reliability

  • Model family: On Olmo-3-7B-Instruct-DPO, a checkpoint before RL post-training, the RSD initialization improves average pass@1 by 2.9 points under GRPO and 2.6 under RLRT.
  • Out of domain: RSD is distilled from mathematical data only. On logical reasoning (Knights and Knaves, Web of Lies), the RSD-initialized model is better on every benchmark, both when the math-trained models are evaluated directly (e.g., 77.2 → 87.1 on the harder Knights and Knaves set at 8B) and when GRPO is trained on logic.
  • Test-time scaling: Under majority voting with 64 samples, the RSD-initialized model stays ahead (38.6 → 44.8 at 4B and 55.7 → 62.2 at 8B under GRPO).
  • Seeds: Each configuration is trained once. However, pass@k is computed with the unbiased estimator from N=64 samples per problem, far more than the k being evaluated (k ≤ 4), so the reported values have little sampling noise. The RSD initialization is ahead in all six model–algorithm pairs (+2.6 to +9.5 points).

(See Section 6.2, Tables 2–3 and Appendix D.7, Table 12.)

5. Behavior of the RSD-trained policy

Before RL, the policy reaches more distinct final answers per problem (45.3 → 49.7 at 4B and 43.2 → 48.4 at 8B) and solves the hardest problems more often. On Qwen3-4B, it does not write as if an attempt had been provided (0% of its responses, against 4.9% for the student distilled without the warmup). Its reasoning is not shortened.

(See Appendix D.10, Tables 14–16.)

6. Thinking mode

Our experiments disable thinking, following the closest prior work. The policies still reason at length (2.5k–3.2k tokens before RL and 12k–18k after RLRT). With thinking enabled, GRPO from the base Qwen3-4B does not improve on our training data (pass@1 moves from 66.4 to 65.7), so the effect of the initialization cannot be measured there. We note this as a limitation. Our work studies the effect of an initialization on the RL that follows, so the setting of interest is a model that RL can still improve, which holds for Qwen3 with thinking disabled and for Olmo-3.

(See Appendix B.6 and the Limitations in Appendix A.)

We sincerely thank the Area Chair and the reviewers for the time and effort they have devoted to reviewing our work and to the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.