- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a known limitation of group-relative RLVR objectives: when all rollouts in a group receive the same reward, the group-relative advantage vanishes and the information in those trajectories is discarded. The authors propose prospective learning, which uses post-hoc experience to supervise predictions the agent makes before interaction, and instantiate it as Self-Retrospection Distillation (SRD). For each on-policy rollout, a hindsight-conditioned self-teacher (conditioned on the trajectory, the ground-truth answer, and error annotations) supervises a trajectory-blind student that predicts either the KNOWLEDGE a task requires (from successful rollouts) or the PITFALLS it may invite (from failed rollouts). The two are aligned via token-level JSD on the student's own foresight samples. SRD is added as an auxiliary loss () on top of GRPO, OPSD, and RLSD, and foresight is not generated at inference time.
Experiments on 10 benchmarks spanning math, code, search, and agentic tasks with Qwen3.5-4B/9B show improvements over the base objectives in most settings, with gains of up to 24.2 pp. In a code-only setting with dynamic sampling disabled, SRD lifts a 2B model's training success rate from 0.0% to 60.6%. The paper also includes a channel ablation (concluding that PITFALL alone suffices), a token-level analysis of how SRD redistributes the policy's outputs, a theoretical analysis of the reward-uniform regime, and a Fisher-metric decomposition of update directions.
Critical strengths
S1. A clear and potentially broadly applicable reframing of how hindsight is used. Prior on-policy self-distillation work mainly varies what the privileged context contains. This paper separates "what hindsight contains" from "what hindsight supervises," and redirects supervision from the action distribution to pre-interaction anticipation. This matters for my assessment because, if validated, it opens a design axis that is orthogonal to existing self-distillation methods and composable with them as a lightweight auxiliary loss.
S2. Broad and often large empirical gains under controlled conditions. SRD is evaluated across four domains, ten benchmarks, three base objectives, and two model scales, with identical data and prompt ordering across arms and several distribution shifts (stdin-to-functional code format, unseen ALFWorld environments, and the longer horizon of BrowseComp-Plus). Several gains are substantial: for example, +10.0 / +8.3 / +8.1 pp on the Math / Code / Search averages for 4B GRPO, and +17.2 pp on the Math average for 9B OPSD. This matters because it shows the approach is practically relevant and worth investigating, even though, as discussed below, the attribution of these gains remains unclear.
Critical weaknesses
W1. The source of SRD's performance gains is unclear: the gains are confounded with a change in the teacher's privileged information, and the mechanism by which foresight distillation improves task performance is not validated.
(a) Confounding with the change in the teacher's privileged information. In the OPSD and RLSD experiments, introducing SRD involves two simultaneous modifications: (1) replacing the privileged information used by the original teacher with structured hindsight, and (2) adding the SRD auxiliary distillation loss. Consequently, the improvements of OPSD+SRD and RLSD+SRD in Table 1 cannot be fully attributed to the proposed prospective learning. In fact, the ablation in Table 2 already shows that changing the teacher's privileged information alone yields substantial gains. On AIME26, 4B OPSD improves from 53.75% to 63.75% with the hindsight prefix alone, and only reaches 67.08% after additionally adding PITFALL SRD. Similarly, 9B OPSD improves from 65.42% to 77.50%, and reaches 84.17% with SRD. In these experiments, changing the teacher's input therefore accounts for the majority of the performance gain. Moreover, the OPSD baseline results in Table 2 differ markedly from those in Table 1, and no corresponding analysis is provided for RLSD. I recommend that the authors add OPSD and RLSD controls that replace only the hindsight prefix without the SRD loss, under exactly the same training configuration as the main experiments, so that the contribution of each modification can be clearly separated.
(b) Missing mechanistic validation. The core idea of SRD is to use post-hoc information to supervise the foresight generated by the model. However, at inference time, the model does not explicitly generate foresight; it directly performs the original task. This leaves a critical question insufficiently validated: why does distilling the foresight distribution improve the model's reasoning and decision-making on the original task? Although SRD shares parameters with the original policy, this does not imply that improved foresight prediction necessarily transfers to actual task performance. The existing analyses of token distribution shifts and parameter update directions provide mainly correlational evidence and are not sufficient to establish this causal relationship. The gains may instead stem from additional auxiliary training or regularization effects rather than from the proposed prospective learning mechanism. I suggest the following controls: (1) perform the same distillation using randomly shuffled hindsight or hindsight from other tasks, to rule out gains from a generic auxiliary loss; (2) perform SFT directly on the hindsight text, to test whether foresight distillation is actually necessary; and (3) condition the teacher only on the ground-truth answer without the interaction trajectory, to test whether the improvement truly depends on information extracted from interaction experience.
Why this is consequential. Taken together, (a) and (b) mean that none of the main results cleanly isolates prospective learning as the source of improvement: for OPSD and RLSD, most of the gain is attributable to the change in the teacher's input, while for GRPO, the mechanism by which the auxiliary loss improves task performance is untested. Without the controls above, the current experiments do not sufficiently support the paper's central claim or its distinct value relative to existing distillation methods.
W2. The core claim of "learning from reward-uniform groups" lacks sufficient empirical support.
The paper presents exploiting trajectory information from reward-uniform groups (all-correct or all-incorrect) as one of its core contributions, but the main experimental setup is somewhat disconnected from this claim. First, GRPO and RLSD in Table 1 still use dynamic sampling, which retains only groups with reward variance, so SRD does not actually exploit the filtered reward-uniform groups in the main results. Second, the main experiments use only the PITFALL channel by default, which cannot extract any SRD supervision from all-correct groups. This is not fully consistent with the introduction's emphasis that both successful and failed trajectories provide additional supervision.
Although the authors disable dynamic sampling in RQ3 to verify SRD's ability to learn from reward-uniform groups, this experiment covers only the code domain, and its most striking result (the 2B model improving from 0.0% to 60.6%) is reported only as training success rate, without generalization results on an independent test set. In addition, the 100% usable budget of SRD in Figure 4(a) mainly reflects that the auxiliary loss is computable on reward-uniform groups, and does not directly show that these trajectories all provide effective learning signals.
Therefore, while the current experiments show that SRD can continue to provide a training signal under sparse rewards, they are not sufficient to show that exploiting reward-uniform groups leads to higher sample efficiency under standard training configurations. I recommend comparing against standard GRPO with dynamic sampling under the same rollout or compute budget, and reporting results on independent test sets, to further validate this core claim.
Questions for authors
-
(W1a) Under the training configuration of Table 1, how much of the OPSD and RLSD improvement remains attributable to SRD after controlling for the change in the teacher's privileged prefix?
-
(W1b) What evidence distinguishes the benefit of foresight distillation from a generic auxiliary-training effect, given that foresight is not generated at inference time?
-
(W2) With reward-uniform groups retained, does SRD improve held-out performance over standard GRPO with dynamic sampling under a matched rollout or compute budget, rather than only improving training success?
5 comments
Response to Reviewer UmXD - p2
W1(b). Mechanism: why does foresight distillation help if foresight is not generated at inference? We ran the three requested controls on GRPO+SRD, math+code+search, with the same budget. Benchmarks: AIME26 (math), LCB v6 functional (code), HotpotQA (search), Average@8 %.
| control | what changes in SRD target | 4B: AIME26 / LCB / Hotpot (avg) | 9B: AIME26 / LCB / Hotpot (avg) |
|---|---|---|---|
| GRPO | none | 54.6 / 55.4 / 71.0 (60.3) | 70.4 / 67.3 / 75.4 (71.0) |
| SRD (arm e) | – | 70.8 / 63.9 / 74.3 (69.7) | 81.7 / 67.5 / 75.6 (74.9) |
| (1) cross-domain hindsight | hindsight from another domain (code←math, search←code, math←search) | 67.1 / 61.3 / 70.9 (66.4) | 79.6 / 66.5 / 74.9 (73.7) |
| (2) SFT on hindsight text | CE on hindsight tokens instead of distillation | 60.8 / 57.7 / 72.5 (63.7) | 84.2 / 72.8 / 74.1 (77.0) |
| (3) ground-truth-only teacher | no trajectory: answer / first test cases only | 52.9 / 54.2 / 71.5 (59.5) | 75.8 / 67.9 / 76.6 (73.4) |
- (1) Cross-domain hindsight keeps most of the gain. (4B: 66.4 vs. SRD 69.7 vs. GRPO 60.3; 9B: 73.7 vs. 74.9 vs. 71.0). Since one training batch mixes the domains, the signal the policy learns from is cross-domain anyway; the hindsight is therefore general rather than problem-specific, and the method is robust to this change. The benefit does not require hindsight that is specific to the same problem, so this control does not rule out that part of the gain comes from learning to produce foresight-style text (a structured pre-action prediction) rather than from the problem-specific content. Our claim is therefore the prospective-learning objective — predicting post-hoc information before acting, as a distillation target — not that the particular content of each hindsight is what transfers.
- (2) SFT on the hindsight text: mixed. This control simply replaces the JSD target of SRD with an SFT loss on the same hindsight text. At 4B it is worse than distillation (63.7 vs. 69.7; AIME26 60.8 vs. 70.8) but better than GRPO. At 9B it is not worse: 77.0 vs. SRD 74.9 (AIME26 84.2 vs. 81.7, LCB 72.8 vs. 67.5, HotpotQA 74.1 vs. 75.6). We therefore do not claim that distillation beats SFT on the hindsight text; the evidence is that training on the hindsight text itself is a large part of the gain, and the advantage of the distillation form is not established at 9B.
- (3) A ground-truth-only teacher: scale-dependent. Privileged information that is not grounded in the interaction can hurt reasoning [1]. At 4B it gives no gain (59.5 vs. GRPO 60.3 vs. SRD 69.7); it peaked mid-run (AIME26 64.2 at step 29) and fell back while the mean response length shrank from 3385 to 1650 tokens, the collapse trend we had predicted. At 9B it does help (73.4 vs. GRPO 71.0, SRD 74.9; AIME26 75.8 vs. 81.7, LCB 67.9 vs. 67.5, HotpotQA 76.6 vs. 75.6), i.e., most of the 9B gain on LCB and HotpotQA is reachable with the answer alone, and only on AIME26 is the trajectory-informed teacher clearly ahead. Response length at the last rollout: 1.7–2.4 k tokens, from 3.1–3.8 k at the start.
Within the prospective learning framework, every variant is supposed to improve on the base method, so SRD is one setting of a broader family.
Answer to Q(W1b): the controls show that the gain is not specific to the same-problem hindsight (cross-domain keeps most of it at both sizes) and that at 4B information from the trajectory matters (GT-only ≈ GRPO < SFT < SRD). At 9B the picture is weaker: SFT matches or exceeds SRD and GT-only recovers most of the gain, so the 9B data do not separate the mechanism from a generic training-signal effect. We will say this plainly and not claim more.
[1] Kim. et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
Response to Reviewer UmXD - p3
W2. Reward-uniform groups: evidence is thin. (a) Main experiments use dynamic sampling and only the pitfall channel. Correct, by design of the baselines; the code-only setting (RQ3) is where uniform groups are kept. We now extend RQ3 to code-only, both channels, 2B–9B, with held-out test sets (LCB v6 functional and OJBench), not only training success; the results are below. Held-out average@8 %, final step; "base" is the model before training (training and test data are the code domain only); dynamic sampling is not used for either arm, so GRPO has no gradient on reward-uniform groups while SRD still supplies one:
| code-only, no dynamic sampling | LCB v6: base | GRPO | GRPO+SRD | OJBench: base | GRPO | GRPO+SRD |
|---|---|---|---|---|---|---|
| 2B | 17.7 | 25.6 | 27.0 | 0.4 | 2.3 | 6.5 |
| 4B | 41.5 | 48.2 | 58.9 | 6.3 | 23.9 | 23.2 |
| 9B | 56.7 | 66.1 | 68.5 | 10.9 | 30.0 | 35.4 |
SRD is ahead of GRPO on LCB at all three sizes (+1.4, +10.7, +2.4) and on OJBench at 2B and 9B (+4.2, +5.4); the exception is OJBench at 4B (23.2 vs. 23.9). The clearest gain is 4B LCB.
(b) The learning signal on the 100% useable traces We therefore read the distillation signal (KL between teacher and student) on reward-uniform groups over training from the code-only runs: it is non-zero on all-wrong and on all-correct groups at 2B, 4B and 9B. The pitfall term (all-wrong groups) decays as the policy improves (mean per-token loss 0.0020 → 0.0002 at 2B, 0.0024 → 0.0001 at 4B, 0.0015 → 0.0004 at 9B) while the knowledge term (all-correct groups) grows (0.0000 → 0.0001 / 0.0011 / 0.0018), so the total stays non-zero (at 9B it rises, 0.0015 → 0.0023). A non-zero KL shows that SRD has a training signal where policy gradient has none; whether this signal helps is answered by the held-out results above.
(c) Matched budget against dynamic sampling, on the hardest problems. We trained on the LCB v6 training problems that carry the benchmark's hard label only (248 of 422), where most sampled groups are all-wrong, comparing GRPO + dynamic sampling (DS) with GRPO + SRD (both channels, no DS), otherwise the same configuration. Held-out Average@8 %:
| trained on hard problems only | LCB v6 functional | OJBench |
|---|---|---|
| 2B base | 15.5 | 0.5 |
| 2B GRPO + SRD (no DS) | 28.0 | 5.2 |
| 2B GRPO + DS | 18.9 | 1.3 |
What we can say from the runs that exist: (i) SRD keeps improving when almost every group is all-wrong: the 2B SRD run goes from 15.5 to 28.0 on LCB (+12.5) and from 0.5 to 5.2 on OJBench, at ≈3.7 min per step. (ii) The 2B DS arm struggles to obtain informative gradients on these hard problems: assembling its first batch of 32 groups required sampling and discarding 2,278 reward-uniform (all-wrong) groups, leaving fewer than 2% of sampled groups usable. This single update required ≈71 min (4,270 s) of rollout time, about 19× the time of an SRD step. At this observed rate, the same ≈3-hour training budget would allow only about 2–3 DS updates, compared with 51 SRD steps. This highlights the bottleneck of dynamic sampling on extremely difficult problems: obtaining reward contrast becomes prohibitively expensive, whereas SRD can learn directly from the abundant all-wrong trajectories.
We thank the reviewer again for their valuable feedback. We hope the additional experiments and analyses have helped address the remaining concerns and strengthen the empirical evidence for our approach. We would be glad to discuss any further questions.
Response to Reviewer UmXD - p1
We thank the reviewer for the detailed and constructive review and for the positive assessment: S1 SRD separates "what hindsight contains" from "what hindsight supervises" and composes with existing self-distillation methods; S2 the gains are broad under controlled conditions. We answer the attribution questions in W1 and W2 with new controls below.
W1(a). Confounding between the teacher's privileged information and the SRD loss. We agree that the original Table 2 did not separate them. We added the missing control: hindsight prefix, no SRD loss (new arm d), same configuration as the main runs, for OPSD and RLSD, 4B and 9B, math+code+search. The original trace prefix OPSD and hindsight prefix + SRD are the existing runs from Table 1.
In short: yes, most of the OPSD gain in Table 1 comes from the teacher prefix. However, in SRD the hindsight prefix can directly reuse retrospections already generated, avoiding a separate hindsight-extraction pass, and it is shorter than the original trace prefix, which makes training easier. And our SRD can still improve over the hindsight prefix OPSD/RLSD as shown in Table 2 (where we only use the math training data)
Held-out Average@8, % (five benchmarks evaluated on all arms). Trace prefix OPSD and hindsight prefix + SRD are the original runs at the end of training; Hindsight prefix OPSD, no SRD loss, is the new control.
Answer to Q(W1a): Relative to the original trace prefix, replacing it with the hindsight prefix alone changes the five-benchmark average by +7.8 pp (4B OPSD), +7.5 pp (9B OPSD), −1.9 pp (4B RLSD), and +0.7 pp (9B RLSD). Adding the SRD loss on top further improves performance by +0.9, +2.8, +3.0, and +2.6 pp, respectively.
These results show that the prefix change accounts for most of the improvement in OPSD, while the SRD loss provides the larger contribution in RLSD. Importantly, the SRD loss consistently improves performance over the hindsight-prefix-only controls across both model scales and algorithms. The clearest evidence for the SRD loss itself comes from GRPO, where there is no teacher prefix, and SRD is the only modification (see W1(b) controls and the code-only experiments). We will explicitly clarify in Table 1 that the OPSD/RLSD +SRD variants use a hindsight prefix different from their corresponding baselines, making the two sources of improvement transparent.