acceptodds
Under review as a conference paper at ICLR 2027

Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

Abstract

Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: *can hindsight teach an agent what it could have anticipated before acting?* We introduce *prospective learning*, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to pp. Its advantage is especially pronounced when reward contrast is scarce: when –% of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In 2B setting, where % of groups are all-failure, the RLVR training ends up at success, while adding SRD reaches % under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Distillation, Prospective Learning, Reinforcement Learning with Verifiable Reward, Agent, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UmXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a known limitation of group-relative RLVR objectives: when all rollouts in a group receive the same reward, the group-relative advantage vanishes and the information in those trajectories is discarded. The authors propose prospective learning, which uses post-hoc experience to supervise predictions the agent makes before interaction, and instantiate it as Self-Retrospection Distillation (SRD). For each on-policy rollout, a hindsight-conditioned self-teacher (conditioned on the trajectory, the ground-truth answer, and error annotations) supervises a trajectory-blind student that predicts either the KNOWLEDGE a task requires (from successful rollouts) or the PITFALLS it may invite (from failed rollouts). The two are aligned via token-level JSD on the student's own foresight samples. SRD is added as an auxiliary loss (λ=0.01\lambda = 0.01) on top of GRPO, OPSD, and RLSD, and foresight is not generated at inference time.

Experiments on 10 benchmarks spanning math, code, search, and agentic tasks with Qwen3.5-4B/9B show improvements over the base objectives in most settings, with gains of up to 24.2 pp. In a code-only setting with dynamic sampling disabled, SRD lifts a 2B model's training success rate from 0.0% to 60.6%. The paper also includes a channel ablation (concluding that PITFALL alone suffices), a token-level analysis of how SRD redistributes the policy's outputs, a theoretical analysis of the reward-uniform regime, and a Fisher-metric decomposition of update directions.

Critical strengths

S1. A clear and potentially broadly applicable reframing of how hindsight is used. Prior on-policy self-distillation work mainly varies what the privileged context contains. This paper separates "what hindsight contains" from "what hindsight supervises," and redirects supervision from the action distribution to pre-interaction anticipation. This matters for my assessment because, if validated, it opens a design axis that is orthogonal to existing self-distillation methods and composable with them as a lightweight auxiliary loss.

S2. Broad and often large empirical gains under controlled conditions. SRD is evaluated across four domains, ten benchmarks, three base objectives, and two model scales, with identical data and prompt ordering across arms and several distribution shifts (stdin-to-functional code format, unseen ALFWorld environments, and the longer horizon of BrowseComp-Plus). Several gains are substantial: for example, +10.0 / +8.3 / +8.1 pp on the Math / Code / Search averages for 4B GRPO, and +17.2 pp on the Math average for 9B OPSD. This matters because it shows the approach is practically relevant and worth investigating, even though, as discussed below, the attribution of these gains remains unclear.

Critical weaknesses

W1. The source of SRD's performance gains is unclear: the gains are confounded with a change in the teacher's privileged information, and the mechanism by which foresight distillation improves task performance is not validated.

(a) Confounding with the change in the teacher's privileged information. In the OPSD and RLSD experiments, introducing SRD involves two simultaneous modifications: (1) replacing the privileged information used by the original teacher with structured hindsight, and (2) adding the SRD auxiliary distillation loss. Consequently, the improvements of OPSD+SRD and RLSD+SRD in Table 1 cannot be fully attributed to the proposed prospective learning. In fact, the ablation in Table 2 already shows that changing the teacher's privileged information alone yields substantial gains. On AIME26, 4B OPSD improves from 53.75% to 63.75% with the hindsight prefix alone, and only reaches 67.08% after additionally adding PITFALL SRD. Similarly, 9B OPSD improves from 65.42% to 77.50%, and reaches 84.17% with SRD. In these experiments, changing the teacher's input therefore accounts for the majority of the performance gain. Moreover, the OPSD baseline results in Table 2 differ markedly from those in Table 1, and no corresponding analysis is provided for RLSD. I recommend that the authors add OPSD and RLSD controls that replace only the hindsight prefix without the SRD loss, under exactly the same training configuration as the main experiments, so that the contribution of each modification can be clearly separated.

(b) Missing mechanistic validation. The core idea of SRD is to use post-hoc information to supervise the foresight generated by the model. However, at inference time, the model does not explicitly generate foresight; it directly performs the original task. This leaves a critical question insufficiently validated: why does distilling the foresight distribution improve the model's reasoning and decision-making on the original task? Although SRD shares parameters with the original policy, this does not imply that improved foresight prediction necessarily transfers to actual task performance. The existing analyses of token distribution shifts and parameter update directions provide mainly correlational evidence and are not sufficient to establish this causal relationship. The gains may instead stem from additional auxiliary training or regularization effects rather than from the proposed prospective learning mechanism. I suggest the following controls: (1) perform the same distillation using randomly shuffled hindsight or hindsight from other tasks, to rule out gains from a generic auxiliary loss; (2) perform SFT directly on the hindsight text, to test whether foresight distillation is actually necessary; and (3) condition the teacher only on the ground-truth answer without the interaction trajectory, to test whether the improvement truly depends on information extracted from interaction experience.

Why this is consequential. Taken together, (a) and (b) mean that none of the main results cleanly isolates prospective learning as the source of improvement: for OPSD and RLSD, most of the gain is attributable to the change in the teacher's input, while for GRPO, the mechanism by which the auxiliary loss improves task performance is untested. Without the controls above, the current experiments do not sufficiently support the paper's central claim or its distinct value relative to existing distillation methods.

W2. The core claim of "learning from reward-uniform groups" lacks sufficient empirical support.

The paper presents exploiting trajectory information from reward-uniform groups (all-correct or all-incorrect) as one of its core contributions, but the main experimental setup is somewhat disconnected from this claim. First, GRPO and RLSD in Table 1 still use dynamic sampling, which retains only groups with reward variance, so SRD does not actually exploit the filtered reward-uniform groups in the main results. Second, the main experiments use only the PITFALL channel by default, which cannot extract any SRD supervision from all-correct groups. This is not fully consistent with the introduction's emphasis that both successful and failed trajectories provide additional supervision.

Although the authors disable dynamic sampling in RQ3 to verify SRD's ability to learn from reward-uniform groups, this experiment covers only the code domain, and its most striking result (the 2B model improving from 0.0% to 60.6%) is reported only as training success rate, without generalization results on an independent test set. In addition, the 100% usable budget of SRD in Figure 4(a) mainly reflects that the auxiliary loss is computable on reward-uniform groups, and does not directly show that these trajectories all provide effective learning signals.

Therefore, while the current experiments show that SRD can continue to provide a training signal under sparse rewards, they are not sufficient to show that exploiting reward-uniform groups leads to higher sample efficiency under standard training configurations. I recommend comparing against standard GRPO with dynamic sampling under the same rollout or compute budget, and reporting results on independent test sets, to further validate this core claim.

Questions for authors
  • (W1a) Under the training configuration of Table 1, how much of the OPSD and RLSD improvement remains attributable to SRD after controlling for the change in the teacher's privileged prefix?

  • (W1b) What evidence distinguishes the benefit of foresight distillation from a generic auxiliary-training effect, given that foresight is not generated at inference time?

  • (W2) With reward-uniform groups retained, does SRD improve held-out performance over standard GRPO with dynamic sampling under a matched rollout or compute budget, rather than only improving training success?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UmXD - p1

We thank the reviewer for the detailed and constructive review and for the positive assessment: S1 SRD separates "what hindsight contains" from "what hindsight supervises" and composes with existing self-distillation methods; S2 the gains are broad under controlled conditions. We answer the attribution questions in W1 and W2 with new controls below.

W1(a). Confounding between the teacher's privileged information and the SRD loss. We agree that the original Table 2 did not separate them. We added the missing control: hindsight prefix, no SRD loss (new arm d), same configuration as the main runs, for OPSD and RLSD, 4B and 9B, math+code+search. The original trace prefix OPSD and hindsight prefix + SRD are the existing runs from Table 1.

In short: yes, most of the OPSD gain in Table 1 comes from the teacher prefix. However, in SRD the hindsight prefix can directly reuse retrospections already generated, avoiding a separate hindsight-extraction pass, and it is shorter than the original trace prefix, which makes training easier. And our SRD can still improve over the hindsight prefix OPSD/RLSD as shown in Table 2 (where we only use the math training data)

Held-out Average@8, % (five benchmarks evaluated on all arms). Trace prefix OPSD and hindsight prefix + SRD are the original runs at the end of training; Hindsight prefix OPSD, no SRD loss, is the new control.

modelalgoprefix / L_SRDAMOLCB v6OJHotpot2Wikiavg (5)
4BOPSDtrace / –8.849.414.058.052.436.5
hindsight / –15.559.917.471.557.444.3
hindsight / SRD14.857.317.773.363.045.2
RLSDtrace / –13.255.819.071.461.944.3
hindsight / –11.254.817.271.257.542.4
hindsight / SRD17.557.718.071.961.945.4
9BOPSDtrace / –12.053.620.162.455.940.8
hindsight / –16.563.717.476.867.048.3
hindsight / SRD20.864.320.678.671.451.1
RLSDtrace / –18.860.722.273.367.948.6
hindsight / –20.266.119.074.167.049.3
hindsight / SRD22.866.723.477.469.151.9

Answer to Q(W1a): Relative to the original trace prefix, replacing it with the hindsight prefix alone changes the five-benchmark average by +7.8 pp (4B OPSD), +7.5 pp (9B OPSD), −1.9 pp (4B RLSD), and +0.7 pp (9B RLSD). Adding the SRD loss on top further improves performance by +0.9, +2.8, +3.0, and +2.6 pp, respectively.

These results show that the prefix change accounts for most of the improvement in OPSD, while the SRD loss provides the larger contribution in RLSD. Importantly, the SRD loss consistently improves performance over the hindsight-prefix-only controls across both model scales and algorithms. The clearest evidence for the SRD loss itself comes from GRPO, where there is no teacher prefix, and SRD is the only modification (see W1(b) controls and the code-only experiments). We will explicitly clarify in Table 1 that the OPSD/RLSD +SRD variants use a hindsight prefix different from their corresponding baselines, making the two sources of improvement transparent.

AuthorsOpenReview ↗

Response to Reviewer UmXD - p2

W1(b). Mechanism: why does foresight distillation help if foresight is not generated at inference? We ran the three requested controls on GRPO+SRD, math+code+search, with the same budget. Benchmarks: AIME26 (math), LCB v6 functional (code), HotpotQA (search), Average@8 %.

controlwhat changes in SRD target4B: AIME26 / LCB / Hotpot (avg)9B: AIME26 / LCB / Hotpot (avg)
GRPOnone54.6 / 55.4 / 71.0 (60.3)70.4 / 67.3 / 75.4 (71.0)
SRD (arm e)–70.8 / 63.9 / 74.3 (69.7)81.7 / 67.5 / 75.6 (74.9)
(1) cross-domain hindsighthindsight from another domain (code←math, search←code, math←search)67.1 / 61.3 / 70.9 (66.4)79.6 / 66.5 / 74.9 (73.7)
(2) SFT on hindsight textCE on hindsight tokens instead of distillation60.8 / 57.7 / 72.5 (63.7)84.2 / 72.8 / 74.1 (77.0)
(3) ground-truth-only teacherno trajectory: answer / first test cases only52.9 / 54.2 / 71.5 (59.5)75.8 / 67.9 / 76.6 (73.4)
  • (1) Cross-domain hindsight keeps most of the gain. (4B: 66.4 vs. SRD 69.7 vs. GRPO 60.3; 9B: 73.7 vs. 74.9 vs. 71.0). Since one training batch mixes the domains, the signal the policy learns from is cross-domain anyway; the hindsight is therefore general rather than problem-specific, and the method is robust to this change. The benefit does not require hindsight that is specific to the same problem, so this control does not rule out that part of the gain comes from learning to produce foresight-style text (a structured pre-action prediction) rather than from the problem-specific content. Our claim is therefore the prospective-learning objective — predicting post-hoc information before acting, as a distillation target — not that the particular content of each hindsight is what transfers.
  • (2) SFT on the hindsight text: mixed. This control simply replaces the JSD target of SRD with an SFT loss on the same hindsight text. At 4B it is worse than distillation (63.7 vs. 69.7; AIME26 60.8 vs. 70.8) but better than GRPO. At 9B it is not worse: 77.0 vs. SRD 74.9 (AIME26 84.2 vs. 81.7, LCB 72.8 vs. 67.5, HotpotQA 74.1 vs. 75.6). We therefore do not claim that distillation beats SFT on the hindsight text; the evidence is that training on the hindsight text itself is a large part of the gain, and the advantage of the distillation form is not established at 9B.
  • (3) A ground-truth-only teacher: scale-dependent. Privileged information that is not grounded in the interaction can hurt reasoning [1]. At 4B it gives no gain (59.5 vs. GRPO 60.3 vs. SRD 69.7); it peaked mid-run (AIME26 64.2 at step 29) and fell back while the mean response length shrank from 3385 to 1650 tokens, the collapse trend we had predicted. At 9B it does help (73.4 vs. GRPO 71.0, SRD 74.9; AIME26 75.8 vs. 81.7, LCB 67.9 vs. 67.5, HotpotQA 76.6 vs. 75.6), i.e., most of the 9B gain on LCB and HotpotQA is reachable with the answer alone, and only on AIME26 is the trajectory-informed teacher clearly ahead. Response length at the last rollout: 1.7–2.4 k tokens, from 3.1–3.8 k at the start.

Within the prospective learning framework, every variant is supposed to improve on the base method, so SRD is one setting of a broader family.

Answer to Q(W1b): the controls show that the gain is not specific to the same-problem hindsight (cross-domain keeps most of it at both sizes) and that at 4B information from the trajectory matters (GT-only ≈ GRPO < SFT < SRD). At 9B the picture is weaker: SFT matches or exceeds SRD and GT-only recovers most of the gain, so the 9B data do not separate the mechanism from a generic training-signal effect. We will say this plainly and not claim more.

[1] Kim. et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

AuthorsOpenReview ↗

Response to Reviewer UmXD - p3

W2. Reward-uniform groups: evidence is thin. (a) Main experiments use dynamic sampling and only the pitfall channel. Correct, by design of the baselines; the code-only setting (RQ3) is where uniform groups are kept. We now extend RQ3 to code-only, both channels, 2B–9B, with held-out test sets (LCB v6 functional and OJBench), not only training success; the results are below. Held-out average@8 %, final step; "base" is the model before training (training and test data are the code domain only); dynamic sampling is not used for either arm, so GRPO has no gradient on reward-uniform groups while SRD still supplies one:

code-only, no dynamic samplingLCB v6: baseGRPOGRPO+SRDOJBench: baseGRPOGRPO+SRD
2B17.725.627.00.42.36.5
4B41.548.258.96.323.923.2
9B56.766.168.510.930.035.4

SRD is ahead of GRPO on LCB at all three sizes (+1.4, +10.7, +2.4) and on OJBench at 2B and 9B (+4.2, +5.4); the exception is OJBench at 4B (23.2 vs. 23.9). The clearest gain is 4B LCB.

(b) The learning signal on the 100% useable traces We therefore read the distillation signal (KL between teacher and student) on reward-uniform groups over training from the code-only runs: it is non-zero on all-wrong and on all-correct groups at 2B, 4B and 9B. The pitfall term (all-wrong groups) decays as the policy improves (mean per-token loss 0.0020 → 0.0002 at 2B, 0.0024 → 0.0001 at 4B, 0.0015 → 0.0004 at 9B) while the knowledge term (all-correct groups) grows (0.0000 → 0.0001 / 0.0011 / 0.0018), so the total stays non-zero (at 9B it rises, 0.0015 → 0.0023). A non-zero KL shows that SRD has a training signal where policy gradient has none; whether this signal helps is answered by the held-out results above. (c) Matched budget against dynamic sampling, on the hardest problems. We trained on the LCB v6 training problems that carry the benchmark's hard label only (248 of 422), where most sampled groups are all-wrong, comparing GRPO + dynamic sampling (DS) with GRPO + SRD (both channels, no DS), otherwise the same configuration. Held-out Average@8 %:

trained on hard problems onlyLCB v6 functionalOJBench
2B base15.50.5
2B GRPO + SRD (no DS)28.05.2
2B GRPO + DS18.91.3

What we can say from the runs that exist: (i) SRD keeps improving when almost every group is all-wrong: the 2B SRD run goes from 15.5 to 28.0 on LCB (+12.5) and from 0.5 to 5.2 on OJBench, at ≈3.7 min per step. (ii) The 2B DS arm struggles to obtain informative gradients on these hard problems: assembling its first batch of 32 groups required sampling and discarding ​2,278 reward-uniform (all-wrong) groups​, leaving fewer than 2% of sampled groups usable. This single update required ≈71 min (4,270 s) of rollout time, about ​19× the time of an SRD step​. At this observed rate, the same ≈3-hour training budget would allow only about 2–3 DS updates, compared with 51 SRD steps. This highlights the bottleneck of dynamic sampling on extremely difficult problems: obtaining reward contrast becomes prohibitively expensive, whereas SRD can learn directly from the abundant all-wrong trajectories.

We thank the reviewer again for their valuable feedback. We hope the additional experiments and analyses have helped address the remaining concerns and strengthen the empirical evidence for our approach. We would be glad to discuss any further questions.

Reviewer_mwdJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose SRD, a post hoc method to strengthen the underlying LLM's capability by forecasting what may be a pitfall and what may be a useful case for policy generation.

Critical strengths

the performance gain across all evaluations

Critical weaknesses

After reading the paper, I have several concerns that I would like to discuss with the authors:

  • limited novelty (also the term hindsight replay buffer is confusing). In the original hindsight replay buffer paper, the authors' motivation is to tackle the exploration bottleneck of an RL agent by enabling it to learn from previously unsuccessful experiences. In contrast, SRD primarily focuses on better exploiting experiences by distilling the existing knowledge/pitfalls into the policy rather than encouraging the policy to discover new behaviors or solutions. Which, to me, is an exploitation problem about how to leverage existing knowledge to supervise-train a policy, potentially achieved by test-time scaling rather than directly increasing the policy's exploration capability. (Also see the second and fourth points. )

  • While SRD may improve policy performance, it focuses on better exploiting existing experiences. From this perspective, how does SRD differ from test-time scaling? In test-time scaling, improved performance can be achieved by increasing the number and diversity of sampled trajectories. So, at inference time, we scale the inference to include multiple solutions in parallel and discard the unsuccessful trajectories while keeping the working ones. Given sufficient exploration coverage and computational budget, it remains unclear whether the proposed method offers advantages beyond those achievable through test-time scaling.

  • A related concern, in my opinion, would be the limited exploration capability of the resulting policy. Since SRD distills knowledge from previously observed trajectories, the resulting policy may be constrained by the coverage and diversity of the existing experience buffer.

  • Unclear distinction from prior work. The proposed method is conceptually related to prior work on learning from agent experience, including Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024), and generalized distillation with privileged information (Lopez-Paz et al., 2016). These methods have explored how feedback or information available after an interaction can improve subsequent decisions. While SRD introduces a different auxiliary prediction target, it remains unclear whether its advantages are attributable specifically to learning prospective information, rather than to generic hindsight-conditioned distillation. More direct comparisons with these methods would help substantiate the claimed novelty.

Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners.

Shinn N, Cassano F, Gopinath A, et al. Reflexion: Language agents with verbal reinforcement learning[J]. Advances in neural information processing systems, 2023, 36: 8634-8652.

Lopez-Paz D, Bottou L, Schölkopf B, et al. Unifying distillation and privileged information[J]. arXiv preprint arXiv:1511.03643, 2015.

Questions for authors

see weakness and I am happy to discuss with the authors

6 comments
AuthorsOpenReview ↗

Response to Reviewer mwdJ - p1

We thank the reviewer for the thoughtful comments and for noting the performance gain across all evaluations. We welcome the discussion — the questions about how our proposed prospective learning and SRD relate to exploration, test-time scaling, and memory-style methods are exactly the ones that help us sharpen what the contribution is, and we hope the replies below, with new experiments, give a clearer picture of the method.

W1/W3. Novelty, the term "hindsight replay buffer", and exploitation vs. exploration. On the name. The reviewer contrasts SRD with the original hindsight replay buffer paper (Hindsight Experience Replay, which targets exploration). There is no replay buffer in SRD in that sense or in the sense of memory-based methods: hindsight is produced on the fly from the current training group's rollouts, used for that update, and discarded. Unlike Reflexion/ExpeL-style methods, in our setting, nothing external has to be stored and retrieved at test time — the experience and the hindsight are internalized into the policy, and inference is the plain policy with no memory.

On exploitation vs. exploration. We state this explicitly: yes, SRD exploits existing hindsight/hindsights alongside the training to optimize the distribution of the policy's foresight (its pre-action prediction). The SRD formula in the paper is one instance of a more general prospective learning objective — learning to predict, before acting, what post-hoc information would say — and variants that encourage exploration more strongly are possible within the same framework (we test three further instantiations, all on GRPO+SRD with math+code+search, average@8 % over AIME26 / LCB v6 / HotpotQA, 4B | 9B: GRPO 60.3 | 71.0, GRPO+SRD 69.7 | 74.9; hindsight taken from another domain 66.4 | 73.7; SFT on the hindsight text instead of distillation 63.7 | 77.0; a teacher that sees only the ground-truth answer 59.5 | 73.4 — so variants of the same prospective-learning family keep most or all of the gain, except the ground-truth-only teacher at 4B). What SRD adds is a new learning signal: a distillation target available on reward-uniform groups where the policy-gradient signal is exactly zero; it is additional supervision from the same rollouts which help to improve the exploitation from the samples used to be wasted, and sampling strategy itself is untouched.

  • Does exploiting hindsight hurt the policy's coverage/exploration capability (pass@k at large k)? We read the pass@k logged at every eval step of the original GRPO runs on held-out benchmarks (math and search; LCB is left out because the training set of those runs was not the final one). If SRD only sharpened what the base policy already did, pass@1 would rise while pass@8 stayed flat or fell. Measured (9B math+search, GRPO vs. GRPO+SRD; bold = SRD ahead of GRPO at step 9):
9B, held-outmetricGRPO (a): step 0 → 9 → 19 → 49GRPO+SRD (e): step 0 → 9 → 19 → 49
math AIME24pass@119.6 → 19.2 → 57.9 → 76.322.1 → 59.6 → 83.8 → 84.1
pass@843.3 → 43.3 → 83.3 → 93.346.7 → 86.7 → 93.3 → 90.0
math AIME25pass@118.3 → 23.7 → 69.2 → 77.924.2 → 70.8 → 83.3 → 83.3
pass@850.0 → 60.0 → 96.7 → 96.743.3 → 100 → 100 → 96.7
search HotpotQApass@167.9 → 73.8 → 75.4 → 75.472.3 → 76.4 → 76.6 → 75.6
pass@888.0 → 88.0 → 90.0 → 89.089.0 → 89.0 → 91.0 → 88.0
search 2Wikipass@159.6 → 63.0 → 72.5 → 71.660.0 → 64.4 → 70.1 → 71.1
pass@878.0 → 82.0 → 90.0 → 86.080.0 → 79.0 → 85.0 → 89.0

The same check at 4B (math and search, pass@1 / pass@8 in %, steps 0 → 9 → 19 → 49):

4B, held-outmetricGRPO (a)GRPO+SRD (e)
math AIME24pass@124.2 → 22.5 → 37.5 → 69.625.0 → 43.3 → 72.5 → 76.7
pass@853.3 → 60.0 → 80.0 → 90.063.3 → 73.3 → 96.7 → 93.3
math AIME25pass@125.8 → 27.5 → 40.4 → 64.228.3 → 47.1 → 70.0 → 75.4
pass@846.7 → 60.0 → 70.0 → 86.760.0 → 86.7 → 96.7 → 100.0
search HotpotQApass@158.1 → 69.8 → 74.6 → 71.061.9 → 71.8 → 73.1 → 74.3
pass@883.0 → 86.0 → 86.0 → 83.084.0 → 85.0 → 87.0 → 89.0
search 2Wikipass@148.2 → 58.2 → 66.8 → 60.648.9 → 57.5 → 61.5 → 72.0
pass@871.0 → 75.0 → 90.0 → 84.070.0 → 73.0 → 80.0 → 92.0

At 4B, pass@8 under SRD is at or above GRPO's at the final step on all four benchmarks (AIME24 93.3 vs. 90.0, AIME25 100 vs. 86.7, HotpotQA 89 vs. 83, 2Wiki 92 vs. 84), and well above it at step 9 on math.

AuthorsOpenReview ↗

Response to Reviewer mwdJ - p2

The same check on the agentic benchmarks (held-out ALFWorld-OOD and WebShop; ALFWorld pass@1 / pass@8 and WebShop score / pass@8 in %, steps 0 → 9 → 19 → 49):

agentic, held-outmetricGRPOGRPO+SRD
4B ALFWorldpass@139.2 → 42.4 → 46.6 → 67.837.8 → *45.9 → 55.4 → 71.0
pass@871.0 → 75.0 → 69.0 → 83.067.0 → 80.0 → 83.0 → 90.0
4B WebShopscore75.3 → 75.8 → 70.8 → 77.276.7 → 76.1 → 75.1 → 83.1
pass@853.0 → 56.0 → 68.0 → 82.052.0 → 73.0 → 79.0 → 87.0
9B ALFWorldpass@148.6 → 53.5 → 60.8 → 77.149.2 → 57.6 → 66.8 → 78.3
pass@882.0 → 84.0 → 81.0 → 91.080.0 → 87.0 → 85.0 → 85.0
9B WebShopscore86.8 → 87.0 → 87.0 → 78.187.9 → 86.8 → 77.9 → 84.2
pass@857.0 → 57.0 → 75.0 → 88.061.0 → 78.0 → 80.0 → 88.0

What this shows. SRD rises much faster, and the gain is also visible at large k. At step 9, 9B SRD is already at pass@1 / pass@8 of 59.6 / 86.7 on AIME24 and 70.8 / 100.0 on AIME25, where GRPO is at 19.2 / 43.3 and 23.7 / 60.0; at 4B it is 43.3 / 73.3 and 47.1 / 86.7 vs. 22.5 / 60.0 and 27.5 / 60.0. At step 49, 4B pass@8 is at or above GRPO's on all four math/search tasks (AIME24 93.3 vs. 90.0, AIME25 100.0 vs. 86.7, HotpotQA 89.0 vs. 83.0, 2Wiki 92.0 vs. 84.0). At 9B, GRPO catches up by step 49; the exceptions are ALFWorld pass@8 (85.0 vs. 91.0) and WebShop score, where SRD has no early lead.

AuthorsOpenReview ↗

Response to Reviewer mwdJ - p3

W2. How does SRD differ from test-time scaling?

Test-time scaling methods such as best-of-N and Reflexion need a reliable verifier or feedback signal at test time, which is often unavailable in practice; SRD uses the verifier only during training and none at inference. We compare with best-of-N sampling at 2B, where the untrained model is weak. Setup: LiveCodeBench v6 functional (63 problems) and its 20 problems labelled hard (the benchmark's own difficulty label, no model-dependent selection); n = 64 samples per problem, same tool-calling rollout and grader as in our training evaluations; best-of-k is computed with an oracle verifier (unbiased pass@k estimator), which is generous to best-of-N, since SRD needs no verifier at test time. Models: the untrained Qwen3.5-2B ("base") and the same model after GRPO+SRD (code-only training, both channels, no dynamic sampling, 51 steps ≈ 2.9 h on 8 GPUs; ≈ 205 s/step from our timing analysis). Both were evaluated with the same script. Pass@k in %:

2B, LCB v6 functionalk=1248163264
all 63 problems: base (best-of-k)18.225.231.637.342.246.952.4
all 63 problems: GRPO+SRD27.233.538.442.045.148.050.8
hard 20 problems: base (best-of-k)1.52.74.56.89.111.915.0
hard 20 problems: GRPO+SRD5.05.55.96.88.511.315.0

Measured cost of the best-of-N side on the same 8 GPUs: 64 samples for all 63 problems took ≈ 16 min (4,032 generations) and ≈ 7.5 min for the 20 hard problems; the responses are long (median at the 16k-token cap, i.e. many samples are truncated).

What this shows: after training, SRD's pass@1 is much higher than the base model's (27.2 vs. 18.2 on the full set; 5.0 vs. 1.5 on the hard subset), and matching it with best-of-N takes 2–4 samples per problem (4–8 on hard) plus a verifier. At k = 64 SRD is no better than the base (50.8 vs. 52.4; both solve 15.0 on hard), consistent with the exploitation effect discussed above. Most real tasks need no discovery of new solutions and aim for a higher pass@1, which is what SRD improves at a one-off training cost of 2.9 h. The best-of-N cost is generation time only, one seed, same test distribution; SRD's training pays off after roughly 14,000 test problems (full set) or 4,000 (hard set).

Reviewer_1vVBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Self-Retrospection Distillation (SRD), training agents by utilizing hindsight information from agents’ own rollouts to supervise pre-interaction foresight prediction. Before acting, the policy predicts the pitfalls and foresights of a task. It is then distilled toward the same model's predictions when that model is also given hindsight from its own completed rollouts. Added to GRPO, OPSD and RLSD, SRD shows improved results on math, code, search and agentic benchmarks.

Critical strengths
  • Overall, this is a well-motivated, interesting and well-conducted study. My understanding: SRD uses an auxiliary training task to instill a "think before acting" preference, which is a neat way to reuse post-hoc experience.
  • The empirical work is substantial. It compares several base objectives (GRPO, OPSD, RLSD) across a range of representative benchmarks, although some may be relatively easy imo. The GRPO+SRD comparison is particularly clean.
  • The appendix contains informative analyses.
  • The method is composable and can be plugged into different post-training algorithms.
Critical weaknesses
  • A hand-designed and fairly specific behavior. At a high level, SRD trains a particular behavior: anticipating foresights before acting. I don't see it as a clearly universally desirable behavior, and it's not guaranteed to help on every task.

  • Denser feedback as an clear alternative. The motivating problem, no learning signal from reward-uniform groups, arises largely because rewards are binarized? This is done even in environments where partial credit might have been readily available. Denser or partial-credit rewards are an established alternative line of work, and to me seem a more general way to approach the problem. I suggest adding GRPO with partial-credit rewards as a baseline. This seems easy to do in several of the evaluated environments.

  • Some issues in the experiments/presentation or confusing points (for a study of this size I think these are largely acceptable). (a) For OPSD, much of the improvement seems to come from changing the teacher prefix, not from the SRD loss. (b) Foresight is not used at inference. I would like to see two experiments:

    • the untrained model prompted to write foresight before solving, as a zero-training baseline;
    • the SRD-trained model with its own foresight prepended at inference.
  • The improvements on the agentic tasks (ALFWorld, WebShop) are mixed: large in some settings but small or even negative in others.This may be partly because these benchmarks are relatively simple (e.g., WebShop).

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1vVB - p1

We thank the reviewer for the careful and generous reading, and for describing SRD as a well-motivated, interesting and well-conducted study. We are glad the reviewer sees SRD as a neat way to reuse post-hoc experience by instilling a "think before acting" preference, that the GRPO+SRD comparison is clean, that the appendix analyses are informative, and that the method is composable with different post-training algorithms. We also appreciate the concrete, easy-to-run requests (partial-credit baseline, zero-training foresight prompting, foresight at inference); all are now done, and the results are below.

W1. SRD is a hand-designed, specific behavior; not universally desirable. The performance of SRD can be effected by the prompt quality, and we tested how much of the effect depends on the wording of our hindsight prompts. We wrote a second set of hindsight prompts from scratch as follows and re-ran GRPO+SRD on math+code+search with all other settings as in Table 1.

The two prompts that differ most from the original are the pitfall-extraction system prompt (hindsight, from a failed attempt) and the foresight-prediction system prompt (what the student writes before acting); the knowledge-channel prompt is rewritten in the same way (headings [Insight]/[Instance]).

Alternative pitfall-extraction system prompt:

A solver's attempt at the task below ended in a wrong answer, and the reference answer is provided. Your job is diagnosis, not solving: work out where the attempt went off track and record what a future solver should keep in mind. Write 1-3 entries, each laid out exactly as:

[Mistake]
<what exactly went wrong in the attempt, stated concretely>
[Principle]
<the general fact or method that prevents this kind of mistake>
[Illustration]
<a very small worked instance of the principle (tiny numbers or a short snippet); it must not be this task's answer>

Concrete example of the expected style:
[Mistake]
Lost the factor 2 while expanding the square.
[Principle]
a^2+b^2 = (a+b)^2 - 2ab.
[Illustration]
With u+v=6 and uv=4: u^2+v^2 = 36 - 8 = 28.

Too vague (avoid): 'Take care with algebra.'

Requirements:

- Start every entry with the literal headings [Mistake], [Principle], [Illustration].
- Be specific; the principle must carry over to other problems.
- Do not reveal the reference answer and do not write out a full solution.
- Return only the entries, with no extra text.

Alternative foresight-prediction system prompt (student, sees only the problem):

You will see a problem and nothing else (no attempt, no answer). Anticipate the mistakes a solver is most likely to make on it and write them down as 1-3 entries, each laid out exactly as:

[Mistake]
<the specific trap a solver is likely to step into here>
[Principle]
<the general fact or method that avoids it>
[Illustration]
<a tiny worked instance of the principle, not this problem's answer>

Requirements:

- Start every entry with the literal headings [Mistake], [Principle], [Illustration]; be specific rather than generic.
- Do not solve the problem or outline its full method; list only the traps and how to avoid each.
- Never state a final answer.
- Return only the entries, nothing else.
GRPO+SRD (arm e)AIME24AIME26AMOLCB v6OJHotpot2Wikiavg (7)Δ\Delta Improv.
4B GRPO (a, no SRD)69.654.614.555.418.371.060.649.1--
4B GRPO+SRD, original prompt76.770.821.263.926.574.372.057.9+8.8
4B GRPO+SRD, alternative prompt69.667.114.860.522.170.466.453.0+3.9
9B GRPO (a, no SRD)76.270.418.567.327.675.471.658.1--
9B GRPO+SRD, original prompt84.281.728.067.531.875.671.162.8+4.7
9B GRPO+SRD, alternative prompt81.375.020.368.330.877.171.060.5+2.4

(Average@8 %, final step. GRPO and original-prompt rows are the original Table 1 runs. Δ\Delta Improv. is the improvement over GRPO)

The alternative prompt is somewhat less effective than the one we tuned for the paper, but it still improves clearly over standard GRPO: the seven-benchmark average rises by +3.9 at 4B and +2.4 at 9B (original prompt: +8.8 and +4.7). So the gain does not depend on one particular wording, although its size does, and we will say so in the limitations.

AuthorsOpenReview ↗

Response to Reviewer 1vVB - p2

W2. Whether denser / partial-credit rewards are a more general alternative; add GRPO with partial credit.

Reward changes only for code (fraction of tests passed, capped below 1 for incomplete solutions) and search (LLM judge score 0–4, scaled; truncation → 0); math stays binary. Evaluation is unchanged (binary pass, where agents should pass all the test units in coding benchmarks and QA should be the correct answer, Average@8). We also report the share of reward-uniform groups which is dropped due to dynamic sampling under each reward. (GRPO and GRPO+SRD rows are the original Table 1 runs; bold are the best in each model-benchmark pair. )

AIME26 / LCB v6 / HotpotQA (avg)dropped groups
4B GRPO (binary)54.6 / 55.4 / 71.0 (60.3)40%
4B GRPO-PR (partial credit)67.9 / 52.8 / 70.1 (63.6)54%
4B GRPO+SRD (binary)70.8 / 63.9 / 74.3 (69.7)–
9B GRPO (binary)70.4 / 67.3 / 75.4 (71.0)56%
9B GRPO-PR (partial credit)77.5 / 67.9 / 73.4 (72.9)50%
9B GRPO+SRD (binary)81.7 / 67.5 / 75.6 (74.9)–

GRPO+SRD (binary reward) is ahead of GRPO-PR on average (+6.1 at 4B, +2.0 at 9B), so the denser reward does not close the gap. Partial credit also does not remove the reward-uniform groups: their share stays large (9B: 56% binary vs. 50% PR; 4B: 40% vs. 54%), so many groups still carry no policy-gradient signal, and this is where SRD still supplies one. Within the baselines, GRPO-PR vs. binary GRPO shows no consistent gain on LCB/HotpotQA (−2.6 / −0.9 at 4B, +0.6 / −2.0 at 9B); the AIME26 columns differ by +13.3 (4B) and +7.1 (9B) even though the math reward is binary in both runs, so we read differences of that size as run-to-run variation.

W3(a). For OPSD, a portion of improvement comes from the teacher prefix. Agreed, and measured (table below). Most of the OPSD gain in Table 1 comes from the teacher prefix. However, in SRD the hindsight prefix can directly reuse retrospections already generated, avoiding a separate hindsight-extraction pass, and it is shorter than the original trace prefix, which makes training easier. We will mark in Table 1 of the paper that the OPSD/RLSD rows with SRD use a different prefix from their base rows.

Average@8, %. Trace prefix OPSD, no SRD and hindsight prefix + SRD are the Table 1 runs at the end of training; hindsight prefix OPSD, no SRD loss is the new control.

modelalgoprefix / L_SRDAMOLCB v6OJHotpot2Wikiavg (5)
4BOPSDtrace / –8.849.414.058.052.436.5
hindsight / –15.559.917.471.557.444.3
hindsight / SRD14.857.317.773.363.045.2
RLSDtrace / –13.255.819.071.461.944.3
hindsight / –11.254.817.271.257.542.4
hindsight / SRD17.557.718.071.961.945.4
9BOPSDtrace / –12.053.620.162.455.940.8
hindsight / –16.563.717.476.867.048.3
hindsight / SRD20.864.320.678.671.451.1
RLSDtrace / –18.860.722.273.367.948.6
hindsight / –20.266.119.074.167.049.3
hindsight / SRD22.866.723.477.469.151.9

Relative to the trace prefix, using the hindsight prefix alone changes the five-benchmark average by +7.8 pp (4B OPSD), −1.9 pp (4B RLSD), +7.5 pp (9B OPSD), and +0.7 pp (9B RLSD). Adding the SRD loss on top further improves performance by +0.9, +3.0, +2.8, and +2.6 pp, respectively.

These results show that the teacher prefix accounts for most of the improvement in OPSD, whereas the additional SRD loss provides more substantial gains in RLSD. Importantly, the SRD loss consistently improves performance over the corresponding hindsight-prefix-only controls across both model scales.

AuthorsOpenReview ↗

Response to Reviewer 1vVB - p3

W3(b). Foresight is not used at inference. We ran both requested experiments: (1) the untrained model prompted to write a foresight first, and (2) the SRD-trained model with its own foresight prepended.

Seven benchmarks (AIME24, AIME26, AMO, LCB v6, OJBench, HotpotQA, 2Wiki), Average@8 %. "+ foresight" = the model first writes a pitfall-style foresight for the problem (same model, prompt only), which is placed in the prompt before it solves. The presented results are from checkpoints of OPSD and RLSD in Table 1.

4B modelavg (7), no foresightavg (7), + foresightΔAIME26 / LCB / HotpotQA, no foresight → + foresight
Vanilla (untrained)31.026.4−4.626.7 / 42.9 / 63.4 → 22.9 / 42.5 / 51.0
GRPO48.645.0−3.658.3 / 56.2 / 70.2 → 48.3 / 56.5 / 65.8
GRPO+SRD58.055.8−2.271.2 / 63.5 / 74.0 → 70.8 / 65.9 / 67.9
OPSD41.542.8+1.352.1 / 48.6 / 54.9 → 55.8 / 49.0 / 58.5
OPSD + SRD50.646.1−4.564.2 / 58.7 / 74.6 → 59.2 / 57.1 / 63.1
RLSD49.647.4−2.363.7 / 55.6 / 68.8 → 59.2 / 54.2 / 65.8
RLSD + SRD52.747.7−5.172.9 / 57.1 / 72.6 → 63.7 / 59.5 / 61.6
9B modelavg (7), no foresightavg (7), + foresightΔAIME26 / LCB / HotpotQA, no foresight → + foresight
Vanilla (untrained)34.239.5+5.314.6 / 57.9 / 70.8 → 38.8 / 67.1 / 63.5
GRPO58.155.4−2.772.1 / 68.1 / 73.8 → 68.3 / 63.7 / 71.5
GRPO+SRD63.760.6−3.181.2 / 72.0 / 77.4 → 80.4 / 69.8 / 71.0
OPSD45.747.2+1.555.4 / 55.4 / 62.0 → 54.2 / 55.6 / 61.6
OPSD + SRD59.055.3−3.780.0 / 64.7 / 75.8 → 70.0 / 67.7 / 72.4
RLSD56.354.4−1.968.3 / 60.5 / 73.8 → 69.6 / 59.1 / 70.8
RLSD + SRD59.656.8−2.877.5 / 68.7 / 78.1 → 75.4 / 64.3 / 71.8

Across all 14 checkpoints (7 per size), adding the foresight lowers the 7-benchmark average in 11 and raises it in 3 (4B OPSD +1.3, 9B OPSD +1.5, and 9B Vanilla +5.3). The drop is mostly on the search tasks: averaged over the seven checkpoints of a size, HotpotQA changes by −6.4 (4B) / −4.1 (9B) and 2Wiki by −5.9 / −4.9, while LCB and AMO are essentially unchanged (|mean| ≤ 2.7) and AIME is mixed (4B −4.3; 9B ≈ 0). The one clear gain is the untrained 9B model (+5.3: AIME24 22.5 → 40.8, AIME26 14.6 → 38.8, LCB 57.9 → 67.1, with search −7 / −11); its base accuracy on AIME is low enough that any extra scaffold may help.

(1) For the untrained model, prompting for a foresight first helps at 9B (math/code) but hurts at 4B, so it gives no consistent substitute for SRD training. (2) For the SRD-trained checkpoints, prepending the model's own foresight lowers the average at both sizes (−2.2 to −5.1), so the SRD gain is in the weights and the foresight is not needed at inference. This agrees with what we claim. We do not claim the foresight is useless as text; only that adding it to the prompt does not give a consistent improvement and wastes more test-time inference.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their thoughtful evaluations and constructive feedback. We appreciate Reviewer 1vVB for recognizing SRD as a well-motivated and carefully conducted study, highlighting its clean GRPO comparison, informative analyses, and composability. We thank Reviewer mwdJ for acknowledging the performance improvements across evaluations. We particularly appreciate Reviewer UmXD for recognizing our central conceptual contribution—separating what hindsight contains from ​what hindsight supervises​—as a potentially broadly applicable design axis, and for highlighting the broad empirical gains under controlled conditions.

To address the reviewers' concerns, we conducted additional experiments on: (i) attribution and mechanism, (ii) alternative approaches and test-time scaling, and (iii) learning from reward-uniform groups​. The key results are summarized below, with complete experimental details in our individual responses.

Concern (Reviewer)New experimentsKey findings
a) Teacher prefix vs. SRD loss(1vVB W3a; UmXD W1a)OPSD/RLSD prefix controls at 4B/9BPrefix explains most OPSD gains; SRD adds +0.9–3.0 pp beyond prefix-only controls.
b) Mechanism validation(UmXD W1b; mwdJ W4)Cross-domain hindsight, SFT, GT-only teacherCross-domain retains most gains; SFT is competitive at 9B, clarifying the limits of mechanism attribution.
c) Partial-credit rewards(1vVB W2)GRPO with denser code/search rewardsSRD remains ahead by +6.1/+2.0 pp at 4B/9B.
d) Foresight at inference(1vVB W3b; UmXD W1b)14 checkpoint comparisons with/without foresightExplicit foresight lowers performance in 11/14 comparisons; unnecessary for SRD inference.
e) Test-time scaling / prior methods(mwdJ W1–W4)Best-of-N, ExpeL-style, Reflexion-styleSRD improves small-k performance without inference-time verification; Reflexion remains complementary.
f) Reward-uniform learning(UmXD W2)Code-only, no-DS, 2B–9B, held-out LCB/OJBenchSRD improves LCB at all scales and OJBench at 2B/9B; non-zero auxiliary signals on uniform groups.
g) Hardest-problem setting(UmXD W2)2B GRPO+DS vs. GRPO+SRDDS discarded 2,278 all-wrong groups and required ≈19× SRD for one step.
h) Prompt sensitivity(1vVB W1)Alternative hindsight/foresight promptsGains persist (+3.9/+2.4 pp at 4B/9B), though smaller than with the original prompts.

Overall, these experiments strengthen the evidence for SRD as a composable auxiliary learning objective, clarify the contributions of the teacher prefix and SRD loss, and demonstrate its ability to learn from reward-uniform trajectories. We also acknowledge the limits of mechanism attribution and test-time scaling comparisons.

In the revision, we will ​clarify the teacher-prefix differences (a) and dynamic-sampling/channel configurations (f) in Table 1​, incorporate the new reward-uniform learning results (f) into RQ3, and include the remaining experimental analyses in the appendix as appropriate. We will also expand the limitations to acknowledge that SRD uses hand-designed prompts and is sensitive to their wording, while noting that the gains persist with independently rewritten prompts preserving the same structured foresight format (h).

We thank all reviewers again for their constructive feedback and welcome any further questions or discussion.

Dear reviewers,

Kindly go through and acknowledge the author responses to your review comments and follow up with any questions or comments you may have.

Thanks,

AC

AuthorsOpenReview ↗

Paper revision: summary of changes

We thank all reviewers again. We have uploaded a revised manuscript that incorporates the additional experiments from our rebuttal. The main changes are:

  1. Teacher prefix vs. SRD loss (UmXD W1a; 1vVB W3a). Table 1 now marks the OPSD+SRD and RLSD+SRD rows (†) to indicate that they use our generated hindsight as the teacher prefix. The matched hindsight-prefix-only controls for OPSD and RLSD at 4B and 9B are in Appendix D.2 (Table 7).
  2. Mechanism controls (UmXD W1b; mwdJ W4). Cross-domain hindsight, SFT on the hindsight text and a ground-truth-only teacher are in Appendix D.7 (Table 11).
  3. Learning from reward-uniform groups (UmXD W2). Section 5 (RQ3) now reports held-out results of code-only training without dynamic sampling and the hard-problem comparison against GRPO with dynamic sampling (Table 3). The 2B–9B held-out table is in Appendix D.3 (Table 8).
  4. Exploitation vs. exploration (mwdJ W1/W3). RQ5 now compares pass@1 and pass@8 during training (Figure 5); the per-benchmark curves are in Appendix D.4 (Figure 6).
  5. Foresight at inference (1vVB W3b). The 14-checkpoint comparison with and without explicit foresight is in Appendix D.6 (Table 10).
  6. Prompt sensitivity (1vVB W1). The alternative prompts are in Appendix B.3 and the corresponding results in Appendix D.5 (Table 9). The Reproducibility Statement now notes that SRD relies on handcrafted hindsight and foresight prompts.
  7. Partial-credit rewards (1vVB W2). GRPO with partial-credit rewards, including the share of reward-uniform groups dropped by dynamic sampling, is in Appendix D.8 (Table 12).
  8. Comparison with test-time methods (mwdJ W2/W4). ExpeL-style and Reflexion-style baselines, alone and on top of SRD, are in Appendix D.9 (Table 13); we state that Reflexion with an oracle verifier is a strong baseline that SRD is complementary to, not a replacement for.
  9. Related work (mwdJ W4). We now discuss Reflexion, ExpeL, GEPA, test-time scaling, and generalized distillation with privileged information (Lopez-Paz et al., 2016).
  10. Metric clarification. Appendix C.5 states that WebShop reports the mean task score (0–100).

We would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.