- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PR-OPD, which combines GRPO with self-distillation by aligning the hidden representations with those of the same policy conditioned on self-generated hindsight skills, and reports performance gains on ALFWorld and WebShop using two LLM backbones.
Critical strengths
-
PR-OPD integrates representation-level self-distillation with self-generated hindsight, without requiring an external teacher or additional inference-time computation.
-
The reported gains across two benchmarks and two backbones provide preliminary evidence of its utility, particularly the 7.8-point WebShop accuracy improvement over Skill-SD with the 3B backbone.
Critical weaknesses
-
PR-OPD differs from the main baselines in both its GRPO warm start and its use of trajectory-specific, self-generated skills. To attribute the gains specifically to hidden-state distillation, a more controlled comparison is needed—for example, starting PR-OPD, SDAR, and Skill-SD from the same GRPO checkpoint at step 100 and providing hindsight skills generated by the same procedure, while retaining each method’s distillation objective. This would help distinguish the benefit of representation-level supervision from that of the training schedule and privileged information.
-
The empirical evidence is insufficient to establish reliable and sustained improvements: Each configuration uses a single training seed, and some gains over the strongest baseline are only 0.8–2.3 percentage points on 128 evaluation episodes. All runs end at step 150, leaving only 30–50 updates with representation supervision. Independent training runs, comparative task-performance curves, and extended training are needed to determine whether the reported gains are reproducible and persist beyond the selected endpoint.
Questions for authors
-
Clarification of the token-OPD ablation: What exact loss, gradient treatment, and distillation coefficient were used? Were importance weighting or gating applied? These details are necessary to assess whether its degradation reflects a limitation of output-level supervision or the particular implementation.
-
Evidence isolating the proposed contribution: Have you compared PR-OPD with SDAR and Skill-SD starting from the same GRPO checkpoint and using the same hindsight-generation procedure? If available, these results would help establish whether hidden-state distillation itself accounts for the gains.
-
Reliability and training dynamics: Do the gains persist across independent training seeds and beyond step 150? Please provide any available comparative task-performance curves, particularly after distillation begins. Evaluation-seed repetitions and alignment-loss curves do not resolve uncertainty about training variability or sustained performance.
7 comments
Q2:Have you compared PR-OPD with SDAR and Skill-SD starting from the same GRPO checkpoint and using the same hindsight-generation procedure?
Answer: Yes. We ran this comparison on Qwen3-1.7B in both ALFWorld and WebShop. Within each training seed, SDAR, Skill-SD, and PR-OPD started from the same GRPO checkpoint at step 100 and trained for 50 additional updates. All three used the same procedure to generate one hindsight skill per completed trajectory. Each method collected its own on-policy rollouts, so the trajectories and generated skills could differ. SDAR and Skill-SD retained their respective output-level distillation objectives, while PR-OPD used projected hidden-state alignment. Continued GRPO served as the reward-only baseline.
At step 150, we evaluated each policy once on each of the same 140 ALFWorld seen tasks and 500 WebShop test instructions. The student acted without skills at temperature 0.4. Values are mean ± sample SD across two independent training seeds. ALFWorld All weights each task equally. WebShop Acc measures full success, and Score gives partial credit on a 0–100 scale.
Qwen3-1.7B at step 150
| Method | Pick (35) | Look (13) | Clean (27) | Heat (16) | Cool (25) | Pick2 (24) | ALFWorld All | WebShop Acc | WebShop Score |
|---|---|---|---|---|---|---|---|---|---|
| Continued GRPO | 48.57±0.00 | 42.31±5.44 | 18.52±0.00 | 37.50±17.68 | 28.00±0.00 | 27.08±2.95 | 33.57±1.01 | 38.10±1.56 | 63.45±7.00 |
| SDAR from GRPO@100 | 51.43±0.00 | 42.31±5.44 | 27.78±2.62 | 37.50±0.00 | 32.00±5.66 | 18.75±8.84 | 35.36±0.51 | 35.50±4.95 | 57.80±5.94 |
| Skill-SD from GRPO@100 | 51.43±4.04 | 50.00±5.44 | 35.19±18.33 | 53.13±4.42 | 28.00±5.66 | 16.67±11.79 | 38.21±2.53 | 41.20±5.09 | 67.80±1.13 |
| PR-OPD from GRPO@100 | 54.29±4.04 | 46.15±10.88 | 35.19±2.62 | 43.75±17.68 | 32.00±0.00 | 27.08±2.95 | 40.00±2.02 | 44.40±1.98 | 72.15±3.32 |
Skill-SD was the stronger matched output-level baseline on all three aggregate metrics in both seeds. Relative to Skill-SD, PR-OPD solved 2 and 3 additional ALFWorld tasks and 5 and 27 additional WebShop goals in seeds 1 and 2, respectively. The mean paired gains were 1.79 percentage points in ALFWorld success, 3.20 points in WebShop accuracy, and 4.35 points in WebShop score. The advantage remained under a shared starting checkpoint and hindsight-generation procedure, supporting a contribution from representation-level supervision in this setting. The ALFWorld margin was small in both seeds, and the magnitude of the WebShop gain varied across seeds.
Q3:Do the gains persist across independent training seeds and beyond step 150? Please provide any available comparative task-performance curves, particularly after distillation begins.
Answer: Yes. At step 150, PR-OPD exceeded continued GRPO on both ALFWorld success and WebShop accuracy in each of two independent training seeds, on both Qwen3-1.7B and Qwen2.5-3B. We extended both methods to step 200 on Qwen3-1.7B. PR-OPD retained a positive margin on both benchmarks in both seeds. Full-split gains over continued GRPO. For each seed, we subtracted continued GRPO from PR-OPD at the same checkpoint on the same 140 ALFWorld tasks or 500 WebShop goals. Values are mean ± sample SD of those paired differences, in percentage points.
| Step | Δ ALFWorld success (pp) | Δ WebShop accuracy (pp) |
|---|---|---|
| 150 | +6.43±1.01 | +6.30±0.42 |
| 200 | +3.57±1.01 | +5.60±6.51 |
Qwen3-1.7B results after 200 training steps. Within each seed, continued GRPO and PR-OPD started from the same GRPO checkpoint at step 100 and trained for 100 additional updates. We evaluated both step-200 checkpoints on all 140 ALFWorld seen tasks and 500 WebShop test instructions. Values are mean ± sample SD across two training seeds.
| Method | Step | Pick | Look | Clean | Heat | Cool | Pick2 | ALFWorld All | WebShop Acc | WebShop Score |
|---|---|---|---|---|---|---|---|---|---|---|
| Continued GRPO | 200 | 61.43±6.06 | 46.15±0.00 | 29.63±0.00 | 50.00±8.84 | 30.00±8.49 | 39.58±8.84 | 43.21±0.51 | 45.00±6.79 | 70.00±1.41 |
| PR-OPD | 200 | 60.00±4.04 | 42.31±5.44 | 38.89±7.86 | 50.00±0.00 | 48.00±0.00 | 35.42±8.84 | 46.79±0.51 | 50.60±0.28 | 75.85±0.07 |
At step 200, PR-OPD solved 4 and 6 more ALFWorld tasks than continued GRPO in seeds 1 and 2, respectively, for a mean gain of 3.57 percentage points. On WebShop, the accuracy gains were 1.0 and 10.2 percentage points, averaging 5.60 points. Mean WebShop score increased by 5.85 points.
The following are the reported checkpoint accuracies for the Qwen3-1.7B run whose full-split endpoints correspond to seed 2. Both methods start from the same GRPO checkpoint at step 100.
| WebShop accuracy (%) | Step 100 | Step 125 | Step 150 | Step 175 | Step 200 |
|---|---|---|---|---|---|
| Continued GRPO | 27.4 | 32.2 | 37.0 | 38.6 | 40.2 |
| PR-OPD | 27.4 | 35.8 | 43.0 | 48.8 | 50.4 |
| Difference (pp) | 0.0 | +3.6 | +6.0 | +10.2 | +10.2 |
PR-OPD led at the first measured checkpoint after distillation began and at every subsequent checkpoint shown. The full-split evaluations show that the size of this lead evolved differently across seeds. On WebShop, the margin widened from +6.0 to +10.2 points in seed 2 and narrowed from +6.6 to +1.0 point in seed 1. On ALFWorld, the step-200 margins were +2.86 and +4.29 points. Thus, the advantage over continued GRPO persisted through step 200 in both Qwen3-1.7B runs, with substantial seed-to-seed variation on WebShop.
We sincerely appreciate your acknowledgment and constructive comments to improve our work.
Answer:What we changed. We use token-OPD as shorthand for the ablation in which we replace PR-OPD’s hidden-state alignment loss with an output-level distillation loss on the student’s sampled tokens. Both branches start from the same GRPO@100 checkpoint and use the same hindsight-generation procedure. After each rollout, the current policy writes one skill per completed trajectory. The same policy then acts as the teacher, with that skill prepended to each turn, and re-scores the student’s sampled response. The student sees only the original context. This setup lets us compare two ways of using the same privileged context. For the output-level objective, we use SDAR’s sigmoid gate.
Loss and gradient. Let (M) denote the response-token mask, and let (t_k) and (s_k) denote the teacher and student log-probabilities of the sampled token (y_k). We compute the teacher log-probabilities once per training step, before the update. With (\mathrm{sg}) denoting stop-gradient, the implemented loss is
\mathcal L_{\mathrm{tok}} =\frac{1}{|M|}\sum_{k\in M}g_k\bigl(\mathrm{sg}(t_k)-s_k\bigr).$$ We minimize $(\mathcal L_{\mathrm{GRPO}}+0.01,\mathcal L_{\mathrm{tok}})$. The distillation coefficient matches the alignment weight in PR-OPD. The GRPO term retains PPO clipping, reference-policy KL regularization (coefficient 0.01), and entropy regularization (0.001). The PPO importance ratio applies only to the GRPO term. The distillation term uses the soft gate above and includes every response token from both successful and failed trajectories. **Why token-OPD may underperform GRPO.** The clearest explanation suggested by the gradient is self-imitation on failed trajectories. Since both the teacher score and the gate are detached, token (k) contributes the following gradient to the total loss: $$-\frac{0.01}{|M|},g_k,\nabla_\theta\log\pi_\theta(y_k).$$ Under gradient descent, this term always favors the sampled token. The teacher controls the strength of that update: even when it assigns the token a lower probability than the student does, the gate remains positive and the token still receives a smaller reinforcing update. Consider an incorrect action in a failed trajectory with negative group-relative advantage. GRPO pushes that action’s probability down while its policy gradient is active, whereas token-OPD pushes its sampled tokens up. This conflict offers a concrete explanation for the drop from 66.4% with continued GRPO to 57.8% with token-OPD on 3B ALFWorld. The gradient establishes the direction of the auxiliary update.