acceptodds
Under review as a conference paper at ICLR 2027

PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

Abstract

Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. **Invisible Advantage**: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. **Much to Align**: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://anonymous.4open.science/r/PR-OPD-A666.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic reinforcement learning, on-policy distillation, representation distillation, self-evolving agents, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_hABzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PR-OPD, which combines GRPO with self-distillation by aligning the hidden representations with those of the same policy conditioned on self-generated hindsight skills, and reports performance gains on ALFWorld and WebShop using two LLM backbones.

Critical strengths
  • PR-OPD integrates representation-level self-distillation with self-generated hindsight, without requiring an external teacher or additional inference-time computation.

  • The reported gains across two benchmarks and two backbones provide preliminary evidence of its utility, particularly the 7.8-point WebShop accuracy improvement over Skill-SD with the 3B backbone.

Critical weaknesses
  • PR-OPD differs from the main baselines in both its GRPO warm start and its use of trajectory-specific, self-generated skills. To attribute the gains specifically to hidden-state distillation, a more controlled comparison is needed—for example, starting PR-OPD, SDAR, and Skill-SD from the same GRPO checkpoint at step 100 and providing hindsight skills generated by the same procedure, while retaining each method’s distillation objective. This would help distinguish the benefit of representation-level supervision from that of the training schedule and privileged information.

  • The empirical evidence is insufficient to establish reliable and sustained improvements: Each configuration uses a single training seed, and some gains over the strongest baseline are only 0.8–2.3 percentage points on 128 evaluation episodes. All runs end at step 150, leaving only 30–50 updates with representation supervision. Independent training runs, comparative task-performance curves, and extended training are needed to determine whether the reported gains are reproducible and persist beyond the selected endpoint.

Questions for authors
  1. Clarification of the token-OPD ablation: What exact loss, gradient treatment, and distillation coefficient were used? Were importance weighting or gating applied? These details are necessary to assess whether its degradation reflects a limitation of output-level supervision or the particular implementation.

  2. Evidence isolating the proposed contribution: Have you compared PR-OPD with SDAR and Skill-SD starting from the same GRPO checkpoint and using the same hindsight-generation procedure? If available, these results would help establish whether hidden-state distillation itself accounts for the gains.

  3. Reliability and training dynamics: Do the gains persist across independent training seeds and beyond step 150? Please provide any available comparative task-performance curves, particularly after distillation begins. Evaluation-seed repetitions and alignment-loss curves do not resolve uncertainty about training variability or sustained performance.

7 comments

We sincerely appreciate your acknowledgment and constructive comments to improve our work.

Q1: Clarification of the token-OPD ablation: What exact loss, gradient treatment, and distillation coefficient were used? Were importance weighting or gating applied?

Answer:

What we changed. We use token-OPD as shorthand for the ablation in which we replace PR-OPD’s hidden-state alignment loss with an output-level distillation loss on the student’s sampled tokens. Both branches start from the same GRPO@100 checkpoint and use the same hindsight-generation procedure. After each rollout, the current policy writes one skill per completed trajectory. The same policy then acts as the teacher, with that skill prepended to each turn, and re-scores the student’s sampled response. The student sees only the original context. This setup lets us compare two ways of using the same privileged context. For the output-level objective, we use SDAR’s sigmoid gate.

Loss and gradient. Let (M) denote the response-token mask, and let (t_k) and (s_k) denote the teacher and student log-probabilities of the sampled token (y_k). We compute the teacher log-probabilities once per training step, before the update. With (\mathrm{sg}) denoting stop-gradient, the implemented loss is

\mathcal L_{\mathrm{tok}} =\frac{1}{|M|}\sum_{k\in M}g_k\bigl(\mathrm{sg}(t_k)-s_k\bigr).$$ We minimize $(\mathcal L_{\mathrm{GRPO}}+0.01,\mathcal L_{\mathrm{tok}})$. The distillation coefficient matches the alignment weight in PR-OPD. The GRPO term retains PPO clipping, reference-policy KL regularization (coefficient 0.01), and entropy regularization (0.001). The PPO importance ratio applies only to the GRPO term. The distillation term uses the soft gate above and includes every response token from both successful and failed trajectories. **Why token-OPD may underperform GRPO.** The clearest explanation suggested by the gradient is self-imitation on failed trajectories. Since both the teacher score and the gate are detached, token (k) contributes the following gradient to the total loss: $$-\frac{0.01}{|M|},g_k,\nabla_\theta\log\pi_\theta(y_k).$$ Under gradient descent, this term always favors the sampled token. The teacher controls the strength of that update: even when it assigns the token a lower probability than the student does, the gate remains positive and the token still receives a smaller reinforcing update. Consider an incorrect action in a failed trajectory with negative group-relative advantage. GRPO pushes that action’s probability down while its policy gradient is active, whereas token-OPD pushes its sampled tokens up. This conflict offers a concrete explanation for the drop from 66.4% with continued GRPO to 57.8% with token-OPD on 3B ALFWorld. The gradient establishes the direction of the auxiliary update.

Q2: Have you compared PR-OPD with SDAR and Skill-SD starting from the same GRPO checkpoint and using the same hindsight-generation procedure?

Answer: Yes. We ran this comparison on Qwen3-1.7B in both ALFWorld and WebShop. Within each training seed, SDAR, Skill-SD, and PR-OPD started from the same GRPO checkpoint at step 100 and trained for 50 additional updates. All three used the same procedure to generate one hindsight skill per completed trajectory. Each method collected its own on-policy rollouts, so the trajectories and generated skills could differ. SDAR and Skill-SD retained their respective output-level distillation objectives, while PR-OPD used projected hidden-state alignment. Continued GRPO served as the reward-only baseline.

At step 150, we evaluated each policy once on each of the same 140 ALFWorld seen tasks and 500 WebShop test instructions. The student acted without skills at temperature 0.4. Values are mean ± sample SD across two independent training seeds. ALFWorld All weights each task equally. WebShop Acc measures full success, and Score gives partial credit on a 0–100 scale.

Qwen3-1.7B at step 150

MethodPick (35)Look (13)Clean (27)Heat (16)Cool (25)Pick2 (24)ALFWorld AllWebShop AccWebShop Score
Continued GRPO48.57±0.0042.31±5.4418.52±0.0037.50±17.6828.00±0.0027.08±2.9533.57±1.0138.10±1.5663.45±7.00
SDAR from GRPO@10051.43±0.0042.31±5.4427.78±2.6237.50±0.0032.00±5.6618.75±8.8435.36±0.5135.50±4.9557.80±5.94
Skill-SD from GRPO@10051.43±4.0450.00±5.4435.19±18.3353.13±4.4228.00±5.6616.67±11.7938.21±2.5341.20±5.0967.80±1.13
PR-OPD from GRPO@10054.29±4.0446.15±10.8835.19±2.6243.75±17.6832.00±0.0027.08±2.9540.00±2.0244.40±1.9872.15±3.32

Skill-SD was the stronger matched output-level baseline on all three aggregate metrics in both seeds. Relative to Skill-SD, PR-OPD solved 2 and 3 additional ALFWorld tasks and 5 and 27 additional WebShop goals in seeds 1 and 2, respectively. The mean paired gains were 1.79 percentage points in ALFWorld success, 3.20 points in WebShop accuracy, and 4.35 points in WebShop score. The advantage remained under a shared starting checkpoint and hindsight-generation procedure, supporting a contribution from representation-level supervision in this setting. The ALFWorld margin was small in both seeds, and the magnitude of the WebShop gain varied across seeds.

Q3: Do the gains persist across independent training seeds and beyond step 150? Please provide any available comparative task-performance curves, particularly after distillation begins.

Answer: Yes. At step 150, PR-OPD exceeded continued GRPO on both ALFWorld success and WebShop accuracy in each of two independent training seeds, on both Qwen3-1.7B and Qwen2.5-3B. We extended both methods to step 200 on Qwen3-1.7B. PR-OPD retained a positive margin on both benchmarks in both seeds. Full-split gains over continued GRPO. For each seed, we subtracted continued GRPO from PR-OPD at the same checkpoint on the same 140 ALFWorld tasks or 500 WebShop goals. Values are mean ± sample SD of those paired differences, in percentage points.

StepΔ ALFWorld success (pp)Δ WebShop accuracy (pp)
150+6.43±1.01+6.30±0.42
200+3.57±1.01+5.60±6.51

Qwen3-1.7B results after 200 training steps. Within each seed, continued GRPO and PR-OPD started from the same GRPO checkpoint at step 100 and trained for 100 additional updates. We evaluated both step-200 checkpoints on all 140 ALFWorld seen tasks and 500 WebShop test instructions. Values are mean ± sample SD across two training seeds.

MethodStepPickLookCleanHeatCoolPick2ALFWorld AllWebShop AccWebShop Score
Continued GRPO20061.43±6.0646.15±0.0029.63±0.0050.00±8.8430.00±8.4939.58±8.8443.21±0.5145.00±6.7970.00±1.41
PR-OPD20060.00±4.0442.31±5.4438.89±7.8650.00±0.0048.00±0.0035.42±8.8446.79±0.5150.60±0.2875.85±0.07

At step 200, PR-OPD solved 4 and 6 more ALFWorld tasks than continued GRPO in seeds 1 and 2, respectively, for a mean gain of 3.57 percentage points. On WebShop, the accuracy gains were 1.0 and 10.2 percentage points, averaging 5.60 points. Mean WebShop score increased by 5.85 points.

The following are the reported checkpoint accuracies for the Qwen3-1.7B run whose full-split endpoints correspond to seed 2. Both methods start from the same GRPO checkpoint at step 100.

WebShop accuracy (%)Step 100Step 125Step 150Step 175Step 200
Continued GRPO27.432.237.038.640.2
PR-OPD27.435.843.048.850.4
Difference (pp)0.0+3.6+6.0+10.2+10.2

PR-OPD led at the first measured checkpoint after distillation began and at every subsequent checkpoint shown. The full-split evaluations show that the size of this lead evolved differently across seeds. On WebShop, the margin widened from +6.0 to +10.2 points in seed 2 and narrowed from +6.6 to +1.0 point in seed 1. On ALFWorld, the step-200 margins were +2.86 and +4.29 points. Thus, the advantage over continued GRPO persisted through step 200 in both Qwen3-1.7B runs, with substantial seed-to-seed variation on WebShop.

Reviewer_gVCjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an approach (PR-OPD) to train agents using hindsight information. Skills are generated from completed trajectories and their rewards. These skills are then provided to a teacher that shares the same weights as the student but has access to the additional hindsight information (privileged). The proposed method aligns the student's hidden representations with those of the teacher. The key motivation underlying this approach is that hindsight information may be reflected more strongly in the teacher's hidden representations than in its output probabilities. Experiments on ALFWorld and WebShop show improvements over GRPO and other baselines.

Critical strengths
  • I found the idea of exploiting hindsight information to provide a better learning signal than sparse terminal rewards interesting.
  • The observation that hindsight information can more strongly affect the hidden representations without producing similarly large changes in its output probabilities is interesting. This motivates alignment with the hidden states representation rather than conventional output-level distillation.
  • The empirical results are promising, showing improvements over GRPO and several baselines. Also, the experimental study is solid having several ablations examining the contribution of representation alignment, the projector, and the warm-start stage.
Critical weaknesses
  • The main claim of the paper is that the induced hidden representations (when skills are prepended) contain useful information that can improve the student's decision-making. The teacher and the student have identical weights, and process the same responses and actions, so they only differ in the input (with and without the skills). It is natural then to have different hidden representations but this does not necessarily mean that the differences would lead to improved decisions. Why should aligning these representations lead to better policies? The ablation studies provide useful empirical evidence that representation alignment can outperform output-level distillation, but a more convincing explanation of what task-relevant information is being transferred, and why this alignment is effective are needed. Basically the approach lacks clear justification and explainability.

  • The motivation is to overcome the limited feedback provided by sparse terminal rewards. However, there are other well-established approaches to improve and speed up learning in sparse reward settings. For example, reward machines and automata-based approaches use explicit task specifications or learned task structures to guide RL. These methods can also learn and exploit task rules, rather than relying on predefined specifications. It would be important to discuss relationship to these approaches and clarify the advantages and limitations of learning from implicit representations rather than explicit or learned task structures.

Questions for authors

Please refer to my comments above concerning a clear justification and explainability of the hidden representation alignment, as well as relationship to existing approaches addressing the sparse rewards challenge.

3 comments

We sincerely appreciate your acknowledgment and constructive comments to improve our work.

Q1: Why should aligning these representations lead to better policies?

Answer: We agree that a change in hidden states alone does not establish that the change is useful for decision-making. Our rationale is that hindsight can help the teacher identify which parts of an interaction history matter for the next decision. For example, having already reached and searched a location should influence where the agent goes next. During training, the student sees the ordinary interaction history and receives an auxiliary target from the same policy interpreting that history with hindsight. Across trajectories, alignment can encourage the student to recover distinctions that are predictable from its own observations, while GRPO continues to train action selection from task reward.

Two findings support this interpretation. First, Fig. 1(b) tests whether the representation is sensitive to which skill is supplied. Both conditions contain a hindsight skill, but only one uses the skill generated for the current trajectory. Middle-layer hidden states distinguish these conditions with 65.4% accuracy at 3B and 79.4% at 1.7B, compared with approximately 50–52% for the output features tested in the paper. Thus, the hidden states carry information about the relationship between the skill and the particular trajectory, beyond the mere presence of additional text.

Second, we observe a concrete change in the student’s decisions when no skill is available at evaluation. In ALFWorld, the fraction of all actions that attempted to navigate to the agent’s current location fell from 5.84% under GRPO to 2.33% under PR-OPD on the 140 seen games, and from 4.82% to 2.03% on the 134 unseen games. The matched 3B ablation provides complementary evidence: with the same starting checkpoint and hindsight-generation procedure, hidden-state alignment achieved 75.7% and 72.1% success across the two seeds, compared with 67.9% and 70.0% for token-OPD.

Taken together, these results support a specific interpretation: trajectory-specific hindsight provides a representation-level training signal, and the resulting policy makes better use of an observable fact—its current location—in at least one identifiable class of decisions. The probe does not identify a particular hidden feature as the cause of that behavioral change. We will make this distinction explicit in the revision.

Q2: Advantages and limitations of learning from implicit representations rather than explicit or learned task structures

Answer: We agree that reward machines and subgoal automata are closely related to our motivation. Both can exploit structure that a terminal reward alone leaves unspecified. This structure can also be learned from experience. Toro Icarte et al. (2019) learn reward machines from interaction, and Furelos-Blanco et al. (2020) induce subgoal automata from traces of observable events. An advantage of these approaches is that task progress can be inspected explicitly. For an ALFWorld instruction to heat an object and then place it, for example, an automaton can represent whether the heating step has been completed and which event advances the task.

PR-OPD uses hindsight to provide auxiliary supervision during policy training. After a rollout, the policy writes a natural-language skill and reprocesses the trajectory with that skill in context. The resulting hidden states provide training targets for the student, while GRPO continues to use the original environment reward. This is useful when relevant details vary across instructions, as with product attributes and options in WebShop. The same procedure can use those details directly from text without first defining an event vocabulary or fitting a task automaton. At inference, the student acts from the ordinary interaction history.

The trade-off is interpretability and control. We can inspect the generated skill, but the student’s hidden states do not tell us which rule it retained or when it will apply that rule. An inaccurate hindsight skill can also supply a misleading training target. Explicit task structure is therefore attractive when subgoals recur and their completion can be reliably detected. PR-OPD targets settings where trajectory-specific language is readily available and useful events are harder to formalize. We will add this distinction to the paper and clarify that our experiments leave the relative sample efficiency of the two approaches open.

[1] Toro Icarte, et. al. (2019). Learning Reward Machines for Partially Observable Reinforcement Learning. Advances in Neural Information Processing Systems, 32 (NeurIPS 2019).

[2] Furelos-Blanco, D., Law, M., Russo, A., Broda, K., & Jonsson, A. (2020). Induction of Subgoal Automata for Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 3890–3897. DOI 10.1609/aaai.v34i04.5802.

Reviewer_gVCjOpenReview ↗

Re:Official Comment by Authors

Thank you for the response. The new evidence strengthens the empirical justification for the proposed approach, although the underlying mechanism remains somewhat unclear. Also, thanks for the clarification regarding the relationship to reward machines and learned automata.

Many thanks for your positive feedback and for the time and effort you have devoted to improving our work. We sincerely appreciate your constructive comments and valuable suggestions.

Reviewer_MsUiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper looks at privileged self-distillation for LLM agents, the idea is the same model, given a helpful "skill" in its prompt, acts as the teacher for itself without the skill. The authors first observe that the skill makes the teacher act better but barely changes its token probabilities, while it changes the hidden states a lot and in a skill specific way. Based on this, they propose PR-OPD: after a GRPO warm start, the model writes a hindsight skill for each of its own trajectories, and its hidden states, passed through a small projector, are pulled toward the skill conditioned hidden states at every layer on top of the GRPO loss. On ALFWorld and WebShop with two small Qwen models, PR-OPD gets the best numbers against GRPO and token-level self-distillation baselines.

Critical strengths
  1. The diagnostic analysis are the most convincing part of the paper. The per-layer probes and CKA has good ablation, and they show that the skill changes the hidden states in a skill specific way, not just because the prompt got longer.
  2. The method is simple and adds nothing at inference. The policy writes its own skills, so unlike the baseline it needs no hand-written skill library and no separate teacher.
Critical weaknesses

I don't think the performance claims are statistically significant. Each method is trained only once, and the numbers in Table 1 and 2 are single evaluation of the final checkpoint on 128 episodes, where one standard error at 70% success is already about 4 points. On ALFWorld, PR-OPD beats the best baseline by 2 episodes out of 128 at 3B and by 1 episode at 1.7B. The ablations look like noise as well. Without projector, the alignment loss is tiny and stays within 0.01 of its starting value, while ALFWorld moves by +6.3 points at 3B and -7.1 at 1.7B. I find it hard to believe such a small term moves success this much in both directions. To me it looks more like run-to-run noise. I would need several training seeds and the full evaluation splits before I could trust the ranking in Table 1.

Questions for authors
  1. Could you rerun Table 1 and 2 ablations with several training seeds and report the confidence interval?
  2. How exactly is a "change" defined for tokens and hidden states in Figure 1, and do the numbers hold under other thresholds? Do the output probes still stay near chance with a richer output feature, for example, the top-2048 log-probabilities?
  3. Without the projector, the alignment loss is tiny and nearly flat, but ALFWorld success moves by +6.3 points at 3B and −7.1 at 1.7B. With the projector, how much do the policy's own hidden states actually move toward the teacher's, rather than the projector just absorbing the difference?
4 comments

We sincerely appreciate your acknowledgment and constructive comments to improve our work.

Q1:Could you rerun Table 1 and 2 ablations with several training seeds and report the confidence interval?

Answer: We rerun and compare the main baseline methods on both models and tasks with two different training seeds. Results are reported as mean ± sample standard deviation across two independent training seeds.

BackboneMethodPickLookCleanHeatCoolPick2ALFWorld AllWebShop AccWebShop Score
Qwen3-1.7BGRPO48.6±0.042.3±5.418.5±0.037.5±17.728.0±0.027.1±2.933.6±1.038.1±1.663.5±7.0
Qwen3-1.7BSDAR47.1±10.138.5±10.925.9±0.046.9±4.420.0±0.025.0±5.933.6±2.037.7±2.763.3±8.3
Qwen3-1.7BSkill-SD55.7±2.038.5±0.037.0±5.243.8±8.826.0±2.814.6±2.936.8±1.537.8±4.565.7±6.0
Qwen3-1.7BPR-OPD54.3±4.046.2±10.935.2±2.643.8±17.732.0±0.027.1±2.940.0±2.044.4±2.072.2±3.3
Qwen2.5-3BGRPO92.9±6.150.0±5.481.5±0.046.9±13.360.0±22.658.3±5.969.6±2.561.1±0.179.3±2.2
Qwen2.5-3BSDAR90.0±2.061.5±0.072.2±13.140.6±13.356.0±17.047.9±14.765.0±4.061.9±3.880.1±1.8
Qwen2.5-3BSkill-SD92.9±6.161.5±0.083.3±2.656.3±0.066.0±2.847.9±8.871.4±1.063.0±0.875.2±0.4
Qwen2.5-3BPR-OPD88.6±4.069.2±10.981.5±5.268.8±8.866.0±8.558.3±11.873.9±2.567.0±0.681.4±2.3

Qwen2.5-3B on AlfWorld:

MethodSeed 1 (%)Seed 2 (%)Mean (%)Δ vs. GRPO [95% CI]
GRPO71.4 (100/140)67.9 (95/140)69.64—
Token-OPD67.9 (95/140)70.0 (98/140)68.93−0.71 [−37.02, +35.59]
PR-OPD w/o projector77.9 (109/140)73.6 (103/140)75.71+6.07 [+1.53, +10.61]
PR-OPD w/ projector75.7 (106/140)72.1 (101/140)73.93+4.29 [+4.29, +4.29]†

Due to resource constraints, we used eight A100 GPUs for the rebuttal runs (the original experiments used eight B200 GPUs). The GRPO warm-up checkpoints achieved lower success rates in some model–task settings, but all methods in each matched comparison started from the same checkpoint for a given seed, and PR-OPD continued to outperform GRPO on both benchmarks with both backbones across the two seeds. We evaluated each step-150 checkpoint once on each of the same full 140 ALFWorld seen games and 500 WebShop test instructions. PR-OPD exceeded Skill-SD, the strongest baseline by two-seed mean in Table 1, on both primary metrics in both seeds. The paired 95% t intervals across seeds are reported above (df = 1). The most consistent margin was on 3B WebShop (+4.00 points, 95% CI [+1.46, +6.54]).

Q2: How exactly is a "change" defined for tokens and hidden states in Figure 1, and do the numbers hold under other thresholds? Do the output probes still stay near chance with a richer output feature, for example, the top-2048 log-probabilities?

Answer: In Fig. 1(a,b), we hold the ALFWorld GRPO@100 checkpoint and the sampled response fixed. At each response-token position, we compare two passes through the same model, one with the trajectory’s hindsight skill and one without it. The output shift is the absolute change in the log-probability of the sampled token. The hidden shift is the cosine distance between the two hidden states at that position, averaged across layers. We count a position as changed when its shift exceeds the 95th percentile for the same statistic under length-matched unrelated filler. This gives each channel a nominal 5% change rate under filler. The output cutoffs are 0.2496 and 0.5506 nats for 3B and 1.7B, respectively. The corresponding layer-mean hidden cutoffs are 0.00249 and 0.00626.

Threshold sensitivity. The table reports the percentage of positions that exceed each cutoff when the model receives its own trajectory’s skill. The layer-mean column reproduces the statistic in Fig. 1. We also report the last layer separately.

BackboneFiller percentileSampled-token outputLayer-mean hidden (Fig. 1)Last-layer hidden
3B80th38.3%99.1%96.4%
3B95th23.7%81.3%66.7%
3B99th14.9%31.5%13.0%
1.7B80th29.7%100.0%98.8%
1.7B95th16.4%93.3%89.8%
1.7B99th10.3%49.7%61.6%

The ordering in Fig. 1 holds at all three cutoffs for both backbones. At the strictest cutoff, the 3B last-layer result shows a small reversal (13.0% hidden vs. 14.9% output), while the original layer-mean statistic remains higher (31.5%). On action tokens specifically, the original 95th-percentile criterion flags 64.0% of positions through hidden states versus 6.6% through outputs at 3B, and 89.5% versus 1.8% at 1.7B. A threshold-free analysis gives the same overall pattern: the AUROC for distinguishing own-skill shifts from filler shifts is 0.97 for layer-mean hidden states versus 0.60 for outputs at 3B, and 0.98 versus 0.55 at 1.7B.

Richer output probes. The richer output features we tested also stayed close to chance. On the own-skill versus other-trajectory-skill task, expanding the probe input from the sampled token to the log-probabilities of the top 64 candidates yielded 51.9% accuracy at 3B and 51.8% at 1.7B (chance: 50%). We also tested a 256-dimensional random projection of the full vocabulary probability vector, which includes contributions from tokens outside the top 64. This probe reached 51.0% and 50.4%, respectively. On the same classification task, middle-layer hidden states reached 65.4% and 79.4%. Thus, the advantage of hidden states persisted with both richer output features we evaluated.

Q3: Without the projector, the alignment loss is tiny and nearly flat, but ALFWorld success moves by +6.3 points at 3B and −7.1 at 1.7B. With the projector, how much do the policy's own hidden states actually move toward the teacher's, rather than the projector just absorbing the difference?

Answer: With the projector, the policy’s unprojected hidden states became modestly closer to the skill-conditioned teacher’s states. Raw cosine similarity increased approximately 6%. Across the eight runs, the remaining raw alignment gap decreased by 8% on average. On 3B ALFWorld specifically, it decreased by 27% and 10% in the two seeds.

We measured the same raw gap, 2−2cos⁡(hS,hT)2-2\cos(h_S,h_T), in both training arms. Here hSh_S is the student state before projection and hTh_T is the skill-conditioned teacher state. With the projector, this gap is logged as a diagnostic. Without the projector, it is the alignment objective. For each run, we compared the mean gap over steps 101–105 with its mean over steps 146–150. Negative values indicate that the gap narrowed.

BackboneEnvironmentWith projector, seed 1 / seed 2Without projector, seed 1 / seed 2
1.7BALFWorld+2% / −8%0% / −8%
1.7BWebShop−10% / −13%+27% / −10%
3BALFWorld−27% / −10%−20% / −24%
3BWebShop−9% / +9%−15% / +24%
Mean across eight runs−8%−3%

Over all 50 updates, the raw-gap slope was negative in 7 of 8 runs with the projector and 6 of 8 without it. The projected loss fell by approximately 80% over the same period. Thus, the large reduction in projected loss was accompanied by a smaller, measurable reduction before projection. Because the teacher shares the evolving policy weights and each update uses fresh rollouts, these measurements quantify the observed student–teacher gap. They cannot isolate how much of the projected loss decrease came from updating the projector’s parameters.

The original +6.3 and −7.1 points in Table 2 compare direct alignment against GRPO at 3B and 1.7B, respectively. For the effect of the projector itself, the direct 3B comparison was 72.7% without versus 71.1% with the projector. In the full-split rerun, adding the projector reduced success by 3/140 and 2/140 games across the two seeds, a mean change of −1.79 percentage points (paired 95% CI [−6.32, +2.75]). The raw-state measurements establish that alignment also occurs before projection.

Many thanks for your positive feedback and for the time and effort you have devoted to improving our work. We sincerely appreciate your constructive comments and valuable suggestions.

Reviewer_tGiSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors propose a new method for RL fine tuning of LLMs with privileged information called PR-OPD. In this setting the "privileged information" comes in the form of hindsight skills chosen by the LLM being trained after observing the outcome of its proposed solution. Standard approaches in this space, to the best of my knowledge, then use the same LLM policy + the hindsight skill as a teacher for the student policy. The authors note though that the teacher's output tokens don't change by much when using the skill despite their internal activations changing significantly. Therefore they hypothesize that a lot of important information for the student is lost by learning on tokens, so they propose learning directly on the teacher's activations instead. They evaluate their method on two smaller QWen models in the ALFWorld and WebShop environments.

Critical strengths
  • The paper's core motivation, that teacher tokens don't provide enough information while their hidden activations do, appears solid and is interesting.

  • The results appear promising given these smaller scale LLMs.

  • It's nice that the approach uses the same model weights for the teacher and student, from a memory efficiency perspective.

Critical weaknesses
  • I feel the paper is missing some analysis of instances where the approach doesn't out-perform the baselines. Specifically it would be interesting to know why the method isn't strictly dominant on ALFWorld (out performs baselines on all tasks), while it appears to perform much better on WebShop.

  • Similarly, it would be interesting to have some discussion as to why the method seems to degrade in performance a bit after adding the projection component of the method for ALFWorld with the larger QWen model. Is there a reason behind this, or is it just something that is roughly captured by uncertainty/variance?

  • There is a some work in standard RL domains showing that privileged information can actually be harmful for learning as the teacher's access to extra information can bias the learning objective of the student [1]. In PR-OPD's setting the asymmetry comes not from partial observability but from the teacher having access to a kind of "cheat sheet" for solving the task. It would be interesting to know if a similar kind of bias can show up here, resulting in the student policy hallucinating skills that it doesn't have or that are irrelevant to the task.

[1] Baisero, Amato "Unbiased asymmetric reinforcement learning under partial observability"

Questions for authors
  • Can you explain why you think PR-OPD falls short in some ALFWorld tasks and why the projection approach doesn't work as well for the larger QWen model?

  • Have you observed any skill hallucination in the student as a result of this approach?

4 comments
AuthorsOpenReview ↗

Rerun of Table 1/2

We rerun and compare the main baseline methods on both models and tasks with two different training seeds. Results are reported as mean ± sample standard deviation across two independent training seeds.

BackboneMethodPickLookCleanHeatCoolPick2ALFWorld AllWebShop AccWebShop Score
Qwen3-1.7BGRPO48.6±0.042.3±5.418.5±0.037.5±17.728.0±0.027.1±2.933.6±1.038.1±1.663.5±7.0
Qwen3-1.7BSDAR47.1±10.138.5±10.925.9±0.046.9±4.420.0±0.025.0±5.933.6±2.037.7±2.763.3±8.3
Qwen3-1.7BSkill-SD55.7±2.038.5±0.037.0±5.243.8±8.826.0±2.814.6±2.936.8±1.537.8±4.565.7±6.0
Qwen3-1.7BPR-OPD54.3±4.046.2±10.935.2±2.643.8±17.732.0±0.027.1±2.940.0±2.044.4±2.072.2±3.3
Qwen2.5-3BGRPO92.9±6.150.0±5.481.5±0.046.9±13.360.0±22.658.3±5.969.6±2.561.1±0.179.3±2.2
Qwen2.5-3BSDAR90.0±2.061.5±0.072.2±13.140.6±13.356.0±17.047.9±14.765.0±4.061.9±3.880.1±1.8
Qwen2.5-3BSkill-SD92.9±6.161.5±0.083.3±2.656.3±0.066.0±2.847.9±8.871.4±1.063.0±0.875.2±0.4
Qwen2.5-3BPR-OPD88.6±4.069.2±10.981.5±5.268.8±8.866.0±8.558.3±11.873.9±2.567.0±0.681.4±2.3

Qwen2.5-3B on AlfWorld:

MethodSeed 1 (%)Seed 2 (%)Mean (%)Δ vs. GRPO [95% CI]
GRPO71.4 (100/140)67.9 (95/140)69.64—
Token-OPD67.9 (95/140)70.0 (98/140)68.93−0.71 [−37.02, +35.59]
PR-OPD w/o projector77.9 (109/140)73.6 (103/140)75.71+6.07 [+1.53, +10.61]
PR-OPD w/ projector75.7 (106/140)72.1 (101/140)73.93+4.29 [+4.29, +4.29]†

Due to resource constraints, we used eight A100 GPUs for the rebuttal runs (the original experiments used eight B200 GPUs). The GRPO warm-up checkpoint achieved lower success rates in some model–task settings, but all methods in each matched comparison started from the same checkpoint for a given seed, and PR-OPD continued to outperform GRPO on both benchmarks with both backbones across the two seeds.

We sincerely appreciate your acknowledgment and constructive comments to improve our work.

Q1: Why does PR-OPD fall short on some ALFWorld tasks?

Answer: Our interpretation is that hindsight is most useful when it highlights a requirement the agent can already act on. In an ALFWorld Heat task, for example, the instruction specifies that the object must be heated before placement. After the agent searches an empty container in a Pick task, however, the object’s location may still be unknown. Hindsight can suggest how to continue searching, but the student must ground its next action in the observations available at that point. The 3B results illustrate this distinction. Across the two seeds, PR-OPD solved 12 and 10 of 16 Heat games, compared with 6 and 9 for GRPO. On Pick, PR-OPD solved 32 and 30 of 35 games, compared with 34 and 31 for GRPO. GRPO’s high Pick success rate also left limited room for improvement.

On WebShop, a requirement in the instruction, such as a requested size or material, can guide several successive decisions: forming a search query, comparing products, and selecting an option. This gives the student repeated opportunities to apply the same hindsight lesson. The full-split results also clarify the size of the difference. At 3B, PR-OPD improved over GRPO by 5.9 percentage points on WebShop and 4.3 on ALFWorld. At 1.7B, the gains were nearly identical (6.3 and 6.4 points, respectively). We therefore interpret the results in terms of which decisions hindsight can inform, with the clearest variation appearing across ALFWorld task types.

Q2: Why does the projector help less on the 3B backbone?

Answer: On 3B ALFWorld, the two-seed evaluation reproduced the direction of the original projector ablation, but the gap was smaller. Adding the projector reduced the number of games solved from 109 to 106 in one seed and from 103 to 101 in the other, out of 140 games per seed. The mean change was −1.79 percentage points (paired 95% CI [−6.32, +2.75]), compared with −6.3 points in the original run. Given this uncertainty, the two runs do not establish a systematic performance penalty.

Our explanation is that the larger backbone benefits less from the flexibility of a learned projection. In the original ALFWorld runs initialized at GRPO step 100, direct alignment yielded a smaller discrepancy between the student’s raw hidden states and its skill-conditioned teacher’s states at 3B than at 1.7B (0.012 vs. 0.031, averaged over steps 146–150; Table 4). A projector lets the student match the teacher through a learned mapping, giving its raw representation more freedom to retain features useful for action selection. That flexibility may matter more at 1.7B, where a larger discrepancy remained after direct alignment. This offers a plausible explanation for the difference across backbones, while the small 3B success-rate gap remains compatible with run-to-run variation.

Q3: Have we observed skill hallucination?

Answer: This is a very interesting point to worry about. We found little evidence of widespread skill hallucination in the student. We screened 34,476 ALFWorld responses across PR-OPD, GRPO, and the starting checkpoint, including 14,040 from PR-OPD. Manual inspection of every keyword match found no instance of the student reciting a hindsight rule, lesson, or skill. We also examined behavior directly. Attempts to take, move, examine, or open an object that had never appeared in the agent’s observations or available commands accounted for only 0.14–0.89% of steps across checkpoints, with no increase under PR-OPD.

We did identify one recurring navigation error, which we think is interesting to be shared. On the 140 seen games, failed commands naming a previously unobserved instance occurred at similar rates under PR-OPD and GRPO (1.48% vs. 1.60% of all steps). On the 134 unseen games, the rates were 1.82% and 0.74%, respectively (difference +1.08 percentage points, 95% game-bootstrap CI [+0.37, +1.91]). A single command, go to fridge 2, accounted for 58 of the 75 excess steps: after searching the first fridge, the student sometimes inferred that there was a second one. The student’s reasoning suggests a possible explanation. In one representative trace, it correctly recalled that it had already searched fridge 1 and found nothing, then concluded that it should “check the other fridges.” The inference that failed was the existence of a second fridge. The numbered name fridge 1 may have made fridge 2 a plausible next search target even though the agent had never observed it. This reading is consistent with the concentration of excess errors in one navigation command and with the absence of a corresponding increase in attempts to take or open unobserved objects.

Reviewer_tGiSOpenReview ↗

Response to the Authors

Thanks for your response to my questions/weaknesses. I like your responses to Q1 and Q2 a lot, I think the paper would improve greatly with this kind of analysis present in the main body. While having higher benchmark scores is great, understanding where the method helps most and why is where readers are going to get the most value. On Q3 I agree that it's a somewhat out there concern, though the cited prior work does show that supervising with asymmetric information can lead to learning signal bias, so I was curious if the same kind of effect would show up here. It's good to know that the method doesn't result in hallucinated skills, and I appreciate the interesting failure case you showed here. Overall I would support the paper for acceptance.

Many thanks for your positive feedback and for the time and effort you have devoted to improving our work. We sincerely appreciate your constructive comments and valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.