acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Self-Distillation with Outcome-Aligned Privileged Information for Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) trains large language model agents through long-horizon interactions, but sparse trajectory-level rewards provide limited guidance on individual decisions. Recent on-policy self-distillation (OPSD) methods supplement sparse rewards with dense supervision derived from natural-language skills or reflections used as training-time privileged information (PI). However, discrete textual PI prevents gradients from flowing directly through the generated tokens to optimize PI construction. Learnable latent PI alleviates this bottleneck, but learnability alone does not ensure that its influence on sampled actions is aligned with relative trajectory outcomes. To address this gap, we propose PILOT (Privileged Information Learning with Outcome-aligned Training), a framework that learns outcome-aligned latent PI to guide on-policy self-distillation for agentic RL. PILOT constructs latent PI from source trajectories and measures its influence on distinct same-task query trajectories through sampled-action log-likelihood differences with and without PI. With the actor held fixed, it trains the PI analyzer to align each query’s average bounded influence with relative outcome credit derived from the same-task on-policy rollout group. The measured influence weights an OPSD objective optimized jointly with direct RL, yielding an actor that requires neither privileged inputs nor the analyzer at inference. We evaluate PILOT with Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on ALFWorld and WebShop. PILOT outperforms the evaluated RL and OPSD baselines on all aggregate task metrics across both benchmarks and model scales.Our code is available at https://anonymous.4open.science/status/PILOT-478B.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, on-policy self-distillation, privileged information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QkurOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a common issue in Agentic RL + OPSD training: due to the sources of privileged information (PI) and various other factors, the teacher's judgment of whether the current action is correct may not always be accurate, and may not align with outcome feedback. To address this issue, the paper builds on the idea of LOPD and proposes Pilot. Based on learnable latent PI, it trains an analyzer to generate outcome-aligned PI, so that the teacher conditioned on this PI makes judgments about actions that align with environmental feedback. When updating the actor, the paper uses PI-Guided Actor Optimization to introduce a finer-grained loss on top of GRPO.

The authors evaluate Pilot on ALFWorld and WebShop using Qwen2.5-3B/7B-Instruct as backbones, and compare it with a wide range of mainstream RL and OPD/OPSD methods. Compared with existing baselines, Pilot achieves consistent improvements.

Critical strengths
  1. The paper is clearly written and easy to understand. Readers can quickly follow the logical progression from the problem identified by the authors to their proposed solution and method analysis. The analyzer architecture, training procedure, and loss derivations are also clear and easy to follow.
  2. Pilot's approach of improving PI quality by training a latent PI analyzer is insightful. It also provides a promising direction for future work on dynamically generating PI for the current trajectory to prevent the teacher from being misled.
  3. Pilot achieves relatively consistent improvements over existing baselines across two backbones and two benchmarks, demonstrating the effectiveness of the method. The experiments are also quite solid, with fairly comprehensive ablation studies supporting the effectiveness of most design choices.
Critical weaknesses
  1. Some figures are not vector graphics but PNG images that become blurry when zoomed in (e.g., Figures 1 and 2). This is not quite in line with the presentation standards expected of an academic paper.
  2. In my view, the purpose of combining RL with OPSD is that RL provides reliable but relatively coarse-grained supervision, while OPSD can provide fine-grained token-level supervision. However, the proposed method forces the PI-conditioned teacher's average assessment over an entire trajectory to align with the coarse-grained supervision from the final outcome. In this setting, even an ultimately successful trajectory may contain incorrect or redundant steps. The Analyzer's training makes no distinction between the correct and incorrect steps within that trajectory, which may result in the PI-conditioned teacher still being unable to distinguish good and bad steps within the same trajectory at a fine-grained level.
  3. The gate in the current L_OPSD design is always positive, and the L_OPSD term always encourages the sampled actions. This is undesirable for actions that are inherently poor or redundant.
Questions for authors
  1. Are there any experiments that directly demonstrate the effectiveness of the Outcome Alignment mechanism? For example, after training with Outcome Alignment, do queries with high outcome credit actually receive more positive PI influence, while queries with low outcome credit receive more negative PI influence?
  2. During Analyzer training, its learning signal depends on sufficient outcome variation within the reference group. For tasks that are too difficult, where many rollout groups have all-zero rewards, would Pilot fail? Traditional RL + OPSD can also alleviate this issue to some extent.
6 comments

W1. Figure Quality

Response: Thank you for pointing this out. We have replaced Figures 1 and 2 with vector graphics and checked the remaining figures to ensure that text and graphical elements remain clear when zoomed in.

W2. Step-Level Discrimination under Trajectory-Level Alignment

Response: Thank you for raising this concern. We agree that successful trajectories may contain incorrect or redundant steps. Outcome Alignment constrains average trajectory-level influence while retaining step-specific scoring and weighting. To examine whether these weights reflect differences in action utility within the same trajectory, we conducted an additional within-trajectory counterfactual continuation analysis.

1. Trajectory-level alignment and step-specific weighting.

For query trajectory jj, the average bounded PI influence is

\sum_{t\in\mathcal T_j} \tanh\left(\frac{\Delta_{jt}}{T_\Delta}\right).$$ The Analyzer’s alignment term encourages queries with higher relative outcome credit to receive higher average influence, subject to regularization. **This average constraint neither assigns the trajectory’s final outcome label to every action nor requires all steps to have identical or positive influence.** At each step, influence is computed by comparing the log-probability of the same sampled action with and without PI, conditioned on its interaction history, and is used to determine the auxiliary imitation weight. **2. Within-trajectory counterfactual analysis of action utility.** Across four ALFWorld tasks, we selected one high-gate and one low-gate nonterminal valid decision position from **the same original trajectory** for each task. Positions were selected before continuation evaluation, with the Actor, Analyzer, PI pairings, and gate weights fixed. Each pair therefore shares the same original trajectory outcome, allowing us to examine whether the gates contain decision information beyond overall trajectory success or failure. At each position, we restored the complete environment state and interaction history before the action and estimated: - $Q_t$: the success rate (%) when the original action is forced and execution then continues with the frozen PI-free Actor. - $V_t$: the success rate (%) when the current action is resampled from the same Actor and execution then continues. Both branches used the same remaining environment-step budget, with **eight independent continuations per branch**. We measured the original action’s local advantage over resampling from the current policy as $$\widehat A_t=\widehat Q_t-\widehat V_t.$$ This advantage is estimated independently through environment continuations and can be positive, zero, or negative; it is distinct from the gate weight, which remains positive. Success rates, estimated advantages, and advantage gaps below are expressed in %. | Task | Gate weights (High / Low) | High-gate advantage (%) | Low-gate advantage (%) | Advantage gap (%) | | :--- | :---: | :---: | :---: | :---: | | Examine CD | 0.70 / 0.35 | +37.5 | −12.5 | +50.0 | | Clean fork | 0.68 / 0.40 | +25.0 | 0.0 | +25.0 | | Examine laptop | 0.75 / 0.30 | +25.0 | −12.5 | +37.5 | | Clean lettuce | 0.72 / 0.25 | +25.0 | −37.5 | +62.5 | **In all four comparisons, the high-gate action had a higher estimated local advantage than the low-gate action within the same trajectory, with a mean advantage gap of 43.75%.** High-gate actions consistently had positive estimated advantages, whereas low-gate actions had zero or negative estimated advantages and received lower auxiliary imitation weights. **Despite sharing the same original trajectory outcome, high-gate actions consistently exhibited higher estimated local advantages, showing that the gates captured differences in action utility beyond overall trajectory success or failure at the evaluated positions.** We have added a related discussion in **Appendix E.3** to clarify how trajectory-level alignment and step-specific auxiliary weighting complement each other.

W3. Potential Limitations of Positive Gating in Action Reinforcement

Response: Thank you for pointing this out. We agree that positive gating assigns nonzero auxiliary imitation weights even to actions with negative influence. Pilot is designed to weaken auxiliary reinforcement of these actions rather than directly convert negative influence into a penalty. PI influence measures how PI changes the sampled action’s log-probability; it is not a direct label of action correctness. Negative influence therefore does not necessarily indicate an incorrect or redundant action.

1. How soft gating modulates auxiliary reinforcement.

With g=σ(sΔ)g=\sigma(s\Delta) and s>0s>0, negative influence receives g<0.5g<0.5, whereas positive influence receives g>0.5g>0.5. Negative-influence actions thus receive lower auxiliary imitation weights. This gate is used jointly with the GRPO objective, and both objectives determine the overall policy update. A positive gate neither reinforces all sampled actions equally nor implies that the overall update necessarily increases every action’s probability.

2. Empirical comparison of gating variants.

To investigate whether completely filtering or explicitly suppressing negative-influence actions is beneficial, we evaluated four additional variants:

MethodGate or auxiliary updateValidation success rate (%)Validation score (%)
Pilot (originally reported)g=σ(sΔ)g=\sigma(s\Delta)82.090.6
Hard-zero nonpositive influenceg=1[Δ>0]σ(sΔ)g=\mathbf{1}[\Delta>0]\sigma(s\Delta)65.674.9
Smooth positive-only gateg=max⁡(0,2σ(sΔ)−1)g=\max(0,2\sigma(s\Delta)-1)63.382.4
Signed auxiliary updatew=tanh⁡(sΔ/2)w=\tanh(s\Delta/2), with clipping60.277.1
No Actor auxiliary objectiveZero auxiliary loss coefficient; retain Analyzer training58.676.7

The signed variant promotes actions with positive weights and suppresses actions with negative weights within a clipped log-probability-change interval of [−0.2,0.2][-0.2,0.2]. Both the weights and old-policy log-probabilities are detached. The variant without the Actor auxiliary objective retains Analyzer training and therefore is not an independent GRPO baseline.

None of these alternatives exceeded Pilot’s originally reported results. Among the supplementary variants, the signed update also achieved lower success rate and score than smooth positive-only gating. These observations are consistent with the rationale for soft gating: reducing auxiliary imitation weights for negative-influence actions can be an effective choice without directly converting negative influence into a penalty.

We have added a related discussion in Appendix D.4 to clarify that sigmoid gating continuously modulates auxiliary imitation strength rather than directly determining action correctness or guaranteeing that all incorrect or redundant actions are filtered out. We have also added the experimental settings and results of these variants in Appendix D.4 to explain the empirical basis of this design.

Q1. Experimental Validation of Outcome Alignment Effectiveness

Response: Thank you for suggesting this direct examination of the mechanism. We conducted a controlled experiment to measure whether Outcome Alignment makes PI influence more favorable for higher-credit queries than for lower-credit queries. The results show substantially stronger credit–influence alignment and a larger high–low credit influence gap after training with correctly associated outcome credits, including on tasks excluded from Analyzer training.

1. Controlled experimental setup. We held the Actor checkpoint, rollout corpus, source–query pairings, and evaluation credits fixed. The Actor remained frozen throughout, and we compared four Analyzer variants:

  • Initialized Analyzer: evaluated the common initialization without further training.
  • No Alignment: removed the Outcome Alignment term and trained using only the original anchor and influence regularizers.
  • Shuffled Credit: retained the alignment objective and regularizers but shuffled query-level training credits within each task.
  • Full Alignment: trained with correctly associated query outcome credits and the original regularizers.

The three trained variants started from the same Analyzer initialization and used the same data order, learning rate, and budget of 150 Analyzer updates, with random seed 0. We used four training tasks comprising 313 turns and four held-out tasks comprising 23 valid query trajectories and 294 turns. The held-out tasks did not participate in Analyzer updates.

2. Direct measurement of credit–influence alignment. For query trajectory jj, we measure its average bounded PI influence as

Ij=1Tj∑t=1Tjtanh⁡(Δj,t),I_j = \frac{1}{T_j}\sum_{t=1}^{T_j}\tanh(\Delta_{j,t}),

where Δj,t\Delta_{j,t} is the mean sampled-action token log-probability with PI minus that without PI. Let C~j\widetilde{C}_j denote the normalized outcome credit. We report two metrics:

  • Credit–influence alignment: mean⁡j(C~jIj)\operatorname{mean}_j(\widetilde{C}_j I_j).
  • High–low credit influence gap: E[Ij∣C~j>0]−E[Ij∣C~j<0]\mathbb{E}[I_j \mid \widetilde{C}_j > 0] - \mathbb{E}[I_j \mid \widetilde{C}_j < 0].

Both metrics are computed within each held-out task and then averaged across tasks. Higher values indicate stronger alignment between outcome credit and PI influence.

Analyzer variantCredit–influence alignment ↑High–low credit influence gap ↑
Initialized Analyzer0.007760.01604
No Alignment0.000620.00147
Shuffled Credit0.006200.01300
Full Alignment0.081590.22275

3. Findings. Full Alignment increased credit–influence alignment from 0.00776 to 0.08159 and the high–low credit influence gap from 0.01604 to 0.22275 relative to initialization. Its influence gap also exceeded that of every control on each of the four held-out tasks. Removing the alignment term substantially weakened this separation, and shuffling credits also reduced it, supporting the importance of the correct credit–query correspondence.

These controlled results provide direct evidence that Outcome Alignment learns a relative trajectory-level relationship: higher-credit queries receive more favorable PI influence than lower-credit queries, and this relationship extends to the held-out tasks in this experiment.

Reviewer_UyMBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PILOT, which learns latent privileged information from other on-policy trajectories of the same task. An analyzer aligns its influence on sampled-action probabilities with relative trajectory outcomes. The actor is trained using GRPO and an influence-gated auxiliary objective, without privileged inputs at inference. Experiments cover ALFWorld and WebShop with Qwen2.5-3B and 7B.

Critical strengths

Strong benchmark performance: PILOT achieves the highest aggregate scores among the evaluated baselines on ALFWorld and WebShop with both Qwen2.5-3B and 7B.

Critical weaknesses

The choice of PI source is insufficiently justified. It is unclear why PI should be extracted from the current policy’s own trajectories rather than expert demonstrations, offline experience, or external feedback. When the policy is weak, its rollouts may repeatedly contain similar errors, limiting the information they provide. Controlled comparisons would help establish the value of self-generated experience relative to alternative sources.

Questions for authors

Could the authors provide a clearer rationale for using self-generated trajectories as PI, or controlled comparisons with alternative PI sources, to address the Critical Weakness?

10 comments

We thank the reviewer for the careful review, constructive feedback, and recognition of PILOT’s strong benchmark performance. We particularly appreciate the suggestion to clarify the choice of PI source and strengthen its empirical justification through controlled comparisons.

W1 & Q1. Justification for Self-Generated Trajectories as PI: Rationale and Controlled Comparisons

Response: We clarify our rationale and provide controlled comparisons with alternative PI sources below.

Q1.1. Rationale for using self-generated trajectories

(a) Complementary information across trajectories. Even when generated by the same policy, different trajectories of the same task can contain different observations, action sequences, and outcomes. Other completed trajectories can therefore provide hindsight context beyond the query’s own interaction history.

(b) Experience that evolves with the actor. As the actor’s capabilities and behavior change, the states it visits, actions it takes, and difficulties it encounters also change. Using current-policy trajectories refreshes the PI source accordingly, providing experience relevant to the policy being optimized.

(c) Reuse of existing rollouts. Self-generated trajectories are already collected during on-policy RL and can be used for both direct RL and PI construction. This design requires neither additional expert demonstrations nor external model calls to generate feedback, allowing PI learning to be integrated into the existing training pipeline.

Q1.2. Controlled comparisons with alternative PI sources

We compared current-policy PI with three alternatives: a fixed offline trajectory library generated by the initial policy, expert trajectories, and trajectories generated by a lagged policy. Across the PILOT variants, we kept the query rollout collection procedure and outcome-credit computation rules fixed, retained a trainable analyzer, and matched the number of source trajectories and the visible token budget. The encoder, Q-Former, and training objectives remained unchanged.

MethodPI sourceSuccess rateTask score
GRPONo PI68.0%78.4
PILOT–CurrentCurrent-policy trajectories82.0%90.6
PILOT–Initial/OfflineFixed initial-policy trajectory library66.4%75.3
PILOT–ExpertExpert trajectories68.0%79.5
PILOT–LaggedLagged-policy trajectories72.7%82.0

Current-policy PI performs best among the evaluated sources, achieving a success rate of 82.0%, compared with 66.4% for fixed offline PI, 68.0% for expert PI, and 72.7% for lagged-policy PI. Compared with GRPO, it increases success from 68.0% to 82.0% and task score from 78.4 to 90.6.

These comparisons support the effectiveness of current-policy experience within PILOT. In particular, expert demonstrations do not automatically provide more useful PI under the same analyzer and learning objective. The advantage of current-policy PI over lagged-policy and fixed offline PI also supports the design choice of continually refreshing source experience as the actor evolves.

Together, these results provide direct empirical support for choosing current-policy trajectories as PILOT’s PI source: they achieve the highest success rate and task score among the evaluated sources under the same PI learning framework, while requiring no additional expert demonstrations.

We have added the PI-source comparison to Appendix D.5. We hope that the rationale and controlled comparisons above clarify our choice of PI source and address the reviewer’s concern. We are happy to answer any further questions.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer UyMB,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_xqyPOpenReview ↗
Rating
3(+2)
Confidence
3(−1)
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes PILOT, an on-policy self-distillation method for agentic RL. For each task, it samples multiple trajectories and uses other rollouts as privileged information (PI) for a self-teacher. A Q-Former compresses these trajectories into latent tokens, which are optimized to align their influence on the actor's sampled-action probabilities with relative trajectory outcomes. This influence is then used to weight an auxiliary self-distillation loss alongside GRPO.

The authors evaluate the approach on ALFWorld and WebShop using Qwen2.5-3B/7B-Instruct and report improvements over GRPO and several self-distillation baselines.

Critical strengths

The paper addresses an important limitation of agentic RL: trajectory-level rewards provide limited supervision for individual decisions.

The reported improvements over GRPO, SDAR, and LOPD are encouraging. The ablations also suggest that source–query pairing, continued analyzer training contribute to performance.

Critical weaknesses

1. Limited novelty and insufficient justification for the complexity of the proposed method.

The central idea of using other on-policy rollouts as privileged information for self-distillation is not new.

MOPD (arXiv:2605.12652) already conditions a teacher on successful and unsuccessful peer rollouts from the same prompt. GRSD (arXiv:2607.28076) constructs privileged guidance from group-level trajectory reflections. LOPD (arXiv:2608.13040) already uses a trainable latent-context compressor with an outcome-dependent privileged-margin objective.

PILOT introduces a different source–query pairing strategy, relative-outcome alignment and other elements. However, the paper does not convincingly establish why these modifications constitute a substantial advance over the existing approaches.

I would like to know whether such a complicated PI-learning pipeline necessary? The method introduces a frozen encoder, trainable Q-Former, source–query pairing with additional elements and a separate optimization stage. OPSD + RL itself is a complicated setup to tune and optimize but the current pipeline has more complex elements to optiimize. Why not directly optimize the privileged prompts constructed from other trajectories? Alternatively, could a JEPA-style predictive objective learn outcome-relevant PI from successful and failed trajectories without optimizing the Q-Former through the frozen actor's sampled-token log-probabilities?

The paper should justify its choice of PI-learning mechanism against these simpler or alternative formulations

2. Experimental validation and objective design are not sufficiently convincing.

The baseline implementation details are not sufficiently clear. Appendix C.2 provides brief descriptions of the baselines, but it is unclear how the privileged information was constructed for OPSD and GRPO+OPSD, and whether the methods were evaluated under comparable training budgets.

Several baseline results in Table 1 are identical to those reported in SDAR. However, PILOT reports 240 policy updates, whereas SDAR uses 150. Were these baselines rerun or taken directly from previous work? This should be clarified, particularly since training budgets can substantially affect performance.

I also could not find the number of independent training seeds or any standard deviations. Some of the improvements are relatively small. For example, on WebShop with Qwen2.5-7B, PILOT achieves 83.6% success compared to SDAR's 82.8%. Since validation uses 128 sampled tasks per evaluation, it is difficult to determine how significant these differences are without multiple seeds and uncertainty estimates.

The evaluation is also relatively limited. The experiments only consider ALFWorld and WebShop with Qwen2.5-Instruct models. Both benchmarks are relevant for agentic RL, but it is unclear whether the approach generalizes to more demanding reasoning tasks or reasoning-oriented models. An evaluation on Qwen3 and additional benchmarks such as Reasoning Gym, mathematical reasoning, or code generation would strengthen the paper.

Finally, I have some concerns about the proposed outcome-alignment objective. The Q-Former is optimized using trajectory-level outcome credit and average PI-induced likelihood differences, whereas the resulting gates are applied at the action-turn level. It is unclear whether this alignment actually identifies useful individual decisions. Although the authors provide gate-shuffling ablations, these do not directly establish that the learned turn-level weights correspond to action quality. A more direct analysis would be useful.

Questions for authors
  1. How does PILOT differ fundamentally from MOPD, GRSD, and LOPD? In particular, what does the outcome-alignment objective add beyond existing methods for learning privileged information from other trajectories?

  2. Why is the Q-Former-based PI-learning pipeline necessary? Have the authors considered optimizing textual or soft prompts directly, or using a JEPA-style predictive objective to learn outcome-relevant privileged information?

  3. What privileged information is used for the OPSD and GRPO+OPSD baselines? Were all baselines rerun under the same training budget, or were results taken from prior papers?

  4. How many independent training seeds were used? Can the authors report means and standard deviations, especially for the relatively small improvements on Qwen2.5-7B?

  5. Since the analyzer is trained with trajectory-level credit, how do the authors establish that its turn-level gates reflect useful decisions rather than simply correlating with overall trajectory success?

  6. Can the authors provide evidence that the approach generalizes to reasoning-oriented backbones and more diverse tasks beyond ALFWorld and WebShop?

13 comments

We thank the reviewer for the careful review and constructive feedback. We appreciate the opportunity to clarify our methodology and provide additional evidence addressing the concerns raised.

W1 & Q1: Novelty relative to MOPD, GRSD, and LOPD

Q1.1. How does PILOT differ fundamentally from MOPD, GRSD, and LOPD?

Response: We agree with the reviewer that prior work has established two important advances: peer rollouts for the same task can provide useful privileged information (PI), and PI itself can be learned.

  • MOPD conditions the teacher on successful and failed rollouts from the same group.
  • GRSD contrasts reflections on these rollouts to construct guidance that modulates policy advantages.
  • LOPD learns continuous latent context from retrieved experience and uses an outcome-weighted privileged-margin objective to regulate its effect on sampled-token probabilities.

Building on these advances, PILOT trains the Analyzer to address a more specific question: given PI from another trajectory, how should the current Actor’s scores for its own sampled actions change?

  1. Measuring PI influence. We hold the policy snapshot, interaction history, and action prefix fixed, score the same sampled action with and without PI, and measure PI influence as the log-probability difference.

  2. Aligning influence with relative outcomes. We then compute the query’s relative outcome credit against a reference group that excludes the source trajectories, encouraging higher average PI influence on queries with better relative outcomes within the same task. This alignment operates at the trajectory level.

  3. Using influence for Actor optimization. During Actor optimization, PI influence at each interaction turn determines the weight of that turn’s auxiliary imitation loss, which is combined with GRPO for training (Sections 3.2–3.4).

Thus, PILOT learns not only the latent PI representation but also how it should influence the policy, and uses that influence to regulate the Actor’s uptake of supervision.

Q1.2. In particular, what does the outcome-alignment objective add beyond existing methods for learning privileged information from other trajectories?

Response: Learning latent PI enables information extraction from other trajectories; outcome alignment additionally specifies how that information should affect the policy. As established in Appendix B.1, the alignment statistic equals the within-task covariance between relative outcome credit and mean bounded PI influence. The objective therefore encourages higher average influence on queries with better relative outcomes. Compared with LOPD’s outcome-weighted margin, PILOT introduces an explicit within-task relative alignment objective based on a same-policy comparison with and without PI.

Our experiments support both the full method and the additional value of this objective. The table below presents the results on WebShop with Qwen2.5-3B-Instruct:

MethodSuccess (%) ↑WebShop Score (%) ↑
LOPD76.685.2
PILOT w/ Privileged-Margin71.982.7
PILOT (Ours)82.090.6

PILOT achieves 82.0% success versus 76.6% for LOPD, an absolute improvement of 5.4%. To further examine the learning objective, we retain PILOT’s Q-Former latent-PI architecture and replace outcome alignment with the outcome-weighted privileged-margin objective. Compared with this variant, PILOT achieves absolute improvements of 10.1% in success (71.9% → 82.0%) and 7.9% in WebShop score (82.7% → 90.6%), supporting the effectiveness of relative outcome alignment for learning latent PI within our framework.

Furthermore, shuffling query outcome credits reduces success from 82.0% to 78.1% (Section 4.4, Table 2), providing complementary evidence that correctly matching outcome credit to PI influence matters.

W1 & Q2: Necessity and complexity of the Q-Former-based PI-learning pipeline

Q2.1. Why is the Q-Former-based PI-learning pipeline necessary?

Response: The Q-Former is designed to selectively extract and compress privileged information from source trajectories that is useful for the Actor's decisions. Providing trajectory text directly requires the Actor to identify and use relevant information within the additional context, whereas Pooling–MLP aggregates trajectory representations through predefined pooling operations. In contrast, the Q-Former uses learnable query tokens for selective aggregation, while Outcome Alignment connects PI learning to its influence on the frozen Actor's action probabilities. This design combines trajectory compression with information selection tailored to the Actor.

To evaluate whether this design is necessary, we compared PILOT with the following simplified alternatives:

  • Trajectory-text PI: We convert the same source trajectories into outcome-annotated textual context and use it directly as PI, retaining the same Actor update procedure.
  • Pooling–MLP compressor: We retain the frozen encoder and replace the Q-Former with segment-wise pooling followed by an MLP. The compressor produces PI tokens with the same token count and dimensionality, while retaining the original Outcome Alignment objective and Actor update procedure.
  • Outcome-only gating: We remove PI extraction and scoring, and construct auxiliary training weights directly from the query trajectory's within-group relative outcome. We retain the same form of the Actor's auxiliary loss and control its total weight.

The results are as follows:

MethodSuccess (%) ↑WebShop Score (%) ↑
PILOT82.090.6
Trajectory-text PI73.684.4
Pooling–MLP compressor66.482.0
Outcome-only gating75.885.3

All differences below are absolute differences on the percentage scale.

PILOT outperforms trajectory-text PI by 8.4% in Success and 6.2% in WebShop Score, showing that directly providing trajectory information is insufficient to replace learned PI extraction. More importantly, replacing the Q-Former with Pooling–MLP reduces the two metrics by 15.6% and 8.6%, respectively, while keeping the frozen encoder, PI token count and dimensionality, Outcome Alignment objective, and Actor update procedure unchanged. This controlled comparison provides direct evidence for retaining the Q-Former: with the same latent capacity and learning objective, the simpler compressor does not extract equally effective PI.

PILOT also outperforms outcome-only gating by 6.2% and 5.3%, respectively. Thus, assigning auxiliary training weights solely from trajectory outcomes is insufficient to recover the full method's gains. This further supports the value of extracting PI from source trajectories and constructing training signals based on its influence on the Actor.

These gains come with quantifiable training overhead. As shown in Appendix D.2, on the same model and benchmark, PILOT takes 156.51 seconds per update, compared with 117.27 seconds for GRPO, corresponding to 1.33× the update time.

Overall, these experiments support the need for the Q-Former within the evaluated PI-learning pipeline: direct textual context, simpler representation compression, and outcome-only weighting all fall short of the full method's performance.

We have added these comparisons and the corresponding analysis to Appendix D.5 of the revised manuscript.

Q2.2. Have the authors considered optimizing textual or soft prompts directly, or using a JEPA-style predictive objective to learn outcome-relevant privileged information?

Response: We thank the reviewer for suggesting these alternatives. Outcome prediction and JEPA-style latent prediction are reasonable approaches to learning PI. The key distinction lies in their optimization targets: predictive objectives train representations to predict outcomes or trajectory content, whereas PILOT's Outcome Alignment trains PI so that its influence on the frozen Actor's action log-probabilities aligns with relative outcome credit. PILOT therefore optimizes not only for outcome-related information, but also for how that information affects the Actor's decisions.

To evaluate this distinction, we retained the same frozen encoder, Q-Former, PI token count and dimensionality, and Actor update procedure, while replacing the Analyzer's learning objective:

  • Outcome-weighted privileged-margin: We replace Outcome Alignment with an outcome-weighted privileged-margin objective, retaining the same PI scoring interface to compare alternative outcome-aware learning criteria.
  • Outcome prediction: We predict the source trajectory's own outcome from its latent PI, using score regression for WebShop and success classification for ALFWorld. The Analyzer receives no gradients through the Actor's action probabilities.
  • JEPA-style prediction + outcome prediction: We predict a masked target's stop-gradient embedding from a compressed representation of the source context, and add the same outcome-prediction auxiliary objective as above. The Analyzer likewise receives no gradients through the Actor's action probabilities. This baseline combines latent prediction with explicit outcome supervision, directly testing whether their combination is sufficient to learn effective PI.

For these alternatives, we still compute PI influence and the resulting gate through forward scoring with the frozen Actor, and retain the same Actor update procedure. This focuses the comparison on how the Analyzer learns PI.

Analyzer objectiveSuccess Rate (%) ↑WebShop Score ↑
PILOT Outcome Alignment82.090.6
Outcome-weighted privileged-margin71.982.7
Outcome prediction65.680.0
JEPA-style prediction + outcome prediction65.678.5

The success-rate differences below are absolute differences expressed in %, while WebShop Score differences are reported on the score scale.

PILOT outperforms direct outcome prediction by 16.4% in success rate and 10.6 in WebShop Score. It also outperforms JEPA-style prediction with outcome supervision by 16.4% in success rate and 12.1 in WebShop Score. Thus, with the same PI architecture and interface, training PI to predict outcomes or latent representations does not match directly optimizing its influence on the Actor. PILOT also improves over the outcome-weighted privileged-margin objective by 10.1% in success rate and 7.9 in WebShop Score, indicating that the specific alignment criterion matters beyond incorporating outcome-aware supervision.

These results support the motivation for Outcome Alignment: predictive objectives constrain what PI can predict, whereas Outcome Alignment directly constrains how PI influences the Actor. Because PI is ultimately used through the Actor's action probabilities to construct gates for auxiliary training, Outcome Alignment connects the Analyzer's objective to the downstream use of PI. The results therefore support retaining the training path through the frozen Actor; the evaluated outcome-prediction and JEPA-style alternatives do not provide equally effective training signals.

We have added these comparisons and the corresponding analysis to Section 4.4 and Appendix D.6 of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank Reviewers xqyP, UyMB, and Qkur for their careful reading and constructive feedback. Their comments helped us clarify PILOT’s methodological contributions and strengthen its empirical evaluation. Below, we summarize the strengths recognized by the reviewers, the main evidence provided in our responses, and the discussion status.

Strengths Highlighted by the Reviewers

We appreciate the reviewers’ recognition of the following strengths:

  • Important problem: PILOT addresses a key limitation of agentic RL: trajectory-level rewards provide limited supervision for individual decisions.
  • Insightful methodology: Learning a latent privileged-information (PI) analyzer offers a promising approach to improving PI quality and the guidance provided to the teacher.
  • Strong empirical performance: PILOT achieves the highest aggregate scores among the evaluated baselines on ALFWorld and WebShop with both Qwen2.5-3B and Qwen2.5-7B.
  • Clear presentation and informative ablations: The method is clearly presented, with ablation studies supporting key design choices, including source–query pairing and continued analyzer training.

Main Clarifications and Additional Evidence

  • To Reviewer xqyP: We clarified PILOT’s novelty in learning how privileged information should influence the Actor, documented baseline PI sources and training budgets, and provided comparisons under matched policy-update budgets with three-seed evaluation. Controlled comparisons against trajectory-text PI, Pooling–MLP, outcome-only gating, and alternative learning objectives further support the contributions of the Q-Former and Outcome Alignment. Within-trajectory counterfactual experiments provide evidence that the gates distinguish action utility beyond overall trajectory success. Evaluations across seven Search-QA datasets and a reasoning-capable backbone broaden the empirical coverage beyond ALFWorld and WebShop.

  • To Reviewer UyMB: We provided controlled PI-source comparisons, per-seed results, and training diagnostics, with current-policy trajectories performing best among the evaluated sources. We explained the identical mean success rates of expert PI and GRPO through per-seed success counts and clarified that expert PI reweights the Actor’s sampled actions rather than directly providing imitation targets. Further diagnostics showed that unsuccessful trajectories can still provide relative outcome supervision when task-completion scores differ, even if specific errors recur. Queries without valid outcome contrasts are filtered out, clarifying both the availability of supervision during low-success training and its limitations when outcome differences are absent.

  • To Reviewer Qkur: We addressed figure quality with vector graphics and clarified how trajectory-level Outcome Alignment preserves step-specific weighting. High-gate actions had higher estimated local advantages in all four within-trajectory counterfactual comparisons, with a mean gap of 43.75 percentage points. Additional ablations supported soft weighting over filtering or explicitly penalizing negative-influence actions, while controlled Analyzer experiments provided evidence of credit–influence alignment on held-out tasks. Training-signal analysis and weak-initialization experiments further showed that PILOT retained effective supervision despite frequent all-zero reward groups and outperformed GRPO and GRPO+OPSD under the evaluated conditions.

Discussion Summary

  • Reviewer xqyP acknowledged that the additional baseline comparisons, alternative objectives, and multi-seed evaluation strengthened the experimental evidence, and raised their rating to 3.
  • Reviewer UyMB: In response to the follow-up concerns, we provided further explanations and evidence concerning expert PI results and the learning value of weak-policy trajectories. We believe these additions address the remaining concerns.
  • Reviewer Qkur: We appreciate the reviewer’s positive assessment and provided detailed responses to their questions and concerns, supported by additional mechanism analysis and robustness experiments.

Together, these clarifications and experiments provide clearer methodological justification, more transparent comparisons, and stronger evidence for PILOT’s gating mechanism and applicability across the evaluated settings. All corresponding revisions have been incorporated into the updated manuscript, with all modifications highlighted in blue to facilitate review. We believe these revisions have strengthened our work. We sincerely thank all reviewers again for their valuable feedback and constructive engagement, and remain happy to provide further clarification on any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.