acceptodds
Under review as a conference paper at ICLR 2027

Latent On-Policy Self-Distillation

Abstract

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student’s own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher’s privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_12XiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to learn privileged context from prior experience for on-policy self-distillation. The proposed method, LOPD, uses a trainable composer to compress retrieved successful trajectories into continuous latent tokens, which condition a frozen teacher to provide token-level supervision on the student’s own rollouts. The student and composer are jointly optimized through distillation, with an outcome-weighted privilege-margin constraint and an initialization anchor intended to prevent the teacher’s supervision from becoming uninformative.

Critical strengths
  1. A sensible and conceptually appealing approach. Using a trainable composer to encode retrieved experience into latent privileged context is an elegant extension of on-policy self-distillation. This design allows the representation of experience to be optimized directly for the distillation objective, providing a clear motivation for the proposed method.

  2. Substantial empirical evaluation. The paper evaluates the method on multiple benchmarks spanning tool use and code generation, with comparisons against a broad set of relevant baselines. The overall experimental coverage and reported performance improvements provide reasonable empirical support for the method’s effectiveness.

Critical weaknesses
  1. The effects of experience selection and latent conditioning are not sufficiently disentangled. Latent conditioning is a plausible and potentially useful design. However, we note that several baselines differ from LOPD in how they obtain and select the experiences provided to the teacher (Appendix B.1). The observed gains may therefore reflect two potentially intertwined factors: (i) which experiences the teacher receives, and (ii) how those experiences are represented and used to condition the teacher. The proposed composer primarily addresses the second factor, while the baseline comparisons also vary the first, making it difficult to attribute the improvements specifically to latent conditioning. Figure 3 provides useful ablations of composer training and the privilege margin, but all variants still use latent context. A textual-context ablation within the LOPD pipeline, using the same retrieval method, consistent experience-bank construction and update rules where applicable, and matched teacher and distillation settings, would provide stronger evidence for the contribution of latent conditioning.

  2. Limited evidence for training stability and reproducibility. Jointly optimizing the student and composer introduces a risk of degenerate teacher–student agreement, making the privilege-margin constraint particularly important. Although the margin ablation is informative, the paper does not clearly report the number of independent training seeds or variability across runs, and the statistical meaning of the shaded regions in Figure 4 is unspecified. Results across multiple training seeds would help assess the reliability of the gains. Additionally, tracking the achieved margin and dual variable would clarify whether the constraint is consistently satisfied and whether the joint optimization remains stable throughout training.

Questions for authors
  1. Could the authors add a textual-context ablation within the LOPD pipeline, using the same retrieval method and consistent procedures for constructing and, if applicable, updating the experience bank, while matching the teacher and distillation settings? Please clarify whether the bank is updated during training; where feasible, matching the retrieved experiences would help more directly isolate the contribution of the composer.

  2. Could the authors report results across multiple independent training seeds, including the number of runs and variability, and clarify what the shaded regions in Figure 4 represent? Reporting how the achieved margin and dual variable evolve during training would also help assess whether the constraint is consistently satisfied and whether the joint optimization remains stable.

Overall, we believe this paper makes an interesting contribution and would be willing to raise our score to support acceptance if the above concerns are satisfactorily addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 12Xi (1/2)

We sincerely thank the reviewer for the careful and constructive review, and for finding latent conditioning an elegant extension of on-policy self-distillation! In response, we provide a textual-context ablation within the LOPD pipeline, results across three independent training seeds, and the training dynamics of the achieved margin and the dual variable. We address each comment in detail below.


W1 & Q1 Disentangling experience selection from latent conditioning: a textual-context ablation within the LOPD pipeline with matched retrieval, bank, teacher, and distillation settings; and whether the bank is updated during training.

Thank you for this insightful suggestion, which helps isolate the core contribution of our method!

In the main comparison, each baseline obtains its privileged information as designed in its original paper (Appendix B.1), since what the teacher receives is an integral part of each method; notably, OPSD and SDFT even condition on oracle traces partly generated by stronger models, whereas LOPD uses only the model's own rollouts. We nevertheless agree that a controlled comparison is the most direct evidence, and run it as follows.

With the experiences held fixed, latent conditioning clearly outperforms textual conditioning. Following the reviewer's suggestion, we run a textual-context ablation within the LOPD pipeline (Qwen3-4B, EnvScaler). The teacher receives the same top-3 experiences, retrieved by the same method from the same bank, but as raw text placed directly in its context instead of latent tokens; the teacher (the same frozen backbone), the distillation objective (reverse KL with the top-MM teacher logits plus a tail bucket), and the training budget are all unchanged. Since the textual context is fixed, there is no composer to optimize, so the cold start and the margin do not apply.

How the teacher receives the same top-3 experiencesEnvScaler
Base model (no distillation)48.6
Text50.6
Latent (LOPD)63.7

Holding the experiences fixed, textual conditioning barely improves over the base model (50.6 vs. 48.6), whereas latent conditioning reaches 63.7, an improvement of 13.1 points with exactly the same experiences.

For completeness, which experiences are provided also matters: in our response to Reviewer CqJo, replacing the top-3 retrieval with no retrieval or random retrieval reduces LOPD to 49.7 and 50.2. Both factors therefore contribute, and latent conditioning is what turns relevant experiences into effective supervision.

The bank is fixed during training. It is built offline from the model's own successful rollouts on the training split before LOPD training and is never updated (Sec. 3.2 and Appendix A.4); all variants above share this same bank.

We hope this clarifies the contribution of latent conditioning, and we thank the reviewer again for suggesting this controlled comparison.

AuthorsOpenReview ↗

Response to Reviewer 12Xi (2/2)

W2 & Q2 Training stability and reproducibility: results across independent training seeds, the meaning of the shaded regions in Figure 4, and how the achieved margin and the dual variable evolve during training.

Thank you for this constructive suggestion on the reliability of our results!

LOPD's gains are stable across independent training seeds. We rerun the main setting (Qwen3-4B, trained on EnvScaler) with three independent training seeds and evaluate each resulting student on EnvScaler and on the held-out ACEBench:

Seed 1Seed 2Seed 3Mean ± stdBase modelGRPO
EnvScaler63.163.464.463.7 ± 0.748.661.8
ACEBench58.659.361.359.7 ± 1.450.656.0

On both benchmarks, the variation across seeds is small relative to the gains over the base model and the strongest baseline.

Figure 4: the shaded regions show the fluctuation within a moving window of a single training run, i.e., the variation of the evaluation results across neighboring checkpoints around the smoothed curve, rather than the variance across seeds, which is reported above.

The margin constraint is effectively enforced, and the joint optimization remains stable. We track the achieved margin Δ(ϕ)\Delta(\phi) (Eq. 12) and the dual variable β\beta over a representative training run with m=0.05m = 0.05:

Training progress0%25%50%75%100%
Achieved margin Δ(ϕ)\Delta(\phi)0.0760.0620.0530.0430.051
Dual variable β\beta00.0480.1170.1830.164

Δ(ϕ)\Delta(\phi) starts well above mm after initialization and is gradually pulled down by the distillation gradient. As it approaches mm, β\beta grows and pulls it back: Δ(ϕ)\Delta(\phi) briefly dips to 0.043 and then recovers to 0.051, while β\beta stays bounded and eases from 0.183 to 0.164 once the constraint is restored. The margin thus remains around mm throughout training, and β\beta neither diverges nor oscillates, so the joint optimization stays stable. This is exactly how the constraint keeps the teacher from collapsing toward the student, the degenerate agreement that otherwise lowers performance to 55.1 without the margin (Figure 3).


We hope these additional results address the reviewer's concerns. In the revision, we have included the textual-context ablation (Sec. 4.3, Table 4), the results across independent training seeds (Appendix B.6), and the dynamics of the achieved margin and the dual variable (Appendix B.7), and clarified the meaning of the shaded regions in Figure 4 (Appendix B.4). We would be glad to discuss any remaining questions.

Thank you for the detailed rebuttal and additional experiments. The controlled textual-context ablation provides useful evidence for the contribution of trainable latent conditioning under matched retrieval settings, although retrieval differences across existing baselines remain an open limitation. That said, we appreciate the authors' efforts to address our concerns and find the additional evidence convincing overall. We therefore raise our rating from 2 to 3 and support acceptance.

AuthorsOpenReview ↗

Thank you for your follow-up and support

We sincerely thank the reviewer for the thoughtful engagement, for raising the score, and for supporting acceptance. We are glad that the controlled textual-context ablation helped clarify the contribution of latent conditioning.

We agree that how privileged information is obtained and selected across self-distillation methods is an interesting question in its own right. Since each existing method defines what its teacher receives as part of its own design, we compared against them as originally proposed and used the controlled ablation within LOPD to isolate the effect of latent conditioning. A unified comparison of these design choices across methods would indeed be a valuable direction for this line of work.

Thank you again for your constructive feedback, which has helped strengthen the paper.

Reviewer_CqJoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LOPD, which learns the privileged context used by a self-teacher instead of relying on predefined answers or skills. It compresses retrieved experiences into latent tokens and jointly optimizes the context composer and student through on-policy distillation. A reward-based margin prevents the teacher from collapsing toward the student. Experiments across tool-use and coding benchmarks show consistent improvements over existing methods. Ablations further demonstrate the importance of jointly learning the context.

Critical strengths

I think learning the privileged context is an interesting and useful idea. It addresses a limitation of existing self-distillation methods while keeping student inference unchanged.

The experimental results are generally strong across different models and benchmarks. The ablations also provide useful evidence that joint optimization improves performance over a frozen composer.

Critical weaknesses

It is unclear how much of the improvement actually comes from the retrieved experiences. The paper does not compare against a composer with no retrieval or randomly shuffled experiences. Such controls would help establish whether the model learns useful information from experience rather than simply benefiting from trainable latent prompts.

The paper appears to select the margin and other hyperparameters based on EnvScaler test-set performance (Figures 3 and 5). This raises some concerns about test-set tuning, especially since the evaluation set contains only 200 tasks.

Questions for authors

How does LOPD perform with random or no retrieved experiences? This would help isolate the actual contribution of experience retrieval.

Were the margin, latent-token count, and retrieval count selected using a separate validation set?

How does the total training cost compare with GRPO and Skill-SD, including composer initialization? The rollout efficiency gains are promising, but computational efficiency remains unclear.

9 comments
AuthorsOpenReview ↗

Response to Reviewer CqJo (1/2)

We sincerely thank the reviewer for the thoughtful review, and for finding the idea of learning the privileged context interesting and our results strong! In response, we provide two retrieval controls (no retrieval and random retrieval), clarify how the hyperparameters were selected together with our preliminary ALFWorld results, and report the total training cost including the composer cold start. We address each comment in detail below.


W1 & Q1 How much of the improvement comes from the retrieved experiences rather than from trainable latent prompts; performance with random or no retrieved experiences.

Thank you for this important question, which goes to the heart of our claim!

The gains come from the content of the retrieved experiences, not merely from extra trainable latent tokens. Following the reviewer's suggestion, we train two controls from scratch (cold start and LOPD training; Qwen3-4B, EnvScaler), changing only what the composer receives: (i) no retrieval, where the composer receives only the task description in place of each retrieved experience, yielding the same number of latent tokens (3×323 \times 32), i.e., a task-conditioned trainable latent prompt; and (ii) random retrieval, where three experiences are drawn at random from the bank instead of the top-3.

Composer inputTop-3 similarityEnvScaler
No retrieval (task description only)–49.7
Random retrieval0.411 ± 0.06750.2
Top-3 retrieval (LOPD)0.877 ± 0.04363.7
Base model–48.6

Both controls barely improve over the base model (49.7 and 50.2 vs. 48.6), whereas top-3 retrieval reaches 63.7. A trainable latent prompt alone, or experiences unrelated to the task, provides almost no benefit; the gains of LOPD come from retrieving and learning from relevant experiences.

This is further supported by the bank-size ablation we conducted for Reviewer 4KS1. There, we subsample the bank and retrain LOPD from scratch. As both the volume and the relevance of the retrieved experiences decrease, performance drops accordingly:

Bank size25%50%100%
Top-3 similarity0.8040.8420.877
EnvScaler56.462.163.7

Combined with the random-retrieval result, this shows that LOPD's gains are driven by the relevance of the retrieved experiences, as measured by their similarity to the task, and remain robust to a moderate drop in this similarity.

Together with the simpler compressors from our response to Reviewer 4KS1, a clear picture emerges. LOPD's gain comes from learning to extract privileged information from experience, and this fails in two ways. Without relevant experiences, there is nothing useful to extract, so the composer cannot learn extraction at all (49.7 with no retrieval, 50.2 with random retrieval). With relevant experiences but a single vector per experience (mean pooling or the last token), the compressor lacks the room and the learnable mechanism to extract them (49.3 / 51.1). Only when both are present does the gain appear:

No relevant experienceRelevant experiences
One vector per experience–49.3 / 51.1
QFormer (3×323 \times 32 tokens)49.7 / 50.263.7

We have added both controls to Sec. 4.3 (Table 4a), together with the bank-size ablation, as an ablation of the retrieved experiences.

AuthorsOpenReview ↗

Response to Reviewer CqJo (2/2)

W2 & Q2 Whether the margin, latent-token count, and retrieval count were selected on the EnvScaler test set, and whether a separate validation set was used.

Thank you for raising this important concern about evaluation rigor.

The hyperparameters were not selected on the EnvScaler test set. Our work began on ALFWorld [1] from AgentGym [2], which served as our development environment and is entirely separate from EnvScaler and all reported benchmarks. After validating the method there, we moved to the more recent and diverse settings reported in the paper, and therefore did not include these preliminary results. KK, nretn_{\text{ret}}, and mm were fixed during this ALFWorld stage and then kept unchanged across all models and domains. With Qwen3-4B-Instruct-2507 on ALFWorld, we first swept KK with nret=1n_{\text{ret}} = 1:

Base modelK=8K=8K=16K=16K=32K=32K=64K=64
ALFWorld success (%)24.534.042.554.053.0

K=32K = 32 performed best, with no further gain at 64. With K=32K = 32, we then swept nret∈{1,2,3}n_{\text{ret}} \in \lbrace 1, 2, 3 \rbrace and selected the best setting, nret=3n_{\text{ret}} = 3. The margin m=0.05m = 0.05 was set in the same preliminary stage.

We apologize that our presentation may have given this impression. Figures 3 and 5 are post-hoc ablation and sensitivity analyses, conducted after the hyperparameters had been fixed, rather than the basis for choosing them. We have stated the provenance of these hyperparameters in Sec. 4.1 and Appendix B.5, and revised the related wording accordingly.

Two further observations support this. First, the same hyperparameters are used without re-tuning for all backbones (Qwen3-4B, Qwen3-8B, and OLMo-3-7B) and both training domains (tool use and coding), and LOPD achieves the best aggregate result on every benchmark, including the held-out BFCL-v3 and ACEBench. Second, mm acts only as a lower bound that keeps the teacher from collapsing toward the student, rather than a finely tuned value: in Figure 3, every mm from 0.02 to 0.20 leads to effective joint optimization, with the student surpassing the frozen-composer variant.


Q3 Total training cost, including composer initialization, compared with GRPO and Skill-SD.

Thank you for this practical question!

Including the composer cold start, LOPD's total training cost is lower than that of both GRPO and Skill-SD. We train each method on EnvScaler with Qwen3-4B until its training reward converges, and report the full wall-clock time on one node with 8×A100 (80GB) GPUs:

MethodCold startTrainingTotalGPU hours
GRPO–9.72 h9.72 h77.8
Skill-SD–8.19 h8.19 h65.5
LOPD2.39 h3.16 h5.55 h44.4

Compared with a standard OPSD-style step, on which LOPD builds, each LOPD step additionally runs the composer to encode the retrieved experiences and updates it. However, LOPD extracts a denser learning signal from each rollout and therefore converges much faster, so its rollout efficiency does translate into compute efficiency: including the cold start, LOPD needs 57% of the time of GRPO and 68% of that of Skill-SD.

Like other self-distillation methods, LOPD needs privileged information beyond the on-policy rollouts themselves. OPSD and SDFT prepare oracle traces in advance, part of which come from stronger models (Appendix B.1); Skill-SD prepares a skill library; and SDPO obtains its privileged information online from successful sibling rollouts, which ties it to group sampling at every training step. LOPD's bank is instead built once, offline, from the model's own rollouts without any external API. Such one-off preparation is excluded from the table for all methods.


In summary, the revision includes: (1) the retrieval controls, combined with the bank-size ablation, as an ablation of the retrieved experiences (Sec. 4.3); and (2) an explicit statement of the provenance of KK, nretn_{\text{ret}}, and mm (Sec. 4.1 and Appendix B.5), with the related wording revised. We hope these address the reviewer's concerns, and we would be glad to discuss any remaining questions.

[1] Shridhar et al. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR 2021.

[2] Xi et al. AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments. ACL 2025.

AuthorsOpenReview ↗

Brief follow-up on our responses to Reviewer CqJo

Dear Reviewer CqJo,

Thank you again for your thoughtful review. As the discussion period progresses, we would like to kindly check whether our responses and revised PDF have addressed your concerns. In brief:

  • Retrieval Controls: Added from-scratch no-retrieval (49.7) and random-retrieval (50.2) controls on EnvScaler (Sec. 4.3), confirming that LOPD's gain (63.7 vs. Base 48.6) stems from retrieving relevant experiences rather than generic trainable latent prompts.
  • Hyperparameter Provenance: Clarified that K=32,nret=3,m=0.05K=32, n_{\text{ret}}=3, m=0.05 were fixed on a separate development environment (ALFWorld) and kept unchanged across all models and domains, with Figures 3 and 5 serving as post-hoc analyses rather than test-set tuning (Sec. 4.1, Appendix B.5).
  • Full Training Cost: Reported the complete 8×A100 wall-clock cost including cold start, showing LOPD (5.55 h / 44.4 GPUh) requires only 57% of GRPO's time (9.72 h / 77.8 GPUh) and 68% of Skill-SD's (8.19 h / 65.5 GPUh).

We would be happy to clarify anything further. Thank you for your time!

Reviewer_4KS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes replacing the handcrafted "privilege" in privileged distillation with latent embeddings, basically letting an encoder learn whatever best suits the model. this context embedding (instead of the raw tokens) is then fed into the policy prediction to guide the learning of the student. Specifically, they create a "trajectory bank" with succesful trajectories and then retrieve the top-J relevant trajectories for each task. These are then encoded into a continuous representation, and the token-level embeddings are length neutralized using learnable-query cross attention (QFormer-style). Feeding the teacher this information guides the student better than choosing custom privilege for each task separately.

The experiments are shown on both agentic tool-use and coding tasks. While the performance on EnvScaler is quite impressive, other datasets show a consistent improvement as well. My biggest concern is the lack of analysis or insight into the choice of trajectory bank or how sensitive it is, which are elaborated in other sections next.

Critical strengths
  • The paper tackles an important limitation of on-policy self distillation papers, which is to choose the right form of privileged information which helps teacher gather enough expertise to guide the student while also not making it trivially easy for the student to match teacher. Instead, they rely on learnable prevelige information, which is just an encoded form of retrieved successful trajectories. The flexibility here is that the encoder and the subsequent learnable query matrices can choose to pick or drop certain concepts or information based on the task.

  • The formulation is domain-agnostic, and the choice based on creating an offline trajectory bank is well explained.

  • Sufficient ablations are provided which help us understand the critical components of the paper, like choice of #nearest neighbors or token lengths.

Critical weaknesses
  1. My biggest concern is the lack of sufficient analysis into the trajectory bank constructed at the beginning of the training. Specifically,

a) The authors did not explain how do we choose the subset of successful tasks which would be populated in this bank. b) If this is fixed at the beginning of the training, or do we continuously or periodically update the bank to add more trajectories as training progresses. c) How "in-distribution" do these seed trajectories need to be, compared to the domain of the tasks in the training?

  1. The performance of LOPD is quite strong on EnvScaler, but the gains are limited on BFCL-v3 and other coding-heavy tasks. The authors could provide some intuition or reasoning behind why, for example, RLVR/GRPO is strong on coding tasks, and if we can potentially combine these approaches to yield complementary benefits.
Questions for authors
  1. Key question repeated from above: How "in-distribution" do the seed trajectories need to be, compared to the domain of the tasks in the training? Additionally, the effect of "volume" of seed tasks in the trajectory bank is unspecified.

  2. Apart from using learnable query matrix to handle the variable-state latent embeddings, the authors should also experiment with more simpler alternatives such as mean pooling or classifier embedding as baselines.

  3. Behavioral internalization is interesting - does this trend hold for other benchmarks as well? Or is this just a effect of improved overall accuracy in EnvScaler?

  4. Other relevant papers needs to be cited - starting from experience replay in Q-learning - to provide a more rounded context on using episodic memory for agentic training.

Minor

  1. Eq 1 overloads the notation πθS\pi_\theta^S is used to denote policy conditioned on task and action/state.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4KS1 (1/3)

We sincerely thank the reviewer for the careful reading and the constructive feedback! In response, we provide a bank-size ablation with retrieval-similarity statistics, two simpler compressor baselines, and the behavioral analysis of Table 3 on ACEBench, and we extend the related work on experience replay and episodic memory. We address each comment in detail below.


W1 & Q1 Analysis of the trajectory bank: (a) how the trajectories that populate the bank are selected, (b) whether the bank is fixed or updated during training, (c) how in-distribution the seed trajectories need to be, and the effect of the bank's volume.

Thank you for this insightful question, which touches the core of our design. We address each part in turn.

(a) The bank contains only the model's own successful rollouts on the training split. As described in Sec. 3.2 and Appendix A.4, no evaluation task, external data, or stronger model is involved in building it. We are happy to provide further details: for EnvScaler, each training task is rolled out multiple times, and one successful trajectory (reward ≥ 0.5) is kept per task. For Qwen3-4B, this yields 1,824 trajectories, covering 1,824 of the 2,349 training tasks (77.7%).

(b) The bank is fixed throughout training, by design. It is built offline before LOPD training and never updated. Since the source of experience stays unchanged, any gain must come from learning how to represent the experience, rather than from collecting more of it.

(c) & Q1: How in-distribution the experiences need to be, and the effect of the bank's volume. Since "in-distribution" can be defined at many granularities, we measure it directly as how close the retrieved experiences are to each specific task, i.e., the cosine similarity between the task and its top-3 retrieved experiences. All experiences come from the training split that all baselines already use. To vary this similarity, together with the volume of the bank, we subsample the bank and retrain LOPD from scratch (Qwen3-4B, EnvScaler):

Bank sizeTrajectoriesTop-3 similarityEnvScaler
25%4560.804 ± 0.04856.4
50%9120.842 ± 0.04262.1
100%1,8240.877 ± 0.04363.7
GRPO––61.8
Base model––48.6

The gain follows this similarity. LOPD still matches GRPO at a similarity of 0.842 (62.1 vs. 61.8): it does not need the closest experiences to learn effectively. At 0.804, it still improves over the base model, though by a smaller margin (56.4 vs. 48.6). When the experiences are unrelated to the task, as with random retrieval in our response to Reviewer CqJo (0.411), the gain vanishes. This steady dependence on similarity shows that LOPD genuinely draws on the content of the experiences.

Meanwhile, although the bank is built only from EnvScaler, the trained student transfers to the held-out BFCL-v3 and ACEBench, where it achieves the best average results.

All in all, LOPD needs no data beyond the training split shared with all baselines. Measured by their similarity to the task, the experiences do not need to be the closest ones: LOPD remains effective as the top-3 similarity drops from 0.877 to about 0.80. Its gains are rooted in the content of the experiences themselves.

We have added the construction details to Appendix A.4 and the bank-size ablation to Sec. 4.3 (Table 4). We hope this clarifies the role of the trajectory bank, and we thank the reviewer again for prompting this analysis!

AuthorsOpenReview ↗

Response to Reviewer 4KS1 (2/3)

W2 Limited gains on BFCL-v3 and coding tasks; why RLVR/GRPO is strong on coding, and whether the two approaches can be combined.

Thank you for this thoughtful suggestion! We address it from three perspectives.

1. On these benchmarks, LOPD's gains remain consistently larger than GRPO's. We understand that these gains may look limited next to the +15 points on EnvScaler. Measured against the base model, however, LOPD improves more than GRPO in each of the following settings:

SettingGRPOLOPD
BFCL-v3 Avg (Qwen3-4B)+2.37+4.50
BFCL-v3 Avg (Qwen3-8B)+0.62+1.50
LiveCodeBench Avg (Qwen3-4B)+2.68+3.17
LiveCodeBench Avg (OLMo-3-7B)+1.95+4.64
EvalPlus Avg (Qwen3-4B)+0.55+2.57
EvalPlus Avg (OLMo-3-7B)+2.39+3.13

2. Two intuitions behind this gap. First, in the agentic domain, BFCL-v3 and ACEBench are held-out benchmarks whose tools, environments, and interaction formats differ from EnvScaler, so their gains reflect transfer rather than in-domain fitting. Second, LOPD's advantage grows with the complexity of the experience. In agentic tasks, trajectories record rich interactions with the environment, such as tool responses, state changes, and recovery from errors. Such experience is long and hard to exploit directly as text, and this is exactly where LOPD's learnable privileged context helps: LOPD learns to compress the experience into a compact latent context that the teacher can turn into dense, token-level supervision. Single-turn coding tasks involve little environment interaction, so the experience is simpler, and their exact verifiers already provide a precise outcome signal, which makes RLVR particularly effective. Even so, LOPD still improves more than GRPO on coding, as shown above.

3. The two approaches are complementary, and combining them is feasible and simple. Several recent works already combine on-policy self-distillation with RLVR, including CrEST [1], RLSD [2], and RLCSD [3]: the verifier decides whether a trajectory is reinforced, and the self-teacher decides which tokens within it deserve more credit. Given these works, we believe LOPD can be combined with GRPO in the same way with little effort, since LOPD already computes the per-token privilege δt,n\delta_{t,n} (Eq. 11) that these methods use to rescale the GRPO advantage:

A^t,n=A⋅clip(exp⁡(sign(A)δt,n),1−ϵ,1+ϵ).\hat A_{t,n} = A \cdot \mathrm{clip}\big(\exp(\mathrm{sign}(A)\delta_{t,n}), 1-\epsilon, 1+\epsilon\big).

LOPD's teacher is even better suited to this role than the self-teachers in [1–3]: its margin objective (Eq. 12) already trains it to agree with the verifier, and, unlike them, it does not require ground-truth answers or tool outputs.

We thank the reviewer for this perspective and have added this discussion to Sec. 5 as future work.

[1] Wang et al. Teach the Magnitude, Not the Direction: Verifier-Anchored Credit Assignment for Multi-Turn Multi-Step LLM Agents. EMNLP 2026.

[2] Yang et al. Self-Distilled RLVR. arXiv:2604.03128, 2026.

[3] Pan et al. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. arXiv:2606.11709, 2026.

AuthorsOpenReview ↗

Response to Reviewer 4KS1 (3/3)

Q2 Simpler alternatives to the learnable query matrix, such as mean pooling or a classifier embedding, as baselines.

Thank you for this constructive suggestion!

Both simpler compressors fall far short of the QFormer. We replace the QFormer with mean pooling and with the last-token hidden state (the counterpart of a classifier embedding in a decoder-only model), keep all other settings unchanged, and run the full LOPD training (Qwen3-4B, EnvScaler):

CompressorEnvScaler
Mean pooling49.3
Last token51.1
QFormer (ours)63.7
Base model48.6

All variants are trained end-to-end under the same objective, so the gap comes from the compressor itself, in two ways. First, capacity: mean pooling and the last token compress a long multi-turn trajectory into a single embedding, so whatever is learned, the privileged information from each experience is bounded by one vector. Repeating this vector to fill more latent slots would not change this, since the copies carry identical information. Second, learnable extraction: their aggregation rule is fixed, as mean pooling weights all tokens equally and the last token relies on the encoder to route everything into one position, whereas the QFormer's queries learn what to extract from each experience. Between the two, the last token does slightly better, since the encoder can at least learn what to route into that position, whereas mean pooling has no such flexibility. As stated in Sec. 3.2, we do not claim the attention block itself as a contribution: LOPD's contribution is to make the privileged context learnable, which requires a compressor that can learn what to extract and has room to hold it; the QFormer is a simple instance.


Q3 Whether behavioral internalization holds on other benchmarks, or is merely an effect of improved accuracy on EnvScaler.

Thank you for this interesting question!

Yes, the trend also holds on ACEBench, and it is not merely a side effect of higher accuracy. Using the same configuration as Table 3 (Qwen3-4B):

BaseBase + ComposerLOPD student
ACEBench score50.655.360.6
Interaction steps9.2810.4410.10
Tool calls / step0.7620.7020.695
First-step length (chars)4,6824,5244,213
Score / tool call0.1200.1240.135
Repeated tool calls1.881.361.32

All five behavioral metrics move in the same direction as on EnvScaler. The key column is Base + Composer: its backbone is not trained at all, yet the latent context alone already produces the same behavioral shift, which the student then inherits while surpassing the teacher (60.6 vs. 55.3). Moreover, score per tool call and repeated calls measure efficiency, which higher accuracy does not directly imply.


Q4 Citing related work on experience replay and episodic memory.

Thank you for this helpful suggestion! We have added a discussion to Sec. 2, covering methods that use experience directly as training data, from experience replay [4, 5] and self-imitation [6] to their recent agentic and reasoning counterparts [7, 8], and methods that retrieve it at inference time, from episodic memory [9] to language-agent memory [10, 11]. Within this line of research on learning from experience, LOPD shares the same goal of reusing past experience, but uses it in a different way: experience only conditions the teacher during training, and the result is internalized into the policy, so no memory is needed at deployment. We would welcome any further works the reviewer may suggest.


Minor Notation overload in Eq. 1.

Thank you for the careful reading! We have denoted the trajectory distribution separately as pθ(τ∣x)p_\theta(\tau \mid x) in Eq. 1 and the related equations.


In summary, the revision includes: (1) bank construction details and ablations of the bank size and the compressor (Appendix A.4, Sec. 4.3); (2) a discussion of combining LOPD with RLVR (Sec. 5); (3) related work on experience replay and episodic memory (Sec. 2); and (4) the notation fix in Eq. 1. We hope these address the reviewer's concerns, and we would be glad to discuss any remaining questions.

[4] Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 1992.

[5] Mnih et al. Human-level control through deep reinforcement learning. Nature, 2015.

[6] Oh et al. Self-Imitation Learning. ICML 2018.

[7] Qin et al. Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning. ICLR 2026.

[8] Zhan et al. ExGRPO: Learning to Reason from Experience. ICLR 2026.

[9] Pritzel et al. Neural Episodic Control. ICML 2017.

[10] Liu et al. Contextual Experience Replay for Self-Improvement of Language Agents. ACL 2025.

[11] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory. ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We sincerely thank all reviewers and the Area Chair for their time and constructive feedback. We are encouraged that the reviewers found learning the privileged context "an interesting and useful idea" (CqJo) and "an elegant extension of on-policy self-distillation" (12Xi) that "tackles an important limitation of on-policy self-distillation" with a "domain-agnostic" formulation (4KS1), with "generally strong" results across models and benchmarks (CqJo). We are also glad that our responses addressed the concerns of Reviewers 12Xi and CqJo, both of whom raised their ratings. Below we summarize the main additions and revisions; all changes are highlighted in blue in the revised PDF.

Additional evidence

  • Where the gain comes from (Table 4a): no-retrieval and random-retrieval controls barely improve over the base model (49.7 and 50.2 vs. 48.6), while top-3 retrieval reaches 63.7. Subsampling the experience bank further shows that the gain tracks experience relevance (62.1 at 50% size and 56.4 at 25%) and does not strictly require a large bank or the closest in-distribution match (Reviewers CqJo and 4KS1).
  • How experience is represented (Table 4b): with the same top-3 experiences, a raw-text teacher reaches 50.6 and single-vector compressors (mean pooling 49.3, last token 51.1) stay near the base model, whereas LOPD reaches 63.7 (Reviewers 12Xi and 4KS1). These additional experiments also allow us to uncover a compressor-design insight (Sec. 4.3): an effective compressor needs both sufficient capacity and a learnable aggregation, of which the QFormer is a simple instance.
  • Stability and reproducibility: results across three training seeds (63.7 ± 0.7 on EnvScaler, 59.7 ± 1.4 on ACEBench; Appendix B.6) and the dynamics of the achieved margin and dual variable (Appendix B.7) (Reviewer 12Xi).
  • Experimental rigor: the hyperparameters were fixed on a separate development environment before the reported experiments (Sec. 4.1, Appendix B.5), and the bank construction is fully specified (Appendix A.4) (Reviewers CqJo and 4KS1).

Manuscript revisions

  • Sec. 2: a new paragraph on learning from experience (experience replay, self-imitation, episodic memory, and language-agent memory).
  • Eq. 1 and related equations: the trajectory distribution is denoted separately as pθ(τ∣x)p_\theta(\tau \mid x).
  • Abstract, Sec. 1, and Sec. 5: the efficiency claims now state explicitly that they refer to the on-policy rollout budget during training.
  • Sec. 4.1 and Appendix B.5: the provenance of the hyperparameters, with the related wording in Sec. 4.3 and Appendix A.2 revised accordingly.
  • Sec. 4.3 and Table 4: the experience and representation ablations above.
  • Sec. 5: a discussion of combining LOPD with RLVR.
  • Appendix A.4, B.4–B.7: bank construction details, ablation setups and the meaning of the shaded regions in Figure 4, hyperparameter selection, multi-seed results, and margin dynamics.

Overall, the method, training setup, and main results (Tables 1–3) remain unchanged. All added experiments and analyses serve to verify the reliability of our method and directly substantiate the paper's core claims, confirming that the gains come from experience rather than extra parameters, that learning the latent representation matters, and that the joint optimization is stable. Combined with the consistent improvements over GRPO and self-distillation baselines across three backbones and seven benchmarks, these results suggest that the revised manuscript offers a clear and well-supported contribution to LLM post-training.

We would like to express our sincere gratitude to all reviewers and the Area Chair for their time, careful reading, and thoughtful suggestions throughout the discussion period. Their rigorous questions and constructive feedback have substantially strengthened this work, and we deeply appreciate their dedicated service and contributions to the research community.

Sincerely,
The Authors of LOPD

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.