acceptodds
Under review as a conference paper at ICLR 2027

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

Abstract

On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student’s rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, with relative improvements over GRPO of 7% to 58% in average score across settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-turn LLM agents, on-policy self-distillation, agentic RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u7g9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the failure of on-policy self-distillation (OPSD) in training multi-turn LLM agents with privileged information (PI). The authors observe that OPSD teaches the student to mimic the teacher's confident actions without understanding the underlying reasoning, which can degrade performance below that of an untrained model. To solve this, they introduce Privileged Self-Practice (PSP), a framework that shifts PI from the training objective to the rollout sampler. When a task's success rate falls below a specific threshold, PSP uses an analyzer model to write short instructions based on the agent's failures, re-samples trajectories with these instructions in context, and trains using standard Group-Relative Policy Optimization (GRPO).

Critical strengths

PSP integrates into standard RL pipelines (like GRPO) without requiring complex auxiliary loss balancing or cross-view objectives.

The failure analysis of OPSD in agents is concrete, with the Fig. 1 example, the entropy blow-up in Fig. 5, and the link to the irreducible information gap from prior work.

The gate threshold, an “always” variant, removing the reference trajectory, and frozen instruction pools (Table 8) all isolate real design choices. The frozen-instruction result is especially informative: the gain comes from fresh, diverse instructions, not just extra re-rollouts.

Critical weaknesses

The analyzer's success heavily depends on a pre-computed pool of successful reference trajectories generated by the 27B model. The authors' own ablation shows a significant performance drop if the analyzer must infer the correct procedure solely from the student's failures.

The method also introduces additional computational overhead. PSP needs an online analyzer (half the GPUs on AppWorld), extra re-rollouts, and an offline pool of 16 trajectories per task from a 27B model. A compute-matched comparison with GRPO is only partly addressed (the 128-sample check in D.1).

Each number comes from one training run per method and model. The reported ± is over evaluation seeds only, so it does not capture training variance. Several key margins are small (Qwen3-8B average 27.4 vs. 25.6 for GRPO; SWE-bench Qwen3-4B has RLSD ahead of PSP on both metrics). The claim that PSP “consistently outperforms” GRPO is not well supported at this level of evidence.

Questions for authors

The data shows a significant performance drop when the 27B analyzer must write instructions based solely on the student's failures, rather than having access to a successful reference trajectory. How viable is PSP in novel or highly complex domains where generating these successful reference trajectories upfront is impossible?

PSP requires invoking a 27B analyzer model during the training loop and re-rolling failed groups. How does the wall-clock time and total FLOP count of PSP compare to plain GRPO or RLSD, and is the performance gain strictly worth the computational premium in resource-constrained environments?

You avoided using frontier models to limit compute. If you scale the analyzer up to a frontier model (e.g., GPT-4o or Claude 3.5 Sonnet), does the quality of the instructions improve enough to rescue the "failures only" ablation, or does the student model hit a capacity ceiling regardless of instruction quality?

The ablations show that using frozen instructions pre-written by the base model provides little gain unless a massive pool of 64 instructions per task is used. Why does the student model overfit so rapidly to a single static instruction, and does this imply that PSP acts more as a regularizer through text variance than as a knowledge injector?

9 comments

We thank the reviewer for the careful and constructive comments. We address each point below and have run new experiments where more evidence was requested.

Before the individual points, we would like to restate the core claim of the paper, because it bears on several of them. The paper concentrates on one question: given privileged information (PI) about a training task, how should it be trained into the weights of a multi-turn agent? OPSD distills it, RLSD and OPID use it to reweight the advantage, and PSP uses it in the rollout sampler. Where the PI comes from is a separate matter. Prior work in this line obtains it from a stronger model (Skill-SD, OPID) or from gold solutions (OPSD). We use an open 27B model with reference trajectories as an inexpensive and reproducible method, and in practice the PI can be an instruction written by expert engineers or by a giant frontier model. All methods in the paper receive exactly the same PI, so the comparison is only about how the PI is trained into the weights.

W1 & Q1: PSP relies on reference trajectories from the 27B model, which may not be available in a new domain.

As stated at the beginning, the paper is about how PI should be trained into the weights of a multi-turn agent. The Qwen3.6-27B analyzer is a moderately sized open-weight model, which keeps the experiments reproducible, and its reference trajectories are only an inexpensive way to simulate such information. Real applications can use better instructions, for example hand-written ones or ones written by a frontier model.

Table R1 puts the ablation the reviewer points to next to the other ways of producing the instruction.

Table R1. Test average on AppWorld by how the instruction is produced. The numbers are collected from Tables 1, 2, 5 and 9 of the paper.

instruction written byreference it reads27B call during trainingQwen3-4BQwen3-8B
no instruction (GRPO)-no10.625.6
27B analyzernone, failures onlyyes10.525.5
the student itselfone stored trajectory per taskno17.6-
27B analyzerpool of 16 trajectories per taskyes16.727.4

(1) Without the reference, the instruction is worse and PSP falls back to the level of GRPO. This shows that a weak instruction removes the gain. (2) The requirement could be very light. One stored trajectory per task is enough, and the 27B model does not have to be called during training (Appendix D.2).

What PSP needs in a new domain is a good instruction for the training tasks, and a reference trajectory is one way to obtain it. First, many agent benchmarks already come with reference solutions, from which such instructions can be written. Second, in practice many good instructions do not rely on a reference solution at all, e.g., they are written by engineers who know the task. Whether a reference is available is therefore a separate question from the one this paper studies, which is how to train the PI into the weights. All methods in the paper receive their instructions from the same mechanism, so they are compared at the same instruction quality.

W2 & Q2: PSP adds compute, and the comparison with GRPO is not compute-matched.

The gain of PSP comes from using the privileged information and not from the extra compute. On Qwen3-4B, GRPO does not reach the accuracy of PSP when it is given the same number of rollouts, twice the rollouts per step, or its full budget of 500 steps, as the comparisons below show. The most direct comparison is validation accuracy at the same number of rollouts. Table R2 gives the best validation TGC reached within a given rollout budget on Qwen3-4B. A GRPO step uses 256 rollouts, and a PSP step uses 256 plus the re-rollouts of the gated groups. We also add a new control, GRPO with n = 16, which uses 512 rollouts per step.

Table R2. Best validation TGC (%) within a rollout budget, Qwen3-4B.

rollouts20k40k60k80k100k256k
GRPO (n = 8)8.410.910.915.420.0-
GRPO (n = 16, new)8.110.211.611.611.612.3
PSP9.817.523.226.330.2-
PSP, student as analyzer (no 27B call in training)14.724.925.328.429.8-

PSP is ahead at every budget, both with the 27B analyzer and with the student as the analyzer, and giving GRPO twice the rollouts per step does not change its curve. On the test sets GRPO with n = 16 scores 7.5 on average, against 7.4 ± 3.1 for n = 8. Table R3 gives the wall-clock cost per step on the same node, together with the test average.

Table R3. Wall-clock cost per step and test average, Qwen3-4B. Seconds per step are measured on the same node and averaged over the whole run of the paper. The test average of GRPO and PSP is over three training runs, and RLSD is the single run of the paper.

methodseconds per stepwhere the overhead comes fromtest Avg
GRPO154-7.4 ± 3.1
RLSD341extra scoring of every rollout with the PI in context, at every step12.1
PSP444re-rollouts and analyzer calls on the gated groups only (86% of the groups at the start, 42% at the end)16.3 ± 1.5

A PSP step takes about 2.9 times as long as a GRPO step, and this overhead decreases during training as fewer groups are gated, whereas the overhead of RLSD is constant. Compared with GRPO, PSP still reaches a given accuracy earlier. It passes the best validation score of GRPO (20.0) after 49k rollouts and 15.1 node-hours, where GRPO needs 98k rollouts and 17.0 node-hours. We report rollouts and node-hours on identical hardware rather than FLOPs. All methods train the same student, so FLOPs are roughly proportional to rollouts.

As for whether the premium is worth it, the gain depends on how often the student's own rollouts succeed. It is large for Qwen3-4B (+8.9 on average) and small for Gemma4-E4B (+1.0) as Table R4 shows.

W3: Each number comes from one training run, so the claim that PSP “consistently outperforms” GRPO is not well supported.

We thank the reviewer for raising this point, and we agree that the reported ± covers evaluation seeds only. We have added two training seeds for GRPO and for PSP on each of the three students, each with its own validation-based checkpoint selection and 5-seed test, as in the paper. With them the picture is clearer. The gain is large on Qwen3-4B, moderate on Qwen3-8B, and within the variation across runs on Gemma4-E4B.

Table R4. Test average, mean ± std over three training runs (the run of the paper and two new seeds). Tables R5 to R7 give all metrics of every run.

studentGRPOPSPdifference
Qwen3-4B7.4 ± 3.116.3 ± 1.5+8.9
Qwen3-8B25.3 ± 2.827.9 ± 0.5+2.6
Gemma4-E4B29.5 ± 2.030.5 ± 1.7+1.0

Table R5. Qwen3-4B, all test metrics of every training run.

methodrunnormal TGCnormal SGCnormal TGC@5normal SGC@5challenge TGCchallenge SGCchallenge TGC@5challenge SGC@5Avg
GRPOpaper13.2 ± 1.74.3 ± 1.428.016.14.6 ± 0.60.7 ± 0.512.05.810.6
GRPOnew seed 19.9 ± 1.41.8 ± 1.322.012.52.6 ± 0.80.0 ± 0.07.22.27.3
GRPOnew seed 25.0 ± 1.90.7 ± 1.013.75.42.5 ± 0.40.0 ± 0.07.20.74.4
GRPOmean ± std over runs9.4 ± 4.12.3 ± 1.821.2 ± 7.211.3 ± 5.53.2 ± 1.20.2 ± 0.48.8 ± 2.82.9 ± 2.67.4 ± 3.1
PSPpaper21.8 ± 2.46.4 ± 2.439.923.29.4 ± 0.43.2 ± 0.619.910.116.7
PSPnew seed 120.0 ± 1.56.8 ± 0.837.521.46.8 ± 1.21.1 ± 0.417.05.814.6
PSPnew seed 224.4 ± 2.010.7 ± 1.841.123.29.5 ± 1.13.2 ± 1.318.29.317.5
PSPmean ± std over runs22.1 ± 2.28.0 ± 2.439.5 ± 1.822.6 ± 1.08.6 ± 1.52.5 ± 1.218.4 ± 1.48.4 ± 2.316.3 ± 1.5

Table R6. Qwen3-8B, all test metrics of every training run.

methodrunnormal TGCnormal SGCnormal TGC@5normal SGC@5challenge TGCchallenge SGCchallenge TGC@5challenge SGC@5Avg
GRPOpaper30.8 ± 1.013.9 ± 2.453.635.714.9 ± 1.15.2 ± 1.530.020.925.6
GRPOnew seed 131.4 ± 2.414.6 ± 4.650.632.110.4 ± 0.72.7 ± 0.624.512.222.3
GRPOnew seed 235.2 ± 2.517.5 ± 5.058.346.414.6 ± 1.45.3 ± 0.428.817.327.9
GRPOmean ± std over runs32.5 ± 2.415.3 ± 1.954.2 ± 3.938.1 ± 7.413.3 ± 2.54.4 ± 1.527.7 ± 2.916.8 ± 4.425.3 ± 2.8
PSPpaper31.6 ± 2.012.1 ± 1.357.739.316.8 ± 0.65.9 ± 0.832.123.727.4
PSPnew seed 133.7 ± 2.015.4 ± 1.658.346.416.6 ± 0.85.5 ± 0.631.920.128.5
PSPnew seed 234.6 ± 1.916.4 ± 3.257.135.716.8 ± 1.17.0 ± 0.931.923.727.9
PSPmean ± std over runs33.3 ± 1.614.6 ± 2.357.7 ± 0.640.5 ± 5.516.7 ± 0.16.1 ± 0.832.0 ± 0.122.5 ± 2.127.9 ± 0.5

Table R7. Gemma4-E4B, all test metrics of every training run. PSP uses γ = 0.3, as in the paper.

methodrunnormal TGCnormal SGCnormal TGC@5normal SGC@5challenge TGCchallenge SGCchallenge TGC@5challenge SGC@5Avg
GRPOpaper33.9 ± 1.318.2 ± 1.354.839.315.3 ± 0.84.3 ± 0.631.920.127.2
GRPOnew seed 136.1 ± 4.719.6 ± 4.460.146.417.8 ± 0.96.2 ± 1.335.223.030.6
GRPOnew seed 236.4 ± 2.719.6 ± 2.558.942.918.3 ± 0.94.9 ± 1.638.625.930.7
GRPOmean ± std over runs35.5 ± 1.419.2 ± 0.858.0 ± 2.842.9 ± 3.617.1 ± 1.65.1 ± 1.035.3 ± 3.423.0 ± 2.929.5 ± 2.0
PSPpaper38.6 ± 1.820.4 ± 4.361.346.420.3 ± 1.77.8 ± 1.438.426.632.5
PSPnew seed 133.8 ± 3.018.9 ± 4.156.546.416.5 ± 1.04.9 ± 1.434.521.629.2
PSPnew seed 235.8 ± 1.920.0 ± 2.657.741.118.1 ± 1.16.3 ± 1.636.523.729.9
PSPmean ± std over runs36.1 ± 2.419.8 ± 0.858.5 ± 2.544.6 ± 3.118.3 ± 1.96.3 ± 1.536.5 ± 1.924.0 ± 2.530.5 ± 1.7

Based on the results, we observe the following. (1) On Qwen3-4B every PSP run is above every GRPO run. (2) On Qwen3-8B, PSP is 2.6 points higher on average. (3) On Gemma4-E4B, the margin is about one point, so there PSP is on par with GRPO. The gain thus decreases as the student's own rollouts succeed more often. The train pass rate at the start of training is 3% for Qwen3-4B, 8% for Qwen3-8B and 11% for Gemma4-E4B, and PSP is designed to add signal where this rate is low.

In short, the margin depends strongly on how well the base model already solves the training tasks, and PSP stays at or above GRPO in every setting. This is different from loss-side methods such as OPSD, which can fall below GRPO. In PSP, the privileged information never enters the loss, and every ungated group is trained exactly as in GRPO. We therefore replace “consistently outperforms GRPO” with “matches or exceeds GRPO in every setting”, with the largest gains where the base model solves the fewest training tasks”, and we now bold all entries that are not significantly different from the best. The new seeds cover AppWorld only, because a SWE-bench run takes much longer and did not fit in the rebuttal period. There the single runs show PSP above GRPO on both students (10.57 against 8.35 on average) and tied with RLSD on Qwen3-4B (5.92 ± 0.37 against 6.20 ± 0.80), as the reviewer notes.

Reviewer_r6ChOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is an interesting paper with useful insight into RL with privileged information (PI) and a strategy for applying it more successfully in longer-horizon settings. Fundamentally the problem is this: when there is no possible way the student (without the PI) would be able to produce the desired behavior given the information it does have available, then we shouldn't try to coerce it to do so; but when it can produce the desired behavior it based on other cues, we should try to train it to do that. The difficulty is in distinguishing these cases. The paper provides a method for doing this, provides some interesting analysis, and shows that it provides some measurable performance improvements.

Unlike some other methods for addressing this problem, the technique in the paper doesn't distort the original objective function, which is important.

Critical strengths

The greatest strength of this paper is that it solves a problem in RL-based post-training by actually understanding the problematic phenomenon and then leveraging that understanding to design a mechanism to fix it.

The paper was also generally well written and easy to follow, with good use of intuition-building examples.

Critical weaknesses

I can't name any major weaknesses. Some smaller (and some tiny) ones are below.

  • It's not clear how to generalize from the performance of this method on the two benchmarks presented. Presumably the world of possible relationships between PI and non-PI settings is huge, and it's likely that this method works better in some circumstances than others. It would have been nice to see that addressed conceptually or empirically or even just raised as a question. For example, teacher-student distillation is common in robotics in cases where the teacher is trained with ground-truth information about the world state but the student learns to map images to actions. In that case, we know that the necessary information is available in the input signal to the student---it's just easier not to do the initial RL in image space. Or, sometimes the student can recover the information it needs from an observation history.

  • It's better practice to, in tables, put in bold all entries that are not significantly different from the best (using an appropriately-selected significance test for the data).

  • Captions for tables 1 and 2 should say what benchmark they apply to

Questions for authors

What do you think about the first "weakness" listed above? That is, are there kinds of PI (either helpful or unhelpful) that are not well represented in your benchmarks? Would it be useful to disentangle the effects of your method a bit, for example, by constructing subsets of the benchmarks (or making synthetic benchmarks) where the PI was completely unhelpful due to being miraculous, or due to being redundant; or where it was recoverable via history of available information?

3 comments

We thank the reviewer for the careful reading and the encouraging assessment. The question about the kinds of privileged information (PI) is the one we find most important, so we answer it first.

W1 & Q1: How does the method depend on the relation between the privileged and the non-privileged setting, and which kinds of PI are represented in the benchmarks?

We agree that this deserves an explicit discussion, and we have added a short one to the appendix (Appendix G). Following the categories the reviewer proposes, we separate the PI by what the student could do without it (Table R1).

Table R1. Kinds of privileged information and how PSP behaves on each.

kind of PIwhat it means for the studentwhat PSP doesevidence
redundantthe student already solves the taskthe gate skips the group, so PSP is GRPOafter SFT only 11 of 32 groups are gated and PSP ties with GRPO (Appendix D.2), and the gain shrinks as the student gets stronger
recoverablethe student could find the behavior by its own exploration, and the PI only points it therethe guided rollout succeeds, the reward reinforces it, and the behavior transfers to the setting without PImain results
uninformativethe PI does not help the student succeedthe re-rollouts fail, the group carries no signal, and PSP falls back to GRPOwithout the reference trajectory PSP matches GRPO (Table 5)
miraculousthe PI contains information the student can never obtain at test timenot isolated in our benchmarksdiscussed below

The PI in AppWorld and SWE-bench is of the recoverable kind. An instruction typically tells the agent which API to call or in which order, and the agent could also find this by exploring the environment.

The robotics example is a helpful contrast. There the teacher's information is present in the current observation of the student, so the student can produce the action from its own input and distillation is appropriate. Our setting is multi-turn agents, where the instruction usually matters at one decisive turn. Given the instruction, the student gets past this turn and can practice the remaining easier turns. The decisive turn itself still has to be learned by the student on its own.

This also marks the limit of the method. When the decisive turn depends on information that the student cannot obtain by any means, which is the miraculous case, we expect PSP to help much less. Our benchmarks do not isolate this case, and we state it as a limitation in the revised paper (Appendix G). A synthetic benchmark that controls the kind of PI, as the reviewer suggests, would be the right way to study it, and we leave it to future work.

W2: Entries that are not significantly different from the best should be in bold.

We agree. In the revised tables we bold the best value of each column together with every value within one standard deviation of it. We also add two training seeds for GRPO and PSP on AppWorld and report the mean and standard deviation over three training runs in Appendix D.7.

W3: The captions of Tables 1 and 2 should say which benchmark they apply to.

Fixed. The captions of Tables 1 and 2 now state that the results are on AppWorld.

We sincerely thank the reviewer for the careful reading and the encouraging assessment. We hope our response has answered your questions, and we are happy to discuss any further points. Thank you again for your time.

This categorization is definitely helpful

Reviewer_kbBsOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that on-policy self-distillation from a teacher conditioned on privileged information fails for multi-turn agents because the student copies the teacher's confident steps without the information behind them. It proposes Privileged Self-Practice (PSP). When a task's rollouts mostly fail, an analyzer model writes a short per-task instruction, the task is re-sampled with that instruction in context, and training uses unchanged GRPO, so the privileged information never enters the loss. On AppWorld and SWE-bench Verified with three student models, PSP reports the best average score in every setting and 7% to 58% relative gains over GRPO.

Critical strengths

The idea is clear and simple. The paper distinguishes three uses of privileged information (as a target, a weight, or a sampler) and uses it to guide rollouts while leaving the GRPO objective unchanged. This makes the method easy to adopt and avoids asking the student to match a teacher conditioned on information unavailable to it.

Critical weaknesses

The paper's causal claims are not sufficiently isolated. The central story is that privileged-information mismatch causes OPSD to fail, and that moving this information to the sampler explains PSP's success. Neither side is established. The experiments show collapse under the chosen configurations, but do not rule out optimization instability or sensitivity to distillation hyperparameters, which limits the conclusion that the mismatch itself causes the failure and motivates replacing distillation with PSP. PSP's gains are similarly confounded: it relies on a stronger 27B analyzer and its reference trajectories, and spends extra rollouts on gated groups, while the main text reports no compute-matched GRPO comparison. Section 5.2 reinforces this concern, as PSP and GRPO tie when both start from an SFT checkpoint on the same trajectories. Much of the benefit may therefore come from stronger-model supervision and additional sampling rather than the sampler design itself, limiting its practical value to weak students whose own rollouts rarely succeed.

Questions for authors
  1. What tuning rule rules out optimization instability or sensitivity to distillation hyperparameters, rather than privileged-information mismatch, as the cause of OPSD's collapse?

  2. Given that PSP and GRPO perform similarly after SFT initialization, when does PSP offer a meaningful advantage over SFT followed by GRPO? Please account for total training cost, including re-rollouts and analyzer calls, and compare against GRPO under matched settings.

8 comments

We thank the reviewer for the thoughtful feedback. We address the concerns in turn and have added experiments for both.

Before the individual points, we would like to restate the core claim of the paper, because we believe it resolves most of the concerns. The paper concentrates on one question: given privileged information (PI) about a training task, how should it be trained into the weights of a multi-turn agent? OPSD distills it, RLSD and OPID use it to reweight the advantage, and PSP uses it in the rollout sampler. Where the PI comes from is a separate matter. Prior work in this line obtains it from a stronger model (Skill-SD, OPID) or from gold solutions (OPSD). We use an open 27B model with reference trajectories as an inexpensive and reproducible method, and in practice the PI can be an instruction written by expert engineers or by a giant frontier model. All methods in the paper receive exactly the same PI, so the comparison is only about how the PI is trained into the weights.

W1 (first part) & Q1: The collapse of OPSD may come from optimization instability or from the distillation hyperparameters rather than from the privileged-information mismatch.

The collapse of OPSD is structural and is not caused by the hyperparameters. As Fig. 1 illustrates, OPSD asks a student without the PI to directly imitate actions that were taken with the PI. The student learns the confident form but does not have the information behind it. At inference, it skips the exploration and confidently commits to a wrong answer. The problem is in the OPSD imitation objective, not in the hyperparameters.

Regarding the settings, all methods share the same learning rate, batch size and rollout settings (Table 6), and each baseline uses the coefficient of its source paper.

To check this directly, we reran OPSD with learning rates of 3e-7 and 1e-7 (1e-6 in the paper), and GRPO+OPSD with a coefficient of 0.001 (0.01 in the paper), on Qwen3-4B. The full curves are in the revised Fig. 5, which we recommend reading first because the trend is easiest to see there. Table R1 gives the same curves as averages over windows of 10 training steps for quick reference.

Table R1. The revised Fig. 5 in tabular form (Qwen3-4B), averaged over windows of 10 training steps. "-" means that the run has ended.

(a) Actor entropy

training steps1-1011-2021-3031-4041-5051-6061-70
OPSD, lr 1e-60.141.04-----
OPSD, lr 3e-70.010.080.55----
OPSD, lr 1e-70.010.010.010.020.100.360.61
GRPO+OPSD, coef 0.010.010.142.653.84---
GRPO+OPSD, coef 0.0010.010.010.010.010.010.010.01
GRPO0.010.010.010.010.010.010.01

(b) Train-batch reward (%)

training steps1-1011-2021-3031-4041-5051-6061-70
OPSD, lr 1e-61.20.0-----
OPSD, lr 3e-72.11.60.8----
OPSD, lr 1e-72.92.52.21.61.71.50.8
GRPO+OPSD, coef 0.013.13.50.10.0---
GRPO+OPSD, coef 0.0015.413.115.617.821.522.524.1
GRPO4.26.89.311.811.418.921.6

First, in the three OPSD runs and in GRPO+OPSD with the coefficient 0.01, the entropy rises sharply, while the train-batch reward never exceeds its initial level and falls towards zero. GRPO keeps its entropy at 0.01 and its reward rises from 4.2% to 21.6% over the same steps. Second, a smaller learning rate only delays this. The entropy reaches 0.5 around step 10, 23 and 60 for the learning rates 1e-6, 3e-7 and 1e-7, so the same curve is repeated on a longer time scale. Third, with the coefficient 0.001 the run shows no sign of collapse and its reward rises as it does for GRPO, which suggests that at this weight the GRPO term dominates the update.

Appendix D.4 shows the same effect without any distillation loss. Two runs use PI-guided rollouts with identical settings and differ only in which view is updated (Table R2).

Table R2. Appendix D.4: train-batch pass rate, averaged over 10 steps, when the same kind of rollouts is used to update two different views.

updated viewnon-privileged pass rate (steps 1-10 → 31-40)privileged pass rate (steps 1-10 → 31-40)
privileged view, as in PSP0.031 → 0.0560.120 → 0.264
non-privileged view, with importance-sampling correction0.018 → 0.0060.109 → 0.088

Updating the privileged view improves both views, and they keep improving afterwards. Updating the non-privileged view on PI-guided rollouts makes both views worse. A view that does not see the privileged information cannot learn the privileged behavior directly, and this is the same failure that OPSD shows.

We do not claim that distillation losses are problematic in general. We claim that directly distilling PI-conditioned trajectories into a student that does not see the PI tends to make a multi-turn agent give up its early exploration and commit to a wrong answer.

W1 (second part): PSP's gain may come from the stronger 27B analyzer, its reference trajectories and the extra rollouts rather than from the sampler design.

As stated at the beginning, the paper is about how PI should be trained into the weights of a multi-turn agent. The 27B analyzer and its reference trajectories are only an inexpensive way to simulate such information. Qwen3.6-27B is a moderately sized open-weight model, which keeps the experiments reproducible. Real applications can use better instructions, for example ones hand-written by expert engineers or written by a giant frontier model. For this reason all baselines in the paper use exactly the same mechanism to generate the PI, so the stronger model helps every method equally.

The 27B model also does not have to be present during training. In Appendix D.2 the student itself is the analyzer and reads one stored trajectory per task, so no 27B call is made in the training loop. This variant reaches 17.6 on average, compared with 7.4 ± 3.1 for GRPO and 16.3 ± 1.5 for PSP with the 27B analyzer in the loop (three training runs each).

For the extra rollouts, we added experiments of GRPO with group size 16, which doubles the number of rollouts per step. Table R3 gives the best validation TGC reached within a given rollout budget on Qwen3-4B.

Table R3. Best validation TGC (%) within a rollout budget, Qwen3-4B.

rollouts20k40k60k80k100k256k
GRPO (n = 8)8.410.910.915.420.0-
GRPO (n = 16, new)8.110.211.611.611.612.3
PSP9.817.523.226.330.2-
PSP, student as analyzer (no 27B call in training)14.724.925.328.429.8-

Simply adding rollouts does not improve performance. GRPO with n = 16 follows the n = 8 curve, while PSP is ahead at every budget, also when the student itself is the analyzer. Table R4 gives the full test results of GRPO with n = 16 next to GRPO with n = 8 (three training runs) and PSP with the student as the analyzer, that is, without any 27B call in training.

Table R4. Full test results on Qwen3-4B. PSP here uses the student itself as the analyzer (Appendix D.2, Table 9). The GRPO (n = 8) row is the mean over three training runs, with ± the standard deviation across runs; in the other rows ± is over 5 evaluation seeds.

methodnormal TGCnormal SGCnormal TGC@5normal SGC@5challenge TGCchallenge SGCchallenge TGC@5challenge SGC@5Avg
GRPO (n = 8, mean of 3 runs)9.4 ± 4.12.3 ± 1.821.211.33.2 ± 1.20.2 ± 0.48.82.97.4 ± 3.1
GRPO (n = 16, new)9.2 ± 0.81.8 ± 0.023.816.12.0 ± 0.40.0 ± 0.06.50.77.5
PSP, student as analyzer22.6 ± 1.312.1 ± 3.738.725.09.6 ± 0.43.7 ± 0.920.48.617.6

Doubling the rollouts of GRPO gives 7.5 on average, against 7.4 ± 3.1 for n = 8, while PSP without the 27B model in the training loop reaches 17.6. The extra rollouts are therefore not the source of the gain. What matters is how the PI is used, which is the question this paper studies.

Q2: PSP and GRPO tie after SFT initialization, so when does PSP offer an advantage over SFT followed by GRPO once the total cost is counted?

This result is expected, and it is the reason we report it. PSP uses the instruction only on groups whose own rollouts carry no reward signal and trains every other group exactly as GRPO does. After SFT the student already solves most training tasks (11 of 32 groups are gated at the first step, against 28 of 32 from the base model), so there is little left for the instruction to do and PSP is close to GRPO. We show this negative result on purpose because it matches the design of the method. If RL can already solve nearly everything, PSP reduces to GRPO.

We do not propose PSP as a replacement for SFT. The two start from different resources. SFT needs complete expert trajectories in the agent's action format. Sometimes what is available is an instruction for the task, such as a system prompt written by engineers or by a frontier model. PSP is one way to train such an instruction into the weights. Table R5 collects the results of Appendix D.2, where SFT and PSP use the same 90 trajectories.

Table R5. SFT and PSP on the same 90 trajectories (Qwen3-4B, test average, from Tables 9 and 10 of the paper). For PSP, γ = 0.3 is selected on validation.

methodAvg
SFT14.8
PSP from the base model17.6
SFT, then GRPO17.7
SFT, then PSP19.1

First, from the base model, PSP reaches the level of SFT followed by GRPO (17.6 against 17.7) by only reading the trajectories, without training on any trajectory token. Second, PSP also works on top of SFT (19.1 against 17.7 for GRPO). The margin there is small, as expected, because few groups are gated after SFT and the two methods train almost the same batches. When expert trajectories are available, SFT followed by GRPO is a good recipe, and PSP remains compatible with it. As stated at the beginning, the paper studies how PI should be trained into the weights. SFT uses a different resource, namely expert trajectories, and we include the comparison in Appendix D.2 to show where PSP stands when such trajectories are also available.

On cost, a PSP step takes about 2.9 times as long as a GRPO step (444 s vs. 154 s) because of the re-rollouts of the gated groups and the analyzer calls. From the base model this cost pays off. PSP passes the best validation score of GRPO (20.0) after 49k rollouts and 15.1 node-hours, where GRPO needs 98k rollouts and 17.0 node-hours (Table R3). After SFT fewer groups are gated, so the overhead is smaller as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.