- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the failure of on-policy self-distillation (OPSD) in training multi-turn LLM agents with privileged information (PI). The authors observe that OPSD teaches the student to mimic the teacher's confident actions without understanding the underlying reasoning, which can degrade performance below that of an untrained model. To solve this, they introduce Privileged Self-Practice (PSP), a framework that shifts PI from the training objective to the rollout sampler. When a task's success rate falls below a specific threshold, PSP uses an analyzer model to write short instructions based on the agent's failures, re-samples trajectories with these instructions in context, and trains using standard Group-Relative Policy Optimization (GRPO).
Critical strengths
PSP integrates into standard RL pipelines (like GRPO) without requiring complex auxiliary loss balancing or cross-view objectives.
The failure analysis of OPSD in agents is concrete, with the Fig. 1 example, the entropy blow-up in Fig. 5, and the link to the irreducible information gap from prior work.
The gate threshold, an “always” variant, removing the reference trajectory, and frozen instruction pools (Table 8) all isolate real design choices. The frozen-instruction result is especially informative: the gain comes from fresh, diverse instructions, not just extra re-rollouts.
Critical weaknesses
The analyzer's success heavily depends on a pre-computed pool of successful reference trajectories generated by the 27B model. The authors' own ablation shows a significant performance drop if the analyzer must infer the correct procedure solely from the student's failures.
The method also introduces additional computational overhead. PSP needs an online analyzer (half the GPUs on AppWorld), extra re-rollouts, and an offline pool of 16 trajectories per task from a 27B model. A compute-matched comparison with GRPO is only partly addressed (the 128-sample check in D.1).
Each number comes from one training run per method and model. The reported ± is over evaluation seeds only, so it does not capture training variance. Several key margins are small (Qwen3-8B average 27.4 vs. 25.6 for GRPO; SWE-bench Qwen3-4B has RLSD ahead of PSP on both metrics). The claim that PSP “consistently outperforms” GRPO is not well supported at this level of evidence.
Questions for authors
The data shows a significant performance drop when the 27B analyzer must write instructions based solely on the student's failures, rather than having access to a successful reference trajectory. How viable is PSP in novel or highly complex domains where generating these successful reference trajectories upfront is impossible?
PSP requires invoking a 27B analyzer model during the training loop and re-rolling failed groups. How does the wall-clock time and total FLOP count of PSP compare to plain GRPO or RLSD, and is the performance gain strictly worth the computational premium in resource-constrained environments?
You avoided using frontier models to limit compute. If you scale the analyzer up to a frontier model (e.g., GPT-4o or Claude 3.5 Sonnet), does the quality of the instructions improve enough to rescue the "failures only" ablation, or does the student model hit a capacity ceiling regardless of instruction quality?
The ablations show that using frozen instructions pre-written by the base model provides little gain unless a massive pool of 64 instructions per task is used. Why does the student model overfit so rapidly to a single static instruction, and does this imply that PSP acts more as a regularizer through text variance than as a knowledge injector?
9 comments
W2 & Q2: PSP adds compute, and the comparison with GRPO is not compute-matched.
The gain of PSP comes from using the privileged information and not from the extra compute. On Qwen3-4B, GRPO does not reach the accuracy of PSP when it is given the same number of rollouts, twice the rollouts per step, or its full budget of 500 steps, as the comparisons below show. The most direct comparison is validation accuracy at the same number of rollouts. Table R2 gives the best validation TGC reached within a given rollout budget on Qwen3-4B. A GRPO step uses 256 rollouts, and a PSP step uses 256 plus the re-rollouts of the gated groups. We also add a new control, GRPO with n = 16, which uses 512 rollouts per step.
Table R2. Best validation TGC (%) within a rollout budget, Qwen3-4B.
| rollouts | 20k | 40k | 60k | 80k | 100k | 256k |
|---|---|---|---|---|---|---|
| GRPO (n = 8) | 8.4 | 10.9 | 10.9 | 15.4 | 20.0 | - |
| GRPO (n = 16, new) | 8.1 | 10.2 | 11.6 | 11.6 | 11.6 | 12.3 |
| PSP | 9.8 | 17.5 | 23.2 | 26.3 | 30.2 | - |
| PSP, student as analyzer (no 27B call in training) | 14.7 | 24.9 | 25.3 | 28.4 | 29.8 | - |
PSP is ahead at every budget, both with the 27B analyzer and with the student as the analyzer, and giving GRPO twice the rollouts per step does not change its curve. On the test sets GRPO with n = 16 scores 7.5 on average, against 7.4 ± 3.1 for n = 8. Table R3 gives the wall-clock cost per step on the same node, together with the test average.
Table R3. Wall-clock cost per step and test average, Qwen3-4B. Seconds per step are measured on the same node and averaged over the whole run of the paper. The test average of GRPO and PSP is over three training runs, and RLSD is the single run of the paper.
| method | seconds per step | where the overhead comes from | test Avg |
|---|---|---|---|
| GRPO | 154 | - | 7.4 ± 3.1 |
| RLSD | 341 | extra scoring of every rollout with the PI in context, at every step | 12.1 |
| PSP | 444 | re-rollouts and analyzer calls on the gated groups only (86% of the groups at the start, 42% at the end) | 16.3 ± 1.5 |
A PSP step takes about 2.9 times as long as a GRPO step, and this overhead decreases during training as fewer groups are gated, whereas the overhead of RLSD is constant. Compared with GRPO, PSP still reaches a given accuracy earlier. It passes the best validation score of GRPO (20.0) after 49k rollouts and 15.1 node-hours, where GRPO needs 98k rollouts and 17.0 node-hours. We report rollouts and node-hours on identical hardware rather than FLOPs. All methods train the same student, so FLOPs are roughly proportional to rollouts.
As for whether the premium is worth it, the gain depends on how often the student's own rollouts succeed. It is large for Qwen3-4B (+8.9 on average) and small for Gemma4-E4B (+1.0) as Table R4 shows.
W3: Each number comes from one training run, so the claim that PSP “consistently outperforms” GRPO is not well supported.
We thank the reviewer for raising this point, and we agree that the reported ± covers evaluation seeds only. We have added two training seeds for GRPO and for PSP on each of the three students, each with its own validation-based checkpoint selection and 5-seed test, as in the paper. With them the picture is clearer. The gain is large on Qwen3-4B, moderate on Qwen3-8B, and within the variation across runs on Gemma4-E4B.
Table R4. Test average, mean ± std over three training runs (the run of the paper and two new seeds). Tables R5 to R7 give all metrics of every run.
| student | GRPO | PSP | difference |
|---|---|---|---|
| Qwen3-4B | 7.4 ± 3.1 | 16.3 ± 1.5 | +8.9 |
| Qwen3-8B | 25.3 ± 2.8 | 27.9 ± 0.5 | +2.6 |
| Gemma4-E4B | 29.5 ± 2.0 | 30.5 ± 1.7 | +1.0 |
Table R5. Qwen3-4B, all test metrics of every training run.
| method | run | normal TGC | normal SGC | normal TGC@5 | normal SGC@5 | challenge TGC | challenge SGC | challenge TGC@5 | challenge SGC@5 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | paper | 13.2 ± 1.7 | 4.3 ± 1.4 | 28.0 | 16.1 | 4.6 ± 0.6 | 0.7 ± 0.5 | 12.0 | 5.8 | 10.6 |
| GRPO | new seed 1 | 9.9 ± 1.4 | 1.8 ± 1.3 | 22.0 | 12.5 | 2.6 ± 0.8 | 0.0 ± 0.0 | 7.2 | 2.2 | 7.3 |
| GRPO | new seed 2 | 5.0 ± 1.9 | 0.7 ± 1.0 | 13.7 | 5.4 | 2.5 ± 0.4 | 0.0 ± 0.0 | 7.2 | 0.7 | 4.4 |
| GRPO | mean ± std over runs | 9.4 ± 4.1 | 2.3 ± 1.8 | 21.2 ± 7.2 | 11.3 ± 5.5 | 3.2 ± 1.2 | 0.2 ± 0.4 | 8.8 ± 2.8 | 2.9 ± 2.6 | 7.4 ± 3.1 |
| PSP | paper | 21.8 ± 2.4 | 6.4 ± 2.4 | 39.9 | 23.2 | 9.4 ± 0.4 | 3.2 ± 0.6 | 19.9 | 10.1 | 16.7 |
| PSP | new seed 1 | 20.0 ± 1.5 | 6.8 ± 0.8 | 37.5 | 21.4 | 6.8 ± 1.2 | 1.1 ± 0.4 | 17.0 | 5.8 | 14.6 |
| PSP | new seed 2 | 24.4 ± 2.0 | 10.7 ± 1.8 | 41.1 | 23.2 | 9.5 ± 1.1 | 3.2 ± 1.3 | 18.2 | 9.3 | 17.5 |
| PSP | mean ± std over runs | 22.1 ± 2.2 | 8.0 ± 2.4 | 39.5 ± 1.8 | 22.6 ± 1.0 | 8.6 ± 1.5 | 2.5 ± 1.2 | 18.4 ± 1.4 | 8.4 ± 2.3 | 16.3 ± 1.5 |
Table R6. Qwen3-8B, all test metrics of every training run.
| method | run | normal TGC | normal SGC | normal TGC@5 | normal SGC@5 | challenge TGC | challenge SGC | challenge TGC@5 | challenge SGC@5 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | paper | 30.8 ± 1.0 | 13.9 ± 2.4 | 53.6 | 35.7 | 14.9 ± 1.1 | 5.2 ± 1.5 | 30.0 | 20.9 | 25.6 |
| GRPO | new seed 1 | 31.4 ± 2.4 | 14.6 ± 4.6 | 50.6 | 32.1 | 10.4 ± 0.7 | 2.7 ± 0.6 | 24.5 | 12.2 | 22.3 |
| GRPO | new seed 2 | 35.2 ± 2.5 | 17.5 ± 5.0 | 58.3 | 46.4 | 14.6 ± 1.4 | 5.3 ± 0.4 | 28.8 | 17.3 | 27.9 |
| GRPO | mean ± std over runs | 32.5 ± 2.4 | 15.3 ± 1.9 | 54.2 ± 3.9 | 38.1 ± 7.4 | 13.3 ± 2.5 | 4.4 ± 1.5 | 27.7 ± 2.9 | 16.8 ± 4.4 | 25.3 ± 2.8 |
| PSP | paper | 31.6 ± 2.0 | 12.1 ± 1.3 | 57.7 | 39.3 | 16.8 ± 0.6 | 5.9 ± 0.8 | 32.1 | 23.7 | 27.4 |
| PSP | new seed 1 | 33.7 ± 2.0 | 15.4 ± 1.6 | 58.3 | 46.4 | 16.6 ± 0.8 | 5.5 ± 0.6 | 31.9 | 20.1 | 28.5 |
| PSP | new seed 2 | 34.6 ± 1.9 | 16.4 ± 3.2 | 57.1 | 35.7 | 16.8 ± 1.1 | 7.0 ± 0.9 | 31.9 | 23.7 | 27.9 |
| PSP | mean ± std over runs | 33.3 ± 1.6 | 14.6 ± 2.3 | 57.7 ± 0.6 | 40.5 ± 5.5 | 16.7 ± 0.1 | 6.1 ± 0.8 | 32.0 ± 0.1 | 22.5 ± 2.1 | 27.9 ± 0.5 |
Table R7. Gemma4-E4B, all test metrics of every training run. PSP uses γ = 0.3, as in the paper.
| method | run | normal TGC | normal SGC | normal TGC@5 | normal SGC@5 | challenge TGC | challenge SGC | challenge TGC@5 | challenge SGC@5 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | paper | 33.9 ± 1.3 | 18.2 ± 1.3 | 54.8 | 39.3 | 15.3 ± 0.8 | 4.3 ± 0.6 | 31.9 | 20.1 | 27.2 |
| GRPO | new seed 1 | 36.1 ± 4.7 | 19.6 ± 4.4 | 60.1 | 46.4 | 17.8 ± 0.9 | 6.2 ± 1.3 | 35.2 | 23.0 | 30.6 |
| GRPO | new seed 2 | 36.4 ± 2.7 | 19.6 ± 2.5 | 58.9 | 42.9 | 18.3 ± 0.9 | 4.9 ± 1.6 | 38.6 | 25.9 | 30.7 |
| GRPO | mean ± std over runs | 35.5 ± 1.4 | 19.2 ± 0.8 | 58.0 ± 2.8 | 42.9 ± 3.6 | 17.1 ± 1.6 | 5.1 ± 1.0 | 35.3 ± 3.4 | 23.0 ± 2.9 | 29.5 ± 2.0 |
| PSP | paper | 38.6 ± 1.8 | 20.4 ± 4.3 | 61.3 | 46.4 | 20.3 ± 1.7 | 7.8 ± 1.4 | 38.4 | 26.6 | 32.5 |
| PSP | new seed 1 | 33.8 ± 3.0 | 18.9 ± 4.1 | 56.5 | 46.4 | 16.5 ± 1.0 | 4.9 ± 1.4 | 34.5 | 21.6 | 29.2 |
| PSP | new seed 2 | 35.8 ± 1.9 | 20.0 ± 2.6 | 57.7 | 41.1 | 18.1 ± 1.1 | 6.3 ± 1.6 | 36.5 | 23.7 | 29.9 |
| PSP | mean ± std over runs | 36.1 ± 2.4 | 19.8 ± 0.8 | 58.5 ± 2.5 | 44.6 ± 3.1 | 18.3 ± 1.9 | 6.3 ± 1.5 | 36.5 ± 1.9 | 24.0 ± 2.5 | 30.5 ± 1.7 |
Based on the results, we observe the following. (1) On Qwen3-4B every PSP run is above every GRPO run. (2) On Qwen3-8B, PSP is 2.6 points higher on average. (3) On Gemma4-E4B, the margin is about one point, so there PSP is on par with GRPO. The gain thus decreases as the student's own rollouts succeed more often. The train pass rate at the start of training is 3% for Qwen3-4B, 8% for Qwen3-8B and 11% for Gemma4-E4B, and PSP is designed to add signal where this rate is low.
In short, the margin depends strongly on how well the base model already solves the training tasks, and PSP stays at or above GRPO in every setting. This is different from loss-side methods such as OPSD, which can fall below GRPO. In PSP, the privileged information never enters the loss, and every ungated group is trained exactly as in GRPO. We therefore replace “consistently outperforms GRPO” with “matches or exceeds GRPO in every setting”, with the largest gains where the base model solves the fewest training tasks”, and we now bold all entries that are not significantly different from the best. The new seeds cover AppWorld only, because a SWE-bench run takes much longer and did not fit in the rebuttal period. There the single runs show PSP above GRPO on both students (10.57 against 8.35 on average) and tied with RLSD on Qwen3-4B (5.92 ± 0.37 against 6.20 ± 0.80), as the reviewer notes.
We thank the reviewer for the careful and constructive comments. We address each point below and have run new experiments where more evidence was requested.
Before the individual points, we would like to restate the core claim of the paper, because it bears on several of them. The paper concentrates on one question: given privileged information (PI) about a training task, how should it be trained into the weights of a multi-turn agent? OPSD distills it, RLSD and OPID use it to reweight the advantage, and PSP uses it in the rollout sampler. Where the PI comes from is a separate matter. Prior work in this line obtains it from a stronger model (Skill-SD, OPID) or from gold solutions (OPSD). We use an open 27B model with reference trajectories as an inexpensive and reproducible method, and in practice the PI can be an instruction written by expert engineers or by a giant frontier model. All methods in the paper receive exactly the same PI, so the comparison is only about how the PI is trained into the weights.
W1 & Q1: PSP relies on reference trajectories from the 27B model, which may not be available in a new domain.
As stated at the beginning, the paper is about how PI should be trained into the weights of a multi-turn agent. The Qwen3.6-27B analyzer is a moderately sized open-weight model, which keeps the experiments reproducible, and its reference trajectories are only an inexpensive way to simulate such information. Real applications can use better instructions, for example hand-written ones or ones written by a frontier model.
Table R1 puts the ablation the reviewer points to next to the other ways of producing the instruction.
Table R1. Test average on AppWorld by how the instruction is produced. The numbers are collected from Tables 1, 2, 5 and 9 of the paper.
(1) Without the reference, the instruction is worse and PSP falls back to the level of GRPO. This shows that a weak instruction removes the gain. (2) The requirement could be very light. One stored trajectory per task is enough, and the 27B model does not have to be called during training (Appendix D.2).
What PSP needs in a new domain is a good instruction for the training tasks, and a reference trajectory is one way to obtain it. First, many agent benchmarks already come with reference solutions, from which such instructions can be written. Second, in practice many good instructions do not rely on a reference solution at all, e.g., they are written by engineers who know the task. Whether a reference is available is therefore a separate question from the one this paper studies, which is how to train the PI into the weights. All methods in the paper receive their instructions from the same mechanism, so they are compared at the same instruction quality.