- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PCSD is an on-policy self-distillation framework giving token-level credit assignment based on local persistence. It adds on top of GRPO for multi-turn agents. The teacher is a frozen copy of the initial checkpoint and is prompted with skills selected through keyword retrieval. It evaluates tokens sampled by the student for OPSD. They evaluate PCSD on ALFWorld and WebShop using Qwen2.5-3B and Qwen3-1.7B. In single-run experiments of 150 training steps, PCSD obtains ALFWorld overall scores 6.2 and 5.5 higher than SDAR, and maintain SDAR’s accuracy on WebShop.
Critical strengths
- per-token credit assignment is a reasonable teacher sits between pointwise and step-level weighting, and the fix is cheap, detached post-hoc weighting, without extra model.
- having ablations to see if the adaptive aggregation is needed or not, and shows t
Critical weaknesses
- there's no random seed, confidence interval and etc.
- The performance gain is not huge, it's just two models on one dataset. I would have more confidence interval based on 1.
- more baselines would be helpful, like StepOPSD (arxiv: 2605.27140) and etc.
- In Table 1, it's not convincing that the model doesn't have a performance gain based on the teacher prompt, at 1.7B ALFworld the performance is worse than the vanilla model. So the skills might not be helpful for small models.
Questions for authors
- Seeds and CIs for Tables 1–3 - is the evaluation set fixed across methods?
- End-to-end PCSD with a no-skill teacher and with shuffled skills.
- What's the distribution of r_{k,i} and η_{k,i}? what fraction of tokens have r>0 or η<0.9, and gate activation split between <think> and <action> spans?
- Training curves for all methods: does the gap hold at convergence or only early?
2 comments
W4/Q2: Can skills provide effective teacher supervision?
We agree that Table 1 shows that directly providing skills to Qwen3-1.7B on ALFWorld does not consistently improve its ability to complete tasks autonomously. However, the Skill-Prompt baseline and the distillation teacher assess different capabilities. Skill-Prompt uses general skills without retrieving task-specific skills. For the teacher, we use keyword matching on the observable task instruction to retrieve task-specific skills and combine them with general skills. The two settings therefore differ.
In addition, Skill-Prompt requires the model to use general skills to autonomously generate and execute an entire successful trajectory. The distillation teacher scores student-generated tokens conditioned on the existing interaction prefix. A smaller model may struggle to use skills to complete an entire task successfully while still benefiting from their guidance on individual decisions.
To directly examine the contribution of the skill context, we conducted end-to-end PCSD training experiments under three conditions: general skills only (no retrieval), random skills, and the full skill configuration. The random-skill condition uses the same general skills as the full configuration and replaces only the retrieved task-specific skill with a randomly sampled skill. All three conditions use the same distillation objective and training settings.
| Setting | Pick | Look | Clean | Heat | Cool | Pick2 | Overall |
|---|---|---|---|---|---|---|---|
| GRPO | 71.1 | 41.7 | 36.4 | 40.0 | 31.8 | 31.6 | 46.1 |
| General skills only (no retrieval) | 56.2 | 25.0 | 71.8 | 60.6 | 69.0 | 22.2 | 54.7 |
| Random skills | 56.7 | 33.3 | 46.4 | 65.2 | 51.2 | 47.2 | 50.0 |
| Full skills (PCSD) | 63.3 | 68.8 | 50.0 | 69.2 | 48.1 | 63.6 | 59.4 |
Comparing full skills with general skills alone tests whether the retrieved skill context provides additional training value. Comparing full and random skills tests the importance of matching skills to the task. PCSD achieved overall success rates of 54.7% with general skills alone and 50.0% with random skills, both above the GRPO baseline of 46.1%, with general skills outperforming random skills by 4.7 percentage points. Under the same distillation settings, combining general skills with retrieved, task-matched skills raised the success rate to 59.4%, a gain of 4.7 percentage points over general skills alone and 9.4 over random skills, indicating that task-matched skills provide additional value during training.
Q3: Distributions of and and gate activation across spans.
We analyzed 148,152 valid response tokens from 50 distinct ALFWorld tasks. 37.57% of tokens used information from the long window (), and 12.96% of all tokens relied entirely on the long window (). For trend modulation, 48.88% of tokens retained their full base weight (), while 11.67% had their base weight reduced by more than 10% (). Using the number of valid tokens within each span type as the denominator, the gate activation rates () were 29.07% for <think> spans and 11.17% for <action> spans.
Q4: Training dynamics and convergence.
We have added training curves in Appendix A.12, Figure 7 of the revised PDF.
Response to Reviewer grd8
We thank the reviewer for recognizing the value of token-level credit assignment and the role of the adaptive aggregation ablation experiments. We respond to each point below.
W1/W2/Q1: Experimental coverage, statistical validation, and evaluation protocol.
Our experiments cover two benchmarks, ALFWorld and WebShop, and two model backbones. For the comparisons in Tables 1–3, all methods use the same fixed evaluation task IDs, and all reported results come from the checkpoint after 150 training updates.
Given the available compute and rebuttal time, we completed two additional independent training runs with different random seeds for each method (SDAR and PCSD) on each benchmark (ALFWorld and WebShop) using Qwen2.5-3B. The additional runs followed the same protocol. Including the original run, this gives three runs per method on each benchmark.
Both methods used random seeds 0, 42, and 100. On ALFWorld, PCSD outperformed SDAR across all three seeds, achieving a mean overall success rate of 90.80% compared with 84.13% for SDAR, a gain of 6.67 percentage points. On WebShop, PCSD achieved a higher Score than SDAR at all three seeds, with a mean Score of 84.67 compared with 83.40 for SDAR, a gain of 1.27 points. WebShop accuracy was the same for both methods at seed 0. At seeds 42 and 100, PCSD exceeded SDAR by 3.1 and 0.8 percentage points, respectively. Mean accuracy increased from 67.47% to 68.77%, a gain of 1.30 percentage points.
W3: Comparison with StepOPSD.
In the previous version, we cited StepOPSD as Zhang et al. (2026b) in Section 2.2. We agree that the relationship between StepOPSD and PCSD should be discussed more precisely. We have added StepOPSD as a baseline in the main results table (see Table 1 in the revised PDF).