acceptodds
Under review as a conference paper at ICLR 2027

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Abstract

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

Primary area:
reinforcement learning
Keywords:
Agentic RL, OPD

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_grd8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PCSD is an on-policy self-distillation framework giving token-level credit assignment based on local persistence. It adds on top of GRPO for multi-turn agents. The teacher is a frozen copy of the initial checkpoint and is prompted with skills selected through keyword retrieval. It evaluates tokens sampled by the student for OPSD. They evaluate PCSD on ALFWorld and WebShop using Qwen2.5-3B and Qwen3-1.7B. In single-run experiments of 150 training steps, PCSD obtains ALFWorld overall scores 6.2 and 5.5 higher than SDAR, and maintain SDAR’s accuracy on WebShop.

Critical strengths
  • per-token credit assignment is a reasonable teacher sits between pointwise and step-level weighting, and the fix is cheap, detached post-hoc weighting, without extra model.
  • having ablations to see if the adaptive aggregation is needed or not, and shows t
Critical weaknesses
  • there's no random seed, confidence interval and etc.
  • The performance gain is not huge, it's just two models on one dataset. I would have more confidence interval based on 1.
  • more baselines would be helpful, like StepOPSD (arxiv: 2605.27140) and etc.
  • In Table 1, it's not convincing that the model doesn't have a performance gain based on the teacher prompt, at 1.7B ALFworld the performance is worse than the vanilla model. So the skills might not be helpful for small models.
Questions for authors
  • Seeds and CIs for Tables 1–3 - is the evaluation set fixed across methods?
  • End-to-end PCSD with a no-skill teacher and with shuffled skills.
  • What's the distribution of r_{k,i} and η_{k,i}? what fraction of tokens have r>0 or η<0.9, and gate activation split between <think> and <action> spans?
  • Training curves for all methods: does the gap hold at convergence or only early?
2 comments
AuthorsOpenReview ↗

Response to Reviewer grd8

We thank the reviewer for recognizing the value of token-level credit assignment and the role of the adaptive aggregation ablation experiments. We respond to each point below.

W1/W2/Q1: Experimental coverage, statistical validation, and evaluation protocol.

Our experiments cover two benchmarks, ALFWorld and WebShop, and two model backbones. For the comparisons in Tables 1–3, all methods use the same fixed evaluation task IDs, and all reported results come from the checkpoint after 150 training updates.

Given the available compute and rebuttal time, we completed two additional independent training runs with different random seeds for each method (SDAR and PCSD) on each benchmark (ALFWorld and WebShop) using Qwen2.5-3B. The additional runs followed the same protocol. Including the original run, this gives three runs per method on each benchmark.

Both methods used random seeds 0, 42, and 100. On ALFWorld, PCSD outperformed SDAR across all three seeds, achieving a mean overall success rate of 90.80% compared with 84.13% for SDAR, a gain of 6.67 percentage points. On WebShop, PCSD achieved a higher Score than SDAR at all three seeds, with a mean Score of 84.67 compared with 83.40 for SDAR, a gain of 1.27 points. WebShop accuracy was the same for both methods at seed 0. At seeds 42 and 100, PCSD exceeded SDAR by 3.1 and 0.8 percentage points, respectively. Mean accuracy increased from 67.47% to 68.77%, a gain of 1.30 percentage points.

Dataset and MetricMethodSeed 0Seed 42Seed 100Mean ± Sample SD95% CI for the Mean
ALFWorld OverallSDAR84.482.885.284.13 ± 1.22[81.10, 87.17]
ALFWorld OverallPCSD90.690.691.290.80 ± 0.35[89.94, 91.66]
WebShop ScoreSDAR83.481.885.083.40 ± 1.60[79.43, 87.37]
WebShop ScorePCSD85.082.986.184.67 ± 1.63[80.63, 88.71]
WebShop AccuracySDAR67.267.268.067.47 ± 0.46[66.32, 68.61]
WebShop AccuracyPCSD67.270.368.868.77 ± 1.55[64.92, 72.62]

W3: Comparison with StepOPSD.

In the previous version, we cited StepOPSD as Zhang et al. (2026b) in Section 2.2. We agree that the relationship between StepOPSD and PCSD should be discussed more precisely. We have added StepOPSD as a baseline in the main results table (see Table 1 in the revised PDF).

W4/Q2: Can skills provide effective teacher supervision?

We agree that Table 1 shows that directly providing skills to Qwen3-1.7B on ALFWorld does not consistently improve its ability to complete tasks autonomously. However, the Skill-Prompt baseline and the distillation teacher assess different capabilities. Skill-Prompt uses general skills without retrieving task-specific skills. For the teacher, we use keyword matching on the observable task instruction to retrieve task-specific skills and combine them with general skills. The two settings therefore differ.

In addition, Skill-Prompt requires the model to use general skills to autonomously generate and execute an entire successful trajectory. The distillation teacher scores student-generated tokens conditioned on the existing interaction prefix. A smaller model may struggle to use skills to complete an entire task successfully while still benefiting from their guidance on individual decisions.

To directly examine the contribution of the skill context, we conducted end-to-end PCSD training experiments under three conditions: general skills only (no retrieval), random skills, and the full skill configuration. The random-skill condition uses the same general skills as the full configuration and replaces only the retrieved task-specific skill with a randomly sampled skill. All three conditions use the same distillation objective and training settings.

SettingPickLookCleanHeatCoolPick2Overall
GRPO71.141.736.440.031.831.646.1
General skills only (no retrieval)56.225.071.860.669.022.254.7
Random skills56.733.346.465.251.247.250.0
Full skills (PCSD)63.368.850.069.248.163.659.4

Comparing full skills with general skills alone tests whether the retrieved skill context provides additional training value. Comparing full and random skills tests the importance of matching skills to the task. PCSD achieved overall success rates of 54.7% with general skills alone and 50.0% with random skills, both above the GRPO baseline of 46.1%, with general skills outperforming random skills by 4.7 percentage points. Under the same distillation settings, combining general skills with retrieved, task-matched skills raised the success rate to 59.4%, a gain of 4.7 percentage points over general skills alone and 9.4 over random skills, indicating that task-matched skills provide additional value during training.

Q3: Distributions of rk,ir_{k,i} and ηk,i\eta_{k,i} and gate activation across spans.

We analyzed 148,152 valid response tokens from 50 distinct ALFWorld tasks. 37.57% of tokens used information from the long window (r>0r>0), and 12.96% of all tokens relied entirely on the long window (r=1r=1). For trend modulation, 48.88% of tokens retained their full base weight (η=1\eta=1), while 11.67% had their base weight reduced by more than 10% (η<0.9\eta<0.9). Using the number of valid tokens within each span type as the denominator, the gate activation rates (w>0.5w>0.5) were 29.07% for <think> spans and 11.17% for <action> spans.

Q4: Training dynamics and convergence.

We have added training curves in Appendix A.12, Figure 7 of the revised PDF.

Reviewer_3Lt2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Persistent Consistency Self-Distillation (PCSD) method, which is an on-policy self-distillation method that estimates teacher credibility from the local persistence of teacher-favoring signals. An adaptive window aggregation method is developed that adjusts its effective range according to the local statistics of teacher–student discrepancies. Trend-aware modulation is introduced that transforms credibility estimates into smooth token-level distillation weights. Experiments are conducted on the two benchmarks of ALFWorld and WebShop, and the results demonstrate performance improvements over baselines.

Critical strengths

As explained in the paper, the key insight of their local persistent consistency method is that "informative teacher advantage should persist across a local neighborhood, whereas isolated spikes are more likely to reflect sampling noise or incidental variation", which is interesting. They also propose to adaptively adjust the local window. Some experimental performance demonstrates clear improvements.

Critical weaknesses

There is no detail of the privileged skills used to train the teacher and justification of these skills. Table 1 indicates the performances of some models with skills are worse than those without skills. This raises the question if the skills are useful or the teacher is indeed functioning well during the reinforcement learning process.

Questions for authors

Besides the questions in critical weakness, a minor comment is that in some figures, the font sizes are too small.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Lt2

We thank the reviewer for recognizing the motivation behind persistent local support and adaptive aggregation, and for pointing out that the privileged skills and their role in training need further clarification.

W1: Response to concerns about privileged skills and teacher effectiveness.

1. What the skills contain and how they are used.

Appendix A.1 describes the skill categories, the keyword-based retrieval method, and the information isolation protocol. Appendix A.2 explains how the retrieved skills are added to the teacher’s context. We agree that these details should be presented more clearly and concretely in the main text, particularly the source of the skill library and the content of individual skills.

We reuse the skill resources provided by SDAR (arxiv: 2605.15155), which adopts the SkillBank proposed by SkillRL(arxiv: 2602.08234). For example, ALFWorld skills describe ordered procedures such as acquiring the target object, cleaning it, and placing it at the target location. WebShop skills describe how to construct a search query based on the user’s requirements, verify product attributes and prices, and select the required options before making a purchase. These skills provide reusable procedural priors rather than answers to individual evaluation instances.

For each trajectory, we use keyword matching on the observable task instruction to retrieve a task-specific skill and combine it with general skills. Only the teacher receives the skill text. Given the same interaction prefix, the teacher scores the same tokens generated by the student. The student does not receive retrieved skills during training or evaluation. We will clarify this point and provide actual skill entries, a complete retrieval example, and the source of the skill library.

2. Why a lower task success rate with skill prompting does not necessarily imply ineffective local supervision.

We agree that Table 1 shows that directly providing skills does not consistently improve autonomous task completion, particularly for smaller models. However, the Skill-Prompt baseline and the distillation teacher assess different capabilities. Their settings also differ: Skill-Prompt uses general skills without retrieving task-specific skills, whereas the teacher’s context includes both general skills and a retrieved skill matched to the task. In addition, Skill-Prompt requires the model to use general skills to autonomously generate and execute an entire successful trajectory. The distillation teacher instead scores student-generated tokens conditioned on an existing interaction prefix. A smaller model may struggle to use skills to complete an entire task successfully while still benefiting from skill guidance at individual decisions.

3. End-to-end skill ablation.

To directly examine the contribution of the skill context, we conducted end-to-end PCSD training experiments under three conditions: general skills only (no retrieval), random skills, and the full skill configuration. The random-skill condition uses the same general skills as the full configuration and replaces only the retrieved task-specific skill with a randomly sampled skill. All three conditions use the same distillation objective and training settings.

SettingPickLookCleanHeatCoolPick2Overall
GRPO71.141.736.440.031.831.646.1
General skills only (no retrieval)56.225.071.860.669.022.254.7
Random skills56.733.346.465.251.247.250.0
Full skills (PCSD)63.368.850.069.248.163.659.4

Comparing full skills with general skills alone tests whether the retrieved skill context provides additional training value. Comparing full and random skills tests the importance of matching skills to the task. PCSD achieved overall success rates of 54.7% with general skills alone and 50.0% with random skills, both above the GRPO baseline of 46.1%, with general skills outperforming random skills by 4.7 percentage points. Under the same distillation settings, combining general skills with retrieved, task-matched skills raised the success rate to 59.4%, a gain of 4.7 percentage points over general skills alone and 9.4 over random skills, indicating that task-matched skills provide additional value during training.

W2: Response to the comment on figure readability.

Thank you for pointing this out. We have enlarged the figures in the revised PDF.

Reviewer_L44pOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Persistent Consistency Self-Distillation (PCSD), a framework for improving agentic reinforcement learning through reliable token-level teacher supervision. The authors argue that existing on-policy self-distillation methods either rely on noisy token-level teacher–student discrepancies or use coarse step-level aggregation that overlooks positional differences. PCSD addresses this by estimating teacher reliability through persistent local support, using adaptive window aggregation, exponential decay, trend modulation, and sigmoid gating to assign token-level distillation weights. The resulting objective is jointly optimized with GRPO. Experiments on ALFWorld and WebShop using two model backbones demonstrate improvements over GRPO and existing self-distillation methods, particularly on ALFWorld.

Critical strengths
  1. Well-motivated problem: The paper addresses the important challenge of unreliable teacher supervision in agentic RL. Aggregating local evidence to reduce sensitivity to noisy token-level signals is an intuitive approach.

  2. Strong empirical improvements: PCSD consistently outperforms GRPO and SDAR across both backbones, with gains of 5.5–6.2 percentage points over SDAR. The improvements also extend to unseen ALFWorld environments.

  3. Useful ablation studies: The authors examine adaptive windowing, exponential weighting, trend modulation, and distillation strength. These experiments support the contributions of the individual components.

Critical weaknesses
  1. Limited validation of teacher reliability estimation: The method assumes that persistent teacher–student probability gaps indicate reliable supervision. While the appendix provides robustness and teacher-quality analyses, the correlations between weight changes and teacher-quality changes remain weak. More direct evidence linking the proposed weights to the correctness of teacher guidance would strengthen the central motivation.

  2. Limited experimental breadth and statistical validation: Experiments are restricted to two relatively small models and two agentic benchmarks, with the strongest improvements concentrated on ALFWorld. Results across multiple independent training seeds or confidence intervals are not reported, making it difficult to assess the robustness of the improvements over existing methods.

Questions for authors
  1. PCSD shows substantial improvements on ALFWorld, but its gains on WebShop are relatively modest, with identical success rates to SDAR. What explains this difference, and how well does the method generalize to more challenging agentic tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer L44p

We thank the reviewer for the positive assessment of the paper’s motivation, experimental improvements, and component ablations, as well as for the constructive suggestions on reliability validation and experimental robustness.

W1: Interpretation of persistent consistency weight.

We thank the reviewer for suggesting a more direct test of the reliability of teacher guidance. We agree that a persistent teacher–student probability gap does not guarantee that the teacher’s guidance is correct. In PCSD, this gap measures how strongly the teacher supports the same student-sampled token relative to the student. Local aggregation uses information from neighboring positions to make the allocation of distillation strength less sensitive to isolated fluctuations. Therefore, the resulting weight is a proxy for the extent to which the local teacher signal can be used for distillation. It cannot be directly interpreted as the probability that the teacher’s guidance is correct. We will clarify this limitation in the revised manuscript.

The existing experiments examine this mechanism from different angles. Appendix A.11 and Figure 6 show that when isolated perturbations affect the teacher–student probability gaps at more positions, PCSD better preserves the original ranking of token weights. When 5% of positions are perturbed, the rank correlations before and after perturbation are 0.934 and 0.929 for PCSD and SDAR, respectively. When the proportion increases to 10%, the corresponding values are 0.887 and 0.864. Table 2 provides evidence from downstream task performance. Full PCSD achieves a success rate of 90.6% on ALFWorld, while the variants with independent per-token weighting, a fixed window, and trend modulation removed achieve success rates of 82.8%, 88.3%, and 83.6%, respectively. These results support the practical value of allocating distillation weights based on persistent teacher support. We will describe persistent consistency more explicitly as a heuristic for allocating supervision strength and clarify the extent to which the existing experiments validate its relationship with the correctness of teacher guidance.

W2: Statistical validation and experimental coverage.

Due to training costs, our original results were based on a single training run for each setting, using the checkpoint at step 150. To assess the robustness of our method, we conducted two additional independent training runs with different random seeds for both SDAR and PCSD using Qwen2.5-3B on ALFWorld and WebShop. Including the original run, this gives three runs for each method on each benchmark.

Both methods used random seeds 0, 42, and 100. On ALFWorld, PCSD outperformed SDAR across all three seeds, with the mean overall success rate increasing from 84.13% to 90.80%, a gain of 6.67 percentage points. On WebShop, PCSD achieved a higher Score than SDAR across all three seeds, with the mean Score increasing from 83.40 to 84.67, a gain of 1.27 points. WebShop accuracy was the same for both methods at seed 0 and improved by 3.1 and 0.8 percentage points at seeds 42 and 100, respectively. Mean accuracy increased from 67.47% to 68.77%, a gain of 1.30 percentage points.

Dataset and MetricMethodSeed 0Seed 42Seed 100Mean ± Sample SD95% CI for the Mean
ALFWorld OverallSDAR84.482.885.284.13 ± 1.22[81.10, 87.17]
ALFWorld OverallPCSD90.690.691.290.80 ± 0.35[89.94, 91.66]
WebShop ScoreSDAR83.481.885.083.40 ± 1.60[79.43, 87.37]
WebShop ScorePCSD85.082.986.184.67 ± 1.63[80.63, 88.71]
WebShop AccuracySDAR67.267.268.067.47 ± 0.46[66.32, 68.61]
WebShop AccuracyPCSD67.270.368.868.77 ± 1.55[64.92, 72.62]

Q1: Reasons for the smaller gains on WebShop and the generalization ability of PCSD.

The reviewer’s observation is correct. In the originally reported experiments, PCSD did not improve the strict task success rate on WebShop relative to SDAR. However, PCSD increased the Score from 83.4 to 85.0 for Qwen2.5-3B and from 76.8 to 78.9 for Qwen3-1.7B, while maintaining success rates of 67.2% and 58.6%, respectively.

As defined in Appendix A.9, the two metrics measure different outcomes. Score measures how closely the purchased products match the users’ requirements on average, while the success rate counts only purchases that satisfy all requirements. The results therefore indicate an improvement in average match quality, while the number of fully completed tasks remained unchanged.

The difference between the two benchmarks may be related to the type of supervision that is effective for each. ALFWorld involves reusable procedures, such as locating an object, performing the required operation, and placing it at the target location. These procedures may benefit more directly from persistent local support. WebShop also depends on instance-specific product retrieval, attribute verification, and option selection. Improved local supervision may help satisfy more of the user’s requirements, but a single unmet constraint can still prevent a task from counting as fully successful.

In the three-seed experiments reported in W2, PCSD also showed some gains in strict task success rate (Acc) on WebShop with Qwen2.5-3B. Both methods achieved a success rate of 67.2% at seed 0. At seeds 42 and 100, PCSD outperformed SDAR by 3.1 and 0.8 percentage points, respectively. Across the three seeds, PCSD achieved a mean success rate of 68.77%, an improvement of 1.30 percentage points over SDAR’s 67.47%.

Regarding generalization, the results on the ALFWorld unseen split provide evidence of transfer to new environments and layouts. PCSD achieved a success rate of 86.7%, compared with 72.7% for SDAR and 70.9% for GRPO, without requiring skill retrieval during evaluation. These results support generalization to unseen environments within ALFWorld and suggest that PCSD may be more effective on similar tasks that involve reusable procedures and longer-range dependencies between steps.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all three reviewers for their careful reading and constructive feedback. Their questions helped us clarify how PCSD assigns token-level supervision, test the contribution of task-matched skills, and assess how consistently the gains hold across training runs.

Robustness and generalization (Reviewer L44p and grd8): We ran two additional seeds for both PCSD and SDAR on ALFWorld and WebShop with Qwen2.5-3B. All comparisons used the same evaluation tasks and the checkpoint after 150 training updates. Across three seeds, PCSD outperformed SDAR on ALFWorld in every run, raising mean success from 84.13% to 90.80%. On WebShop, PCSD also improved Score in every run, from 83.40 to 84.67 on average. The gain in strict success was smaller: the methods tied on one seed, and PCSD improved on the other two, raising the mean from 67.47% to 68.77%. This distinction matters because Score gives partial credit for meeting purchase requirements, while strict success requires meeting all of them. On the ALFWorld unseen split, PCSD achieved 86.7% success versus 72.7% for SDAR, providing evidence of transfer to new environments within this benchmark.

Contribution of privileged skills (Reviewer 3Lt2 and grd8): Skill-Prompt asks a model to complete an entire task using general skills. In PCSD, the teacher receives both general skills and a skill retrieved from the observable task instruction, then scores student-generated tokens given the same interaction history. The student receives no skill text during training or evaluation. In a separate end-to-end ALFWorld ablation, success reached 59.4% with task-matched retrieval, compared with 54.7% using general skills alone and 50.0% when a random skill replaced the matched one. With the training objective and settings held fixed, these results show that matching the teacher’s skill context to the task improves downstream performance. All three teacher-guided settings outperformed GRPO(46.1%) trained without a teacher. These results support the effectiveness of teacher supervision during training.

Interpretation of token weights (Reviewer L44p): PCSD uses persistent local teacher support as a heuristic for allocating distillation strength. The perturbation experiments show that local aggregation makes token-weight rankings more stable, and replacing it with independent per-token weighting reduced ALFWorld success from 90.6% to 82.8%. These experiments establish the practical value of the weighting rule. Whether its weights predict the correctness of individual teacher judgments remains an open question.

The revised PDF includes StepOPSD in the main comparison, training curves in Appendix A.12, and larger figures. We will also clarify the interpretation of the weights in the main text and document the skill library’s source with actual entries and a worked retrieval example.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.