- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose PLUME, which leverages rollout groups generated during GRPO training to extract skills and maintain a skill bank. The framework incorporates OPSD, using the previous training step’s checkpoint augmented with the skill bank as a teacher to guide student training. The authors report two benefits: (1) improved student performance within the training domain compared with GRPO, OPSD, and related methods; and (2) the transferability of the accumulated skills to third-party models, improving their performance in the same domain.
Critical strengths
The paper explores the untapped value of rollout groups in GRPO and combines reinforcement learning with on-policy distillation during training.
Critical weaknesses
-
The core training mechanism is largely inherited from prior work, while the mechanistic justification for the newly introduced components remains insufficient. The paper explains the motivations behind within-group comparison, cross-task generalization, and skill persistence, but does not adequately clarify why skills generated and continuously accumulated by the current policy itself can provide more effective supervision than existing RL and self-distillation methods, or what assumptions this expectation relies on.
-
The paper lacks generalization experiments, leaving it unclear whether this training procedure degrades the model's capabilities in other domains. From the perspective of training small models specialized for a particular domain, comparisons with SFT and KD are missing. Although PLUME does not require an external model, the current results do not establish sufficient performance or cost advantages over simpler approaches, especially considering the additional computational overhead introduced by skill summarization
Questions for authors
-
The experimental results in Table 1 show that PLUME generally outperforms GRPO+OPSD. What accounts for this improvement?
-
How can skills learned for specific tasks during training be applied in real-world scenarios?
-
Do the authors view PLUME as a general-purpose training approach, or as a method intended to specialize models for particular domains?
5 comments
Author Response to the Official Review of Submission 17146 by Reviewer FqkL [2/2]
Q3. Is PLUME intended for general-purpose training or domain specialization? How could the learned skills be useful in practice?
PLUME is a training procedure that can be applied across interactive environments. In this paper we use it for domain specialization, training a separate policy in ALFWorld, WebShop, and SearchQA. A practical setting is a shopping sandbox where the final order can be checked against the customer’s requirements. An error on one product can yield a lesson about verifying the selected variant, which can inform training on other products. At deployment, the resulting policy uses its ordinary interaction history and requires no skill lookup.
The three environments establish applicability across distinct tasks. Whether a WebShop-trained policy retains its capabilities in other domains remains open. Table 3 tests a different form of transfer: whether another model can use the explicit bank without parameter updates.
Q4. Why are SFT and KD comparisons missing if the goal is to train a domain specialist?
We agree that SFT and KD are relevant approaches to training specialized agents. Many agent tasks provide an interactive environment and an outcome verifier but no expert demonstrations. In this setting, training on teacher trajectories requires first generating them by having the teacher interact with the environment. A common way to learn from a teacher’s successful behavior is to use rejection sampling to collect demonstrations for SFT or trajectory-based KD. The teacher makes multiple attempts, and a verifier selects successful trajectories for training. In a multi-turn agent task, every attempt requires repeated teacher inference and environment interaction, including attempts that are ultimately discarded. Synthesizing trajectories therefore incurs both teacher inference costs and the cost of executing actions in the environment, even for trajectories discarded by rejection sampling. Direct KD also incurs the cost of querying the teacher for action preferences.
PLUME studies a complementary setting in which the agent has access to an environment and verifiable outcomes and learns from its own interactions. As in reasoning RLVR, outcome rewards drive policy updates. PLUME also compares the actions and environment responses across successful and failed attempts, extracts lessons, and generalizes them across tasks. The resulting skills guide subsequent updates through on-policy self-distillation. This approach incurs additional training computation for skill extraction and distillation, while requiring neither an external teacher nor teacher-generated demonstrations. Our comparisons with GRPO, GRPO+OPSD, and SDAR examine how effectively the agent can use its rollout experience under this setting.
The same training process also produces a skill bank that other models can use through prompting alone, without access to the original trajectories or any parameter updates. As shown in Table 3, the bank learned by Qwen3-4B improved all three recipient models on WebShop and on both the seen and unseen ALFWorld splits.
Response to authors
Thank you for the clarifications and the additional ablation results. The revision explains the motivation for extracting reusable skills from rollouts, and the new ablations show gains within PLUME. The paper does not yet establish whether these gains justify the extra compute and training complexity. I still consider the methodological contribution incremental, and whether the trained policies retain their capabilities outside their training domains remains untested. I therefore maintain my original rating.
Author Response to the Official Review of Submission 17146 by Reviewer FqkL [1/2]
Q1. Why can skills generated by the current policy provide useful supervision, and what assumptions does this require?
The policy observes different evidence when acting and when analyzing its experience. Before choosing an action, it sees one partial interaction history. After rollout, it can inspect alternative attempts at the same task, the environment’s responses, and the eventual outcomes. For example, two shopping attempts may reach the same product but differ in whether the requested size was selected before checkout. Comparing them can suggest a candidate lesson about checking the selected variant. Retaining that lesson allows the teacher to draw on it when scoring a later trajectory involving a different product, even if the later batch contains no similarly informative contrast.
Our approach relies on two assumptions. First, a decision can be easier to assess in hindsight, when the policy can compare complete attempts and their outcomes, than to discover during exploration from a partial interaction history. Second, when a lesson from one task also applies to other tasks, on-policy self-distillation can help the policy internalize that lesson and act without the skill bank.
Q2. What accounts for the improvement over GRPO+OPSD in Table 1?
GRPO+OPSD baseline draws its teacher guidance from the fixed skill library released by SkillRL using strong teacher. PLUME builds its bank from the policy’s own rollouts and updates it throughout training. As the policy changes, newly observed successes and failures can therefore inform the guidance used in later updates.
Table 2 provides a direct test of continued updates on WebShop with Qwen3-4B. Both variants generate skills from the policy’s own experience. Freezing the bank once it reaches its token budget reduces success from 72.93% to 69.60%. Clearing the skill bank after every update gives 69.13%, showing the benefit of retaining earlier lessons. The largest drop occurs when each lesson is extracted from a single trajectory: success falls to 56.33%. Among the choices tested, comparing alternative attempts at the same task had the largest observed effect, with smaller gains from retaining and updating the resulting skills.