acceptodds
Under review as a conference paper at ICLR 2027

PLUME: PERSISTENT LEARNING FROM UNIFIED MULTI-ROLLOUT EXPERIENCE FOR AGENTIC REIN- FORCEMENT LEARNING

Abstract

Large language models are increasingly trained to act in multi-step agent environments, and group-relative policy optimization (GRPO) with verifiable rewards is widely used to train such agents. However, its trajectory-level advantages provide limited guidance on which decisions contributed to the outcomes, and the sampled trajectories are discarded after policy updates without their textual experience being distilled into reusable lessons. We introduce PLUME (Persistent Learning from Unified Multi-Rollout Experience), a framework that learns from rollout groups not only in numerical space through GRPO’s relative advantages but also in text space through contrastive analysis. Within each task, PLUME compares alternative trajectories to extract lessons from successes and failures. It then compares these task-specific lessons across a batch to distill generalizable skills, which are accumulated in a persistent skill bank across training steps. Both trajectory analysis and skill generalization are performed by the policy model itself, requiring neither a stronger external teacher nor externally supplied skills. The accumulated skills guide policy optimization through self-distillation, without being inserted into the policy’s rollout or inference-time prompts. PLUME thus yields two artifacts: an improved policy and a standalone skill bank for transfer. Experiments on ALFWorld, WebShop, and search-augmented QA show consistent gains across three model scales. The resulting skill bank also transfers zero-shot to larger models from different families, improving performance in almost all evaluations across the three benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FqkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose PLUME, which leverages rollout groups generated during GRPO training to extract skills and maintain a skill bank. The framework incorporates OPSD, using the previous training step’s checkpoint augmented with the skill bank as a teacher to guide student training. The authors report two benefits: (1) improved student performance within the training domain compared with GRPO, OPSD, and related methods; and (2) the transferability of the accumulated skills to third-party models, improving their performance in the same domain.

Critical strengths

The paper explores the untapped value of rollout groups in GRPO and combines reinforcement learning with on-policy distillation during training.

Critical weaknesses
  1. The core training mechanism is largely inherited from prior work, while the mechanistic justification for the newly introduced components remains insufficient. The paper explains the motivations behind within-group comparison, cross-task generalization, and skill persistence, but does not adequately clarify why skills generated and continuously accumulated by the current policy itself can provide more effective supervision than existing RL and self-distillation methods, or what assumptions this expectation relies on.

  2. The paper lacks generalization experiments, leaving it unclear whether this training procedure degrades the model's capabilities in other domains. From the perspective of training small models specialized for a particular domain, comparisons with SFT and KD are missing. Although PLUME does not require an external model, the current results do not establish sufficient performance or cost advantages over simpler approaches, especially considering the additional computational overhead introduced by skill summarization

Questions for authors
  1. The experimental results in Table 1 show that PLUME generally outperforms GRPO+OPSD. What accounts for this improvement?

  2. How can skills learned for specific tasks during training be applied in real-world scenarios?

  3. Do the authors view PLUME as a general-purpose training approach, or as a method intended to specialize models for particular domains?

5 comments
AuthorsOpenReview ↗

Author Response to the Official Review of Submission 17146 by Reviewer FqkL [1/2]

Q1. Why can skills generated by the current policy provide useful supervision, and what assumptions does this require?

The policy observes different evidence when acting and when analyzing its experience. Before choosing an action, it sees one partial interaction history. After rollout, it can inspect alternative attempts at the same task, the environment’s responses, and the eventual outcomes. For example, two shopping attempts may reach the same product but differ in whether the requested size was selected before checkout. Comparing them can suggest a candidate lesson about checking the selected variant. Retaining that lesson allows the teacher to draw on it when scoring a later trajectory involving a different product, even if the later batch contains no similarly informative contrast.

Our approach relies on two assumptions. First, a decision can be easier to assess in hindsight, when the policy can compare complete attempts and their outcomes, than to discover during exploration from a partial interaction history. Second, when a lesson from one task also applies to other tasks, on-policy self-distillation can help the policy internalize that lesson and act without the skill bank.

Q2. What accounts for the improvement over GRPO+OPSD in Table 1?

GRPO+OPSD baseline draws its teacher guidance from the fixed skill library released by SkillRL using strong teacher. PLUME builds its bank from the policy’s own rollouts and updates it throughout training. As the policy changes, newly observed successes and failures can therefore inform the guidance used in later updates.

Table 2 provides a direct test of continued updates on WebShop with Qwen3-4B. Both variants generate skills from the policy’s own experience. Freezing the bank once it reaches its token budget reduces success from 72.93% to 69.60%. Clearing the skill bank after every update gives 69.13%, showing the benefit of retaining earlier lessons. The largest drop occurs when each lesson is extracted from a single trajectory: success falls to 56.33%. Among the choices tested, comparing alternative attempts at the same task had the largest observed effect, with smaller gains from retaining and updating the resulting skills.

AuthorsOpenReview ↗

Author Response to the Official Review of Submission 17146 by Reviewer FqkL [2/2]

Q3. Is PLUME intended for general-purpose training or domain specialization? How could the learned skills be useful in practice?

PLUME is a training procedure that can be applied across interactive environments. In this paper we use it for domain specialization, training a separate policy in ALFWorld, WebShop, and SearchQA. A practical setting is a shopping sandbox where the final order can be checked against the customer’s requirements. An error on one product can yield a lesson about verifying the selected variant, which can inform training on other products. At deployment, the resulting policy uses its ordinary interaction history and requires no skill lookup.

The three environments establish applicability across distinct tasks. Whether a WebShop-trained policy retains its capabilities in other domains remains open. Table 3 tests a different form of transfer: whether another model can use the explicit bank without parameter updates.

Q4. Why are SFT and KD comparisons missing if the goal is to train a domain specialist?

We agree that SFT and KD are relevant approaches to training specialized agents. Many agent tasks provide an interactive environment and an outcome verifier but no expert demonstrations. In this setting, training on teacher trajectories requires first generating them by having the teacher interact with the environment. A common way to learn from a teacher’s successful behavior is to use rejection sampling to collect demonstrations for SFT or trajectory-based KD. The teacher makes multiple attempts, and a verifier selects successful trajectories for training. In a multi-turn agent task, every attempt requires repeated teacher inference and environment interaction, including attempts that are ultimately discarded. Synthesizing trajectories therefore incurs both teacher inference costs and the cost of executing actions in the environment, even for trajectories discarded by rejection sampling. Direct KD also incurs the cost of querying the teacher for action preferences.

PLUME studies a complementary setting in which the agent has access to an environment and verifiable outcomes and learns from its own interactions. As in reasoning RLVR, outcome rewards drive policy updates. PLUME also compares the actions and environment responses across successful and failed attempts, extracts lessons, and generalizes them across tasks. The resulting skills guide subsequent updates through on-policy self-distillation. This approach incurs additional training computation for skill extraction and distillation, while requiring neither an external teacher nor teacher-generated demonstrations. Our comparisons with GRPO, GRPO+OPSD, and SDAR examine how effectively the agent can use its rollout experience under this setting.

The same training process also produces a skill bank that other models can use through prompting alone, without access to the original trajectories or any parameter updates. As shown in Table 3, the bank learned by Qwen3-4B improved all three recipient models on WebShop and on both the seen and unseen ALFWorld splits.

Reviewer_FqkLOpenReview ↗

Response to authors

Thank you for the clarifications and the additional ablation results. The revision explains the motivation for extracting reusable skills from rollouts, and the new ablations show gains within PLUME. The paper does not yet establish whether these gains justify the extra compute and training complexity. I still consider the methodological contribution incremental, and whether the trained policies retain their capabilities outside their training domains remains untested. I therefore maintain my original rating.

Reviewer_Ai4BOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A single model is given the GRPO advantages and reflects on a useful skill/lesson from the successes/failures. This is checked and may make it into a skill bank. The student is then conditioned on the skill to get new "teacher" logprobs for on-policy self-distillation. The policy is then updated with GPRO+distillation.

Critical strengths
  • Sensible method which is made up of lots of parts that all fit together nicely - essentially having the model generate its own privileged teacher information (ie the skill) and also reusing it in the form of a skill bank.
  • Explained clearly with a good set of baselines and ablations.
Critical weaknesses
  • Would be interesting to have a proper discussion of compute cost comparisons to the baselines.
  • Would also be interesting to have some analysis/plots of how often skills are used in other trajectories (apologies if I missed it).
  • Not sure how novel and how much of a contribution it is given the related work. I think this is probably the main potential issue.
Questions for authors
  • I am not that familiar with the current existing work - is the only change compared to prior work is the addition of a skill bank? If not then what is the closest work and what are the differences?

  • See weaknesses

3 comments
AuthorsOpenReview ↗

Author Response to the Official Review of Submission 17146 by Reviewer Ai4B

Q1. Is the main difference from prior work the addition of a skill bank? Which work is closest to PLUME?

The skill bank is one part of PLUME. Its broader contribution is to connect group-wise trajectory comparison, persistent skill accumulation, and on-policy self-distillation with outcome-based GRPO in a single learning loop. This process yields both a trained policy that acts without the bank and an explicit set of reusable skills. As Table 3 shows, these skills can also improve other models’ performance on held-out tasks through prompting alone.

GRSD is the closest comparison in how the two methods learn from groups of rollouts. Both compare attempts at the same task. GRSD contrasts reflections from successful and failed attempts to produce guidance for assigning credit across turns within that task. PLUME takes the lesson from each task and compares it with lessons from other tasks to derive reusable skills. It retains those skills in a persistent bank, allowing experience from one task to guide the teacher when scoring actions on later tasks.

We also compared PLUME with GRSD on WebShop across three Qwen3 model scales. We evaluated the step-150 checkpoints on the same held-out goals and averaged the results over three decoding seeds. PLUME achieved higher test scores and success rates at all three scales.

BackbonePLUME scoreGRSD scorePLUME success rateGRSD success rate
Qwen3-1.7B74.6061.7456.40%36.33%
Qwen3-4B85.5177.0072.93%65.87%
Qwen3-8B86.5180.2479.67%71.20%

Q2. How often are skills reused in other trajectories?

PLUME maintains one shared skill bank during training run. At each update, the policy compares trajectories from the same prompt to extract a group-level lesson, then compares lessons across tasks to update the bank. During OPSD, the teacher uses the same updated skill bank to re-score every sampled trajectory in the batch. We share the same skills across prompts because our goal is to extract strategies that generalize across task instances within an environment. Once a skill enters the bank, the teacher use it to re-score trajectories from all prompt in the current batch and in subsequent batches while the skill remains in the bank.

We also tracked all 195 distinct skills added during the 150-step WebShop Qwen3-4B run. Every skill stayed in the bank for at least one subsequent batch, and 149 (76.4%) stayed for at least ten subsequent batches. Among the 161 skills that left the bank before training ended, the median was 25 training steps, including the step when the skill was added.

Training steps in the skill bankLeft the bank before training endedStill in the bank at step 150Total skills
1–5415
6–10261541
11–2031536
21–30351348
31–5065065
Total16134195

Each skill is counted once. For the 34 skills still in the bank at step 150, the table counts updates through the end of training. These counts measure how often skills were available to the teacher. Table 2 also provides complementary evidence of their collective value: clearing the bank after every update while continuing to generate new skills reduced WebShop test success from 72.93% to 69.13%.

Q3. What is the computational cost relative to the baselines?

We ran GRPO with the same rollout implementation and settings used for PLUME. On eight H800 GPUs, 150 WebShop updates took 40.48 training GPU-hours for PLUME and 33.27 for GRPO, including rollout collection and all method-specific training computation. On the same goal test, PLUME achieved 85.51 score / 72.93% success, compared with 75.93 / 64.20% for GRPO. At this interaction budget, the additional 7.21 GPU-hours accompanied gains of 9.58 score points and 8.73 percentage points in success.

Thank you for your response and the new results. My main concern remains how much of a contribution it is given the surrounding literature. I am going to maintain my score but note that other reviewers' stronger concerns of novelty may be more accurate as I am not that familiar with the related work.

AuthorsOpenReview ↗

Clarification of PLUME’s Contributions and Relation to Prior Work

Dear Reviewer Ai4B,

Thank you for the follow-up. To address the remaining concern about PLUME’s contribution relative to prior work, we summarize its three main distinctions and the supporting evidence below.

  1. No external strong model is required. The policy derives skills from its own interactions and environment feedback, without external skill annotations or a separate teacher or judge model.

  2. No predefined skill library is required. PLUME compares attempts within tasks, consolidates lessons across tasks, and continually updates a persistent bank. The skills guide self-distillation, and the updated policy generates new experience, forming a closed loop of exploration, skill extraction, and policy improvement.

  3. The extracted skills transfer to other models. In Table 3, skills learned by Qwen3-4B improve all three recipient models on WebShop and both ALFWorld splits through prompting alone. PLUME produces reusable skills alongside an improved policy that acts without skill retrieval.

Related methodComparison with PLUME
SEEDStarts with externally annotated skill-SFT and derives skills to supervise individual trajectories. PLUME learns without this SFT stage and retains skills induced across tasks for later updates.
OPIDUses an external model for hindsight guidance. PLUME generates skill supervision with the policy itself.
GRSDUses policy-generated reflections and an external judge to guide learning for each prompt. PLUME requires no external judge and accumulates skills across prompts and updates.
SkillRLRetrieves skills from an evolving library during task execution. PLUME internalizes skills through teacher-side distillation and deploys without skill retrieval.
SDARUses a fixed, externally generated skill library in our comparison. PLUME constructs and continually updates its own skill bank from rollout experience.

The new ablation holds the persistent bank and distillation objective fixed: cross-task lesson comparison improves WebShop score from 82.94 to 85.51. Existing ablations test persistence and continued updates, and additional comparisons show higher scores and success rates than GRSD at all three evaluated model scales.

We hope this clarifies our contribution and would appreciate your assessment of these distinctions and results.

Thank you for your time.

Reviewer_evQSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PLUME, which extracts reusable skills from multiple rollouts, maintains a persistent skill bank, and internalizes these skills through on-policy self-distillation. Experiments on ALFWorld, WebShop, and SearchQA show improvements over GRPO and several self-distillation baselines.

Critical strengths
  1. PLUME achieves strong results, particularly on ALFWorld and WebShop, across three model scales.
  2. The paper is clearly written, with useful ablations and skill-transfer experiments.
Critical weaknesses
  1. The main components have been explored in prior work. SEED and OPID investigate skill internalization through self-distillation, GRSD uses group-level reflection, and SkillRL maintains evolving skill libraries. PLUME mainly combines these existing ideas, and adopting the same distillation objective from SDAR. However, the paper does not clearly justify the added value of combining these components.

  2. Insufficient validation of the key contributions. The ablations do not isolate the gains of cross-task skill generalization. Moreover, the self-distillation baselines use externally generated skills, while PLUME uses dynamically updated skills. This makes it difficult to identify which components drive for the improvements.

Questions for authors
  1. What distinguishes PLUME from SEED, GRSD, and SkillRL beyond combining their existing ideas?

  2. Could the authors provide an ablation removing cross-task skill generalization?

  3. Does PLUME still outperform the baselines under a matched training compute budget, given its additional analysis and distillation overhead?

3 comments
AuthorsOpenReview ↗

Author Response to the Official Review of Submission 17146 by Reviewer evQS [1/2]

Q1. What distinguishes PLUME from SEED, OPID, GRSD, and SkillRL beyond combining their components?

We agree that prior work established several ingredients we use, including skill internalization, group reflection, and persistent skill libraries. We explicitly adopt the gated distillation objective from SDAR. PLUME’s contribution is a way to construct persistent supervision from the policy’s own experience. The scope of that supervision provides a concrete distinction among the methods.

MethodWhere a lesson comes from and how it is used
SEEDAfter SFT on externally generated skill annotations, SEED extracts a hindsight skill from each completed trajectory and uses it to supervise that trajectory in the same policy update.
OPIDEpisode- and step-level hindsight from a completed trajectory guides scoring at decisions within that trajectory.
GRSDReflections on attempts at one prompt produce guidance that modulates turn-level advantages for that prompt.
SkillRLAn evolving skill library supplies retrieved guidance to the policy during task execution.
PLUMEThe policy compares attempts within a task and lessons across tasks. Retained rules then condition the teacher on subsequent batches, while the student acts on ordinary task inputs.

The connection between these stages matters. A group-level lesson may name the exact product or action that distinguished two attempts. Comparing lessons across tasks gives the analyzer evidence for which part of that decision can carry over. Retention makes the resulting rule available after its source batch, and teacher-side distillation gives it a route into policy updates. The following experiment tests the cross-task comparison while preserving the other stages.

Q2. Can the authors isolate the contribution of cross-task skill generalization?

We ran the requested ablation on WebShop with Qwen3-4B. Each group still produced one lesson from eight attempts at the same task. The Skill Analyzer then processed one task’s lesson at a time. It could still rewrite or skip that lesson, and all accepted rules entered the same shared, persistent bank. Consequently, both variants could reuse rules across tasks; only the full method could compare different tasks’ lessons when inducing a rule.

WebShop Qwen3-4B, step 150ScoreSuccess
Independent lesson induction82.94 ± 1.6271.87% ± 1.18
Ours85.51 ± 0.2272.93% ± 0.38

Joint induction improved mean score by 2.57 points and mean success by 1.07 percentage points. Its score was higher under all three evaluation seeds. The variation shown is across decoding seeds on the same tests for each checkpoint. We will add the independent-induction row to Table 2. And under our WebShop protocol, PLUME outperformed GRSD in both score and success rate at all three Qwen3 scales tested (1.7B, 4B, and 8B). This comparison evaluates the two complete methods. The ablation below isolates the contribution of comparing lessons across tasks while retaining the same persistent bank and distillation objective.

AuthorsOpenReview ↗

Author Response to the Official Review of Submission 17146 by Reviewer evQS [2/2]

Q3. External fixed skills and PLUME’s dynamic skills differ in more than one respect. Can the gains be attributed to individual components?

Table 1 includes SDAR to show what the same GRPO and gated self-distillation objectives achieve when the teacher draws on a fixed library of externally supplied skills. Using SDAR’s skill library and retrieval procedure in place of PLUME’s online skill generation and bank updates recovers the SDAR baseline.

Table 2 then examines how individual choices in skill extraction, retention, and updating affect performance within PLUME. In particular, freezing PLUME’s own bank keeps the source of the skills fixed and tests the value of continued updates. The largest drop occurs when the analyzer sees only one attempt at a task, which supports using alternative trajectories to extract a lesson. The frozen-bank comparison directly addresses the question about dynamic skills: the policy generates the skills in both runs. Allowing the bank to continue updating after it fills improves score by 2.47 points and success by 3.33 percentage points. The new independent-induction control tests a different question. Both variants continually update a shared bank, but only full PLUME compares lessons across tasks when writing skills. This comparison yields a further 2.57-point difference in score.

Change from full PLUMETest scoreSuccess rate
Full PLUME85.5172.93%
Extract each lesson from a single trajectory70.7856.33%
Process each task’s lesson independently during skill induction82.9471.87%
Clear the skill bank after each update80.3669.13%
Freeze the skill bank once it reaches its token budget83.0469.60%

Together, these controls provide evidence for the individual design choices within PLUME. Each difference is measured against the full method and should be interpreted separately.

Q4. Does PLUME still outperform the baselines under a matched compute budget?

We compare PLUME, GRPO, and SDAR on WebShop with Qwen3-4B using verl on eight NVIDIA H800GPUs. We matched PLUME’s rollout implementation and settings. Training GPU-hours include rollout collection and all method-specific computation, including skill extraction, bank updates, and teacher scoring for PLUME.

Matched training compute. We use mean training episode reward to measure learning progress at the same GPU-hour budget. Each episode receives a reward of 10 for a successful purchase and 0 otherwise. We average rewards across rollouts within each update, then average the update means over the 3 GPU-hours preceding each budget point. At the largest budget shared by all three runs, 33.27 GPU-hours, PLUME reached a mean reward of 6.11, compared with 5.92 for GRPO and 5.67 for SDAR. PLUME and GRPO traded the lead during training and achieved comparable rewards at this budget.

Training GPU-hoursPLUMEGRPOSDAR
101.821.260.62
153.152.821.47
204.624.402.85
255.245.344.09
306.026.345.62
33.276.115.925.67

Matched rollout budget. All three methods sampled 16 tasks per update and eight trajectories per task, for a total of 19,200 trajectories over 150 updates. We evaluated the resulting policies on the same held-out WebShop goals and averaged the results over three decoding seeds.

At 150 updatesPLUMEGRPOSDAR
Training GPU-hours40.4833.2747.19
Score85.5175.9378.25
Success rate72.93%64.20%66.80%

Relative to GRPO, PLUME used 7.21 additional GPU-hours and achieved a 9.58-point higher held-out score and an 8.73 percentage-point higher success rate. It also outperformed SDAR while using fewer training GPU-hours in these runs. The held-out gains at a fixed trajectory budget suggest that PLUME makes more effective use of its rollout data: each group supplies both an outcome-based policy update and experience from which the policy can extract skills for subsequent updates.

AuthorsOpenReview ↗

Gentle reminder of the author-reviewer discussion deadline

Dear Reviewer evQS,

Thank you again for your detailed review. We would like to check whether our responses have addressed your concerns. In brief, we provided the following.

  • A mechanism-level comparison with SEED, OPID, GRSD, SkillRL and SDAR, clarifying how the components form a closed learning loop.

  • The requested cross-task induction ablation. With the same persistent skill bank and distillation objective, PLUME achieves a WebShop score of 85.51, compared with 82.94 when each task’s lesson is processed independently. Ablation results examining group-level trajectory comparison, skill persistence, and continued bank updates.

  • Full-run GPU-hour measurements and comparisons at matched compute and rollout budgets. PLUME and GRPO achieve comparable training rewards at matched compute. At the same rollout budget, PLUME uses 7.21 additional GPU-hours and achieves an 8.73 percentage-point gain in held-out success.

If any point remains unclear, we would be glad to clarify it. If our responses have resolved your concerns, we would be grateful if you could consider updating your assessment.

Thank you for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Author Response to Submission 17146

We thank the reviewers for their thoughtful feedback. We have grouped overlapping points from the weaknesses and questions for authors by topic. Reviewers Ai4B and evQS asked how PLUME differs from closely related methods and what its components contribute when used together. Reviewer FqkL asked why skills derived from the policy’s own experience can guide subsequent policy updates. All three reviewers raised questions about computational cost. We address these shared concerns below and report the relevant experiments and analyses in our individual responses.

Relation to prior work.

Several reviewers asked how PLUME relates to prior work on reflection, skill libraries, and self-distillation. PLUME connects these mechanisms with outcome-based GRPO in a closed loop: the policy’s exploration produces the experience that supervises its subsequent updates, and the updated policy generates the next round of experience. The same rollout groups serve two purposes. GRPO uses their relative rewards to update the policy, while the policy compares the trajectories in text space to extract lessons and generalize them across tasks. Retaining these skills allows experience from one batch to guide self-distillation on later batches. The policy itself generates and updates the skills throughout training, without externally supplied skills or a separate skill-generation model.

MechanismPLUMESEEDOPIDGRSDSkillRLSDAR
Compares rollouts within each task, then compares the resulting lessons across tasks✓—————
Generates new skills or reflections from rollout experience during RL✓✓✓✓✓—
Retains and updates an explicit skill bank across training steps✓———✓—
Uses the policy itself to generate skill or reflection text during RL✓✓¹—✓²△³—
Uses skills or reflections as privileged context for on-policy self-distillation✓✓✓✓—✓
Deploys the policy without skill-augmented prompts✓✓✓✓—✓

¹ SEED first fine-tunes the policy on externally generated skill annotations.
² GRSD uses the policy to generate reflections and an external judge to score them.
³ SkillRL uses an external teacher in its main experiments and also evaluates policy-generated skills.

This loop produces two outputs from the same on-policy learning process: a policy that acts without skill context, and an explicit bank of skills that can be inspected and transferred independently of the policy weights. Table 3 tests the latter directly. Skills extracted during training improved all three recipient models on ALFWorld and WebShop when supplied through prompting alone, including on the ALFWorld unseen split.

Computational cost.

PLUME reuses the rollout groups collected for GRPO. Its additional training cost comes from analyzing these rollouts to extract and update skills, and from one additional teacher forward pass to re-score the sampled actions with the skill bank in context. As reported in section 4.5, these operations together took 33.0 seconds in the profiled first training step on WebShop with Qwen3-4B, accounting for approximately 19% of that step’s wall-clock time. We provide the full-run GPU-hour measurements and matched-budget comparisons in our individual responses below.

Manuscript revisions.

We revised the visual style of Figures 1 and 2, clarified the comparisons with SEED and SkillRL in section 2, and specified in section 4.1 that we train a separate policy in each environment. Table 2 now includes the cross-task induction ablation. The new section 4.4 discusses how the policy’s own rollout experience supplies privileged information and under what conditions updating the skill bank can improve supervision. We will include the GRSD comparison and compute analysis in the supplementary material.

AuthorsOpenReview ↗

Summary of Responses and Additional Results for Submission 17146

Dear Area Chair and Reviewers,

Thank you for your thorough and insightful reviews. We sincerely appreciate your valuable feedback, which has helped us improve the clarity and rigor of our paper. Below, we summarize the primary concerns raised and our corresponding responses:

  • Comparison with prior work (Reviewers evQS, Ai4B). We provide a systematic comparison with SEED, OPID, GRSD, SkillRL, and SDAR, clarifying how PLUME connects rollout comparison, reusable skill extraction, persistent updates, and self-distillation in a closed learning loop. We also report comparisons with GRSD on WebShop at three model scales.

  • Cross-task induction, persistence, and updates (Reviewers evQS, FqkL). The new ablation retains the same persistent bank and distillation objective while processing each task’s lesson independently. Comparing lessons across tasks improves WebShop score from 82.94 to 85.51. Our other ablations examine group-level trajectory comparison, retaining skills across updates, and continuing to update the bank.

  • Mechanistic justification and skill reuse (Reviewers FqkL, Ai4B). We explain how comparing completed rollouts and their outcomes provides hindsight information for subsequent learning. We clarify how trajectories share the skill bank during self-distillation and provide statistics on how long skills remain available across training updates.

  • Computational cost and comparison fairness (all three reviewers). We report full-run GPU-hours and comparisons at matched interaction and compute budgets. On WebShop 4B, PLUME uses 7.21 additional GPU-hours relative to GRPO over 150 updates and improves held-out success by 8.73 percentage points at the same sampled-trajectory budget. At matched training compute, the two methods achieve comparable training rewards.

  • Generalization and the training setting (Reviewers Ai4B, FqkL). We clarify the goal of domain-specific agent adaptation and highlight the policy evaluation on ALFWorld unseen room layouts in Appendix F, alongside skill-bank transfer in Table 3. We also explain the supervision setting motivating our comparisons with RL and self-distillation baselines and its relationship to SFT and KD with expert supervision.

  • Retention of general capabilities (Reviewer FqkL). In our latest response, we report new evaluations of the original Qwen3-4B and PLUME policies trained separately on ALFWorld, SearchQA, and WebShop. These cover instruction following (IFEval, 541 prompts), academic knowledge (MMLU, 14,042 questions), mathematical reasoning (GSM8K, 1,319 problems), and code completion (HumanEval, 164 problems). All three PLUME policies matched or exceeded the base model on IFEval and remained within 0.4 percentage points of its MMLU accuracy. GSM8K accuracy declined by 1.06–1.97 percentage points, and HumanEval results differed by 1–4 fewer solved problems. These results support that PLUME largely preserves the capabilities measured by these benchmarks after domain-specific training.

As the discussion period draws to a close, we would appreciate your assessment of whether these clarifications and results address your concerns. Please let us know if any issues remain that we can clarify.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.