acceptodds
Under review as a conference paper at ICLR 2027

CASU: Case-to-Class Skill Synthesis with Adaptive Skill Utilization for LLM Agents

Abstract

Large Language Model (LLM) agents increasingly solve complex tasks through interaction, producing trajectories that could benefit future related tasks by preserving successful strategies and failure lessons. Yet this experience often remains underused: raw trajectories are too noisy and instance-specific to become transferable skills, and even when such skills are extracted, it remains unclear how to use them effectively, since some skills are better internalized into model parameters while others are better retained as external guidance. To address these challenges, we introduce CASU, a three-stage framework for hierarchical skill synthesis and adaptive skill utilization that uses only the agent's own experience, without stronger external supervision. Specifically, CASU distills multiple successful and failed trajectories of each task instance into Case Skills, which capture reusable strategies while filtering instance-specific details. It then synthesizes Class Skills from Case Skills collected across tasks in the same category, and optimizes the Class Skill generator with paired environment feedback on held-out tasks. It further proposes Skill Absorption Efficiency (SAE) to estimate the internalization suitability of each learned skill by measuring the consistency of its induced parameter updates across disjoint task subsets. High-SAE skills are internalized through distillation on skill-guided responses, while remaining useful skills are retained for external invocation. Across ALFWorld and WebShop, CASU improves GRPO-trained Qwen2.5-3B and Qwen2.5-7B agents, delivering up to 10.95 points of improvement in ALFWorld average success and 7.0 points in WebShop accuracy, surpassing strong RL, skill-augmented RL, and self-distillation baselines.

Primary area:
reinforcement learning
Keywords:
agent skills, reinforcemance learning, self distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V7MAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how agents can learn skills from their own task experience and what skills should be retained in the model parameters and what skills should be kept external for retrieval. The CASU first summarizes the Case Skills from multiple successful and failed attempts at the same task, and then consolidates experience from similar tasks into Class Skills. The skill generator is trained from environment feedback. The authors then propose SAE, which selects skills for distillation according to the consistency of their gradients across tasks and the interference that these updates would have on existing behaviors. The remaining skills are kept external as guidance. Experiments on ALFWorld and WebShop show that CASU further improves upon the performance of GRPO-trained 3B and 7B models.

Critical strengths
  1. I find the paper's central question compelling: a skill that proves helpful when provided as a prompt might not be valuable when trained into the model. Instead of distilling all accumulated skills into the model, the authors seek to find how different skills should be used. Using gradient consistency across tasks to accomplish this task is what I find most compelling.

  2. The pipeline builds upon the model's own trial and error and environment feedback without the aid of an outside teacher, which has practical value. The authors also compare random selection, low-SAE and high-SAE selection, internalizing all skills, and keeping all skills external, rather than just reporting the performance of the final system. These experiments lend credence to the thesis of selectively learning skills.

Critical weaknesses
  1. I find it challenging to separate the contributions of the design choices and additional training in CASU from the baseline's differences in the experimental setup. The ALFWorld results are taken from the SDAR baseline, whereas CASU's starting point is the authors' GRPO checkpoint, and they use their evaluation pipeline. The gap between GRPO and GRPO+CASU in the provided table, therefore, potentially conflates the effect of updating the model with the design choices made in CASU. The latter introduces skill-generator training and distillation on top of GRPO, whereas the changes to the baseline include replacing keyword matching with embedding-based retrieval and model reranking. I would like to see, at the very least, the evaluation of the starting checkpoint in the same pipeline, along with controls that better approximate the additional training budget (e.g., number of training iterations) and retrieval changes. Your splits for the tasks used to select the skill bank are unclear to me as well: the Appendix only mentions the tasks used to select the skill set but does not specify whether any of them were used in the final evaluation. I do not think you have ruled out the possibility of leakage, which would explain the results if it were the case, but this detail should be clarified before taking these results as evidence.

  2. SAE is the most interesting contribution, but I do not see strong evidence that such a sophisticated metric was needed in your particular setup. On the ALFWorld-7B benchmark, the higher selection score only improves performance by approximately 1.7% over randomly chosen skills. You do not report performance distribution over different training runs, and it is unclear how SAE compares to an alternative approach of, say, selecting skills according to their observed performance gain. On the other hand, your control experiments convincingly demonstrate how skill selection affects downstream performance, and I would like to see more comparisons to simple heuristics, such as selecting skills that are most beneficial for the tasks that the model failed. How much do each of the two factors (Fisher weighting and loss normalization) affect the performance? The theoretical justification for the local update is compelling, but it remains unclear how it shapes the global optimization in SDL with multiple skills and update steps. Most importantly, the only demonstration of internalization in Fig 5 actually compares the behavior of a model with and without a skill prompt before distillation: it only shows that adding a prompt helps, but not that the skill was internalized and can successfully be removed later.

Questions for authors

Could the authors explicitly report the source, reward-validation, bank-selection, and final evaluation task splits and clarify whether there is any overlap between them? In particular, are the 140 ALFWorld tasks and 500 WebShop tasks used for selecting the best bank in Appendix C.3 the same tasks that are used for final evaluation?

Reviewer_2uDqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Experience reuse is a major challenge for recent LLM based agents. Raw trajectories are noisy and not transferable to many tasks, and extracted skills need to be carefully applied to relevant tasks. Further, there is a choice whether the skill is stored in external memory or should the model parameters be updated. This paper introduces CASU, a framework that enables LLM agents to synthesize and selectively utilize reusable skills from their own interaction trajectories without a stronger external teacher. Key highlights are: (1) the agent summarizes multiple attempts at solving tasks into concise Case Skills, and then uses GRPO to train a generator that merges similar Case Skills into broader Class Skills, rewarding candidates based on net task repairs minus regressions on held-out validation tasks. (2) CASU computes Skill Absorption Efficiency (SAE) to internalize consistently learnable, low-interference skills into model parameters via offline self-distillation, while retaining intermediate-scoring skills in an external retrieval bank and discarding the rest. (3) Across ALFWorld and WebShop using Qwen2.5-3B and 7B models, GRPO+CASU improves over standard GRPO baselines by up to 10.95 points on ALFWorld and 7.0 points on WebShop.

Critical strengths
  • Paper focuses on a key question in agent skill synthesis pipelines “which self-synthesized skills should be internalized into weights versus kept in context?”. Prior works e.g., SKILL0, Skill-SD, Skill0.5, SLIM typically internalize everything or filter only by skill type or external utility. This internalization decision is made systematically via cross-task gradient agreement.

  • Proposed method uses the exact same 3B/7B checkpoint LMM as extractor, generator, teacher, and student without using any stronger external teacher. Related work is current and clearly positions CASU relative to SDAR, Skill-SD, Trace2Skill, and SkillRL.

Critical weaknesses
  • From my understanding there is a mismatch in the evaluation protocol. All ALFWorld baselines (including GRPO) are copied from SDAR paper which uses a different 128-episode protocol, whereas CASU is evaluated on 420 episodes (140 valid-seen tasks * 3 attempts each). Authors' own 150-step GRPO starting checkpoint under that same 420-episode setup is not reported and Figure 4a suggests it is between 82–88%, making the real gain from CASU smaller than +10.95). Both the Stage 2 bank checkpoints and Stage 3 hyperparameters appear to be selected on splits matching the exact dimensions of the evaluation sets.

  • Presented experimental results indicate that Stage 3 only improves over the Stage 2 external skill bank on ALFWorld-7B (raising success from 87.38% to 90.48%, the sole setting where ablations are reported). On the 3B models, the final post-Stage-3 GRPO+CASU performance in Table 1 is actually slightly lower than using the selected Stage 2 external bank directly with the GRPO executor on both ALFWorld-3B (85.95% final vs. 88.33% Stage 2 bank) and WebShop-3B (72.4% final vs. 73.4% Stage 2 bank).

Questions for authors
  1. Can you report the performance numbers for 150-step GRPO starting checkpoint before applying the CASU method?

  2. Can you add the definition of the bank selection split D_sel in the main paper of in Appendix C.1?

  3. Clarify how Stage 2 bank checkpoints and Stage 3 hyperparameters are selected and how the proposed approach avoids overfitting on evaluation sets.

  4. Can you add discussion about why stage3 (Skill Internalization and Externalization Allocation) improved performance beyond stage2 (Class Skill Synthesis) only for 7B models and not for 3B models?

My current rating assumes that the authors will satisfactorily address the questions above during the rebuttal period.

Reviewer_vUACOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CASU, which extracts Case Skills from the agent's own success and failure trajectories and then synthesizes experience from tasks of the same type into Class Skills via GRPO; the generator's reward is the difference in success rate of a frozen executor on independent tasks with and without the skill. The method then uses cross-task gradients and anchor Fisher to construct Skill Absorption Efficiency (SAE), selects a subset of skills for offline distillation, and calibrates the retained external skills after distillation. The paper reports performance gains on ALFWorld and WebShop, and examines the role of each component by removing the synthesis stage, varying the internalized skill selection, and changing the anchor ratio.

Critical strengths
  • Strength 1: Skill synthesis is directly supervised by execution outcomes. The Case-to-Class design first compresses multiple attempts on a single task and then distills strategies across tasks; paired evaluation on the frozen executor counts both repairs and regressions, enabling the generator to learn from actual behavioral benefit rather than merely producing plausible-looking summaries of experience.

  • Strength 2: Separating internalization selection from external-use benefit is a valuable design direction. SAE introduces cross-task update compatibility and sensitivity of existing behavior, providing a skill-selection signal distinct from success rate. High-SAE internalization outperforms random and low-SAE selection at the same number of skills per category giving direct empirical support for this selection strategy.

Critical weaknesses
  • Weakness 1: The qualified skill set EκE_\kappa already requires Urv(s)>0U_{\mathrm{rv}}(s)>0 and Usrc(s)>0U_{\mathrm{src}}(s)>0, yet the method further specifies $ C_{\mathrm{ext}} =\bigcup_\kappa {s\in E_\kappa\setminus I_\kappa:\ 0.3\leq\bar r_s\leq0.5}. $ Consequently, skills with rˉs<0.3\bar r_s<0.3 are deleted outright and never reach the post-distillation. A low SAE ranking, however, does not imply $ N_{\mathrm{repair}}(s\mid\theta^\star) -N_{\mathrm{regression}}(s\mid\theta^\star)<\nu. $ That is, unsuitable for internalization does not entail no longer sufficiently beneficial as a prompt.

  • Weakness 2: Inconsistency between the SAE metric and the actual distillation process. First, anchor Fisher averages the token gradients within a response before squaring, which can underestimate the parameter directions that are sensitive to token behavior owing to gradient cancellation. Second, SAE's local analysis relies on an update direction adjusted by the inverse Fisher, yet the actual SDL training does not state that such an adjustment is used; under the two updates, cross-task effects may even have opposite signs. Existing ablations support the effectiveness of the SAE selection strategy, but the inconsistencies above weaken its basis as a metric of internalization compatibility and risk of interference with existing capabilities.

Questions for authors
  • Quesion 1: How do the GRPO checkpoints that the authors trained themselves and used to initialize CASU perform on the same ALFWorld tasks and under the same execution settings as CASU? Please add this result so that the increment relative to its own initialization model is made explicit.

  • Quesion 2: Could a control be added that selects internalization skills ranked by the existing Usrc(s)U_{\mathrm{src}}(s)? Such a control could reuse the existing skill library and trajectories, hold initialization, the qualified candidate pool, the number of skills selected per category, and the distillation settings fixed, and evaluate without providing external skills, changing only the selection criterion. This would directly test whether SAE provides more useful information than the known skill benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.