- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how agents can learn skills from their own task experience and what skills should be retained in the model parameters and what skills should be kept external for retrieval. The CASU first summarizes the Case Skills from multiple successful and failed attempts at the same task, and then consolidates experience from similar tasks into Class Skills. The skill generator is trained from environment feedback. The authors then propose SAE, which selects skills for distillation according to the consistency of their gradients across tasks and the interference that these updates would have on existing behaviors. The remaining skills are kept external as guidance. Experiments on ALFWorld and WebShop show that CASU further improves upon the performance of GRPO-trained 3B and 7B models.
Critical strengths
-
I find the paper's central question compelling: a skill that proves helpful when provided as a prompt might not be valuable when trained into the model. Instead of distilling all accumulated skills into the model, the authors seek to find how different skills should be used. Using gradient consistency across tasks to accomplish this task is what I find most compelling.
-
The pipeline builds upon the model's own trial and error and environment feedback without the aid of an outside teacher, which has practical value. The authors also compare random selection, low-SAE and high-SAE selection, internalizing all skills, and keeping all skills external, rather than just reporting the performance of the final system. These experiments lend credence to the thesis of selectively learning skills.
Critical weaknesses
-
I find it challenging to separate the contributions of the design choices and additional training in CASU from the baseline's differences in the experimental setup. The ALFWorld results are taken from the SDAR baseline, whereas CASU's starting point is the authors' GRPO checkpoint, and they use their evaluation pipeline. The gap between GRPO and GRPO+CASU in the provided table, therefore, potentially conflates the effect of updating the model with the design choices made in CASU. The latter introduces skill-generator training and distillation on top of GRPO, whereas the changes to the baseline include replacing keyword matching with embedding-based retrieval and model reranking. I would like to see, at the very least, the evaluation of the starting checkpoint in the same pipeline, along with controls that better approximate the additional training budget (e.g., number of training iterations) and retrieval changes. Your splits for the tasks used to select the skill bank are unclear to me as well: the Appendix only mentions the tasks used to select the skill set but does not specify whether any of them were used in the final evaluation. I do not think you have ruled out the possibility of leakage, which would explain the results if it were the case, but this detail should be clarified before taking these results as evidence.
-
SAE is the most interesting contribution, but I do not see strong evidence that such a sophisticated metric was needed in your particular setup. On the ALFWorld-7B benchmark, the higher selection score only improves performance by approximately 1.7% over randomly chosen skills. You do not report performance distribution over different training runs, and it is unclear how SAE compares to an alternative approach of, say, selecting skills according to their observed performance gain. On the other hand, your control experiments convincingly demonstrate how skill selection affects downstream performance, and I would like to see more comparisons to simple heuristics, such as selecting skills that are most beneficial for the tasks that the model failed. How much do each of the two factors (Fisher weighting and loss normalization) affect the performance? The theoretical justification for the local update is compelling, but it remains unclear how it shapes the global optimization in SDL with multiple skills and update steps. Most importantly, the only demonstration of internalization in Fig 5 actually compares the behavior of a model with and without a skill prompt before distillation: it only shows that adding a prompt helps, but not that the skill was internalized and can successfully be removed later.
Questions for authors
Could the authors explicitly report the source, reward-validation, bank-selection, and final evaluation task splits and clarify whether there is any overlap between them? In particular, are the 140 ALFWorld tasks and 500 WebShop tasks used for selecting the best bank in Appendix C.3 the same tasks that are used for final evaluation?