- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Continual-Skill, a method that incorporates skills into agentic GRPO training to improve agents’ ability to invoke skills. It makes skills available during LLM rollouts according to prioritized exposure probabilities and uses these probabilities to weight group returns for GRPO updates. Experiments demonstrate its advantages in settings with perturbed skill banks.
Critical strengths
- The paper addresses an important problem, and the proposed approach is relatively simple and reasonable.
- The paper is clearly written and easy to follow.
Critical weaknesses
- Reliance on hand-designed heuristics. The method relies heavily on manual design choices, particularly in its central exposure probability mechanism. It also introduces numerous hyperparameters, making their selection potentially challenging. Clearer guidance on how to choose these hyperparameters would improve the method’s practical usability.
- Unclear source and practical relevance of the gains. The main advantages appear in the larger, perturbed skill banks, while the method shows no clear improvement over the baselines with a clean skill bank. This raises questions about its intended application scenarios and the reasons for its stronger performance in the Medium and Large settings. Are the gains attributable to the larger number of skills, the presence of perturbations, or an interaction between the two? My interpretation is that the improvements may largely stem from the manually designed exposure probabilities suppressing interference from perturbed skills. However, such settings may be less common in practice, where smaller, cleaner skill banks resembling the Small setting may be more representative. Further analysis is needed to clarify when and why the method is beneficial.
- Insufficient hyperparameter sensitivity analysis. How sensitive is the method to its many hyperparameters, and how do their values affect performance? A more systematic analysis would help establish the robustness of the reported results.
- Potential confounding effect of the number of injected skills. The proposed method and the baselines inject different numbers of skills. Could this difference substantially affect performance? Comparisons that control for the number of injected skills would help isolate the contribution of the proposed method.
- Unclear contribution of exposure-based weighting. What effect do the weights based on exposure probabilities in Equation (6) have on performance? An ablation of this weighting mechanism would help establish its contribution.
- Limited scope of the ablation studies. The effects of individual components may vary substantially across skill banks. Could the authors conduct ablation studies across different bank settings, rather than restricting them to the Large bank in a single environment?
Questions for authors
See Weaknesses.
3 comments
W2 & W4: Sources of improvement and the number of provided skills
The improvements in expanded skill banks directly demonstrate the value of our method under skill accumulation. The paper retains the original skills, adds redundant, overly specific, outdated, or conflicting content, and initializes all methods with the same skill bank. On WebShop, our method exceeds the strongest comparison baseline reported in the original paper by 31.40 and 40.40 percentage points in success rate under the Medium and Large conditions, respectively. These are direct comparisons under the same initial skill-bank conditions.
Existing methods perform well in the Small setting, showing that they can effectively use relatively clean initial skill banks. As the banks expand, their performance diverges substantially, indicating that skill accumulation changes the relative performance of these methods. Together, these results clearly identify the conditions in which our method offers its main advantages.
Synthetic expansion enables reproducible evaluation of problems arising from skill accumulation. Expansion simultaneously increases the number of skills and changes their content composition, so the results evaluate the combined effects of these two changes. The perturbation types listed in the paper describe the content being tested; their prevalence in real skill banks would need to be measured using data from actual accumulation.
The number of skills provided by each method is reported in Appendix Table 14. The main experiments retain each method's own exposure strategy and compare complete systems. Matching the number of provided skills would enable a further comparison of skill selection under the same budget. The conclusion supported by the current main results is that, given the same initial skill bank, our complete method achieves a clear advantage under expanded-bank conditions. This advantage reflects the combined effects of skill selection, the number of provided skills, and the management mechanism.
W5 & W6: Probability weighting and component ablations
Equation (6) corrects for bias arising from differences in exposure probability. A skill that is provided more frequently receives more feedback, so directly accumulating this feedback makes the estimate depend on exposure frequency. Equation (6) weights observations by their actual exposure probabilities so that the estimate corresponds to the difference in mean feedback between providing and withholding the skill.
This follows directly from the formula. Let the mean feedback in the two groups be and , and let the exposure probability be . Under the assumptions stated in the appendix, the weighted expectation is ; after removing the weights, the expectation becomes . The latter varies with exposure probability. The paper therefore explains the role of weighting in utility estimation. Quantifying its contribution to final success rate would require a control experiment that removes these weights.
The component experiments provide two concrete pieces of evidence. In Table 3, the full configuration outperforms the configuration with management disabled across all three seeds, improving mean success rate by 2.49 percentage points. In Table 4, the configuration combining semantic information and utility achieves the highest mean success rate. These results support the roles of the corresponding mechanisms in the ALFWorld Large setting.
The evaluation of the complete method covers two environments and three skill-bank sizes, while the decomposition of component contributions focuses on ALFWorld Large. Accordingly, we claim that the complete method is competitive across the tested settings and use ablations to explain the sources of performance in one of those settings. Component experiments across additional settings could further examine how individual contributions vary; the current conclusions are supported by the two bodies of evidence described above.
Thank you for the rebuttal. I recognize that time for additional experiments is limited during the rebuttal period, and I find the authors’ explanations for W1 and W3 satisfactory, as these concerns are less central to my assessment. For W2 and W4, however, experimental evidence is essential to understanding why the method works, under what conditions it succeeds, and when it fails. The results provided so far are therefore insufficient for me to raise my score.
We sincerely appreciate the thorough and constructive feedback provided on our manuscript. In response to the reviewers’ comments, we have carefully addressed each point below.
We believe the current results support the use of this parameter configuration across environments. The paper uses the same default routing and management parameters on ALFWorld and WebShop and reports results at three skill-bank sizes. These experiments directly test whether the parameters transfer across settings.
Exposure probabilities also follow an explicit rule. The system first specifies the budget for providing skills and then computes probabilities based on the number of candidates. For example, with 10 general-skill candidates and a budget of 2, each candidate has an exposure probability of 0.2; with 10 task-specific skill candidates and a budget of 6, the probability is 0.6. In practice, users choose the exposure budget, and the system computes the corresponding probabilities.
Skill-management parameters specify the evidence required to take action. The default configuration requires at least 40 observations in total, with at least 12 observations in both the exposed and unexposed groups; archival due to negative utility additionally requires two consecutive confirmations. These rules link every state transition to specific observations and decision criteria.
The paper therefore provides both the rationale for parameter selection and results obtained using the same parameter configuration across settings. The sensitivity experiments suggested by the reviewer could further quantify how parameter changes affect performance, while the current results already support the effectiveness of a shared default configuration in the evaluated settings.