- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper points out that traditional skill evaluations primarily focus on measuring performance improvements on specific but fail to capture the impact of adding new skills to a generalist agent. The authors propose a "2×2" evaluation framework that compares the agent before and after the addition of a skill. The authors also introduce the capability correctness (CC) to assess atomic capability execution, accounting for both precise tool invocations and functionally equivalent alternatives.
Experiments on two generalist agents (CUGA and Pi), encompassing seven domains and 320 test cases, show that adding skills can enhance performance on new tasks but may also lead to regression on previously ones. This phenomenon reveals redundancy and cross-skill interference hidden by aggregate metrics. A capability-oriented pruning case study further demonstrate the diagnostic value of the CC, which acieves 94.1% agreement with human annotations.
Critical strengths
1. A practical framework for identifying latent regressions.
This 2×2 evaluation framework considers both newly acquired and pre-existing capabilities, revealing cross-skill interference that aggregate scores may overlook. This addresses an important limitation in evaluating generalist agents with expanding skill libraries.
2. Fine-grained evaluation with diagnostic value.
The Capability Correctness (CC) metric enables analysis of redundancy and interference at the capability level, achieving 94.1% consistency with human annotations. A case study on model pruning further demonstrates its potential value for skill refinement, although the improvements are not statistically significant.
Critical weaknesses
1. Limited statistical evidence regarding cross-skill interference.
Although the paper demonstrates performance regression after adding new skills, the magnitude of the decline in Capability Correctness (CC) are generally minor, and the confidence intervals overlap. This weakens the evidence that skill interference is a systematic and significant issue rather than merely a result of experimental fluctuation.
2. Insufficient validation of the proposed metric's practical utility.
The CC metric relies on LLM-based assessments, while human validation covers only 118 capabilities. Furthermore, experiments involving capability-guided pruning fail to demonstrate statistically significant improvements, and the pruning process also alters skill descriptions. This makes it difficult to isolate the contribution of CC and establish its practical effectiveness.
Questions for authors
1. Statistical significance of skill interference.
In CC scenarios involving previously supported tasks, the observed performance drops are mostly relatively minor, and the confidence intervals overlap. Could the authors provide results from paired statistical tests or confidence intervals regarding these performance differences? How many of these performance drops are statistically significant? This would help determine whether cross-skill interference is systematic phenomenon rather than merely the result of experimental fluctuations.
2. Effectiveness of CC-guided pruning.
The pruning experiments show no statistically significant improvements, and the CC-guided process also modifies skill descriptions. Could the authors clarify whether the random-pruning baseline includes comparable level of description modifications? Furthermore, could the authors isolate the effects of capability pruning from description refinement to demonstrate that CC offers substantial benefits over manual skill editing?
4 comments
Response to Reviewer 9K8M (Part 2)
Ques 1: Statistical significance of skill interference. In CC scenarios involving previously supported tasks, the observed performance drops are mostly relatively minor, and the confidence intervals overlap. Could the authors provide results from paired statistical tests or confidence intervals regarding these performance differences? How many of these performance drops are statistically significant? This would help determine whether cross-skill interference is systematic phenomenon rather than merely the result of experimental fluctuations.
Response: Thank you for the suggestion. In response, we add a new paired difference bootstrap analysis of the performance differences between the old and new systems, per-sample. We report 95% paired bootstrap confidence intervals for per-task differences between the new and old systems, with significance defined by exclusion of zero. For , CC changes are modest, with all paired confidence intervals including zero; however, we observe significant declines in several skill precision and recall comparisons, and one mean progress comparison reveal changes in routing and execution. Conversely, a majority of the CC comparisons improve significantly.
Taken together, the results show significant CC improvements on most comparisons and only modest CC changes on . However, other metrics reveal significant changes in skill selection and an isolated regression in task progress that would otherwise remain undetected. We include the above discussion and the paired differences tables in the revised paper in a new Appendix Section A.17.
Table: Paired bootstrap significance of (new − old) on general+single-domain tasks for CUGA and PI Claude Opus 4.6. Each cell reports the paired mean difference with its 95% bootstrap CI; * marks a CI that excludes zero (statistical significance). Skill precision/recall are reported on T_old only. † the CI upper bound equals 0.000 at the displayed precision but excludes zero.
| Agent | Set. | Capability correctness (T_old) | Capability correctness (T_new) | Mean progress rate (T_old) | Mean progress rate (T_new) | Skill prec. (T_old) | Skill rec. (T_old) |
|---|---|---|---|---|---|---|---|
| CUGA | S1 | −0.003 [−0.036, +0.031] | +0.090 [−0.074, +0.245] | +0.001 [−0.032, +0.034] | +0.114 [−0.023, +0.245] | −0.045* [−0.094, −0.002] | −0.045* [−0.090, −0.008] |
| CUGA | S2 | −0.030 [−0.070, +0.000] | +0.399* [+0.229, +0.597] | −0.035*† [−0.080, +0.000] | +0.064 [−0.075, +0.255] | −0.200* [−0.360, −0.080] | −0.200* [−0.360, −0.080] |
| CUGA | S3 | −0.002 [−0.046, +0.041] | +0.382* [+0.285, +0.483] | +0.006 [−0.029, +0.041] | +0.326* [+0.234, +0.421] | −0.003 [−0.023, +0.016] | +0.000 [−0.019, +0.019] |
| PI | S1 | −0.022 [−0.070, +0.022] | +0.104* [+0.042, +0.174] | −0.004 [−0.036, +0.027] | +0.091* [+0.025, +0.165] | −0.029 [−0.076, +0.018] | −0.024 [−0.073, +0.020] |
| PI | S2 | +0.040 [+0.000, +0.080] | +0.254* [+0.162, +0.353] | −0.005 [−0.020, +0.010] | +0.075 [−0.010, +0.170] | +0.000 [+0.000, +0.000] | +0.000 [+0.000, +0.000] |
| PI | S3 | −0.008 [−0.061, +0.047] | +0.120* [+0.037, +0.215] | −0.039 [−0.092, +0.011] | +0.099* [+0.024, +0.185] | −0.111* [−0.210, −0.020] | −0.097*† [−0.200, +0.000] |
Table: Paired bootstrap significance of (new − old) on cross-domain tasks for CUGA and PI Claude Opus 4.6. Each cell reports the paired mean difference over its 95% bootstrap CI; * marks a CI that excludes zero, which represents statistical significance. "–" denotes cells with no paired T_old cross-domain tasks. Skill precision/recall are reported on T_old only.
| Agent | Set. | Capability correctness (T_old) | Capability correctness (T_new) | Mean progress rate (T_old) | Mean progress rate (T_new) | Skill prec. (T_old) | Skill rec. (T_old) |
|---|---|---|---|---|---|---|---|
| CUGA | S1 | −0.008 [−0.076, +0.061] | −0.015 [−0.126, +0.102] | −0.044 [−0.129, +0.026] | +0.022 [−0.095, +0.135] | −0.320* [−0.490, −0.159] | −0.376* [−0.565, −0.200] |
| CUGA | S2 | – | +0.153* [+0.031, +0.288] | – | +0.031 [−0.025, +0.094] | – | – |
| CUGA | S3 | −0.008 [−0.048, +0.034] | +0.227* [+0.162, +0.295] | −0.009 [−0.041, +0.026] | +0.137* [+0.090, +0.184] | −0.038 [−0.086, +0.005] | −0.060* [−0.117, −0.012] |
| PI | S1 | −0.017 [−0.104, +0.071] | +0.008 [−0.046, +0.069] | −0.009 [−0.085, +0.062] | +0.048 [−0.028, +0.141] | −0.141* [−0.247, −0.037] | −0.018 [−0.106, +0.059] |
| PI | S2 | – | −0.001 [−0.077, +0.058] | – | +0.027 [−0.062, +0.098] | – | – |
| PI | S3 | +0.002 [−0.039, +0.043] | +0.131* [+0.089, +0.173] | +0.014 [−0.023, +0.052] | +0.089* [+0.054, +0.126] | −0.074 [−0.166, +0.017] | −0.074 [−0.164, +0.012] |
Response to Reviewer 9K8M (Part 3)
Ques 2: Effectiveness of CC-guided pruning. The pruning experiments show no statistically significant improvements, and the CC-guided process also modifies skill descriptions. Could the authors clarify whether the random-pruning baseline includes comparable level of description modifications? Furthermore, could the authors isolate the effects of capability pruning from description refinement to demonstrate that CC offers substantial benefits over manual skill editing?
Response: This concern overlaps with Reviewer GqYJ, Q2. We repeat the response below to keep each reviewer response self-contained.
We revised the CC-guided pruning experiment to remove only the six identified redundant tools, while leaving the skill description and all remaining unpruned tool schemas and documentation unchanged. This isolates the effects of tool removal from those of description clarification or documentation shortening. Note that the random pruned baseline used in the original paper already apply the correct procedure and thus remained unchanged. To further assess performance variation across the random pruned setting, we also expand the random baseline to four independently sampled pruning sets (A to D), where each has six randomly selected tools that are pruned. Random D is the result reported in the original paper, while Random A to C are newly added. All random pruning runs follow the same modification procedure as CC-guided pruning and differ only in which six tools are removed.
As shown in the Table below, random pruning is highly dependent on the sampled set: on general and single-domain new tasks, CC ranges from to and mean progress from to , yielding and , respectively. In comparison, CC-guided pruning achieves CC on general and single domain tasks---the highest among all configurations evaluated---and mean progress, exceeding the old system, unpruned system, and random average, although not the best individual random draw. This controlled comparison provides evidence that CC identifies a useful pruning signal beyond arbitrary tool removal.
The advantage is task-dependent rather than universal. On general and single-domain old tasks, CC-guided pruning reduces CC from (unpruned) to , revealing that removing capability tools based on redundancy for new-skill capabilities alone does not also guarantee retention of capabilities on previously supported tasks. This result reinforces the need for our old/new task decomposition as what appears beneficial on new tasks can still impose a measurable cost on previously supported tasks. Besides that, cross-domain tasks are more difficult because they require a combination of capabilities from the pruned skill and from the other skills. CC-guided pruning improves new-task CC from (unpruned) to , but does not consistently outperform random pruning on the remaining cross-domain metrics.
These results provide preliminary evidence that CC can identify pruning candidates for general and single-domain tasks, while cross-domain tasks require more sophisticated, interaction-aware optimization beyond the scope of this paper.
Table: Performance after skill pruning in Setting 1. G+S denotes general + single-domain. Sample sizes: G+S, , ; cross-domain, , . Random averages are mean ± standard deviation over runs A–D. Bold marks where CC-guided pruning beats the random average.
General + single-domain
| System | CC T_old | CC T_new | Prog. T_old | Prog. T_new |
|---|---|---|---|---|
| Old system | 0.789 | 0.735 | 0.603 | 0.577 |
| Unpruned | 0.786 | 0.825 | 0.604 | 0.691 |
| Random A | 0.792 | 0.817 | 0.640 | 0.682 |
| Random B | 0.791 | 0.776 | 0.647 | 0.546 |
| Random C | 0.807 | 0.831 | 0.647 | 0.748 |
| Random D | 0.771 | 0.736 | 0.617 | 0.623 |
| Random avg | 0.790 ± 0.015 | 0.790 ± 0.043 | 0.638 ± 0.014 | 0.650 ± 0.086 |
| CC-guided | 0.758 | 0.838 | 0.631 | 0.700 |
Cross-domain
| System | CC T_old | CC T_new | Prog. T_old | Prog. T_new |
|---|---|---|---|---|
| Old system | 0.845 | 0.805 | 0.585 | 0.687 |
| Unpruned | 0.837 | 0.790 | 0.541 | 0.709 |
| Random A | 0.827 | 0.901 | 0.579 | 0.768 |
| Random B | 0.899 | 0.877 | 0.594 | 0.679 |
| Random C | 0.870 | 0.870 | 0.609 | 0.709 |
| Random D | 0.838 | 0.860 | 0.535 | 0.725 |
| Random avg | 0.859 ± 0.033 | 0.877 ± 0.017 | 0.579 ± 0.032 | 0.720 ± 0.037 |
| CC-guided | 0.819 | 0.844 | 0.524 | 0.682 |
We update Table 5 with the new results, revise Section 5.3 to incorporate the above discussion, and report the complete results with confidence intervals in Appendix A.3. Due to the extent of these changes, we refer readers to the revised paper for the complete section.
Follow-up on Our Rebuttal – Reviewer 9K8M
Dear Reviewer 9K8M,
Thank you for your constructive feedback. As the rebuttal period comes to an end, please let us know if you have any remaining questions or concerns. If our response has addressed them, we would appreciate it if you would reconsider your rating in light of our clarifications.
Response to Reviewer 9K8M (Part 1)
We thank the reviewer for the constructive comments and address the concerns raised below. Should any questions remain, we are happy to provide additional details. We hope our responses clarify the reviewer’s concerns and respectfully request reconsideration of the ratings in light of the clarifications provided. The revised paper is uploaded where the revision are in purple. The code and logs for the rebuttal is under the code branch
rebuttal.Weakenesses 1: Limited statistical evidence regarding cross-skill interference.Although the paper demonstrates performance regression after adding new skills, the magnitude of the decline in Capability Correctness (CC) are generally minor, and the confidence intervals overlap. This weakens the evidence that skill interference is a systematic and significant issue rather than merely a result of experimental fluctuation.
Response: We agree that small CC declines should not be interpreted as systematic regression without quantifying their uncertainty. However, overlap between the systems’ marginal confidence intervals does not determine whether their paired difference is statistically significant. As also requested in Question 1, we therefore report paired confidence intervals for the per-task performance differences and identify which observed declines are statistically significant. Please see our detailed response to Question 1 below.
Weaknesses 2: Insufficient validation of the proposed metric's practical utility. The CC metric relies on LLM-based assessments, while human validation covers only 118 capabilities. Furthermore, experiments involving capability-guided pruning fail to demonstrate statistically significant improvements, and the pruning process also alters skill descriptions. This makes it difficult to isolate the contribution of CC and establish its practical effectiveness.
Response: We strengthened the metric validation by further expanding our human validation study from 118 to 148 capability judgments, including targeted analysis of functionally equivalent executions. We have revised the human validation discussion originally presented in Section 5.4 and Appendix A.14; these section numbering refer to the original submission and differ in the revised paper. We also provide the expanded human validation study results here below:
Table: Human validation of capability judgments. Entries report counts (percentages) over all 148 annotated capabilities.
Moreover, please also see our response to Reviewer GqYJ, Question 1, for a detailed analysis of CC’s added value and practical utility, including comparisons with existing subgoal evaluation and human validation of cases in which the CC and prior approach disagree.
In addition, we revised the pruning experiments to isolate capability removal from skill description refinement and documentation shortening. The revised CC-guided setup removes only the identified redundant tools, leaving the skill description and all remaining unpruned tool schemas and documentation unchanged. We also expanded the random-pruning baseline to four independently sampled sets, each removing the same number of tools under the same procedure. The results provide preliminary evidence that CC offers useful pruning signals for general and single-domain new tasks, although we do not claim consistent improvements across all task types. Please see our detailed response to Question 2 below.