acceptodds
Under review as a conference paper at ICLR 2027

Aggregate Scores Mask Generalist Agent’s Skill Usefulness

Abstract

Recent advances in AI agents are moving toward generalist agents that acquire new capabilities through modular skills—reusable capabilities, knowledge, and tools that let agents perform new tasks. This raises a key evaluation question: when an agent already has many skills, what does adding another skill actually change? Standard with- or without-skill benchmarks evaluate skills in isolation, measuring whether they help on target tasks while overlooking whether they duplicate existing skills or degrade capabilities the agent already has. We reframe skill evaluation as an incremental, system-level attribution problem: we compare the agent before and after a skill addition across a grid of system state and test case context, and introduce a capability correctness (CC) metric that scores the capability atomic operations inside a skill, crediting exact and functionally equivalent invocations. Applied to a generalist agent extended with seven open-source domains as skills (320 released test cases), we find that a single skill addition produces both gains on its target tasks and regressions on previously supported ones—including redundancy and cross-skill interference that aggregate benchmarks hide. Because CC localizes these effects to specific capabilities, it also yields an actionable diagnostic signal for skill refinement, which we probe with a simple targeted pruning method. Therefore, we treat aggregate scores as insufficient to attribute the true usefulness of a skill.

Primary area:
datasets and benchmarks
Keywords:
skill evaluation, generalist agent evaluation, skill-specific testcase generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9K8MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that traditional skill evaluations primarily focus on measuring performance improvements on specific but fail to capture the impact of adding new skills to a generalist agent. The authors propose a "2×2" evaluation framework that compares the agent before and after the addition of a skill. The authors also introduce the capability correctness (CC) to assess atomic capability execution, accounting for both precise tool invocations and functionally equivalent alternatives.

Experiments on two generalist agents (CUGA and Pi), encompassing seven domains and 320 test cases, show that adding skills can enhance performance on new tasks but may also lead to regression on previously ones. This phenomenon reveals redundancy and cross-skill interference hidden by aggregate metrics. A capability-oriented pruning case study further demonstrate the diagnostic value of the CC, which acieves 94.1% agreement with human annotations.

Critical strengths

1. A practical framework for identifying latent regressions.

This 2×2 evaluation framework considers both newly acquired and pre-existing capabilities, revealing cross-skill interference that aggregate scores may overlook. This addresses an important limitation in evaluating generalist agents with expanding skill libraries.

2. Fine-grained evaluation with diagnostic value.

The Capability Correctness (CC) metric enables analysis of redundancy and interference at the capability level, achieving 94.1% consistency with human annotations. A case study on model pruning further demonstrates its potential value for skill refinement, although the improvements are not statistically significant.

Critical weaknesses

1. Limited statistical evidence regarding cross-skill interference.

Although the paper demonstrates performance regression after adding new skills, the magnitude of the decline in Capability Correctness (CC) are generally minor, and the confidence intervals overlap. This weakens the evidence that skill interference is a systematic and significant issue rather than merely a result of experimental fluctuation.

2. Insufficient validation of the proposed metric's practical utility.

The CC metric relies on LLM-based assessments, while human validation covers only 118 capabilities. Furthermore, experiments involving capability-guided pruning fail to demonstrate statistically significant improvements, and the pruning process also alters skill descriptions. This makes it difficult to isolate the contribution of CC and establish its practical effectiveness.

Questions for authors

1. Statistical significance of skill interference.

In CC scenarios involving previously supported tasks, the observed performance drops are mostly relatively minor, and the confidence intervals overlap. Could the authors provide results from paired statistical tests or confidence intervals regarding these performance differences? How many of these performance drops are statistically significant? This would help determine whether cross-skill interference is systematic phenomenon rather than merely the result of experimental fluctuations.

2. Effectiveness of CC-guided pruning.

The pruning experiments show no statistically significant improvements, and the CC-guided process also modifies skill descriptions. Could the authors clarify whether the random-pruning baseline includes comparable level of description modifications? Furthermore, could the authors isolate the effects of capability pruning from description refinement to demonstrate that CC offers substantial benefits over manual skill editing?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9K8M (Part 1)

We thank the reviewer for the constructive comments and address the concerns raised below. Should any questions remain, we are happy to provide additional details. We hope our responses clarify the reviewer’s concerns and respectfully request reconsideration of the ratings in light of the clarifications provided. The revised paper is uploaded where the revision are in purple. The code and logs for the rebuttal is under the code branch rebuttal.


Weakenesses 1: Limited statistical evidence regarding cross-skill interference.Although the paper demonstrates performance regression after adding new skills, the magnitude of the decline in Capability Correctness (CC) are generally minor, and the confidence intervals overlap. This weakens the evidence that skill interference is a systematic and significant issue rather than merely a result of experimental fluctuation.

Response: We agree that small CC declines should not be interpreted as systematic regression without quantifying their uncertainty. However, overlap between the systems’ marginal confidence intervals does not determine whether their paired difference is statistically significant. As also requested in Question 1, we therefore report paired confidence intervals for the per-task performance differences and identify which observed declines are statistically significant. Please see our detailed response to Question 1 below.


Weaknesses 2: Insufficient validation of the proposed metric's practical utility. The CC metric relies on LLM-based assessments, while human validation covers only 118 capabilities. Furthermore, experiments involving capability-guided pruning fail to demonstrate statistically significant improvements, and the pruning process also alters skill descriptions. This makes it difficult to isolate the contribution of CC and establish its practical effectiveness.

Response: We strengthened the metric validation by further expanding our human validation study from 118 to 148 capability judgments, including targeted analysis of functionally equivalent executions. We have revised the human validation discussion originally presented in Section 5.4 and Appendix A.14; these section numbering refer to the original submission and differ in the revised paper. We also provide the expanded human validation study results here below:

Table: Human validation of capability judgments. Entries report counts (percentages) over all 148 annotated capabilities.

Total capabilitiesAgreeDisagreeAmbiguous
148141 (95.3%)6 (4.1%)1 (0.7%)

Moreover, please also see our response to Reviewer GqYJ, Question 1, for a detailed analysis of CC’s added value and practical utility, including comparisons with existing subgoal evaluation and human validation of cases in which the CC and prior approach disagree.

In addition, we revised the pruning experiments to isolate capability removal from skill description refinement and documentation shortening. The revised CC-guided setup removes only the identified redundant tools, leaving the skill description and all remaining unpruned tool schemas and documentation unchanged. We also expanded the random-pruning baseline to four independently sampled sets, each removing the same number of tools under the same procedure. The results provide preliminary evidence that CC offers useful pruning signals for general and single-domain new tasks, although we do not claim consistent improvements across all task types. Please see our detailed response to Question 2 below.

AuthorsOpenReview ↗

Response to Reviewer 9K8M (Part 2)

Ques 1: Statistical significance of skill interference. In CC scenarios involving previously supported tasks, the observed performance drops are mostly relatively minor, and the confidence intervals overlap. Could the authors provide results from paired statistical tests or confidence intervals regarding these performance differences? How many of these performance drops are statistically significant? This would help determine whether cross-skill interference is systematic phenomenon rather than merely the result of experimental fluctuations.

Response: Thank you for the suggestion. In response, we add a new paired difference bootstrap analysis of the performance differences between the old and new systems, per-sample. We report 95% paired bootstrap confidence intervals for per-task differences between the new and old systems, with significance defined by exclusion of zero. For ToldT_{\mathrm{old}}, CC changes are modest, with all paired confidence intervals including zero; however, we observe significant declines in several skill precision and recall comparisons, and one mean progress comparison reveal changes in routing and execution. Conversely, a majority of the TnewT_{\mathrm{new}} CC comparisons improve significantly.

Taken together, the results show significant CC improvements on most TnewT_{\mathrm{new}} comparisons and only modest CC changes on ToldT_{\mathrm{old}}. However, other ToldT_{\mathrm{old}} metrics reveal significant changes in skill selection and an isolated regression in task progress that would otherwise remain undetected. We include the above discussion and the paired differences tables in the revised paper in a new Appendix Section A.17.

Table: Paired bootstrap significance of (new − old) on general+single-domain tasks for CUGA and PI Claude Opus 4.6. Each cell reports the paired mean difference with its 95% bootstrap CI; * marks a CI that excludes zero (statistical significance). Skill precision/recall are reported on T_old only. † the CI upper bound equals 0.000 at the displayed precision but excludes zero.

AgentSet.Capability correctness (T_old)Capability correctness (T_new)Mean progress rate (T_old)Mean progress rate (T_new)Skill prec. (T_old)Skill rec. (T_old)
CUGAS1−0.003 [−0.036, +0.031]+0.090 [−0.074, +0.245]+0.001 [−0.032, +0.034]+0.114 [−0.023, +0.245]−0.045* [−0.094, −0.002]−0.045* [−0.090, −0.008]
CUGAS2−0.030 [−0.070, +0.000]+0.399* [+0.229, +0.597]−0.035*† [−0.080, +0.000]+0.064 [−0.075, +0.255]−0.200* [−0.360, −0.080]−0.200* [−0.360, −0.080]
CUGAS3−0.002 [−0.046, +0.041]+0.382* [+0.285, +0.483]+0.006 [−0.029, +0.041]+0.326* [+0.234, +0.421]−0.003 [−0.023, +0.016]+0.000 [−0.019, +0.019]
PIS1−0.022 [−0.070, +0.022]+0.104* [+0.042, +0.174]−0.004 [−0.036, +0.027]+0.091* [+0.025, +0.165]−0.029 [−0.076, +0.018]−0.024 [−0.073, +0.020]
PIS2+0.040 [+0.000, +0.080]+0.254* [+0.162, +0.353]−0.005 [−0.020, +0.010]+0.075 [−0.010, +0.170]+0.000 [+0.000, +0.000]+0.000 [+0.000, +0.000]
PIS3−0.008 [−0.061, +0.047]+0.120* [+0.037, +0.215]−0.039 [−0.092, +0.011]+0.099* [+0.024, +0.185]−0.111* [−0.210, −0.020]−0.097*† [−0.200, +0.000]

Table: Paired bootstrap significance of (new − old) on cross-domain tasks for CUGA and PI Claude Opus 4.6. Each cell reports the paired mean difference over its 95% bootstrap CI; * marks a CI that excludes zero, which represents statistical significance. "–" denotes cells with no paired T_old cross-domain tasks. Skill precision/recall are reported on T_old only.

AgentSet.Capability correctness (T_old)Capability correctness (T_new)Mean progress rate (T_old)Mean progress rate (T_new)Skill prec. (T_old)Skill rec. (T_old)
CUGAS1−0.008 [−0.076, +0.061]−0.015 [−0.126, +0.102]−0.044 [−0.129, +0.026]+0.022 [−0.095, +0.135]−0.320* [−0.490, −0.159]−0.376* [−0.565, −0.200]
CUGAS2–+0.153* [+0.031, +0.288]–+0.031 [−0.025, +0.094]––
CUGAS3−0.008 [−0.048, +0.034]+0.227* [+0.162, +0.295]−0.009 [−0.041, +0.026]+0.137* [+0.090, +0.184]−0.038 [−0.086, +0.005]−0.060* [−0.117, −0.012]
PIS1−0.017 [−0.104, +0.071]+0.008 [−0.046, +0.069]−0.009 [−0.085, +0.062]+0.048 [−0.028, +0.141]−0.141* [−0.247, −0.037]−0.018 [−0.106, +0.059]
PIS2–−0.001 [−0.077, +0.058]–+0.027 [−0.062, +0.098]––
PIS3+0.002 [−0.039, +0.043]+0.131* [+0.089, +0.173]+0.014 [−0.023, +0.052]+0.089* [+0.054, +0.126]−0.074 [−0.166, +0.017]−0.074 [−0.164, +0.012]
AuthorsOpenReview ↗

Response to Reviewer 9K8M (Part 3)

Ques 2: Effectiveness of CC-guided pruning. The pruning experiments show no statistically significant improvements, and the CC-guided process also modifies skill descriptions. Could the authors clarify whether the random-pruning baseline includes comparable level of description modifications? Furthermore, could the authors isolate the effects of capability pruning from description refinement to demonstrate that CC offers substantial benefits over manual skill editing?

Response: This concern overlaps with Reviewer GqYJ, Q2. We repeat the response below to keep each reviewer response self-contained.

We revised the CC-guided pruning experiment to remove only the six identified redundant tools, while leaving the skill description and all remaining unpruned tool schemas and documentation unchanged. This isolates the effects of tool removal from those of description clarification or documentation shortening. Note that the random pruned baseline used in the original paper already apply the correct procedure and thus remained unchanged. To further assess performance variation across the random pruned setting, we also expand the random baseline to four independently sampled pruning sets (A to D), where each has six randomly selected tools that are pruned. Random D is the result reported in the original paper, while Random A to C are newly added. All random pruning runs follow the same modification procedure as CC-guided pruning and differ only in which six tools are removed.

As shown in the Table below, random pruning is highly dependent on the sampled set: on general and single-domain new tasks, CC ranges from 0.7360.736 to 0.8310.831 and mean progress from 0.5460.546 to 0.7480.748, yielding 0.790±0.0430.790\pm0.043 and 0.650±0.0860.650\pm0.086, respectively. In comparison, CC-guided pruning achieves 0.8380.838 CC on general and single domain tasks---the highest among all configurations evaluated---and 0.7000.700 mean progress, exceeding the old system, unpruned system, and random average, although not the best individual random draw. This controlled comparison provides evidence that CC identifies a useful pruning signal beyond arbitrary tool removal.

The advantage is task-dependent rather than universal. On general and single-domain old tasks, CC-guided pruning reduces CC from 0.7860.786 (unpruned) to 0.7580.758, revealing that removing capability tools based on redundancy for new-skill capabilities alone does not also guarantee retention of capabilities on previously supported tasks. This result reinforces the need for our old/new task decomposition as what appears beneficial on new tasks can still impose a measurable cost on previously supported tasks. Besides that, cross-domain tasks are more difficult because they require a combination of capabilities from the pruned skill and from the other skills. CC-guided pruning improves new-task CC from 0.7900.790 (unpruned) to 0.8440.844, but does not consistently outperform random pruning on the remaining cross-domain metrics.

These results provide preliminary evidence that CC can identify pruning candidates for general and single-domain tasks, while cross-domain tasks require more sophisticated, interaction-aware optimization beyond the scope of this paper.

Table: Performance after skill pruning in Setting 1. G+S denotes general + single-domain. Sample sizes: G+S, nold=49n_{old}=49, nnew=12n_{new}=12; cross-domain, nold=17n_{old}=17, nnew=13n_{new}=13. Random averages are mean ± standard deviation over runs A–D. Bold marks where CC-guided pruning beats the random average.

General + single-domain

SystemCC T_oldCC T_newProg. T_oldProg. T_new
Old system0.7890.7350.6030.577
Unpruned0.7860.8250.6040.691
Random A0.7920.8170.6400.682
Random B0.7910.7760.6470.546
Random C0.8070.8310.6470.748
Random D0.7710.7360.6170.623
Random avg0.790 ± 0.0150.790 ± 0.0430.638 ± 0.0140.650 ± 0.086
CC-guided0.7580.8380.6310.700

Cross-domain

SystemCC T_oldCC T_newProg. T_oldProg. T_new
Old system0.8450.8050.5850.687
Unpruned0.8370.7900.5410.709
Random A0.8270.9010.5790.768
Random B0.8990.8770.5940.679
Random C0.8700.8700.6090.709
Random D0.8380.8600.5350.725
Random avg0.859 ± 0.0330.877 ± 0.0170.579 ± 0.0320.720 ± 0.037
CC-guided0.8190.8440.5240.682

We update Table 5 with the new results, revise Section 5.3 to incorporate the above discussion, and report the complete results with confidence intervals in Appendix A.3. Due to the extent of these changes, we refer readers to the revised paper for the complete section.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal – Reviewer 9K8M

Dear Reviewer 9K8M,

Thank you for your constructive feedback. As the rebuttal period comes to an end, please let us know if you have any remaining questions or concerns. If our response has addressed them, we would appreciate it if you would reconsider your rating in light of our clarifications.

Reviewer_GqYJOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the incremental value of adding skills to a generalist agent with an existing skill library. It proposes a 2×2 evaluation over old and new agent configurations and old and new task sets, together with a capability correctness metric that credits exact and functionally equivalent executions.

Critical strengths

The paper addresses a practical evaluation spot: improvements on tasks targeted by an added skill should not substitute for checking retention on previously supported tasks. The explicit separation of these effects is useful for maintaining deployed agents.

Critical weaknesses
  1. The necessity and novelty of the proposed evaluation are insufficiently established. The 2×2 protocol largely applies standard before or after retention evaluation, while fine-grained assessment is already supported by some existing works (e.g., ToolSandbox[1] and TED[2]). CC introduces skill-derived capability units and equivalent-execution judgments, but its added value over existing subgoal evaluation remains unclear. Human agreement and parallel metric reporting can not establish that CC provides better diagnosis or skill-selection decisions than success-rate or progress-rate differences on identical old or new task splits.
  2. The pruning study is confounded, and task grouping is inconsistent. S1 simultaneously removes interfaces, clarifies skill descriptions, and shortens documentation, preventing attribution of improvements specifically to CC-guided component selection. The lack of a clearly separated held-out evaluation further limits the refinement claim. Additionally, the released experimental code uses a task grouping inconsistent with the paper's taxonomy. The S3 grouping configuration includes 70 domain-plus-general tasks in the “general + single-domain” results, although Table 11 explicitly classifies these combinations as cross-domain. The authors should reconcile this discrepancy between the code and the paper, and verify whether the domain-specific results and conclusions remain valid under a consistent grouping.

[1] arxiv 2408.04682

[2] arxiv 2603.15483

Questions for authors
  1. What does CC contribute beyond existing subgoal or milestone evaluation? On identical trajectories and task splits, does it yield more accurate diagnoses or better skill-selection decisions than ordinary performance differences? What independent evidence validates cases where the metrics disagree?
  2. Can the pruning effects be separated from description-only clarification and documentation shortening? Was the random-pruning baseline evaluated using multiple randomly sampled sets of six tools to remove? How much did performance vary across these sets, and did CC-guided pruning show a consistent advantage?
  3. Please reconcile S3's grouping with Table 11 and apply a consistent definition across settings.
8 comments
AuthorsOpenReview ↗

Response to Reviewer GqYJ (Part 1)

We thank the reviewer for the constructive comments and address the concerns raised below. Should any questions remain, we are happy to provide additional details. We hope our responses clarify the reviewer’s concerns and respectfully request reconsideration of the ratings in light of the clarifications provided. The revised paper is uploaded where the revision are in purple. The code and logs for the rebuttal is under the code branch rebuttal.


Weaknesses 1: The necessity and novelty of the proposed evaluation are insufficiently established. The 2×2 protocol largely applies standard before or after retention evaluation, while fine-grained assessment is already supported by some existing works (e.g., ToolSandbox[1] and TED[2]). CC introduces skill-derived capability units and equivalent-execution judgments, but its added value over existing subgoal evaluation remains unclear. Human agreement and parallel metric reporting can not establish that CC provides better diagnosis or skill-selection decisions than success-rate or progress-rate differences on identical old or new task splits.

Response: Our evaluation is necessary because skill-specific evaluation that is done in prior works cannot reveal whether adding a skill to an existing library degrades previously supported capabilities. The novelty of our paper lies in treating skill addition in an existing system as an incremental system-level intervention and jointly measuring capability expansion and retention on new tasks and previously supported task (also see response to Reviewer 9Bvm, W1). Existing fine-grained evaluations such as TED and ToolSandbox measure task progress through predefined milestones or subgoals, often requiring either literal exact match or judge-assessed semantic exact match. This works well for agents following known workflows with fixed tools, but is limiting for generalist coding agents, which may bypass the supplied skill and achieve the same result using built-in coding tools or alternative operations. We further explain this distinction and the added value of CC in our response to Q1 below.


Weaknesses 2: The pruning study is confounded, and task grouping is inconsistent. S1 simultaneously removes interfaces, clarifies skill descriptions, and shortens documentation, preventing attribution of improvements specifically to CC-guided component selection. The lack of a clearly separated held-out evaluation further limits the refinement claim. Additionally, the released experimental code uses a task grouping inconsistent with the paper's taxonomy. The S3 grouping configuration includes 70 domain-plus-general tasks in the "general + single-domain" results, although Table 11 explicitly classifies these combinations as cross-domain. The authors should reconcile this discrepancy between the code and the paper, and verify whether the domain-specific results and conclusions remain valid under a consistent grouping.

Response: We thank the reviewer for these observations. Both points are addressed in our responses to the reviewer's specific questions:

Pruning Study Confounded: See our detailed response to Q2 below, which addresses the attribution concern regarding simultaneous skill pruning changes (interface removal, skill description clarification, and documentation shortening).

S3 Task Grouping Inconsistency: See our detailed response to Q3 below, which reconciles the discrepancy between the released code and the paper's taxonomy (Table 11) together with revised results.

AuthorsOpenReview ↗

Response to Reviewer GqYJ (Part 2)

Ques 1: What does CC contribute beyond existing subgoal or milestone evaluation? On identical trajectories and task splits, does it yield more accurate diagnoses or better skill-selection decisions than ordinary performance differences? What independent evidence validates cases where the metrics disagree?

Response: Subgoal and milestone evaluation measure whether predefined task outcomes are reached, but generally do not identify which skill capability produced them. Although TED can evaluate both task outcomes and also tool use via exact match of the tool name specified in a subgoal; this becomes limiting when a skill is removed or bypassed and the agent uses another function or its own code. ToolSandbox combines state-based milestones with tool-call assertions tied to specific tool names and arguments, which also reject equivalent alternatives. CC addresses these metrics' limitations by assigning an exact match when the exact reference operation is used and a fuzzy match when a different operation fulfills the same intended capability (see Appendix A.1 in the paper for judge template). Additionally, we provide two new analysis illustrating the added value of our CC metric over existing exact match evaluation approach:

(a) Comparing New System Exact Match and Old System Fuzzy Match on Identical Task. In the new system where the new skill is added, the agent invokes the reference operation defined in the skill; without it as in old system, the agent performs a functionally equivalent operation using built-in coding tools or an alternative function.

For the same task in Setting 1, we compare a pair of CUGA trajectories from a.1) new system vs. a.2) old system. We use the test case Setting 1; tc_301be4c0, whose expected ground truth capability is:

Name: cancel_booking. Description: Cancel an existing booking. First call without confirm to preview cancellation terms, then call with confirm=true to execute. Expected output: {"booking_id": "BK-B70889", "status": "cancelled", "cancellation_fee": 0, "refund_amount": 1180}.

  • a.1) [New system with new skill]: We observe that the agent invokes the reference cancel_booking capability through the skill CLI, first to preview the cancellation and then to confirm it. The judge returns an exact match (E) conclusion for the agent trace CUGA_raw_logs/CUGA_opus4_6_setting1_new_system/tc_301be4c0/trial_1/trace_6_20260921_013305.json. The relevant trajectory snippet and the judge’s corresponding assessment are shown below:

Trajectory snippet: (In L2232-L2236 (preview) and L2277-L2281 (confirm))

$ python cli.py call cancel_booking
--args '{"booking_id":"BK-B70889"}' {"tool": "cancel_booking", "is_write": true, "result": {"status": "preview", "booking_id": "BK-B70889", "cancellation_fee": 0, "refund_amount": 1180, "reason": "Within 24-hour free cancellation window"}}\

$ python cli.py call cancel_booking
--args '{"booking_id":"BK-B70889","confirm":true}' {"tool": "cancel_booking", "is_write": true, "result": {"status": "cancelled", "booking_id": "BK-B70889", "refund_amount": 1180, "cancellation_fee": 0, "reason": "Within 24-hour free cancellation window"}}

Judge Output:

The agent directly invoked the underlying cancel_booking capability via the skill CLI, first as a preview call without confirm, then as an execution call with confirm:true, and both calls succeeded. The final verified booking output matches the expected cancelled state and values for booking BK-B70889 (status: cancelled, cancellation_fee: 0, refund_amount: 1180}. Because this is the same underlying command/script capability, successfully executed with matching output, it is an exact match. GRADE: E

  • a.2) [Old system without the skill]: Without the supplied skill, the reference capability cancel_booking is unavailable. We observe that the agent instead writes ad-hoc Python scripts that directly modify the database, rather than following the required preview-then-confirm procedure. The judge returns a fuzzy match (F) conclusion for the agent trace CUGA_raw_logs/CUGA_opus4_6_setting1_old_system/tc_301be4c0/trial_1/trace_5_20260921_013154.json. The relevant trajectory snippet and the judge's corresponding assessment are shown below:

Trajectory snippet: In L2561-L2565 (cancel_flight.py) and L3923-L3927 (cancel_all.py)

python ./cancel_flight.py Flight booking update: 1 row(s) affected Verification - Booking: BK-B70889, Status: cancelled, Price Paid: 1180, Refund Amount: 1180 Flight cancellation committed to database

python ./cancel_all.py Hotel HR-68D9 updated: 1 row(s) Car CR-68B7 updated: 1 row(s) FLIGHT BK-B70889: status=cancelled, price_paid=$1180, refund=$1180, cabin=business ...

Judge Output:

The agent did achieve a booking cancellation outcome, but it did not invoke the named cancel_booking capability in the required preview-then-confirm flow. ...GRADE: F.

AuthorsOpenReview ↗

Response to Reviewer GqYJ (Part 3)

Ques 1 Response Continuation: CC is particularly important in the no-skill old-system setting because it credits functionally equivalent executions that exact tool-call and predefined-subgoal evaluations often miss. We next use human evaluation to examine disagreements between CC and TED-subgoal and determine which metric better aligns with human judgments.

b) Human Validation of CC Fuzzy Matches vs TED Subgoal Evaluation. We focus this analysis on trajectories that CC classified as fuzzy matches, since recognizing alternative execution paths is a central purpose of CC. From this pool, we randomly sample 15 CUGA and 15 Pi agent ground truth capabilities together with their corresponding trajectories. For comparison, we convert each ground truth capability into a TED-style subgoal and evaluate the same trajectories. This differs from the progress rate evaluation in our paper, which uses TED to assess outcome-based subgoals rather than capability execution. Table 1 reports lenient match agreement, where both exact and fuzzy executions count as valid matches, while Table 2 reports strict label agreement, where CC’s fuzzy classification must agree specifically with the human exact and fuzzy labels. Human evaluation confirmed all 30 samples as valid capability matches, comprising 26 fuzzy and 4 exact matches. CC classified all 30 as fuzzy capability matches, agreeing with humans on match validity in every case and on match type in 26/30 cases (87%). TED-subgoal marked only 4/30 (13%) as completed, including only 1/26 (4%) human-confirmed fuzzy cases, showing that it often misses functionally equivalent execution paths.

Table 1: Lenient human alignment on n=30n=30 human-confirmed capability matches. Both exact and fuzzy executions count as matches. All samples were drawn from CC fuzzy-match predictions; TED-subgoal reports a match when is_completed = True.

MetricMatch (n=30n=30)
Capability Correctness30/30 (100%)
TED-subgoal4/30 (13%)

Table 2: Strict evaluation on the same 30 samples, partitioned by human label. CC's fuzzy verdict agrees only with the 26 human-fuzzy cases, while TED-subgoal reports a match when is_completed = True.

MetricExact (n=4n=4)Fuzzy (n=26n=26)Overall (n=30n=30)
Capability Correctness0/4 (0%)26/26 (100%)26/30 (87%)
TED-subgoal3/4 (75%)1/26 (4%)4/30 (13%)

We will add the above analysis and discussion as a new Subsection 5.4, with further details in Appendix A.9 of the revised paper.

Reviewer_9BvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a more systematic evaluation of the effects of an added skill on the existing tasks and new tasks by comparing them using both the system state and context, each for two cases: without the new skill and with the skill. Since reported aggregate scores on tasks are a sum, they cannot reflect whether the agents perform better or worse on the new problems or old problems; their proposed refined measurements can. It also includes a capability correctness metric to quantify the percentage of the capabilities required for a task is generated by an agent with the skills. The capability correctness metric can also be used to edit skills to remove redundant capabilities.

Critical strengths

The paper identifies a gap in evaluating skills for agents and proposes a more fine-grained evaluation to overcome the limitations using aggregate scores only.

Critical weaknesses

The contributions of the paper seem to be very limited. While aggregate scores may not reflect a usefulness of an added skill, the proposed change seems to be straightforward.

Questions for authors
  • For each task, you assume a list of ground truth capabilities. Do they correspond to API calls for the task? If so, would they depend more on the agent?

  • Even when the list of ground truth capabilities is available, it seems to me a more important question is when and how they are applied to a task, including the needed parameters. Could you explain why did you choose to focus on matching the capabilities?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Bvm (Part 1)

We thank the reviewer for the constructive comment and address the concerns raised below. Should any questions remain, we are happy to provide additional details. We hope our responses clarify the reviewer’s concerns and respectfully request reconsideration of the ratings in light of the clarifications provided. The revised paper is uploaded where the revision are in purple.


Weaknesses: The contributions of the paper seem to be very limited. While aggregate scores may not reflect a usefulness of an added skill, the proposed change seems to be straightforward.

Response: We agree that the 2 x 2 comparison is straightforward; our paper's contribution is to define a different and practically important evaluation paradigm that is overlooked by the literature: the marginal value of a new skill conditioned on an existing skill library, including its effects on previously supported tasks. Prior evaluations based on no-skill baselines or skill-specific tasks cannot determine whether a skill addition extends, duplicates, or disrupts capabilities already supported by the system. We operationalize this setting by jointly measuring capability expansion and retention, introducing CC to assess atomic operations while recognizing functionally equivalent execution, and building a unified seven-domain testbed that captures shared-state interactions, which is often overlooked by prior work that evaluates skills in separate environments.

We will revise the original paper's contributions i) to iii) to the following:

i) We formulate skill evaluation as incremental, system-level attribution through a 2×22\times2 comparison of agents before and after skill addition on new and previously supported tasks. Unlike prior evaluations centered on no-skill baselines or skill-specific tasks, this measures a skill’s value within an existing library and its effects on established capabilities.

ii) We introduce capability correctness (CC), which explicitly evaluates function execution within arbitrarily authored skills. Unlike subgoal evaluation based on exact or semantic matches that overlook alternative actions taken by generalist agents, CC credits both exact matches and functionally equivalent execution.

iii) We integrate seven open-source domains as skills into two generalist agents within a unified, shared-state environment, exposing interactions missed by isolated domain setups in prior work. We provide an automated test-generation pipeline and 320 grounded general, single-domain, and cross-domain test cases.


Ques 1: For each task, you assume a list of ground truth capabilities. Do they correspond to API calls for the task? If so, would they depend more on the agent?

Response: Each ground truth capability cmGT(x)c^{\mathrm{GT}}_m(x) represents an atomic operation required by the task and is derived from the skills. It need not be an API call and may be implemented as a Python function, a shell command such as cat or ls, or another operation. The ground truth is fixed by the task and skill specification, independent of the evaluated agent. The ground truth capability specifies what each capability should accomplish without enforcing how the agent must accomplish it. For example, an agent that bypasses the supplied skill and uses its general coding tools to produce the same outcome still receives credit. CC is therefore agent-agnostic in both its ground truth definition and scoring.

For clarity, we will revise the original paper at Line 215 to include the following detail:

Each ground truth capability is derived from the skill documentation and grounded in the shared environment state, not dependent on the agent nor its execution. Examples of a capability can be an API call, Python function, a shell command such as cat or ls, or other operations.

In addition, we will add to the original paper at Line 223 the following explanation:

Therefore the scoring is independent of how an agent achieves the capability expected outcome. A capability counts as fulfilled either when the agent calls the reference function or when it produces the same expected outcome by any other means (e.g., a different agent that bypasses the skill and solves the task with its own general coding ability still gets credit).

AuthorsOpenReview ↗

Response to Reviewer 9Bvm (Part 2)

Ques 2: Even when the list of ground truth capabilities is available, it seems to me a more important question is when and how they are applied to a task, including the needed parameters. Could you explain why did you choose to focus on matching the capabilities?

Response: We focus on capability matching because existing subgoal-based progress rate evaluation in the literature measures exact matches to predefined milestone outcomes, but does not identify which skill capabilities produced those outcomes. This is particularly limiting for generalist agents, which may bypass the supplied skill and achieve the same result through built-in coding tools or alternative operations. An exact milestone match can measure task progress, but cannot determine whether a skill contributed functionality beyond what the agent could already perform.

CC complements progress rate by evaluating the operations used to achieve the task. Each (cmGT(x))(c^{\mathrm{GT}}_m(x)) contains a reference function or command name, when available, a natural-language capability description, and an expected outcome. When the reference operation appears in the trajectory, CC checks for an exact match against the described capability and outcome. When the reference operation is absent, CC uses the capability description and expected outcome to identify functionally equivalent execution through fuzzy matching. Parameters are represented through the task-specific description and expected outcome rather than requiring a literal argument match. This design provides a common basis for evaluating differently authored skills and allows us to compare agents with and without a skill, even when they solve the task through different implementations.

For clarification, we will include the following paragraph at the end of Section 3.3 in the paper:

While subgoal-based progress rate measures whether predefined milestone outcomes are reached, it does not identify which capability produced them---a key gap for generalist agents, which may bypass the supplied skill and reach the same outcome through built-in coding tools or alternative operations. CC complements progress rate by scoring the operations used: when the exact reference function appears it checks an exact capability-and-outcome match, otherwise it credits a functionally equivalent invocation via fuzzy matching, so parameters are verified through the expected outcome rather than a literal argument match. This lets us compare agents with and without a skill even when they solve the task through different implementations.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal – Reviewer 9Bvm

Dear Reviewer 9Bvm,

Thank you for your constructive feedback. As the rebuttal period comes to an end, please let us know if you have any remaining questions or concerns. If our response has addressed them, we would appreciate it if you would reconsider your rating in light of our clarifications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We thank all reviewers for their thoughtful and constructive feedback, which has helped us improve the clarity and quality of our paper. Please note that our rebuttal references line numbers and sections from the original submission (unless otherwise stated), which we assume remains accessible to the reviewers; if not, please let us know.

In addition, we have uploaded a revised version of the paper in which all relevant changes are highlighted in purple. All code and logs used for the rebuttal can be found in the rebuttal code branch.

Below is a summary of the key concerns raised across reviews, each of which we address in detail in our individual responses below:

  • Clarified the paper's contribution and provided further analysis of the value of the proposed capability correctness (CC) metric. (Reviewer 9Bvm W1, Reviewer GqYJ W1, Reviewer GqYJ Q1)
  • Addressed concerns regarding the pruning experiments. (Reviewer GqYJ W2, Reviewer GqYJ Q2, Reviewer 9K8M W2, Reviewer 9K8M Q2)
  • Resolved the Setting 3 group taxonomy issue. (Reviewer GqYJ W2, Reviewer GqYJ Q3)
  • Strengthened the statistical evidence for our proposed approach by providing per-task performance-difference confidence intervals. (Reviewer 9K8M W1, Reviewer 9K8M Q1)
  • Expanded the human validation study of the CC metric. (Reviewer 9K8M W2)
  • Addressed other minor questions or clarifications. (Reviewer 9Bvm Q1, Reviewer 9Bvm Q2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.