acceptodds
Under review as a conference paper at ICLR 2027

SkillDiver: Constraint-Guided Generation and Adaptive Differential Testing to Evaluate Skill Selection

Abstract

Large language model (LLM) agents increasingly use reusable skills, but selecting relevant skills becomes less reliable as skill libraries grow. Evaluating skill selection at scale remains challenging for two reasons: extending labeled benchmarks to new skills and usage scenarios requires costly task construction and label validation, while a task may admit multiple valid skill selections, making per-task correctness labels difficult to define. We develop SKILLDIVER, a framework that combines constraint-guided task generation with adaptive differential testing to evaluate skill selection without requiring per-task correctness labels. Constraint-guided generation extracts and samples fine-grained constraints from skill documents to generate tasks. Adaptive differential testing treats the selection from the full skill library as the selection under test and assesses it by comparing with selections from smaller reference pools. We use two types of reference pools: the target pool, which contains the skills used to generate the task, and the adaptive pool, which additionally includes skills in the selection under test. When the selection under test is not reproduced in either reference execution, SKILLDIVER reports a selection deviation as a potential selection error. Across two agents and four settings, SKILLDIVER exposes more selection deviations than GoS-Random-Direct, increasing the selection deviation rate by 9.8% to 90.4%. For validation on four labeled benchmarks, the differential module achieves an average F1 of 84.58% in identifying selections that disagree with benchmark labels, outperforming an LLM judge by 28.46 percentage points. With skill chains held fixed, constraint-guided generation increases within-chain semantic diversity by 76.6% on SkillsBench and 61.6% on R3-Skill. Overall, SKILLDIVER explores how task generation and differential testing can support automated skill selection testing without requiring per-task correctness labels.

Primary area:
datasets and benchmarks
Keywords:
Agent evaluation, Skill selection, Automated testing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AJvDOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I do not have sufficient expertise to assess this submission reliably and have alerted the AC.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

1 comment

Thank you for taking the time to review our submission and for communicating your concerns about your ability to assess the paper to the Area Chair. We would be happy to provide further clarifications on our methodology or evaluation if any specific questions arise during the discussion.

Reviewer_sGrvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillDiver generates tasks from constraints extracted from skill documents and tests skill selection by varying the available pool. It flags a full-pool selection when repeated executions in smaller source and adaptive pools fail to reproduce it. Feedback guides further task sampling. Experiments cover two agents and four settings, with benchmark and human validation of the reported deviations.

Thank you for the detailed rebuttal and additional experiments. The new comparisons, sampling-adjusted results, and clarified annotation criteria substantially address my main concerns and provide stronger evidence for the effectiveness of SkillDiver. I am satisfied with the responses and will raise my rating from 2 to 3

Critical strengths
  1. The adaptive pool accommodates valid alternatives to the source skills. The audio-conversion example in Appendix E.3 demonstrates why checking against the source alone can reject a valid selection. The paper appropriately distinguishes a deviation from a proven error.

  2. SkillDiver finds more deviations in all eight comparisons. Human annotators judge 80–83% of its reports incorrect, with substantial agreement across the full audit (kappa = 0.786).

Critical weaknesses
  1. The benchmark comparison supplies the answer through the target pool. Section 4.2 constructs this pool from the labeled skill set, then scores correctness by exact agreement with that set. Simply flagging selections that differ from the target pool would score perfectly. Table 2 therefore tests recovery of a supplied label rather than independent correctness judgment. The LLM judge receives less information and follows a different criterion: it can accept a skill-free solution, which exact-match scoring rejects when the label is nonempty. These differences complicate the claimed 28.46-point F1 advantage. The human-labeled generated tasks are the more convincing test; comparisons with source-chain mismatch and target-pool-only testing are missing there.

  2. The human audit needs reweighting and clearer annotation criteria. It samples equal numbers of reported and unreported cases, although reports constitute only 18.4% of SkillDiver's tests. Reweighting Table 7 gives an estimated recall of 56–63%, versus 85–88% in Table 8; uncertainty intervals are needed. Also, 78.5% of reports are omissions, and a relevant skill need not be necessary. In Appendix E.5.2, the reference runs select erlang-distribution for a request that never specifies Erlang. This does not establish that selecting no skill is wrong. The annotation instructions should explain how optional skills and valid skill-free solutions were handled.

Questions for authors
  1. On human-labeled tasks, how do source-chain mismatch and target-pool-only testing compare with the adaptive rule?

  2. Does the advantage over the LLM judge persist after sampling adjustment and alignment of its instructions with the scoring criterion?

  3. How did annotators distinguish optional skills from required ones, particularly for empty selections such as Appendix E.5.2?

3 comments

Thank you for your detailed feedback. We have added simple-rule comparisons, revised the LLM judge, corrected for stratified sampling, and clarified the annotation criteria, and respectfully ask you to reconsider your rating in light of these revisions. Detailed responses follow.

Q1&&W1: Comparison of simple judgment rules and adaptive differential testing.

Thank you for this suggestion. We add a comparison of adaptive testing with two simpler judgment rules on human-annotated cases. We also extend the original 400-case set with 200 cases from GoS-Coverage-Direct, a baseline added in response to another reviewer (R2-Q4), using the same sampling and annotation protocol.

Experiment. We compare three judgment rules on the resulting 600 cases: (1) source-chain mismatch, which flags selections differing from the source skills; (2) target-only testing, which checks whether target-pool executions reproduce the full-pool selection; and (3) adaptive testing, which additionally considers adaptive-pool executions. We evaluate matching thresholds (k=1,2,3) for the latter two rules using the same recorded executions. We report both unweighted and sampling-adjusted results to account for the stratified sampling.

R3-Table 1. Judgment reliability on 600 human-annotated cases (%). Each entry reports Precision / Recall / F1.

RulekA1 UnweightedA1 ReweightedA2 UnweightedA2 Reweighted
Source-chain mismatch–68.10 / 98.07 / 80.3846.12 / 92.26 / 61.5070.51 / 98.87 / 82.3251.02 / 95.70 / 66.56
Target-only169.58 / 95.37 / 80.4647.39 / 81.52 / 59.9371.83 / 95.86 / 82.1352.17 / 84.15 / 64.41
Adaptive179.67 / 92.28 / 85.5179.96 / 69.28 / 74.2480.67 / 90.98 / 85.5180.61 / 65.49 / 72.27
Target-only269.83 / 96.53 / 81.0448.77 / 86.18 / 62.2972.07 / 96.99 / 82.6953.43 / 88.52 / 66.64
Adaptive278.71 / 94.21 / 85.7675.36 / 77.01 / 76.1879.68 / 92.86 / 85.7675.92 / 72.74 / 74.30
Target-only368.96 / 96.91 / 80.5847.18 / 87.76 / 61.3771.43 / 97.74 / 82.5452.45 / 91.48 / 66.67
Adaptive375.38 / 95.75 / 84.3563.24 / 83.10 / 71.8276.90 / 95.11 / 85.0466.01 / 81.34 / 72.88

A1 and A2 denote the two human annotators. Reweighting uses the same method- and reporting-status-specific sampling weights as the 600-case comparison in R3-Q2.

Results. At the default threshold (k=1), adaptive testing improves sampling-adjusted F1 over target-only testing by 14.31 and 7.86 percentage points for the two annotators, respectively. Across all thresholds, adaptive testing consistently achieves higher precision and F1 than both simpler rules, at the cost of lower recall. Among the 55 cases reported by target-only testing but filtered out by adaptive testing at (k=1), the annotators judge 47 and 42 as correct selections, respectively, while the remaining 8 and 13 are genuine errors. These results show that adaptive testing reduces false positives by accommodating valid alternatives, while potentially missing some genuine errors.

Q2&&W2: The LLM judge comparison after sampling adjustment and criterion alignment.

Thank you for this valuable suggestion. We revise the LLM judge to address the differences in available information and judgment criteria, and reevaluate it on the four benchmarks and human-annotated generated-task cases. The results indicate that, even after sampling adjustment, SkillDiver still outperforms the new LLM judge.

Revised LLM judge. We preserve task-level selection appropriateness as the judgment objective while improving the information and criteria available to the judge. Specifically, we make the following changes (- indicates removed text; + indicates added text):

1. Provide the available skill catalog and execution trace. This allows the judge to assess the necessity of unselected skills and examine the observed selection in context.

+ Available skill catalog:
+ {available_skill_catalog}

+ Recorded full-pool execution trace:
+ {execution_trace}

2. Clarify the criteria for omissions and unnecessary selections. We align the judge with the human annotation guidelines, including the role of skills in reducing exploratory effort, and apply consistent criteria to empty and non-empty selections.

- When the selected skill set is empty, consider whether the agent using the same model could complete the task with its existing knowledge and capabilities without consulting a skill.
+ An omission requires applicable, task-specific guidance that addresses a concrete gap in the observed execution or could substantially reduce exploratory effort, without adequate coverage from the selected skills. Mere topical relevance or potential usefulness is insufficient.
+ A selected skill is unnecessary if it has no concrete task-related role. Supporting guidance and reasonable applicability checks can be valid.

3. Update the evidence instruction to reflect the additional information.

- Base your decision only on the task and the descriptions of the selected skills.
+ Base your decision only on the task, the available skill catalog, the selected skills and their documentation, and the recorded full-pool execution trace.

Sampling adjustment. For both case sets, we weight each case by the population-to-sample ratio of its generation-method and reporting-status stratum, then compute accuracy, precision, recall, and F1 from the weighted confusion counts. Approximate 95% confidence intervals use stratified sampling variance, finite-population correction, and the delta method, retaining within-case covariance for paired differences. We retain the original 400-case set from SkillDiver and GoS-Random-Direct to compare the original and revised judges on the same cases, and additionally report the final 600-case comparison between the revised judge and differential testing, matching Appendix D.2 of the revised manuscript.

R3-Table 2a. Judge revision on the original 400 human-annotated cases: sampling-adjusted metrics (%). F1 is reported with 95% confidence intervals.

AnnotatorMethodAccuracyPrecisionRecallF1 (95% CI)
1Original LLM judge86.6771.4246.5156.34 [47.06, 65.61]
1Revised LLM judge86.9368.1654.9860.86 [52.24, 69.49]
1Differential testing (k=1)90.5977.7468.7672.98 [66.12, 79.83]
2Original LLM judge84.9370.4942.4152.96 [43.63, 62.29]
2Revised LLM judge85.4568.3150.9058.34 [49.77, 66.90]
2Differential testing (k=1)89.6679.5665.0271.56 [64.67, 78.45]

Effect of revising the judge (400 cases). On this fixed case set, the revised LLM judge improves sampling-adjusted F1 from 56.34% to 60.86% and from 52.96% to 58.34% for the two annotators, respectively. Differential testing achieves F1 scores of 72.98% and 71.56%, exceeding the revised judge by 12.11 pp [0.93, 23.29] and 13.22 pp [2.79, 23.66], respectively (paired 95% CIs).

Final comparison (600 cases). After including 200 cases from GoS-Coverage-Direct, the following sampling-adjusted results are obtained.

R3-Table 2b. Final sampling-adjusted judgment reliability on 600 human-annotated cases (%). F1 is reported with 95% confidence intervals.

AnnotatorMethodAccuracyPrecisionRecallF1 (95% CI)
1Revised LLM judge86.7566.5254.0659.64 [52.53, 66.76]
1Differential testing (k=1)91.2979.9669.2874.24 [68.53, 79.95]
2Revised LLM judge85.9667.9051.7558.73 [51.64, 65.83]
2Differential testing (k=1)90.2980.6165.4972.27 [66.50, 78.03]

Results. On the final 600 cases, differential testing exceeds the revised judge in sampling-adjusted F1 by 14.59 pp [5.52, 23.66] and 13.53 pp [4.74, 22.33] for the two annotators, respectively (paired 95% CIs). Before weighting, its F1 is 85.51% for both annotators, compared with 66.22% and 66.96% for the revised judge. On the four benchmarks, differential testing also achieves higher macro-average F1 (84.58% vs. 62.15%). The advantage therefore persists after revising the judge, expanding the human evaluation, and adjusting for stratified sampling.

Q3&&W2: Annotation of optional skills and valid empty selections.

Thank you for this clarification request. Annotators assess selection appropriateness based on the task requirements, available skill catalog, skill documentation, and execution trace, without seeing reference selections or differential verdicts. An omission is considered an error when an applicable, unselected skill addresses a concrete execution gap or could substantially reduce unnecessary exploration, without adequate coverage from selected skills. Mere topical relevance or potential usefulness is insufficient. A selected skill is unnecessary only when it has no concrete task-related role. Empty selections follow the same criteria, and execution traces serve as evidence rather than task success or failure determining correctness.

In Appendix E.5.2, the full-pool execution selects no skill and explores general cluster-access mechanisms, while a reference execution using erlang-distribution develops a concrete workflow for remote message delivery and receipt verification. This illustrates how skill guidance may reduce unnecessary exploration. However, since the request does not specify Erlang, the reference selection alone does not establish that the empty selection is incorrect. Appendix E.5.2 presents this case as an omission deviation reported by the differential module. Conversely, both annotators accept an empty selection for a clipboard-checking task generated from using-wpf-clipboard, because the request does not specify a Windows/WPF context and the skill is not applicable in the Linux execution environment.

Finally, the reported 78.5% refers to omission-only selection deviations, not human-confirmed errors. Whether an omission constitutes a genuine error requires case-specific assessment.

W1: Benchmark reference-pool construction and label dependence.

Thank you for highlighting this concern. To assess this effect, we construct label-independent target pools by retrieving the top-5 or top-10 skills using text-embedding-3-small, while retaining the same 1,164 benchmark tasks and full-pool selections.

R3-Table 3. Macro-average judgment reliability across four benchmarks (%).

Reference-pool constructionPrecisionRecallF1
Original label-derived99.8074.3084.58
Retrieved Top-594.1473.5982.32
Retrieved Top-1092.9065.7576.57
Original LLM Judge95.9640.9156.12
Revised LLM Judge97.4447.2162.15

Results. Replacing label-derived target pools with retrieved Top-5 and Top-10 pools reduces macro-average F1 by 2.26 and 8.01 percentage points, respectively. Nevertheless, both configurations retain higher F1 scores (82.32% and 76.57%) than the original LLM judge (56.12%) and revised LLM judge (62.15%, R3-Q2). These results indicate that label-derived pool construction affects judgment reliability but does not fully explain differential testing's advantage. Additionally, our human-annotated evaluation (see R3-Q1 and R3-Q2) provides supplementary evidence that further demonstrates the effectiveness of differential testing.

Reviewer_45wJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillDiver, a framework for automatically detecting potential errors in skill selection by LLM agents. SkillDiver extracts capabilities and constraints from skill documents to automatically generate diverse test cases, and identifies potential selection errors by comparing the skills selected by an agent across full and reduced skill candidate pools. It also adaptively adjusts subsequent test generation based on previously observed selection deviation rates and test counts. The authors demonstrate that SkillDiver discovers more potential selection errors than the baseline method and evaluate the effectiveness of its error detection approach using ground-truth labels from existing benchmarks and human judgments.

Critical strengths

SkillDiver provides an automated framework for assessing how reliably LLM agents select appropriate skills for a given task.

Critical weaknesses

1. Limited practical applicability due to oracle-like assumptions

SkillDiver generates test cases from known target skills, which are subsequently used to construct reference pools (Sections 3.2 and 3.3). However, in realistic scenarios involving arbitrary user requests, the appropriate skills are unknown, and identifying them is precisely the problem being evaluated. More importantly, the benchmark evaluation (Section 4.2, Table 2) explicitly constructs reference pools using ground-truth skill labels, potentially favoring the proposed method. This raises substantial concerns about whether SkillDiver can reliably detect selection errors without privileged information and limits the practical significance of the reported evaluation results.

2. Insufficient experimental evidence to establish the framework's superiority

The primary experimental comparison (Section 4.1, Table 1) relies on GoS-Random-Direct, a relatively simple baseline, without direct comparisons against more competitive automated testing or evaluation approaches. Furthermore, the reported improvements in Selection Deviation Rate (SDR) measure selection inconsistencies rather than confirmed errors. Although the paper includes human validation and ablation studies (Sections 4.1 and 4.4), the experimental evidence remains insufficient to establish that SkillDiver identifies substantially more genuine selection errors than existing alternatives under comparable evaluation conditions.

Questions for authors
  1. Practical applicability: Is SkillDiver intended exclusively for testing tasks generated from known skills, or can it also evaluate arbitrary real-world user requests? In the latter case, how would the target pool be constructed without knowing which skills are appropriate for the request (Section 3.3)?

  2. Oracle dependency in evaluation: In Section 4.2, benchmark ground-truth skill labels are used to construct the reference pools and evaluate selection correctness. To what extent does this design influence the reported F1 scores in Table 2? Would the conclusions remain valid if the reference pools were constructed independently of the ground-truth labels?

  3. Actual error discovery: Does a higher Selection Deviation Rate (SDR) translate into discovering more genuine skill-selection errors? Can the authors provide a comparison of human-confirmed error discovery rates between SkillDiver and GoS-Random-Direct under the same testing budget, accounting for the stratified sampling procedure used in the human evaluation (Section 4.1 and Appendix D.2)?

  4. Comparison with stronger baselines: The primary comparison in Table 1 relies on GoS-Random-Direct, which uses random skill-chain sampling and direct task generation. Would SkillDiver's advantages remain significant against stronger alternatives, such as coverage-guided or adaptive test generation strategies, under equivalent computational budgets? In particular, can the authors demonstrate that SkillDiver discovers more genuine selection errors per unit of testing cost, rather than simply producing more selection deviations?

4 comments

Thank you for your feedback. We have added label-independent reference pools and human-confirmed error discovery comparisons against random and coverage-guided baselines, and respectfully ask you to reconsider your rating in light of these results. Detailed responses follow.

Q1&&W1: Reference-pool construction for arbitrary user requests.

Thank you for this valuable suggestion. We'd like to clarify that SkillDiver is designed to automatically test agents' skill-selection capabilities by proactively generating tasks from skill constraints (i.e., the former you mentioned). The source skills are therefore known and form the target pool, but are not assumed to be the correct selection.

Experiment. Regarding the latter case of evaluating arbitrary real-world user requests, we view this as a promising practical extension of SkillDiver. To explore its feasibility, we introduce a retrieval-based approach using text-embedding-3-small to identify the top-k skills based on semantic similarity between user requests and skill names and descriptions. These retrieved skills form the target pool without requiring ground-truth labels, while the remaining differential testing procedure remains unchanged.

Results. We evaluate this approach with k=5 and k=10 on 1,164 benchmark requests, achieving macro-average F1 scores of 82.32% and 76.57%, respectively. These results provide evidence that our differential testing method can operate without known source skills. Detailed results are provided in Q2&&W1.

Q2&&W1: The effect of label-derived reference pools on evaluation results.

Experiment. Thank you for raising this important concern. To examine the influence of ground-truth labels on reference-pool construction, we conduct additional experiments using the retrieval-based approach described in R2-Q1, with k=5k=5 and k=10k=10. We retain the same 1,164 benchmark requests and full-pool selections, changing only the construction of target pools.

R2-Table 2. Judgment reliability under different reference-pool constructions. Per-benchmark entries report Precision / Recall / F1.

BenchmarkMetricLabel-derivedRetrieved Top-5Retrieved Top-10Original LLM Judge
Macro Avg.Precision99.8094.1492.9095.96
Recall74.3073.5965.7540.91
F184.5882.3276.5756.12
SkillsBenchP / R / F1100.00 / 68.25 / 81.1390.57 / 76.19 / 82.7695.12 / 61.90 / 75.0094.74 / 28.57 / 43.90
SkillRetP / R / F1100.00 / 95.95 / 97.9399.65 / 89.10 / 94.0897.89 / 86.60 / 91.9098.48 / 60.75 / 75.14
R3-SkillP / R / F1100.00 / 61.50 / 76.1693.28 / 66.84 / 77.8888.28 / 60.43 / 71.7596.74 / 47.59 / 63.80
SRA-BenchP / R / F199.19 / 71.51 / 83.1193.04 / 62.21 / 74.5690.29 / 54.07 / 67.6493.88 / 26.74 / 41.63

Results. Replacing label-derived pools with retrieved Top-5 and Top-10 pools reduces macro-average F1 by 2.26 and 8.01 percentage points, respectively, to 82.32% and 76.57%. Both remain above the original judge (56.12%) and revised judge (62.15%, R3-Q2). Thus, reference-pool construction affects reliability, but SkillDiver's advantage is not solely attributable to label-derived reference pools.

Q3&&W2: The relationship between SDR and genuine error discovery.

Thanks for this valuable suggestion. We agree that a higher SDR does not necessarily imply the discovery of more genuine skill-selection errors. After adjusting for stratified sampling, SkillDiver still achieves a higher estimated human-confirmed error discovery rate than GoS-Random-Direct under the same test-case budget, for both annotators.

Experiment. Since the human audit stratifies cases by reporting status, we estimate EDR^=(Nreported/Ntotal)×(Nconfirmed/Nsampled reported)\widehat{\mathrm{EDR}} = (N_{\mathrm{reported}}/N_{\mathrm{total}}) \times (N_{\mathrm{confirmed}}/N_{\mathrm{sampled\ reported}}). We further compute 95% confidence intervals using a design-based normal approximation with finite-population correction.

R2-Table 3. Estimated human-confirmed error discovery rates (EDR) with 95% confidence intervals. Differences are SkillDiver minus GoS-Random-Direct.

AnnotatorSkillDiverGoS-Random-DirectDifference (pp)Relative gain
115.25% [13.94, 16.56]10.17% [8.95, 11.39]+5.08 [3.29, 6.87]+49.95%
214.70% [13.30, 16.10]11.32% [10.22, 12.41]+3.38 [1.61, 5.16]+29.90%

Results. SkillDiver achieves 29.90%–49.95% higher estimated human-confirmed error discovery rates, with both 95% confidence intervals for the differences above zero. These results support its advantage in discovering human-confirmed selection errors per test.

Q4&&W2: Comparison with stronger test-generation baselines.

Thanks for this suggestion. We add GoS-Coverage-Direct, a coverage-guided baseline, and find that SkillDiver still achieves higher SDR and sampling-adjusted human-confirmed error discovery rates (EDR) under the same test-case budget.

Experiment. GoS-Coverage-Direct prioritizes less frequently sampled skills. It retains GoS-Random-Direct's task generation and differential testing procedures, but replaces random chain sampling with coverage-guided sampling, assigning each skill a weight of 1/(1+n(s))1/(1+n(s)), where n(s)n(s) is its previous sampling count, and sampling chains proportionally to the L2 norm of their skill weights.

R2-Table 4a. Comparison under the eight settings. Each pair reports SkillDiver / GoS-Coverage-Direct, with 1,000 test cases per method per setting.

AgentSettingSDR (%)SCovDAS
CodexSkillsBench / 19918.70 / 15.20199 / 19997 / 78
R3 / 507.20 / 5.20860 / 89776 / 58
R3 / 10015.80 / 9.20856 / 905133 / 105
R3 / 20029.50 / 26.20881 / 906264 / 253
GLMSkillsBench / 19916.30 / 12.20199 / 19993 / 67
R3 / 5013.60 / 9.80848 / 873165 / 110
R3 / 10019.10 / 15.60872 / 894179 / 147
R3 / 20026.80 / 21.50862 / 905245 / 226

Results. SkillDiver achieves higher SDR and DAS across all eight settings, with SDR improvements of 2.00–6.60 percentage points. GoS-Coverage-Direct achieves higher SCov in all R3 settings, consistent with its explicit prioritization of less frequently sampled skills.

Experiment. To assess genuine error discovery, we extend the sampling-adjusted EDR analysis in R2-Q3. We randomly sample 100 reported and 100 unreported cases from GoS-Coverage-Direct for evaluation by the same two annotators (Cohen's κ=0.808\kappa=0.808), and compare against SkillDiver's existing human annotations.

R2-Table 4b. Estimated human-confirmed error discovery rates (EDR, 95% CI). Differences are SkillDiver minus GoS-Coverage-Direct.

AnnotatorSkillDiverGoS-Coverage-DirectDifference (pp)
115.25% [13.94, 16.56]12.21% [11.24, 13.17]+3.04 [1.41, 4.67]
214.70% [13.30, 16.10]11.92% [10.91, 12.94]+2.78 [1.05, 4.51]

Results. Across 8,000 tests per method, SkillDiver reports 1,470 deviations compared with 1,149 for GoS-Coverage-Direct. More importantly, SkillDiver achieves 23.31%–24.93% higher estimated human-confirmed error discovery rates, with both 95% confidence intervals for the differences above zero. These results demonstrate its advantage in genuine error discovery under the same test-case budget.

I thank the authors for the additional experiments, which address some of my concerns. However, I have decided to maintain my rating for the following reasons.

First, the difference in F1 between Top-5 (82.32%) and Top-10 (76.57%) suggests that the results depend on how the reference pools are constructed. While this is not necessarily a problem, it is difficult to tell how much of the performance comes from retrieval and how much from differential testing. I would have liked to see comparisons with other retrieval methods, such as random selection, BM25, and different embedding models, as well as comparisons with other error detection methods using the same retrieved pools. These would help clarify the contribution of SkillDiver itself.

Second, the EDR analysis is a useful addition, but the estimates are based on 100 human-annotated reported cases per method, sampled across eight settings. This makes it difficult to tell whether the reported improvements hold consistently across different agents and skill pools. Also, the new coverage-guided baseline only changes how skills are sampled, while keeping direct task generation and the same differential testing procedure. I therefore remain unconvinced that the experiments sufficiently demonstrate SkillDiver's advantage over stronger testing alternatives.

Thanks for your feedback.

More retrieval and error detection.

Regarding retrieval and error detection, one possible explanation for the F1 decrease from Top-5 (82.32%) to Top-10 (76.57%) is that a larger reference pool increases the chance that the agent repeats an incorrect full-pool selection during reference executions. Under our differential testing rule, such matches prevent deviation reports, leading to more false negatives and lower recall. This is consistent with our motivation for using smaller reference pools.

To further examine the impact of reference-pool construction, we add BM25 and Qwen-based embedding retrieval. We also introduce a retrieval-aware LLM judge that receives the same retrieved candidates.

Retrieval-aware LLM judge. We retain the revised judge’s existing inputs and judgment criteria, while adding the retrieved candidates and instructions on their interpretation:

+ Retrieved candidate skills (Top-{k}):
+ {retrieved_skill_catalog}
+ The retrieved candidates are the Top-{k} skills recommended from the available catalog for this task. They are recommendations, not ground-truth labels, and may include irrelevant skills or omit appropriate ones. Consider them alongside the full available catalog under the criteria above. Do not assume that every retrieved skill is required or judge the selection as incorrect solely because it differs from the retrieved set.

R2-Table 5. Macro-averaged judgment reliability across retrieval methods and reference-pool sizes (%). Each entry reports Precision / Recall / F1. For each benchmark, retrieval method, and Top-k setting, both judgment methods use the same retrieved candidates.

Top-kJudgment methodtext-embedding-3-smallBM25Qwen-based embedding
5Differential testing94.14 / 73.59 / 82.3298.75 / 66.27 / 78.5294.37 / 72.34 / 81.64
Retrieval-aware LLM judge93.35 / 48.65 / 62.1894.98 / 46.96 / 61.2095.64 / 49.31 / 63.49
10Differential testing92.90 / 65.75 / 76.5798.24 / 66.92 / 78.9691.02 / 69.85 / 78.83
Retrieval-aware LLM judge95.37 / 51.58 / 65.3896.49 / 48.15 / 62.5296.62 / 49.13 / 63.51

Results. As shown in R2-Table 5, differential testing consistently outperforms the retrieval-aware LLM judge across all three retrieval methods and both reference-pool sizes.

EDR across configurations.

Regarding EDR across configurations, our human validation primarily evaluates differential judgment reliability, using 100 reported and 100 unreported cases per method. We agree that this sample size limits configuration-specific EDR analysis, while obtaining reliable estimates across eight configurations and three methods requires substantially more human annotations than the rebuttal period permits.

Importantly, all three methods use the same differential testing procedure, and SkillDiver achieves higher SDR than both baselines in all eight settings. The sampling-adjusted human validation further shows higher estimated aggregate EDR. We believe these results provide sufficient evidence for SkillDiver's aggregate error-discovery advantage.

More stronger test-generation baselines.

To our knowledge, there is no established automated test-generation baseline directly applicable to agent skill-selection setting. Therefore, we construct GoS-Random-Direct and GoS-Coverage-Direct, using random and coverage-guided skill-chain sampling, respectively. The latter is introduced in response to your suggestion.

Regarding task generation, both baselines generate tasks directly from skill documents, drawing on the document-based generation approach used in existing skill-selection benchmarks such as R3-Skill. We additionally examine the contribution of constraint-guided generation through an ablation experiment (Section 4.4, Table 3), replacing it with direct generation while keeping the skill chains and full pools fixed. SkillDiver achieves higher SDR across all four Codex settings, supporting the contribution of constraint-guided generation to deviation discovery. Of course, potentially more effective task-generation strategies may exist, and exploring them remains a direction for future work.

Regarding error detection, we hold the differential testing procedure fixed across task-generation methods to isolate the contribution of task generation. Changing both generation and detection would make their individual contributions difficult to distinguish, and a higher SDR alone could reflect more false positives. We separately evaluate the differential testing module in RQ2 (Section 4.2 and Appendices D.2–D.3), assessing its judgment reliability against benchmark labels and human annotations and comparing adaptive differential testing with other judgment rules. These experiments separately assess the reliability and contribution of the differential testing module.

Reviewer_uj6EOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They propose an automated testing tool, SkillDiver, for checking whether agents choose the correct skills. It generates various test tasks, and then judges the agent’s skill choice using LLM judge. Evaluate GPT 5.5 with Codex and GLM-5.3-Flash with OpenHands on skill libraries of up to 200 skills. They show that their tool exposes more selection deviations than GoS-Random-Direct, increasing the selection deviation rate. They demonstrate that their tool explores how task generation and differential testing can support automated skill selection testing without requiring per-task correctness labels.

Critical strengths
  • The idea is sensible
  • Human validation with inter-annotator agreement reported, and threshold sensitivity checks.
Critical weaknesses

My main concern is that the paper only looks at skill selection, not task performance. If an agent picks the wrong skill, I’d expect its performance to drop, but this is never checked. Even with a “wrong” skill, the agent could still finish the task correctly, especially since most reported deviations are omissions. So I’m not sure how much these deviations actually matter. It’d be helpful to check the correlation between skill choice and task performance. I also think the evaluation is a bit limited. They only test two agents (Codex with GPT-5.5 and OpenHands with GLM-5.3-Flash), and the skill pool is capped at 200 even though the paper is motivated by large skill libraries. Evaluating more agents and larger pools would make the results more convincing.

Questions for authors

Please see the weaknesses above.

2 comments

Thank you for the constructive feedback. We would like to clarify that SkillDiver identifies potential selection errors through differential testing across skill pools, rather than relying on an LLM judge, which is used only as an evaluation baseline. We have also added downstream task-performance analysis and experiments with an additional model and 500-skill pools. We respectfully ask you to reconsider your rating in light of these results. Our detailed responses follow.

Q1&&W1: The correlation between skill selection and task performance.

Thanks for your useful suggestion. Our evaluation focuses on skill selection, as SkillDiver is designed to test this capability and generated tasks often lack reliable outcome verifiers. Moreover, prior studies have examined the impact of skill usage and selection on downstream task performance [1, 2]. Nevertheless, we agree that examining the relationship between skill selection and downstream task performance would further strengthen our evaluation. We conducted experiments using SkillsBench, and the results indicate a positive correlation between skill selection and downstream task performance.

Experiment. We conduct an end-to-end evaluation using Codex with GPT-5.5 on 86 SkillsBench tasks with available benchmark verifiers, using the full SkillsBench skill library and repeating each task six times. We compare skill selections against the SkillsBench reference labels and evaluate task performance using the original benchmark tests. We report the mean task score (0–1) and the full-score success rate (task score = 1). To account for differences in task difficulty, we further compare executions within the 29 tasks exhibiting both GT-aligned and GT-mismatched selections.

R1-Table 1. Relationship between skill-selection agreement and task performance.

AnalysisSelection statusRunsMean task scoreFull-score success rate
All tasks (86)GT-aligned850.55051.76% (44/85)
GT-mismatched4310.35833.41% (144/431)
Omission-only3520.36634.38% (121/352)
Within-task (29)GT-aligned790.64060.29%
GT-mismatched950.46344.08%
Difference (95% CI)—+0.176 [0.075, 0.290]+16.21 pp [6.26, 27.59]

GT-aligned indicates exact agreement with the SkillsBench reference skills. Omission-only denotes selections that omit reference skills. The all-task results are averaged across runs.

Results. Within the same tasks, GT-aligned selections achieve higher mean task scores by 0.176 [0.075, 0.290] and full-score success rates by 16.21 percentage points [6.26, 27.59] (95% CIs). Notably, omission-only selections nevertheless succeed in 34.38% of runs. These results establish a positive association between skill-selection agreement and downstream performance. Moreover, an omitted skill does not necessarily prevent task completion, and its impact warrants further investigation.

[1] Li Y, Liu W, Gao B, et al. SkillsInjector: Dynamic Skill Context Construction for LLM Agents. arXiv preprint arXiv:2605.29794, 2026.

[2] Li X, Liu Y, Chen W, et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv preprint arXiv:2602.12670, 2026.

Q2&&W2: More agents and larger skill pools.

Experiment. Thank you for the suggestion. The original evaluation covers only two agent–model configurations due to cost considerations. We initially limit the skill pool size to 200 to accommodate Codex's default skill-catalog context budget (2% of its context window), and adopt the same settings for OpenHands to ensure comparability. This range is also consistent with Li (2026), which studies skill libraries containing 5–200 skills [1]. However, we also agree with your suggestion. Therefore, we additionally evaluate OpenHands with DeepSeek-V4.1-Flash-1017 across all four original settings and extend both OpenHands models to 500-skill pools.

R1-Table 2. Selection deviation discovery across models and skill-pool sizes. Each pair reports SkillDiver / GoS-Random-Direct. SDR denotes selection deviation rate (%), SCov and DAS denote skill coverage and deviation-associated skills.

SettingGLM SDR (%)GLM SCovGLM DASDeepSeek SDR (%)DeepSeek SCovDeepSeek DAS
SkillsBench / 19916.30 / 11.70199 / 19893 / 6913.00 / 7.90199 / 19979 / 46
R3 / 5013.60 / 12.20848 / 773165 / 9511.70 / 6.70854 / 800100 / 56
R3 / 10019.10 / 17.40872 / 781179 / 13717.00 / 9.80889 / 815145 / 75
R3 / 20026.80 / 21.20862 / 801245 / 18223.00 / 17.80911 / 843209 / 146
R3 / 50033.60 / 23.50901 / 849343 / 23134.70 / 27.20912 / 854317 / 233

Results. SkillDiver achieves higher SDR than GoS-Random-Direct across all settings for both OpenHands models. At 500 skills, the SDR gains are 10.10 percentage points for GLM and 7.50 points for DeepSeek. These results extend the observed discovery advantage to an additional model and larger candidate pools.

[1] Li X. When single-agent with skills replace multi-agent systems and when they fail[J]. arXiv preprint arXiv:2601.04748, 2026.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We have addressed their comments in our responses and revised the manuscript with the corresponding experiments and clarifications.

  1. Task performance and generalizability (Reviewer uj6E-Q1/Q2). We add an analysis of the relationship between skill-selection agreement and downstream task performance. We also extend the evaluation to an additional model and larger skill pools (Sections 2 and 4.1, Appendix D.8).

  2. Applicability and label dependence (Reviewer 45wJ-Q1/Q2, Reviewer sGrv-W1). We introduce retrieval-based target-pool construction without ground-truth labels and evaluate Top-5 and Top-10 pools on the same benchmark requests (Section 4.2, Appendices B.2 and C.2).

  3. Genuine error discovery and stronger baselines (Reviewer 45wJ-Q3/Q4). We add the coverage-guided baseline GoS-Coverage-Direct and compare the methods using human-confirmed error-discovery rates under the same test-case budget. The revised evaluation includes sampling-adjusted results and confidence intervals (Section 4.1, Appendices C.1–C.4 and D.2).

  4. Judgment reliability and annotation criteria (Reviewer sGrv-Q1/Q2/Q3). We compare the adaptive rule with simpler judgment rules, revise the LLM-judge evaluation, and report both unweighted and sampling-adjusted results. We also clarify the annotation criteria, optional skills, empty selections, and the interpretation of the Erlang example as a selection deviation (Section 4.2, Appendices D.2–D.3, E.5.2, and F.5).

Additionally, adding the coverage-guided baseline and expanding the experimental configurations required updating related results and downstream analyses, including threshold sensitivity and human-annotated evaluations. We also updated the LLM-judge comparisons to reflect the revised prompt. Accordingly, we synchronized the related analyses and summaries throughout the manuscript, including the Abstract, Introduction, Conclusion, and Appendices D.1 and D.4–D.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.