1. Limited practical applicability due to oracle-like assumptions
SkillDiver generates test cases from known target skills, which are subsequently used to construct reference pools (Sections 3.2 and 3.3). However, in realistic scenarios involving arbitrary user requests, the appropriate skills are unknown, and identifying them is precisely the problem being evaluated. More importantly, the benchmark evaluation (Section 4.2, Table 2) explicitly constructs reference pools using ground-truth skill labels, potentially favoring the proposed method. This raises substantial concerns about whether SkillDiver can reliably detect selection errors without privileged information and limits the practical significance of the reported evaluation results.
2. Insufficient experimental evidence to establish the framework's superiority
The primary experimental comparison (Section 4.1, Table 1) relies on GoS-Random-Direct, a relatively simple baseline, without direct comparisons against more competitive automated testing or evaluation approaches. Furthermore, the reported improvements in Selection Deviation Rate (SDR) measure selection inconsistencies rather than confirmed errors. Although the paper includes human validation and ablation studies (Sections 4.1 and 4.4), the experimental evidence remains insufficient to establish that SkillDiver identifies substantially more genuine selection errors than existing alternatives under comparable evaluation conditions.
Thank you for taking the time to review our submission and for communicating your concerns about your ability to assess the paper to the Area Chair. We would be happy to provide further clarifications on our methodology or evaluation if any specific questions arise during the discussion.