- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces SKILLROOT, a framework for identifying faulty directives in Skill files used by LLM-based agents. To address the limited diagnostic information in conventional execution traces, SKILLROOT enriches each agent action with its supporting context—Skill content, task instructions, prior observations, or the agent’s own judgment—and records the relevant evidence. It then represents Skill directives, entities, and semantic relations in a knowledge graph, enabling related information distributed across the file to be analyzed jointly. By aggregating directive activations across successful and failed executions, the framework computes Ochiai-based suspiciousness scores for both individual directives and related directive groups. Finally, an LLM-based judge combines these historical priors with evidence from a new failure trace to identify the directives most likely responsible for the observed failure.
Critical strengths
- The paper addresses an important and under-explored problem by localizing faulty directives in Skill files, which increasingly serve as persistent, task-specific specifications for LLM-based agents. Unlike prior work that mainly identifies erroneous agents, actions, or steps within a single trajectory, SKILLROOT targets reusable instructions shared across executions. This formulation is practically meaningful because correcting one faulty directive may prevent an entire class of recurring failures.
- The paper presents a well-motivated combination of action-level attribution, cross-execution statistical analysis, and failure-specific LLM reasoning. Instead of asking an LLM to inspect a Skill file and a single failed trace directly, SKILLROOT first aggregates evidence from successful and failed executions to construct a suspiciousness prior. This design reduces reliance on unconstrained LLM reasoning and makes effective use of historical execution evidence.
- The Skill Knowledge Graph provides a structured representation of complex Skill files by connecting directives, entities, procedures, examples, dependencies, refinements, and contradictions distributed across different sections.
- The evaluation is reasonably comprehensive and supports the main claims of the paper.
Critical weaknesses
- The paper does not provide an empirical evaluation of efficiency or scalability, despite motivating SKILLROOT partly through the substantial token overhead of analyzing large trace collections with LLMs. The framework introduces additional costs at several stages, including action-level attribution and counterfactual generation for every tool call, LLM-based knowledge-graph construction, NLI-based trigger checking, co-occurrence search, and the final LLM judge. However, the paper reports neither token consumption nor wall-clock latency, API cost, storage overhead, or preprocessing time.
- The Ochiai score measures whether a directive is disproportionately active in failed executions, while the action-level attribution is self-reported by the executing agent and may reflect post-hoc rationalization. Consequently, a highly ranked directive may correlate with difficult tasks or failure-prone contexts without actually causing the failure. A stronger evaluation would repair or remove the predicted culprit directives and rerun the affected tasks to verify whether the failures are eliminated.
- The benefits of several components are not fully consistent across error types and judge models. The co-occurrence merge improves performance for Opus-4.6 but provides limited or mixed gains for GPT-5.5, while ambiguity remains difficult for all methods.
Questions for authors
Please refer to the Weaknesses
3 comments
Response
I appreciate the authors' response. While W1 has been addressed and the remaining weaknesses have been partially resolved, my concern regarding the SKILLROOT limitation persists. I will maintain my score.
Dear Reviewer,
Thank you for your prompt response. We are glad that our additional results on computational efficiency and scalability have addressed your concerns.
Could you please share your remaining doubts or concerns regarding the other weaknesses you highlighted? We would be happy to provide further clarifications or additional experimental results to address them.
Thank you again for your time and constructive feedback.
Regards,
Authors
W1. Empirical evaluation of SKILLROOT’s efficiency and scalability, including token consumption and other details
Please refer to our response to reviewer FYY4 W2 (a), Q2 for further details about the computational overhead and scalability of self attribution and other compoents.
W2. Distinguishing Faulty Directives from Those Correlated with Failures Due to Difficult Tasks
We thank the reviewer for raising this important question. We would like to clarify that SkillROOT does not aim to repair faulty Skill directives but rather to identify potentially responsible directives using historical evidence and their associated Ochiai suspiciousness scores. The final decision on whether a flagged directive should be removed remains with the agent developer, who can examine the evidence provided by SkillROOT.
We acknowledge that task difficulty may influence suspiciousness scores, as harder tasks may fail for reasons unrelated to the Skill directives. One possible extension is to incorporate task difficulty into Ochiai scoring by using a classifier or an LLM judge to estimate task complexity and assign lower weights to failures on harder tasks, thereby reducing their contribution to the suspiciousness score. This would allow the scoring mechanism to account for both historical execution outcomes and task difficulty. We will further explore this direction as part of our future work.
W3. Inconsistency of performance across judge model and error type.
We thank the reviewer for raising this insightful observation. The performance of our method varies across error types and RCA judge models, largely depending on two factors: (1) whether the available context (execution log and Skill file), together with the model's internal knowledge, is sufficient to identify the faulty directive, and (2) how the RCA judge utilizes the Ochiai suspiciousness scores alongside the provided context.
Across error types, the main benefit of our method arises when the execution log and Skill file alone do not provide sufficient evidence to identify the faulty directive. Contradiction and ambiguity errors are relatively easier to identify, as conflicting instructions are explicitly present in the Skill file, while ambiguous instructions can often be recognized from their wording and possible interpretations. In contrast, wrong-content errors are more challenging because verifying the correctness of a directive may require proprietary or domain-specific knowledge that is neither available in the execution log or Skill file nor part of the model's internal knowledge. Consequently, as the available contextual evidence becomes less sufficient to determine correctness, moving from contradiction and ambiguity to wrong-content errors, historical evidence becomes increasingly valuable, as it allows SkillROOT to contrast directive activations across successful and failed executions and identify which directives are predominantly associated with failures.
Similarly, different RCA judge models utilize the candidate directives, contextual information, and Ochiai scores differently. We observe that Opus tends to rely more heavily on Ochiai suspiciousness scores, whereas GPT-5.5 places greater emphasis on reasoning over the available context, relying on Ochiai scores primarily when contextual evidence is insufficient. These differences in model behavior, potentially arising from their training, are also reflected in our ablation results and help explain the variation in performance across models and error types. We will include this discussion in the revised manuscript.