acceptodds
Under review as a conference paper at ICLR 2027

KETriage: Incident Triage with Evolving Hierarchical Symbolic Knowledge

Abstract

Cloud incidents can degrade user experience and cause financial losses, while identifying the responsible service through incident triage remains challenging due to large blast radii and complex fault propagation. Despite extensive efforts, existing approaches still face two fundamental challenges: the absence of integration mechanisms for domain knowledge and brittle adaptation to evolving systems. We present KETriage, a knowledge-enhanced and self-improving agentic framework that identifies the responsible service. To bridge the knowledge gap of LLM agents in incident triage, KETriage constructs a hierarchical symbolic knowledge base (HSKB) offline, a symbolic graph organized into ontology, instance, and case layers. During online inference, KETriage anchors incident terminologies to the HSKB ontology taxonomy and retrieves relevant knowledge segments to identify the responsible service. Given the inherent dynamic nature of cloud systems, KETriage further introduces post-resolution knowledge consolidation, enabling continual experience evolution without prohibitive retraining. We conducted comprehensive experiments using real-world incidents collected from an industrial cloud service system serving millions of users. The experimental results show that KETriage outperforms existing methods, achieving Acc@1 of 79.48% and 93.66% Acc@3 in incident triage.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM Agents, Incident Triage, AIOps, Knowledge-Augmented Reasoning, Self-Improving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WYtrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KETriage, a knowledge-enhanced agentic framework for cloud incident triage. It constructs a Hierarchical Symbolic Knowledge Base (HSKB) from historical incidents, organized into ontology, instance, and case layers. During online triage, the agent performs terminology alignment, knowledge retrieval, and concurrent incident filtering, then identifies the responsible service based on current failure evidence, upstream responsibility boundaries, and fault propagation relationships. After incidents are resolved, confirmed historical experience is incorporated into the knowledge base. Experiments on 804 real-world industrial incidents show that KETriage achieves 79.48% Acc@1 and 93.66% Acc@3, outperforming several existing baselines.

Critical strengths

S1. Hierarchical knowledge organization and evidence-constrained attribution are well suited to cross-service incident triage. HSKB connects service dependencies, failure patterns, and historical cases while distinguishing current incident evidence from historical knowledge. This helps avoid attributing responsibility based solely on similar past incidents. The design is particularly useful for cross-service incidents, where KETriage achieves 74.36% Acc@3, compared with 62.56% for TRIANGLE.

S2. The experimental results on real-world industrial incidents are encouraging. The paper evaluates 804 real-world incidents and includes component ablations, knowledge evolution experiments, execution cost comparisons, and error analysis. In particular, the full system reduces average execution time from 679.79 seconds to 190.08 seconds compared with the w/o HC variant, which directly accesses raw evidence. This suggests that structured knowledge can reduce redundant retrieval and reasoning overhead for the agent.

Critical weaknesses

W1. Key implementation details of knowledge construction, retrieval, and updating are missing. Section 3 mainly describes the method through abstract functions such as Retrieve, Filter, and Update. It does not clearly explain how ontology properties are induced, how entities and relations are extracted, how local knowledge is retrieved and expanded, or how new knowledge is merged with existing knowledge and conflicts are handled. Although the paper defines the F/B/D constraints, their implementation and the ranking of candidates that satisfy these constraints are not described in sufficient detail for reproduction. These steps are central to the system's performance.

W2. The experiments do not sufficiently isolate the contributions of hierarchical knowledge organization and knowledge evolution. The ablations in Table 1 demonstrate the overall benefit of HSKB, but there is no direct comparison with flat RAG under the same model, evidence, and computational budget. For knowledge evolution, Figure 3 mainly shows performance changes as more historical knowledge is added to a fixed test set. This does not establish whether the system can reliably adapt to changing service dependencies, conflicting knowledge, or outdated information. In addition, the overall Acc@1 improvement over DeepTriage is only 2.12 percentage points, and no statistical significance analysis is provided.

Questions for authors

Q1. How are HSKB construction and online retrieval implemented?

Could the authors clarify how entities and relations are extracted, how terminology is aligned, how knowledge segments are retrieved and expanded, and how the F/B/D checks and candidate ranking are performed? In particular, how does the agent decide which graph nodes and historical cases to retrieve, and when to stop retrieving?

Q2. How much of the improvement comes specifically from hierarchical knowledge organization rather than conventional RAG?

Could the authors compare flat RAG and a service-dependency-only graph under the same LLM, raw evidence, and computational budget? The distinction between w/o HSKB and w/o HC also needs clarification to determine whether the gains come from knowledge structure or differences in how evidence is accessed.

Q3. Can knowledge evolution adapt to changing service relationships rather than simply accumulating historical cases?

How does the system update or revise existing knowledge when new incident records conflict with established service dependencies or failure patterns? Could the authors evaluate scenarios involving changes in service relationships and analyze the 14 Top-1 regressions caused by knowledge updates? This would help substantiate the claimed continual self-improvement capability.

Reviewer_7sjROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces KETriage, a knowledge-enhanced and self-improving agentic framework for identifying the service responsible for causing the root incident. The framework proposes hierarchical symbolic knowledge base built from the knowledge about resolved incidents.

Critical strengths
  • Well-designed system with potential for practical applications.
  • Relevant and important topic.
  • The manuscript is easy to read and follow.
Critical weaknesses
  • The novelty of the proposed method is not identified clearly enough in comparison to existing works. While the authors provide related works, they do not showcase how the closest other papers (such as COMET, Comfey, etc) are different from their proposed method.
  • It is imppossible to assess the impact of Hierarchical Symbolic Knowledge (even though it is included in ablation), as there is no comparison to flat RAG memory in the same setup and system.
  • The introduction discusses "Brittle adaptation to evolving systems" as well as "post-resolution knowledge consolidation that continuously assimilates domain knowledge from resolved incidents". However, in the experiments, the only varied element is the knowledge base related to incidents. This is a limited evaluation of adaptation, focused only on operational knowledge, but not environmental change in the system.
Questions for authors
  • What exactly is the novelty in comparison to other existing works?
Reviewer_Xcm1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies incident triage in cloud systems. Given a new incident and its data, the goal is to name the service that is responsible. The hard case is a cross-service incident, where the responsible service is not the one that reports the impact.

KETriage is an LLM agent that works over a hierarchical symbolic knowledge base (HSKB). The HSKB has three layers. The ontology layer holds object types and relation types that on-call engineers supply. The instance layer is a typed graph of (services, teams, monitors, their dependencies). The case layer stores resolved incidents and links them to the claims that they support. At inference time, the agent maps incident terms onto the ontology and pulls the matching part of the knowledge base, which it then uses to filter the concurrent incidents. Before it names an external service as responsible, the agent needs evidence that this service is failing now and sits upstream of the target. It also needs a link between the two that is specific to this incident. Once the incident is resolved, an offline step writes the confirmed resolution, rather than the agent's prediction, into the next HSKB revision.

Critical strengths

The task and data are realistic, and the paper reports the hard case separately. Cross-service incidents are where triage costs engineers the most time (it is a real and important problem), and the paper breaks results out for them and for the data-plane and control-plane subsets. I appreciate the effort.

Critical weaknesses
  1. The Acc@1 gain is small once you account for how the test set is made up, and the paper does not show that it is statistically reliable. Of the 804 test incidents, 609 are self-inflicted, so a rule that always predicts the impacted service already scores 75.75% Acc@1 with no model at all. KETriage scores 79.48%. That is 3.7 points above the rule, and 2.12 points (about 17 incidents) above DeepTriage. KETriage correctly attributes 70 cross-service incidents to an upstream service, but it also sends 40 self-inflicted incidents to some external service, a mistake the rule never makes. On self-inflicted incidents DeepTriage still has the higher Acc@1 (95.07% vs. 93.43%), and on the cross-service incidents that motivate the paper, KETriage's Top-1 answer is wrong for 125 of 195. There is also no measure of uncertainty. Each configuration runs only once with an LLM agent. I cannot tell from a single run whether a 2-point gap is real. The ablations have the same problem: removing EI lowers Acc@1 from 79.48% to 77.24%, a difference that might not survive a second run of the same agent. And because every result comes from one month at one provider, even a stable gap might not carry over to another system.
  2. Missing details to reproduce the results or to judge whether the comparison is fair. I could not find what training data DeepTriage and DeepCT used, or whether it covered the same period as the HSKB. COMET and TRIANGLE may therefore have run on the same model, but the paper does not state it. If they ran on a weaker model or without concurrent incidents, part of the reported advantage comes from the setup and not from the HSKB.
  3. The claims about interpretability are not tested. The introduction argues that on-call engineers need reasoning that they find convincing, and the contribution list promises "compelling reasoning chains". However, there is little evaluation of this interpretability.
Questions for authors
  1. Whether the gains come from the HSKB or from gpt-5.6-sol? It would be better if we can have results with a second model, like an open-weight model.
  2. At what point in an incident’s life is the input x_i taken (the input to the triage agent, Section 3.1)? If x_i or the concurrent records were exported after resolution, they could name the responsible service outright. Please state when the input is captured for KETriage and for each baseline. DeepCT, for example, sees only the first 10 minutes of discussion. If KETriage reads later records, it has more information than the baselines, and Table 1 would overstate how well it triages.
  3. Why does w/o HC (70.52% Acc@1) score lower than w/o HSKB (75.50%)?
  4. How is the responsible service labeled for cross-service incidents? How did you handle incidents with several contributing services or with ownership that changed after the first assignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.