- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a method that leverages large language models (LLMs) to automatically build small models for root cause analysis (RCA) in microservice systems. Traditional approaches mainly rely on LLMs to directly analyze large volumes of logs, metrics, and traces, but LLMs are not good at precise numerical comparisons and can therefore make incorrect judgments; meanwhile, using specialized machine learning models requires manually designing features and training pipelines for different systems. The core idea of HatchRCA is to shift the LLM from directly performing the entire fault analysis process to first automatically designing small models, generating their training code, and training them based on system-specific data. An independent validation mechanism then selects reliable small models, rules, and historical knowledge, which are finally combined with the LLM for fault diagnosis. This process can identify the fault occurrence time, faulty components, and root causes without requiring humans to manually write system-specific training programs. On the OpenRCA benchmark, which contains 87 tasks, HatchRCA improves the exact-match accuracy from 29.9% to 52.9% and the average score from 0.397 to 0.673, while using the same underlying LLM.
Critical strengths
-
Strong Innovation: The paper proposes a method that enables LLMs to automatically design, train, and validate task-specific small models, reducing the need for manually designing models and training pipelines for different microservice systems and improving automation.
-
Significant Experimental Improvements: Under the same backbone model, test set, and evaluation protocol, HatchRCA improves the full-score rate from 29.9% to 52.9%. The paper also provides per-task statistical significance tests, strengthening the credibility of its experimental results.
Critical weaknesses
-
Reliance on Manually Designed Domain Knowledge: The best-performing configuration of HatchRCA still relies on manually written prompts and rules. After removing these components, the full-score rate drops from 52.9% to 42.5%, indicating limitations in the framework's automation capabilities.
-
Lack of Ablation Studies: The paper does not independently disable the model admission mechanism or replace the LLM-generated small models with conventional models within the same framework. Therefore, it is difficult to determine the individual contributions of these core components to the overall performance improvement.
Questions for authors
-
Effectiveness of Core Components: Could the authors provide ablation studies that replace the LLM-generated small models with conventional models and separately disable the admission and confidence-capping mechanisms to clarify each component's contribution to the overall performance improvement?
-
Fairness of Experimental Comparisons: The best-performing HatchRCA configuration uses additional manually designed domain knowledge and nine hand-crafted rules that are not provided to the official RCA-Agent baseline. Could the authors compare the two methods under the same domain knowledge conditions to determine how much of the improvement comes from these additional inputs?
-
Degree of Automation in Practice: Although the paper emphasizes eliminating manual model design, HatchRCA-auto still relies on pre-built reference models, shared feature extraction pipelines, and human-prepared hint cards. When applying the framework to a completely new microservice system, what manual preparation is still required, and how much effort would it involve?
4.In Section 5, authors mentioned that “The anonymized supplement contains the admission logs, the per-task predictions and the aggregation scripts that regenerate the tables”. However, I didn’t find where are the supplement materials.
6 comments
Q2. The manually rules
Under the same domain knowledge, with no human-written prompts or rules on either side, HatchRCA-auto reaches a full-score rate of 41.4% over the 87 tasks against 29.9% for the official agent. Adding the human-written prompts and rules gives the full HatchRCA, which reaches 55.2%. These inputs therefore add 13.8 percentage points, while HatchRCA-auto alone is already 11.5 points above the official agent. On Bank alone, HatchRCA-auto reaches 47.1% against 32.4% for the official agent.
We also added the nine human-written rules to the prompt of the official agent and re-ran the 34 Bank tasks. Its full-score rate is 29.4% with the rules against 32.4% without them (Table R3). The rules therefore show no gain for the official agent, and only 2 of the 9 rules ever fired on Bank.
Without human-written knowledge, the gain of HatchRCA-auto is larger in partial answers, where 72.4% of the 87 tasks have at least one correct field against 49.4% for the official agent (Table R4). This gain of 23.0 percentage points is twice the full-score gain of 11.5 points.
Table R3. The nine human rules given to the official agent, Bank, 34 tasks.
| System | Full score | At least one field | Total | Mean |
|---|---|---|---|---|
| Official agent, no rules | 11/34 (32.4%) | 17/34 (50.0%) | 13.58 | 0.399 |
| Official agent + nine human rules | 10/34 (29.4%) | 17/34 (50.0%) | 13.25 | 0.390 |
| HatchRCA-auto | 16/34 (47.1%) | 24/34 (70.6%) | 20.42 | 0.601 |
| HatchRCA, full | 23/34 (67.6%) | 28/34 (82.4%) | 26.33 | 0.774 |
Table R4. Full-score, partial, total and mean scores without human-written knowledge on either side, all 87 tasks.
| System | Full score | At least one field | Total | Mean |
|---|---|---|---|---|
| Official agent | 26/87 (29.9%) | 43/87 (49.4%) | 34.50 | 0.397 |
| HatchRCA-auto | 36/87 (41.4%) | 63/87 (72.4%) | 50.43 | 0.580 |
Q3. A new system
For HatchRCA-auto, a new system needs two inputs from a person, namely a map from its telemetry names to the shared feature extraction and an adapter that reads its telemetry. The hint cards are the human-written prompt text of the main configuration, and HatchRCA-auto does not use them, because its prompt checklist is empty (Section 4.4). We have not measured the effort in person-hours or lines of code, so we report the required inputs rather than an estimate.
Q1. Ablations
Yes, we ran both ablations on the 34 Bank test tasks. Replacing the LLM-generated models with a conventional XGBoost model lowers the full-score rate from 58.8% to 50.0% (Table R1). For the admission and cap mechanisms, forcing the admission-rejected models into the pipeline lowers the human full-score count from 23 to 14 of 34, and turning the cap off on the deployed models lowers it from 23 to 19 (Table R2). Admission therefore selects materially better models, and the cap protects the human configuration's score rather than lowering it.
Table R1. LLM-generated models versus the frozen reference model inside the same agent. Bank, 34 tasks, one run per setting.
Table R2. Admission and confidence cap switched on and off separately. Bank, 34 tasks, one run per setting. The official agent row is the paper's main run on the same 34 tasks.