acceptodds
Under review as a conference paper at ICLR 2027

HatchRCA: Self-Incubated Small Models from LLMs for Root Cause Analysis

Abstract

Root cause analysis (RCA) in cloud systems requires both semantic reasoning over heterogeneous telemetry and quantitative pattern recognition at scale—two capabilities that existing approaches can only combine by manually engineering features and training pipelines for each new dataset, and by admitting empirically summarized rules without quantitative evidence of their reliability. We propose HatchRCA, a framework in which an LLM autonomously designs, trains, and invokes task-specific small models: numerical pattern recognition is delegated to the small models while semantic adjudication remains with the LLM, and both the models and the experience rules are produced by the LLM itself, requiring no human involvement. To make such autonomy safe, every self-generated artifact must meet measured acceptance criteria before deployment—criteria that are data-derived rather than preset, non-degrading, and falsifiable on held-out cases. Models are trained from LLM-written code under an adaptive validation scheme and calibrated before use; those with insufficient accuracy, leakage across splits, or unbounded overfitting are retrained automatically. Rules mined from failure cases are admitted only if they carry sufficient support, correct at least 80% of the cases they fire on, and leave the overall score unchanged or improved, then survive re-verification on held-out cases; artifacts whose benefit cannot be confirmed are never enabled. Under a matched backbone, scorer, and task set, HatchRCA improves the full-score rate by 140% (23.0% to 55.2%) and the mean score by 115% (0.322 to 0.694) over the official RCA agent across 87 tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Root cause analysis, Large language model agents, Autonomous machine learning pipelines, Large-small model collaboration, Gated self-improvement, Rule mining, Telemetry analysis, Confidence calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_QtMzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method that leverages large language models (LLMs) to automatically build small models for root cause analysis (RCA) in microservice systems. Traditional approaches mainly rely on LLMs to directly analyze large volumes of logs, metrics, and traces, but LLMs are not good at precise numerical comparisons and can therefore make incorrect judgments; meanwhile, using specialized machine learning models requires manually designing features and training pipelines for different systems. The core idea of HatchRCA is to shift the LLM from directly performing the entire fault analysis process to first automatically designing small models, generating their training code, and training them based on system-specific data. An independent validation mechanism then selects reliable small models, rules, and historical knowledge, which are finally combined with the LLM for fault diagnosis. This process can identify the fault occurrence time, faulty components, and root causes without requiring humans to manually write system-specific training programs. On the OpenRCA benchmark, which contains 87 tasks, HatchRCA improves the exact-match accuracy from 29.9% to 52.9% and the average score from 0.397 to 0.673, while using the same underlying LLM.

Critical strengths
  1. Strong Innovation: The paper proposes a method that enables LLMs to automatically design, train, and validate task-specific small models, reducing the need for manually designing models and training pipelines for different microservice systems and improving automation.

  2. Significant Experimental Improvements: Under the same backbone model, test set, and evaluation protocol, HatchRCA improves the full-score rate from 29.9% to 52.9%. The paper also provides per-task statistical significance tests, strengthening the credibility of its experimental results.

Critical weaknesses
  1. Reliance on Manually Designed Domain Knowledge: The best-performing configuration of HatchRCA still relies on manually written prompts and rules. After removing these components, the full-score rate drops from 52.9% to 42.5%, indicating limitations in the framework's automation capabilities.

  2. Lack of Ablation Studies: The paper does not independently disable the model admission mechanism or replace the LLM-generated small models with conventional models within the same framework. Therefore, it is difficult to determine the individual contributions of these core components to the overall performance improvement.

Questions for authors
  1. Effectiveness of Core Components: Could the authors provide ablation studies that replace the LLM-generated small models with conventional models and separately disable the admission and confidence-capping mechanisms to clarify each component's contribution to the overall performance improvement?

  2. Fairness of Experimental Comparisons: The best-performing HatchRCA configuration uses additional manually designed domain knowledge and nine hand-crafted rules that are not provided to the official RCA-Agent baseline. Could the authors compare the two methods under the same domain knowledge conditions to determine how much of the improvement comes from these additional inputs?

  3. Degree of Automation in Practice: Although the paper emphasizes eliminating manual model design, HatchRCA-auto still relies on pre-built reference models, shared feature extraction pipelines, and human-prepared hint cards. When applying the framework to a completely new microservice system, what manual preparation is still required, and how much effort would it involve?

4.In Section 5, authors mentioned that “The anonymized supplement contains the admission logs, the per-task predictions and the aggregation scripts that regenerate the tables”. However, I didn’t find where are the supplement materials.

6 comments
AuthorsOpenReview ↗

Q1. Ablations

Yes, we ran both ablations on the 34 Bank test tasks. Replacing the LLM-generated models with a conventional XGBoost model lowers the full-score rate from 58.8% to 50.0% (Table R1). For the admission and cap mechanisms, forcing the admission-rejected models into the pipeline lowers the human full-score count from 23 to 14 of 34, and turning the cap off on the deployed models lowers it from 23 to 19 (Table R2). Admission therefore selects materially better models, and the cap protects the human configuration's score rather than lowering it.

Table R1. LLM-generated models versus the frozen reference model inside the same agent. Bank, 34 tasks, one run per setting.

ModelTotalFull-score rate
LLM-generated model22.9258.8%
Reference model21.3450.0%

Table R2. Admission and confidence cap switched on and off separately. Bank, 34 tasks, one run per setting. The official agent row is the paper's main run on the same 34 tasks.

ModelsConfigCapFull scoreAt least one fieldTotalMean
Official agent (main run)----11/34 (32.4%)17/34 (50.0%)13.580.399
Good (admitted)humanon23/34 (67.6%)28/34 (82.4%)26.330.774
Good (admitted)humanoff19/34 (55.9%)24/34 (70.6%)22.250.654
Good (admitted)autoon16/34 (47.1%)24/34 (70.6%)20.420.601
Good (admitted)autooff18/34 (52.9%)24/34 (70.6%)21.080.620
Bad (gate off)humanon14/34 (41.2%)21/34 (61.8%)18.420.542
Bad (gate off)humanoff14/34 (41.2%)18/34 (52.9%)16.420.483
Bad (gate off)autoon16/34 (47.1%)23/34 (67.6%)19.580.576
Bad (gate off)autooff15/34 (44.1%)22/34 (64.7%)18.000.529
AuthorsOpenReview ↗

Q2. The manually rules

Under the same domain knowledge, with no human-written prompts or rules on either side, HatchRCA-auto reaches a full-score rate of 41.4% over the 87 tasks against 29.9% for the official agent. Adding the human-written prompts and rules gives the full HatchRCA, which reaches 55.2%. These inputs therefore add 13.8 percentage points, while HatchRCA-auto alone is already 11.5 points above the official agent. On Bank alone, HatchRCA-auto reaches 47.1% against 32.4% for the official agent.

We also added the nine human-written rules to the prompt of the official agent and re-ran the 34 Bank tasks. Its full-score rate is 29.4% with the rules against 32.4% without them (Table R3). The rules therefore show no gain for the official agent, and only 2 of the 9 rules ever fired on Bank.

Without human-written knowledge, the gain of HatchRCA-auto is larger in partial answers, where 72.4% of the 87 tasks have at least one correct field against 49.4% for the official agent (Table R4). This gain of 23.0 percentage points is twice the full-score gain of 11.5 points.

Table R3. The nine human rules given to the official agent, Bank, 34 tasks.

SystemFull scoreAt least one fieldTotalMean
Official agent, no rules11/34 (32.4%)17/34 (50.0%)13.580.399
Official agent + nine human rules10/34 (29.4%)17/34 (50.0%)13.250.390
HatchRCA-auto16/34 (47.1%)24/34 (70.6%)20.420.601
HatchRCA, full23/34 (67.6%)28/34 (82.4%)26.330.774

Table R4. Full-score, partial, total and mean scores without human-written knowledge on either side, all 87 tasks.

SystemFull scoreAt least one fieldTotalMean
Official agent26/87 (29.9%)43/87 (49.4%)34.500.397
HatchRCA-auto36/87 (41.4%)63/87 (72.4%)50.430.580
AuthorsOpenReview ↗

Q3. A new system

For HatchRCA-auto, a new system needs two inputs from a person, namely a map from its telemetry names to the shared feature extraction and an adapter that reads its telemetry. The hint cards are the human-written prompt text of the main configuration, and HatchRCA-auto does not use them, because its prompt checklist is empty (Section 4.4). We have not measured the effort in person-hours or lines of code, so we report the required inputs rather than an estimate.

Reviewer_4Yu1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle the problem of root cause analysis in software systems. Their contributions are twofold: the first root cause analysis framework in which an LLM generates its own task-specific small models and an evidence-based admission scheme. They pair these developments with an evaluation on the OpenRCA benchmark and careful ablations of components that require domain knowledge from humans.

Critical strengths
  1. The delegation of telemetry analysis to small machine learning models that are automatically learned is novel.
  2. The gains on OpenRCA are impressive: 29.9% with the official RCA-agent to 52.9% with the RCA-agent paired with the HatchRCA framework
Critical weaknesses
  1. The authors only considered the official RCA-agent as the "base agent". It's unclear what extent the HatchRCA framework applies to general-purpose agents. Such evidence would strengthen the argument that HatchRCA is a general-purpose self-evolving framework for RCA.
Questions for authors
  1. Do the authors have results pairing HatchRCA with another general-purpose agent? A candidate agent harness could be the Terminus-2 agent from the Harbor Framework, which pairs an LLM with only a terminal tool/environment.
1 comment
AuthorsOpenReview ↗

Q1. Terminus-2

Yes, we replaced the official RCA-agent with Terminus-2 (deepseek-v4-flash) and gave it the predicted fault type and the top-5 candidate components from the admitted HatchRCA models. On 20 Bank test tasks, this raises the mean score of Terminus-2 from 0.3125 to 0.5125, with a 95% bootstrap interval of +0.038+0.038 to +0.379+0.379 for the gain (Table R1). The 20 tasks are the 18 Bank test tasks whose scoring includes the root-cause component, plus q13 and q30 to cover latency and packet-loss faults, and we fixed them before both runs. With the HatchRCA outputs, Terminus-2 also consumes 2.69M input tokens over the 20 tasks, 56% fewer than the 6.14M it consumes without them.

Table R1. Terminus-2 with and without the HatchRCA candidates, 20 Bank tasks, deepseek-v4-flash.

No candidatesWith candidates
Total6.2510.25
Mean0.31250.5125
Full score2/205/20
Input tokens6.14M2.69M
Reviewer_rSrgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HatchRCA has an LLM onboard each OpenRCA system by writing training programs for small models (fault type, component, cause, onset) and mining rules/experience. A fixed host admits each artifact only when training-set measurements pass explicit criteria, and caps candidate confidence by host-measured rank-1 accuracy. At diagnosis, admitted models supply ranked candidates; the LLM agent adjudicates over telemetry and may overrule them.

Under a matched backbone and public scorer on 87 OpenRCA tasks, HatchRCA improves full-score rate and mean score over the official RCA-agent on every unit. HatchRCA-auto removes authored RCA prompt/rule material and still beats the baseline but trails the main configuration. Admission rejects several training programs and withholds weak cause/time models.

Critical strengths
  1. Gated LLM→small-model pipeline. Separating generation from host-side admission (relative bars, leakage/reload checks, measured accuracy caps, rule re-verification) is a concrete way to make self-written numeric models usable in RCA agents rather than trusting the LLM to grade its own outputs.

  2. Matched baseline comparison. Holding backbone, scorer, and task set fixed against the official RCA-agent, plus the auto variant without authored RCA knowledge, makes the reported gains interpretable.

Critical weaknesses
  1. No ablation isolating the claimed mechanisms. The paper attributes gains to generated small models and evidence-based admission, but never disables admission alone or swaps generated models for the frozen reference set inside the same agent. Without those controls, the headline lift does not establish which component drives performance. This is decisive for acceptance.

  2. Incomplete autonomy and weak bars. The main system still relies on authored prompts and hand-ported rules; removing them costs a non-trivial share of solved tasks. Feature tables are a shared non-LLM extraction. When the reference scores zero on a class, the relative admission bar cannot reject a zero-recall model, so “measured reliability” collapses to a floor cap. Revise the claim to match what is actually automated, and replace vacuous relative bars with absolute ones.

Questions for authors
  1. Please provide ablations that (a) keep generated models but turn off admission / accuracy capping, and (b) replace generated models with the frozen reference under the same agent—even on one dataset.
  2. What absolute admission bar will replace relative bars when the reference recall is zero, and how does cloudbed-1 latency change under it?
  3. Align the OpenReview abstract numbers with Table 2 in the PDF; they currently disagree.
6 comments
AuthorsOpenReview ↗

Q1(a). Admission and cap, separately

We ran (a) on the 34 Bank test tasks (Table R1). To let the rejected models reach the agent we switched the admission gate off, so the two rows differ in both model quality and the gate. Each row uses its own measured cap ceiling, and every arm is one run on the same 34 tasks with the official scoring program.

Forcing the rejected models into the pipeline lowers the score, most clearly for the full configuration. With the cap on, the human full-score count falls from 23 to 14 of 34 and the mean score from 0.774 to 0.542.

The cap is a clean within-model switch, and on the deployed good models it protects the human score. Turning the cap off lowers the human Bank full-score count from 23 to 19 and the total from 26.33 to 22.25. The autonomous configuration changes little (total +0.66, interval crosses zero), and on the bad models turning the cap off does not help either configuration. We therefore keep the cap. It does not lower the deployed score, and it guards the human configuration against overconfident candidates.

Table R1. Admission and confidence cap switched on and off separately. Bank, 34 tasks, one run per setting. The official agent row is the paper's main run on the same 34 tasks.

ModelsConfigCapFull scoreAt least one fieldTotalMean
Official agent (main run)----11/34 (32.4%)17/34 (50.0%)13.580.399
Good (admitted)humanon23/34 (67.6%)28/34 (82.4%)26.330.774
Good (admitted)humanoff19/34 (55.9%)24/34 (70.6%)22.250.654
Good (admitted)autoon16/34 (47.1%)24/34 (70.6%)20.420.601
Good (admitted)autooff18/34 (52.9%)24/34 (70.6%)21.080.620
Bad (gate off)humanon14/34 (41.2%)21/34 (61.8%)18.420.542
Bad (gate off)humanoff14/34 (41.2%)18/34 (52.9%)16.420.483
Bad (gate off)autoon16/34 (47.1%)23/34 (67.6%)19.580.576
Bad (gate off)autooff15/34 (44.1%)22/34 (64.7%)18.000.529
AuthorsOpenReview ↗

Q1(b). A reference model

For (b), the LLM-generated model outperforms the frozen reference model inside the same agent, with a full-score rate of 58.8% against 50.0% on Bank (Table R2). The reference model uses default XGBoost settings, and both runs use the same 34 Bank test tasks and the official scoring program, with one run each.

Table R2. LLM-generated models versus the frozen reference model inside the same agent. Bank, 34 tasks, one run per setting.

ModelTotalFull-score rate
LLM-generated model22.9258.8%
Reference model21.3450.0%
AuthorsOpenReview ↗

Q2. An absolute bar

We agree that the relative bar fails in this case. It is the reference recall minus 0.05, floored at 0, so when the reference recall is 0, the bar is 0 and even a zero-recall model passes. We will therefore add an absolute bar that does not depend on the reference.

The absolute bar asks whether a component model finds the root cause more often than random guessing. A random top-5 list contains the true root cause with probability 5/n5/n, where nn is the number of candidate components. A model passes only if its hits on the validation cases beat this rate in a one-sided binomial test at the 0.05 level. The bar thus follows from nn and the number of validation cases and has no tuned constant.

For cloudbed-1 latency, n=56n = 56 and there are 3 validation cases, so the random hit rate is 5/56=0.0895/56 = 0.089 and a model needs at least 2 hits. The cloudbed-1 latency model hits none, so the absolute bar rejects it, although the relative bar admitted it (Table R3). Re-running the 17 cloudbed-1 tasks without this model gives a total score of 9.33 against 8.92 with it, with 5 tasks at full score in both runs. Removing the model therefore costs nothing on cloudbed-1.

On the floor cap, a rejected model receives no cap at all, because the agent never sees it. Among the admitted models in the revised runs, 5 of 12 have a measured accuracy at or below 0.30 and receive the floor of 0.30. The other 7 receive their measured accuracy, from 0.40 to 0.80 (Section 4.4).

Table R3. Relative and absolute bars on the original reference set. Needed hits is the minimum number of validation cases that the model must hit to beat random guessing at the 0.05 level.

DatasetFault typeReference recallModel recallNeeded hitsAbsolute bar
cloudbed-1LATENCY0.00.02Reject
cloudbed-1PACKET_LOSS0.00.44Keep
TelecomPACKET_LOSS0.00.83Keep
TelecomRESOURCE0.050.856Keep
cloudbed-2LATENCY0.3330.6672Keep
Reviewer_XF1nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper approaches root cause analysis through LLMs training small task-specific models to detect faults. The small models along with rules and experience will be used by the LLM under qualified settings, and result in significant improvements in the OpenRCA benchmark.

Critical strengths

The paper introduces letting an LLM build a model that performs numeric comparisons for RCA.

Its criterion provides quality gating with evidence-based admission.

The LLM builds the model that performs numeric comparison while carrying measured accuracy into diagnosis.

Critical weaknesses

Limitations of smaller backbone models, overfitting to OpenRCA benchmark.

Questions for authors

Do results scale to larger backbone models? How brittle are the constants and limitations in appendix G?

3 comments
AuthorsOpenReview ↗

Q1. A larger model

We replaced deepseek-v4-flash with the larger deepseek-v4-pro as the LLM that selects the input features of the small models, and the small models trained on the two sets of features differ by at most 2 of the 34 Bank test tasks. We ran this feature selection twice with each LLM, trained XGBoost models on each selected feature set with the same training program, kept every other part fixed, and scored the models on the 34 Bank test tasks (Table R1). We score the small models with two metrics, strict@kk and hit@3. The metric strict@kk is the fraction of these tasks on which the predicted fault type is correct and the ground-truth root-cause component lies within the top-kk candidates of that type's component model. The metric hit@3 is the fraction of these tasks on which the ground-truth root-cause component lies within the top-3 retrieved candidates, regardless of the predicted type. The two LLMs give identical strict@3 and strict@5 in all four runs. Across these runs, strict@1 ranges from 21 to 23 of 34, and hit@3 ranges from 85.3% to 88.2%.

Table R1. Small models trained on the features selected by each LLM, 34 Bank test tasks.

LLMRunstrict@1strict@3strict@5hit@3
deepseek-v4-flash021/3424/3425/3488.2%
deepseek-v4-flash122/3424/3425/3488.2%
deepseek-v4-pro022/3424/3425/3485.3%
deepseek-v4-pro123/3424/3425/3488.2%
AuthorsOpenReview ↗

Q2. The constants and the limitations

We changed each constant alone and re-ran the admission check on the metrics recorded in our runs (Table R2). Only one of the six fixed constants, the cause floor, changes an admission decision when moved by 12.5% of its current value. Every other constant can change by less than 50% without changing any decision.

The limitations in Appendix G stand as follows. (1) Without the human-written prompts and rules, the number of the 87 tasks solved at full score falls from 48 to 36 (Table R3). (2) The models that admission rejects are materially worse: forcing them into the pipeline lowers the human full-score count on Bank from 23 to 14 of 34 (Table R4). (3) The weakness of relative bars is addressed by an absolute bar that requires each component model to beat random guessing in a one-sided binomial test at the 0.05 level, which rejects the zero-recall cloudbed-1 latency model (Table R5).

Table R2. Fixed constants changed one at a time. The last column gives the first change, relative to the current value, that alters an admission decision. Any smaller change leaves every decision unchanged.

ConstantRoleCurrent valueFirst value that alters a decisionChange that alters a decision
Cause floorminimum accuracy to use the cause model0.400.35 or 0.4512.5%
Component margincomponent bar is reference recall minus margin0.050.07550%
Overfit boundmaximum gap of training over validation accuracy0.320.169%
Rule supportminimum fired training cases to keep a rule25150%
Fault-type marginfault-type bar is reference accuracy minus margin0.020.1400%
Rule precisionminimum precision to keep a rule0.8not alter from 0.5 to 1.0100%

Table R3. Full-score, partial, total and mean scores without human-written knowledge on either side, all 87 tasks.

SystemFull scoreAt least one fieldTotalMean
Official agent26/87 (29.9%)43/87 (49.4%)34.500.397
HatchRCA-auto36/87 (41.4%)63/87 (72.4%)50.430.580

Table R4. Admission and confidence cap switched on and off separately. Bank, 34 tasks, one run per setting. The official agent row is the paper's main run on the same 34 tasks.

ModelsConfigCapFull scoreAt least one fieldTotalMean
Official agent (main run)----11/34 (32.4%)17/34 (50.0%)13.580.399
Good (admitted)humanon23/34 (67.6%)28/34 (82.4%)26.330.774
Good (admitted)humanoff19/34 (55.9%)24/34 (70.6%)22.250.654
Good (admitted)autoon16/34 (47.1%)24/34 (70.6%)20.420.601
Good (admitted)autooff18/34 (52.9%)24/34 (70.6%)21.080.620
Bad (gate off)humanon14/34 (41.2%)21/34 (61.8%)18.420.542
Bad (gate off)humanoff14/34 (41.2%)18/34 (52.9%)16.420.483
Bad (gate off)autoon16/34 (47.1%)23/34 (67.6%)19.580.576
Bad (gate off)autooff15/34 (44.1%)22/34 (64.7%)18.000.529

Table R5. Relative and absolute bars on the original reference set. Needed hits is the minimum number of validation cases that the model must hit to beat random guessing at the 0.05 level.

DatasetFault typeReference recallModel recallNeeded hitsAbsolute bar
cloudbed-1LATENCY0.00.02Reject
cloudbed-1PACKET_LOSS0.00.44Keep
TelecomPACKET_LOSS0.00.83Keep
TelecomRESOURCE0.050.856Keep
cloudbed-2LATENCY0.3330.6672Keep
AuthorsOpenReview ↗

Q3. Overfitting

We do not think the results are overfit to the OpenRCA benchmark, for two reasons. First, no program can read test data. The split is fixed before the LLM builds any model, training reads only the training queries, and the sandbox blocks the files that define the test split. Second, the models can find root-cause components that never appear as a root cause in the training data. In the cloudbed-1 test set, 5 of the 16 distinct root-cause components never appear as a root cause in training, and in the cloudbed-2 test set 3 of the 18 never do. A correct answer on these components therefore cannot come from memorizing the training answers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their constructive and detailed reviews, and we are encouraged that all four reviewers now recommend acceptance. The main points raised in the discussion concerned the mechanisms behind the gains and what HatchRCA automates relative to human-written knowledge. Our revisions address both, alongside evidence on transfer to another agent.

The added ablations confirm that the admission mechanism contributes directly to performance. In the main configuration, forcing admission-rejected models into the pipeline lowers the full-score count on Bank from 23 to 14 of 34, showing that admission selects better models. We further strengthen admission with an absolute bar based on a binomial test, which requires no tuned constant.

We have clarified that the LLM writes every training program without human editing (Section 3.2). With no human-written knowledge on either side, HatchRCA-auto still raises the full-score rate from 29.9% to 41.4% on all 87 tasks. Giving the human rules to the baseline agent brings no gain, so the rules alone do not reproduce the improvement.

The gains also generalize. Supplying HatchRCA candidates to Terminus-2 raises its mean score from 0.3125 to 0.5125 while reducing input tokens by 56%. The results are also stable under a larger feature-selection LLM and under changes to most fixed constants.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.