acceptodds
Under review as a conference paper at ICLR 2027

TopoGAT: Topology-Aware Unsupervised Root-Cause Localization in Distributed Training via Spatio-Temporal Graphs

Abstract

Large language model (LLM) training relies on parallelism for efficiency, yet this architecture introduces a vulnerability: a single faulty rank propagates delays through collective synchronization barriers, slowing the entire cluster and causing substantial computational waste. Existing fault localization methods depend on fault labels or hand-crafted rules, which are scarce in production and generalize poorly across new training workloads. They also treat ranks independently, ignoring the training topology and failing to distinguish root-cause ranks from victim ranks slowed by synchronization. We propose TopoGAT, an unsupervised spatio-temporal graph framework for root-cause rank localization. TopoGAT exploits the inherent parallelism topology by reconstructing each rank’s behavioral profile from its topological neighbors and its own history, learning normal synchronization patterns without fault labels. Using universally available telemetry—collective communication events, GPU metrics, and network counters—it generalizes across diverse workloads. Trained once on a single benchmark topology and evaluated zero-shot on 11 real-world datasets spanning 3 model families, 10 parallelism configurations, and both computation- and communication-side faults, TopoGAT achieves an average top-3 localization accuracy (Hit@3) of 94.6%, significantly outperforming state-of-the-art baseline methods.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Unsupervised root-cause localization, Distributed model training, Spatio-temporal heterogeneous graph, Graph attention network

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JHPPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TopoGAT, which leverages the parallelism topology and normal spatiotemporal patterns of distributed training to capture relationships between nodes. It further introduces a two-stage error calibration mechanism to mitigate common anomalies caused by synchronization propagation, enabling root-cause localization without requiring fault labels. Overall, the paper is well motivated, and the experimental evaluation is fairly comprehensive and convincing.

Critical strengths
  1. I think the paper is motivated by a very good and important problem. One particularly interesting observation is that, in distributed training, the actual faulty node may not have the highest anomaly score, as synchronization delays can also slow down other nodes. I find this insight particularly compelling.

  2. The experimental evaluation is thorough and provides convincing evidence to support the paper’s main claims.

  3. The paper is well written and well organized, making the motivation, proposed method, and experimental results easy to follow.

Critical weaknesses
  1. On D1, TopoGAT performs worse than VAE despite its topology-aware design, while generalizing better to unseen datasets. It would be helpful to clarify the reason for this difference.

  2. The learned attention weights are nearly uniform, and replacing them with uniform weights has little impact on performance. This raises questions about the necessity of GAT over simpler aggregation methods. Could the authors better isolate the contribution of the graph structure compared with strong non-graph baselines and further explain the model’s cross-topology generalization?

  3. In Table 23, localization performance improves as more ranks are missing (from 4 to 8 to 16 ranks). This unexpected trend warrants further investigation, ideally with repeated trials to rule out sampling variability.

  4. In Equation (5), the term “neighbor mean” is not clearly defined. Please specify which neighbors are included in the aggregation.

Questions for authors

See Weaknesses.

Reviewer_LRGNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attacks a real production pain point: figuring out which rank is dragging down a distributed LLM training job. The idea is to train a 0.16M-param graph network over per-iteration telemetry — nodes are ranks, edges follow the TP/PP/DP topology plus time — using only normal data from one 64-rank setup, then ship it zero-shot to other models and parallelism configs. What makes transfer work is a two-stage calibration (subtract the group median, divide by each rank's p99) plus cross-rank z-scoring at scoring time, with CUSUM+MAD cutting out anomaly windows. They test on 11 production datasets, 3 model families, 10 parallelism configs and 12 injected fault types, reaching 94.6% offline / 88.9% online Hit@3 zero-shot against 8 baselines, with bootstrap CIs, paired tests, 10 seeds and decomposition ablations.

Critical strengths
  • The authors are refreshingly honest with themselves: they report that their learned attention ends up essentially uniform and that a plain typed-mean aggregator matches it, and they show the model falls apart when ranks drop out.
Critical weaknesses
  • Every "fault" in the evaluation is injected into otherwise healthy runs (hardware-side or via a modified Megatron-LM). Real incidents come with confounders — concurrent faults, network congestion, operator interventions — and none of that is tested, so I don't know what precision to expect in actual deployment.
  • The "GAT" in TopoGAT doesn't pull its weight: as shown above, the authors' own ablation shows attention is effectively uniform and a typed-mean aggregator is statistically indistinguishable, so the architectural story oversells what actually drives the results (typed topology + calibration + learned residuals).
Questions for authors
  1. Are D1–D11 all from one hardware generation and interconnect? No dataset states GPU model, interconnect, or cluster identity.
  2. Would the zero-shot story and the cross-rank z-scoring survive a move across GPU generations or a different telemetry cadence?
  3. What happens when a fault saturates a whole parallelism group and contaminates the group median your de-commoning step subtracts? Your victim-dominance assumption rules this case out, but production won't. For example, in a real deployment the ranks of a single TP group may all reside on the same machine, so a single motherboard failure can take down the entire group at once.
Reviewer_VckQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Synchronization in distributed training propagates delays from a faulty rank to other ranks, so the most anomalous device need not be the source. This paper studies root-cause rank localization across models and TP/PP/DP topologies using training on normal data only, reducing reliance on scarce fault labels. It is a timely and important problem for LLM training. The main approach infers iteration boundaries from collective-communication events and detects anomalous segments with CUSUM. TopoGAT constructs a heterogeneous spatio-temporal graph containing ranks, TP/PP/DP relations, temporal edges, and global nodes, and predicts each rank's next-iteration deviation from its neighbors.

Critical strengths
  1. The paper clearly distinguishes anomaly detection from root-cause localization: synchronization can make healthy ranks highly anomalous, so raw anomaly magnitude is not a reliable attribution signal. The communication-event example in Figure 2 and the discussion of group-shared delays give the method a concrete systems motivation.
Critical weaknesses
  1. Although model weights are not retrained, §4.5 scales errors using target-task normal windows for each rank/metric. Transfer without weight updates should therefore be distinguished from transfer without target-task calibration. The paper should specify whether normal windows are selected by the online scanner or labels and whether inference uses only statistics available at the current time. The appendix describes one-time σ computation, whereas the main text describes dynamic maintenance; these deployment protocols need alignment.
  2. The 11 datasets primarily contain injected faults and do not directly establish performance on naturally occurring production incidents. Whole-group failures, shared switch/NIC causes, and multiple simultaneous causes warrant further study: median subtraction may suppress shared fault signals, and hitting one cause within Top-3 does not establish complete localization.
  3. Each window spans six training iterations, with iteration periods of roughly 15–124 seconds, and the pipeline also depends on event collection, boundary inference, anomaly detection, and EMA. The 10–29.4 ms forward time therefore should not be interpreted as fault-to-actionable-alert latency.
Questions for authors
  1. Why does losing one non-root-cause rank reduce offline Hit@3 from 94.6% to 49.4%? How are missing values encoded, are nodes and edges removed, and are normal baselines recomputed?
  2. Please precisely define bare/full protocols and event-level/dataset-average metrics, including how online windows aggregate into events. Please also reconcile Table 7's counts with precision/recall and explain the online tie with Minder in Table 17.
  3. How much normal data is needed to establish σ and normalization statistics for a new task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.