- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AlignDiag, which checks training consistency across hardware via labeled comparisons and grouped data splits to select metrics and calibrate thresholds for different training stages. It then compares computations under the same inputs, replaces suspected sources with reference implementations, and rescans for remaining errors.
Critical strengths
-
The calibration protocol is carefully designed. Labels are defined independently of the tested metrics, related samples stay together during data splitting, and rules are fixed before testing.
-
The H200 and Ascend experiments provide useful cross-vendor evidence. They also offer a concrete lesson: a partial repair can improve one endpoint while worsening another. For example, fixing RMSNorm on the NPU reduces logit error but increases loss error to 3.11 times the baseline.
Critical weaknesses
-
My main concern is the limited evidence from realistic migration workloads. Localization uses Qwen3-0.6B, four registered sites, two training steps, and injected precision changes. Recovering all 16 subsets is useful, but the search space is small. The long-run cases are also limited: MoE repeatedly trains on one sequence, while Dense shows strong overfitting, with validation NLL rising from about 3.3 to 7. Both run on H200. These examples show that loss can hide internal differences, but provide limited evidence for real cross-hardware debugging. The paper also reports no newly discovered real-world migration bugs.
-
The method seems incremental. Same-input checks already appear in TTrace, numerical-error analysis in RENDER, and intervention-based reasoning in delta debugging. The baselines mainly compare individual metrics and simple search strategies, so the gain over complete systems such as TTrace and OpGuard remains unclear. Only offline CPU analysis is timed, leaving the actual debugging cost unmeasured.
-
There are also presentation issues. The introduction confuses pair-weighted accuracy (99.07% and 94.17%) with group-balanced accuracy (91.10% and 90.17%; Table 2). Appendix D.1 and E.3 need better organization and explanation. The repeated CPU timing analysis could be shortened.
Questions for authors
- What are the 17 fault mechanism categories, and how many cases come from real bugs instead of controlled injections?
- Is there evidence that localization works with more candidate sites, nonzero normal numerical differences, or faults beyond the injected precision changes?
- How does the full system compare with TTrace or OpGuard on supported workloads, including the cost of collecting evidence and rerunning training
3 comments
Reply to Q1
Thank you for encouraging us to clarify the categories and fault sources. The “17 mechanism categories” describe the model architectures and computational mechanisms covered by the evidence library, rather than 17 fault types. We will clarify this wording and add a category table to the appendix.
The controlled localization experiment includes 24 fault-injection test inputs and 8 normal test inputs, evaluated in pairs on H200 and NPU. All faults in this experiment were deliberately introduced by changing computation precision; they were not independently discovered software bugs. The 1,000-step case study likewise uses a controlled precision intervention.
The full evidence library also includes labeled cases involving anomalous gradients at masked positions and changes in the order of sparse Top-k selection and causal masking. These extend coverage beyond precision changes, but an abnormal label alone does not establish that a case represents a real-world bug discovered in practice.
Reply to Q2
Thank you for the question. AlignDiag localizes sources using local execution evidence and responses to interventions. The four candidate locations and precision injections are settings of this controlled test, rather than predefined limits on candidate locations or fault types. The localization procedure knows which candidate locations to inspect, but does not access the injected fault locations or fault count; this ground truth is used only for subsequent evaluation. The current experiment demonstrates complete source-set recovery across all 16 combinations at four locations (Appendix D.2). Labeled cases involving anomalous gradients at masked positions and changes in the order of Top-k selection and causal masking provide additional coverage beyond precision-related issues.
AlignDiag does not require identical results across hardware. It uses calibrated tolerances to distinguish permitted drift from anomalies. During localization, we evaluate the actual computation and a trusted reference on the same input, replace identified candidate sources with the reference computation, and rescan to assess their downstream effects. This does not require rewriting an operator for each candidate location. The zero local threshold is a calibration result for this controlled experiment, rather than a fixed requirement of the method. It also does not directly establish localization performance when normal local differences are nonzero. We will clarify the “same-input comparison—reference replacement—rescan” procedure in the flowchart and captions, distinguish the local experimental settings from permitted cross-hardware drift, and explain that AlignDiag supports localization and repair verification without requiring high precision or bitwise alignment.
Summary of Revisions
Thank you for your suggestions. We have revised the paper as follows:
Corrected the metric names in the introduction, identifying 99.07% and 94.17% as accuracies weighted by the number of comparison pairs.
Clarified that the long-run case studies examine consistency between the reference and target training paths, and that overfitting is outside the scope of this work.
Clarified the experimental settings involving four candidate locations and a zero local threshold, and explained the “same-input comparison—reference replacement—rescan” procedure in the localization flowchart caption.
Reorganized Appendices D.1 and E.3, adding guidance for interpreting the figures, tables specifying sample coverage, and explanations of the branches corresponding to each panel. Condensed repeated CPU timing descriptions while preserving the original results and timing conditions.
Reply to W1
Thank you for your comments on the experimental settings and practical use. AlignDiag supports developers who encounter training differences during cross-hardware migration by assessing whether those differences are abnormal, localizing their sources, and verifying repairs. In practice, an anomaly need not be a previously unknown bug to warrant diagnosis: known issues may also recur on new hardware or under new configurations. We therefore use controlled faults with known sources to objectively evaluate localization accuracy and completeness. The localization procedure itself does not access the injected fault locations or fault count. On H200 and Ascend NPU, we evaluate both the recovered source sets and their effects on outputs, loss, and parameter updates through joint repairs and repair reversals (§5.2, Figure 2). These results directly assess whether AlignDiag provides useful evidence for diagnosis and repair. Discovering previously unreported bugs would be a separate application outcome, beyond the diagnostic capabilities evaluated here.
The two long-run H200 case studies examine consistency between the reference and target training paths and how it evolves during training. We focus on whether migration preserves the same training computation requirements and keeps differences within permitted tolerances, rather than the model’s absolute predictive performance or generalization. Thank you for noting the overfitting in the Dense case. This trend appears in both paths and does not, by itself, indicate a migration-induced fault; addressing overfitting is also outside AlignDiag’s scope. Your feedback also highlights the need to make the comparison objectives clearer when presenting these cases, helping readers distinguish model training quality from migration consistency.
Reply to W2 & Q3
Thank you for this suggestion. We would like to clarify the research objectives and the scope of the comparisons.
AlignDiag addresses cross-hardware training migration with permitted numerical drift, focusing on stage-aware assessment, multi-source localization, and recovery verification. TTrace focuses on differential diagnosis between distributed execution and a trusted reference, while OpGuard uses bitwise alignment to locate the first mismatch. Applying their original systems directly to our setting would introduce additional factors, including backend support, execution alignment, and reference adaptation, making it difficult to compare the diagnostic methods fairly.
We were also unable to obtain runnable, complete official implementations, which prevented end-to-end reproduction and comparison. Implementing selected criteria or search procedures from their papers would not amount to reproducing the full systems.
We have conducted method-level comparisons. §5.3, Figure 3 compares representative numerical criteria, while Appendix Figure D.6 compares first-anomaly localization with iterative multi-source search. These experiments assess decision quality, source-set recovery, and scan overhead; they compare methodological components rather than rank the full TTrace and OpGuard systems. The reported CPU timings cover only offline analysis, while scan counts provide an additional measure of localization effort. Neither represents end-to-end runtime, including evidence collection and training reruns.