acceptodds
Under review as a conference paper at ICLR 2027

AlignDiag: Stage-Aware Multi-Source Diagnosis for Cross-Hardware Training Migration

Abstract

Large language model training increasingly spans accelerator generations and vendors as model development outpaces datacenter renewal. Common frameworks make such migration practical, yet backend-specific implementations can introduce legitimate numerical differences under finite-precision arithmetic. Meanwhile, similar losses may conceal internal faults whose effects propagate through subsequent computations. Reliable diagnosis must distinguish permissible drift from material errors and separate their sources from downstream effects. We introduce AlignDiag, the first staged multi-source diagnostic framework for training migration. It builds a contract-grounded evidence library that distinguishes noise, permitted drift, and controlled faults while preserving their provenance and lineage. Building on this structured evidence, a label-efficient procedure selects role-aware rules without using evaluated metrics to define labels. With these rules fixed, AlignDiag then detects and localizes material divergence automatically and uses targeted replay to verify whether identified candidates are genuine sources of observed deviations. Across our experiments, we organize 5,861 comparison pairs into 844 controlled groups covering expected variation and injected faults. Overall, AlignDiag achieves 98.61% accuracy distinguishing drift from faults. In prospective evaluations spanning training windows from 2 to 1,000 steps across GPU and NPU platforms, AlignDiag detects faults missed by loss-only checks, recovers injected source sets, and exposes sustained routing divergence in MoE models despite near-zero training loss. Multi-source localization reduces scans by 36.0% over iterative single-source strategies while preserving complete recovery.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Heterogeneous computing, Model training, Numerical accuracy, Fault diagnosis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Rg9nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AlignDiag, which checks training consistency across hardware via labeled comparisons and grouped data splits to select metrics and calibrate thresholds for different training stages. It then compares computations under the same inputs, replaces suspected sources with reference implementations, and rescans for remaining errors.

Critical strengths
  1. The calibration protocol is carefully designed. Labels are defined independently of the tested metrics, related samples stay together during data splitting, and rules are fixed before testing.

  2. The H200 and Ascend experiments provide useful cross-vendor evidence. They also offer a concrete lesson: a partial repair can improve one endpoint while worsening another. For example, fixing RMSNorm on the NPU reduces logit error but increases loss error to 3.11 times the baseline.

Critical weaknesses
  1. My main concern is the limited evidence from realistic migration workloads. Localization uses Qwen3-0.6B, four registered sites, two training steps, and injected precision changes. Recovering all 16 subsets is useful, but the search space is small. The long-run cases are also limited: MoE repeatedly trains on one sequence, while Dense shows strong overfitting, with validation NLL rising from about 3.3 to 7. Both run on H200. These examples show that loss can hide internal differences, but provide limited evidence for real cross-hardware debugging. The paper also reports no newly discovered real-world migration bugs.

  2. The method seems incremental. Same-input checks already appear in TTrace, numerical-error analysis in RENDER, and intervention-based reasoning in delta debugging. The baselines mainly compare individual metrics and simple search strategies, so the gain over complete systems such as TTrace and OpGuard remains unclear. Only offline CPU analysis is timed, leaving the actual debugging cost unmeasured.

  3. There are also presentation issues. The introduction confuses pair-weighted accuracy (99.07% and 94.17%) with group-balanced accuracy (91.10% and 90.17%; Table 2). Appendix D.1 and E.3 need better organization and explanation. The repeated CPU timing analysis could be shortened.

Questions for authors
  • What are the 17 fault mechanism categories, and how many cases come from real bugs instead of controlled injections?
  • Is there evidence that localization works with more candidate sites, nonzero normal numerical differences, or faults beyond the injected precision changes?
  • How does the full system compare with TTrace or OpGuard on supported workloads, including the cost of collecting evidence and rerunning training
3 comments

Reply to W1

Thank you for your comments on the experimental settings and practical use. AlignDiag supports developers who encounter training differences during cross-hardware migration by assessing whether those differences are abnormal, localizing their sources, and verifying repairs. In practice, an anomaly need not be a previously unknown bug to warrant diagnosis: known issues may also recur on new hardware or under new configurations. We therefore use controlled faults with known sources to objectively evaluate localization accuracy and completeness. The localization procedure itself does not access the injected fault locations or fault count. On H200 and Ascend NPU, we evaluate both the recovered source sets and their effects on outputs, loss, and parameter updates through joint repairs and repair reversals (§5.2, Figure 2). These results directly assess whether AlignDiag provides useful evidence for diagnosis and repair. Discovering previously unreported bugs would be a separate application outcome, beyond the diagnostic capabilities evaluated here.

The two long-run H200 case studies examine consistency between the reference and target training paths and how it evolves during training. We focus on whether migration preserves the same training computation requirements and keeps differences within permitted tolerances, rather than the model’s absolute predictive performance or generalization. Thank you for noting the overfitting in the Dense case. This trend appears in both paths and does not, by itself, indicate a migration-induced fault; addressing overfitting is also outside AlignDiag’s scope. Your feedback also highlights the need to make the comparison objectives clearer when presenting these cases, helping readers distinguish model training quality from migration consistency.

Reply to W2 & Q3

Thank you for this suggestion. We would like to clarify the research objectives and the scope of the comparisons.

AlignDiag addresses cross-hardware training migration with permitted numerical drift, focusing on stage-aware assessment, multi-source localization, and recovery verification. TTrace focuses on differential diagnosis between distributed execution and a trusted reference, while OpGuard uses bitwise alignment to locate the first mismatch. Applying their original systems directly to our setting would introduce additional factors, including backend support, execution alignment, and reference adaptation, making it difficult to compare the diagnostic methods fairly.

We were also unable to obtain runnable, complete official implementations, which prevented end-to-end reproduction and comparison. Implementing selected criteria or search procedures from their papers would not amount to reproducing the full systems.

We have conducted method-level comparisons. §5.3, Figure 3 compares representative numerical criteria, while Appendix Figure D.6 compares first-anomaly localization with iterative multi-source search. These experiments assess decision quality, source-set recovery, and scan overhead; they compare methodological components rather than rank the full TTrace and OpGuard systems. The reported CPU timings cover only offline analysis, while scan counts provide an additional measure of localization effort. Neither represents end-to-end runtime, including evidence collection and training reruns.

Reply to Q1

Thank you for encouraging us to clarify the categories and fault sources. The “17 mechanism categories” describe the model architectures and computational mechanisms covered by the evidence library, rather than 17 fault types. We will clarify this wording and add a category table to the appendix.

The controlled localization experiment includes 24 fault-injection test inputs and 8 normal test inputs, evaluated in pairs on H200 and NPU. All faults in this experiment were deliberately introduced by changing computation precision; they were not independently discovered software bugs. The 1,000-step case study likewise uses a controlled precision intervention.

The full evidence library also includes labeled cases involving anomalous gradients at masked positions and changes in the order of sparse Top-k selection and causal masking. These extend coverage beyond precision changes, but an abnormal label alone does not establish that a case represents a real-world bug discovered in practice.

Reply to Q2

Thank you for the question. AlignDiag localizes sources using local execution evidence and responses to interventions. The four candidate locations and precision injections are settings of this controlled test, rather than predefined limits on candidate locations or fault types. The localization procedure knows which candidate locations to inspect, but does not access the injected fault locations or fault count; this ground truth is used only for subsequent evaluation. The current experiment demonstrates complete source-set recovery across all 16 combinations at four locations (Appendix D.2). Labeled cases involving anomalous gradients at masked positions and changes in the order of Top-k selection and causal masking provide additional coverage beyond precision-related issues.

AlignDiag does not require identical results across hardware. It uses calibrated tolerances to distinguish permitted drift from anomalies. During localization, we evaluate the actual computation and a trusted reference on the same input, replace identified candidate sources with the reference computation, and rescan to assess their downstream effects. This does not require rewriting an operator for each candidate location. The zero local threshold is a calibration result for this controlled experiment, rather than a fixed requirement of the method. It also does not directly establish localization performance when normal local differences are nonzero. We will clarify the “same-input comparison—reference replacement—rescan” procedure in the flowchart and captions, distinguish the local experimental settings from permitted cross-hardware drift, and explain that AlignDiag supports localization and repair verification without requiring high precision or bitwise alignment.

AuthorsOpenReview ↗

Summary of Revisions

Thank you for your suggestions. We have revised the paper as follows:

Corrected the metric names in the introduction, identifying 99.07% and 94.17% as accuracies weighted by the number of comparison pairs.

Clarified that the long-run case studies examine consistency between the reference and target training paths, and that overfitting is outside the scope of this work.

Clarified the experimental settings involving four candidate locations and a zero local threshold, and explained the “same-input comparison—reference replacement—rescan” procedure in the localization flowchart caption.

Reorganized Appendices D.1 and E.3, adding guidance for interpreting the figures, tables specifying sample coverage, and explanations of the branches corresponding to each panel. Condensed repeated CPU timing descriptions while preserving the original results and timing conditions.

Reviewer_1vW7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AlignDiag, a diagnostic framework for identifying numerical discrepancies during cross-hardware LLM training migration. Its primary goal is to distinguish permissible numerical drift arising from hardware and backend implementation differences from actual training faults, and to localize multiple sources of errors. AlignDiag constructs an evidence library to calibrate stage-aware alignment criteria and employs intervention-based diagnosis to distinguish root causes from propagated errors.

Critical strengths
  • The paper addresses an important and practical challenge in cross-hardware LLM training migration by distinguishing acceptable numerical differences from actual training faults. The evaluation covers a broad evidence library of 844 experimental groups and 5,861 comparison pairs, and reports an overall diagnostic accuracy of 98.61% (91.90% balanced accuracy). In particular, the 1,000-step MoE experiment demonstrates the limitations of loss-only monitoring. Despite both training runs achieving near zero training loss, 95.32% of layer-token positions exhibit different Top 8 expert sets at step 1,000. This result effectively illustrates why internal training state monitoring can be valuable even when conventional loss metrics appear normal.
Critical weaknesses
  • The paper does not sufficiently clarify the relationship between its notion of "intended training semantics," permissible numerical drift, and actual training correctness. Although AlignDiag employs declared numerical contracts and empirically calibrated thresholds, it remains unclear what aspects of training correctness these criteria establish and how the acceptable numerical differences are justified.

  • LLM training relies on a relatively limited set of core operations, such as MatMul, Softmax, and RMSNorm. Previously validated operator-level numerical properties may be reusable across models and training configurations when execution conditions remain unchanged. Although the paper explores evidence reuse and target-specific calibration, its practical advantages over an incremental validation approach are not sufficiently established. Furthermore, while the evidence library spans 17 mechanism categories, the explicitly described fault-injection experiments primarily involve controlled precision changes, raising concerns about generalizability to more diverse training faults.

Questions for authors
  1. What are the precise definitions of "intended training semantics" and "permitted numerical drift" in AlignDiag? In particular, how are the numerical contracts and acceptable tolerance ranges determined and justified?

  2. What additional advantages does AlignDiag provide over an incremental validation approach that reuses previously validated operator-level numerical properties and checks only new or modified execution conditions? How much new evidence collection and calibration are required when models or training configurations change?

  3. What specific fault types are represented by the 17 mechanism categories? Beyond the described controlled precision changes (e.g., FP32 to BF16), how effectively can AlignDiag detect and localize realistic training faults such as incorrect operator implementations, synchronization errors, communication faults, or configuration-dependent bugs?

3 comments

Response to Q1 & W1

Thank you for your patience and helpful comments.

We first clarify two terms used in the paper. “Intended training semantics” refers to the computational requirements that must be preserved across hardware migration, including operator precision, gradient reduction procedures, and optimizer update rules. “Permitted numerical drift” refers to acceptable differences caused by finite-precision arithmetic and hardware-specific implementations that satisfy these same requirements, such as differences in reduction order or intermediate rounding. Bitwise agreement between executions is therefore not required. These requirements are specified in advance for each migration task, and samples are independently labeled against them. Calibration thresholds help detect deviations from the requirements; they do not define the requirements themselves.

We do not prescribe a single fixed tolerance. Instead, we build a sample library covering both normal variation and controlled faults, then use independently labeled samples to calibrate decision boundaries for different training stages. Normal variation includes, for example, differences in accumulation order or intermediate rounding that remain consistent with the prescribed reduction requirements.

Response to Q2 & W2

(1)

Thank you for this suggestion. We agree that verified operator properties can be reused when execution conditions remain unchanged, which is compatible with AlignDiag. Rewriting or replacing operators can also resolve some migration issues. To guide these changes, however, developers first need to determine which differences indicate actual problems, where they originate, and which operators require modification. AlignDiag supports this process by assessing differences and localizing their sources, helping developers make targeted repairs and reducing manual inspection and adjustment of individual operators. It can also help validate operators after modification.

The same operator may use different implementations across hardware, producing different finite-precision results even when the mathematical operation is identical. Consider 100000000 + (-100000000) + 1, whose exact result is 1. Adding the first two numbers first yields 0, followed by a final result of 1. Adding the last two numbers first in FP32 rounds -100000000 + 1 to −100000000, giving a final result of 0. Thus, evaluation order alone can change the result of the same summation. Parallel implementations across hardware may also use different evaluation orders. A difference in results therefore does not necessarily imply that an operator needs to be rewritten. Distinguishing permitted numerical variation from actual faults helps determine whether—and where—changes are needed.

(2)

For hardware pairs already covered by the sample library, such as H200/NPU, the library can be reused directly. For new hardware, calibration combines existing evidence with data from the target hardware. For example, adaptation to the RTX 4090 reused 44 existing evidence groups, added 30 groups for calibration, and was validated on another 50 groups. Forward and backward balanced accuracy reached 97.78% and 92.89%, respectively (§5.5, Table 4(b)).

In a separate forward-pass few-shot experiment, with the metrics and decision structure fixed, six labeled comparison pairs per fold were sufficient to match the performance of the full calibration dataset across all 20 sampling trials (§5.4, Figure 4). For model changes, we reused the selected metrics across Llama, OLMo, and Granite, while recalibrating the numerical parameters using training samples from each model (§5.5, Table 4(a)).

Response to Q3

The “17 mechanism categories” refer to the model architectures and computational mechanisms covered by the sample pairs, rather than 17 fault types. We will add the following table to the appendix to clarify these categories.

GroupCategories
Basic architecturesDense, Dense-GQA, MoE, Sparse
Other computational mechanismsGDN, KDA, Sparse-CSA
Hybrid and composite architecturesHybrid, KDA-GDN, GDN-Attention-Hybrid, GDN-MoE-Hybrid, KDA-MLA-MoE, MLA-MoE, Sparse-MoE
Other architectures and precision configurationsLow-Precision-Hybrid, MXFP4-MoE, Mixer-Supernet

Our focus is on diagnosing cross-hardware training differences while allowing reasonable numerical drift, rather than comparing numerical formats. For example, in the 1,000-step precision intervention experiment on Qwen3-30B-A3B, AlignDiag localized the source of the difference to the first layer’s router Softmax at step 1. By step 1,000, the two execution paths selected different Top-8 expert sets at 95.32% of layer–token positions, although both training losses were close to zero. This illustrates the value of early source localization and shows how monitoring training loss alone can miss differences in internal behavior (Appendix F, Figure F.1).

Beyond precision changes, the evidence library includes cases involving anomalous gradients at masked positions and changes in the order of sparse Top-k selection and causal masking, with corresponding normal and abnormal labels. These cases extend coverage to backward-pass implementation and configuration issues.

To localize sources, AlignDiag compares local computations under identical inputs, distinguishing locally generated differences from those propagated from upstream. It then rescans after removing the effects of the identified sources. In the controlled fault localization experiment on Qwen3-0.6B, we injected faults by changing computation precision at four locations spanning RMSNorm, RoPE, and Softmax, covering all 16 source combinations. On each of H200 and NPU, AlignDiag correctly identified the complete source set for all 24 abnormal test inputs, with no false positives on the eight normal test inputs (Appendix D.2, Figure D.2).

AuthorsOpenReview ↗

Summary of Revisions

Thank you for your suggestions. We have revised the paper as follows:

Clarified “intended training semantics” and “permitted numerical drift” within the existing discussion, specifying the training computation requirements that must be preserved.

Explained the basis for calibrating the forward and backward decision thresholds, including the roles of normal-variation and fault samples.

Clarified that the “17 mechanism categories” refer to model architectures and computational mechanisms, and added a category table to the appendix.

Reviewer_UJFdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores numerical divergence across different hardware types. The focus is on producing metrics that can help assess and diagnose divergence. The authors introduce AlignDiag to help triage the source of error, and seek to build an apples-to-apples comparison.

Critical strengths
  • Important problem being addressed
  • Metric-based approach is decent
Critical weaknesses
  • Missing some critical related work (LayerCast: https://arxiv.org/abs/2506.09501)
  • Discusses numerical divergence, but the paper does not go over any particular number format at all (BF16, FP8, MxFP, etc)
  • The metrics discussion was interesting, but it seems like the authors provided many metrics, and tried to figure out which ones worked and which didn't. I didn't really extract a methodical metrics approach, nor a key insight after reading the paper.
Questions for authors

Please address the points above in the Weaknesses. In particular, the lack of number formats evaluation is a big omission. For example, can layer cast address everything above for code portability (albeit at a performance hit)? Are there insights into which metrics work/are important, and which are just statistical covers?

2 comments
AuthorsOpenReview ↗

Clarifying Our Scope and Metric Selection Rationale

Reply to W1 & W2:

Thank you for your helpful comments and for pointing us to LayerCast. We appreciate your suggestions and would like to clarify the scope of our work.

We have carefully read the LayerCast paper. It examines how changes in hardware, execution configurations, and numerical precision affect the reproducibility of LLM inference. LayerCast mitigates numerical nondeterminism by retaining low-precision weight storage while performing computations in FP32. We will include this work in the numerical precision discussion under “Validation and diagnosis methods” in Related Work, clarifying its connection to and differences from our work.

We would like to clarify that the two works address different tasks. AlignDiag focuses on cross-hardware training migration. Its goal is not to make two executions as numerically similar as possible by increasing precision, but to determine whether discrepancies violate the intended training requirements while allowing permissible numerical drift, localize error sources, and verify recovery after intervention. Increasing precision can therefore mitigate some numerical differences, but does not directly provide the fault detection, source localization, and recovery verification addressed by our work.

Consistent with this distinction, we study discrepancies in cross-hardware training while allowing reasonable numerical drift, rather than comparing different numerical formats. Our experiments include BF16 and FP32. For example, our cross-hardware comparisons examine differences between BF16 execution on H200 and BF16 execution on an NPU, rather than differences between BF16 and FP32. Comparing numerical formats is therefore not the focus of this work.

Reply to W3 & Q:

Thank you for your thoughtful feedback on our metric selection.We would like to clarify the reasoning behind our choices.

As discussed in our response to W1 and W2, LayerCast does not address the diagnostic task we formulate. We further clarify below the rationale for selecting our metrics, beyond simply listing metrics and their performance.

Our metric selection starts from the questions that need to be checked during training: whether numerical differences exceed acceptable bounds, whether optimizer states evolve as expected, and whether parameter updates are applied correctly. We then select corresponding metrics and use independently labeled data to calibrate and validate the decision rules. The key idea is that different training stages require different checks, and correctness of a local computation does not by itself establish correctness of the entire training chain.

We do not assume that every metric makes an independent contribution to fault detection. For example, the lost-update metric identifies cases where the expected update is nonzero but the actual update is zero. These cases also constitute sign disagreements covered by our sign check. Under the current decision criteria, the lost-update metric therefore helps characterize the anomaly rather than providing an independent vote. Other supplementary checks improve detection coverage or suppress false positives; their roles should be interpreted alongside the specific decision rules and ablation results.

Our core contribution is a systematic method that connects stage-aware assessment, multi-source localization, and recovery verification, rather than a fixed set of metrics or thresholds. Our experiments demonstrate its effectiveness in the evaluated settings.

AuthorsOpenReview ↗

Summary of Revisions

Thank you for your suggestions. We have revised the paper as follows:

  1. Added a discussion and citation of LayerCast in Related Work, clarifying how its task and objectives differ from those of AlignDiag and emphasizing our focus on cross-hardware training diagnosis with permitted numerical drift.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.