- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the numerical mismatch that arises when the same LLM runs on an NVIDIA A100 and an AMD MI210. The authors characterize the mismatch with 5 families of output metrics and trace one source to the accumulation order inside matrix instructions. They further compare precision interventions and an output-head LoRA, and test whether the mismatch carries into knowledge distillation. The results show that FP32 upcasting reduces the logit error of the dense model at three times the runtime, that keeping only the MLPs in BF16 retains most of this gain, and that higher precision gives mixed results on MoE models.
Critical strengths
- The research problem is timely and practically important. As training and serving increasingly span heterogeneous hardware, whether the same model produces the same numbers on different vendors directly affects reproducibility and evaluation comparability.
- The motivation is clearly presented. In Section 2.2, the paper argues that a numerical difference matters through what downstream computation does with it. Fig. 3 then illustrates this with two constructed examples, where a one-ULP change either only reorders lower-ranked candidates or flips the leading token and leads to a wrong answer. This makes a convincing case that a single measure, such as bitwise equality or task accuracy, cannot show how much the mismatch matters.
- The analysis is systematic. The paper traces the mismatch through the whole stack, from instruction-level accumulation probes (Tables 1, 8) through layer-wise propagation (Fig. 2) and MoE routing to distilled students in Table 4.
- The experiments are carefully designed. The evaluation shows concrete disagreements between metrics. For example, FP32 upcasting on Llama lowers KL from 0.218 to 0.130 while halving exact rollout agreement in Table 2.
Critical weaknesses
-
The novelty is limited, as most findings are either well-known or expected. Prior work has already studied hardware-dependent numerical deviations in neural network inference [1], and recent work even uses these deviations to identify the hardware behind a model's outputs. The numerical behavior of NVIDIA and AMD matrix units has also been described in FTTN (cited). FP32 upcasting is the LayerCast method applied to a new setting, and both the FP16 overflow on a dense model and the removal of the mismatch under FP32 on both vendors are expected. The new parts are the MoE near-tie routing analysis, the native-baseline cost frontier and selective FP32, and the last two are not well supported (Weakness 5).
-
None of the interventions closes the gap in the paper's own setting. The paper keeps NVIDIA BF16 as a fixed reference and only changes the AMD side (Sec. 3). In this setting, full FP32 reduces only part of the gap for the dense model and makes the MoE models worse on logit metrics (Sec. 5.2), FP16 produces non-finite outputs on Jais, and the output-head LoRA increases the error in every configuration (Sec. 5.4). The only method that fully works, FP32 on both vendors, requires changing the reference, which is not possible in this setting. The suggestion in the conclusion to share the accumulation order across vendors is also not tested. Under a simple model with independent errors, a one-sided fix can reduce the MSE by at most about 50%, and the 42.8% on Jais is already close to this. Confirming such a limit, for example with an FP64 reference, would make the negative results more useful.
-
The multi-metric analysis shows that the metrics disagree, but it does not explain why or which metric to use. For example, FP32 on Llama reduces KL by about 40% but roughly halves exact rollout agreement, and the paper gives no explanation. The paper also does not match metrics to use cases, such as bitwise equality for reproducibility, KL for distillation, log-probability ratios for RL and top-K agreement for sampling. Some metrics overlap (logit MSE, probability MSE and KL), and task accuracy changes by less than 1 pp in every setting, so it is not informative here. Finally, the point that a single metric can be misleading has been made in prior work [2].
-
The experiments do not support the motivation. The paper motivates the problem with knowledge distillation and RL, but it has no RL experiment and does not measure log-probability ratios on sampled tokens, which is where the training-inference mismatch affects RL training. The only downstream experiment is offline distillation on fixed NVIDIA-generated responses (Sec. 5.5), with only 512 responses and 2 epochs (Table 6). On-policy settings, where the mismatch affects both sampling and scoring, are therefore not tested. The distillation result itself is also weak. The student is near chance that each condition is run only once, and changing the data-order seed changes more answers than changing the hardware (2,445 vs. 431). Moreover, the FP32 teacher is the closest to the NVIDIA teacher, yet its student drifts 15 times further, which suggests that the student is simply sensitive to any small change rather than to vendor differences. Together with the small changes in task accuracy (less than 1 pp in every setting), this means that the paper shows that the mismatch exists, but not that it harms any downstream use.
-
The selective FP32 result is not well supported. All 11 configurations are selected and evaluated on the same 200 development questions of Jais, without confidence intervals (Appendix E.3, Table 15), so it is unclear whether the result generalizes. The claim that "most of the cost of full upcasting buys little" only holds for runtime, since memory is still 2.02×, compared with 2.55× for full FP32. The cost comparison is also not consistent: 1.30× is forward time only, while 3.15× includes model loading, and full FP32 takes 3.02× forward time in Appendix E.3.
-
Some claims are overstated, and the presentation needs improvement. The claim that distributions move "most of the way to the reference" contradicts an actual KL reduction of only 37.1%. Additionally, the main tables lack confidence intervals, bit-equal rates for FP32/FP16 are inconsistently reported across tables (Table 2 vs. Tables 9 and 13), and Fig. 4 illustrates stepwise BF16 accumulation despite matrix instructions natively accumulating in FP32.
[1] Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks, NeurIPS 2023.
[2] Accuracy is Not All You Need, NeurIPS 2024.
Questions for authors
- Since the student model operates near random-chance accuracy on 4-choice MMLU (~26%, Table 4) and data ordering induces far greater variation than hardware (2,445 vs. 431 answers), can the authors demonstrate that the teacher mismatch has a genuine, statistically meaningful impact on a converged model or in an on-policy setting?
- The proposed module configuration was selected and tested exclusively on the same 200 development questions of Jais. Does this heuristic retain its ~90% numerical gain and runtime advantage on a held-out evaluation split or another dense model (such as Llama-3.1-8B)?
5 comments
Responses to Reviewer U951 Part (2/3): Metrics, RL and distillation, and selective FP32
Comment 3. (1) Why the metrics disagree, (2) which metric to use (Weakness 3)
3.1 The disagreement between metrics is a finding of the paper, and the main reason we report five families. KL measures the whole distribution while identical rollouts depend only on the winning token, so FP32 on Llama lowers KL by 40% yet identical responses fall from 16% to 9%. Part of this happens at exact ties in the BF16 reference, where greedy decoding picks the lower token index, and AMD BF16 often ties at the same positions while AMD FP32 has no tie there and can pick either token. Accuracy hides the mismatch even more, since 1.1% to 4.7% of answers change in both directions while accuracy barely moves (global official comment, N1)
3.2 Since no single family captures every effect, the choice follows from how the AMD outputs will be used (Table 2). Logit MSE weighs every vocabulary entry equally and KL mainly the likely tokens, which is why the two can move in opposite directions.
Table 2. Which metric to use
| Use | Metric |
|---|---|
| Reproducing a published result | Bit-equality, identical rollouts |
| Distillation and sampling | KL, top-k agreement |
| RL | Probability ratio on sampled tokens |
| Reported scores | Accuracy |
Comment 4. (1) RL training, (2) the probability ratio on sampled tokens, (3) distillation, (4) the distillation scale, (5) the data-order seed (Weakness 4, Question 1)
4.1 Training on the other vendor gives a different model. We ran the same GRPO training of Qwen2.5-0.5B on GSM8K on NVIDIA A100 and AMD MI210 GPUs, and after 60 steps the two models answer 42% to 44% of the test questions differently (global official comment, N1). We tested the same run on the same hardware and this is not happening, indicating that this is a cross-vendor issue.
4.2 The ratio on sampled tokens is comparable to the training-inference mismatch that RL methods already correct. Scoring the same sampled tokens on both vendors puts 0.53% of them outside PPO's clip range, against 0.85% between AMD's own sampling engine and trainer. Moving RL between vendors therefore adds a second source of the mismatch that methods such as truncated importance sampling target (Yao et al., 2025; Qi et al., 2025).
4.3 The distillation result holds with students well above chance. The OLMo-7B and Llama-3.2-1B students score 44% and 46% on MMLU, against 25% for chance, and moving the teacher to AMD changes 9.4% and 3.2% of their answers (global official comment, N1). The paper's 15x drift of the FP32 teacher's student disappears with four other data-order seeds, so we will correct this claim and the statement that the teacher's distance does not predict the student's, since on Llama-3.1-8B the closer teacher gives the closer student (Reviewer VYuC, 2.2).
4.4 Distillation scale. Each distillation run needs the teacher’s outputs transferred by hand from the NVIDIA machine to the AMD machine, which makes larger runs expensive and not viable for our capability. We expect a larger scale to increase the difference rather than reduce it, since the student trains longer on the shifted targets, so the small scale used here does not affect our conclusion.
4.5 Data-order seed. A different data-order seed changes more answers than the vendor, but a practitioner chooses and fixes the seed, while the GPU behind a shared cluster or cloud endpoint is often outside their control. We will post the seed comparison for the new distillation and RL runs here in the coming days.
Comment 5. (1) Selective FP32 on held-out data and another dense model, (2) its cost (Weakness 5, Question 2)
5.1 Selective FP32 keeps its gain on held-out data and on another dense model. On 1,200 held-out Jais responses, it lowers the logit MSE by 32.3% and KL by 33.5%, against 34.3% and 36.4% for full FP32. On Llama-3.1-8B, it lowers KL by 35.1% and logit MSE by 5.4% at 1.40x the forward time, while full FP32 lowers them by 40.1% and 11.9% at 2.50x (95% intervals within ±2 points).
5.2 The paper measures cost in two different ways. In the main-text cost table, the full FP32 row comes from the complete run on the full workload, while the selective FP32 row, like the appendix, gives forward time and peak memory on the development responses only. Measured the same way, selective FP32 still needs clearly less time and memory than full FP32, and we will report all costs this way.
Responses to Reviewer U951 Part (3/3): Presentation
Comment 6. (1) Wording, (2) Fig. 4, (3) intervals and bit-equality (Weakness 6)
6.1 Wording. "Most of the way to the reference" refers to the part of the gap that a change on the AMD side can remove, which full FP32 reaches (2.1). We will state this together in the camera-ready version with the measured reductions on the development set, 42.8% in MSE and 37.1% in KL.
6.2 Fig. 4. We will draw the FP32 accumulator explicitly in the camera-ready version, so the figure shows how each vendor's matrix instructions group and round products before they reach it (2.2). We will also add the BF16 additions in the residual stream, where our ablations on Jais find FP32 matters most.
6.3 Intervals and bit-equality. Every number in the main tables on camera-ready version will have an interval, with best and second-best entries marked only where the intervals separate them, and all tables will compare FP32 and FP16 outputs after rounding them to BF16.
Responses to Reviewer U951 Part (1/3): Novelty and the limit of one-sided precision
We thank the reviewer for the thorough review and the pointers to related work. Below we show how our study differs from prior work (Comment 1), why changing precision on AMD alone has a limit and where the mismatch arises (Comment 2), why the metrics disagree and which one to use (Comment 3), that the mismatch changes the results of RL training and distillation (Comment 4), and that selective FP32 keeps its gain on held-out data and on another model (Comment 5).
To support our argument, we will refer to the table that we wrote in the global official comment above.
Comment 1. (1) Novelty compared with prior work, (2) relation to deterministic inference, (3) the expected findings (Weakness 1)
1.1 Unlike prior work, our study compares NVIDIA and AMD on full language models and follows the difference into inference, distillation, and RL. Prior work compares GPUs of a single vendor (Yuan et al., 2025; Cooper et al., 2026) or describes the matrix units of both vendors without running a model (Li et al., 2024), as Table 1 shows. We will add Schlögl et al. (2023) and the platform-identification work to the related work.
Table 1. What each work compares and measures
1.2 Deterministic-inference work changes the reference execution itself, while our goal is to match the NVIDIA execution as it is. Batch-invariant kernels and fixed reduction orders make repeated runs on one vendor identical by replacing its kernels (He and Thinking Machines Lab, 2025; Zhang et al., 2025; Yang et al., 2026), which would change the NVIDIA outputs that practitioners already rely on.
1.3 Expected findings. We do not propose full FP32 upcasting, since, as in LayerCast, it lowers the mismatch at about three times the forward time and more than twice the peak memory, which is not realistic for practitioners to deploy. We use it only as a reference, and selective FP32 is the practical alternative at 1.30x the forward time.
Comment 2. (1) The limit of one-sided precision, (2) a shared accumulation order (Weakness 2)
2.1 FP32 on both vendors is only a diagnostic, and changing only the AMD side can remove part of the gap, but not all of it. Each BF16 output is the exact result plus its own rounding, and computing more precisely on AMD removes the AMD rounding but not the rounding in the NVIDIA reference. With AMD FP32 as the exact result, since it is within 1e-8 of FP64, full FP32 removes 42.8% of the logit MSE on the development set and 34.3% on 1,200 held-out responses, which is the most any precision change on AMD can remove.
2.2 The mismatch arises inside the matrix instructions. The kernel study in Appendix C.2 shows that a projection still differs on almost every output when both vendors run one kernel with the same tile order, and becomes identical when both add the products one by one in the same order. This is because the A100 adds BF16 products in blocks of 8 and rounds toward zero, while the MI210 adds them in groups of 2 or 4 and rounds to nearest (Fasi et al., 2021; Khattak and Mikaitis, 2025; Khattak, Mikaitis and Graziani, 2026). A shared accumulation order, as the paper suggests, would therefore remove this source of the mismatch, as fixed orders already make outputs identical across tensor-parallel sizes (Zhang et al., 2025) and NVIDIA generations (Cooper et al., 2026).