acceptodds
Under review as a conference paper at ICLR 2027

Same Model, Different Numbers: Measuring and Reducing Cross-Vendor Mismatch in Language Models

Abstract

Running the same language model on different graphics processing unit (GPU) vendors can produce different logits, even when the model weights and inputs are the same. We analyze cross-vendor mismatch in two dense and two mixture-of-experts (MoE) models with five metric families, namely bitwise equality, logit differences, top-K consistency, token agreement, and task accuracy. We trace one source of the mismatch to accumulation order inside vendors' matrix instructions. Upcasting to FP32 reduces the dense model's logit error by 43% at three times the runtime, yet keeping only the MLPs in BF16 retains 94% of this gain at 1.3 times the runtime, so most of the cost of full upcasting buys little. In the MoE models, FP32 and FP16 both lower the probability error but raise the logit error and change expert selection, and FP16 fails in the dense model. An output-head low-rank adapter (LoRA) does not help either, since the final hidden state does not predict the mismatch. The mismatch also carries into training. With every seed fixed, a student distilled from a teacher running on AMD answers 431 MMLU questions differently from one distilled from the same teacher on NVIDIA. Under FP32 upcasting, bitwise equality barely changes while the output distributions move most of the way to the reference, so judging cross-vendor agreement by a single measure misreads both its cost and its gains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, cross-vendor numerical mismatch, numerical reproducibility, floating-point arithmetic, low-rank adaptation, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U951OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the numerical mismatch that arises when the same LLM runs on an NVIDIA A100 and an AMD MI210. The authors characterize the mismatch with 5 families of output metrics and trace one source to the accumulation order inside matrix instructions. They further compare precision interventions and an output-head LoRA, and test whether the mismatch carries into knowledge distillation. The results show that FP32 upcasting reduces the logit error of the dense model at three times the runtime, that keeping only the MLPs in BF16 retains most of this gain, and that higher precision gives mixed results on MoE models.

Critical strengths
  1. The research problem is timely and practically important. As training and serving increasingly span heterogeneous hardware, whether the same model produces the same numbers on different vendors directly affects reproducibility and evaluation comparability.
  2. The motivation is clearly presented. In Section 2.2, the paper argues that a numerical difference matters through what downstream computation does with it. Fig. 3 then illustrates this with two constructed examples, where a one-ULP change either only reorders lower-ranked candidates or flips the leading token and leads to a wrong answer. This makes a convincing case that a single measure, such as bitwise equality or task accuracy, cannot show how much the mismatch matters.
  3. The analysis is systematic. The paper traces the mismatch through the whole stack, from instruction-level accumulation probes (Tables 1, 8) through layer-wise propagation (Fig. 2) and MoE routing to distilled students in Table 4.
  4. The experiments are carefully designed. The evaluation shows concrete disagreements between metrics. For example, FP32 upcasting on Llama lowers KL from 0.218 to 0.130 while halving exact rollout agreement in Table 2.
Critical weaknesses
  1. The novelty is limited, as most findings are either well-known or expected. Prior work has already studied hardware-dependent numerical deviations in neural network inference [1], and recent work even uses these deviations to identify the hardware behind a model's outputs. The numerical behavior of NVIDIA and AMD matrix units has also been described in FTTN (cited). FP32 upcasting is the LayerCast method applied to a new setting, and both the FP16 overflow on a dense model and the removal of the mismatch under FP32 on both vendors are expected. The new parts are the MoE near-tie routing analysis, the native-baseline cost frontier and selective FP32, and the last two are not well supported (Weakness 5).

  2. None of the interventions closes the gap in the paper's own setting. The paper keeps NVIDIA BF16 as a fixed reference and only changes the AMD side (Sec. 3). In this setting, full FP32 reduces only part of the gap for the dense model and makes the MoE models worse on logit metrics (Sec. 5.2), FP16 produces non-finite outputs on Jais, and the output-head LoRA increases the error in every configuration (Sec. 5.4). The only method that fully works, FP32 on both vendors, requires changing the reference, which is not possible in this setting. The suggestion in the conclusion to share the accumulation order across vendors is also not tested. Under a simple model with independent errors, a one-sided fix can reduce the MSE by at most about 50%, and the 42.8% on Jais is already close to this. Confirming such a limit, for example with an FP64 reference, would make the negative results more useful.

  3. The multi-metric analysis shows that the metrics disagree, but it does not explain why or which metric to use. For example, FP32 on Llama reduces KL by about 40% but roughly halves exact rollout agreement, and the paper gives no explanation. The paper also does not match metrics to use cases, such as bitwise equality for reproducibility, KL for distillation, log-probability ratios for RL and top-K agreement for sampling. Some metrics overlap (logit MSE, probability MSE and KL), and task accuracy changes by less than 1 pp in every setting, so it is not informative here. Finally, the point that a single metric can be misleading has been made in prior work [2].

  4. The experiments do not support the motivation. The paper motivates the problem with knowledge distillation and RL, but it has no RL experiment and does not measure log-probability ratios on sampled tokens, which is where the training-inference mismatch affects RL training. The only downstream experiment is offline distillation on fixed NVIDIA-generated responses (Sec. 5.5), with only 512 responses and 2 epochs (Table 6). On-policy settings, where the mismatch affects both sampling and scoring, are therefore not tested. The distillation result itself is also weak. The student is near chance that each condition is run only once, and changing the data-order seed changes more answers than changing the hardware (2,445 vs. 431). Moreover, the FP32 teacher is the closest to the NVIDIA teacher, yet its student drifts 15 times further, which suggests that the student is simply sensitive to any small change rather than to vendor differences. Together with the small changes in task accuracy (less than 1 pp in every setting), this means that the paper shows that the mismatch exists, but not that it harms any downstream use.

  5. The selective FP32 result is not well supported. All 11 configurations are selected and evaluated on the same 200 development questions of Jais, without confidence intervals (Appendix E.3, Table 15), so it is unclear whether the result generalizes. The claim that "most of the cost of full upcasting buys little" only holds for runtime, since memory is still 2.02×, compared with 2.55× for full FP32. The cost comparison is also not consistent: 1.30× is forward time only, while 3.15× includes model loading, and full FP32 takes 3.02× forward time in Appendix E.3.

  6. Some claims are overstated, and the presentation needs improvement. The claim that distributions move "most of the way to the reference" contradicts an actual KL reduction of only 37.1%. Additionally, the main tables lack confidence intervals, bit-equal rates for FP32/FP16 are inconsistently reported across tables (Table 2 vs. Tables 9 and 13), and Fig. 4 illustrates stepwise BF16 accumulation despite matrix instructions natively accumulating in FP32.

[1] Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks, NeurIPS 2023.

[2] Accuracy is Not All You Need, NeurIPS 2024.

Questions for authors
  1. Since the student model operates near random-chance accuracy on 4-choice MMLU (~26%, Table 4) and data ordering induces far greater variation than hardware (2,445 vs. 431 answers), can the authors demonstrate that the teacher mismatch has a genuine, statistically meaningful impact on a converged model or in an on-policy setting?
  2. The proposed module configuration was selected and tested exclusively on the same 200 development questions of Jais. Does this heuristic retain its ~90% numerical gain and runtime advantage on a held-out evaluation split or another dense model (such as Llama-3.1-8B)?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer U951 Part (1/3): Novelty and the limit of one-sided precision

We thank the reviewer for the thorough review and the pointers to related work. Below we show how our study differs from prior work (Comment 1), why changing precision on AMD alone has a limit and where the mismatch arises (Comment 2), why the metrics disagree and which one to use (Comment 3), that the mismatch changes the results of RL training and distillation (Comment 4), and that selective FP32 keeps its gain on held-out data and on another model (Comment 5).

To support our argument, we will refer to the table that we wrote in the global official comment above.

Comment 1. (1) Novelty compared with prior work, (2) relation to deterministic inference, (3) the expected findings (Weakness 1)

1.1 Unlike prior work, our study compares NVIDIA and AMD on full language models and follows the difference into inference, distillation, and RL. Prior work compares GPUs of a single vendor (Yuan et al., 2025; Cooper et al., 2026) or describes the matrix units of both vendors without running a model (Li et al., 2024), as Table 1 shows. We will add Schlögl et al. (2023) and the platform-identification work to the related work.

Table 1. What each work compares and measures

WorkHardware comparedModelsMeasured
Schlögl et al., 2023CPUs and NVIDIA GPUsImage classifiersOutput deviations
Li et al., 2024 (FTTN)NVIDIA and AMD matrix unitsKernels onlyRounding and accumulation
Yuan et al., 2025NVIDIA onlyLLMsAccuracy
Cooper et al., 2026NVIDIA onlyLLMsToken divergence
This workNVIDIA and AMD GPUsDense and MoE LLMsOutputs, routing, distillation, RL

1.2 Deterministic-inference work changes the reference execution itself, while our goal is to match the NVIDIA execution as it is. Batch-invariant kernels and fixed reduction orders make repeated runs on one vendor identical by replacing its kernels (He and Thinking Machines Lab, 2025; Zhang et al., 2025; Yang et al., 2026), which would change the NVIDIA outputs that practitioners already rely on.

1.3 Expected findings. We do not propose full FP32 upcasting, since, as in LayerCast, it lowers the mismatch at about three times the forward time and more than twice the peak memory, which is not realistic for practitioners to deploy. We use it only as a reference, and selective FP32 is the practical alternative at 1.30x the forward time.

Comment 2. (1) The limit of one-sided precision, (2) a shared accumulation order (Weakness 2)

2.1 FP32 on both vendors is only a diagnostic, and changing only the AMD side can remove part of the gap, but not all of it. Each BF16 output is the exact result plus its own rounding, and computing more precisely on AMD removes the AMD rounding but not the rounding in the NVIDIA reference. With AMD FP32 as the exact result, since it is within 1e-8 of FP64, full FP32 removes 42.8% of the logit MSE on the development set and 34.3% on 1,200 held-out responses, which is the most any precision change on AMD can remove.

2.2 The mismatch arises inside the matrix instructions. The kernel study in Appendix C.2 shows that a projection still differs on almost every output when both vendors run one kernel with the same tile order, and becomes identical when both add the products one by one in the same order. This is because the A100 adds BF16 products in blocks of 8 and rounds toward zero, while the MI210 adds them in groups of 2 or 4 and rounds to nearest (Fasi et al., 2021; Khattak and Mikaitis, 2025; Khattak, Mikaitis and Graziani, 2026). A shared accumulation order, as the paper suggests, would therefore remove this source of the mismatch, as fixed orders already make outputs identical across tensor-parallel sizes (Zhang et al., 2025) and NVIDIA generations (Cooper et al., 2026).

AuthorsOpenReview ↗

Responses to Reviewer U951 Part (2/3): Metrics, RL and distillation, and selective FP32

Comment 3. (1) Why the metrics disagree, (2) which metric to use (Weakness 3)

3.1 The disagreement between metrics is a finding of the paper, and the main reason we report five families. KL measures the whole distribution while identical rollouts depend only on the winning token, so FP32 on Llama lowers KL by 40% yet identical responses fall from 16% to 9%. Part of this happens at exact ties in the BF16 reference, where greedy decoding picks the lower token index, and AMD BF16 often ties at the same positions while AMD FP32 has no tie there and can pick either token. Accuracy hides the mismatch even more, since 1.1% to 4.7% of answers change in both directions while accuracy barely moves (global official comment, N1)

3.2 Since no single family captures every effect, the choice follows from how the AMD outputs will be used (Table 2). Logit MSE weighs every vocabulary entry equally and KL mainly the likely tokens, which is why the two can move in opposite directions.

Table 2. Which metric to use

UseMetric
Reproducing a published resultBit-equality, identical rollouts
Distillation and samplingKL, top-k agreement
RLProbability ratio on sampled tokens
Reported scoresAccuracy

Comment 4. (1) RL training, (2) the probability ratio on sampled tokens, (3) distillation, (4) the distillation scale, (5) the data-order seed (Weakness 4, Question 1)

4.1 Training on the other vendor gives a different model. We ran the same GRPO training of Qwen2.5-0.5B on GSM8K on NVIDIA A100 and AMD MI210 GPUs, and after 60 steps the two models answer 42% to 44% of the test questions differently (global official comment, N1). We tested the same run on the same hardware and this is not happening, indicating that this is a cross-vendor issue.

4.2 The ratio on sampled tokens is comparable to the training-inference mismatch that RL methods already correct. Scoring the same sampled tokens on both vendors puts 0.53% of them outside PPO's clip range, against 0.85% between AMD's own sampling engine and trainer. Moving RL between vendors therefore adds a second source of the mismatch that methods such as truncated importance sampling target (Yao et al., 2025; Qi et al., 2025).

4.3 The distillation result holds with students well above chance. The OLMo-7B and Llama-3.2-1B students score 44% and 46% on MMLU, against 25% for chance, and moving the teacher to AMD changes 9.4% and 3.2% of their answers (global official comment, N1). The paper's 15x drift of the FP32 teacher's student disappears with four other data-order seeds, so we will correct this claim and the statement that the teacher's distance does not predict the student's, since on Llama-3.1-8B the closer teacher gives the closer student (Reviewer VYuC, 2.2).

4.4 Distillation scale. Each distillation run needs the teacher’s outputs transferred by hand from the NVIDIA machine to the AMD machine, which makes larger runs expensive and not viable for our capability. We expect a larger scale to increase the difference rather than reduce it, since the student trains longer on the shifted targets, so the small scale used here does not affect our conclusion.

4.5 Data-order seed. A different data-order seed changes more answers than the vendor, but a practitioner chooses and fixes the seed, while the GPU behind a shared cluster or cloud endpoint is often outside their control. We will post the seed comparison for the new distillation and RL runs here in the coming days.

Comment 5. (1) Selective FP32 on held-out data and another dense model, (2) its cost (Weakness 5, Question 2)

5.1 Selective FP32 keeps its gain on held-out data and on another dense model. On 1,200 held-out Jais responses, it lowers the logit MSE by 32.3% and KL by 33.5%, against 34.3% and 36.4% for full FP32. On Llama-3.1-8B, it lowers KL by 35.1% and logit MSE by 5.4% at 1.40x the forward time, while full FP32 lowers them by 40.1% and 11.9% at 2.50x (95% intervals within ±2 points).

5.2 The paper measures cost in two different ways. In the main-text cost table, the full FP32 row comes from the complete run on the full workload, while the selective FP32 row, like the appendix, gives forward time and peak memory on the development responses only. Measured the same way, selective FP32 still needs clearly less time and memory than full FP32, and we will report all costs this way.

AuthorsOpenReview ↗

Responses to Reviewer U951 Part (3/3): Presentation

Comment 6. (1) Wording, (2) Fig. 4, (3) intervals and bit-equality (Weakness 6)

6.1 Wording. "Most of the way to the reference" refers to the part of the gap that a change on the AMD side can remove, which full FP32 reaches (2.1). We will state this together in the camera-ready version with the measured reductions on the development set, 42.8% in MSE and 37.1% in KL.

6.2 Fig. 4. We will draw the FP32 accumulator explicitly in the camera-ready version, so the figure shows how each vendor's matrix instructions group and round products before they reach it (2.2). We will also add the BF16 additions in the residual stream, where our ablations on Jais find FP32 matters most.

6.3 Intervals and bit-equality. Every number in the main tables on camera-ready version will have an interval, with best and second-best entries marked only where the intervals separate them, and all tables will compare FP32 and FP16 outputs after rounding them to BF16.

Reviewer_fwxaOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the differences that arise when running the same model on GPUs from different vendors (NVIDIA vs. AMD), investigates the sources of these discrepancies, and proposes methods to mitigate them. Specifically, the authors focus on NVIDIA’s A100 and AMD’s MI210 GPUs. They observe significant divergences in logits outputs between the two cards and, through block-wise matrix experiments, demonstrate that these differences stem from hardware-level variations in instruction accumulation mechanisms. Furthermore, the authors explore several remediation strategies and examine the impact of such discrepancies on downstream tasks, such as knowledge distillation.

Critical strengths
  1. I think the problem addressed is both real and underestimated. "Cross-vendor reproducibility" is a practical challenge in deployment and distributed training. Existing works remain confined within single-vendor ecosystems, leaving a gap in cross-vendor system measurements. Therefore, this topic holds clear and significant value.
  2. The experimental evidence chain for root-cause analysis is solid. The three-layer investigation in Table 1, combined with the instruction-level probing in Appendix C.2, precisely pinpoints the source of discrepancies to the instruction level. This delves into the hardware domain, and I believe the research depth of this paper surpasses that of many typical deep learning papers.
  3. The authors explore several remediation strategies in the paper and transparently report their limitations, along with an analysis of the underlying bottlenecks. This honesty provides significant reference value to the community.
Critical weaknesses

I think the biggest limitation of this paper is that the scale of the discussion is somewhat insufficient. This is mainly reflected in two aspects:

  1. The authors only compared one specific model from NVIDIA with one from AMD. It remains unclear whether similar issues exist for other GPU models (such as NVIDIA’s H200 or B300), and whether there is a discernible pattern regarding the severity of these discrepancies across different hardware generations.

  2. The scale of the downstream distillation experiments is also relatively limited. The results are demonstrated using only a single pair of models. It is yet to be verified whether this phenomenon can be generalized to other more mainstream models (such as Qwen).

Questions for authors

To be honest, I am not an expert in this specific field, but I consider this to be an excellent paper. The only issue I can identify relates to the scale of the experiments, which I find somewhat conservative. Therefore, I am curious whether similar issues would persist across a wider range of GPU models and downstream tasks.

2 comments
AuthorsOpenReview ↗

Responses to Reviewer fwxa: Other GPUs, hardware generations, and larger downstream experiments

We thank the reviewer for the encouraging assessment and for pointing to the scale of the study as its main limitation. Below we show why the mismatch appears on every GPU pair whose matrix units differ and that it already appears between NVIDIA generations (Comment 1), and that it persists across more models and in RL training (Comment 2).

To support our argument, we will refer to the table that we wrote in the global official comment above.

Comment 1. (1) Other GPUs, (2) hardware generations (Weakness 1, Question)

1.1 The mismatch comes from how each vendor designs its matrix units, so it appears on every pair whose units differ. Bit-accurate models of the matrix units show that each generation changes how many products are added in one step (Table 1), and that NVIDIA rounds this sum toward zero while the MI210 rounds to nearest. We chose the A100 and the MI210 because they belong to the same generation, so the comparison isolates the vendor rather than the generation.

Table 1. BF16 products added in one step of the matrix unit (Li et al., 2024; Fasi et al., 2021; Khattak and Mikaitis, 2025; Khattak, Mikaitis and Graziani, 2026)

GPUProducts added in one step
NVIDIA A1008
NVIDIA H100, B20016
AMD MI2102-4
AMD MI3008

1.2 The same mechanism already separates NVIDIA generations, with a mismatch of the same size. Cooper et al. (2026) find that BF16 outputs diverge between NVIDIA's A100, L40S and H100 on 31% to 100% of problems. We measured an H100 against our A100 reference with the paper's metrics (global official comment, N2). The H100’s logit MSE to the A100 is 0.83x to 1.04x that of the MI210, and as across vendors, 90% to 100% of its token changes occur at near-ties of the reference.

Comment 2. Downstream experiments at a larger scale (Weakness 2)

2.1 The downstream results now cover three teacher-student pairs and cross-vendor RL training of a Qwen model. We added an OLMo-7B student well above chance, a Llama-3.1-8B teacher with a Llama-3.2-1B student, and GRPO training of Qwen2.5-0.5B on NVIDIA A100 and AMD MI210 GPUs (global official comment, N1). In each, moving to AMD changes 3% to 44% of the answers. We tested the same run on the same hardware and this is not happening, indicating that this is a cross-vendor issue.

Dear Reviewer fwxa,

This is a kind reminder that the discussion period is approaching its end. We would appreciate it if you could let us know whether our response addresses your concerns. Should you have any further questions, we are more than happy to provide clarifications.

Reviewer_VYuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper characterizes numerical differences when identical language-model checkpoints and inputs are executed on NVIDIA A100 and AMD MI210 GPUs. It evaluates two dense and two MoE models using numerical, ranking, generation, accuracy, and routing metrics. A matrix-multiplication case study illustrates instruction-level accumulation differences. The paper also evaluates precision changes and output-head LoRA: selective FP32 retains most of the full-upcasting improvement on Jais, while output-head adapters fail to improve agreement. Finally, distillation experiments show that changing the teacher’s execution environment can change student outputs under otherwise fixed training conditions. Most precision experiments measure fidelity to a fixed NVIDIA BF16 reference; an additional matched-precision experiment shows substantially higher cross-vendor agreement when both executions use FP32.

Critical strengths

The controlled comparisons lend credibility to the reported measurements. In particular, same-device repeats, deterministic distillation repeats, the alternative data-order seed, and matched-precision comparisons help separate cross-vendor execution differences from ordinary run-to-run variation. Evaluating both numerical agreement and task accuracy also avoids treating these as interchangeable properties.

Critical weaknesses
  1. Limited substantive insight beyond documenting numerical differences.

For readers familiar with numerical validation across GPU stacks, the central observations largely confirm familiar consequences of finite-precision arithmetic. Numerical mismatch alone does not establish that an execution is incorrect or operationally unacceptable. The paper does not sufficiently establish how its logit, ranking, or routing metrics inform judgments about consequential downstream effects. The distillation experiment demonstrates sensitivity to changed teacher targets, but the student is near chance on MMLU, accuracy changes are small, and an alternative data-order seed changes substantially more answers than the hardware change. This is a valid existence result, but it provides limited additional guidance for assessing the significance of cross-hardware differences. I therefore found little new actionable understanding despite the breadth of measurements.

  1. The interventions provide limited additional methodological or practical value.

Full upcasting follows an existing approach, and the main constructive extension—selective FP32—is validated only on Jais under a batch-size-one, eager-attention, non-cached evaluation protocol. Its improvement in reference fidelity is not connected to a demonstrated downstream learning benefit, while it still requires approximately twice the peak memory. Output-head LoRA is a reasonable diagnostic of a restricted correction hypothesis, but its failure, even with the accompanying linear-fit analysis, offers limited insight into how to address the underlying mismatch. Overall, the paper does not establish a sufficiently general mitigation or a validated basis for deciding when the measured mismatch warrants intervention.

Questions for authors
  1. Interpretation and practical significance: What decision about cross-vendor execution can a practitioner make more reliably using the proposed measurements? Is there evidence, within the studied setting, connecting these metrics to consequential downstream effects rather than simply detecting disagreement? This would materially affect my assessment of informativeness.

  2. Contribution of the interventions: Beyond reproducing a fixed BF16 reference more closely, what validated benefit does selective FP32 provide, and does its cost–fidelity tradeoff transfer to another model? For output-head LoRA, what generalizable conclusion do the authors draw from the negative result beyond the failure of the tested linear correction family?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer VYuC: Practical decisions, selective FP32, and the LoRA result

We thank the reviewer for the careful review and for asking what the mismatch means in practice. Below we show when our measurements tell a practitioner that AMD is reliable and how they connect to downstream effects in distillation and RL (Comment 1), how selective FP32 behaves under serving settings and on another model (Comment 2), and what the output-head LoRA result shows (Comment 3).

To support our argument, we will refer to the table that we wrote in the global official comment above.

Comment 1. (1) What a practitioner can decide, (2) the downstream evidence (Question 1, Weakness 1)

1.1 Our measurements tell a practitioner whether AMD is reliable for a given use, which accuracy alone cannot. Most inference and training results are computed on NVIDIA GPUs, so NVIDIA is the reference a practitioner relies on, and the question when moving to AMD is whether its outputs can be trusted for the same purpose. Accuracy alone would say yes, since it barely moves, yet 1% to 5% of individual benchmark answers change (global official comment, N1). Each metric family has their own purpose: bit-equality and identical rollouts for reproducing outputs, logit differences and top-k agreement for distillation and sampling, and accuracy for benchmark scores.

1.2 The metrics connect to downstream effects. A logit difference changes the generated token only where the reference's top two tokens are nearly tied, and almost every token change between vendors happens at such a position (global official comment, N3). Once one token changes, the rest of the response differs, which is how a small logit difference becomes a different answer. In training the effect is larger. After 60 RL steps, NVIDIA-trained and AMD-trained models answer 42% to 44% of test questions differently. Distilled students also answer 3% to 9% of their answers differently. We tested the same run on the same hardware and this is not happening, indicating that this is a cross-vendor issue (global official comment, N1). Logits also has an effect on this, since across three distillation teachers, the teacher closer to NVIDIA in KL gives the student closer to the NVIDIA-teacher student (Comment 2). A different data-order seed may introduce worse result, but the focus of our work is when both are in deterministic setup. We will post the seed comparison for the new distillation and RL runs here in the coming days.

Comment 2. (1) Selective FP32 under serving settings, (2) its downstream benefit, (3) its transfer to another dense model, (4) a mixture-of-experts model (Weakness 2, Question 2)

2.1 Selective FP32 keeps most of the gain of full FP32 at a fraction of its cost + under some serving settings. Selective FP32 computes every module except the MLPs in FP32. We repeated it on Jais with batch sizes 1 to 32, fused attention (SDPA) and KV-cache decoding (global official comment, N4). With batch sizes 1 to 32 and SDPA, selective FP32 lowers the logit MSE by 12% to 14%, against 13% to 15% for full FP32, at 1.16 to 1.51x the time of native BF16. Under KV-cache decoding, 92.5% to 93% of its greedy rollouts stay identical to NVIDIA, against 92% to 93% for native BF16, at 1.16 to 1.48x the prefill time, while decoding takes 0.76 to 1.12x the time.

2.2 A teacher closer to NVIDIA also gives a student closer to NVIDIA. We distilled Llama-3.2-1B from Llama-3.1-8B on AMD with three teachers. The selective-FP32 teacher gives a student with 5.0% lower KL to the student of the NVIDIA teacher than the native BF16 teacher, close to the 5.7% of the full-FP32 teacher.

2.3 Selective FP32 transfers to another dense model. To test whether the tradeoff holds beyond Jais, we applied selective FP32 to Llama-3.1-8B with the same modules in FP32 (global official comment, N4). It lowers KL by 35.1% and logit MSE by 5.4% at 1.40x the time and 1.46x the peak memory of native BF16, while full FP32 lowers them by 40.1% and 11.9% at 2.50x and 2.76x.

2.4 On OLMoE, higher precision on AMD alone does not help. Selective FP32 raises the logit MSE to NVIDIA by 12.0% and lowers KL by only 2.8%, as full FP32 does in the paper (global official comment, N4). We think this happens because the reference is computed in BF16, and in a mixture-of-experts model FP32 picks different experts than BF16 does. With FP32 on both vendors, 98.4% of OLMoE responses become identical.

Comment 3. What the output-head LoRA result shows (Question 2)

3.1 No correction applied at the output can remove the mismatch, because the difference spreads over far more directions than the logits. A rank-32 LoRA on the head can only add a correction within 32 directions. A singular value decomposition shows that the logits need only 4 to 8 directions, while the cross-vendor difference needs 954 to 1,592 (global official comment, N5). A fix therefore has to act inside the matrix multiplications, where the difference arises, as selective FP32 does (Comment 2).

Dear Reviewer VYuC,

This is a kind reminder that the discussion period is approaching its end. We would appreciate it if you could let us know whether our response addresses your concerns. Should you have any further questions, we are more than happy to provide clarifications.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Highlight of Responses to Reviewers Part (1/2): Main points and new experiment N1

We thank the reviewers for their careful reading and appreciate all the feedback. We grouped their points into five main topics, and the final version of the paper will include every result and clarification below. In each reviewer's reply, paragraphs are numbered by comment, so 2.1 is the first paragraph of Comment 2.

Cross-vendor mismatch is a practical problem that no current method removes. Practitioners rely on results computed on NVIDIA GPUs, in inference as well as in training. Recent work treats the same kind of variation within NVIDIA hardware as a problem worth removing (Yuan et al., 2025; He and Thinking Machines Lab, 2025; Yang et al., 2026). To our knowledge, this is the first study of what changes when the same language model moves from NVIDIA to AMD. Both NVIDIA and AMD round to BF16 in their own way, and the NVIDIA reference already contains its own BF16 rounding. A change on the AMD side therefore has to match the NVIDIA rounding, not only reduce its own.

We focus on the gap between GPU vendors, from NVIDIA to AMD (U951). Prior work shows numerical deviations in matrix kernels across GPU vendors (Li et al., 2024), in image classifiers across CPUs and NVIDIA GPUs (Schlögl et al., 2023), and in language models within NVIDIA hardware (Yuan et al., 2025; Cooper et al., 2026). We measure what these deviations change when a language model moves between GPU vendors, from bit-equality to routing, rollouts and training.

Results of downstream tasks differ between the two vendors under the same data and training setup (VYuC, U951, fwxa). Training the same model with RL on NVIDIA and on AMD gives models that answer many test questions differently, and even evaluating one trained model on the other vendor changes its answers. We also observe that distillation and plain inference change answers too (N1). We tested the same run on the same hardware and this is not happening, indicating that this is a cross-vendor issue.

Selective FP32 gives similar alignment results to NVIDIA compared to Full FP32 while saving time and memory (VYuC, U951). It computes everything except the MLPs in FP32, so it runs much faster and uses less memory than full FP32 (LayerCast) while keeping most of its gain, also under batching, KV-cache decoding and on another model (N4).

The cross-vendor mismatch happens because of the underlying operation of each vendor and it is not a specific architecture issue (U951, VYuC, fwxa). The mismatch comes from how each vendor's matrix instructions group and round products. In the paper's kernel study, a matrix multiplication from the model gives identical outputs on both vendors once the products are added one by one in the same order. Neither higher precision on the AMD side nor a learned correction at the output can remove it. This is because the BF16 rounding in the NVIDIA reference is itself part of the gap, and the final hidden state that methods such as LoRA read does not predict the difference (N5). The mismatch is also not limited to one pair of GPUs, as an H100 differs from the A100 reference about as much as the MI210 does (N2).

Five metrics are necessary because a single metric is flawed and does not measure the entire picture (U951). FP32 on Llama lowers KL by 40%, but the share of responses identical to NVIDIA’s falls from 16% to 9%. Accuracy hides the mismatch because answers change in both directions where moving from NVIDIA to AMD instead can improve the performance or reduce the performance. Moving to AMD changes 1.1% to 4.7% of benchmark answers.

New experiments in the rebuttal

We thank the reviewer for suggesting experiments that make our finding more solid. We gather all the new experiments that we did during review period and will add it in the final print of the paper.

N1. Moving to AMD changes downstream results (VYuC, U951, fwxa). We ran the same GRPO training of Qwen2.5-0.5B on GSM8K on both vendors, distilled an OLMo-7B and a Llama-3.2-1B student from teachers run on each vendor, and evaluated four models on their benchmarks (Tables N1a and N1b). We show that changing the vendor can make the output different on downstream tasks. In RL, the difference grows during training, because every step samples new responses from a slightly different model.

Table N1a. Share of answers changed by moving from NVIDIA to AMD

SettingAnswers changed by the vendorAnswers changed by repeating the run
Inference, 4 models on MMLU, ArabicMMLU and ARC-Challenge1.1-4.7%0%
Distillation, OLMo-7B and Llama-3.2-1B students (MMLU)3.2-9.4%0%
RL, same trained model evaluated on the other vendor (GSM8K)20-24%0%
RL, model trained on the other vendor (GSM8K)42-44%0%
AuthorsOpenReview ↗

Highlight of Responses to Reviewers Part (2/2): New experiments and references

Table N1b. AMD against NVIDIA on the paper's benchmarks

Model and benchmarkAnswers changedRight to wrongWrong to rightAccuracy change (points)
Jais, MMLU2.35%105112+0.05
Jais, ArabicMMLU1.68%93100+0.05
OLMoE, MMLU4.66%186199+0.09
OLMoE, ARC-Challenge3.41%219-1.02
Llama-3.1-8B, MMLU1.71%7489+0.11
Llama-3.1-8B, ARC-Challenge1.11%46+0.17
Qwen1.5-MoE, MMLU3.25%130166+0.26
Qwen1.5-MoE, ARC-Challenge1.28%75-0.17

N2. H100->A100 differs about as much as towards AMD does (fwxa, U951).

We show that this problem is similar with cross-architecture where going from H100 to A100 also give differentiation like from A100 to MI210.

Table N2. H100 and MI210 against the A100 reference

ModelLogit MSE, H100Logit MSE, MI210Identical rollouts, H100Identical rollouts, MI210
Jais0.2380.287181/200187/200
OLMoE0.02110.021551/12844/128
Llama-3.1-8B0.01040.009919/12821/128

N3. Tokens change almost only at near-ties of the reference (VYuC, U951). A token changes between vendors when the top two tokens have almost the same logit, so the small AMD-NVIDIA difference can swap their order. Such near-ties are a small share of all tokens, yet they account for almost every token change (Table N3).

Table N3. Share of token changes that occur at near-ties

ModelToken changes at near-tiesTokens that are near-ties
Jais97%12%
OLMoE100%1.6%
Llama-3.1-8B95%1.7%

N4. Selective FP32 still works under real serving settings (VYuC, U951). Batching and KV-cache decoding change how the kernels accumulate, so they add a mismatch of their own, even on the same GPU. Under both, selective FP32 keeps most of full FP32's gain at a fraction of its cost (Table N4a). It also holds on another model, Llama-3.1-8B (Table N4b).

Table N4a. Selective FP32 on Jais under serving settings, relative to native BF16

Batch sizeTime (SDPA)Peak memory (SDPA)Logit MSE change, selective / full FP32 (SDPA)Identical rollouts with KV cache, of 200
11.16x1.48x−12.0% / −14.7%185
81.48x1.45x−14.0% / −13.9%186
321.51x1.41x−12.0% / −12.9%186

Table N4b. Selective FP32 on Llama-3.1-8B, change from native BF16, on all 128 development responses (the paper's appendix used 32)

SettingTimePeak memoryLogit MSEKL
Selective FP321.40x1.46x−5.4%−35.1%
Full FP322.50x2.76x−11.9%−40.1%

N5. The difference is too spread out for LoRA at the output, which explains why LoRA fails (VYuC). A rank-32 LoRA on the output head can only correct the logits within 32 directions, yet the cross-vendor difference needs far more directions than the logits themselves (Table N5).

Table N5. Directions needed to capture most of the logits and of the cross-vendor difference

JaisOLMoE
Logits48
Cross-vendor difference1,592954

References (works not already cited in the paper)

Dutta et al. (2024). Accuracy is Not All You Need. https://arxiv.org/abs/2407.09141

Khattak and Mikaitis (2025). Accurate Models of NVIDIA Tensor Cores. https://arxiv.org/abs/2512.07004

Qi et al. (2025). Defeating the Training-Inference Mismatch via FP16. https://arxiv.org/abs/2510.26788

Schlögl, Hofer and Böhme (2023). Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks. https://proceedings.neurips.cc/paper_files/paper/2023/hash/af076c3bdbf935b81d808e37c5ede463-Abstract-Conference.html

Yao et al. (2025). Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. https://fengyao.notion.site/off-policy-rl

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.