acceptodds
Under review as a conference paper at ICLR 2027

Loss Units Change the Algorithm: Consistent Multi-Optimizer AMP

Abstract

Replacing a batch-mean loss with a batch sum, with a compensating learning-rate change, preserves the real-arithmetic SGD update. Under automatic mixed precision (AMP), however, the larger loss can cause one optimizer to skip while another steps. We study how this numerical decision changes multi-optimizer training. Transactional Covariant AMP (TC-AMP) adjusts each loss scale inversely to its loss units and uses joint replay: if any block overflows, no optimizer steps and the same minibatch is retried. We prove exact execution equivalence for disjoint SGD and classical-momentum blocks under power-of-two unit changes. The proof assumes matched scaled backward expressions and normal-range arithmetic. In partial DINOv2 fine-tuning on CUB-200-2011, native AMP produces eight partial updates per sum-unit run and different mean–sum model endpoints. TC-AMP matches parameters, predictions, update events, and reference-unit scaler histories between loss units in each of three seeds. It reaches development validation accuracy, versus for FP32, at its committed-image throughput in this cached-feature workload. A per-block replay baseline also matches model endpoints, but requires eight additional retries. Checkpoint interventions show why this distinction matters: saved scaler state alone can change retry cost even when model and optimizer states match. Loss units can thus change either the updates performed or the computation required to perform them. Consistency tests for mixed-precision training should track update decisions and consumed minibatches alongside final weights.

Primary area:
optimization
Keywords:
mixed-precision training, loss scaling, stochastic optimization, multi-optimizer training, numerical reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uWZcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Transactional Covariant AMP (TC-AMP), a method for preserving the execution trajectory in AMP training. The paper finds that under AMP, loss formulations that are originally equivalent may exhibit different actual training trajectories, and thus are no longer equivalent under mixed precision. TC-AMP proposes a loss-unit-aware transactional update mechanism for AMP, ensuring that the training execution process remains consistent under different loss units.

Critical strengths
  1. The paper identifies a problem in AMP training: mathematically equivalent loss formulations have different execution trajectories under AMP.
  2. The theoretical analysis is relatively clear.
  3. Experiments verify trajectory consistency.
Critical weaknesses
  1. The paper identifies that changes in loss units can lead to inconsistent training trajectories under AMP. However, the experiments are mainly conducted in controlled multi-optimizer settings. The prevalence and practical impact of this issue in broader large-scale training scenarios remain insufficiently validated.
  2. TC-AMP introduces additional mechanisms, including loss-unit-aware scaling, transactional updates, and replay. However, the paper provides limited analysis of the computational overhead, synchronization cost, and practical efficiency impact of these mechanisms, especially in large-scale training scenarios.
  3. The theoretical analysis relies on assumptions such as SGD/classical momentum optimizers, disjoint parameter ownership, and specific scaling conditions. However, modern training pipelines often involve AdamW, gradient clipping, and shared parameter structures. Whether TC-AMP can still guarantee trajectory consistency under these settings remains unclear.
Questions for authors

Can the authors compare TC-AMP with existing AMP implementations or mixed-precision strategies (e.g., BF16 training and alternative loss scaling methods) to clarify when TC-AMP provides practical advantages beyond traditional FP16 GradScaler?

Reviewer_gFzcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how automatic mixed precision (AMP) interacts with multi-optimizer training, specifically when a loss is reduced by summing rather than averaging over the batch. The authors observe that switching SGD from a batch-mean to a batch-sum loss (scaled by batch size) yields an equivalent update in exact arithmetic, but under FP16 AMP the larger sum-reduced loss inflates gradient magnitudes and can trigger per-optimizer gradient-scaler skips, so one optimizer steps while another does not. They propose TC-AMP, which rescales each block’s loss to cancel the batch-size factor and retries the minibatch if any block overflows, and show that mean- and sum-reduced runs then produce identical results.

Critical strengths

The proposed modification appears computationally cheap and low-overhead, which is a point in its favor if the underlying problem matters in practice.

Critical weaknesses
  • Severe clarity problems throughout. The paper repeatedly introduces terms and claims without definition or motivation, making it hard to follow even on careful reading. The title’s “loss units” is not a term I recognize and is never made intuitive. The abstract opens directly into batch-mean vs. batch-sum loss with no motivation, and phrases like “the larger loss can cause one optimizer to skip while another steps” and “this numerical decision” are used before the reader has any basis to understand them. In the intro, notation is undefined (e.g., cic_i and the index ii in Eq. 1), “feature network” and “class centers” are used without saying this is a classification setting, and center loss is invoked with no background despite being central to the setup. The paper should be self-contained for a reader from a neighboring area, and currently it is not.

  • Weak motivation and positioning. It is never made clear why a reader should care that the update “still executes the same updates” under AMP, nor why asynchronous updates across blocks are inherently a problem — many methods (e.g., GANs) update different components at different times without issue, yet the paper treats this as a problem without justifying it. Relatedly, the mean-to-sum switch increases overflow risk, and it is unclear why a practitioner would choose sum reduction in FP16 at all; the paper would be far stronger if it showed concretely when sum-reduced losses arise in practice. The related work is too thin to locate the paper’s contribution in the literature.

  • Limited experimental scope. For a paper about mixed precision, experiments are confined to CIFAR and CUB. At this scale there is little practical reason to use AMP over full-precision training, so the experiments do not convincingly demonstrate that the problem or the fix matters where AMP is actually needed.

  • Unclear technical novelty. TC-AMP appears implementable from existing PyTorch components (per-optimizer GradScalers with adjusted init_scale). The authors should clarify what is technically novel beyond this configuration.

Questions for authors

See weakness

Reviewer_aDbZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how gradient overflow in distributed training can lead to an results discrepancy under auto mixed precision FP16 for gradients. It proposes a method for loss aware gradient scaling with additional synchronization across optimizers.

The reported failure mode is valid, but was not discovered by the authors, and not clear if it is practically significant.

Critical strengths

none

Critical weaknesses
  1. Limited novelty. The problem is known:

PyTorch admits in their documentation: "Each optimizer checks its gradients for infs/NaNs and makes an independent decision whether or not to skip the step. This may result in one optimizer skipping the step while the other one does not. Since step skipping occurs rarely (every several hundred iterations) this should not impede convergence. If you observe poor convergence after adding gradient scaling to a multiple-optimizer model, please report a bug."

The proposed method also does not seem to be novel: loss rescaling, synchronization and replay fallback.

  1. Limited practical significance. I am not convinced that the problem occurs often in practice. Also, there exist easier fixes: for example, authors report no issues when changing the gradients data type from FP16 to BF16.
Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.