- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Transactional Covariant AMP (TC-AMP), a method for preserving the execution trajectory in AMP training. The paper finds that under AMP, loss formulations that are originally equivalent may exhibit different actual training trajectories, and thus are no longer equivalent under mixed precision. TC-AMP proposes a loss-unit-aware transactional update mechanism for AMP, ensuring that the training execution process remains consistent under different loss units.
Critical strengths
- The paper identifies a problem in AMP training: mathematically equivalent loss formulations have different execution trajectories under AMP.
- The theoretical analysis is relatively clear.
- Experiments verify trajectory consistency.
Critical weaknesses
- The paper identifies that changes in loss units can lead to inconsistent training trajectories under AMP. However, the experiments are mainly conducted in controlled multi-optimizer settings. The prevalence and practical impact of this issue in broader large-scale training scenarios remain insufficiently validated.
- TC-AMP introduces additional mechanisms, including loss-unit-aware scaling, transactional updates, and replay. However, the paper provides limited analysis of the computational overhead, synchronization cost, and practical efficiency impact of these mechanisms, especially in large-scale training scenarios.
- The theoretical analysis relies on assumptions such as SGD/classical momentum optimizers, disjoint parameter ownership, and specific scaling conditions. However, modern training pipelines often involve AdamW, gradient clipping, and shared parameter structures. Whether TC-AMP can still guarantee trajectory consistency under these settings remains unclear.
Questions for authors
Can the authors compare TC-AMP with existing AMP implementations or mixed-precision strategies (e.g., BF16 training and alternative loss scaling methods) to clarify when TC-AMP provides practical advantages beyond traditional FP16 GradScaler?