acceptodds
Withdrawn from ICLR 2027

Beyond Fixed Precision: Measuring and Tracking Minimum Sufficient Precision in Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixed-precision training typically assigns one of a few hardware-supported formats to each layer, operator, or graph region. Such assignment rules choose among available formats without measuring the precision that a given learning computation actually requires. We introduce Minimum Sufficient Precision (MSP), the lowest precision that preserves a training step's numerical or learning-level behavior for one operator execution. We measure MSP with controlled precision sweeps across four model families and a controlled stress workload, relate its variation to numerical and learning-side factors, and use it to provision executable precision levels during training. In 56% of 120 valid comparisons the numerical and learning-level criteria select different MSPs, so numerical accuracy alone does not guarantee that training behavior is preserved; the requirement also moves with training state, and refreshing more often does not consistently improve fidelity. Precision is therefore a local, criterion-specific, and time-varying computational requirement to be measured and tracked, rather than a fixed datatype assignment.

Primary area:
optimization
Keywords:
Adaptive Precision, Mixed-Precision Learning, Numerical Stability, Low-Precision Computing, Efficient Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BpttOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies minimum sufficient precision for LLMs. The authors propose two metrics: numerical MSP and learning MSP. The former measures numerical difference with different precision setting whereas the latter measures the effect on downstream tasks. Experiments are conducted over multiple models to compare how those two metrics relate with each other.

Critical strengths
  1. The idea of isolating/comparing local changes (numerical) vs global effects (learning) is interesting. Precision selection based only on local output error can miss changes in real operations that affects training, e.g. gradient update, which have different tolerence on precision requirements.

  2. Experiment results are candid and transparent on limitations and failures. These disclosures make the work valuable as an emprical study.

Critical weaknesses
  1. It's unclear how this is useful in practice. The results seem pretty noisy. Models can have two to three selected MSP and the numerical MSP often disagrees with learning MSP. And this is only across four precision setting. In practice, the setting is exponentially much more. The measurement also seems expensive under large number of settings. The authors should discuss and show how this method is useful in practice. Otherwise, this work remains an interesting finding, but not complete.

  2. The robustness of the threshold and method is unclear. First, the two metrics, numerical and learning, are not on the same scale and needs to be normalized in certain ways. Then it's unclear how the threshold are selected to decide MSP. This is crutial and seems to be manually picked without justification or analysis.

Questions for authors
  1. how's this useful in practice, espeically for guiding training process?

  2. how robustness are the threshold and normalization method? does it changes with models, sizes, data?

  3. Does "operator execution" means both forward GEMM and backward GEMM, or only forward?

  4. Which is more favorable, numerical or learning? Can you provide actual evidence on using MSP produces better training outcomes, e.g. training loss?

Reviewer_xh1COpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the the lowest precision one single training step's numerical or learning-level behavior for one operator step. It distinguishes numerical output accuracy from downstream effects on gradients, updates, or loss. Using four calibrated precision levels across four model families and a controlled workload, the authors find that numerical and learning-level criteria disagree in 67 of 120 comparisons. They also examine factors associated with MSP, changes during training, and the trade-offs of using measured values to select precision.

Critical strengths
  1. A useful and carefully controlled measurement approach. Changing one operator’s arithmetic while holding its execution context fixed helps isolate precision effects. Separating output accuracy from downstream fidelity makes the study more informative than measuring numerical error alone.
  2. The experiments show both the value and limits of the measurements. The paper reports operator-level variation, temporal changes, refresh-policy comparisons, and unfavorable provisioning outcomes. Together, these provide useful evidence about when local precision measurements can guide execution and where they need further validation.
Critical weaknesses
  1. The explanatory contribution is less developed than the measurement contribution. The feature analysis shows associations, but some learning-side features are closely related to the deviations used to define MSP. The results therefore provide limited insight into how much can be explained independently of measuring those deviations.
  2. The tracking method is difficult to assess fully from the paper. The validity definition is clear and the refresh comparisons are useful, but the state-aware decision rule and its reference requirements are not explicit. This leaves uncertainty about how the policy would operate outside the measurement setup.
Questions for authors
  1. Could you briefly summarize the strict/default/loose threshold results reported in the archive? In particular, how stable are the numerical-stricter and learning-stricter counts?
  2. What triggers the state-aware refresh policy, and what reference information does it require? Appendix F describes the reuse/refresh framework, but the decision rule is unclear.
  3. Does Figure 4’s right panel show a subset of the conditions used in the left-panel fits? Please clarify the regression model and bootstrap sampling unit.
  4. Could you summarize which downstream quantity and horizon define the learning-level labels for each workload? Appendix A.3 places these settings in the archive, but a short explanation would make the main results easier to interpret.
Reviewer_4GVeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Minimum Sufficient Precision (MSP) as the lowest tested arithmetic precision that preserves a specified numerical or learning-level criterion for an individual operator execution. Using four emulated precision levels, it studies variation across operators and training states, disagreement between fidelity criteria, and the consequences of reusing or provisioning measured values. The contribution is a useful but incremental measurement framework and empirical study, without a practical adaptive training system.

Critical strengths

The paper includes a useful experimental separation of arithmetic error from learning sensitivity. Holding the checkpoint, inputs and surrounding computation fixed makes the local intervention easy to interpret. It is interesting to see examples where satisfying the numerical criterion does not satisfy the learning criterion.

The submission includes a good deal of supporting material. The supplementary archive contains measurement code and raw timing repeats.

Critical weaknesses

The novelty is modest. Prior work already adapts precision to learning sensitivity. The paper needs to establish more clearly what its measurement framework reveals beyond existing approaches.

The practical benefit remains unclear. Guided provisioning sometimes violates the fidelity threshold or runs slower than FP32, while the reported timings exclude measurement and preparation costs.

The explanatory analysis is not very convincing. The reported fits are in-sample and include learning deviations closely related to the target being explained. It's not clear to what extent this is transferable.

Questions for authors

What does MSP measurement reveal that existing adaptive precision methods can't?

Does guided provisioning preserve fidelity while reducing total runtime, including measurement and preparation costs? If so, in what situations?

Does the explanatory model retain its benefit on held-out conditions?

Reviewer_1D1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Minimum Sufficient Precision (MSP), defined as the lowest calibrated arithmetic precision that preserves a specified numerical or learning-level fidelity criterion for an individual operator execution. The authors measure MSP through controlled precision sweeps, investigate its dependence on numerical characteristics and optimization sensitivity, and study how precision requirements change during training. They further evaluate MSP-guided precision provisioning using a four-level precision-emulation backend across several model families. The results show substantial disagreement between numerical and learning-level precision requirements, while also revealing workload-dependent trade-offs between numerical precision, training fidelity, and arithmetic cost.

Critical strengths

1.The paper presents an interesting perspective on mixed-precision training by distinguishing the precision required by a computation from the numerical formats supported by hardware. Defining precision demand at the operator-execution level provides a systematic way to examine precision redundancy and sensitivity across training states. 2.Separating numerical fidelity from learning-level fidelity is conceptually useful. The controlled precision sweeps demonstrate that numerical output accuracy and downstream training behavior can impose different precision requirements, motivating a more learning-aware approach to precision allocation. 3.The experiments investigate precision heterogeneity across several model families, operators, and training states. The authors also report cases where MSP-guided provisioning fails to preserve cumulative fidelity or increases arithmetic cost, providing a useful characterization of the limitations of local precision measurements.

Critical weaknesses

1.Local MSP measurements do not reliably guarantee fidelity when applied to multi-step training. The MSP Oracle measures one operator execution while keeping the surrounding computation at reference precision. However, practical precision allocation changes multiple operations simultaneously, and the resulting errors can interact and accumulate across training steps. This limitation is directly reflected in Section 4.5 and Appendix H.4, where MSP-guided provisioning produces a ten-step loss deviation of 1.116 on one BERT-tiny window, substantially exceeding the predefined tolerance of 10−310^{-3}. Although the paper acknowledges this failure, it does not establish a reliable mechanism for translating locally sufficient precision into safe joint allocation over longer training trajectories. 2.The practical efficiency benefits are not established by the reported runtime measurements. The runtime comparisons focus on managed GEMM kernels and explicitly exclude packing, scale preparation, workspace setup, and other execution overheads. Moreover, MSP-guided provisioning is slower than the FP32 baseline on some workloads, with runtime ratios of 1.57 for distilgpt2 and 1.28 for MLP-Mixer-B/16. These results suggest that reducing effective arithmetic precision does not necessarily improve execution efficiency. Without end-to-end measurements and comparisons against conventional native mixed-precision implementations, the practical advantage of the proposed precision provisioning remains unclear. 3.The practical cost and reliability of MSP tracking remain insufficiently characterized. While the paper explicitly presents the Oracle as an offline measurement tool and the fitted relation as explanatory rather than predictive, the tracking experiments cover only a small number of short representative trajectories. Validity is defined through a reference-based learning-level deviation, yet the reported tracking results quantify cost primarily through refresh counts rather than the full cost of validity checks and recalibration. The evidence therefore remains insufficient to establish when tracking provides a favorable fidelity–cost trade-off over longer training trajectories. 4.The additional practical value of learning-level MSP over numerical MSP needs clearer evidence. Although the two criteria disagree in 67 of 120 valid comparisons, the numerical criterion is stricter in 63 cases, whereas the learning-level criterion is stricter in only four. Thus, the observed disagreement primarily indicates that numerical fidelity may be overly conservative, rather than that numerical accuracy frequently fails to preserve learning behavior. Since the two criteria also use different deviation metrics and tolerances, the extent to which this mismatch reflects intrinsic learning sensitivity remains uncertain. More direct comparisons of numerical-MSP and learning-MSP allocation policies under matched fidelity and runtime constraints would help establish the benefit of the proposed distinction.

Questions for authors

1.How can MSP measured for a single operator execution be extended to simultaneous precision allocation across multiple operators? Can the authors evaluate whether a joint allocation or longer-horizon fidelity constraint prevents the large cumulative loss deviations observed on BERT-tiny? 2.After including precision calibration, operand decomposition, reconstruction, dispatch, and data conversion, how does MSP-guided provisioning compare with native FP16/BF16/FP32 mixed-precision training in end-to-end training time, peak memory, and achieved model quality? 3.For the evaluated tracking policies, how is staleness detected, what reference computations are required, and what is the total cost of validity checks and recalibration? Do the observed fidelity–cost trade-offs persist over longer training horizons? 4.How sensitive are the 67/120 numerical-versus-learning MSP disagreements to the selected fidelity thresholds? Under matched cumulative training fidelity, how much computational benefit does learning-level MSP provide over numerical-MSP-based allocation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.