acceptodds
Withdrawn from ICLR 2027

How Accurate Is Accurate Enough?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How accurate must a numerical approximation be within a learning system? Primitive error alone cannot answer this question: errors of the same magnitude can have very different consequences for losses, predictions, and gradients at different learning states. We study this question through the learning objective itself. The objective weights classwise numerical errors nonuniformly according to the current state, so the importance of an error depends not only on its magnitude but also on the class it affects and the weight that class receives. For softmax cross-entropy, we characterize this coupling between class weights and errors and derive the exact extrema of the signed loss change over pairings of fixed non-target probability and score-error multisets, with the target probability and target score error held fixed. Building on this structure, we establish finite-error guarantees that propagate primitive error to losses, probabilities, predictions, and feature gradients, then invert these guarantees to obtain a certified primitive tolerance for the current state under prescribed learning-level error requirements. We give a complete instantiation of the framework in high-dimensional von Mises–Fisher learning. Controlled interventions and a large collection of saved learning states show that identical primitive error can produce substantially different learning consequences, while certified numerical tolerances vary by orders of magnitude across states under the same learning-level requirements. These results show that the adequacy of a numerical approximation must be assessed in relation to the current learning state and the quantity to be preserved; numerical accuracy should itself be treated as part of the learning objective.

Primary area:
learning theory
Keywords:
numerical accuracy, numerical approximation, state-conditioned tolerance, error propagation, certified bounds, probabilistic learning, special functions, von Mises-Fisher learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x9AwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work tries to back propagate prediction error back and examine how numerical errors come in. There is no clean conclusion (as expected in very general setups), and the takeaway is ambiguous "the adequacy of a numerical approximation must be assessed in relation to the current learning state and the quantity to be preserved; numerical accuracy should itself be treated as part of the learning objective."

Critical strengths

The high level question can be interesting: how compute error changes model performance. This can interesting when LLMs now are usually compressed / quantized after training.

The writing style is very clean, with perfect english, and clean structure.

Critical weaknesses

I think there are two main problems.

First, this work is not readable. The terms and setups are not put into previous works for readers to understand what each term means. Instead, it follows AI writing style, where random terms and notations come in. I will just give a few examples: a. "primitive error" refers to numerical error in each computation step. b. "learning state" means the computed values given a data and a model param in the compute graph of a learned model. c. "learning budget" means the final compute error

Second, there is no interesting technique or interesting conclusion. saying that 'it depends' is okay if surprising and interesting setups are shown to given different behavior. Simply showing that it depends on the logit distribution, for example, is too trivial.

Questions for authors

See weaknesses.

Reviewer_Rp7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how numerical errors in intermediate computations affect the accuracy of a learning objective. The problem can be formulated within a standard framework of numerical error analysis. Given a computational state ss, an intermediate quantity QQ is obtained through a mapping F\mathcal F, and the final objective LL is evaluated through another mapping G\mathcal G. When F\mathcal F is replaced by a numerical approximation F^\widehat{\mathcal F}, the exact and approximate computations are

Q &= \mathcal F(s), & L &= \mathcal G(Q),\ \widehat Q &= \widehat{\mathcal F}(s), & \widehat L &= \mathcal G(\widehat Q). \end{aligned}$$ For small intermediate errors, $$\widehat L-L \approx abla\mathcal G(Q)^\top(\widehat Q-Q).$$ Thus, the same intermediate error may lead to different final errors depending on the sensitivity of the downstream objective. Conversely, given an error budget for the final objective, one can determine how much numerical error is admissible in the intermediate computation. The authors apply this analysis to high-dimensional von Mises–Fisher (vMF) learning. In this example, the computational state, intermediate quantity, and downstream objective correspond to $$\begin{aligned} s &= \left(f,y,\tau,{(\mu_j,k_j,c_j)}_{j=1}^C\right),\ Q &= q,\ \mathcal G(Q) &= \operatorname{LSE}(Q)-Q_y, \end{aligned}$$ where $f$ is the neural network feature, $y$ is the target label, $\tau$ is the temperature, and $(\mu_j,k_j,c_j)$ are the class parameters. The mapping $\mathcal F$ computes class logits involving modified Bessel functions. Specifically, the exact and approximate logits are $$\begin{aligned} q_j &= \Phi_ u(r_j)-\Phi_ u(k_j)+c_j,\ \widehat q_j &= F(r_j)-F(k_j)+c_j, \end{aligned}$$ where $r_j=|k_j\mu_j+f/\tau|$, \$ u=p/2-1\$, and \$\Phi_ u(x)=\log I_ u(x)- u\log x\$. Here, $F$ approximates \$\Phi_ u\$, with a paired derivative approximation $A=F'$ for gradient computation. Starting from a certified error bound for the Bessel function ratio, the authors derive error bounds for the logits, cross-entropy loss, predicted probabilities, and feature gradients. They also analyze how the correspondence between class probabilities and logit errors affects the loss, using the classical rearrangement inequality to characterize the extreme loss changes under fixed probability and error multisets. The authors then invert these error bounds. Given loss and gradient error budgets $\varepsilon_L$ and $\varepsilon_g$, they define the certified primitive tolerance $$b^\star(s;\varepsilon)=\sup\left{ b\geq 0: B_L(s,b)\leq\varepsilon_L,; B_g(s,b)\leq\varepsilon_g \right}.$$ They establish the monotonicity of the certificates, which permits scalar numerical inversion, and derive first-order approximations to the resulting tolerance. For the chosen analytical Bessel approximation, they also prove a uniform error bound of order \$O( u^{-3})\$. The experiments evaluate 5,120 saved learning states using high-precision reference calculations. They demonstrate that identical primitive errors can produce different downstream errors, and that certified numerical tolerances vary substantially across learning states. Under a joint loss and gradient error budget of $10^{-5}$, the full range of certified tolerances spans approximately 7.16 orders of magnitude. Additional experiments examine local sensitivity and the effects of pairing class probability weights with numerical errors. Overall, the paper specializes numerical error propagation and goal-oriented accuracy control to a softmax-vMF learning objective, deriving certified downstream error bounds and state-dependent numerical tolerances for a particular Bessel function approximation.
Critical strengths
  1. The paper provides a systematic error analysis covering losses, predicted probabilities, classification decisions, and feature gradients. The resulting certificates can be evaluated from approximate outputs and valid primitive error bounds, without requiring exact model outputs.
  2. The analytical Bessel function approximation and its uniform error bound involve nontrivial technical work. The numerical study uses saved learning states and high-precision reference computations to examine the resulting certificates.
Critical weaknesses
  1. Limited theoretical novelty. The central methodology is a standard application of numerical conditioning and goal-oriented error estimation: propagate errors through a composite objective, characterize their sensitivity to the current state, and invert the resulting bounds to determine admissible numerical errors. Most of the main theoretical results follow this established approach. The downstream error estimates use standard properties of softmax and cross-entropy; the class-weight–error coupling result follows from the cross-entropy identity and the classical rearrangement inequality; the tolerance inversion relies on monotonicity; and the local analysis uses a Taylor expansion. Although the resulting bounds are valid and useful for the specific objective, they do not appear to establish a substantially new mathematical principle. The more specific technical contribution is the construction and error analysis of the Bessel function approximation. However, analytical bounds and stable numerical methods for Bessel function ratios are already well studied. The paper does not establish a clear advantage over existing methods in accuracy, computational complexity, or numerical stability.
  2. Unclear practical significance. The experiments primarily confirm the state dependence of numerical error sensitivity, which is already expected from standard error propagation analysis. The substantial variation in certified tolerances is mathematically consistent with this framework, but does not by itself establish a new computational capability. Moreover, the reported tolerances are sufficient bounds rather than the actual maximum admissible errors. The authors' perturbation experiments indicate that the certificates can be considerably conservative: the median detected gradient-response threshold is 24.61 times the corresponding certified threshold under the prescribed perturbation family. More importantly, the paper does not demonstrate how these certificates improve practical computation. A natural application would be to adapt numerical accuracy to the current learning state, thereby reducing computational costs while maintaining prescribed learning accuracy. However, the experiments evaluate a fixed approximation on saved states rather than implementing such an adaptive scheme. There is no comparison of actual computational costs against existing stable Bessel function implementations, nor evidence of improved training efficiency or model performance. The downstream error budgets are also specified externally, without a quantitative connection to optimization accuracy or convergence. Consequently, it remains unclear whether the proposed certification provides meaningful benefits beyond a conventional error analysis of the chosen numerical approximation.
Questions for authors
  1. What is the main theoretical contribution beyond existing numerical conditioning and goal-oriented error estimation? In particular, how does the proposed Bessel approximation compare with established methods in terms of accuracy, computational complexity, and numerical stability?
  2. Can the certified tolerances be used in an adaptive numerical scheme that demonstrably reduces computational costs? A comparison with existing stable Bessel function implementations in actual training tasks would help establish the practical value of the framework.
  3. How should the loss and gradient error budgets be selected in practice? Given the conservativeness of the certificates, to what extent do the observed variations in certified tolerances reflect differences in actual numerical accuracy requirements?
Reviewer_v5fAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives bounds linking numerical approximation errors to changes in losses, predictions, and feature gradients, then inverts them to obtain state-dependent accuracy tolerances. Experiments on von Mises–Fisher learning show that these tolerances vary widely across learning states, but do not demonstrate end-to-end training benefits or computational savings.

Critical strengths

The authors vMF example connects special-function approximation error to downstream learning quantities and is novel. Distinguishes conservative certification from actual tolerance and fixed-state guarantees from training-level claims.

Critical weaknesses

Results are Somewhat Trivial

  • Perturbations matter according to the derivatives of the downstream objective.
  • Softmax ignores common logit shifts and weights class-specific errors differently.
  • Rescaling a differentiation coordinate rescales its gradient.
  • Inverting a monotone error bound gives an admissible tolerance. The exact finite-error bounds and rigorous vMF approximation analysis require technical work. But technical correctness does not establish that the resulting insight is surprising or consequential. The authors need to distinguish the contribution of those bounds from illustrating expected conditioning effects.

Experiment The experiments primarily illustrate the proposed sensitivity bounds rather than demonstrate their practical value. The coordinate-rescaling intervention reproduces a direct chain-rule effect, while the probability-rearrangement experiments confirm that differently weighted errors have different consequences—both useful checks, but limited evidence of a substantive advance. The central missing experiment is an adaptive-accuracy training run showing that the proposed certificates deliver a useful computational saving while preserving learning quality.

Questions for authors
  1. Can the framework deliver practical computational savings? Demonstrate adaptive accuracy or precision during training that preserves final performance and outperforms simpler policies.

  2. How should the error budgets be chosen? What connects the prescribed loss and feature-gradient tolerances to optimizer updates, convergence, or generalization?

  3. How do fixed-state guarantees extend to actual training? How do model Jacobians, optimizer dynamics, and accumulated numerical errors affect the conclusions?

  4. What is the substantive advance over standard sensitivity analysis? When are the finite-error certificates necessary, given that first-order approximations closely match most reported tolerances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.