acceptodds
Withdrawn from ICLR 2027

Same Loss, Different Gradients

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentiable learning typically assumes that the scalar objective evaluated in the forward pass and the gradient supplied to the optimizer in the backward pass describe the same mathematical object. We show that this correspondence can fail when probabilistic objectives rely on finite special-function recurrences, custom backward rules, and numerical clipping. In high-dimensional von Mises–Fisher (vMF) learning, real numerical implementations can produce identical forward scores and losses at the same learning state while supplying different gradients and following different optimization trajectories. We characterize the structure of this mismatch in finite-start Bessel recurrence and show that classwise radial mismatch can compose through Softmax coupling into a locally nonconservative update field. Evaluating special-function values and derivatives separately is therefore insufficient to characterize the realized learning objective. Motivated by this observation, we introduce AR/FR, a fixed-depth analytic realization that constructs a potential and its derivative jointly, ensuring forward–backward coherence by construction. We establish a uniform cubic-order error bound relative to the exact Bessel ratio over the entire nonnegative concentration axis and propagate this guarantee to learning scores and objectives. As representation dimension increases, the original finite recurrence becomes sequentially deeper, whereas the worst-case AR/FR error guarantee tightens cubically, jointly providing coherence, certified fidelity, and fixed-depth computation. These results suggest that a differentiable numerical primitive is defined by both the values it realizes and the derivatives it actually supplies to the optimizer; together, they constitute the numerical realization of the learning algorithm.

Primary area:
learning theory
Keywords:
Differentiable numerical computation, Forward–backward coherence, von Mises–Fisher learning, Bessel function approximation, Efficient probabilistic learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_21SaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For high-dimensional von Mises–Fisher learning, real numerical implementations can produce identical forward losses while supplying different gradients. The paper introduces the AR/FR, a fixed-depth analytic realization that constructs a potential and its derivative jointly, ensuring forward–backward coherence by construction, and establishes a uniform cubic-order error bound. It also tests classification and OOD detection under different numerical realizations.

Critical strengths
  1. Clear presentation. The figures and mathematical formulation clearly illustrate the distinction between approximation accuracy and forward–backward coherence, making the central problem and proposed construction easy to follow.

2.Theoretical support. The paper provides theoretical analyses of finite-recurrence errors and their implications for gradient consistency. It also establishes a uniform cubic-order error bound for the proposed Bessel ratio approximation and extends this guarantee to scores and fixed-state objectives.

Critical weaknesses

1.Limited applicability and insufficient evidence of practical significance. The proposed remedy targets vMF objectives involving high-order Bessel functions. Most models may do not have this numerical structure, which demonstrate limited applicability. And the current experiments provide limited evidence of its benefits across different learning applications. Moreover, Table 1 shows mixed downstream results, leaving insufficiently established when the identified gradient mismatch causes meaningful performance degradation and when correcting it is beneficial. Evaluations on additional vMF-based tasks, would help establish the scope and practical significance of the contribution.

2.Missing comparison with a closely related paired approximation. Scott et al. (2021) already approximate the Bessel ratio analytically and integrate it to obtain a corresponding normalizer surrogate, thereby providing a mathematically coherent, recurrence-free construction. However, the paper does not directly compare AR/FR against this approach. Consequently, the experiments do not establish the incremental benefit of the proposed approximation over an existing method that already provides these properties. Comparisons of numerical accuracy, and downstream performance of Scott et al. (2021) are needed to substantiate the advantage of AR/FR.

Questions for authors

1.The exposition lacks sufficient context and motivation. For example, AR and FR are introduced in lines 84–86 without a clear explanation of what they represent or how their roles in the method should be understood.

2.The derivation and motivation of Eq. (6) are not clearly explained. Specifically, it is unclear how this expression is obtained and why this particular functional form is chosen.

Reviewer_4G99OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates forward–backward incoherence in numerical implementations of von Mises–Fisher (vMF) learning objectives. The authors demonstrate that finite-start Bessel recurrences and clipped custom backward rules can produce identical forward losses while supplying different gradients, resulting in different optimization trajectories. They characterize the numerical mechanisms underlying this mismatch, including recurrence parity, high-concentration behavior, and the possibility of locally nonconservative update fields after Softmax composition.

To address this issue, the paper proposes AR/FR, a paired analytic approximation of the vMF potential and its derivative, ensuring forward–backward coherence in exact arithmetic. The authors establish a uniform O(p−3)O(p^{-3}) approximation error bound and derive corresponding guarantees for learning scores and objectives. Experiments on PATT and ProCo validate the mismatch, while numerical benchmarks and long-tailed classification/OOD experiments evaluate the approximation accuracy, training behavior, and computational efficiency. The method maintains broadly comparable downstream performance and substantially reduces the cost of the local vMF computation.

Critical strengths
  1. The paper provides a systematic investigation of forward–backward incoherence in existing vMF-based learning implementations. Beyond demonstrating identical forward losses with different gradients through controlled experiments, it characterizes the underlying numerical mechanisms using finite-recurrence parity, high-concentration asymptotics, and local Jacobian analysis. These results provide a mathematically grounded explanation of how numerical implementation choices can alter the optimization dynamics of an otherwise unchanged learning objective.

  2. The proposed AR/FR construction jointly defines the numerical potential and its derivative, guaranteeing coherence by construction in exact arithmetic. The uniform cubic-order approximation bound provides a strong theoretical fidelity guarantee that improves with representation dimension. Furthermore, the empirical results demonstrate substantial reductions in the cost of the local vMF computation while maintaining broadly comparable downstream classification and OOD detection performance.

Critical weaknesses
  1. While the paper convincingly identifies forward–backward incoherence in specific vMF learning implementations, the practical significance of this phenomenon for machine learning remains insufficiently established. The observed gradient discrepancy in PATT is relatively small and is largely associated with gradient magnitude rather than direction. The ProCo analysis also indicates that the discrepancy is highly state-dependent and becomes negligible in several later training states. Although the paper demonstrates different optimization trajectories, this does not by itself establish that the original gradients lead to inferior optimization or learning outcomes. The downstream experiments show mixed performance changes rather than a consistent improvement from restoring coherence. Consequently, it remains unclear how broadly consequential this numerical issue is beyond the particular implementations examined. This is one of my major concerns after reading the paper.

  2. The paper would benefit from a clearer assessment of what is fundamentally new in AR/FR relative to existing vMF numerical approximations. As acknowledged in the related work, Scott et al. (2021) already construct a paired potential and derivative by integrating an analytic approximation of the Bessel ratio. This raises questions about the specific novelty of the proposed pairing principle beyond the particular approximation formula and its error certificate. Moreover, the numerical benchmarks show that CUSF achieves substantially smaller approximation errors than AR/FR in several feature dimensions, yet a matched computational-efficiency comparison is not provided. The largest reported speedup also combines the analytic replacement with additional structure-aware execution optimizations. These factors make it difficult to fully isolate the advantage of the proposed numerical construction over established alternatives.

Questions for authors
  1. The paper demonstrates that forward–backward incoherence can lead to different optimization trajectories. However, Appendix E.4 and E.5 suggest that the observed gradient discrepancy is relatively small and is dominated by magnitude differences in the analyzed PATT states, while Appendix C reports much smaller defects in several later ProCo states. Could the authors further clarify the practical consequences of these discrepancies for optimization quality, beyond trajectory separation? In particular, to what extent can the mismatch be explained or mitigated by gradient rescaling or optimizer adaptation, and is there evidence that it leads to materially different learning outcomes in the actual training regimes studied?

  2. The related-work discussion acknowledges that Scott et al. (2021) already construct a coherent potential–derivative pair through integration of an analytic Bessel-ratio approximation. Could the authors clarify which aspects of AR/FR are fundamentally new relative to this prior construction, beyond the specific approximation formula? A direct comparison with this existing paired approach, as well as a matched efficiency–accuracy comparison against strong numerical evaluators such as CUSF, would help establish the distinctive advantages of the proposed method. It would also be useful to distinguish evaluator-level improvements from the additional structure-aware execution gains when assessing practical efficiency.

Reviewer_Y8wAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies von Mises–Fisher learning. In this setting, there are technical/numerical issues that prevent correspondence between the objective function and the gradients used for training. The paper explains how two identical objectives can have different gradients. To address this, they propose a new approach that is guaranteed to have full correspondence by design. They demonstrate their method in experiments.

Critical strengths
  • The paper identifies a failure mode of existing methods.
  • The paper presents a way to fix it.
  • The new approach has theoretical guarantees.
Critical weaknesses
  • My main concern is that the experiments do not show a clear advantage over existing methods. Most results fall within the standard deviation.
  • Additionally, it is not clear to me how often the reported mismatch matters during training. What are the practical disadvantages of current methods in terms of performance?

On the writing side:

  • Some concepts are not introduced in the paper. For example, the von Mises–Fisher distribution, the high-order modified Bessel function, and its relation to the von Mises–Fisher distribution.
Questions for authors
  1. Please comment on the two concerns that I raised in the weaknesses section.
  2. Scott et al. (2021) also derived bounds on the Bessel ratio (see Appendix B, Eq. 9). How would the results of the experiment change if we were to use their expressions instead of yours (Eq. 6)?

References: Scott et al. (2021) - von Mises–Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.