acceptodds
Under review as a conference paper at ICLR 2027

Is Softmax Loss all you need? A Principled Analysis of Softmax-family Loss

Abstract

The **Softmax Cross-Entropy** Loss is one of the most widely employed surrogate objectives for classification and ranking tasks. To elucidate its theoretical properties, the **Fenchel-Young** framework situates it as a canonical instance within a broad family of surrogates. Concurrently, another line of research has addressed scalability when the number of classes is exceedingly large, proposing alternatives that reduce the computational cost of the exact objective. Building on these two perspectives, we present a principled investigation of **Softmax-family** losses. We examine their consistency with task-specific classification and ranking metrics, characterize their gradient structures and score-space smoothness, and derive convergence bounds in a regularized fixed-feature setting. We also analyze approximate methods through a loss-level bias–variance decomposition and per-epoch computational costs, clarifying trade-offs between approximation fidelity and efficiency. Experiments across representative recommendation backbones reveal setting-dependent differences in predictive performance and training behavior, complementing the theoretical analysis. Together, these results establish a principled basis for comparing surrogate objectives and offer practical guidance for loss selection in large-scale machine learning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Fenchel-Young, Consistency, Softmax Approximation, Bias-Variance Decomposition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aFLSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper attempts to develop new insights into the surrogate losses space, focusing on the softmax-family, by looking into whether members of the family preserve softmax's consistency guarantees, how their optimization is different, and whether computationally they offer different trade-offs.

Critical strengths

The paper looks into a very important direction of getting further insights into surrogate losses that play a crucial role in the ML space. The objective is clear, as the authors focus on consistency of these losses and effects on the resulting optimization. Studying them as a family instead of individually has the potential of finding new patterns that can help in improvements or novel insights on how these losses differ from each other.

The experimental results are solid and support some theoretical observations.

The paper gives a counterexample showing that Rankmax is not Bayes-consistent. This is not covered by the original paper [3].

The analyses in sections 3.4 and 3.5 an incremental but useful comparative synthesis [1,4,5].

References:

[1] Sparse Sequence-to-Sequence Models, Peters et al. 2019

[2] Learning with Fenchel-Young Losses, Blondel et al. 2020

[3] Rankmax: An Adaptive Projection Alternative to the Softmax Function, Kong et al. 2020

[4] Dissipativity Theory for Nesterov’s Accelerated Method, Hu & Lessard, 2017

[5] Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference, Nowozin, 2018

Critical weaknesses

Several theoretical ingredients appear already present in the literature, which is not a problem by itself as long as the paper can provide novel insights by merging or looking at these results differently. However, the additional insights beyond existing theory are not sufficiently clear to me.

The softmax/sparsemax/α-entmax have already been jointly analyzed [1], including general ordering properties and Fisher consistency for probability estimation [2, Proposition 1 and §4.2].

The relationship to prior works is not discussed clearly especially in terms of the overlap and additional contributions.

References:

[1] Sparse Sequence-to-Sequence Models, Peters et al. 2019

[2] Learning with Fenchel-Young Losses, Blondel et al. 2020

[3] Rankmax: An Adaptive Projection Alternative to the Softmax Function, Kong et al. 2020

[4] Dissipativity Theory for Nesterov’s Accelerated Method, Hu & Lessard, 2017

[5] Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference, Nowozin, 2018

Questions for authors

Could the authors clarify what elements of their analysis are different from the literature mentioned above and what additional insights emerge from studying these results together?

Reviewer_Fw5dOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper compares softmax and its alternatives from two perspectives: loss properties (consistency, smoothness, and convergence) and computational efficiency (bias, variance, and cost).

Critical strengths
  1. The paper is well organized.
  2. It provides a detailed theoretical comparison of different losses.
  3. The author did a lot of experiments on recommendation tasks to complement their theoretical findings.
Critical weaknesses
  1. My main concern is that the paper studies loss functions and computational efficiency separately, although they could be closely related. For example, in the F–Y framework, the dual variable corresponds to the log-partition for softmax and the threshold for sparse losses [1]. Optimizing this variable jointly with the model can avoid exact normalizer computation [2]. Without discussing this connection, the analysis might be incomplete.
  2. The author did experiments only on recommendation task. It would be more convincing if the author could do experiments on other tasks, such as contrastive learning, extreme classification, to verify their findings.
Questions for authors
  1. Could the authors discuss the connection between loss functions and computational efficiency? For example, would optimizing the F–Y dual variable as in [2] change the conclusions about approximation bias and computational cost?
  2. Theorem 16 assumes a regularized linear head trained with full-batch gradient descent, while the experiments use non-convex models trained with Adam. Thus, the experiments do not directly support the explanation in Finding 1 (Appendix G.2). Could the authors verify this finding under the setting of Theorem 16?
  3. For GRADIENT-MAGNITUDE analysis in section G.5, could the authors discuss how GRADIENT-MAGNITUDE vary during the training process? And how sharpness of the gradient signal distribution on different losses?

References

[1] Loss Functions and Operators Generated by f-Divergences

[2] A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization

Reviewer_NFaxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors study softmax loss and its variants for classification and ranking tasks. The analysis is studied based on the framework of Fenchel–Young losses, which gives a unified entropy-based generation of Softmax, Sparsemax and α-Entmax. Under this framework, the authors show that Rankmax loss is not calibrated with Top-2 and DCG losses. The authors further conduct gradient analyses and compare the smoothness of these losses. Computational alternatives to Softmax are discussed through their loss-level bias, variance and training cost. Experiments on three recommendation datasets compare the performance of these losses under different backbones.

Critical strengths
  1. The paper considers several commonly used alternatives to softmax loss and provides a comparison from their statistical consistency, optimization and computational cost. This comparison is useful for understanding the differences between these losses and is a good tutorial for practitioners.

  2. The consistency analysis distinguishes different classification and ranking targets. The Rankmax counterexample further shows that label-wise weak order preservation alone does not ensure consistency, which is an interesting and counterintuitive result.

  3. The experiments include different recommendation datasets and backbones. Additional comparisons with different learning rates and negative sampling settings provide further observations on the training behavior of these losses.

Critical weaknesses
  1. The decomposition in Section 3.5 is interesting. However, unlike the surrogate regret bound in Appendix C.2, it does not directly control the target regret. The authors are encouraged to further analyze how the bias and variance affect the target loss.

  2. While this work provides interesting theoretical results on the efficiency and consistency of Fenchel-Young losses, several important recent works on this topic is missing. For example, an analysis on the consistency of FY losses for classification is included in [1, Theorem 5.10]; a special kind of FY losses that uses convolutional entropy for better statistical efficiency is provided in [2]; and the dynamic of FY losses in deep learning theory is provided in [3]. It is encouraged to include more recent works on FY losses for a better literature review.

1] Yutong Wang and Clayton Scott. Unified Binary and Multiclass Margin-Based Classification. Journal of Machine Learning Research, 25(143):1–51, 2024.

[2] Yuzhou Cao, Han Bao, Lei Feng, and Bo An. Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel–Young Losses. NeurIPS, 2025.

[3] Han Bao, Shinsaku Sakaue, and Yuki Takezawa. Any-stepsize Gradient Descent for Separable Data under Fenchel–Young Losses. NeurIPS, 2025.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.