- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper theoretically proves that the stochastic gradients of log-softmax loss with class sub-sampling are biased. For the case of large-scale class categories in classification tasks, it is an efficient approach to draw a subset of classes to attend a softmax cross-entropy loss. Based on the theoretical result about softmax [Lin+25], the stochastic gradients w.r.t model parameters for the class-subset softmax loss on mini-batch sampling are proven to be biased, through detailed definition of gradient biasedness. The paper also discusses the extension to this theoretical consequence toward loss functions built upon log-sum-exp functions.
Critical strengths
- In a large-scale classification scenario, class-subset sampling plays a pivotal role for efficient learning. Thus, biasedness of the stochastic gradients addressed in this paper is important for analyzing and further sophisticating optimizers.
Critical weaknesses
- Theoretical novelty is limited. The main claim, Theorem 3.4, is straightforward (trivial) extension from [Lin+25]; biased loss gradients [Lin+25] convey biased parameter gradients in an obvious way.
- It is hard to find novelty or theoretical effectiveness from Definitions 3.1-3.3, though the authors insist that ours may be the first to give a formal and complete definition in line 347. It sounds quite standard to me.
- Most discussions are theoretically not surprising or less clear. Specifically, the weak points of the prior works [Lin+25, Bengio+08, Blanc+18] discussed in this paper are not clear, though the authors aggressively argue them.
Questions for authors
- Please clarify theoretical novelty in comparison to the prior works such as [Lin+25] through clearly pointing out the issues in those works that this paper addresses.
We thank the reviewer for the comments.
For W1
Theorem 3.4 does not follow directly from Lin et al. (2025). Their result concerns logits, while ours concerns model parameters. As explained in our response to Reviewer jTZ9 (Weakness 1), biased logit gradients do not necessarily imply biased parameter gradients.
For W2
Although unbiasedness is a standard concept, its precise definition matters for our existence question. In particular, the order of quantifiers determines whether we require one estimator to be unbiased for all settings or allow a different estimator for each setting. These are different theoretical claims. Please see our response to Reviewer jTZ9 (Weakness 2).
For W3
Our intention is not to question the correctness of previous results, but to clarify their scope.
In particular, both Lin et al. (2025) and Blanc and Rendle (2018, Theorem 2.1) study the unbiasedness condition for logits but not directly for parameter gradients. Bengio and Sénécal (2008) show that their proposed parameter-gradient estimator is biased, but do not establish the impossibility of universal unbiasedness or provide a formal definition for this existence question.
For more discussion on related works, please see our response to Reviewer GxEF (Weakness 1). We will revise our discussion on prior works.