- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper attempts to develop new insights into the surrogate losses space, focusing on the softmax-family, by looking into whether members of the family preserve softmax's consistency guarantees, how their optimization is different, and whether computationally they offer different trade-offs.
Critical strengths
The paper looks into a very important direction of getting further insights into surrogate losses that play a crucial role in the ML space. The objective is clear, as the authors focus on consistency of these losses and effects on the resulting optimization. Studying them as a family instead of individually has the potential of finding new patterns that can help in improvements or novel insights on how these losses differ from each other.
The experimental results are solid and support some theoretical observations.
The paper gives a counterexample showing that Rankmax is not Bayes-consistent. This is not covered by the original paper [3].
The analyses in sections 3.4 and 3.5 an incremental but useful comparative synthesis [1,4,5].
References:
[1] Sparse Sequence-to-Sequence Models, Peters et al. 2019
[2] Learning with Fenchel-Young Losses, Blondel et al. 2020
[3] Rankmax: An Adaptive Projection Alternative to the Softmax Function, Kong et al. 2020
[4] Dissipativity Theory for Nesterov’s Accelerated Method, Hu & Lessard, 2017
[5] Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference, Nowozin, 2018
Critical weaknesses
Several theoretical ingredients appear already present in the literature, which is not a problem by itself as long as the paper can provide novel insights by merging or looking at these results differently. However, the additional insights beyond existing theory are not sufficiently clear to me.
The softmax/sparsemax/α-entmax have already been jointly analyzed [1], including general ordering properties and Fisher consistency for probability estimation [2, Proposition 1 and §4.2].
The relationship to prior works is not discussed clearly especially in terms of the overlap and additional contributions.
References:
[1] Sparse Sequence-to-Sequence Models, Peters et al. 2019
[2] Learning with Fenchel-Young Losses, Blondel et al. 2020
[3] Rankmax: An Adaptive Projection Alternative to the Softmax Function, Kong et al. 2020
[4] Dissipativity Theory for Nesterov’s Accelerated Method, Hu & Lessard, 2017
[5] Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference, Nowozin, 2018
Questions for authors
Could the authors clarify what elements of their analysis are different from the literature mentioned above and what additional insights emerge from studying these results together?