- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies when sparsemax improves prediction over softmax. Assuming a decomposition between useful signal and residual noise at the output of the model, the theoretical analysis derives a lower bound for the amount of residual noise transmitted by bounded softmax to the output, and an upper bound for the expected excess risk of sparsemax based on the probability of incorrect signal support recovery. Combining the bounds leads to a sufficient condition for sparsemax to outperform softmax, interpreted as a trade-off between residual noise transmitted by nonzero weights and the risk of discarding relevant signal with zero weights due to finite-sample support estimation. This result is tested in a synthetic experiment where a linear predictors with weights normalized by softmax or sparsemax is trained on data in which only a fraction of the input variables carry signal. Regimes where sparsemax outperforms softmax are identified through difference in excess-MSE across varying fractions of relevant inputs, noise levels, and sample sizes, and further experiments study then entmax family interpolating between softmax to sparsemax. The study and comparison between softmax and sparsemax is then extended to multiple experimental settings with real-world datasets.
Critical strengths
Decomposing the error into interpretable contributions through theoretical analysis of a relevant minimal setting is a valuable approach. In particular, the trade-off between propagating noise through nonzero weights and discarding relevant signal through zero weights is a realistic mechanism.
Critical weaknesses
The theoretical results follow from two strong assumptions : i) the noise term in theorem 1 follows directly from bounded scores and vanishes in the large B limit. The covariance assumption in Remark 2 excludes zero eigenvalues and prevents cancellation between correlated noise contributions. ii) in theorem 3, the bound on the probability of recovering the wrong support follows directly from assuming a bound on the probability of a given error in each learned score. My concern is that the setting is too ad-hoc : the model assumes that useful signal and noise are already separated at the output ; so zeroing the noise weights helps by construction. The synthetic predictor applies softmax or sparsemax directly to trainable parameters to obtain fixed regression weights, and does not describe how attention mechanism learns to separate relevant inputs from noise.
The presentation is very hard to follow, with multiple statistical tests insufficiently introduced and many acronyms and numerical details reported in the text (eg lines 374-376 what is z, why 12 steps, lines 398-402 what does the margin correspond to, what is the prediction shifted by 0.17, l406 what do these multiple NMSE correspond to ?) Results on real data mostly do not support the theoretical claims and in particular no error advantage of sparsemax was observed in experiments with MISA. "On ETTh1, ETTh2, ETTm1, ETTm2, and the 207-sensor METR-LA, the theory predicts a tie." --> how does the theory predicts ties ?
The synthetic experiments test only a few values of each parameter, and many “ties” are produced by differences too uncertain to resolve, making the reported regime boundaries difficult to interpret. Increasing B does suppress the advantage of sparsemax, and experiments with the entmax family remain descriptive and do not interpret well the results.
Questions for authors
Clarification : please see weaknesses.
Suggestion for future work : can the assumptions of theorem 3 and corollary 4 be proved in a setting where the predictor and the task are specified ; or can you provide experimental evidence for those ?
Revised PDF uploaded: all changes in blue, two new figures (Figs. 3 and 6)
We uploaded a revised PDF. Every change is marked in blue. Figs. 1, 2, 4, 5, Tables 1-2 and all appendix labels keep their numbers, so our earlier replies still point to the right places. Figs. 1, 2, 4 and 5 are redrawn from the same result files with the same panels and numbers, and each figure and table now sits where the text first cites it. The entmax figure moved to Appendix AA (Fig. A13). Line numbers in those replies refer to the original PDF.
New Fig. 3 (page 3): real data.
New Fig. 6 (page 10): mechanism and frozen rule.
Section 6 now also states the FRED-MD test design in the main text: every one of the 121 series is a target, 20 rolling origins, and a moving-block bootstrap that resamples origins jointly across series.
Reviewer bEtt
Reviewer pM4V
Reviewer mjf1