acceptodds
Under review as a conference paper at ICLR 2027

WHEN DOES EXACT-ZERO ATTENTION HELP?

Abstract

When does exact-zero attention improve prediction over bounded dense softmax? We answer this by disentangling three mechanisms: bounded representation, implicit optimization bias, and network-level nuisance propagation. Theoretically, balancing a lower bound on bounded softmax residual-noise risk against a finite-sample support-recovery bound for sparsemax yields a sufficient condition: exact zeros reduce excess risk whenever support-estimation error drops below the retained noise floor. Across 2,400 controlled fits, this trade-off spans an empirical phase map over support sparsity, noise scale, and sample size. Solving both model classes to exact convex optimality decouples representation from optimization. The sparse advantage and its contraction with score range persist without an optimizer, while a closed-form leak term predicts the gap between optima across twelve orders of magnitude. Inside 300 frozen forecasters, the network-level mechanism is isolated: perturbing only the positions zeroed by sparsemax shifts predictions to of what an equally large random set does, against for its softmax twin. On real data, an input coordinate estimated strictly from the training split tracks the regime across the settings we test. Over 798 configurations drawn from a 1.2-million-record cryptocurrency panel and a 121-series US macroeconomic panel, spanning both attention directions, exact zeros lower test error in 773 cases (): by across the cryptocurrency cross-section, along a size-matched time axis, and across the macroeconomic series, with intervals robust to overlapping-window dependence. Finally, a decision rule frozen and registered in advance returned all four of its predicted wins on two untouched datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
sparse attention, attention interpretability, sparsemax, cryptocurrency, time-series forcesating

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bEttOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work studies when sparsemax improves prediction over softmax. Assuming a decomposition between useful signal and residual noise at the output of the model, the theoretical analysis derives a lower bound for the amount of residual noise transmitted by bounded softmax to the output, and an upper bound for the expected excess risk of sparsemax based on the probability of incorrect signal support recovery. Combining the bounds leads to a sufficient condition for sparsemax to outperform softmax, interpreted as a trade-off between residual noise transmitted by nonzero weights and the risk of discarding relevant signal with zero weights due to finite-sample support estimation. This result is tested in a synthetic experiment where a linear predictors with weights normalized by softmax or sparsemax is trained on data in which only a fraction of the input variables carry signal. Regimes where sparsemax outperforms softmax are identified through difference in excess-MSE across varying fractions of relevant inputs, noise levels, and sample sizes, and further experiments study then entmax family interpolating between softmax to sparsemax. The study and comparison between softmax and sparsemax is then extended to multiple experimental settings with real-world datasets.

Critical strengths

Decomposing the error into interpretable contributions through theoretical analysis of a relevant minimal setting is a valuable approach. In particular, the trade-off between propagating noise through nonzero weights and discarding relevant signal through zero weights is a realistic mechanism.

Critical weaknesses

The theoretical results follow from two strong assumptions : i) the noise term in theorem 1 follows directly from bounded scores and vanishes in the large B limit. The covariance assumption in Remark 2 excludes zero eigenvalues and prevents cancellation between correlated noise contributions. ii) in theorem 3, the bound on the probability of recovering the wrong support follows directly from assuming a bound on the probability of a given error in each learned score. My concern is that the setting is too ad-hoc : the model assumes that useful signal and noise are already separated at the output ; so zeroing the noise weights helps by construction. The synthetic predictor applies softmax or sparsemax directly to trainable parameters to obtain fixed regression weights, and does not describe how attention mechanism learns to separate relevant inputs from noise.

The presentation is very hard to follow, with multiple statistical tests insufficiently introduced and many acronyms and numerical details reported in the text (eg lines 374-376 what is z, why 12 steps, lines 398-402 what does the margin correspond to, what is the prediction shifted by 0.17, l406 what do these multiple NMSE correspond to ?) Results on real data mostly do not support the theoretical claims and in particular no error advantage of sparsemax was observed in experiments with MISA. "On ETTh1, ETTh2, ETTm1, ETTm2, and the 207-sensor METR-LA, the theory predicts a tie." --> how does the theory predicts ties ?

The synthetic experiments test only a few values of each parameter, and many “ties” are produced by differences too uncertain to resolve, making the reported regime boundaries difficult to interpret. Increasing B does suppress the advantage of sparsemax, and experiments with the entmax family remain descriptive and do not interpret well the results.

Questions for authors

Clarification : please see weaknesses.

Suggestion for future work : can the assumptions of theorem 3 and corollary 4 be proved in a setting where the predictor and the task are specified ; or can you provide experimental evidence for those ?

Reviewer_pM4VOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper provides a sufficient condition for exact zero attention to outperform dense attention in time series forecasting: when enough irrelevant positions have sufficient noise, and enough training samples exist to identify them accurately. Here, forcing irrelevant attention weights to exactly zero removes a “noise floor” that bounded dense softmax cannot eliminate. Then the theoretical result is demonstrated with controlled experiments in the specific setting.

Critical strengths

I think the question itself is interesting: when sparse attention would have an advantage based on the data distribution in the context of time series forecasting, and whether that advantage comes from optimization procedures or is there an intrinsic gap in what they can learn.

Critical weaknesses

The clarity really makes it difficult to interpret the paper

  • The author should make it more obvious that it is in the context of time series forecasting. The title and abstract do not mention this. It is also overclaiming, as a sufficient condition isn’t the same as “resolves when and why exact zeros outperform dense softmax”. The contribution does not classify when this is the case that sparsemax is better, or sufficiently identify the failure mode of when dense is better.
  • Throughout the paper undefined or unclear when they appear. For example:
    • Intro, “signal-plus-nuisance”, “noisy support estimation”, “support-recovery bound” are all difficult to understand. “normalization never matters” is also not defined or cited anywhere.
    • Contributions (1): What is support-estimation error, or softmax noise floor, or leak term? Twelve orders of magnitude of what? I can have my guess of what they are, but none of them are obvious.
  • There seem to be a lot of experiments and a lot of mention of XX many out of XX scenarios this is better, etc. I don’t think that communicates effectively what the diversity or coverage of these experiments is, why we need so many configurations, and, more importantly, “when” does sparse help. The abstract and intro seem to be overloaded with these numbers that should be experimental details. Many terms are again undefined; for example, what is ETT, METR-LA?
  • “Convex reference solutions show the sparse advantage persists without an optimizer.” Convex problems could still need an optimizer to solve

Significance

  • “data distribution boundary” framework: the theory shows that sparsemax outperforms softmax when there are enough irrelevant positions with sufficient noise, and there are enough training samples to identify them accurately. This result makes sense but seems straightforward to me. I’m not sure if I understand the theoretical significance.
    • What is the surprising part of the theory? Is there any more insight than the fact that sparsemax would remove noisy coordinates if the data is designed specifically to have irrelevant noisy coordinates?
    • How often does this actually happen on practice that there are non-trivial irrelevant positions that meet the threshold?
  • Perturbing sparsemax-zeroed positions shifts predictions less than perturbing dense softmax. Why is this surprising? And why is this important, or a desired quality to have? What are the exact perturbation procedures? 


Novelty and relationship with previous work

  • Sparsemax and its performance or analysis are established before; I’m not a native expert in the field, but the related work also doesn’t tell me which part is new. Is the novelty specifically in time series, and specifically on the designed “ signal-plus-nuisance” data? To me, the importance and practical relevance of this type of data need more demonstration.
  • Are the benchmarks and type/scale of experiments typical in other papers in the field of time series forecasting? How well do the findings generalize to standard time-series forecasting benchmarks?
Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded: all changes in blue, two new figures (Figs. 3 and 6)

We uploaded a revised PDF. Every change is marked in blue. Figs. 1, 2, 4, 5, Tables 1-2 and all appendix labels keep their numbers, so our earlier replies still point to the right places. Figs. 1, 2, 4 and 5 are redrawn from the same result files with the same panels and numbers, and each figure and table now sits where the text first cites it. The entmax figure moved to Appendix AA (Fig. A13). Line numbers in those replies refer to the original PDF.

New Fig. 3 (page 3): real data.

  • (a) Exact zeros lower test error in 773 of 798 real cells.
  • (b) Mean reductions of 2.32%, 12.73% and 15.69%, each with a dependence-robust 90% interval that excludes zero.
  • (c) New count: sparsemax is lower in 12,538 of 15,960 individual rolling origins (78.6%). At n = 128 it is lower in 395 of 399 cells.
  • (d) The training coordinate reads 0.02 to 0.47 where sparsemax wins and 0.50 to 1.00 where the two tie.

New Fig. 6 (page 10): mechanism and frozen rule.

  • (a) New count: sparsemax fits with a wrong support fall from 18.4% at n = 128 to 0 at n = 8,192.
  • (b) In the temporal-attention module trained on windows with known support, sparsemax leaves 0.000 to 0.082 of its attention on irrelevant steps, softmax 0.024 to 0.314 (Table A23).
  • (c) The frozen rule has the lowest mean test error of five strategies.
  • (d) The mean difference in the four predicted-win cells is positive in all three runs, 12 of 12.

Section 6 now also states the FRED-MD test design in the main text: every one of the 121 series is a target, 20 rolling origins, and a moving-block bootstrap that resamples origins jointly across series.

Reviewer bEtt

  • "Increasing B does suppress the advantage of sparsemax": the Data check after Theorem 1 now states that wins fall from 34 to 3 as B grows from 2 to 8, the e−4Be^{-4B} factor of the bound (Fig. 2a gives 34, 17, 4 and 3 at B = 2, 3, 4, 8).
  • Corollary 4: Section 4 now states that the rule check uses the measured failure rates and costs the corollary assumes (57 of 60 regimes).
  • "how does the theory predicts ties ?" Section 6 now opens with a formal definition of the training coordinate (Definition 2) and states how the regime is read from the phase map; the five benchmarks read 0.50 to 1.00 with at least 12,060 training samples.
  • "can you provide experimental evidence for those ?" (assumptions of Theorem 3). A "Data check" now follows Theorem 1, Theorem 3 and Corollary 4. After Theorem 3 it reports the measured wrong-support rate against n (also Fig. 6a), and the bound is never below the measured rate on the 60-cell grid.
  • "helps by construction". A sentence after Definition 1 states that neither model is given the support; Fig. 6b shows that the trained sparsemax module puts 0.000 to 0.082 of its attention on irrelevant steps, against 0.024 to 0.314 for softmax.
  • "what is z, why 12 steps ... what does the margin correspond to, what is the prediction shifted by 0.17 ... what do these multiple NMSE correspond to ?" Each is now defined where it appears in Section 6, and TOST is expanded at first use.

Reviewer pM4V

  • "resolves when and why": the Discussion now reads "gives a sufficient condition for when exact zeros outperform bounded softmax in sequential prediction, and traces why".
  • The abstract's first sentence now names sequence forecasting.
  • The Introduction now defines the support, support estimation, the noise floor and the leak term at first use, and names and cites ETT and METR-LA.
  • Related Work now states that the result is a closed-form risk comparison covering attention over any input groups, with forecasting as the test bed.
  • "How often does this actually happen on practice": Fig. 3a-c.

Reviewer mjf1

  • "Which confirmatory run was designated primary before evaluation": the Confirmatory paragraph now states it (fixed-origin run, registered 2026-09-19 before execution).
  • "The benefit of the selection rule seems limited": the paragraph and Fig. 6c now give the margin over each of the four fixed strategies; each paired 90% interval over the 240 fits excludes zero (Table A11).
  • "The decision margin was also revised": the paragraph now names the two thresholds; all four predicted wins hold under both.
  • "the methods perform similarly when the evaluation samples are redrawn": the paragraph now states that the two redraws score about 1,000 test rows, against 3,780 and 69,527 in the primary run, which widens their intervals; Fig. 6d shows all three runs, and the mean difference is positive in all 12 (four cells, three runs).
  • "how far the practical conclusions extend to neural models": Fig. 6b, and Section 6 now states that, against the whole context, the trained sparsemax forecaster is less sensitive to its zeroed positions than its softmax twin in 140 of 150 pairs (Fig. 5).
Reviewer_mjf1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when sparse attention improves prediction by separating representational constraints, fitting effects, and nuisance propagation through a network. It combines a conditional risk analysis with controlled experiments and proposes a rule for choosing the attention normalization.

Critical strengths
  • The distinction between representation, fitting, and nuisance propagation is useful. The convex reference solutions help explain where prediction gains come from.
  • The experiments include tuned softmax and entmax baselines and neural interventions. The selection rule has the lowest mean error in the confirmatory test. I also appreciate the reporting of failed criteria and protocol changes.
Critical weaknesses
  • The benefit of the selection rule seems limited. The confirmatory improvement over always choosing sparsemax is small and concentrated in one source, while the cross-sectional test favors always choosing sparsemax.
  • The confirmatory results leave uncertainty about the reliability of the selection rule. In two of the four cases initially reported as improvements, the methods perform similarly when the evaluation samples are redrawn. The decision margin was also revised after the original outcome table existed. The appendix reports these details, but the main conclusions give little attention to this sensitivity.
  • It is unclear how far the practical conclusions extend to neural models. The largest prediction gains come from constrained linear probes, while the neural comparisons largely tie. Which findings support the guidance for choosing an attention normalization?
Questions for authors
  • Which confirmatory run was designated primary before evaluation, and how should the redraws affect the conclusion?
  • How does the proposed training statistic explain the cases where always choosing sparsemax performs better?
1 comment
AuthorsOpenReview ↗

Revised PDF uploaded: all changes in blue, two new figures (Figs. 3 and 6)

We uploaded a revised PDF. Every change is marked in blue. Figs. 1, 2, 4, 5, Tables 1-2 and all appendix labels keep their numbers, so our earlier replies still point to the right places. Figs. 1, 2, 4 and 5 are redrawn from the same result files with the same panels and numbers, and each figure and table now sits where the text first cites it. The entmax figure moved to Appendix AA (Fig. A13). Line numbers in those replies refer to the original PDF.

New Fig. 3 (page 3): real data.

  • (a) Exact zeros lower test error in 773 of 798 real cells.
  • (b) Mean reductions of 2.32%, 12.73% and 15.69%, each with a dependence-robust 90% interval that excludes zero.
  • (c) New count: sparsemax is lower in 12,538 of 15,960 individual rolling origins (78.6%). At n = 128 it is lower in 395 of 399 cells.
  • (d) The training coordinate reads 0.02 to 0.47 where sparsemax wins and 0.50 to 1.00 where the two tie.

New Fig. 6 (page 10): mechanism and frozen rule.

  • (a) New count: sparsemax fits with a wrong support fall from 18.4% at n = 128 to 0 at n = 8,192.
  • (b) In the temporal-attention module trained on windows with known support, sparsemax leaves 0.000 to 0.082 of its attention on irrelevant steps, softmax 0.024 to 0.314 (Table A23).
  • (c) The frozen rule has the lowest mean test error of five strategies.
  • (d) The mean difference in the four predicted-win cells is positive in all three runs, 12 of 12.

Section 6 now also states the FRED-MD test design in the main text: every one of the 121 series is a target, 20 rolling origins, and a moving-block bootstrap that resamples origins jointly across series.

Reviewer bEtt

  • "Increasing B does suppress the advantage of sparsemax": the Data check after Theorem 1 now states that wins fall from 34 to 3 as B grows from 2 to 8, the e−4Be^{-4B} factor of the bound (Fig. 2a gives 34, 17, 4 and 3 at B = 2, 3, 4, 8).
  • Corollary 4: Section 4 now states that the rule check uses the measured failure rates and costs the corollary assumes (57 of 60 regimes).
  • "how does the theory predicts ties ?" Section 6 now opens with a formal definition of the training coordinate (Definition 2) and states how the regime is read from the phase map; the five benchmarks read 0.50 to 1.00 with at least 12,060 training samples.
  • "can you provide experimental evidence for those ?" (assumptions of Theorem 3). A "Data check" now follows Theorem 1, Theorem 3 and Corollary 4. After Theorem 3 it reports the measured wrong-support rate against n (also Fig. 6a), and the bound is never below the measured rate on the 60-cell grid.
  • "helps by construction". A sentence after Definition 1 states that neither model is given the support; Fig. 6b shows that the trained sparsemax module puts 0.000 to 0.082 of its attention on irrelevant steps, against 0.024 to 0.314 for softmax.
  • "what is z, why 12 steps ... what does the margin correspond to, what is the prediction shifted by 0.17 ... what do these multiple NMSE correspond to ?" Each is now defined where it appears in Section 6, and TOST is expanded at first use.

Reviewer pM4V

  • "resolves when and why": the Discussion now reads "gives a sufficient condition for when exact zeros outperform bounded softmax in sequential prediction, and traces why".
  • The abstract's first sentence now names sequence forecasting.
  • The Introduction now defines the support, support estimation, the noise floor and the leak term at first use, and names and cites ETT and METR-LA.
  • Related Work now states that the result is a closed-form risk comparison covering attention over any input groups, with forecasting as the test bed.
  • "How often does this actually happen on practice": Fig. 3a-c.

Reviewer mjf1

  • "Which confirmatory run was designated primary before evaluation": the Confirmatory paragraph now states it (fixed-origin run, registered 2026-09-19 before execution).
  • "The benefit of the selection rule seems limited": the paragraph and Fig. 6c now give the margin over each of the four fixed strategies; each paired 90% interval over the 240 fits excludes zero (Table A11).
  • "The decision margin was also revised": the paragraph now names the two thresholds; all four predicted wins hold under both.
  • "the methods perform similarly when the evaluation samples are redrawn": the paragraph now states that the two redraws score about 1,000 test rows, against 3,780 and 69,527 in the primary run, which widens their intervals; Fig. 6d shows all three runs, and the mean difference is positive in all 12 (four cells, three runs).
  • "how far the practical conclusions extend to neural models": Fig. 6b, and Section 6 now states that, against the whole context, the trained sparsemax forecaster is less sensitive to its zeroed positions than its softmax twin in 140 of 150 pairs (Fig. 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.