- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors investigate the relationship between attention sharpness and the reasoning depth of Transformer models. They argue that, under standard softmax attention, sharpness emerges indirectly through the magnitude of query–key products and can vary considerably across training runs. To address this limitation, they propose Powered L1 Attention (L1p), an alternative normalization mechanism with a learnable exponent p that explicitly controls attention sharpness and is invariant, in its ideal formulation, to positive rescaling of the query–key products. They additionally introduce a signed form of L1p that permits negative attention weights. The method is evaluated against softmax and several alternative normalization functions on a synthetic multi-hop reasoning task and on language modeling. Under a fixed training budget, L1p solves the 4-hop task in 18 of 32 runs, compared with at most 3 of 32 for any alternative. For the main 152M-parameter language modeling configuration, L1p achieves held-out perplexity comparable to softmax across 24 random seeds. The results are promising, although several questions remain regarding the necessity of learning p, the effects of exponent initialization and signed attention, and generalization beyond the synthetic task.
Critical strengths
S1. A simple and interesting approach to controlling attention sharpness. The paper studies an important architectural component of Transformers: the normalization of attention scores. Introducing a dedicated exponent provides an interpretable mechanism for controlling sharpness, and the mathematical discussion of magnitude invariance offers a clear distinction from standard softmax attention (Sections 3–4).
S2. Substantial gains on a controlled multi-hop reasoning benchmark. The main results show a large improvement in the frequency with which runs solve 4-hop reasoning, relative to multiple alternative normalization functions (Table 2). The experiments across layer counts are also informative, suggesting that the advantage of L1p within the fixed training budget becomes more pronounced in deeper models (Section 5.3).
S3. Extensive comparisons across random seeds and statistical reporting. The use of 32 seeds for the main reasoning comparison and up to 24 for the central language modeling comparison is a notable strength. The authors report uncertainty intervals, statistical tests, and several alternative baselines, which makes the empirical evidence more informative than comparisons based on single runs (Sections 5.1–5.4; Appendix A).
Critical weaknesses
W1. The role of a learnable exponent in the reasoning improvements remains unclear. Although a learnable p is a central feature of L1p, Appendix E.2 shows that fixing p = 4 can yield comparable reasoning performance. On 16 seeds, the fixed-p configuration solves 4 hops in 11 runs versus 9 for the main L1p configuration; when fixed and learnable exponents are both initialized at 4, the corresponding counts are 7 of 8 and 6 of 8. These differences are not statistically significant, and the authors explicitly note that the strongest case for learning p comes from language modeling. Consequently, it is difficult to determine how much of the reasoning gain comes from learning sharpness rather than starting from a favorable sharpness regime.
W2. Initialization sensitivity and model selection merit closer examination. The choice p = 6 is empirically motivated by the sweep in Appendix E.3 and should not be considered arbitrary. Nevertheless, the results vary substantially across initial exponents, and p is initialized identically across layers and shared across heads within a layer. The analysis would be stronger with evidence on convergence across initializations and the diversity of final exponent configurations. In addition, Appendix A.2 states that hyperparameter selection uses the evaluation data of the main comparisons, albeit with different seeds. An independent evaluation split would more clearly separate model selection from final assessment.
W3. The evidence for the proposed reasoning mechanism is primarily associative. The manuscript motivates L1p through the idea that sharp attention suppresses irrelevant information and improves information propagation across hops. However, greater sharpness is not sufficient to explain reasoning performance: some baselines achieve lower attention entropy than L1p without comparable multi-hop success (Appendix B.6). Signed attention and the placement of signed layers are also important (Appendix E). Further mechanistic analysis is needed to disentangle the contributions of sharpness, signed weights, optimization, and the composition of information across layers.
W4. The scope of the reasoning evidence is limited. The main reasoning gains are established on one synthetic directed-graph traversal task. While this benchmark permits precise control over hop count, it does not establish that the same gains transfer to natural-language multi-hop reasoning or other compositional tasks. The language modeling results primarily demonstrate predictive competitiveness, rather than improved reasoning on realistic benchmarks. The advantage is also defined within a fixed training budget; longer training may alter the gap (Sections 5.3–5.4).
W5. Practical implementation cost deserves more emphasis. Appendix F reports useful timing measurements, but L1p lacks a fused attention implementation. At context length 2,048, its compiled unfused training step is about 2.19 times slower than fused softmax. Although this is partly an engineering limitation, it is relevant to the practical attractiveness of the method, particularly at larger model scales.
W6. Key evidence is difficult to find in the present narrative. Several results central to the interpretation of the contribution—especially the learned-versus-fixed exponent comparison, sensitivity to p initialization, and the final layer-wise exponents—are presented primarily in the appendices (Appendices B.4 and E.2–E.3). Moving the most consequential analyses into the main text would make the contribution and its limitations clearer. Figure 1 would also benefit from a more immediate explanation of the rescaled accuracy and the unit of evaluation, currently detailed in Appendix C.3.
Questions for authors
Q1. Is learning p necessary for the improvement in reasoning depth? Given the fixed-exponent results in Appendix E.2, what evidence distinguishes the effect of learning p from that of selecting a useful initial or fixed sharpness? Would a larger matched comparison of fixed and learned exponents at identical initial values and training budgets help isolate their contributions?
Q2. Do different initializations converge to equivalent exponent configurations? The initialization sweep suggests notable sensitivity to the initial exponent. Do runs initialized at different values of p converge toward similar layer-wise exponents, or can substantially different final exponents yield comparable reasoning accuracy? Could the authors report full exponent trajectories and across-seed distributions for each initialization?
Q3. Why use one exponent per layer rather than one exponent per head? Different heads may have different functional roles and optimal sharpness levels. What motivates sharing p among all attention heads within a layer? Have the authors evaluated head-specific exponents or a simpler single-head architecture to isolate the role of head specialization?
Q4. How do changes in p interact with changes in the query–key products? L1p is invariant to common positive rescaling of a row of query–key products, but its sharpness still depends on their relative magnitudes. For nonzero entries, |a_i|/|a_j| = (|z_i|/|z_j|)^p. During training, to what extent does sharpness change through p versus changing relative query–key magnitudes? Can the authors characterize or visualize these coupled dynamics and compare them with softmax?
Q5. Can the claimed reduction in error propagation across hops be tested more directly? Could controlled perturbations of intermediate representations, activation patching, or head- and layer-level interventions identify which computations implement successive hops and whether L1p actually reduces the propagation of intermediate errors? This would strengthen the interpretation of the aggregate accuracy results.
Q6. Why are statistical comparisons unpaired when configurations share seeds? Appendix A.4 states that runs within an experiment share seeds but that no test pairs runs by seed. What motivated this choice? Since seeds also determine training examples in the synthetic task (Appendix A.5), paired confidence intervals and appropriate matched-outcome analyses could be informative alongside the reported Welch tests and Fisher exact tests. Could the authors report whether the conclusions change under such analyses? Furthermore, could the authors report specifically if they perform the comparison between normalizing function with the same initialization (exactly) for the diferent runs and also these will allow to make paired comparisons.
Q7. How sensitive is learning p to regularization and the numerical implementation? The exponents are excluded from weight decay to avoid driving attention toward flatness. Have the authors tested smaller, nonzero regularization strengths for p? Also, since the implemented L1p uses an additive epsilon and row-wise RMS normalization (Section 4.2), could they quantify deviations from exact magnitude invariance and examine sensitivity to epsilon?
Q8. How does the advantage behave under longer training and more realistic reasoning tasks? The reported improvement is measured within a fixed training budget on the graph-traversal benchmark. Would the advantage persist with longer optimization, and can the authors provide evidence of improvement on a natural-language multi-hop or otherwise distinct compositional reasoning task?
7 comments
Comment 2 of 4
Regarding W3, Q4 and Q5, on the mechanism, we agree that sharpness alone does not explain reasoning depth, and the revised paper now separates what the experiments establish from what remains open. Three new analyses bear on it. First, regarding Q4, we measured how the attention entropy changes from initialization to the end of training, and split this change into two parts that sum to it (new Appendix B.7): the part due to the exponent (for softmax, the part due to the overall scale of the query-key products) and the part due to the relative magnitudes of the products. The table gives the total change and the first part, for the attention entropy normalized to lie between 0 (all attention on one token) and 1 (uniform), averaged over layers and over the runs of the main comparisons. A negative value means sharper attention:
| part of the change in attention entropy | reasoning | language modeling |
|---|---|---|
| L1p, total | ||
| L1p, due to the exponent | ||
| softmax, total | ||
| softmax, due to the overall scale of the products |
For L1p, the change comes almost entirely from the exponent, and for softmax mostly from the overall scale of its products. The relative magnitudes of the products account for the remainder, between and .
Second, regarding layer-level interventions, we kept the magnitude of each attention score and dropped its sign at inference, either in all 6 signed layers at once or in one signed layer at a time (new Appendix E.7, on another separately generated question set, on which 19 of the 32 L1p runs solve 4 hops with the signs intact). The table gives the number of L1p runs, out of 32, that solve each hop count. For removal in one layer at a time, it gives the range over the 6 signed layers:
| intervention | 1 hop | 2 hops | 4 hops | 8 hops |
|---|---|---|---|---|
| signs intact | 32 | 32 | 19 | 4 |
| signs removed in all signed layers at once | 0 | 0 | 0 | 0 |
| signs removed in one signed layer at a time | 16 to 26 | 10 to 20 | 3 to 8 | 0 to 1 |
The trained models therefore rely on the signs, and every signed layer contributes. Cog Attention and signed softmax collapse in the same way. Third, since the submission hypothesized that the signs help through cancellation, we measured cancellation directly in the head outputs of layers 7 to 11, which are signed in all four normalizing functions with signed scores (new Appendix E.8). Signed attention cancels part of the summed contribution of the irrelevant tokens, those outside the question and the edges on its path. In L1p, the norm of this contribution is on average 12% smaller than with the signs removed, and the reduction is similar in the runs that solve 4 hops and in those that do not (0.88 and 0.89 of the norm without signs). The revised Section 4 and the new Appendix H therefore state that the mechanism behind the gain remains open, rather than attributing it to cancellation.
Regarding Q5, two analyses bear on error propagation. First, we classified the wrong answers at 4 hops (new Appendix C.13). A wrong answer can name another node on the question's own path, reached in fewer or more than 4 hops, as a traversal that takes too few or too many hops would. Such answers make up 35.6% of the wrong answers of L1p and 46.0% of those of softmax, compared with 23.2% to 23.4% if each wrong answer named a randomly chosen node. Second, on the question sets of new Appendix C.12, which extend to 11 hops, only L1p solves any hop count beyond 4 among the normalizing functions of the main comparison, in 10, 5, 4, 4, 4, 1 and 1 of its 32 runs at 5 to 11 hops. A paired intervention study of intermediate errors needs runs of both functions that solve the same hop count; the 300,000-step runs of Appendix C.14 now provide such pairs, and we leave this study to future work.
Comment 3 of 4
Regarding W4 and Q8, on longer training, the gap widens at the paper's setting, and against softmax at a further tuned setting, L1p reaches depth sooner. We trained both functions for three times the paper's budget, 300,000 steps, with softmax at both the paper's setting and the best setting of a further grid of 4 learning rates and 3 weight decays, which an extension beyond that setting did not improve on (2 seeds each, ranked by mean rescaled accuracy at 4 hops; new Appendices C.7 and C.14). The table gives, for the first 7 seeds of the main comparison, which all three configurations share, the number of runs that solve 4, 8 and 16 hops at 300,000 steps, and the median step at which the runs that solve 4 hops first do so:
| configuration | 4 hops | 8 hops | 16 hops | median first step at 4 hops |
|---|---|---|---|---|
| L1p, paper's setting | 7 | 6 | 4 | 67,500 |
| softmax, paper's setting | 0 | 0 | 0 | not reached |
| softmax, best grid setting | 5 | 5 | 2 | 207,000 |
At the paper's setting, on these seeds, L1p goes from solving 4 hops in 2 runs at 100,000 steps to all 7 at 300,000, while softmax does not solve it in any run at either budget (Fisher's exact test at 300,000 steps, ). At its best setting, softmax also acquires depth with three times the steps, but L1p reaches 4 hops first on every seed, at a median of 67,500 steps against 207,000 (exact log-rank test on the step at which 4 hops is first solved, with the 2 softmax runs that never solve it censored at 300,000; , the smallest possible with 7 runs each). The final counts at 300,000 steps are closer, 7 against 5 at 4 hops and 6 against 5 at 8 hops, a difference that 7 seeds do not resolve (Fisher's exact test, and ). We therefore state the result as faster acquisition of reasoning depth, the revised abstract now states it for a fixed training budget, and Appendix H leaves open whether softmax would match L1p with still longer training.
Regarding natural-language tasks, we evaluated the 120 language-modeling checkpoints on GSM8K and BABILong (new Appendix D.8). At 152M parameters, every configuration scores below the 3.0% of always giving the most common answer on GSM8K, and on BABILong the differences between L1p and softmax are smaller than the standard deviations across seeds (Welch's -test, Holm-adjusted over the four tasks above both floors, from 0.079 to 0.88), so these benchmarks do not resolve differences in reasoning depth. This agrees with Allen-Zhu (2025, Physics of Language Models, Part 4.1), from whose work our task is adapted: at academic scale, models fail even the simplest 2-hop reasoning tasks, which masks differences in deeper reasoning, so controlled synthetic tasks isolate the skill, and several trends observed on them also appear in real-world pretraining. At this scale, the controlled task is therefore the most reliable instrument available for separating normalizing functions by reasoning depth, and the revised paper states the claim for this task (Section 4, the conclusion, and Appendix H).
Regarding W5, we agree that the implementation cost matters, and Section 5 now states it in the main text: at 2,048 tokens, a training step of L1p takes 1.14 times as long as one of softmax with the same implementation, and 2.19 times as long as one with softmax's fused kernel, which L1p does not have yet. Most of this gap therefore comes from the fused kernel, which runs 1.91 times as fast as the unfused implementation of softmax. On the same implementation, L1p allocates 47.5 rather than 39.2 GiB at its peak (Appendix F). The unsigned form is a softmax over , so it can be computed by a fused kernel that transforms the scores before the softmax; the signed form only multiplies each score by the sign of its product before the values are accumulated; and the row's root mean square, by which the products are divided, is a quadratic form in the query that can be accumulated ahead of the attention.
Regarding W6, Section 5.5 now presents the results on the exponent in the main text: its value is important for reasoning depth, a fixed exponent reaches the reasoning depth and the perplexity of a learned one, and the learned exponent ends near 8 on average over layers, with Appendices E.2, E.3 and B.4 reporting them in full. We have also added the definition of the unit of evaluation (first-token accuracy on the questions of each hop count, in a fixed set of 2,560 questions) to Figure 1's caption, which already explained the rescaled accuracy.
Comment 1 of 4
We appreciate the reviewer's positive assessment of the dedicated exponent as an interpretable way to control sharpness, of the gains in reasoning depth, and of the statistical reporting across seeds, as well as the detailed reading and the specific questions. We address the weaknesses and questions in turn with new experiments and analyses, grouping those that concern the same point.
Regarding W1 and Q1, on whether learning p is necessary, we ran a matched comparison at the settings of the main comparisons (revised Appendix E.2). On reasoning, we hold the exponent fixed at 6, its initial value in L1p, changing nothing else, so that the two configurations differ only in whether the exponent is learned, and also at 8, near the value it reaches when learned. On language modeling, we selected a learning rate for each fixed value from three values on one seed, and trained the better of the two, 8, on 8 seeds. The table gives the number of runs, out of 8, that solve 4 and 8 hops on reasoning (a run solves a hop count when its rescaled accuracy reaches 50, as in the paper), with the p-value of Fisher's exact test against the learned exponent on the same seeds in parentheses. On language modeling, it gives the perplexity of the fixed exponent minus that of L1p, with its 95% confidence interval and the p-value of Welch's t-test:
On reasoning, each fixed exponent solves 4 hops in at least as many runs as the learned one. On language modeling, the exponent fixed at 8 has a perplexity 0.07 lower than L1p's (p=0.32) and matches softmax by the paper's criterion, since the 95% confidence interval on their difference, [−0.13,+0.15], lies within ±0.20. The perplexity gap reported in the submission compared an exponent fixed at 4 with one learned from 4, at a learning rate of 6×10−4 rather than L1p's selected 4.8×10−3. Learning p is therefore not needed for either result, and we have revised the paper so that no sentence presents learnability as the source of the gain. The exponent stays learnable at a cost of one parameter per layer, so that each layer can adapt its sharpness during training. What the reasoning gain does need is an exponent above 1 (fixed at 1, it does not solve 4 hops in any of 16 runs, whereas fixed at 4 it solves it in 11; Fisher's exact test, p=6.8×10−5; E.3) and the signed form (trained without it, L1p does not solve 4 hops in any of 16 runs, whereas with it L1p solves it in 9; Fisher's exact test, p=8.2×10−4; E.6).
Regarding W2 and Q2, on initialization, we examined the exponent of every layer at the end of training for each initial value (new Appendix E.4). Averaged over layers and seeds, the final exponent keeps the order of the initial values, at 4.0, 5.8, 8.0, 8.6, 9.6 and 16.2 on reasoning for initial values of 1, 2, 4, 6, 8 and 16, and at 4.4, 5.0, 6.3, 7.3, 8.3 and 13.7 on language modeling. Substantially different final exponents yield comparable accuracy: the 20 runs of the sweep that solve 4 hops end with mean exponents in their signed layers between 7.4 and 14.6. Within each initial value, runs whose signed layers end with larger exponents tend to score higher at 4 hops (Spearman's ρ between +0.19 and +0.90, with 8 runs each and p between 2.4×10−3 and 0.65), whereas over all 48 runs together ρ=−0.02 (p=0.89), so we read this as an association rather than an effect of the exponent. The mean final exponent of every layer for each initial value is in the new Appendix E.4, which also bounds its range across seeds, and that of the main comparisons in Appendix B.4.
Regarding an independent evaluation split, we evaluated the final checkpoints of all 256 runs of the main reasoning comparison on question sets generated from seeds that neither training nor the original evaluation set uses (new Appendix C.12). On these questions, L1p solves 4 hops in 21 of 32 runs (18 on the original set), Cog Attention in 3, signed softmax in 1, and no other function in any run, so the comparison does not depend on the evaluation set used for selection. On language modeling, the evaluations of Appendix D.4 played no part in selection. On WikiText, an independent held-out perplexity, the mean of L1p over 24 seeds is 35.50, against 35.55 for softmax.