acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Deeper through Controlled Sharpness

Abstract

Self-attention in Transformers uses softmax to normalize query-key products into a distribution of attention scores. A sharp distribution concentrates attention on a small number of tokens, whereas a flat one spreads it more evenly across tokens. Softmax, however, has no parameter that directly controls this sharpness. Instead, sharpness depends indirectly on the magnitudes of the query-key products, emerging through training and varying markedly across random seeds. We identify an empirical association between attention sharpness and reasoning depth under softmax. Motivated by this association, we introduce powered L1 attention (L1p), a normalizing function whose sharpness is controlled by a learnable per-layer exponent and whose attention scores are invariant to the overall magnitude of the query-key products. We further introduce a signed form of L1p that permits negative attention scores, which our ablations show is critical for reasoning depth. On a synthetic multi-hop reasoning task with an exactly controllable number of hops, L1p reaches greater reasoning depth than softmax and five established alternative normalizing functions within the same training budget. At hops, L1p solves the task in of runs with different random seeds, whereas every alternative succeeds in at most of . At the same time, across random seeds, L1p matches the language modeling perplexity of softmax within a pre-specified margin. These results show that directly controlling attention sharpness through an exponent, together with signed attention scores, can substantially increase the reasoning depth a Transformer achieves within a single forward pass.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer, Softmax, Multi-hop Reasoning, Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D4WFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate the relationship between attention sharpness and the reasoning depth of Transformer models. They argue that, under standard softmax attention, sharpness emerges indirectly through the magnitude of query–key products and can vary considerably across training runs. To address this limitation, they propose Powered L1 Attention (L1p), an alternative normalization mechanism with a learnable exponent p that explicitly controls attention sharpness and is invariant, in its ideal formulation, to positive rescaling of the query–key products. They additionally introduce a signed form of L1p that permits negative attention weights. The method is evaluated against softmax and several alternative normalization functions on a synthetic multi-hop reasoning task and on language modeling. Under a fixed training budget, L1p solves the 4-hop task in 18 of 32 runs, compared with at most 3 of 32 for any alternative. For the main 152M-parameter language modeling configuration, L1p achieves held-out perplexity comparable to softmax across 24 random seeds. The results are promising, although several questions remain regarding the necessity of learning p, the effects of exponent initialization and signed attention, and generalization beyond the synthetic task.

Critical strengths

S1. A simple and interesting approach to controlling attention sharpness. The paper studies an important architectural component of Transformers: the normalization of attention scores. Introducing a dedicated exponent provides an interpretable mechanism for controlling sharpness, and the mathematical discussion of magnitude invariance offers a clear distinction from standard softmax attention (Sections 3–4).

S2. Substantial gains on a controlled multi-hop reasoning benchmark. The main results show a large improvement in the frequency with which runs solve 4-hop reasoning, relative to multiple alternative normalization functions (Table 2). The experiments across layer counts are also informative, suggesting that the advantage of L1p within the fixed training budget becomes more pronounced in deeper models (Section 5.3).

S3. Extensive comparisons across random seeds and statistical reporting. The use of 32 seeds for the main reasoning comparison and up to 24 for the central language modeling comparison is a notable strength. The authors report uncertainty intervals, statistical tests, and several alternative baselines, which makes the empirical evidence more informative than comparisons based on single runs (Sections 5.1–5.4; Appendix A).

Critical weaknesses

W1. The role of a learnable exponent in the reasoning improvements remains unclear. Although a learnable p is a central feature of L1p, Appendix E.2 shows that fixing p = 4 can yield comparable reasoning performance. On 16 seeds, the fixed-p configuration solves 4 hops in 11 runs versus 9 for the main L1p configuration; when fixed and learnable exponents are both initialized at 4, the corresponding counts are 7 of 8 and 6 of 8. These differences are not statistically significant, and the authors explicitly note that the strongest case for learning p comes from language modeling. Consequently, it is difficult to determine how much of the reasoning gain comes from learning sharpness rather than starting from a favorable sharpness regime.

W2. Initialization sensitivity and model selection merit closer examination. The choice p = 6 is empirically motivated by the sweep in Appendix E.3 and should not be considered arbitrary. Nevertheless, the results vary substantially across initial exponents, and p is initialized identically across layers and shared across heads within a layer. The analysis would be stronger with evidence on convergence across initializations and the diversity of final exponent configurations. In addition, Appendix A.2 states that hyperparameter selection uses the evaluation data of the main comparisons, albeit with different seeds. An independent evaluation split would more clearly separate model selection from final assessment.

W3. The evidence for the proposed reasoning mechanism is primarily associative. The manuscript motivates L1p through the idea that sharp attention suppresses irrelevant information and improves information propagation across hops. However, greater sharpness is not sufficient to explain reasoning performance: some baselines achieve lower attention entropy than L1p without comparable multi-hop success (Appendix B.6). Signed attention and the placement of signed layers are also important (Appendix E). Further mechanistic analysis is needed to disentangle the contributions of sharpness, signed weights, optimization, and the composition of information across layers.

W4. The scope of the reasoning evidence is limited. The main reasoning gains are established on one synthetic directed-graph traversal task. While this benchmark permits precise control over hop count, it does not establish that the same gains transfer to natural-language multi-hop reasoning or other compositional tasks. The language modeling results primarily demonstrate predictive competitiveness, rather than improved reasoning on realistic benchmarks. The advantage is also defined within a fixed training budget; longer training may alter the gap (Sections 5.3–5.4).

W5. Practical implementation cost deserves more emphasis. Appendix F reports useful timing measurements, but L1p lacks a fused attention implementation. At context length 2,048, its compiled unfused training step is about 2.19 times slower than fused softmax. Although this is partly an engineering limitation, it is relevant to the practical attractiveness of the method, particularly at larger model scales.

W6. Key evidence is difficult to find in the present narrative. Several results central to the interpretation of the contribution—especially the learned-versus-fixed exponent comparison, sensitivity to p initialization, and the final layer-wise exponents—are presented primarily in the appendices (Appendices B.4 and E.2–E.3). Moving the most consequential analyses into the main text would make the contribution and its limitations clearer. Figure 1 would also benefit from a more immediate explanation of the rescaled accuracy and the unit of evaluation, currently detailed in Appendix C.3.

Questions for authors

Q1. Is learning p necessary for the improvement in reasoning depth? Given the fixed-exponent results in Appendix E.2, what evidence distinguishes the effect of learning p from that of selecting a useful initial or fixed sharpness? Would a larger matched comparison of fixed and learned exponents at identical initial values and training budgets help isolate their contributions?

Q2. Do different initializations converge to equivalent exponent configurations? The initialization sweep suggests notable sensitivity to the initial exponent. Do runs initialized at different values of p converge toward similar layer-wise exponents, or can substantially different final exponents yield comparable reasoning accuracy? Could the authors report full exponent trajectories and across-seed distributions for each initialization?

Q3. Why use one exponent per layer rather than one exponent per head? Different heads may have different functional roles and optimal sharpness levels. What motivates sharing p among all attention heads within a layer? Have the authors evaluated head-specific exponents or a simpler single-head architecture to isolate the role of head specialization?

Q4. How do changes in p interact with changes in the query–key products? L1p is invariant to common positive rescaling of a row of query–key products, but its sharpness still depends on their relative magnitudes. For nonzero entries, |a_i|/|a_j| = (|z_i|/|z_j|)^p. During training, to what extent does sharpness change through p versus changing relative query–key magnitudes? Can the authors characterize or visualize these coupled dynamics and compare them with softmax?

Q5. Can the claimed reduction in error propagation across hops be tested more directly? Could controlled perturbations of intermediate representations, activation patching, or head- and layer-level interventions identify which computations implement successive hops and whether L1p actually reduces the propagation of intermediate errors? This would strengthen the interpretation of the aggregate accuracy results.

Q6. Why are statistical comparisons unpaired when configurations share seeds? Appendix A.4 states that runs within an experiment share seeds but that no test pairs runs by seed. What motivated this choice? Since seeds also determine training examples in the synthetic task (Appendix A.5), paired confidence intervals and appropriate matched-outcome analyses could be informative alongside the reported Welch tests and Fisher exact tests. Could the authors report whether the conclusions change under such analyses? Furthermore, could the authors report specifically if they perform the comparison between normalizing function with the same initialization (exactly) for the diferent runs and also these will allow to make paired comparisons.

Q7. How sensitive is learning p to regularization and the numerical implementation? The exponents are excluded from weight decay to avoid driving attention toward flatness. Have the authors tested smaller, nonzero regularization strengths for p? Also, since the implemented L1p uses an additive epsilon and row-wise RMS normalization (Section 4.2), could they quantify deviations from exact magnitude invariance and examine sensitivity to epsilon?

Q8. How does the advantage behave under longer training and more realistic reasoning tasks? The reported improvement is measured within a fixed training budget on the graph-traversal benchmark. Would the advantage persist with longer optimization, and can the authors provide evidence of improvement on a natural-language multi-hop or otherwise distinct compositional reasoning task?

7 comments
AuthorsOpenReview ↗

Comment 1 of 4

We appreciate the reviewer's positive assessment of the dedicated exponent as an interpretable way to control sharpness, of the gains in reasoning depth, and of the statistical reporting across seeds, as well as the detailed reading and the specific questions. We address the weaknesses and questions in turn with new experiments and analyses, grouping those that concern the same point.

Regarding W1 and Q1, on whether learning pp is necessary, we ran a matched comparison at the settings of the main comparisons (revised Appendix E.2). On reasoning, we hold the exponent fixed at 6, its initial value in L1p, changing nothing else, so that the two configurations differ only in whether the exponent is learned, and also at 8, near the value it reaches when learned. On language modeling, we selected a learning rate for each fixed value from three values on one seed, and trained the better of the two, 8, on 8 seeds. The table gives the number of runs, out of 8, that solve 4 and 8 hops on reasoning (a run solves a hop count when its rescaled accuracy reaches 50, as in the paper), with the pp-value of Fisher's exact test against the learned exponent on the same seeds in parentheses. On language modeling, it gives the perplexity of the fixed exponent minus that of L1p, with its 95% confidence interval and the pp-value of Welch's tt-test:

exponent4 hops8 hopsperplexity minus L1p's (95% CI)
learned from 6 (L1p)31
fixed at 64 (p=1.0p = 1.0)4 (p=0.28p = 0.28)
fixed at 84 (p=1.0p = 1.0)0 (p=1.0p = 1.0)−0.07-0.07 [−0.20,+0.07][-0.20, +0.07], p=0.32p = 0.32

On reasoning, each fixed exponent solves 4 hops in at least as many runs as the learned one. On language modeling, the exponent fixed at 8 has a perplexity 0.07 lower than L1p's (p=0.32p = 0.32) and matches softmax by the paper's criterion, since the 95% confidence interval on their difference, [−0.13,+0.15][-0.13, +0.15], lies within ±0.20\pm 0.20. The perplexity gap reported in the submission compared an exponent fixed at 4 with one learned from 4, at a learning rate of 6×10−46 \times 10^{-4} rather than L1p's selected 4.8×10−34.8 \times 10^{-3}. Learning pp is therefore not needed for either result, and we have revised the paper so that no sentence presents learnability as the source of the gain. The exponent stays learnable at a cost of one parameter per layer, so that each layer can adapt its sharpness during training. What the reasoning gain does need is an exponent above 1 (fixed at 1, it does not solve 4 hops in any of 16 runs, whereas fixed at 4 it solves it in 11; Fisher's exact test, p=6.8×10−5p = 6.8 \times 10^{-5}; E.3) and the signed form (trained without it, L1p does not solve 4 hops in any of 16 runs, whereas with it L1p solves it in 9; Fisher's exact test, p=8.2×10−4p = 8.2 \times 10^{-4}; E.6).

Regarding W2 and Q2, on initialization, we examined the exponent of every layer at the end of training for each initial value (new Appendix E.4). Averaged over layers and seeds, the final exponent keeps the order of the initial values, at 4.0, 5.8, 8.0, 8.6, 9.6 and 16.2 on reasoning for initial values of 1, 2, 4, 6, 8 and 16, and at 4.4, 5.0, 6.3, 7.3, 8.3 and 13.7 on language modeling. Substantially different final exponents yield comparable accuracy: the 20 runs of the sweep that solve 4 hops end with mean exponents in their signed layers between 7.4 and 14.6. Within each initial value, runs whose signed layers end with larger exponents tend to score higher at 4 hops (Spearman's ρ\rho between +0.19+0.19 and +0.90+0.90, with 8 runs each and pp between 2.4×10−32.4 \times 10^{-3} and 0.65), whereas over all 48 runs together ρ=−0.02\rho = -0.02 (p=0.89p = 0.89), so we read this as an association rather than an effect of the exponent. The mean final exponent of every layer for each initial value is in the new Appendix E.4, which also bounds its range across seeds, and that of the main comparisons in Appendix B.4.

Regarding an independent evaluation split, we evaluated the final checkpoints of all 256 runs of the main reasoning comparison on question sets generated from seeds that neither training nor the original evaluation set uses (new Appendix C.12). On these questions, L1p solves 4 hops in 21 of 32 runs (18 on the original set), Cog Attention in 3, signed softmax in 1, and no other function in any run, so the comparison does not depend on the evaluation set used for selection. On language modeling, the evaluations of Appendix D.4 played no part in selection. On WikiText, an independent held-out perplexity, the mean of L1p over 24 seeds is 35.50, against 35.55 for softmax.

AuthorsOpenReview ↗

Comment 2 of 4

Regarding W3, Q4 and Q5, on the mechanism, we agree that sharpness alone does not explain reasoning depth, and the revised paper now separates what the experiments establish from what remains open. Three new analyses bear on it. First, regarding Q4, we measured how the attention entropy changes from initialization to the end of training, and split this change into two parts that sum to it (new Appendix B.7): the part due to the exponent pp (for softmax, the part due to the overall scale of the query-key products) and the part due to the relative magnitudes of the products. The table gives the total change and the first part, for the attention entropy normalized to lie between 0 (all attention on one token) and 1 (uniform), averaged over layers and over the runs of the main comparisons. A negative value means sharper attention:

part of the change in attention entropyreasoninglanguage modeling
L1p, total−0.108-0.108−0.146-0.146
L1p, due to the exponent−0.109-0.109−0.136-0.136
softmax, total−0.193-0.193−0.451-0.451
softmax, due to the overall scale of the products−0.179-0.179−0.476-0.476

For L1p, the change comes almost entirely from the exponent, and for softmax mostly from the overall scale of its products. The relative magnitudes of the products account for the remainder, between −0.014-0.014 and +0.025+0.025.

Second, regarding layer-level interventions, we kept the magnitude of each attention score and dropped its sign at inference, either in all 6 signed layers at once or in one signed layer at a time (new Appendix E.7, on another separately generated question set, on which 19 of the 32 L1p runs solve 4 hops with the signs intact). The table gives the number of L1p runs, out of 32, that solve each hop count. For removal in one layer at a time, it gives the range over the 6 signed layers:

intervention1 hop2 hops4 hops8 hops
signs intact3232194
signs removed in all signed layers at once0000
signs removed in one signed layer at a time16 to 2610 to 203 to 80 to 1

The trained models therefore rely on the signs, and every signed layer contributes. Cog Attention and signed softmax collapse in the same way. Third, since the submission hypothesized that the signs help through cancellation, we measured cancellation directly in the head outputs of layers 7 to 11, which are signed in all four normalizing functions with signed scores (new Appendix E.8). Signed attention cancels part of the summed contribution of the irrelevant tokens, those outside the question and the edges on its path. In L1p, the norm of this contribution is on average 12% smaller than with the signs removed, and the reduction is similar in the runs that solve 4 hops and in those that do not (0.88 and 0.89 of the norm without signs). The revised Section 4 and the new Appendix H therefore state that the mechanism behind the gain remains open, rather than attributing it to cancellation.

Regarding Q5, two analyses bear on error propagation. First, we classified the wrong answers at 4 hops (new Appendix C.13). A wrong answer can name another node on the question's own path, reached in fewer or more than 4 hops, as a traversal that takes too few or too many hops would. Such answers make up 35.6% of the wrong answers of L1p and 46.0% of those of softmax, compared with 23.2% to 23.4% if each wrong answer named a randomly chosen node. Second, on the question sets of new Appendix C.12, which extend to 11 hops, only L1p solves any hop count beyond 4 among the normalizing functions of the main comparison, in 10, 5, 4, 4, 4, 1 and 1 of its 32 runs at 5 to 11 hops. A paired intervention study of intermediate errors needs runs of both functions that solve the same hop count; the 300,000-step runs of Appendix C.14 now provide such pairs, and we leave this study to future work.

AuthorsOpenReview ↗

Comment 3 of 4

Regarding W4 and Q8, on longer training, the gap widens at the paper's setting, and against softmax at a further tuned setting, L1p reaches depth sooner. We trained both functions for three times the paper's budget, 300,000 steps, with softmax at both the paper's setting and the best setting of a further grid of 4 learning rates and 3 weight decays, which an extension beyond that setting did not improve on (2 seeds each, ranked by mean rescaled accuracy at 4 hops; new Appendices C.7 and C.14). The table gives, for the first 7 seeds of the main comparison, which all three configurations share, the number of runs that solve 4, 8 and 16 hops at 300,000 steps, and the median step at which the runs that solve 4 hops first do so:

configuration4 hops8 hops16 hopsmedian first step at 4 hops
L1p, paper's setting76467,500
softmax, paper's setting000not reached
softmax, best grid setting552207,000

At the paper's setting, on these seeds, L1p goes from solving 4 hops in 2 runs at 100,000 steps to all 7 at 300,000, while softmax does not solve it in any run at either budget (Fisher's exact test at 300,000 steps, p=5.8×10−4p = 5.8 \times 10^{-4}). At its best setting, softmax also acquires depth with three times the steps, but L1p reaches 4 hops first on every seed, at a median of 67,500 steps against 207,000 (exact log-rank test on the step at which 4 hops is first solved, with the 2 softmax runs that never solve it censored at 300,000; p=5.8×10−4p = 5.8 \times 10^{-4}, the smallest possible with 7 runs each). The final counts at 300,000 steps are closer, 7 against 5 at 4 hops and 6 against 5 at 8 hops, a difference that 7 seeds do not resolve (Fisher's exact test, p=0.46p = 0.46 and 1.01.0). We therefore state the result as faster acquisition of reasoning depth, the revised abstract now states it for a fixed training budget, and Appendix H leaves open whether softmax would match L1p with still longer training.

Regarding natural-language tasks, we evaluated the 120 language-modeling checkpoints on GSM8K and BABILong (new Appendix D.8). At 152M parameters, every configuration scores below the 3.0% of always giving the most common answer on GSM8K, and on BABILong the differences between L1p and softmax are smaller than the standard deviations across seeds (Welch's tt-test, Holm-adjusted over the four tasks above both floors, pp from 0.079 to 0.88), so these benchmarks do not resolve differences in reasoning depth. This agrees with Allen-Zhu (2025, Physics of Language Models, Part 4.1), from whose work our task is adapted: at academic scale, models fail even the simplest 2-hop reasoning tasks, which masks differences in deeper reasoning, so controlled synthetic tasks isolate the skill, and several trends observed on them also appear in real-world pretraining. At this scale, the controlled task is therefore the most reliable instrument available for separating normalizing functions by reasoning depth, and the revised paper states the claim for this task (Section 4, the conclusion, and Appendix H).

Regarding W5, we agree that the implementation cost matters, and Section 5 now states it in the main text: at 2,048 tokens, a training step of L1p takes 1.14 times as long as one of softmax with the same implementation, and 2.19 times as long as one with softmax's fused kernel, which L1p does not have yet. Most of this gap therefore comes from the fused kernel, which runs 1.91 times as fast as the unfused implementation of softmax. On the same implementation, L1p allocates 47.5 rather than 39.2 GiB at its peak (Appendix F). The unsigned form is a softmax over plog⁡∣zj∣p \log |z_j|, so it can be computed by a fused kernel that transforms the scores before the softmax; the signed form only multiplies each score by the sign of its product before the values are accumulated; and the row's root mean square, by which the products are divided, is a quadratic form in the query that can be accumulated ahead of the attention.

Regarding W6, Section 5.5 now presents the results on the exponent in the main text: its value is important for reasoning depth, a fixed exponent reaches the reasoning depth and the perplexity of a learned one, and the learned exponent ends near 8 on average over layers, with Appendices E.2, E.3 and B.4 reporting them in full. We have also added the definition of the unit of evaluation (first-token accuracy on the questions of each hop count, in a fixed set of 2,560 questions) to Figure 1's caption, which already explained the rescaled accuracy.

Reviewer_3ccjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the impact of attention score normalizing functions on the reasoning depth of Transformer models (defined as the number of sequential hops a model can successfully chain in a single forward pass).

Proposed Method: The paper proposes Powered L1L_1 Attention (L1pL_1p), a normalizing function that enforces magnitude invariance while exposing a learnable per-layer exponent pp to directly control sharpness. It defines both an unsigned form and a signed form (which allows negative attention weights and subtraction of value vectors). The final architecture applies the unsigned form in the first half of the layers and the signed form in the second half.

Critical strengths

Rigorous and Statistically Sound Empirical Protocol: This paper conducts a remarkably thorough empirical validation using 32 random seeds per model on multi-hop reasoning and 24 seeds on language modeling, complete with rigorous statistical tests (Welch’s t-tests, Fisher’s exact tests, equivalence testing via TOST). This substantiates the claims beyond random seed noise.

Clean Isolation and Ablation of Architectural Factors: The paper systematically isolates the source of the performance gain. Through detailed ablations, it proves that neither magnitude invariance alone, signed attention alone, nor static scaling factors reproduce L1pL_1p's depth. The layer-wise split (unsigned front half, signed back half) and the behavior of the learned exponent pp are thoroughly mapped and justified.

Critical weaknesses

Limited Domain Diversity for Reasoning Validation: The paper’s claims regarding "reasoning depth" rely heavily on a single synthetic multi-hop graph traversal task. While synthetic tasks allow precise control over hop count, it remains unproven whether the empirical superiority of L1pL_1p translates to realistic multi-step reasoning benchmarks (e.g., GSM8K, MATH, or real-world HotpotQA / multi-hop reading comprehension) where reasoning requires natural language semantic understanding rather than structured graph traversal.

Questions for authors

Clarification on Real-World Reasoning Benchmarks: Have you evaluated L1pL_1p on real-world multi-step symbolic or textual reasoning datasets (e.g., GSM8K, ARC-Challenge, or HotpotQA)? If so, does the reasoning depth advantage observed in the synthetic graph traversal task transfer to natural language reasoning tasks?

1 comment

We appreciate the reviewer's positive assessment of our statistical protocol and of the isolation of the architectural factors, as well as the question on whether the advantage in reasoning depth transfers to reasoning in natural language.

Regarding real-world reasoning benchmarks, the submission provides results for 120 language-modeling checkpoints on 20 downstream tasks, ARC-Challenge among them (Appendix D.4). These are five configurations of the main language-modeling comparison at 152M parameters (L1p, softmax, α\alpha-entmax, and SSA at two settings), with 24 seeds each. At this scale, ARC-Challenge is at chance for every configuration, with L1p at 24.1% and softmax at 24.3%, against 25% for a random choice among four answers. Following the reviewer's suggestion, we have now also evaluated the same checkpoints on GSM8K and on BABILong (new Appendix D.8). We chose BABILong because it poses multi-hop questions in natural language, which makes it the closest standard counterpart of our task. Its first five tasks require one, two or three supporting facts (qa1 to qa3) or relations between two or three arguments (qa4 and qa5), and we report all five, without distracting text and with 2 solved examples in the prompt. The table gives the accuracy of each configuration in percent, as the mean and standard deviation over its 24 seeds, together with two floors: a random answer, and the most common answer given every time.

configurationqa1qa2qa3qa4qa5GSM8K
L1p40.6 ± 8.130.7 ± 4.312.3 ± 4.548.4 ± 12.261.1 ± 6.41.7 ± 0.2
softmax35.4 ± 9.927.6 ± 4.614.1 ± 6.253.2 ± 15.960.7 ± 8.41.5 ± 0.3
random answer16.716.716.716.714.3
most common answer18.718.721.517.718.93.0

L1p has the highest mean of the five configurations on qa1, qa2 and qa5, and softmax on qa4. The differences between L1p and softmax are smaller than the standard deviations across seeds, and with Welch's tt-test and the Holm correction over these four tasks, the adjusted pp-values for L1p against softmax are 0.079 on qa2, 0.16 on qa1, 0.49 on qa4 and 0.88 on qa5. On qa3 and GSM8K, every configuration scores at or below the floors. At 152M parameters, these benchmarks therefore do not resolve differences in reasoning depth between normalizing functions. We did not run MATH or HotpotQA, since GSM8K, the easier of the two mathematical benchmarks, is already at the floor at this scale, and HotpotQA additionally requires reading comprehension over several Wikipedia paragraphs.

Our use of a controlled task follows Allen-Zhu (2025, Physics of Language Models, Part 4.1), from whose work the task is adapted. That work argues that at academic scale (e.g., 1.3B parameters and 100B tokens), comparisons of architectures on real benchmarks are often dominated by noise, and that models at this scale fail even the simplest 2-hop reasoning tasks, which masks differences in deeper reasoning. It therefore isolates skills such as reasoning depth in synthetic pretraining tasks, and it finds that several of the trends observed on these tasks also appear in real-world pretraining. This applies all the more to our models, with 55M and 152M parameters. Our task fixes the number of hops exactly and asks for the answer directly, without intermediate steps, so it measures reasoning depth as the paper defines it: the number of sequential hops a model chains within a single forward pass. At this scale, such a task is the most reliable instrument available for separating normalizing functions by reasoning depth.

We agree that our evidence for reasoning depth comes from one controlled task, and the revised paper states the claim for this task (Section 4, the conclusion, and the new Appendix H). Within this setting, we have also tested the controlled result more strictly. As in the paper, rescaled accuracy is 0 for guessing and 100 for answering every question correctly, and a run solves a hop count when it reaches 50 at the end of training. At the setting with the highest mean rescaled accuracy at 4 hops in a further grid of 4 learning rates and 3 weight decays, softmax does not solve 4 hops in any of 8 runs, whereas L1p at the paper's setting solves it in 3 of them on the same seeds, and L1p's rescaled accuracy is higher on all 8 seeds (Wilcoxon signed-rank test, p=0.0078p = 0.0078; new Appendix C.7). On the first 7 of these seeds, with three times the training steps, softmax at this setting solves 4 hops in 5 runs, against all 7 for L1p, and L1p reaches 4 hops first on every seed, at a median of 67,500 steps against 207,000 for the 5 softmax runs that solve it (exact log-rank test, p=5.8×10−4p = 5.8 \times 10^{-4}, the smallest possible with 7 runs each; new Appendix C.14).

We thank the reviewer again for the question on transfer, and we hope that the new evaluations and the revised scope address it. We would be glad to answer any further questions.

Reviewer_3viFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether controlling attention sharpness can improve Transformers’ reasoning depth and proposes Powered L1 Attention (L1p), which replaces softmax with normalized powers of absolute query–key products, using a learnable per-layer exponent and a signed variant allowing negative attention weights. The final model uses unsigned L1p in the first half of layers and signed L1p in the second half. On a synthetic graph-traversal task, L1p substantially outperforms softmax and other normalizations at 4-hop reasoning (18/32 vs. 0/32 and ≤3/32; with 20 layers, 15/16 vs. none). At 152M parameters, LM perplexity is comparable to softmax (23.20 vs. 23.23, within ±0.20). Ablations show exponent choice, signed attention, and signed-layer placement materially affect reasoning. The results support a substantial reasoning improvement on the synthetic task under a fixed training budget while preserving LM performance in the main configuration.

Critical strengths
  1. Substantial, statistically supported gains in multi-hop reasoning. The strongest evidence is the large advantage over softmax and alternative normalizations on the controlled multi-hop reasoning task. In Table 2, L1p solves the 4-hop task in 18/32 runs, versus 0/32 for softmax and at most 3/32 for competing methods. The gain is statistically significant and grows with model depth: Table 15 reports 15/16 successful L1p runs at 20 layers, versus none for softmax. Appendix C.5 further shows robustness to the success threshold. These results convincingly show that attention normalization can materially affect reasoning under a fixed training budget.

  2. Simple architectural change supported by extensive controlled experiments. L1p adds only one learnable exponent per Transformer layer, making it conceptually simple and parameter-efficient. The paper includes extensive ablations—alternative normalizations, exponent initialization and learning, signed versus unsigned attention, and signed-layer placement. The importance of signed attention is especially clear: removing it reduces successful 4-hop runs from 9/16 to 0/16 (Table 21). Language modeling experiments further show that L1p preserves softmax-level perplexity at 152M parameters, rather than improving reasoning at substantial LM cost. These findings strengthen the empirical contribution and justify further investigation.

Critical weaknesses
  1. Evidence for improved reasoning depth is limited to a narrow synthetic setting. The core reasoning experiments use a single synthetic directed-cycle graph-traversal task (Appendix C). This allows precise control over the number of hops but captures only one form of sequential retrieval and composition. The LM experiments provide complementary perplexity and downstream evidence but do not directly establish improved multi-hop reasoning in pretrained LMs; Appendix D.4 evaluates language understanding and retrieval rather than compositional reasoning depth. Thus, the synthetic gain is convincing, but transfer to more diverse reasoning problems remains uncertain. This limits the scope of the conclusions, though it does not invalidate the controlled result.

  2. The role of learnable sharpness in reasoning is not fully established. L1p is motivated by controlling attention sharpness through a learnable exponent rather than indirectly through query–key magnitudes. However, Appendix E.2 shows that a fixed exponent of 4 performs comparably—and numerically better—on the synthetic reasoning task: Table 21 reports 11/16 successful 4-hop runs for the fixed exponent versus 9/16 for L1p, with no statistically significant difference. Appendix E.2 likewise finds no reasoning advantage from learning the exponent. The authors show that learning the exponent benefits language modeling, which partially justifies the design, but the results suggest that the reasoning gain may stem mainly from powered normalization and signed attention rather than exponent learning. Moreover, Appendix B.6 shows that sharper attention does not necessarily yield greater reasoning depth across normalization mechanisms, and the proposed cancellation explanation remains a hypothesis. This weakens the paper’s explanatory claim: the experiments establish that L1p is effective, but provide weaker evidence that adaptive sharpness control is the essential reason.

Questions for authors
  1. How robust is the reasoning advantage to stronger softmax optimization and longer training? Appendix A.2 indicates that learning rates for the reasoning experiments were screened with only two seeds per configuration, while the main experiments show substantial seed-dependent variability. Section 5.3 also notes that the best 20-layer softmax run approaches the 4-hop success threshold and is still improving at the end of training. Could the authors clarify how sensitive the relative performance is to optimization hyperparameters and training duration? Learning curves or additional training-budget comparisons would help determine whether the advantage reflects faster acquisition of reasoning rather than a difference in eventual attainable depth. This could materially affect interpretation of the central result.

  2. How essential is learning the exponent when the fixed-exponent alternative is independently optimized? Appendix E.2 reports comparable reasoning performance for fixed and learnable exponents, while the language modeling results favor learning the exponent; however, these comparisons use settings different from the main LM experiments. Would a fixed-exponent variant, with its own optimized exponent and training hyperparameters, preserve comparable reasoning performance while closing the reported perplexity gap? This would help establish whether exponent learnability is essential to L1p or mainly a convenient way to adapt powered normalization across tasks.

5 comments
AuthorsOpenReview ↗

Comment 1 of 2

We appreciate the reviewer's positive assessment of the gains in reasoning depth and of the controlled experiments and ablations, as well as the question of whether the advantage reflects faster acquisition of reasoning depth rather than a difference in attainable depth. To address it, we conducted three new sets of experiments and analyzed the learning curves recorded for every run. They show that L1p acquires reasoning depth substantially faster than softmax, including softmax at a further tuned setting.

First, regarding optimization, we tuned softmax further over a grid of four learning rates and three weight decays with 2 seeds per setting (new Appendix C.7), and trained it at the setting with the highest mean rescaled accuracy at 4 hops, a learning rate of 2×10−32 \times 10^{-3} and a weight decay of 0.10.1, on 8 seeds of the main comparison. Since this setting lies at the edge of the grid, we also extended the grid beyond it, with learning rates up to 4×10−34 \times 10^{-3} and weight decays up to 0.30.3, and no further setting ranks above it. As in the paper, rescaled accuracy is 0 for guessing and 100 for answering every question correctly, and a run solves a hop count when it reaches 50 at the end of training. At the best setting, softmax solves 1 and 2 hops in more runs than at the paper's setting, but it does not solve 4 hops on any of the 8 seeds, whereas L1p at the paper's setting solves it on 3 of them. L1p's rescaled accuracy at 4 hops is higher on all 8 seeds (Wilcoxon signed-rank test, p=0.0078p = 0.0078), and the 3-to-0 difference in runs that solve 4 hops gives McNemar's p=0.25p = 0.25 with 8 seeds. L1p at softmax's best setting solves 4 hops in only 1 of 8, so each function is compared at its own tuned setting, as in the paper. Within the paper's budget, neither the paper's setting nor the grid's best setting brings softmax to 4 hops on the seeds of the main comparison, so the conclusion does not rest on the setting chosen by the paper's two-seed learning-rate selection.

Second, regarding training duration, the learning curves of the main comparison (new Appendix C.8) show the number of L1p runs that solve 4 hops growing from 0 of 32 at 25,000 steps to 1 at 50,000, 14 at 75,000 and 18 at 100,000, while no softmax run reaches 4 hops at any evaluation. We then trained both functions for three times the paper's budget, 300,000 steps, with softmax at both the paper's setting and its best grid setting (new Appendix C.14). The table gives, for the first 7 seeds of the main comparison, which all three configurations share, the number of runs that solve 4, 8 and 16 hops at 300,000 steps, and the median step at which the runs that solve 4 hops first do so:

configuration4 hops8 hops16 hopsmedian first step at 4 hops
L1p, paper's setting76467,500
softmax, paper's setting000not reached
softmax, best grid setting552207,000

At the paper's setting, the gap widens with training: on these seeds, L1p goes from solving 4 hops in 2 runs at 100,000 steps to all 7 at 300,000, while softmax does not solve it in any run at either budget (Fisher's exact test at 300,000 steps, p=5.8×10−4p = 5.8 \times 10^{-4}). At its best grid setting, softmax also acquires depth with three times the steps, but L1p gets there first on every seed. It first solves 4 hops at a median of 67,500 steps against 207,000 (exact log-rank test on the step at which 4 hops is first solved, with the 2 softmax runs that never solve it censored at 300,000; p=5.8×10−4p = 5.8 \times 10^{-4}, the smallest possible with 7 runs each), while a complete run of L1p takes only 1.06 times as long as one of softmax on this task, both on the same unfused implementation (Appendix F). At 300,000 steps, the final counts are closer, 7 against 5 at 4 hops and 6 against 5 at 8 hops, a difference that 7 seeds do not resolve (Fisher's exact test, p=0.46p = 0.46 and 1.01.0). In the reviewer's terms, the new runs therefore show faster acquisition of reasoning depth, and the revised abstract states the result for a fixed training budget. Whether softmax would match L1p's final counts with still longer training is left open (Appendix H).

Regarding the 20-layer softmax run that was still improving, it reached a rescaled accuracy of 46.0 at 4 hops at the end of training, just below 50, while 15 of the 16 L1p runs at 20 layers already solve 4 hops. Instead of retraining the 20-layer models, we addressed this question at the 12 layers of the main comparison with the longer runs above.

AuthorsOpenReview ↗

Comment 2 of 2

Regarding the role of learnable sharpness, we agree with the reviewer's reading, and we have revised the paper accordingly. We held the exponent fixed, in the configuration of the main comparisons, at 6, its initial value, and at 8, near the value it reaches when learned (revised Appendix E.2). On language modeling, we selected a learning rate for each fixed value on one seed and trained the better one, 8, on 8 seeds. The table gives the number of runs, out of 8, that solve 4 and 8 hops on reasoning, with the pp-value of Fisher's exact test against the learned exponent on the same seeds in parentheses. On language modeling, it gives the perplexity of the fixed exponent minus that of L1p, with its 95% confidence interval and the pp-value of Welch's tt-test:

exponent4 hops8 hopsperplexity minus L1p's (95% CI)
learned from 6 (L1p)31
fixed at 64 (p=1.0p = 1.0)4 (p=0.28p = 0.28)
fixed at 84 (p=1.0p = 1.0)0 (p=1.0p = 1.0)−0.07-0.07 [−0.20,+0.07][-0.20, +0.07], p=0.32p = 0.32

On reasoning, the exponent fixed at 6 differs from L1p only in not being learned, and it solves at least as many runs. On language modeling, the exponent fixed at 8 has a perplexity 0.07 lower than L1p's (p=0.32p = 0.32) and matches softmax by the paper's criterion, since the 95% confidence interval on their difference, [−0.13,+0.15][-0.13, +0.15], lies within ±0.20\pm 0.20. A fixed exponent thus preserves both results, which answers the reviewer's question affirmatively. The revised abstract, introduction, Table 1 and conclusion no longer present learnability as a property of L1p, and Section 4 motivates it only as a way for each layer to adapt its sharpness during training.

Consistent with the reviewer's suggestion, the reasoning gain depends on powered normalization and on signed attention. An exponent above 1 is necessary: fixed at 1, the exponent does not solve 4 hops in any of 16 runs, whereas fixed at 4 it solves it in 11 (Fisher's exact test, p=6.8×10−5p = 6.8 \times 10^{-5}; E.3). So is the signed form: trained without it, L1p does not solve 4 hops in any of 16 runs, whereas with it L1p solves it in 9 (Fisher's exact test, p=8.2×10−4p = 8.2 \times 10^{-4}; E.6). Trained models also rely on their signs at inference. On a separate question set, keeping the magnitude of each score and dropping its sign in every signed layer leaves no L1p run that solves even 1 hop, and dropping the signs in any single signed layer reduces the runs that solve 4 hops from 19 of 32 to between 3 and 8 (new Appendix E.7).

Regarding the cancellation explanation, we measured cancellation directly in the head outputs (new Appendix E.8). Signed attention cancels part of the summed contribution of the irrelevant tokens, those outside the question and the edges on its path. In L1p, the norm of this contribution is on average 12% smaller than with the signs removed, and the reduction is similar in the runs that solve 4 hops and in those that do not (0.88 and 0.89 of the norm without signs). In line with the reviewer's assessment and Appendix B.6, the revised Section 4 and the new Appendix H therefore state that the mechanism behind the gain remains open.

Regarding the scope of the evidence, our use of a controlled task follows Allen-Zhu (2025, Physics of Language Models, Part 4.1), from whose work the task is adapted. That work argues that at academic scale (e.g., 1.3B parameters and 100B tokens), comparisons of architectures on real benchmarks are often dominated by noise, and that models at this scale fail even the simplest 2-hop reasoning tasks, which masks differences in deeper reasoning. It therefore isolates skills such as reasoning depth in synthetic pretraining tasks, and it finds that several of the trends observed on these tasks also appear in real-world pretraining. This applies all the more to our models, with 55M and 152M parameters. Our task fixes the number of hops exactly and asks for the answer directly, without intermediate steps, so it measures how many hops a model chains within a single forward pass. Consistent with this argument, on the 120 language-modeling checkpoints (new Appendix D.8), every configuration scores below the 3.0% of always giving the most common answer on GSM8K, and on BABILong the differences between L1p and softmax are smaller than the standard deviations across seeds (Welch's tt-test, Holm-adjusted, pp from 0.079 to 0.88), so at 152M parameters these benchmarks do not resolve differences in reasoning depth. At this scale, the controlled task is therefore the most reliable instrument available for separating normalizing functions by reasoning depth, and the revised paper states the claim for this task (Section 4, the conclusion, and the new Appendix H).

We thank the reviewer for these questions, and we hope that the longer-training and fixed-exponent experiments address the reviewer's concerns.

AuthorsOpenReview ↗

Follow-up: matched comparison of fixed and learned exponents

We would like to add a result that completes our answer on learnable sharpness. We held the exponent fixed at 6, its initial value, and trained language models on the same 8 seeds and at the same learning rate as with the learned exponent. This learning rate is also the best of the three that we tried for the fixed exponent, so the two configurations differ only in whether the exponent is learned. With the fixed exponent, the perplexity differs from that with the learned exponent by −0.04-0.04, with a 95% confidence interval of [−0.17,+0.10][-0.17, +0.10] (Welch's tt-test, p=0.57p = 0.57), and from that of softmax by +0.04+0.04, with an interval of [−0.10,+0.18][-0.10, +0.18] (p=0.56p = 0.56). It thus matches both by the paper's criterion. On reasoning, the fixed exponent solves 4 hops in 4 of 8 runs, against 3 for the learned one. An exponent fixed at its initial value therefore preserves both the reasoning depth and the perplexity, which answers the reviewer's question on whether a fixed exponent can close the perplexity gap. We have added this result to Appendix E.2 of the revised submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary and Closing Remarks from the Authors

We thank all reviewers for their careful reading and constructive feedback. Below, we summarize how the revised submission addresses each point.

  • Stronger tuning of softmax (3viF): We extended the tuning of softmax to 15 learning-rate and weight-decay settings. At the best of them, softmax does not solve 4 hops in any of 8 runs, whereas L1p at its own setting solves it in 3 runs and has the higher accuracy on all 8 seeds (Wilcoxon signed-rank test, p=0.0078p = 0.0078; new C.7).
  • Longer training (3viF, D4WF): We trained both L1p and softmax for three times the paper's budget, at which L1p solves 4 hops in all 7 runs, while softmax at the paper's setting solves it in none. Softmax at its best setting also acquires depth, but L1p reaches 4 hops first on every seed, at a median of 67,500 steps against 207,000 (exact log-rank test, p=5.8×10−4p = 5.8 \times 10^{-4}; new C.14). The revised abstract states the main result for a fixed training budget, and the conclusion adds this faster acquisition.
  • Independent evaluation (D4WF): We re-scored every run of the main comparison on new question sets, generated from seeds that neither training nor the original evaluation set uses. On these question sets, L1p solves 4 hops in 21 of 32 runs, and softmax in 0 of 32 (new C.12).
  • The exponent (3viF, D4WF): The submission presented the learnable exponent as a property of L1p. In new experiments, we held the exponent fixed at its initial value, and found that L1p solves 4 hops in at least as many runs as with the learned exponent, and it matches the learned exponent in perplexity (revised E.2). The revised paper accordingly no longer attributes the gain to learning the exponent.
  • Mechanism (3viF, D4WF): We added three analyses: the sources of the change in attention sharpness during training, the removal of the signs at inference, and a direct measurement of cancellation (new B.7, E.7 and E.8). Trained models rely on their signs, but cancellation is similar in runs that solve the task and in those that do not. The revised paper therefore states that the mechanism behind the gain remains open (Section 4 and the new Appendix H).
  • Statistics (D4WF): We verified that runs with the same seed start from identical shared weights and see the same data, and we added tests that pair the runs by seed. These tests lead to the same conclusions as the unpaired ones (new C.11 and a new paragraph in D.2).
  • Natural language (3viF, 3ccj, D4WF): We evaluated the 120 language-modeling checkpoints on GSM8K and BABILong (new D.8). In line with Allen-Zhu (2025), the controlled task remains the most reliable instrument available at this scale for separating normalizing functions by reasoning depth, and the revised paper states the claim for this task (Section 4, the conclusion and the new Appendix H).
  • Presentation (D4WF): We moved the implementation cost and the results on the exponent into Section 5, and we defined the unit of evaluation in Figure 1's caption.

With these additions, the paper offers a comprehensive study of how the normalizing function of attention affects reasoning depth, comparing eight normalizing functions over many seeds with statistical tests throughout. Within this study, L1p reaches greater reasoning depth than softmax on the controlled multi-hop task within the same training budget, while matching softmax in language-modeling perplexity. We thank the reviewers and the chairs for their feedback and time, which have helped us substantially improve the paper, and for the opportunity to discuss and explain our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.