acceptodds
Under review as a conference paper at ICLR 2027

APPLE: Recovering Attention-Attenuated Evidence with a Low-Rank Residual Path

Abstract

Softmax attention forms contextual representations by taking a weighted average of value vectors across tokens for each query, but this aggregation can attenuate token-specific distinctions that remain useful for prediction. We introduce APPLE (Attention-Preserving Path for Local Evidence), a lightweight low-rank branch that can be added to Transformer attention. Given contextual output , APPLE adds a low-rank correction from the local–context residual , where denotes query-aligned local values and equals in standard self-attention. This gives downstream layers access to fine-grained, token-specific information that can be attenuated in the attention output, while retaining the contextual representation. Under the official 300B-token Pythia training recipe, APPLE improves best-validation perplexity at every evaluated scale from 70M to 410M, with reductions of 6.04%, 9.69%, and 0.42%, respectively. The models also improve or tie on most of the 12 zero-shot benchmarks. Complementary controlled experiments show that the attention residual carries label-relevant signals attenuated by attention aggregation, while branch-off and residual-shuffling interventions in trained vision and language models degrade predictions. Across nine prediction settings spanning language, vision, multimodal retrieval, video, audio, and 3D synthesis, APPLE improves matched performance. These results support low-rank access to local–context residuals as a lightweight way to improve prediction while retaining contextual mixing.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
attention mechanisms, transformers, residual representations, low-rank neural networks, efficient neural architectures, language modeling, dense prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wjTNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APPLE, a low-rank residual pathway designed to preserve token-specific information attenuated by attention aggregation. The method adds a learned correction derived from the difference between local value representations and contextual attention outputs, without introducing an additional attention matrix. The authors provide theoretical analysis, controlled synthetic experiments, and residual interventions to investigate the proposed mechanism. Experiments on Pythia models report improvements in validation perplexity, while additional evaluations demonstrate performance gains across language, vision, multimodal, and other prediction tasks. The proposed method introduces a small number of additional parameters, although the magnitude of improvement varies considerably across model sizes and tasks.

Critical strengths
  1. The proposed method has a simple formulation and can be integrated into existing attention modules with minimal architectural changes. The reported parameter and computational overhead is relatively small.

  2. The paper evaluates the method across multiple model families and prediction tasks, including from-scratch language model pretraining. The controlled experiments and residual interventions also provide empirical evidence that the additional pathway contributes to model predictions.

Critical weaknesses
  1. Insufficient baseline comparisons. The experiments primarily compare standard Transformer architectures with their APPLE-augmented counterparts, without comparisons against existing residual attention methods or simpler parameter-matched alternatives. In particular, the absence of a low-rank value bypass baseline is concerning, given that the authors acknowledge its functional equivalence to APPLE under certain conditions. Consequently, it remains unclear whether the improvements come from the proposed residual formulation or simply from adding a learnable pathway.

  2. Limited significance of the theoretical analysis. The theoretical results largely follow from straightforward algebraic properties of the proposed formulation and rely on assumptions such as an orthogonal projection that are not verified for the learned parameters. Furthermore, the synthetic experiments use a fixed averaging operator rather than learned softmax attention, limiting their ability to establish the proposed mechanism in realistic Transformer settings.

  3. Insufficient evidence for robust language modeling improvements. The Pythia experiments use only a single training run per configuration, making it difficult to assess the consistency of the reported improvements. The gain at 410M is relatively small (0.42% in best-validation perplexity), while the 70M baseline exhibits a substantial deterioration in final-validation perplexity. These results raise concerns about training stability and the reliability of the comparisons.

  4. Limited practical significance of several experimental improvements. Although the method is evaluated across diverse model families, several reported improvements are marginal, particularly on T5/XSum and SegFormer/ADE20K. The paper does not consistently provide uncertainty estimates or statistical significance analyses, making it difficult to assess whether these gains are meaningful relative to training variability.

Questions for authors
  1. How does APPLE compare with existing residual attention methods and simpler parameter-matched alternatives, particularly a low-rank value bypass?

  2. Can the authors clarify the significance of the theoretical results and provide evidence that the learned mappings exhibit the proposed properties?

  3. Can the authors provide multiple-seed Pythia results and explain the substantial deterioration in the 70M baseline's final-validation perplexity?

  4. Can the authors provide statistical significance analyses for the relatively small improvements reported across several downstream tasks?

4 comments

Thank you for your careful review and constructive suggestions. We appreciate the time and effort you devoted to evaluating our work, particularly your questions about baseline comparisons, the interpretation of our theoretical results, and the consistency and practical significance of the empirical improvements. Your feedback has helped us clarify the scope of our claims and strengthen the supporting evidence. Below, we address each concern with additional comparisons, analyses, and explanations.

Q1 / W1: Existing methods and matched value bypass

To address your concern regarding baseline comparisons, we added direct comparisons with a parameter-matched value bypass and Twicing Attention (Abdullaev and Nguyen, 2025) under matched training conditions.

We designed a synthetic task requiring both context-based denoising and preservation of small local patterns. Each input is a length-128 signal

\qquad b_i=c+a(i/127-1/2),$$ with independently sampled $c\sim U(-2,2)$ and $a\sim U(-0.2,0.2)$. The motif is $[+1,+1,-1,-1]$ or its sign reversal, with a uniformly sampled starting position from 24 to 120 and $z_i=0$ elsewhere. Independent Gaussian noise has $\sigma\in\{0,0.10,0.25\}$, mixed during training. From the observations alone, the model reconstructs $b_i+0.4z_i$ and predicts background/positive-motif/negative-motif labels defined relative to the background. Copying retains noise, whereas smoothing can weaken the motif; the task therefore requires denoising while preserving local distinctions. We trained all four methods from scratch on the same approximately 9.87M-parameter, eight-layer GPT-NeoX backbone, retaining learned causal softmax attention, standard skips, MLPs, LayerNorm, and RoPE. Each run used 2,048 updates with batch size 128. Backbone initialization and training examples/order were matched within seeds 42, 43, and 44. APPLE and value bypass shared rank 4 per head, the same branch initialization with $U=0$, and 20,480 additional parameters. AdamW learning rates were $3\times10^{-4}$ for the backbone and $10^{-3}$ for the branch. Training comprised 131,072 sequences (32,768 families with four variants each), with 1,536 validation examples. The loss combines MSE normalized by $0.4^2$ and three-class cross-entropy, equally weighting motif and background regions within each loss and excluding the first 16 positions. We evaluated final checkpoints on 3,072 held-out examples, equally split across the three noise levels. The table reports motif-position classification NLL and reconstruction MSE as mean ± SD across the three training seeds, and forward GFLOPs per length-128 sequence. | Model | Trainable parameters | Forward GFLOPs | Motif NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | ---: | ---: | | Baseline | 9,866,244 | 2.6015 | 0.190202 ± 0.002759 | 0.012717 ± 0.000092 | | Value bypass | 9,886,724 | 2.6071 (+0.21%) | 0.191214 ± 0.001832 | 0.012771 ± 0.000090 | | Twicing Attention | 9,866,244 | 2.6867 (+3.27%) | 0.190255 ± 0.003688 | 0.012589 ± 0.000086 | | APPLE | 9,886,724 | 2.6074 (+0.23%) | **0.186989 ± 0.001849** | **0.012430 ± 0.000179** | APPLE’s aggregate motif NLL was 1.69% lower than baseline, 2.21% lower than value bypass, and 1.72% lower than Twicing, with the same direction in all three seeds. APPLE also achieved the lowest mean reconstruction MSE at motif positions. These joint improvements address the intended challenge of the task: distinguishing small local patterns while recovering their clean values from noisy observations. Because APPLE and value bypass share the same branch size, initialization rule, and training budget, the advantage over value bypass supports the usefulness of learning from the local–context residual, beyond simply adding an equally sized pathway. On this controlled task, residual access therefore benefits both local-pattern discrimination and denoising. We additionally evaluated value bypass and Twicing Attention on six GLUE tasks using a reconstruction of the paper's fine-tuning recipe. | Task / metric | Paper baseline | Paper APPLE | Value bypass | Twicing | | --- | ---: | ---: | ---: | ---: | | QNLI accuracy | 91.32 | **91.98** | 91.52 | 91.04 | | CoLA MCC | 57.19 | **59.66** | 58.54 | 51.98 | | MRPC F1 | 89.07 | **89.32** | 88.49 | 86.18 | | SST-2 accuracy | 92.95 | **93.12** | 93.00 | 92.03 | | RTE accuracy | 63.78 | **67.51** | 66.19 | 61.73 | | MNLI accuracy | 84.17 | **84.65** | 84.41 | 84.30 | Together, the synthetic and downstream results support the usefulness of APPLE’s residual formulation beyond the controlled synthetic task.

Q2 / W2: The role of the theory and evidence from learned attention

Thank you for pointing out the need to clarify the role of the projector construction and its connection to general learned corrections. We have added the general low-rank reconstruction-error identity and its improvement condition to Section 3.2 of the revised manuscript, with a derivation in Appendix A.

For one standard self-attention head, write V=V(X)V=V(X), S=A(X)VS=A(X)V for the attention output, R=V−SR=V-S for the residual, and Y=S+RMY=S+RM for the corrected output before output projection. Let Q∈Rd×rQ\in\mathbb{R}^{d\times r} satisfy Q⊤Q=IrQ^\top Q=I_r. Choosing D=QD=Q and U=Q⊤U=Q^\top realizes the orthogonal projector M=P=QQ⊤M=P=QQ^\top, giving

Y=VP+S(I−P).Y=VP+S(I-P).

Consequently, YP=VPYP=VP and Y(I−P)=S(I−P)Y(I-P)=S(I-P): the selected channel subspace preserves the original values exactly, while its orthogonal complement retains the original attention output.

For a general low-rank map M=DUM=DU, no projector assumption is needed for the identities

\qquad \|Y-V\|_F^2-\|S-V\|_F^2 =\|RM\|_F^2-2\langle R,RM\rangle_F.$$ Thus, value-reconstruction error is reduced precisely when $2\langle R,RM\rangle_F>\|RM\|_F^2$: the correction must align sufficiently with the residual without being excessive in magnitude. We do not constrain $DU$ to be a projector because the training objective is task prediction rather than exact reconstruction of $V$; this allows the branch to learn both which residual components to retain and how to transform them for prediction. Additionally, the fixed-averaging experiments you noted were designed to isolate attention-induced attenuation and branch access to the affected information. To address your concern, Q1 / W1 adds matched comparisons with learned softmax attention, while the interventions below test whether the learned branch content contributes to prediction. Building on Q1 / W1, we froze the trained APPLE models and tested branch-content interventions on 2,048 new signal families. After neutralizing motif contrast one layer at a time, we compared correct or reversed contrast from an independent-noise donor with noise matched in norm after output projection. Only branch content was intervened on; the attention, skip, and MLP paths were retained, and subsequent layers were recomputed. The layer-1 results below illustrate the content-specific recovery pattern, with final prediction losses averaged across the three training seeds. | Layer 1 intervention | Motif NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | | Intact branch | 0.281986 | 0.018315 | | Motif contrast neutralized | 1.709315 | 0.045999 | | Correct motif contrast added | 0.292417 | 0.018657 | | Reversed motif contrast added | 4.449763 | 0.086625 | | Norm-matched noise added | 2.015494 | 0.052220 | The content-specific recovery pattern was consistent across most of the tested layers and all three training seeds. Neutralizing the pattern content degraded prediction, while correct-pattern restoration improved both classification NLL and signal-reconstruction MSE and outperformed the opposite-pattern and magnitude-matched-noise controls. These results support the predictive importance of task-appropriate branch content beyond merely injecting an additional signal. Beyond the synthetic setting, the submitted manuscript already examined whether attention attenuates local contrast and APPLE partially restores it in a trained SegFormer-B0 model. Section 5.2 ("Residual reliance") and Figure 4(a), page 7, compare query-aligned local values $V_{\mathrm{local}}$, attention outputs $AV$, and corrected outputs $AV+B$ across four modules in the same checkpoint. Here, $B$ is the APPLE correction, and local contrast (LC) is the mean channel-vector norm of each feature's deviation from its $3\times3$ neighborhood average: | Attention module | $LC(AV)/LC(V_{\mathrm{local}})$ | $LC(AV+B)/LC(AV)$ | | :--- | ---: | ---: | | 0 | 0.1442 | 1.5440 | | 1 | 0.1493 | 1.8078 | | 2 | 0.3244 | 1.7905 | | 3 | 0.3920 | 1.4502 | | Mean | 0.2525 | 1.6482 | Across all four modules, attention reduces local contrast and APPLE partially restores it: $AV$ retains 25.25% of local-value contrast on average, and the correction increases contrast by 64.82% relative to $AV$.

Q3 / W3: Training quality, Pythia variability, and the 70M deterioration

The 70M improvement is not solely a consequence of the baseline's final deterioration. Applying the same best-validation checkpoint-selection criterion to both methods gives Pile validation PPL of 33.195 for baseline and 31.190 for APPLE, a 6.04% reduction. The larger final-checkpoint gap—59.764 versus 31.190 PPL, or 47.81%—is substantially affected by the baseline's deterioration.

Official checkpoints provide an external reference for training quality. Our archived evaluation includes official Hugging Face EleutherAI/pythia-v1.1-{70m,160m} checkpoints. Under the recorded WikiText-2 raw word-perplexity evaluation, all entries below use final checkpoints at step 143,000, rather than best-validation checkpoints:

Model sizeOfficial HF word PPL ↓Our baseline word PPL ↓APPLE word PPL ↓
70M390.208318.513120.303
160M59.80542.85633.964

Our baselines outperform the official references on this evaluation, providing an external check on baseline quality that supports the credibility of our baseline–APPLE comparisons.

Additionally, at 410M, the stronger baseline may leave less predictive headroom, yielding a smaller log-loss reduction and hence a smaller relative PPL gain, 1−e−Δℓ1-e^{-\Delta\ell}, where Δℓ\Delta\ell is the log-PPL reduction under the same evaluation protocol. This modest gain should be viewed alongside the broader improvements across benchmarks.

Q4 / W4: Effect sizes and downstream uncertainty

We would appreciate it if these gains could be considered in the context of the broader improvements observed across model families and tasks. The ADE20K and XSum results complement this evidence: although their absolute gains are modest, APPLE improves in all three paired runs on each task. The tables below report the paired gains and their variability, providing a closer view of how consistently these improvements occur across training seeds.

For ADE20K, we reaggregated seed-level scores and paired improvements from the archived SegFormer runs:

SeedBaseline mIoU ↑APPLE mIoU ↑APPLE − baseline
420.357060.36094+0.00388
430.359250.36184+0.00259
440.360230.36284+0.00261
Mean ± SD0.35885 ± 0.001620.36187 ± 0.00095+0.00303 ± 0.00074

All three seed pairs improved, with a mean paired gain of 0.303 mIoU percentage points (SD 0.074).

For T5/XSum, the three paired runs show the following results:

Seed pairBaseline ROUGE-L ↑APPLE ROUGE-L ↑APPLE − baseline
10.233140.23344+0.00030
20.233370.23390+0.00053
30.232530.23309+0.00056
Mean ± SD0.233013 ± 0.0004340.233477 ± 0.000406+0.000463 ± 0.000142

All three pairs improved. The mean gain is 0.0463 points on the 0–100 ROUGE-L scale, or approximately 0.20% relative to baseline.

Reference

Laziz Abdullaev and Tan M. Nguyen. Transformer Meets Twicing: Harnessing Unattended Residual Information. ICLR 2025.

I thank the authors for the response and the additional comparisons with the low-rank value bypass and Twicing Attention.

However, several concerns remain. First, the controlled comparison, in which APPLE and the value bypass use the same number of additional parameters and the same training budget, is conducted only on a small-scale synthetic task, where APPLE reduces the negative log-likelihood (NLL) by approximately 2%. Second, on GLUE, the value bypass and Twicing Attention results are obtained from the authors’ re-implementation of the fine-tuning setup, whereas the baseline and APPLE results are taken from the paper, so the methods are not compared under identical experimental conditions. Moreover, the value bypass baseline alone already accounts for a large portion of the improvement over the baseline model on several tasks (e.g., RTE and CoLA), which makes it unclear whether the gain stems from the proposed residual formulation or simply from the additional trainable parameters. Third, in the Pythia pretraining experiments, the relative perplexity reduction at the 410M scale is only 0.42%. Finally, the improvements on ADE20K and XSum are small in magnitude (about 0.3 mIoU points and about 0.2% relative improvement in ROUGE-L, respectively), which limits their practical significance.

Reviewer_NFiDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes APPLE, an architectural modification designed to mitigate the attenuation of inherent signals caused by attention. APPLE adds a branch next to each attention module that computes the difference between the value tokens and the attention output, V−AVV - AV. This difference is passed through a low-rank projection, and the result is added back to the attention output AVAV. The authors provide both theoretical and empirical results to validate the effectiveness of APPLE. The evaluation is conducted using relatively small language models like BERT, T5, and Pythia.

Critical strengths
  1. The authors clearly explain why the residual path is expected to help alleviate the variation attenuation. The argument is well supported by intuitive explanations (section 3.2) as well as empirical results (section 4).

  2. Figures effectively deliver main ideas, and thus it is easy to get to know the proposed method. For instance, Figure 1 and 2 well explain how the residual path is designed and which information the residual can hold.

  3. Attenuated signal issue is an interesting and critical problem in machine learning and deep learning community. This study tackles the symptom via relatively simple and practical architectural modifications.

Critical weaknesses

Overall, the idea is simple and clean, but the empirical results do not strongly support the arguments.

  1. [Gap between theory and practice] The paper claims that the proposed residual path is designed to mitigate the attenuation of the signal caused by attention. Section 3.2 provides an intuitive explanation and some theoretical arguments. However, I could not find direct empirical evidence in Section 5 that the residual path actually mitigates this attenuation. How can the authors verify that the improvement stems from reduced attenuation? For example, the authors could measure the signal magnitude (or a comparable attenuation metric) across layers with and without the residual path and report the comparisons.

  2. [Insufficient discussion of observed results] The performance gain becomes smaller as the model size increases. For example, in Table 3, the WikiText-2 PPL difference between Baseline and APPLE is nearly 200 for 70M model, while it becomes 9 for 160M model and 0.35 for 410M model. However, the authors do not provide in-depth discussions regarding this critical observation. Could the authors explain any potential reasons for this decreases in performance?

  3. [Low baseline performance] Several benchmarks in Table 3 are at or below chance level at all three scales. For example, ARC-C stays around 0.22–0.26 (chance 0.25), WinoGrande around 0.50–0.52 (chance 0.50), and LogiQA around 0.25–0.28 (chance 0.25). Particularly, all WSC scores fall below the majority-class baseline of 63.5%, and the value 0.3654 at 160M and 410M shows that both models predict a single class. Since these results are close to the chance level, the performance differences provide little information about model quality. Moreover, several 'wins' correspond to one or a few examples (e.g., 1/651 on LogiQA and 1/1511 on xStoryCloze at 410M). This level of improvement cannot be considered as a meaningful performance gain.

  4. [Strange baseline accuracy values] In Table 4, the authors report 0.6378 of RTE accuracy for baseline. However, RTE dev includes 277 questions, and 0.6378 is in between 176/277 (0.6354) and 177/277 (0.6390). Likewise, SST-2 dev contains 872 questions, and thus 0.9295 is in the middle of 810/872 (0.9289) and 811/827 (0.9300). The authors should justify how the baseline accuracy values were calculated for the baseline.

  5. [Additional validation required] APPLE adds more parameters, although the count is not large. Thus, it should be validated whether the additional parameters improve the performance rather than APPLE's residual design. Because the observed performance gain is not large, this validation plays a critical role in making the proposed ideas convincing.

  6. [Only a few small models used] The authors used only a few small models like BERT, T5, and Pythia with three different sizes. Could we expect the same performance improvement from larger LLMs like GPT, LLaMA, and QWen? The authors used RTX 4090, and 3B models can be trained on 4090 without having memory issues.

Questions for authors

All my questions an concerns are summarized in the above weakness section. Please carefully check and address them.

4 comments

Thank you for your careful review and constructive suggestions.

W1: Direct measurements of attenuation and the predictive relevance of residual content

Thank you for requesting direct measurements. Section 5.2 ("Residual reliance") and Figure 4(a), page 7, report local-contrast measurements in a trained SegFormer-B0, comparing features before and after attention aggregation and after the APPLE correction.

Local contrast (LC) is the mean channel-vector norm of each feature's deviation from its 3×33\times3 neighborhood average. In the same checkpoint, we compare the query-aligned local values VlocalV_{\mathrm{local}}, the attention output AVAV, and AV+BAV+B, where BB is the APPLE correction, across the four modules in Figure 4(a):

Attention moduleLC(AV)/LC(Vlocal)LC(AV)/LC(V_{\mathrm{local}})LC(AV+B)/LC(AV)LC(AV+B)/LC(AV)
00.14421.5440
10.14931.8078
20.32441.7905
30.39201.4502
Mean0.25251.6482

Across all four modules, attention reduces local contrast and APPLE partially restores it: AVAV retains 25.25% of local-value contrast on average, and the correction increases contrast by 64.82% relative to AVAV.

Increased local contrast alone does not establish predictive relevance. We tested whether the content carried by the learned residual branch contributes to prediction through targeted neutralization and restoration. We designed a synthetic task requiring both context-based denoising and preservation of small local patterns. The observed signal is

xi=bi+0.4zi+ϵi,x_i=b_i+0.4z_i+\epsilon_i,

Each input is a length-128 signal. The background is bi=c+a(i/127−1/2)b_i=c+a(i/127-1/2), with independently sampled c∼U(−2,2)c\sim U(-2,2) and a∼U(−0.2,0.2)a\sim U(-0.2,0.2). The local pattern is [+1,+1,−1,−1][+1,+1,-1,-1] or its sign reversal, placed at a uniformly sampled starting position from 24 to 120. Noise is independent across positions, with ϵi∼N(0,σ2)\epsilon_i\sim\mathcal N(0,\sigma^2) and σ∈{0,0.10,0.25}\sigma\in\{0,0.10,0.25\}. The model sees only the observed signal and jointly predicts the clean signal bi+0.4zib_i+0.4z_i and background/positive-pattern/negative-pattern labels. These labels describe deviations from the background, not whether the observed value is above or below zero. Copying the input retains noise, whereas indiscriminate smoothing can weaken the pattern; prediction therefore requires denoising while retaining local distinctions.

Training protocol. We trained approximately 9.87M-parameter, eight-layer GPT-NeoX models from scratch, retaining learned causal softmax attention, standard skips, MLPs, LayerNorm, and RoPE. The APPLE and value-bypass branches use rank 4 per head with U=0U=0 initialization. Each model used seeds 42, 43, and 44, 2,048 optimizer updates, batch size 128, and sequence length 128. Backbone initialization, data order, and budget were matched, and final checkpoints were compared. AdamW learning rates were 3×10−43\times10^{-4} for the backbone and 10−310^{-3} for the branch.

Training used 131,072 sequences (32,768 families with four variants each); validation and the main test used 1,536 and 3,072 original examples, respectively. The loss combines MSE normalized by 0.420.4^2 and three-class cross-entropy, weighting pattern and background regions equally within each loss and excluding the first 16 context positions.

Restoration evaluation. We evaluated 2,048 new signal families, distinct from the training and original evaluation sets. Noise levels 0.10 and 0.25 each account for 1,024 families. With model weights frozen, we neutralized pattern contrast at one layer at a time by averaging branch contributions from paired opposite-sign examples at the known pattern positions. We then compared restoration using correct or opposite pattern contrast from an independent-noise donor, or noise matched in norm after the output projection. The intervened layer's input, attention aggregation, skip connection, and parallel MLP were retained; subsequent layers were recomputed normally.

The layer-1 pattern-classification NLL, averaged across the three training seeds, was:

ConditionPattern-classification NLL ↓
Original APPLE output0.28199
Branch pattern contrast neutralized1.70932
Correct-pattern content restored0.29242
Opposite-pattern content restored4.44976
Matched-norm noise added2.01549

The content-specific recovery pattern was consistent across most of the tested layers and all three training seeds. Neutralizing the pattern content degraded prediction, while correct-pattern restoration improved both classification NLL and signal-reconstruction MSE and outperformed the opposite-pattern and magnitude-matched-noise controls. These results support the predictive importance of task-appropriate branch content beyond merely injecting an additional signal. Figure 4(a) measures attenuation and partial recovery of local contrast, while the intervention experiment shows that task-relevant branch content contributes to prediction across multiple depths.

W2: Size dependence of language-model gains

The scale dependence should be interpreted in terms of the predictive headroom remaining in each baseline, together with the PPL scale. When a baseline assigns insufficient probability to the correct token, there is substantial room for a correction to improve the prediction. At positions where a stronger baseline already assigns high probability to the correct token, the available probability increase is more limited, even when the correction is useful.

This distinction can be expressed directly in terms of NLL. At a prediction position, let pp be the baseline probability of the correct token and δ>0\delta>0 the increase produced by a correction. The NLL improvement is

=\log(p+\delta)-\log p =\log\left(1+\frac{\delta}{p}\right).$$ The same probability increase yields a smaller NLL reduction when the baseline probability is higher, and the possible increase is itself bounded by $\delta\leq 1-p$. Thus, a smaller incremental gain over a stronger baseline does not, by itself, establish that the correction pathway has ceased to be useful. The exponential PPL scale further affects absolute differences. Under the same evaluation and normalization, $$\mathrm{PPL}_{\mathrm{base}}-\mathrm{PPL}_{\mathrm{APPLE}} =\mathrm{PPL}_{\mathrm{base}}\left(1-e^{-\Delta\ell}\right).$$ Consequently, the same NLL reduction produces a larger absolute PPL decrease when baseline PPL is higher. Greater predictive headroom and a higher baseline PPL can therefore both contribute to larger observed gains at smaller scales. We therefore interpret the results relative to each baseline's predictive level and the additional improvement delivered by APPLE. The probability argument explains how diminishing headroom can produce smaller gains. **W3: Absolute benchmark performance and cross-scale comparisons** We acknowledge the low absolute performance and small score differences on some of the cited tasks. Our aim is to characterize the pattern of APPLE’s effects across model scales within each task, considering both the direction and magnitude of the differences. Individual scores should therefore be interpreted alongside the broader cross-scale pattern, while distinguishing absolute task performance from incremental improvements over the matched baseline. ARC-C, WinoGrande, and LogiQA remain near their respective random-choice baselines at the evaluated scales. For example, LogiQA accuracies range from 25.19% to 28.42% against a uniform random-choice baseline of 25%. In contrast, xStoryCloze is near chance at 70M but reaches higher absolute accuracy at the larger scales. These absolute levels should be considered alongside the APPLE–baseline differences across scales: | Task | Accuracy change at 70M | Accuracy change at 160M | Accuracy change at 410M | | :--- | ---: | ---: | ---: | | ARC-C | +0.60 pp | −2.39 pp | 0.00 pp | | WinoGrande | +1.02 pp | 0.00 pp | +0.56 pp | | LogiQA | +0.77 pp | +2.15 pp | +0.16 pp | | xStoryCloze | +1.72 pp | +0.46 pp | +0.06 pp | LogiQA and xStoryCloze show positive differences at all three scales, WinoGrande shows small improvements or a tie, and ARC-C shows a mixed pattern of improvement, degradation, and a tie. We agree that the 410M differences on LogiQA and xStoryCloze each correspond to one additional correct answer; the differences at the smaller scales are not all limited to one example. This directional consistency does not establish statistical significance. Viewed together, the comparisons identify where improvements persist, where their magnitude becomes small, and where the effect is inconsistent, providing a fuller assessment than interpreting each numerical difference in isolation. In particular, increasing absolute accuracy with model scale and an incremental advantage over the matched baseline are distinct observations. We acknowledge that all reported WSC scores fall below the majority-class baseline of approximately 63.46%. However, inspection of the prediction distributions revealed different distributions even among models with the same reported accuracy of 0.3654. The repeated score therefore does not establish that these models all predict a single class. We recognize that aggregate improvement counts and highlighting in the manuscript may make differences of varying magnitude and evidential strength appear equally conclusive. We will revise the tables and discussion to make absolute performance, improvement magnitudes, and cross-scale patterns explicit, and adjust the wording so that small numerical differences are not interpreted as definitive superiority.

W4: Aggregation of RTE and SST-2 accuracies

The reported baseline accuracies are averages of validation-selected accuracies across training seeds, not single-run scores. RTE uses three seeds, and SST-2 uses two. The recorded per-seed accuracies correspond to the following calculations:

=\frac{178+175+177}{3\times277} =0.6377858\ldots\approx0.6378,$$ $$\mathrm{SST\text{-}2} =\frac{810+811}{2\times872} =0.9294725\ldots\approx0.9295.$$ The reported values are therefore explained by averaging across seeds and rounding. An average over multiple runs is not restricted to the $1/N$ increments possible for a single run, so the cited values do not constitute an arithmetic inconsistency. **W5: Separating additional-parameter effects from the residual formulation** To distinguish the effect of additional parameters from the contribution of the residual formulation, we compared baseline, a parameter-matched value bypass, and APPLE using the synthetic task and training protocol described in W1. All three models use the same GPT-NeoX backbone and differ only in the additional attention pathway: $$Y_{\mathrm{base}}=AV,\qquad Y_{\mathrm{bypass}}=AV+VDU,\qquad Y_{\mathrm{APPLE}}=AV+(V-AV)DU.$$ The value bypass transforms the local values $V$ directly, whereas APPLE transforms the local–context residual $V-AV$. The two branches use the same rank, parameter count, insertion point, and initialization rule, allowing us to compare the effect of providing an additional learnable pathway with the effect of using the residual as its input. Within each seed, backbone initialization, training examples and order, and training budget were matched; final checkpoints were compared at that common budget. | Model | Parameters | Pattern-classification NLL ↓ | Pattern-reconstruction MSE ↓ | | :--- | ---: | ---: | ---: | | Baseline | 9,866,244 | 0.190202 ± 0.002759 | 0.012717 ± 0.000092 | | Value bypass | 9,886,724 | 0.191214 ± 0.001832 | 0.012771 ± 0.000090 | | APPLE | 9,886,724 | **0.186989 ± 0.001849** | **0.012430 ± 0.000179** | The table reports metrics at the four pattern positions on the 3,072 original test examples (1,024 per noise level), averaged across noise conditions, with mean ± standard deviation across the three training seeds. Classification NLL measures local-pattern discrimination, while reconstruction MSE measures reconstruction of the clean signal in the presence of noise. The parameter-matched value bypass did not improve either mean metric over the baseline, whereas APPLE improved both. APPLE's mean classification NLL was 2.21% lower than the value bypass. This shows explicitly transforming the local–context difference learned more effectively than directly transforming local values through an equally sized additional pathway. The observed advantage therefore cannot be explained by additional parameter count alone and supports the utility of the residual parameterization. **W6: Scaling to larger language models and computational cost** Based on the existing experimental results, we expect APPLE to improve predictive performance in larger language models as well. Validation PPL improved at every evaluated Pythia scale—70M, 160M, and 410M—under the same 300B-token pretraining protocol, and predictive improvements were also observed across different model families. APPLE provides access to the local–context residual within each attention head, and this formulation is not restricted to a particular model scale. We appreciate the suggestion to evaluate larger models and agree that this would be a valuable extension. To provide context on the resources involved, our experiments use matched full pretraining with a 300B-token budget. Training the 410M model under this protocol required approximately two months on four RTX 4090 GPUs. Extending the same comparison to a 3B model would involve full training and evaluation of both APPLE and a matched baseline, adding a substantial computational commitment. Alongside memory requirements, the time needed to complete these comparative runs is therefore an important practical consideration. We regard larger-model evaluation as a worthwhile direction and hope that the measured training cost helps clarify the scope of the current study.

Thank you for the rebuttal. W1 and W4 have6 been addressed well, however, most of the concerns still remain the same.

W2. The authors explanation generally makes sense. However, it applies to any method and does not show that it explains the diminishing gains of APPLE specifically. An analysis of NLL improvement stratified by the baseline probability of the correct token would directly verify whether the gains are limited by headroom rather than by reduced usefulness of the correction at larger scales.

W3. Without significance tests or confidence intervals, most of the cited differences remain within noisy range, and ARC-C does not even show a consistent direction. Thus, I still think those benchmarks do not strongly support APPLE's benefits.

W6. I understand the computational constraints and do not expect full 3B pretraining during the rebuttal. However, the claim that APPLE will improve larger models is not supported by the current evidence, especially given the shrinking gains with scale discussed in W2.

For the above reasons, I will keep my current score.

Reviewer_yK5xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper claims that attention-based weighted averaging of value vectors attenuates fine-grained token-specific information. The authors propose APPLE, a lightweight module that adds a low-rank correction to the original attention output in each attention head. The correction is based on the residual between the local value and the attention output. APPLE initializes the branch to zero correction, so each branch starts by matching the original attention output. Under a fixed-attention assumption, the authors show that the low-rank correction can preserve token differences within selected channel subspaces. Synthetic tasks and intervention experiments on trained models further suggest that predictions depend on position-aligned residual information. On Pythia models trained with the official setup, APPLE reduces best-validation perplexity at three scales. It also improves or ties on most of the 12 reported zero-shot tasks.

Critical strengths
  1. The method is simple and easy to deploy. Zero initialization preserves the original model's behavior at the start of training. The authors also implement a fused kernel.

  2. The experiments are extensive. They cover nine settings across language, 2D/3D vision, multimodal tasks, etc. The proposed method shows promising results across these settings. The authors also report parameter, latency, and memory overhead.

  3. The paper is generally well-organized.

Critical weaknesses
  1. The paper argues that attention's convex combination attenuates token-specific predictive detail. However, in a standard Transformer, the residual connection x+Attn(x)x+\mathrm{Attn}(x) already passes the current token's representation to subsequent layers, bypassing attention. The paper does not analyze it. The authors should explain why the bottleneck persists despite skip connections.

  2. Novelty seems Limited, and Appendix A complicates the claimed mechanism. Appendix A shows that, when I−MI-M is invertible and WoW_o is trainable, [AV+(V−AV)M]Wo=[AV+VN~]W~o[AV+(V-AV)M]W_o=[AV+V\tilde N]\tilde W_o with rank⁡(N~)=rank⁡(M)\operatorname{rank}(\tilde N)=\operatorname{rank}(M), so APPLE does not enlarge the function class and its gains must reflect optimization or inductive bias rather than added capacity.

  3. The results behind Contribution 2 are proved under a fixed AA, whereas in a trained Transformer A=softmax⁡(QK⊤/d)A=\operatorname{softmax}(QK^\top/\sqrt{d}) is input-dependent and optimized jointly with M=UDM=UD, and exact preservation requires MM to be an orthogonal projection, a constraint APPLE's M=UDM=UD does not enforce, hence the claimed property is not guaranteed for the trained model.

Questions for authors

How does APPLE compare with AV+VDUAV+VDU (a low-rank value bypass) under matched Pythia pretraining settings? This is the core ablation for determining whether the residual parameterization is meaningful.

3 comments

Thank you for your careful review and constructive suggestions.

W1: Token skip versus value-space residual

We agree that the standard skip connection carries the current token representation to subsequent layers. Our claim concerns the benefit of explicitly processing local–context differences when that information is already available through the existing paths. The baseline's learned attention and MLP may also learn to exploit this information; APPLE provides a direct, low-rank transformation of the current head's residual, V−AVV-AV. We therefore test whether this explicit parameterization improves prediction with the skip, attention, and MLP paths retained.

To address your concern about whether APPLE provides additional benefit beyond the standard skip connection, we conducted an additional experiment on a task requiring context-based denoising while preserving small local patterns. Each length-128 input is xi=bi+0.4zi+ϵix_i=b_i+0.4z_i+\epsilon_i, with a slowly varying background bib_i, a four-position motif ziz_i, and Gaussian noise ϵi\epsilon_i. From the observation alone, the model reconstructs bi+0.4zib_i+0.4z_i and predicts background/positive-motif/negative-motif labels defined relative to the background.

We evaluated this task using the same eight-layer GPT-NeoX backbone, retaining its learned causal softmax attention, standard parallel-residual skip, MLP, and input LayerNorms. Let XX denote the block input, F(X)F(X) the MLP path, and WOW_O the attention output projection; suppressing LayerNorm and head-concatenation notation,

\qquad H_{\mathrm{APPLE}}=X+[AV+(V-AV)M]W_O+F(X).$$ The skip term $X$ and MLP path remain in both formulations; APPLE adds $(V-AV)MW_O$. Since softmax attention weights sum to one, $$v_i-\sum_j a_{ij}v_j=\sum_j a_{ij}(v_i-v_j).$$ Thus, the skip carries the block input forward, while APPLE learns how to transform the weighted differences between each token's value and the contextual values it attends to. The comparison tests the additional benefit of this explicit difference processing with the existing paths retained. We trained baseline, value bypass $AV+VDU$, and APPLE from scratch. Within each of three seeds, backbone initialization, training data/order, and the budget of 2,048 updates at batch size 128 were matched. Value bypass controls for adding an equally sized learnable pathway: it shares APPLE's rank, branch initialization, and additional parameter count, but reads $V$ rather than $V-AV$. Table 1 reports motif-position losses on 3,072 held-out examples pooled across the three noise levels, as mean ± SD across seeds. **Table 1. Matched training comparison** | Model | Trainable parameters | Motif classification NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | ---: | | Baseline | 9,866,244 | 0.190202 ± 0.002759 | 0.012717 ± 0.000092 | | Value bypass | 9,886,724 | 0.191214 ± 0.001832 | 0.012771 ± 0.000090 | | APPLE | 9,886,724 | **0.186989 ± 0.001849** | **0.012430 ± 0.000179** | APPLE had the lowest mean loss on both metrics. Its motif NLL was 1.69% lower than baseline and 2.21% lower than value bypass, with both comparisons favoring APPLE in all three seeds. This comparison supports an additional predictive benefit from the residual parameterization in this task, even with the standard skip and MLP available. To test whether the trained models use the branch’s motif content, we froze the APPLE models and evaluated them on 2,048 new signal families. With the skip and MLP in place, we neutralized the branch’s motif contrast at known motif positions, one layer at a time. From the same neutralized state, we added correct or reversed motif contrast from an example with independent noise, or a norm-matched noise control. Table 2 gives the final prediction losses for interventions at layer 1, averaged across the three seeds. **Table 2. Layer-1 branch intervention** | Branch condition | Motif classification NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | | Intact branch | 0.281986 | 0.018315 | | Motif contrast neutralized | 1.709315 | 0.045999 | | Correct motif contrast added | 0.292417 | 0.018657 | | Reversed motif contrast added | 4.449763 | 0.086625 | | Norm-matched noise added | 2.015494 | 0.052220 | Adding the correct motif contrast brought both losses close to their intact values. Reversed contrast and norm-matched noise raised both losses further. This content-specific recovery pattern was consistent across most of the tested layers and all three training seeds: correct restoration improved both metrics over neutralization and outperformed the reversed-contrast and norm-matched-noise controls. The intervention shows that the trained APPLE models use the motif content carried by the branch even with the standard skip in place. Together, Table 1 tests the benefit of adding the residual path during training, while Table 2 tests the use of its motif content after training.

W2: Function class, novelty, and matched value bypass

We agree with the conditional equivalence established in Appendix A: when I−MI-M is invertible and the output projection is freely trainable, APPLE can be reparameterized as a value bypass of the same rank. However, functional equivalence does not imply equivalent learning dynamics. The reparameterization also changes the output projection; it does not make training the two parameterizations with the same optimizer follow equivalent trajectories. Optimization and inductive bias are therefore central to our contribution: APPLE explicitly couples the local-value correction to a correction of the contextual attention output through the same low-rank map.

The equivalence in Appendix A is a coupled change of parameters,

N=M(I−M)−1,W~O=(I−M)WO.N=M(I-M)^{-1},\qquad \widetilde W_O=(I-M)W_O.

It preserves the represented function but does not identify optimizer updates in the two parameterizations. The difference can be seen directly in the gradients of the low-rank factors actually trained by APPLE.

Let S=AVS=AV, R=V−SR=V-S, and Y=S+RDUY=S+RDU. For a scalar training loss L\mathcal L, define G=∂L/∂YG=\partial\mathcal L/\partial Y, including the gradient propagated through the output projection and subsequent computation. The chain rule gives

\nabla_D\mathcal L=R^\top G U^\top,\qquad \left.\nabla_S\mathcal L\right|_{V}=G(I-DU)^\top.$$ For value bypass, $Y_b=S+VD_bU_b$, the corresponding gradients are $D_b^\top V^\top G_b$, $V^\top G_bU_b^\top$, and $G_b$, respectively. The derivative with respect to $S$ isolates the attention-output path, holding $V$ fixed. In particular, consider matched initialization with the same backbone and output projection, $D_b=D$, and $U_b=U=0$. Both methods then have the same output and upstream gradient $G$, and both have zero initial gradient for $D$. Nevertheless, their initial gradients for the zero-initialized factor differ by $$\bigl(\nabla_U\mathcal L\bigr)_{\mathrm{APPLE}} - \bigl(\nabla_{U_b}\mathcal L\bigr)_{\mathrm{bypass}} = -D^\top S^\top G.$$ Unless this term vanishes, the two branches already receive different learning signals at initialization despite representing the same function. APPLE subtracts the contextual contribution from the factor gradient; as $DU$ evolves, it also modifies gradient propagation through the attention-output path. This explicitly identifies the optimization difference induced by residual parameterization. The training comparisons below assess its predictive benefit. To evaluate the residual parameterization beyond the controlled task in Table 1, we additionally evaluated value bypass on six GLUE tasks using a reconstruction of the paper's fine-tuning recipe. The table compares these results with the baseline and APPLE scores reported in the paper. **Table 3. GLUE comparison with value bypass** | Task / metric | Paper baseline | Paper APPLE | Value bypass | | --- | ---: | ---: | ---: | | QNLI accuracy | 91.32 | **91.98** | 91.52 | | CoLA MCC | 57.19 | **59.66** | 58.54 | | MRPC F1 | 89.07 | **89.32** | 88.49 | | SST-2 accuracy | 92.95 | **93.12** | 93.00 | | RTE accuracy | 63.78 | **67.51** | 66.19 | | MNLI accuracy | 84.17 | **84.65** | 84.41 | APPLE scores higher than value bypass on all six tasks, with differences ranging from 0.12 points on SST-2 to 1.32 points on RTE. In the controlled comparison in Table 1, APPLE also achieves 2.21% lower mean motif NLL than value bypass, with the same direction in all three seeds while matching branch size, initialization, and training budget. Together, these results support the practical value of learning the correction from the local–context residual: the choice of branch input matters, and the observed advantage extends beyond the synthetic task.

W3: Theory assumptions and general learned corrections

Thank you for pointing out the need to clarify the connection between the projector construction and general learned corrections. We have added the general reconstruction-error identity and its improvement condition to Section 3.2 of the revised manuscript, with a derivation in Appendix A.

For one standard self-attention head, write V=V(X)V=V(X), S=A(X)VS=A(X)V for the attention output, R=V−SR=V-S for the residual, and Y=S+RMY=S+RM for the corrected output before output projection. With Q⊤Q=IrQ^\top Q=I_r, choosing D=QD=Q and U=Q⊤U=Q^\top realizes the orthogonal rank-rr projector M=P=QQ⊤M=P=QQ^\top and gives

Y=VP+S(I−P),YP=VP,Y(I−P)=S(I−P).Y=VP+S(I-P),\qquad YP=VP,\qquad Y(I-P)=S(I-P).

Thus, the selected channel subspace preserves the original values, while its orthogonal complement retains the original attention output. These identities hold pointwise for input-dependent attention; they do not require one fixed attention matrix across inputs. The separate pairwise-separation result retains its fixed-AA assumption.

For a general low-rank map M=DUM=DU, no projector assumption is needed for

\qquad \|Y-V\|_F^2-\|S-V\|_F^2 =\|RM\|_F^2-2\langle R,RM\rangle_F.$$ Value-reconstruction error therefore decreases precisely when $2\langle R,RM\rangle_F>\|RM\|_F^2$: the correction must align sufficiently with the residual without being excessive in magnitude. We do not constrain $DU$ to be a projector because the training objective is task prediction rather than exact reconstruction of $V$, allowing the branch to learn both which residual components to retain and how to transform them for prediction. The interventions in W1 evaluate the predictive role of these learned residual transformations. With the standard skip and MLP paths retained, restoring the correct motif contrast improves both motif classification NLL and signal-reconstruction MSE relative to neutralization, reversed contrast, and norm-matched noise across most of the tested layers and all three training seeds. These results support the predictive utility of the learned branch content. **Q1: Pythia comparison with low-rank value bypass** To address your question, we trained Pythia-70M with APPLE and a low-rank value bypass from scratch for 100B tokens each. The table compares their final and validation-best checkpoints. Under matched evaluation, APPLE achieved lower token perplexity on the shared Pile validation subset and lower word perplexity on WikiText-2 under both checkpoint-selection schemes. Final denotes the checkpoint at the end of training; best denotes the checkpoint with the lowest validation perplexity on The Pile. Benchmark scores are on a 0–100 scale; the mean is unweighted across the 12 benchmarks. Win/tie/loss counts compare APPLE with value bypass across these 12 benchmarks and exclude perplexity. | Benchmark / metric | Value bypass, final | APPLE, final | Value bypass, best | APPLE, best | | --- | ---: | ---: | ---: | ---: | | ARC-C | 17.24 | **18.17** | 16.81 | **18.00** | | ARC-E | **38.09** | 38.05 | **39.35** | 37.96 | | PIQA | 57.89 | **59.52** | **59.41** | 59.09 | | SciQ | 64.70 | **65.60** | 65.70 | **67.40** | | WinoGrande | **51.22** | 50.75 | 49.80 | **50.67** | | WSC | 36.54 | **46.15** | 36.54 | 36.54 | | LAMBADA | **23.33** | 23.29 | 21.72 | **22.71** | | LogiQA | 19.20 | **22.43** | 20.12 | **24.88** | | BLiMP | 73.33 | **73.66** | 74.04 | **74.26** | | SWAG | 34.97 | **35.15** | 34.90 | **34.98** | | HellaSwag | 26.83 | **27.05** | **26.75** | 26.64 | | xStoryCloze (English) | **52.68** | 52.48 | **52.75** | 51.36 | | Shared Pile validation subset token PPL ↓ | 17.706 | **17.629** | 17.697 | **17.546** | | WikiText-2 word PPL ↓ | 60.738 | **60.043** | 60.727 | **59.782** | | Mean benchmark score ↑ | 41.33 | **42.69** | 41.49 | **42.04** | | APPLE wins / ties / losses vs. value bypass | — | **8 / 0 / 4** | — | **7 / 1 / 4** | APPLE achieved a higher mean benchmark score under both checkpoint-selection schemes, winning 8 of 12 benchmarks at the final checkpoints and 7 of 12, with one tie, at the validation-best checkpoints. The matched evaluation also favored APPLE on both perplexity measures. We would appreciate it if you could consider these Pythia results together with the GLUE comparisons above when assessing the benefits of APPLE’s residual parameterization over value bypass.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.