- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes APPLE, a low-rank residual pathway designed to preserve token-specific information attenuated by attention aggregation. The method adds a learned correction derived from the difference between local value representations and contextual attention outputs, without introducing an additional attention matrix. The authors provide theoretical analysis, controlled synthetic experiments, and residual interventions to investigate the proposed mechanism. Experiments on Pythia models report improvements in validation perplexity, while additional evaluations demonstrate performance gains across language, vision, multimodal, and other prediction tasks. The proposed method introduces a small number of additional parameters, although the magnitude of improvement varies considerably across model sizes and tasks.
Critical strengths
-
The proposed method has a simple formulation and can be integrated into existing attention modules with minimal architectural changes. The reported parameter and computational overhead is relatively small.
-
The paper evaluates the method across multiple model families and prediction tasks, including from-scratch language model pretraining. The controlled experiments and residual interventions also provide empirical evidence that the additional pathway contributes to model predictions.
Critical weaknesses
-
Insufficient baseline comparisons. The experiments primarily compare standard Transformer architectures with their APPLE-augmented counterparts, without comparisons against existing residual attention methods or simpler parameter-matched alternatives. In particular, the absence of a low-rank value bypass baseline is concerning, given that the authors acknowledge its functional equivalence to APPLE under certain conditions. Consequently, it remains unclear whether the improvements come from the proposed residual formulation or simply from adding a learnable pathway.
-
Limited significance of the theoretical analysis. The theoretical results largely follow from straightforward algebraic properties of the proposed formulation and rely on assumptions such as an orthogonal projection that are not verified for the learned parameters. Furthermore, the synthetic experiments use a fixed averaging operator rather than learned softmax attention, limiting their ability to establish the proposed mechanism in realistic Transformer settings.
-
Insufficient evidence for robust language modeling improvements. The Pythia experiments use only a single training run per configuration, making it difficult to assess the consistency of the reported improvements. The gain at 410M is relatively small (0.42% in best-validation perplexity), while the 70M baseline exhibits a substantial deterioration in final-validation perplexity. These results raise concerns about training stability and the reliability of the comparisons.
-
Limited practical significance of several experimental improvements. Although the method is evaluated across diverse model families, several reported improvements are marginal, particularly on T5/XSum and SegFormer/ADE20K. The paper does not consistently provide uncertainty estimates or statistical significance analyses, making it difficult to assess whether these gains are meaningful relative to training variability.
Questions for authors
-
How does APPLE compare with existing residual attention methods and simpler parameter-matched alternatives, particularly a low-rank value bypass?
-
Can the authors clarify the significance of the theoretical results and provide evidence that the learned mappings exhibit the proposed properties?
-
Can the authors provide multiple-seed Pythia results and explain the substantial deterioration in the 70M baseline's final-validation perplexity?
-
Can the authors provide statistical significance analyses for the relatively small improvements reported across several downstream tasks?
4 comments
Q2 / W2: The role of the theory and evidence from learned attention
Thank you for pointing out the need to clarify the role of the projector construction and its connection to general learned corrections. We have added the general low-rank reconstruction-error identity and its improvement condition to Section 3.2 of the revised manuscript, with a derivation in Appendix A.
For one standard self-attention head, write , for the attention output, for the residual, and for the corrected output before output projection. Let satisfy . Choosing and realizes the orthogonal projector , giving
Consequently, and : the selected channel subspace preserves the original values exactly, while its orthogonal complement retains the original attention output.
For a general low-rank map , no projector assumption is needed for the identities
\qquad \|Y-V\|_F^2-\|S-V\|_F^2 =\|RM\|_F^2-2\langle R,RM\rangle_F.$$ Thus, value-reconstruction error is reduced precisely when $2\langle R,RM\rangle_F>\|RM\|_F^2$: the correction must align sufficiently with the residual without being excessive in magnitude. We do not constrain $DU$ to be a projector because the training objective is task prediction rather than exact reconstruction of $V$; this allows the branch to learn both which residual components to retain and how to transform them for prediction. Additionally, the fixed-averaging experiments you noted were designed to isolate attention-induced attenuation and branch access to the affected information. To address your concern, Q1 / W1 adds matched comparisons with learned softmax attention, while the interventions below test whether the learned branch content contributes to prediction. Building on Q1 / W1, we froze the trained APPLE models and tested branch-content interventions on 2,048 new signal families. After neutralizing motif contrast one layer at a time, we compared correct or reversed contrast from an independent-noise donor with noise matched in norm after output projection. Only branch content was intervened on; the attention, skip, and MLP paths were retained, and subsequent layers were recomputed. The layer-1 results below illustrate the content-specific recovery pattern, with final prediction losses averaged across the three training seeds. | Layer 1 intervention | Motif NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | | Intact branch | 0.281986 | 0.018315 | | Motif contrast neutralized | 1.709315 | 0.045999 | | Correct motif contrast added | 0.292417 | 0.018657 | | Reversed motif contrast added | 4.449763 | 0.086625 | | Norm-matched noise added | 2.015494 | 0.052220 | The content-specific recovery pattern was consistent across most of the tested layers and all three training seeds. Neutralizing the pattern content degraded prediction, while correct-pattern restoration improved both classification NLL and signal-reconstruction MSE and outperformed the opposite-pattern and magnitude-matched-noise controls. These results support the predictive importance of task-appropriate branch content beyond merely injecting an additional signal. Beyond the synthetic setting, the submitted manuscript already examined whether attention attenuates local contrast and APPLE partially restores it in a trained SegFormer-B0 model. Section 5.2 ("Residual reliance") and Figure 4(a), page 7, compare query-aligned local values $V_{\mathrm{local}}$, attention outputs $AV$, and corrected outputs $AV+B$ across four modules in the same checkpoint. Here, $B$ is the APPLE correction, and local contrast (LC) is the mean channel-vector norm of each feature's deviation from its $3\times3$ neighborhood average: | Attention module | $LC(AV)/LC(V_{\mathrm{local}})$ | $LC(AV+B)/LC(AV)$ | | :--- | ---: | ---: | | 0 | 0.1442 | 1.5440 | | 1 | 0.1493 | 1.8078 | | 2 | 0.3244 | 1.7905 | | 3 | 0.3920 | 1.4502 | | Mean | 0.2525 | 1.6482 | Across all four modules, attention reduces local contrast and APPLE partially restores it: $AV$ retains 25.25% of local-value contrast on average, and the correction increases contrast by 64.82% relative to $AV$.Q3 / W3: Training quality, Pythia variability, and the 70M deterioration
The 70M improvement is not solely a consequence of the baseline's final deterioration. Applying the same best-validation checkpoint-selection criterion to both methods gives Pile validation PPL of 33.195 for baseline and 31.190 for APPLE, a 6.04% reduction. The larger final-checkpoint gap—59.764 versus 31.190 PPL, or 47.81%—is substantially affected by the baseline's deterioration.
Official checkpoints provide an external reference for training quality. Our archived evaluation includes official Hugging Face EleutherAI/pythia-v1.1-{70m,160m} checkpoints. Under the recorded WikiText-2 raw word-perplexity evaluation, all entries below use final checkpoints at step 143,000, rather than best-validation checkpoints:
| Model size | Official HF word PPL ↓ | Our baseline word PPL ↓ | APPLE word PPL ↓ |
|---|---|---|---|
| 70M | 390.208 | 318.513 | 120.303 |
| 160M | 59.805 | 42.856 | 33.964 |
Our baselines outperform the official references on this evaluation, providing an external check on baseline quality that supports the credibility of our baseline–APPLE comparisons.
Additionally, at 410M, the stronger baseline may leave less predictive headroom, yielding a smaller log-loss reduction and hence a smaller relative PPL gain, , where is the log-PPL reduction under the same evaluation protocol. This modest gain should be viewed alongside the broader improvements across benchmarks.
Q4 / W4: Effect sizes and downstream uncertainty
We would appreciate it if these gains could be considered in the context of the broader improvements observed across model families and tasks. The ADE20K and XSum results complement this evidence: although their absolute gains are modest, APPLE improves in all three paired runs on each task. The tables below report the paired gains and their variability, providing a closer view of how consistently these improvements occur across training seeds.
For ADE20K, we reaggregated seed-level scores and paired improvements from the archived SegFormer runs:
| Seed | Baseline mIoU ↑ | APPLE mIoU ↑ | APPLE − baseline |
|---|---|---|---|
| 42 | 0.35706 | 0.36094 | +0.00388 |
| 43 | 0.35925 | 0.36184 | +0.00259 |
| 44 | 0.36023 | 0.36284 | +0.00261 |
| Mean ± SD | 0.35885 ± 0.00162 | 0.36187 ± 0.00095 | +0.00303 ± 0.00074 |
All three seed pairs improved, with a mean paired gain of 0.303 mIoU percentage points (SD 0.074).
For T5/XSum, the three paired runs show the following results:
| Seed pair | Baseline ROUGE-L ↑ | APPLE ROUGE-L ↑ | APPLE − baseline |
|---|---|---|---|
| 1 | 0.23314 | 0.23344 | +0.00030 |
| 2 | 0.23337 | 0.23390 | +0.00053 |
| 3 | 0.23253 | 0.23309 | +0.00056 |
| Mean ± SD | 0.233013 ± 0.000434 | 0.233477 ± 0.000406 | +0.000463 ± 0.000142 |
All three pairs improved. The mean gain is 0.0463 points on the 0–100 ROUGE-L scale, or approximately 0.20% relative to baseline.
Reference
Laziz Abdullaev and Tan M. Nguyen. Transformer Meets Twicing: Harnessing Unattended Residual Information. ICLR 2025.
I thank the authors for the response and the additional comparisons with the low-rank value bypass and Twicing Attention.
However, several concerns remain. First, the controlled comparison, in which APPLE and the value bypass use the same number of additional parameters and the same training budget, is conducted only on a small-scale synthetic task, where APPLE reduces the negative log-likelihood (NLL) by approximately 2%. Second, on GLUE, the value bypass and Twicing Attention results are obtained from the authors’ re-implementation of the fine-tuning setup, whereas the baseline and APPLE results are taken from the paper, so the methods are not compared under identical experimental conditions. Moreover, the value bypass baseline alone already accounts for a large portion of the improvement over the baseline model on several tasks (e.g., RTE and CoLA), which makes it unclear whether the gain stems from the proposed residual formulation or simply from the additional trainable parameters. Third, in the Pythia pretraining experiments, the relative perplexity reduction at the 410M scale is only 0.42%. Finally, the improvements on ADE20K and XSum are small in magnitude (about 0.3 mIoU points and about 0.2% relative improvement in ROUGE-L, respectively), which limits their practical significance.
Thank you for your careful review and constructive suggestions. We appreciate the time and effort you devoted to evaluating our work, particularly your questions about baseline comparisons, the interpretation of our theoretical results, and the consistency and practical significance of the empirical improvements. Your feedback has helped us clarify the scope of our claims and strengthen the supporting evidence. Below, we address each concern with additional comparisons, analyses, and explanations.
Q1 / W1: Existing methods and matched value bypass
To address your concern regarding baseline comparisons, we added direct comparisons with a parameter-matched value bypass and Twicing Attention (Abdullaev and Nguyen, 2025) under matched training conditions.
We designed a synthetic task requiring both context-based denoising and preservation of small local patterns. Each input is a length-128 signal
\qquad b_i=c+a(i/127-1/2),$$ with independently sampled $c\sim U(-2,2)$ and $a\sim U(-0.2,0.2)$. The motif is $[+1,+1,-1,-1]$ or its sign reversal, with a uniformly sampled starting position from 24 to 120 and $z_i=0$ elsewhere. Independent Gaussian noise has $\sigma\in\{0,0.10,0.25\}$, mixed during training. From the observations alone, the model reconstructs $b_i+0.4z_i$ and predicts background/positive-motif/negative-motif labels defined relative to the background. Copying retains noise, whereas smoothing can weaken the motif; the task therefore requires denoising while preserving local distinctions. We trained all four methods from scratch on the same approximately 9.87M-parameter, eight-layer GPT-NeoX backbone, retaining learned causal softmax attention, standard skips, MLPs, LayerNorm, and RoPE. Each run used 2,048 updates with batch size 128. Backbone initialization and training examples/order were matched within seeds 42, 43, and 44. APPLE and value bypass shared rank 4 per head, the same branch initialization with $U=0$, and 20,480 additional parameters. AdamW learning rates were $3\times10^{-4}$ for the backbone and $10^{-3}$ for the branch. Training comprised 131,072 sequences (32,768 families with four variants each), with 1,536 validation examples. The loss combines MSE normalized by $0.4^2$ and three-class cross-entropy, equally weighting motif and background regions within each loss and excluding the first 16 positions. We evaluated final checkpoints on 3,072 held-out examples, equally split across the three noise levels. The table reports motif-position classification NLL and reconstruction MSE as mean ± SD across the three training seeds, and forward GFLOPs per length-128 sequence. | Model | Trainable parameters | Forward GFLOPs | Motif NLL ↓ | Motif reconstruction MSE ↓ | | --- | ---: | ---: | ---: | ---: | | Baseline | 9,866,244 | 2.6015 | 0.190202 ± 0.002759 | 0.012717 ± 0.000092 | | Value bypass | 9,886,724 | 2.6071 (+0.21%) | 0.191214 ± 0.001832 | 0.012771 ± 0.000090 | | Twicing Attention | 9,866,244 | 2.6867 (+3.27%) | 0.190255 ± 0.003688 | 0.012589 ± 0.000086 | | APPLE | 9,886,724 | 2.6074 (+0.23%) | **0.186989 ± 0.001849** | **0.012430 ± 0.000179** | APPLE’s aggregate motif NLL was 1.69% lower than baseline, 2.21% lower than value bypass, and 1.72% lower than Twicing, with the same direction in all three seeds. APPLE also achieved the lowest mean reconstruction MSE at motif positions. These joint improvements address the intended challenge of the task: distinguishing small local patterns while recovering their clean values from noisy observations. Because APPLE and value bypass share the same branch size, initialization rule, and training budget, the advantage over value bypass supports the usefulness of learning from the local–context residual, beyond simply adding an equally sized pathway. On this controlled task, residual access therefore benefits both local-pattern discrimination and denoising. We additionally evaluated value bypass and Twicing Attention on six GLUE tasks using a reconstruction of the paper's fine-tuning recipe. | Task / metric | Paper baseline | Paper APPLE | Value bypass | Twicing | | --- | ---: | ---: | ---: | ---: | | QNLI accuracy | 91.32 | **91.98** | 91.52 | 91.04 | | CoLA MCC | 57.19 | **59.66** | 58.54 | 51.98 | | MRPC F1 | 89.07 | **89.32** | 88.49 | 86.18 | | SST-2 accuracy | 92.95 | **93.12** | 93.00 | 92.03 | | RTE accuracy | 63.78 | **67.51** | 66.19 | 61.73 | | MNLI accuracy | 84.17 | **84.65** | 84.41 | 84.30 | Together, the synthetic and downstream results support the usefulness of APPLE’s residual formulation beyond the controlled synthetic task.