- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a cumulative attention rollout with layer-wise residue mixing to handle the position bias problem in transformers.
Critical strengths
- The paper attacks something real and under-theorized: why causal Transformers exhibit a systematic, architecture-induced position-bias prior.
- The problem and solution are well formulated. Mathematical proof is provided in appendix.
Critical weaknesses
- The experimental breath is narrower than the abstract implies. The 30-model-dataset experiments is 10 models*3 datasets. Main-text tables (Table 1 and 2) only have results for 10 models. Moreover, FineWeb-Edu, DCLM-Baseline and Wikipedia are all generic English web/encyclopedic text. They vary the input distribution, not the task, the domain, or the architecture.
- Model diversity is limited to three families. No Llama/Mistral/Gemma-class models, no Mixture-of-Experts, no long-context architecture (which is precisely where position bias matters most), no encoder–decoder.
- Results in Figure 1 don’t show how Lost-in-the-Middle problem is alleviated. The paper does not propose a method that explicitly alleviates LiM
- Experiment is insufficient. More ablation studies are needed.
Questions for authors
see weakness
2 comments
With the discussion closing tomorrow, we would appreciate your assessment of our response. We clarified the coverage of 30 model–dataset settings and highlighted the submitted ablations and controlled interventions. These support our structural explanation that closely predicts observed position bias where preceding methods fail. We hope this evidence resolves your concerns about the contribution and experimental support.
We thank the reviewer for their feedback. We address the concerns below.
Position bias has long been observed. What was missing was a structural explanation that could faithfully predict the influence profiles of pretrained Transformers. We provide that explanation. Mean normalized Wasserstein distance falls from 0.62 for attention-only and 0.60 for fixed-residual rollout to 0.09 for ours, without fitting to the measured profiles.
1. Experimental coverage
The abstract explicitly states 30 model–dataset settings, not 30 distinct models. These comprise ten pretrained causal Transformers evaluated on FineWeb-Edu, DCLM-Baseline, and Wikipedia. Table 2 reports FineWeb-Edu; the appendix reports the other corpora. This is exactly the coverage claimed. Using the same influence protocol across different input distributions tests whether the structural prediction persists across corpora.
2. Model diversity and evaluation protocol
Table 1 includes BLOOM, MPT, Falcon-RW, Qwen2.5, Qwen3, and SmolLM3, covering five ALiBi and five RoPE models. This substantially extends the evaluation of the direct predecessor, Wu et al. (ICML 2025), which studied small networks on synthetic retrieval data.
The evaluation also follows established attribution practice: Abnar and Zuidema (ACL 2020) validated rollout using gradients and token blank-out, chiefly on a six-layer unmasked encoder for subject–verb agreement, with BERT and DistilBERT examples. We apply these influence checks to ten pretrained causal Transformers across three corpora and two positional mechanisms. The appendix additionally evaluates 2,048-token contexts. Encoder–decoder architectures fall outside the stated causal-Transformer theory.
3. Explanation versus mitigation
Figure 1 evaluates an explanation of existing position bias; the paper does not claim a mitigation method. This is the scientific question studied by Wu et al., our more recent and direct predecessor. Their meta-review explicitly recorded:
We supply the residual-aware structural theory and compare its cumulative predictions against independently measured input-token influence. Unlike the preceding rollout rules, our model closely predicts the observed profiles. Identifying and testing this architectural mechanism also provides a concrete basis for designing mitigation strategies.
4. Ablations and intervention evidence
The submitted appendix already contains ablations on context length, mixing calibration, gradient versus blank-out influence, and both corpus-level rollout approximations. These directly test the measurement and modeling choices.
We additionally intervene on the attention–residual balance in six pretrained models. Predictions based solely on pre-intervention measurements give the correct direction for all 24 point-estimate shifts, with 20 confidence intervals excluding zero. The evidence therefore includes controlled interventions on the proposed mechanism, not merely agreement with an observed curve.
Could the reviewer specify which additional ablation would address a concern not covered by these experiments?