acceptodds
Under review as a conference paper at ICLR 2027

A Structural Theory of Position Bias in Transformers

Abstract

Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis shows that collapse is instead governed by cumulative attention mixing and is not inevitable. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.

Primary area:
interpretability and explainable AI
Keywords:
position bias, attention rollout, residual connections, causal Transformers, input-token influence, ALiBi, RoPE, Lost-in-the-Middle

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pSxaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cumulative attention rollout with layer-wise residue mixing to handle the position bias problem in transformers.

Critical strengths
  1. The paper attacks something real and under-theorized: why causal Transformers exhibit a systematic, architecture-induced position-bias prior.
  2. The problem and solution are well formulated. Mathematical proof is provided in appendix.
Critical weaknesses
  1. The experimental breath is narrower than the abstract implies. The 30-model-dataset experiments is 10 models*3 datasets. Main-text tables (Table 1 and 2) only have results for 10 models. Moreover, FineWeb-Edu, DCLM-Baseline and Wikipedia are all generic English web/encyclopedic text. They vary the input distribution, not the task, the domain, or the architecture.
  2. Model diversity is limited to three families. No Llama/Mistral/Gemma-class models, no Mixture-of-Experts, no long-context architecture (which is precisely where position bias matters most), no encoder–decoder.
  3. Results in Figure 1 don’t show how Lost-in-the-Middle problem is alleviated. The paper does not propose a method that explicitly alleviates LiM
  4. Experiment is insufficient. More ablation studies are needed.
Questions for authors

see weakness

2 comments

We thank the reviewer for their feedback. We address the concerns below.

Position bias has long been observed. What was missing was a structural explanation that could faithfully predict the influence profiles of pretrained Transformers. We provide that explanation. Mean normalized Wasserstein distance falls from 0.62 for attention-only and 0.60 for fixed-residual rollout to 0.09 for ours, without fitting to the measured profiles.

1. Experimental coverage

Experimental coverage: “narrower than the abstract implies.”

The abstract explicitly states 30 model–dataset settings, not 30 distinct models. These comprise ten pretrained causal Transformers evaluated on FineWeb-Edu, DCLM-Baseline, and Wikipedia. Table 2 reports FineWeb-Edu; the appendix reports the other corpora. This is exactly the coverage claimed. Using the same influence protocol across different input distributions tests whether the structural prediction persists across corpora.

2. Model diversity and evaluation protocol

“Model diversity is limited to three families.”

Table 1 includes BLOOM, MPT, Falcon-RW, Qwen2.5, Qwen3, and SmolLM3, covering five ALiBi and five RoPE models. This substantially extends the evaluation of the direct predecessor, Wu et al. (ICML 2025), which studied small networks on synthetic retrieval data.

The evaluation also follows established attribution practice: Abnar and Zuidema (ACL 2020) validated rollout using gradients and token blank-out, chiefly on a six-layer unmasked encoder for subject–verb agreement, with BERT and DistilBERT examples. We apply these influence checks to ten pretrained causal Transformers across three corpora and two positional mechanisms. The appendix additionally evaluates 2,048-token contexts. Encoder–decoder architectures fall outside the stated causal-Transformer theory.

3. Explanation versus mitigation

“Results in Figure 1 don’t show how Lost-in-the-Middle problem is alleviated.”

Figure 1 evaluates an explanation of existing position bias; the paper does not claim a mitigation method. This is the scientific question studied by Wu et al., our more recent and direct predecessor. Their meta-review explicitly recorded:

“reviewers have raised strong concerns about the mismatch between the theory and empirical observations.” (Meta-review)

We supply the residual-aware structural theory and compare its cumulative predictions against independently measured input-token influence. Unlike the preceding rollout rules, our model closely predicts the observed profiles. Identifying and testing this architectural mechanism also provides a concrete basis for designing mitigation strategies.

4. Ablations and intervention evidence

“More ablation studies are needed.”

The submitted appendix already contains ablations on context length, mixing calibration, gradient versus blank-out influence, and both corpus-level rollout approximations. These directly test the measurement and modeling choices.

We additionally intervene on the attention–residual balance in six pretrained models. Predictions based solely on pre-intervention measurements give the correct direction for all 24 point-estimate shifts, with 20 confidence intervals excluding zero. The evidence therefore includes controlled interventions on the proposed mechanism, not merely agreement with an observed curve.

Could the reviewer specify which additional ablation would address a concern not covered by these experiments?

With the discussion closing tomorrow, we would appreciate your assessment of our response. We clarified the coverage of 30 model–dataset settings and highlighted the submitted ablations and controlled interventions. These support our structural explanation that closely predicts observed position bias where preceding methods fail. We hope this evidence resolves your concerns about the contribution and experimental support.

Reviewer_tv62OpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates position bias in Transformer architectures. Existing work has explored this phenomenon using attention-only and fixed-residual rollout models. Based on this work, this paper further investigates it with an adaptive residual-aware rollout model. The experimental results show that the proposed model can successfully approximate the position bias observed in pretrained Transformers.

Critical strengths
  • S1. The position bias targeted in this paper is an important phenomenon observed in Transformer architectures and warrants study.
  • S2. This paper provides a theoretical model to explain the U-shaped curve of position bias. The model is an extension of existing work, but it incorporates adaptive residual rollout, which enables us to analyze the phenomenon more deeply.
  • S3. The experimental setup covers multiple settings, including RoPE and ALiBi.
  • S4. This paper provides a careful discussion of the scope of the experimental results and architectural limitations.
Critical weaknesses
  • W1. The abstract and introduction sections may overclaim. While the experimental results are discussed carefully, the first few pages of this paper occasionally make claims that are stronger than the results support, such as "resolving the open problem" and "explaining the architectural origins of position bias."
  • W2. The introduction also provides unclear positioning relative to prior work. The current introduction centers its storyline on Wu et al. (2025), while giving less attention to Abnar & Zuidema (2020), who already introduced residual-aware attention rollout. This presentation makes the novelty of the proposed approach less clear.
  • W3. The necessity of adaptive residual rollout is not sufficiently validated because the current experimental setup only uses a single fixed coefficient, \lambda_t = 1/2, as a baseline. From this single fixed-coefficient comparison, I cannot judge whether the improvement comes from the adaptive method or simply from an inappropriate choice of \lambda in the baseline.
  • W4. Although I understand that the purpose of this paper is to investigate position bias through simplified modeling, the paper should more clearly discuss the gap between simplified modeling and mechanistic explanation. Specifically, even if the simplified model can successfully approximate the phenomenon, this does not necessarily mean that the original Transformer produces the phenomenon through the same mechanism.
Questions for authors
  • Q1. Could you clarify the justification for the stronger claims in the abstract and introduction?
  • Q2. I would like to know how the proposed method performs with different fixed values of \lambda.
  • Q3. Could you clarify to what extent the proposed simplified model provides a mechanistic interpretation (or explanation) of position bias in actual Transformers, rather than merely approximating the observed influence profiles?
3 comments

We thank the reviewer for their positive and constructive feedback. We address the concerns below.

1. Justification of the claims (W1 / Q1)

W1 / Q1: “claims that are stronger than the results support, such as ‘resolving the open problem’ and ‘explaining the architectural origins of position bias.’”

“Resolving the open problem.” Wu et al.’s attention-only analysis predicts inevitable first-token collapse at infinite depth; Abnar and Zuidema’s fixed one-half mixing retains this outcome. Both rollout rules fail to predict the observed broad influence profiles. Wu et al.’s meta-review records “strong concerns about the mismatch between the theory and empirical observations.” The authors leave residual effects open:

“It would be interesting to further explore and characterize how residuals influence the emergence and amplification of positional bias.” (Appendix K.2)

Theorem 5.5 resolves the analytical question within the residual-aware model: under its assumptions, divergent cumulative mixing yields collapse; finite total mixing preserves later-token influence. We replace inevitable collapse with a precise dichotomy.

“Explaining the architectural origins of position bias.” We identify the interaction of causal masking, adaptive residual mixing, and positional encoding. Across 30 model–dataset settings, mean normalized Wasserstein distance falls from 0.62 for attention-only and 0.60 for fixed-residual rollout to 0.09 for ours, without fitting to influence profiles. For ALiBi, this agreement holds with content logits removed entirely. Q3 provides the independent intervention test.

2. Positioning relative to prior work (W2)

W2: “Abnar & Zuidema (2020), who already introduced residual-aware attention rollout.”

Our contribution is adaptive residual mixing, its collapse dichotomy, and its predictive accuracy. Abnar and Zuidema (ACL 2020) introduced fixed-residual rollout for attribution. The more recent Wu et al. (ICML 2025) directly studied architectural position bias. Abnar and Zuidema provide the methodological foundation; Wu et al. is the direct scientific predecessor.

3. Comparison with fixed mixing coefficients (W3 / Q2)

W3 / Q2: “I would like to know how the proposed method performs with different fixed values of λ.”

We tested six constants at the original model depths, with unchanged kernels and measured profiles, across 14 settings (13 ALiBi, one RoPE):

Mixing coefficient00.20.40.60.81Adaptive
Mean normalized Wasserstein distance0.360.350.570.630.640.640.09

Adaptive mixing wins all 84 comparisons. Its advantage is not an artifact of the one-half baseline. These results establish superiority over the tested grid, without claiming that every accurate approximation requires layerwise variation.

4. Evidence for the mechanism (W4 / Q3)

W4 / Q3: “Could you clarify to what extent the proposed simplified model provides a mechanistic interpretation (or explanation) of position bias in actual Transformers”

We test the mechanism by intervening in six pretrained Transformers. We rescale the attention branch, hold prompts and targets fixed, and predict changes using only pre-intervention measurements. All 24 point-estimate shifts have the predicted direction; 20 confidence intervals exclude zero. This supports the mechanism beyond static curve agreement. Submitted ablations additionally check influence measurement, context length, mixing calibration, and both corpus-level approximations.

Thank you again for your constructive review. We provided the requested fixed-coefficient comparisons and clarified the theoretical contribution and mechanistic evidence. We hope these responses address your concerns and would appreciate your assessment before the discussion closes tomorrow.

Thank you very much for your clarification and additional experimental results.

W1 and W2: I understand the theoretical findings. I expect the introduction in the next version to clearly explain the scope of this work, as clarified in the rebuttal. Specifically, resolving the analytical problem raised by Wu et al. (2025) under certain assumptions does not necessarily mean revealing the actual mechanisms underlying position bias in Transformers.

W3: I appreciate the additional experimental results, which help address my concern.

W4: I understand that the proposed model can capture some aspects of the mechanisms underlying position bias. However, my concern is that it does not necessarily capture all the mechanisms at work in actual Transformers. This limitation should be carefully discussed in the introduction and conclusion to avoid overstatement.

I was already positive about this paper, and I have increased my ratings for informativeness and soundness, with the expectation that the next version will incorporate the clarifications and additional results provided in the rebuttal.

Reviewer_jq2jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is motivated by better understanding the Lost-in-the-Middle phenomenon.

The paper's approach is as follows:

  • The paper proposes a model of information propagation in transformers through residual connections and attention mixing (the model is not the Transformer itself but rather a simplified architecture)

  • The paper shows that residual connections and attention mixing and positional encodings indeed affect the influence in a location-dependent way, and in particular positional encodings counteract the primacy bias.

  • Experiments on real Transformers validate the model to some extent

Full disclosure: I've struggled quite a bit with figuring out whether the significance of this contribution warrants a better or worse overall score.

Critical strengths

The paper appears correct, and may, if developed, lead to better understanding of the Lost-in-the-Middle phenomenon.

The paper follows a reasonable methodology in terms of connecting the theoretical model to actual transformers both qualitatively and quantitatively, although there are always limitations to connecting the model to the actual objects of interest.

The paper introduces reasonable improvements to prior modelling: a measured and layer-depending residual mixing cofficient; various empirical experiments

Critical weaknesses
  • As the authors acknowledge, the paper builds on much similar work

  • As the authors acknowledge, the actual phenomenon of "Lost-in-the-Middle" in the sense of information retrieval is analytically distinct from the influence of the middle positions

Questions for authors

Your paper is motivated by better understand the Lost-in-the-Middle phenomenon. What are the best argument for this paper's andvancing out understanding of that phenomenon rather than simply improving modelling?

3 comments

We thank the reviewer for their feedback. We address the concerns below.

1. Understanding Lost-in-the-Middle

“Your paper is motivated by better understand the Lost-in-the-Middle phenomenon. What are the best argument for this paper's andvancing out understanding of that phenomenon rather than simply improving modelling?”

Position bias had long been observed, but its origin remained insufficiently explained. Prior accounts involving training data, task behavior, and architectural effects did not establish a mechanism that faithfully predicts the observed influence profiles. We identify an architectural cause: the interaction of causal masking, adaptive residual mixing, and positional encoding. For the first time, our predictions closely match measured influence across pretrained Transformer families, and interventions independently support the mechanism. This advances understanding from observing the bias to identifying and testing a cause, providing a mechanistic basis for mitigation strategies that directly target it.

Across 30 model–dataset settings, mean normalized Wasserstein distance falls from 0.62 for attention-only and 0.60 for fixed-residual rollout to 0.09 for ours, without fitting to influence profiles. For ALiBi, the prediction remains accurate even with content logits removed entirely. Interventions in six models yield all 24 point-estimate shifts in the predicted direction, with 20 confidence intervals excluding zero, using only pre-intervention measurements. Submitted ablations check influence measurement, context length, mixing calibration, and both corpus-level approximations.

2. Advance over prior work

“As the authors acknowledge, the paper builds on much similar work”

Both preceding rollout rules predict inevitable first-token collapse at infinite depth under the analytical assumptions: Wu et al.’s attention-only rule and Abnar and Zuidema’s fixed one-half residual rule. Neither explains the broad influence profiles observed in deep pretrained Transformers, as the finite-depth comparison above demonstrates. Wu et al. (ICML 2025), the more recent and direct predecessor on position bias, explicitly left residual effects open:

“It would be interesting to further explore and characterize how residuals influence the emergence and amplification of positional bias.” (Appendix K.2)

Their meta-review also explicitly records:

“reviewers have raised strong concerns about the mismatch between the theory and empirical observations.” (Meta-review)

We resolve the analytical open problem within the residual-aware model. Our theorem replaces inevitable collapse with a dichotomy governed by cumulative attention mixing: under its assumptions, divergent total mixing yields collapse, while finite total mixing preserves later-token influence. Together with the finite-depth predictions and intervention results, this supplies the structural explanation and quantitative agreement missing from the preceding rollout models.

With the discussion closing tomorrow, we would appreciate your assessment of our response. We identify and test an architectural cause of position bias. This answers a question prior accounts could not resolve. Accurate influence predictions and independent interventions support this explanation. We hope this resolves your concern about advancing understanding beyond improved modelling.

Thank you for the response. I do not currently have more questions. I will think about it and potentially adjust my review as necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.