acceptodds
Under review as a conference paper at ICLR 2027

The Sirens’ Song: When Proximal Background Context Overshadows Distant Evidence

Abstract

Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (**L**ong-context heav**Y**-tailed **R**elevance **A**lignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Our code is public at https://anonymous.4open.science/r/LYRA

Primary area:
learning theory
Keywords:
LLM, Long-Context

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_twNPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether nearby irrelevant context can overwhelm distant evidence through aggregate attention competition. It combines attention diagnostics and interventions with LYRA, which normalizes RoPE-transformed queries and keys and applies a monotone t-distributed transformation to cosine similarity before softmax. The main implementation adapts Qwen3-8B's final block. Evaluation covers LongBench-v2, RULER, LongBench, and ProxBench, a synthetic benchmark of proximal binding distractors. Additional experiments examine a second backbone, the transformation parameter, and layer placement. The theoretical analysis identifies conditions under which the transformation improves the evidence-to-background attention ratio relative to cosine-softmax.

Critical strengths
  1. Useful diagnostic perspective with complementary interventions (PDF pp. 1 and 3–4, Figures 1–3; pp. 14–16, Appendix B). The distinction between locating evidence and successfully using it under attention competition is informative. Figure 1 already varies the number of masked nearby tokens, Figure 2 compares distant evidence with its neighborhood and proximal background across layers, and Figure 3 compares evidence relocation with background-score attenuation at fixed token positions. Together, these observations give the proposed explanation more support than an attention visualization alone. ProxBench complements the natural-context diagnostics by varying entity, attribute, and role-binding confusability while keeping the background logically irrelevant to the answer. Its answer-leakage, span, and length checks make the synthetic construction more interpretable. These are useful contributions even though the extent to which the proposed transformation explains the benchmark gains remains open.

  2. Simple scoring modification with encouraging empirical breadth (PDF pp. 5 and 7–8, Section 3.1 and Tables 1–3; pp. 17–18, Appendices D.2–D.3). LYRA preserves RoPE, causal masking, softmax normalization, and value aggregation, making its architectural change easy to identify. It reports 36.72 versus 32.21 accuracy for Table 1's Baseline on LongBench-v2 and 89.32 average accuracy on RULER. The additional Llama-3.1-8B experiment and the comparison of placements at layers 15, 24, 33, and 35 broaden the evidence beyond the main Qwen3-8B configuration. The placement study also explicitly acknowledges that gains are not uniform across context lengths. This qualified interpretation is appropriate: the results support placement sensitivity without requiring a monotonic improvement with depth.

Critical weaknesses
  1. The main gain cannot yet be attributed to the nonlinear attention rule (PDF pp. 5 and 7, Section 3.1/Table 1; pp. 16–17, Appendix C/Table 5). Appendix C clearly states that training updates the complete final block, approximately 193M parameters, together with 64 added head-specific scalars. The training scope is therefore disclosed, but the paper does not establish whether Table 1's Baseline receives the same LongAlign data, trainable block, optimizer, and training budget. I cannot infer from this ambiguity that the baseline was left unadapted. Clarifying the existing protocol could resolve this part of the concern without a new experiment.

    A separate attribution question remains because LYRA changes both query/key normalization and the nonlinear mapping. Table 5 usefully studies κ in {2, 4, 8, 16}, but none of these settings removes the mapping while retaining normalized attention. Equation 8 identifies κ=0 as precisely that cosine-softmax control. Comparable temperature tuning matters because β also changes attention concentration. Thus, the current sensitivity study does not establish the increment over an adapted, normalized attention baseline. A matched comparison among standard attention, temperature-tuned κ=0, and LYRA would distinguish these explanations. This is the main issue for my recommendation: the reported system improvement is encouraging, but the specific contribution of the proposed scoring transformation is still uncertain.

  2. The connection between the theoretical condition and the empirical explanation remains incomplete (PDF pp. 1, 3–4 and 6, Figures 1–3 and Equations 9–11). Equation 11 gives a valid conditional improvement over cosine-softmax at the same inverse temperature. It compresses an unfavorable background advantage when the gap-scaling factor is below one, or amplifies an existing evidence advantage when that factor exceeds one. The analysis therefore describes identifiable favorable regimes; it does not establish that those regimes dominate the evaluated model's evidence–background competition. Because the transformation preserves similarity ordering, the relevant empirical question is how its change in relative weights affects the competing attention mass. Measuring the prevalence and importance of the favorable comparisons would connect the algebra more directly to the proposed mechanism.

    The existing masking-count sweep, relocation experiment, and fixed-position background attenuation already provide relevant diagnostic evidence. The remaining reproducibility issue concerns their precise construction. Figure 2 defines mean per-token attention density relative to uniform attention, but that definition alone does not specify which examples and spans enter the analysis, how heads and query tokens are aggregated, or the exact relocation positions and attenuation settings in Figure 3. These details affect how broadly one can interpret the diagnostics. This concern limits the strength and reproducibility of the mechanism explanation; it does not imply that the interventions are absent or that the conditional formula is incorrect.

I lean toward weak rejection, primarily because the benefit of the new scoring rule has not been separated from block adaptation and normalized attention. The benchmark gains, controlled synthetic construction, and additional backbone and placement results make the approach promising. A meaningful advantage over matched standard attention and temperature-tuned κ=0 would substantially strengthen the case for acceptance and lead me to reconsider the recommendation. If the existing Baseline already has matched adaptation, making that protocol explicit would remove one uncertainty immediately. The mechanism clarification is secondary and should be weighed alongside the diagnostic evidence already supplied.

Questions for authors
  1. For W1 — main priority: Does Table 1's Baseline use the same LongAlign data, trainable block, optimizer, and budget? Please clarify the existing protocol first. Under matched training, how do standard attention, κ=0 cosine-softmax with comparable temperature tuning, and LYRA compare on the main evaluation? The key distinction is the increment from the nonlinear mapping after adaptation and normalization are controlled; the current positive-κ sweep cannot resolve that distinction.
  2. For W2: Please specify the examples, span labels, head/query-token aggregation, and relocation/attenuation settings for Figures 2–3. How frequently does Equation 11's favorable regime occur in the evaluated model, measured over relevant evidence–background comparisons or their competing attention mass? Relating this measurement to the reported diagnostics would clarify how much of the observed behavior the conditional analysis explains.
2 comments

Thank you for your thoughtful and constructive review. We are sincerely grateful for the care with which you examined our work and were particularly encouraged by your recognition of (1) Diagnostic Insight. Our complementary diagnostics and interventions distinguish locating distant evidence from using it under attention competition. (2) Controlled Benchmark Design. ProxBench systematically varies how easily proximal background can be mistaken for evidence while keeping that background irrelevant to the answer. (3) Method Simplicity and Empirical Breadth. LYRA makes a simple change to attention scoring, and we evaluate it across benchmarks, model backbones, and layer placements.

Your encouraging feedback and recognition are a huge encouragement to our team. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly, highlighted in yellow in the revised manuscript.

  • (W1) Isolating the contribution of the nonlinear mapping. Sincerely thanks for your kind advice. We conduct a controlled ablation on RULER at five context lengths from 8K to 128K. Starting from the same Qwen3-8B checkpoint, we fine-tune the final block on LongAlign dataset under the same training protocol for three variants: standard attention (FT), QKNorm and LYRA. We also report the original Qwen3-8B results as a reference.

    8k16k32k64k128kAvg.
    Qwen-396.0091.8091.2082.1077.4087.70
    Qwen-3 + FT94.8493.2491.0383.7277.3988.04
    Qwen3 + QKNorm95.4093.4891.4784.2477.0988.33
    LYRA96.3394.1993.1485.3977.5689.32

    Standard fine-tuning achieves 88.04% average accuracy, and the QKNorm reaches 88.33%. LYRA further improves the average to 89.32% and outperforms the normalized control at all five context lengths, with gains ranging from 0.47 to 1.67 percentage points. Thus, after controlling for final-block adaptation and QKNorm, the nonlinear mapping provides an additional 0.99 percentage points of average accuracy on RULER.

    The gains across all five context lengths support the utility of the nonlinear mapping. Together with our attention diagnostics, these results suggest that reshaping similarity gaps helps mitigate the Proximity Trap, allowing the model to make better use of distant evidence when it competes for attention with nearby, task-irrelevant background.

    We have added the matched RULER results as Table 9 in Appendix D.4, along with a discussion of LYRA’s gains over the standard attention and QKNorm controls under matched LongAlign fine-tuning. Thank you again for suggesting this comparison. It helped us clarify what LYRA adds beyond fine-tuning and Q/K normalization.

  • (W2) Diagnostic protocol and empirical scope. Thank you for requesting a more reproducible specification and a direct measurement of the regime characterized by Eq. 11. Figures 2 and 3 use the same diagnostic cohort of 400 LongBench-v2 examples. The cohort was selected before inspecting attention outcomes: after applying the evaluation prompt and the 32,768-token head–tail truncation, each example had to retain a single contiguous evidence passage whose final token was at least 2,048 context tokens from the end of <text>. We define the proximal background as the final 256 retained context tokens, excluding all annotated evidence or answer-supporting tokens.

    Regarding empirical relevance of Eq. 11, the favorable condition in Eq. 11 holds for 51.61% of evidence–background token pairs under equal weighting and accounts for 58.00% of the competing attention mass under mass weighting. When we compare LYRA with cosine-softmax on the same queries and keys at a fixed inverse temperature per head, the aggregate evidence-to-background attention-mass ratio increases in 60.16% of head–query instances. Under the checkpoint-faithful evaluation, the ratio increases in 49.22% of instances. These results show that favorable score changes are more prevalent among comparisons that carry more competing attention mass, and that their aggregate effect varies across head–query instances. This complements Figure 3 by showing how often LYRA’s score transformation itself shifts the balance toward distant evidence.

    We have revised our manuscript to clarify the shared diagnostic cohort and span definitions for Figures 2 and 3 in Appendix C. Implementation. We have also added an empirical analysis immediately after Eq. 11 in Section 3.2, reporting how often the favorable condition holds and how often the span-level evidence-to-background attention ratio increases. Thank you again for your thoughtful and detailed comments.

Thank you again for your time and thoughtful review. We appreciate the opportunity to clarify our findings and would be happy to address any remaining questions. If you find our response satisfactory, could you please consider helping us improve our rating score?

Reviewer_5Yy1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Long-context LLMs can identify distant evidence yet fail to use it because many nearby, task-irrelevant tokens can collectively accumulate substantial attention weights, diluting the evidence’s contribution. To address this, the paper introduces LYRA, which applies a nonlinear transformation to query–key cosine similarities to help relevant evidence withstand competition from proximal tokens. LYRA is evaluated on several benchmarks showing improvements in long-context retrieval and understanding. The paper also introduces ProxBench to evaluate robustness to increasingly confusable nearby background.

Critical strengths
  • The masking and attenuation experiments explain the proximal background contribution to dilution of distant attention evidence really well.
Critical weaknesses
  • The baseline scores for RULER and LongBench are missing. Without this, it is unclear whether LYRA improves over the original backbone or just outperforms the other attention approximations.
  • There isn't a matched fine-tuned standard attention baseline. So the gains might have come either from LYRA or from finetuning on the LongAlign dataset.
Questions for authors
  • Since all context tokens compete through softmax, what shows that proximal background dilutes evidence attention more than equally numerous distant background tokens?
  • How does LYRA handle multi-hop reasoning, where intermediate evidence may have low direct similarity to the query? Could its attention transformation suppress such evidence? Benchmarks like AA-LCR might be a good test for this.
3 comments

We thank the reviewer for the constructive feedback, and especially for recognizing how our masking and attenuation experiments clarify the role of proximal background in diluting attention to distant evidence. We address each question below and highlight the corresponding changes in yellow in the revised manuscript.

  • (Weakness) Isolating LYRA’s Gains from LongAlign Fine-Tuning. We thank the reviewer for pointing out the need to compare LYRA with both the original backbone and a standard-attention baseline fine-tuned under matched conditions. We added RULER comparisons at five context lengths from 8K to 128K. The original Qwen3-8B scores 87.70% on average, and standard attention with LongAlign fine-tuning reaches 88.04%. Under the same fine-tuning protocol, LYRA achieves 89.32%, a gain of 1.62 percentage points over the original backbone and 1.28 points over the matched standard-attention baseline. LYRA also outperforms the matched QKNorm baseline at every tested length, with an average gain of 0.99 points and the largest gain of 1.67 points at 32K. The standard-attention comparison shows that LYRA’s gains persist when LongAlign fine-tuning is held fixed, and the QKNorm comparison shows an additional benefit from the nonlinear scoring transformation. Moreover, on LongBench, LYRA also improves over the original Qwen3-8B backbone, raising the score from 46.35 to 50.06, a gain of 3.71 points.

    8k16k32k64k128kAvg.
    Qwen-396.0091.8091.2082.1077.4087.70
    Qwen-3 + FT94.8493.2491.0383.7277.3988.04
    Qwen3 + QKNorm95.4093.4891.4784.2477.0988.33
    LYRA96.3394.1993.1485.3977.5689.32

    We have added these results in Table 9 and discussed both matched comparisons in Appendix D.4 (Isolating the contribution on RULER) of the revised manuscript. Thank you again for your kind suggestion.

  • (Q1) Proximal vs. Distant Background Competition. We thank the reviewer for raising this question. We extended the masking experiment in Figure 1 on LongBench-v2 to compare two conditions on the same examples: masking 64 context tokens near the query and masking 64 context tokens farther away. Masking nearby tokens yielded 1.20 percentage points higher accuracy than masking the same number of distant tokens (p=0.013p = 0.013). This provides behavioral evidence that proximal context interferes more strongly with answering under the tested conditions, consistent with stronger competition for attention. Directly quantifying evidence-attention dilution would require measuring the attention assigned to evidence spans under both masking conditions.

    We have added a discussion of this controlled comparison to Section 2 of the revised manuscript. Thank again for prompting this analysis.

  • (Q2) Multi-Hop Reasoning and Intermediate Evidence We thank the reviewer for raising this concern and suggesting AA-LCR.

    • In multi-hop reasoning, intermediate evidence can be essential to a later reasoning step despite its low direct similarity to the original query. LYRA applies its transformation only in the final attention layer, where the query representation can already incorporate information from earlier layers. The transformation preserves the ranking of query–key similarities within each head. It can still downweight intermediate evidence relative to competing background tokens. For example, if an evidence token slightly outscores a background token and both scores lie in the compression regime, LYRA narrows the score gap and reduces the evidence token’s relative attention advantage.

    • Following the reviewer’s suggestion, we evaluated LYRA on AA-LCR. LYRA achieved 37.89% accuracy, 4.22 percentage points above the 33.67% reported for Qwen3-8B. This comparison provides evidence at the level of end-to-end multi-hop reasoning.

    We have added a discussion of multi-hop reasoning and reported the AA-LCR results in Section 4.2 of the revised manuscript. Many thanks for your kind suggestion.

Thank you once more for dedicating your time. If you have any further questions, we are open to hearing from you at any time. If you find our response satisfactory, we would greatly appreciate your assistance in improving our rating score.

Thank you for the response. My concerns have been addresses and I have revised my rating.

We are deeply grateful for your high recognition. Your positive feedback is incredibly helpful and a tremendous boost for our team. Wishing the reviewers all the best.

Reviewer_meJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Focusing on how long-context language models attend to distant information, this paper diagnoses the Proximity Trap, a phenomenon in which attention to distant evidence is attenuated by both positional distance and cumulative competition from task-irrelevant local context under softmax normalization. To address this limitation, the authors propose LYRA, which computes cosine similarities between RoPE-transformed queries and keys and applies a parameterized nonlinear mapping while preserving standard softmax normalization and value aggregation. In the main experiments, integrating LYRA into only the final Transformer block of Qwen3-8B and fine-tuning that single block on LongAlign yields consistent improvements across LongBench, LongBench-v2, and RULER. To evaluate this failure mode more rigorously, the authors also construct a custom benchmark named ProxBench, which systematically increases the confusability of proximal background context to assess how effectively models utilize distant evidence.

Critical strengths

S1. The paper goes beyond treating context-window length as a sufficient measure of long-context capability. It asks why models can fail to use relevant information that is already present in the input. S2. The main text analyzes the transformation’s monotonicity and identifies the regimes in which similarity gaps are compressed and amplified. It also derives a condition under which the relative weight of evidence improves. S3. The experiments cover general long-context understanding and controlled context lengths, with additional results across task categories.

Critical weaknesses

W1 The problem formulation leaves the meaning of evidence use unclear The second paragraph of the introduction treats distant evidence as a clearly identifiable object, but Section 2.1 only defines it as task-relevant information. The argument moves between finding that information and assigning it attention, before treating improved answer accuracy as better evidence use. The paper needs to explain what counts as evidence in the task and what failure remains after that evidence has been found.

W2 The analysis in the experimental section is not sufficiently in-depth and lacks a clear mechanistic analysis. The introduction argues that abundant nearby background is a central cause of distant-evidence failures. Section 4 mainly shows higher scores on general long-context benchmarks. ProxBench is more focused, but increasing the semantic ambiguity of distractors does not isolate the effect of proximity. The diagnostics in Section 2 are suggestive, but the paper does not show that LYRA's gains come from correcting the failure described there.

W3 The novelty of LYRA is unclear LYRA keeps RoPE and modifies the scoring function applied afterward. Equation (8) is algebraically identical to the t-vMF similarity in Eq. (7) of Kobayashi (2021), a work missing from the references. Applying an established function to long-context attention may be useful, but the contribution must lie in the new insight and independently demonstrated benefit of this adaptation. The paper does not establish why this particular transformation addresses proximal interference.

W4 The baselines are poorly aligned with the claimed contribution The main tables rely heavily on sparse-attention methods such as MInference (Jiang et al., 2024) and FlexPrefill (Lai et al., 2025), which seek to accelerate inference. LYRA retains dense attention and includes fine-tuning on LongAlign. Higher accuracy than efficiency-oriented approximations does not establish an advance in evidence use. The main text also leaves the matched-training standard-attention baseline unclear and omits normalized cosine attention, despite its direct relevance to the scoring change and the existing QKNorm formulation (Henry et al., 2020). The present comparisons cannot show how much of the gain comes from LYRA itself.

W5 Generalization across model sizes has not been established The main experiments and mechanism analyses are centered on Qwen3-8B. No scale comparison establishes that the proposed effect persists as model capacity increases substantially. This matters because the claimed benefit concerns how a model separates relevant information from distracting context.

Reference (1)Henry, A., Dachapally, P. R., Pawar, S. S., & Chen, Y. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246–4253, 2020.

(2)Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 52481–52515, 2024.

(3)Kobayashi, T. T-vMF similarity for regularizing intra-class feature distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6616–6625, 2021.

(4)Lai, X., Lu, J., Luo, Y., Ma, Y., & Zhou, X. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations, 2025.

Questions for authors

Q1. What exactly is the failure that the paper calls insufficient evidence use? Q2. Which result connects LYRA's gains to the claimed Proximity Trap? Q3. What is the substantive contribution beyond the existing t-vMF function? Q4. Does the claimed mechanism hold across substantially different model sizes?

2 comments

We sincerely appreciate your thoughtful and detailed review. Your positive feedback serves as a great source of encouragement for our team, especially recognizing the 1) Motivation beyond context-window as long-context capability, 2) Theoretical analysis of long-context alignment and 3) Comprehensive experiments. Furthermore, we have diligently responded to each of your questions and incorporated your feedback into our manuscript revisions, with these changes highlighted in yellow for your reference in the revised version.

  • (W1 & Q1) The meaning of evidence. We thank the reviewer for pointing out this ambiguity. We clarify what counts as evidence and explain how competition from proximal background can limit evidence use even when the model attends to the evidence.

    • Definition of evidence. We define evidence as the context span containing facts or relations that support deriving the correct answer. Our analysis focuses on evidence located far from the query. Accordingly, the second paragraph of the Introduction motivates our study by considering how such evidence competes for attention with proximal background. In Section 2.1, we refer to these spans as distant evidence and denote them by E\mathcal{E}. For example, in ProxBench (Appendix B.1), the evidence span is the record that states the access code assigned to the entity named in the question.

    • Finding, assigning attention to, and using evidence. All three terms refer to the same evidence span E\mathcal{E}. Finding evidence means locating that span in the input. Assigning attention to evidence refers to the attention weights the model assigns to the tokens in the span, and using evidence means drawing on its content to answer the question. In this paper, evidence finding is outside the scope of our analysis. Given distant evidence already present in the input, we examine how proximal background affects both the attention assigned to that evidence an d the model’s answer.

    • Failure under background competition. Even when the model attends to the distant evidence, that span can receive only a small share of the total attention mass. As Eq. (7) illustrates, many proximal background tokens can collectively outweigh the evidence under softmax, even when each background token receives less attention than an evidence token. We refer to the failure of distant evidence to contribute sufficiently to the model’s answer under such competition as insufficient evidence use.

    We have clarified the evidence definition in the Introduction and Section 2.1, and also clarified the failure that remains after the span is present. Thanks so much for your elaborate and thoughtful review.

  • (W2 & Q2) Mechanistic analysis that LYRA mitigates the Proximity Trap. We sincerely thank the reviewer for suggesting that we connect the Proximity Trap diagnosed in Section 2 to LYRA’s performance gains in Section 4. We first explain why the Proximity Trap is a key failure mode, then present new evidence that mitigating it contributes to LYRA’s gains.

    • The proximity trap as a key failure mode. In Fig. 3, attenuating proximal background with token positions fixed generally increases attention to distant evidence and often raises confidence in the correct answer. Moving the evidence closer yields mixed results. In Fig. 1, masking the 256 context tokens nearest to the query raises LongBench-v2 accuracy from 32.21% to 33.80% overall and from 32.1% to 35.8% on Long In-context Learning. These results support proximal background competition as a contributor to failures in distant evidence use.

    • Linking LYRA’s gains to the failure mode. To test whether LYRA’s gains stem from mitigating proximal background competition, we intervened in opposite directions on LongBench-v2. Attenuating attention to proximal background in the baseline improved overall accuracy from 32.21% to 34.45%. Conversely, increasing attention to proximal background in LYRA reduced its accuracy from 36.72% to 32.31%, narrowing its advantage over the original baseline from 4.51 to 0.10 percentage points. These results support our explanation that reducing competition from proximal background contributes to LYRA’s gains on LongBench-v2.

      MethodsShortMediumLongOverall
      Baseline36.6729.3030.5632.21
      Baseline+Attenuation42.1829.3331.7334.45
      LYRA+Enhancement36.7329.3730.8232.31
      LYRA47.2229.6333.3336.72

    We have added further discussion of the failure modes in the Introduction and a new bidirectional intervention table and analysis in Section 4.1. The table explicitly connects the accuracy changes to the Proximity Trap diagnosed in Section 2. We thank the reviewer again for prompting this clearer mechanistic link.

  • (W3 & Q3) Substantive contribution. Many thank the reviewer for pointing out the connection between ours and prior work. However, the two works differ in their aims, motivations, and how they use the mapping.

    • The Proximity Trap. T-vMF function is used in both our work and Kobayashi (2021). Kobayashi uses the function to regularize feature distributions in image classification. While, we identify a different phenomenon in long-context LLM, which we call the Proximity Trap. Many task-irrelevant tokens near the query can collectively draw attention away from distant evidence, even when the model attends selectively to that evidence. This observation motivates our use of t-vMF to reshape query–key scores after RoPE.

      Moreever, Kobayashi uses the mapping to score image features against classifier weights, thereby regularizing intra-class feature distributions. However, LYRA uses the t-vMF mapping for query–key matching. By reshaping the similarity gaps between query–key pairs, it changes how distant evidence and proximal background compete for attention.

    • Why the transformation can mitigate proximal interference. The mapping preserves the ranking of query–key scores while changing the gaps that determine relative attention weights under softmax. As Eq. (11) shows, compressing a score gap in favor of a proximal background token reduces its relative attention advantage. When distant evidence already scores higher, expanding the gap increases its relative attention weight. These two cases explain when the nonlinear transformation can shift attention toward distant evidence under background competition.

    We have cited Kobayashi (2021) in the revised manuscript and added a discussion clarifying how LYRA adapts the t-vMF mapping for query–key matching in Section 3.1. We thank the reviewer again for highlighting this connection.

  • (W4) Substantive contribution. We thank the reviewer for pointing out the need for baselines that isolate LYRA’s contribution. We added two controls on RULER under the same LongAlign fine-tuning protocol as LYRA: Qwen3-8B with its standard fine-tuning (FT) and Qwen3-8B with QKNorm. The FT model scores 88.04% after fine-tuning, compared with 87.70% for the original Qwen3-8B. LYRA reaches 89.32%, improving over the FTn baseline by 1.28 percentage points and over QKNorm by approximately 1.0 point. LYRA also outperforms the cosine control at all five context lengths, with the largest gain of 1.67 points at 32K. These results quantify the additional benefit of the nonlinear mapping under matched training conditions. We include MInference and FlexPrefill as accuracy references for methods designed to improve inference efficiency.

    8k16k32k64k128kAvg.
    Qwen-396.0091.8091.2082.1077.4087.70
    Qwen-3 + FT94.8493.2491.0383.7277.3988.04
    Qwen3 + QKNorm95.4093.4891.4784.2477.0988.33
    LYRA96.3394.1993.1485.3977.5689.32

    We have added the table of matched RULER comparisons to the revised manuscript and cited papers. We thank the reviewer again for suggestful comments.

  • (W5) The Proximity Trap across model sizes. We thank the reviewer for asking whether the Proximity Trap persists at a larger model scale. We replicated the interventions in Fig. 3 on Qwen3-14B and added the results to the appendix in Fig.5 of our new manuscript. As in Qwen3-8B, moving distant evidence closer produced mixed changes in answer confidence. Attenuating proximal background while holding the input and token positions fixed yielded more consistent increases in attention to distant evidence and improved confidence in the correct answer on a subset of examples. These results show that proximal background can interfere with distant evidence use at both tested scales in the Qwen3 family.

    We have added Fig. 5 to Appendix D.5 (The Proximity Trap Across Model Sizes) and related discussion. We thank the reviewer for suggesting this comparison.

Thanks again for your valuable time and thorough review. Your feedback is immensely valuable to us, and we welcome further communication from you. If you find our response satisfactory, could you please consider helping us improve our rating score?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.