- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether nearby irrelevant context can overwhelm distant evidence through aggregate attention competition. It combines attention diagnostics and interventions with LYRA, which normalizes RoPE-transformed queries and keys and applies a monotone t-distributed transformation to cosine similarity before softmax. The main implementation adapts Qwen3-8B's final block. Evaluation covers LongBench-v2, RULER, LongBench, and ProxBench, a synthetic benchmark of proximal binding distractors. Additional experiments examine a second backbone, the transformation parameter, and layer placement. The theoretical analysis identifies conditions under which the transformation improves the evidence-to-background attention ratio relative to cosine-softmax.
Critical strengths
-
Useful diagnostic perspective with complementary interventions (PDF pp. 1 and 3–4, Figures 1–3; pp. 14–16, Appendix B). The distinction between locating evidence and successfully using it under attention competition is informative. Figure 1 already varies the number of masked nearby tokens, Figure 2 compares distant evidence with its neighborhood and proximal background across layers, and Figure 3 compares evidence relocation with background-score attenuation at fixed token positions. Together, these observations give the proposed explanation more support than an attention visualization alone. ProxBench complements the natural-context diagnostics by varying entity, attribute, and role-binding confusability while keeping the background logically irrelevant to the answer. Its answer-leakage, span, and length checks make the synthetic construction more interpretable. These are useful contributions even though the extent to which the proposed transformation explains the benchmark gains remains open.
-
Simple scoring modification with encouraging empirical breadth (PDF pp. 5 and 7–8, Section 3.1 and Tables 1–3; pp. 17–18, Appendices D.2–D.3). LYRA preserves RoPE, causal masking, softmax normalization, and value aggregation, making its architectural change easy to identify. It reports 36.72 versus 32.21 accuracy for Table 1's Baseline on LongBench-v2 and 89.32 average accuracy on RULER. The additional Llama-3.1-8B experiment and the comparison of placements at layers 15, 24, 33, and 35 broaden the evidence beyond the main Qwen3-8B configuration. The placement study also explicitly acknowledges that gains are not uniform across context lengths. This qualified interpretation is appropriate: the results support placement sensitivity without requiring a monotonic improvement with depth.
Critical weaknesses
-
The main gain cannot yet be attributed to the nonlinear attention rule (PDF pp. 5 and 7, Section 3.1/Table 1; pp. 16–17, Appendix C/Table 5). Appendix C clearly states that training updates the complete final block, approximately 193M parameters, together with 64 added head-specific scalars. The training scope is therefore disclosed, but the paper does not establish whether Table 1's Baseline receives the same LongAlign data, trainable block, optimizer, and training budget. I cannot infer from this ambiguity that the baseline was left unadapted. Clarifying the existing protocol could resolve this part of the concern without a new experiment.
A separate attribution question remains because LYRA changes both query/key normalization and the nonlinear mapping. Table 5 usefully studies κ in {2, 4, 8, 16}, but none of these settings removes the mapping while retaining normalized attention. Equation 8 identifies κ=0 as precisely that cosine-softmax control. Comparable temperature tuning matters because β also changes attention concentration. Thus, the current sensitivity study does not establish the increment over an adapted, normalized attention baseline. A matched comparison among standard attention, temperature-tuned κ=0, and LYRA would distinguish these explanations. This is the main issue for my recommendation: the reported system improvement is encouraging, but the specific contribution of the proposed scoring transformation is still uncertain.
-
The connection between the theoretical condition and the empirical explanation remains incomplete (PDF pp. 1, 3–4 and 6, Figures 1–3 and Equations 9–11). Equation 11 gives a valid conditional improvement over cosine-softmax at the same inverse temperature. It compresses an unfavorable background advantage when the gap-scaling factor is below one, or amplifies an existing evidence advantage when that factor exceeds one. The analysis therefore describes identifiable favorable regimes; it does not establish that those regimes dominate the evaluated model's evidence–background competition. Because the transformation preserves similarity ordering, the relevant empirical question is how its change in relative weights affects the competing attention mass. Measuring the prevalence and importance of the favorable comparisons would connect the algebra more directly to the proposed mechanism.
The existing masking-count sweep, relocation experiment, and fixed-position background attenuation already provide relevant diagnostic evidence. The remaining reproducibility issue concerns their precise construction. Figure 2 defines mean per-token attention density relative to uniform attention, but that definition alone does not specify which examples and spans enter the analysis, how heads and query tokens are aggregated, or the exact relocation positions and attenuation settings in Figure 3. These details affect how broadly one can interpret the diagnostics. This concern limits the strength and reproducibility of the mechanism explanation; it does not imply that the interventions are absent or that the conditional formula is incorrect.
I lean toward weak rejection, primarily because the benefit of the new scoring rule has not been separated from block adaptation and normalized attention. The benchmark gains, controlled synthetic construction, and additional backbone and placement results make the approach promising. A meaningful advantage over matched standard attention and temperature-tuned κ=0 would substantially strengthen the case for acceptance and lead me to reconsider the recommendation. If the existing Baseline already has matched adaptation, making that protocol explicit would remove one uncertainty immediately. The mechanism clarification is secondary and should be weighed alongside the diagnostic evidence already supplied.
Questions for authors
- For W1 — main priority: Does Table 1's Baseline use the same LongAlign data, trainable block, optimizer, and budget? Please clarify the existing protocol first. Under matched training, how do standard attention, κ=0 cosine-softmax with comparable temperature tuning, and LYRA compare on the main evaluation? The key distinction is the increment from the nonlinear mapping after adaptation and normalization are controlled; the current positive-κ sweep cannot resolve that distinction.
- For W2: Please specify the examples, span labels, head/query-token aggregation, and relocation/attenuation settings for Figures 2–3. How frequently does Equation 11's favorable regime occur in the evaluated model, measured over relevant evidence–background comparisons or their competing attention mass? Relating this measurement to the reported diagnostics would clarify how much of the observed behavior the conditional analysis explains.
2 comments
-
(W2) Diagnostic protocol and empirical scope. Thank you for requesting a more reproducible specification and a direct measurement of the regime characterized by Eq. 11. Figures 2 and 3 use the same diagnostic cohort of 400 LongBench-v2 examples. The cohort was selected before inspecting attention outcomes: after applying the evaluation prompt and the 32,768-token head–tail truncation, each example had to retain a single contiguous evidence passage whose final token was at least 2,048 context tokens from the end of <text>. We define the proximal background as the final 256 retained context tokens, excluding all annotated evidence or answer-supporting tokens.
Regarding empirical relevance of Eq. 11, the favorable condition in Eq. 11 holds for 51.61% of evidence–background token pairs under equal weighting and accounts for 58.00% of the competing attention mass under mass weighting. When we compare LYRA with cosine-softmax on the same queries and keys at a fixed inverse temperature per head, the aggregate evidence-to-background attention-mass ratio increases in 60.16% of head–query instances. Under the checkpoint-faithful evaluation, the ratio increases in 49.22% of instances. These results show that favorable score changes are more prevalent among comparisons that carry more competing attention mass, and that their aggregate effect varies across head–query instances. This complements Figure 3 by showing how often LYRA’s score transformation itself shifts the balance toward distant evidence.
We have revised our manuscript to clarify the shared diagnostic cohort and span definitions for Figures 2 and 3 in Appendix C. Implementation. We have also added an empirical analysis immediately after Eq. 11 in Section 3.2, reporting how often the favorable condition holds and how often the span-level evidence-to-background attention ratio increases. Thank you again for your thoughtful and detailed comments.
Thank you again for your time and thoughtful review. We appreciate the opportunity to clarify our findings and would be happy to address any remaining questions. If you find our response satisfactory, could you please consider helping us improve our rating score?
Thank you for your thoughtful and constructive review. We are sincerely grateful for the care with which you examined our work and were particularly encouraged by your recognition of (1) Diagnostic Insight. Our complementary diagnostics and interventions distinguish locating distant evidence from using it under attention competition. (2) Controlled Benchmark Design. ProxBench systematically varies how easily proximal background can be mistaken for evidence while keeping that background irrelevant to the answer. (3) Method Simplicity and Empirical Breadth. LYRA makes a simple change to attention scoring, and we evaluate it across benchmarks, model backbones, and layer placements.
Your encouraging feedback and recognition are a huge encouragement to our team. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly, highlighted in yellow in the revised manuscript.
(W1) Isolating the contribution of the nonlinear mapping. Sincerely thanks for your kind advice. We conduct a controlled ablation on RULER at five context lengths from 8K to 128K. Starting from the same Qwen3-8B checkpoint, we fine-tune the final block on LongAlign dataset under the same training protocol for three variants: standard attention (FT), QKNorm and LYRA. We also report the original Qwen3-8B results as a reference.
Standard fine-tuning achieves 88.04% average accuracy, and the QKNorm reaches 88.33%. LYRA further improves the average to 89.32% and outperforms the normalized control at all five context lengths, with gains ranging from 0.47 to 1.67 percentage points. Thus, after controlling for final-block adaptation and QKNorm, the nonlinear mapping provides an additional 0.99 percentage points of average accuracy on RULER.
The gains across all five context lengths support the utility of the nonlinear mapping. Together with our attention diagnostics, these results suggest that reshaping similarity gaps helps mitigate the Proximity Trap, allowing the model to make better use of distant evidence when it competes for attention with nearby, task-irrelevant background.
We have added the matched RULER results as Table 9 in Appendix D.4, along with a discussion of LYRA’s gains over the standard attention and QKNorm controls under matched LongAlign fine-tuning. Thank you again for suggesting this comparison. It helped us clarify what LYRA adds beyond fine-tuning and Q/K normalization.