acceptodds
Under review as a conference paper at ICLR 2027

Rotary Attention in the Frequency Domain: Value Rotation as a Distance Filter

Abstract

Rotary position embedding (RoPE) encodes position in the attention score but leaves the values unrotated, so what a head reads carries no record of where it came from. We study the variant that also rotates the values and inverse-rotates the head output (QKVO-RoPE), which adds no parameters. We show that this turns each attention head into a filter over distance. Content that a head reads from a single position passes and keeps a phase that records its distance, whereas content that it averages over many positions partly cancels. To test this account, we train matched Transformer pairs from 130M to 1.3B parameters that differ only in the rotation and measure the cancellation in every head. We find that value rotation helps where one position must be read against a diffuse context. Beyond the training length the larger rotated models retrieve needles that their unrotated counterparts miss entirely, and up to 760M the rotated models extrapolate with lower perplexity. We also find that the costs fall where the analysis predicts, in heads that average and in associative recall, whose source lies at a distance set by the query, while the cost at the training length is small. A larger rotary base reduces the cancellation. Our results suggest rotating the values in models that must retrieve from long contexts, preferably with a large rotary base.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformers, rotary position embedding, attention mechanism, length extrapolation, long-context retrieval, complex poles

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rFMnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the existing QKVO-RoPE construction affects content aggregation and retrieval. It represents value rotation as a lag-dependent phase readout and characterizes attenuation of shared value components through the attention-lag distribution’s characteristic function. Paired experiments across 130M–1.3B models evaluate frequency allocation, perplexity, retrieval, copying, and downstream performance, supplemented by rotation and frequency-rescaling ablations. Results include 8K needle-retrieval gains alongside task-dependent regressions and a reversal in length-extrapolation behavior at the largest scale.

Critical strengths
  1. The derivations distinguish attenuation of shared components from preservation of independent zero-mean components’ expected energy.
  2. Matched data, ordering, and hyperparameters across four model scales, together with nine rotation combinations, provide substantial diagnostic breadth.
  3. Contribution beyond an established operator: Quantitative frequency analysis and systematic reporting of benefits, regressions, and failed predictions provide useful mechanistic and empirical insight.
Critical weaknesses
  1. Large absolute phase does not imply unreadability when nearly constant lags can be compensated by a fixed projection. The analysis needs explicit phase-distribution and decoder assumptions, a distinction between sufficient and necessary conditions, and qualification of the continuous approximation for finite integer positions.
  2. Successful 8K retrieval uses repeated-paragraph backgrounds and source-to-query distances within the training range, while all 16K conditions fail. Without non-repetitive backgrounds and paired NTK-scaled retrieval controls, the evidence supports conditional interference robustness rather than general long-distance retrieval recommendations.
  3. The PG-19 diagnostics neither establish that attenuated content is harmful nor measure the relevant retrieval computation.
Questions for authors
  1. Would matched-data-order, same-base QK controls and factorial score/value rescaling support the proposed value-path attribution, given the confounds in the secondary interventions?
  2. Are independent paired training runs available to determine whether small task differences and scale-dependent reversals persist beyond checkpoint-conditional prompt-sampling uncertainty?
  3. See weaknesses.
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the comments, which helped us clarify the assumptions in Section 3.

W1. Whether a fixed projection can read the content depends on how much the lag (how far back the head attends) varies across inputs, not on how large it is. Our fixed-lag result is a special case where the lag does not vary. We have made three changes to Section 3.3.

  • Phase distribution and decoder. The condition is now stated in terms of the spread of the lag, and for a fixed linear readout.
  • Sufficient vs necessary. We now clarify that the condition is sufficient but not necessary, and that we do not claim the content is lost to the model as a whole.
  • Integer positions. Eq. (4) is the geometric sum over integer lags. Only the pair count JWJ_W treats jj as continuous, and we now point to the exact integer counts in the appendix.

The shared-content and zero-mean independent-content assumptions are stated in Section 3.3 and Appendix A.

Revised text (Section 3.3, query-dependent lag)

“What matters is the spread of the lag, not its size. If ℓ⋆(m)\ell^\star(m) stays within ℓ0±Δ\ell_0 \pm \Delta over all inputs the head processes, the fixed projection WoRℓ0W_o R_{\ell_0} removes the common phase and leaves a residual phase of at most θjΔ\theta_j\Delta, so pair jj stays readable whenever θjΔ≪1\theta_j\Delta \ll 1, however large θjℓ0\theta_j\ell_0 is. The fixed-lag case above is Δ=0\Delta = 0. The condition θjℓ⋆≪1\theta_j\ell^\star \ll 1 is therefore sufficient for a readout by a fixed linear projection but not necessary. This limit applies only to such a fixed linear readout. A later layer that reads the phase nonlinearly, or that learns the lag through another path, is not limited in this way, so we do not claim that the content is lost to the model as a whole.”

Revised text (Section 3.3, after the count JWJ_W)

“Eq. (4) is exact for integer lags. Only the count JWJ_W treats the pair index jj as continuous. The exact count is the number of integers jj with Wθj<1W\theta_j < 1, which the appendix lists for a 64-dimensional head.”

W2. We ran two new controls, a non-repetitive background and paired NTK-scaled retrieval, on all four pairs (full numbers in our response to Reviewer WZ2A). In the paired control, both models get the same NTK-aware scaling. NTK-aware scaling raises the rotary base at test time, so that a RoPE model can handle inputs longer than it was trained on.

  • Non-repetitive background. We used continuous web text from LongCrawl64, which is not part of the training data. With NTK-aware scaling on both models, the value-rotated models retrieve more needles at 8K at all four model sizes (for example 28.9 vs 19.9% at 1.3B).
  • Paired NTK-scaled retrieval. Beyond the training length, the value-rotated 760M and 1.3B models still find more needles at 16K, 46.1 vs 3.7% and 24.5 vs 0% (full table in our response to Reviewer WZ2A).
  • Distance sweep at 8K. With NTK-aware scaling, the 760M and 1.3B models find needles up to 5000 tokens back, beyond the training length. There the value-rotated models find more (34.5 vs 27.5% and 40.0 vs 13.0%, significant at 1.3B, p<10−8p < 10^{-8} after Bonferroni correction). From 6000 tokens on, every model finds at most 6%.

Value rotation therefore improves needle retrieval beyond the 4K training length when both models use NTK-aware scaling. On natural text at 8K, the value-rotated models find more needles than QK-RoPE at all four model sizes, significant at each size. On the repeated haystack at 16K, the value-rotated 760M and 1.3B models find 46.1% and 24.5% of needles, against 3.7% and 0% for QK-RoPE. These results address the concern that the gain reflects only robustness to interference on repeated backgrounds. The gain also holds on natural text and with the same NTK-aware scaling on both models, and it reaches needles up to 5000 tokens back.

W3. We agree that the PG-19 diagnostic alone cannot show that the attenuated content is harmful, and we now state this limitation in Section 4.3. The head-level measurement shows where the cancellation happens and how the trained models adapt, by moving output energy to the low-frequency pairs. Whether the cancellation helps or hurts is tested by the task results in the following sections of the paper, and retrieval itself by the needle tests in our response to Weakness 2. The estimate made on a QK-RoPE model also matches the trained value-rotated model. The attenuation that value rotation would cause in the QK-RoPE model at base 10610^6 (mean ∣φ∣|\varphi| 0.708) is almost the same as the attenuation measured in the value-rotated model (0.710).

AuthorsOpenReview ↗

Rebuttal (2/3)

Q1. We trained the matched-data-order, same-base QK-RoPE control, and we completed the factorial score/value rescaling. The paper already had the rescaling table for the value-rotated model, and we added a QK-RoPE column to it.

Same-base QK control. In addition to the 130M QKVO-RoPE model at base 10610^6 from the submission, we trained a 130M QK-RoPE model at base 10610^6 and retrained the 130M QKVO-RoPE model at base 10410^4 with the same setup. The rotary base is the RoPE hyperparameter that sets how fast each pair of dimensions rotates with position. All three models use the same training data, number of tokens, hyperparameters, data order and seed.

  • Effect of the base, with the data order matched. Raising the base from 10410^4 to 10610^6 improves the value-rotated model throughout. PG-19 perplexity at 4K drops from 28.23 to 26.69, and needle accuracy at 4K rises from 33.6% to 82.8%.
  • Effect of value rotation at the same base. At base 10610^6, QKVO-RoPE finds more needles than QK-RoPE, 95.8% vs 90.6% at 2K and 82.8 vs 72.8% at 4K, at nearly the same perplexity at 4K (26.69 vs 26.80).
  • Head level attenuation. The attenuation that value rotation causes in the QK-RoPE model at base 10610^6 (mean ∣φ∣|\varphi| 0.708) is almost the same as the attenuation measured in the value-rotated model (0.710).

So the larger base helps both models, and part of its gain comes through the score. We revised Section 4.7 and the Discussion accordingly.

Factorial score/value rescaling. Here, rescaling means changing the rotary frequencies at test time in the attention score, the value path, or both. Rescaling both paths together corresponds to NTK-aware scaling. The paper already has a table that rescales the score and the value path separately for the value-rotated model, covering all four cases (none, score only, value only, and both). To complete the 2×22\times 2 comparison with the baseline, we added a QK-RoPE column to the table.

With the score rescaled in both models, QK-RoPE has the same or lower perplexity than QKVO-RoPE at every model size (e.g., 18.3 vs. 23.0 at 760M and 8K). Rescaling the value path as well brings QKVO-RoPE to nearly the same perplexity as QK-RoPE at 760M (18.4 vs. 18.3).

The perplexity advantage of the value-rotated model beyond the training length therefore disappears once the attention score is rescaled. This is consistent with Section 4.7, where we show that part of the gain comes through the score. The retrieval advantage, on the other hand, holds when both models use NTK-aware scaling (see our response to W2).

AuthorsOpenReview ↗

Rebuttal (3/3)

Q2. Our significance tests treat the test prompts as random and use one trained model per setting, so they do not include variation between training runs. One estimate of run-to-run variation is already available from two 130M QKVO-RoPE runs at base 10410^4. They reach PG-19 perplexity 28.83 and 28.23 at 4K, and retrieve 38.4% and 33.6% of needles at 4K.

Differences within the 130M pair at the training length are similar, so the revised paper now describes them as inconclusive. The differences we base our conclusions on are much larger. For example, needle retrieval beyond the training length improves by 11-49 points at 760M and 1.3B.

Within-model variance versus QK/QKVO variance. Two additional training runs show whether the differences between QK-RoPE and QKVO-RoPE are larger than the differences between two runs of the same model. We trained 370M QK- and QKVO-RoPE models using two different seeds (42 and 123). A seed changes both the parameter initialization and data order. All other settings are the same for both seeds, and all models were trained on the same hardware.

370M QK-RoPE and QKVO-RoPE models with 64-dimensional heads, each trained with two seeds on 10.7B tokens of FineWeb-Edu at a 4K context with identical hyperparameters. The seeds (42 and 123) change only the initialization and the data order. PG-19 perplexity, lower is better. NTK means NTK-aware scaling at test time. The better model of each seed is in bold.

seedmodel1K2K4K8K16K8K NTK16K NTK
seed 42QK-RoPE23.1421.8021.0532.25196.721.1924.12
QKVO-RoPE22.5821.3920.8124.8861.5620.8623.83
seed 123QK-RoPE22.5921.1920.3735.15110.520.4623.82
QKVO-RoPE22.7921.5320.9325.7387.3521.3724.76

The second seed confirms our main perplexity result. At 8K without NTK-aware scaling, the value-rotated model has lower perplexity at both seeds (24.9 vs 32.3 and 25.7 vs 35.2), and this gap is more than twice the change caused by the seed itself, which is less than 3 points for either model. At 4K, the two encodings stay within one point of each other at both seeds, as the paper already describes for differences at the training length. The perplexity advantage beyond the training length without NTK-aware scaling therefore persists across independent training runs.

Revised text (Appendix G)

“They are therefore conditional on the trained checkpoints and do not include variation between training runs. … Differences within the 130M pair at the training length are of this size and should be read as inconclusive.”

Reviewer_WZ2AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies RoPE with value rotation and inverse output rotation (QKVO-RoPE). It shows that each value carries a distance-dependent phase, which can cancel content shared across attended positions. Language models from 130M to 1.3B parameters are trained to examine proposed method through head level measurements, perplexity, copying, associative recall, and needle retrieval. Results show advantages for some retrieval settings, but also costs and inconsistent gains across model sizes.

Critical strengths
  1. The analysis explains a useful trade-off: value rotation can suppress shared background content, but can also hurt aggregation and reads at varying distances. Head-level measurements support the predicted cancellation and show that trained models place more output energy in low-frequency pairs.

  2. The matched RoPE pairs provide a clean test of value rotation. The paper also reports negative results, frequency-rescaling controls, and training overhead, rather than presenting the change as a consistent improvement.

Critical weaknesses
  1. The advantage over other positional methods is unclear.

XPos and NoPE are included, but their training budgets differ from that of QKVO-RoPE. NTK-aware scaling removes most of the perplexity advantage over standard RoPE, while needle retrieval is not evaluated with the same scaling applied to both models. The current comparisons therefore provide limited support for recommending value rotation for long-context retrieval.

  1. The retrieval evaluation is limited.

The test uses a repeated passage as background. Although the total context exceeds the training length, all successful retrievals beyond that length involve target-to-query distances within the training range. It remains unclear whether the gains hold with more varied background text or targets at larger distances.

  1. Some explanations need controlled ablations.

The paper suggests that head dimension and QK-normalization may explain some differences across model sizes, but does not test these factors separately. The larger-base experiment also lacks a standard RoPE baseline at the same base. These controls would help separate the effects of value rotation from those of other design choices.

Questions for authors
  1. Can you compare QKVO-RoPE with XPos and NoPE under the same training budget, and report needle retrieval with NTK-aware scaling applied to both RoPE variants?

  2. Do the retrieval gains hold with non-repeated backgrounds? At a fixed context length, how does performance change as the target-to-query distance exceeds the 4K training range?

  3. Can you provide controlled ablations for head dimension and QK-normalization?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the thoughtful review. We ran all the suggested experiments. The retrieval tests cover all four model pairs, from 130M to 1.3B parameters, and the controlled comparisons use new 130M and 370M models. The revised text for the paper is quoted under each point. Unless stated otherwise, each needle cell has 200 trials, and we test significance with Fisher’s test and Bonferroni correction.

W1 / Q1. Thank you for these two suggestions. We agree that both suggested experiments would add insight.

(a) Needle retrieval with NTK-aware scaling on both models. NTK-aware scaling increases the rotary base at test time, allowing RoPE models to handle sequences longer than their 4K training length. We applied the same scaling to every rotary module in both models and evaluated needle retrieval at all four model sizes.

haystackcontext130M370M760M1.3B
repeated (as in paper), 200 trials8K29.7 vs 22.1*32.0 vs 40.9*55.2 vs 43.8*59.6 vs 10.7*
repeated (as in paper), 200 trials16K2.2 vs 3.613.5 vs 3.4*46.1 vs 3.7*24.5 vs 0.0*
non-repeated LongCrawl64, mean over distances8K16.2 vs 10.4*22.1 vs 14.3*33.4 vs 28.1*28.9 vs 19.9*

(needle accuracy (%), QKVO-RoPE vs QK-RoPE, higher is better, * significant after Bonferroni correction)

This matches our explanation. NTK-aware scaling brings the rotation angles in the attention scores back into the range seen during training. Value rotation, in addition, reduces the interference from the rest of the context when a retrieval head reads the needle.

(b) Training budgets of XPos and NoPE. For a budget-matched comparison, we trained 370M XPos and NoPE models with the same training data, number of tokens (10.7B), hyperparameters, data order and seed as a 370M QK-RoPE and QKVO-RoPE pair trained on the same hardware. With NTK-aware scaling, QKVO-RoPE retrieves the most needles beyond the training length, and the three RoPE models reach nearly the same perplexity (table below).

Budget-matched 370M models, trained on 10.7B tokens of FineWeb-Edu at a 4K context with identical hyperparameters, data order and seed, on the same hardware. Needle accuracy (%) on non-repeated LongCrawl64 text with NTK-aware scaling, 1000 trials per context (200 per distance), and PG-19 perplexity. Best in bold. * QKVO-RoPE is significantly better than each of the other three models (Fisher’s exact test, Bonferroni correction).

modelneedle 8K ↑needle 16K ↑needle 2000 tokens back, 8K ↑PG-19 ppl 4K ↓PG-19 ppl 8K, NTK ↓PG-19 ppl 16K, NTK ↓
QK-RoPE10.00.313.021.0521.1924.12
QKVO-RoPE18.6*5.2*53.5*20.8120.8623.83
XPos11.00.421.021.1721.0422.19
NoPE0.00.00.023.3690.85467.3

Without NTK-aware scaling, XPos, which is designed for length extrapolation, has the lowest perplexity beyond the training length (PG-19 21.3 at 16K). We will add these results to Tables 2 and 3.

AuthorsOpenReview ↗

Rebuttal (2/3)

W2 / Q2. Thank you for this close look at our setup. We ran new experiments on all four pairs. Beyond the training length, the value-rotated models also find more needles on natural text.

(a) Non-repeated background. The haystack is a continuous passage of web text from a randomly drawn LongCrawl64 document. We already use LongCrawl64 for extrapolation perplexity, and it is not part of the training data (FineWeb-Edu). We insert the needle at a sentence boundary, and its words never appear in the haystack.

Beyond the training length. At 8K with NTK-aware scaling on both models, the value-rotated models find more needles at all four model sizes (last row of the table above). The advantage beyond the training length that the paper reports also holds on natural text.

Within the training length. On natural text within 4K, value rotation gives no consistent advantage (needle accuracy (%), QKVO-RoPE vs QK-RoPE, higher is better).

model size1K2K4K
130M71.3 vs 67.443.5 vs 49.225.6 vs 24.4
370M66.5 vs 58.860.0 vs 51.231.8 vs 28.9
760M69.0 vs 73.861.1 vs 67.940.9 vs 44.0
1.3B76.4 vs 80.968.8 vs 75.645.1 vs 53.9

Without NTK-aware scaling, no model finds more than 6.1% of the needles at 8K on LongCrawl64. The gap is largest on the repeated haystack, where the background content around the needle is the same at every position.

(b) Distance beyond 4K at a fixed context. We kept the context fixed at 8K tokens of LongCrawl64 text and placed the needle 256 to 8000 tokens before the question.

  • With NTK-aware scaling on both models, the 760M and 1.3B models find needles up to 5000 tokens back, which is beyond the training length. At that distance the value-rotated models find more, 34.5 vs 27.5% at 760M and 40.0 vs 13.0% at 1.3B (significant at 1.3B, p<10−8p < 10^{-8} after Bonferroni correction). From 6000 tokens on, every model finds at most 6%.
  • On the repeated haystack at 16K with NTK-aware scaling, the value-rotated 760M and 1.3B models retrieve needles placed 8K tokens back in 62% and 40% of trials, respectively, compared with 17% and 0% for QK-RoPE.

So retrieval from beyond the training length needs NTK-aware scaling, and with it the larger value-rotated models find more needles than QK-RoPE.

Revised text (Section 4.6, end of the new paragraph)

“The benefit of the rotation for needle retrieval therefore depends on the context length and on the background text. This benefit is clear beyond the training length once both models are NTK-aware scaled, and largest on a repetitive haystack. On natural text within the training length it is absent, and at 760M and 1.3B partly reversed.”

Revised text (Abstract)

“With NTK-aware scaling applied to both models, the larger value-rotated models retrieve more needles than their unrotated counterparts beyond the training length.”

AuthorsOpenReview ↗

Rebuttal (3/3)

W3 / Q3. Thank you for these suggestions. We ran new experiments for all three suggested controls, the standard RoPE baseline at the same base in (a), the QK-normalization ablation in (b) and the head-dimension ablation in (c), to separate the effect of value rotation from other design choices.

(a) Standard RoPE baseline at the same base. The rotary base is the RoPE hyperparameter that sets how fast each pair of dimensions rotates with position. A larger base means slower rotation. In addition to the 130M QKVO-RoPE model at base 10610^6 from the submission, we trained a 130M QK-RoPE model at base 10610^6 and retrained the 130M QKVO-RoPE model at base 10410^4 with the same setup. All three models use the same training data, number of tokens, hyperparameters, data order and seed. The three models differ only in the base and in value rotation. In the table, ↓ means lower is better and ↑ means higher is better.

130M, matched data orderPG-19 ppl 4K ↓PG-19 ppl 16K ↓needle 2K ↑needle 4K ↑zero-shot ↑
QKVO-RoPE, base 10410^428.23142.158.033.640.0
QK-RoPE, base 10610^626.80134.990.672.840.6
QKVO-RoPE, base 10610^626.69131.895.882.840.4

The larger base helps both models, so part of its gain comes through the attention score. At the same base of 10610^6, the value-rotated model finds more needles (+10.0 points at 4K) at nearly the same perplexity. The head-level results are consistent with this finding. The attenuation that value rotation would cause in the QK-RoPE model at base 10610^6 (mean ∣φ∣|\varphi| 0.708) is almost the same as attenuation measured in the value-rotated model (0.710).

Revised text (Section 4.7)

“With the data order matched, raising the base from 10410^4 to 10610^6 improves the value-rotated model in every column we measured … At base 10610^6 the value-rotated model still retrieves more needles than the unrotated one (95.8 against 90.6% at 2K and 82.8 against 72.8% at 4K), with nearly equal perplexity … The larger base thus helps both encodings, and part of its gain comes through the score.”

(b) QK-normalization. We trained a new 130M pair with QK-normalization in both models, using the same training data, number of tokens, hyperparameters and data order as our 130M pair. In the table, lower perplexity is better, and a ratio below 1 means the value-rotated model has the lower perplexity.

PG-19 perplexity ↓, QKVO-RoPE / QK-RoPE (ratio)4K8K16K
130M, no QK-normalization28.8 / 28.1 (1.02)58.8 / 67.5 (0.87)224.3 / 309.9 (0.72)
130M, QK-normalization (new)27.1 / 26.9 (1.01)28.8 / 27.1 (1.06)41.8 / 30.7 (1.36)
1.3B, QK-normalization15.3 / 14.7 (1.04)16.6 / 15.8 (1.05)32.0 / 22.1 (1.45)

Without QK-normalization, the value-rotated 130M model has lower perplexity than QK-RoPE beyond the training length, with a QKVO/QK ratio of 0.87 at 8K and 0.72 at 16K. So the 1.3B pattern appears at 130M as soon as QK-normalization is added. This points to the combination of value rotation and QK-normalization, not model size, as the cause of the 1.3B reversal. QK-normalization also keeps the attention scores of both models from growing too large beyond the training length (QK-RoPE perplexity at 16K falls from 309.9 to 30.7).

(c) Head dimension. We trained a new 370M model pair with 128-dimensional heads, with all other settings as in our 370M model pair.

370M QK-RoPE and QKVO-RoPE models with 64- or 128-dimensional heads, trained on 10.7B tokens of FineWeb-Edu at a 4K context with identical hyperparameters, data order and seed, on the same hardware. PG-19 perplexity (lower is better) and MQAR associative-recall accuracy (%, higher is better). The better model at each head size is in bold.

head dimensionmodelPG-19 ppl 4K ↓PG-19 ppl 8K ↓PG-19 ppl 16K ↓MQAR ↑
64QK-RoPE21.0532.25196.727.10
QKVO-RoPE20.8124.8861.5621.34
128QK-RoPE20.4628.9484.4527.32
QKVO-RoPE20.0823.1657.8619.83

With larger heads, the value-rotated model keeps its lower perplexity at every length, including 4K (20.08 vs 20.46).

Reviewer_UDnGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper discusses QKVO-RoPE attention, analyzing where it has an advantage over the standard QK-RoPE attention, where it falls behind, and why. The paper specifically analyses how rotary values cancel each other over longer sequences. A key finding is that QKVO-RoPE attention can be beneficial for some applications, although in absolute terms the reported performance of QKVO-RoPE attention is still far from solving the problem. Experiments with models of different size suggest a nontrivial interaction between model size and the type of attention.

Critical strengths
  • experiments with training language models at different sizes and different attention variants
  • elegant discussion of rotary values cancelling each other out in diffuse contexts
  • clear message and good visuals
  • the math is genuinely helpful
Critical weaknesses
  • the paper does not sufficiently acknowledge continuity with previous work. Specifically, more continuity should have been acknowledged with Su (who literally uses the same term QKVO-RoPE: the authors did not coin it) but also others (RoPER approach by Harik and Jayasiri).
  • perhaps as a result of AI assistance, parts of the text are hard to read due to unusual word choice (such as recurring "cost"), and parts because they seem too condensed. For example, most sentences of Related Work read like the sentence is a one-sentence summaries of what could previously have been a paragraphs that explained the relevance and content of the related work in question in a coherent way. But together, the sentences of Related work do not form a coherent text that a human can read normally, unless they already know the content of all the works mentioned and can guess the links between them. There are also other parts of the text beyond Related Work where the sentences seem similarly condensed.
Questions for authors
  • one example is the sentence "the proportion of each head type can be estimated from a model’s attention distances before retraining." I think I have a reasonable guess on what this means, and it can be an interesting idea, but the paper never goes back to it to elaborate.
  • how practical is the use of QKVO-RoPE in reality, especially given the interaction of model size with attention type?
  • There doesn't seem to be a significant computational penalty to QKVO-RoPE even though the extra rotation seems like it could add latency. Is this the case and why?
  • do you have an explanation for attention type vs model size interactions?
  • I wonder how much AI assistance was actually used in creating the text.
5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the positive assessment and the helpful comments. We have addressed each comment below.

W1. We agree that the original wording (“We call this variant QKVO-RoPE”) did not credit prior work clearly enough. Su’s work was already cited three times in the submission (Related Work, p. 3; Section 3.2, p. 4; Appendix I, p. 16), and the revision now credits it explicitly.

  • The Introduction now attributes the value rotation and the name QKVO-RoPE to Su and cites RoPER as its earliest form.
  • The Introduction also clarifies how our work relates to these papers. They introduced the value rotation operator and motivated it through relative position in the output, and key-value sharing in latent attention. We show that value rotation turns each attention head into a filter over distance, which we characterize through a closed-form analysis of cancellation and measurements on trained heads. We also provide controlled comparisons at model sizes up to 1.3B, showing where rotation helps (single-position retrieval and extrapolation) and where it hurts (averaging and associative recall).

Revised text (Introduction, paragraph 2)

“We study a variant of RoPE that also rotates the values and then rotates the head output back by the position of the query. Su (2025) introduced this variant and named it QKVO-RoPE. Harik and Jayasiri (2022) had proposed an earlier form, RoPER. We derive how much of the content averaged by an attention head this rotation cancels. We then measure its effect in Transformers from 130M to 1.3B parameters. Beyond the training length, the value-rotated 760M and 1.3B models retrieve 11 to 49 points more needles.”

W2. Thank you for pointing this out. Since the submission, we have worked with a native English-speaking domain expert to clarify the previously vague, non-scientific wording. Some examples are provided below.

Introduction. “Beyond the training length, the larger rotated models retrieve needles that their unrotated counterparts fail to retrieve.” is now “At twice the 4K training length, on a repeated haystack, the value-rotated 760M and 1.3B models retrieve 14-18% of needles and their unrotated counterparts retrieve 0% (Section 4.6).”

Related Work. “Barbero et al. (2025) find that the high-frequency pairs of a trained model detect position while the low-frequency pairs carry content, and Section 3.3 gives the corresponding result for the value path.” is now “Barbero et al. (2025) looked at how trained models use rotary frequencies. They find that high-frequency pairs detect position while the low-frequency pairs carry content. Section 3.3 derives the corresponding result for the value path.”

  • Replaced every use of “cost” with a concrete term such as “lowers accuracy”, “increases perplexity”, “impairs copying”, depending on what is meant in each place.
  • Expanded the sentence on head types that the reviewer quotes (see Q1 below).
  • Rewrote Related Work as connected paragraphs that explain what each line of work does and how it relates to value rotation. The section is now organized as three paragraphs: oscillation in sequence models, position in the attention score, and rotation of the values. The last of these brings together the prior work on value rotation (see Weakness 1).
  • Revised the wording throughout the paper. We went through every section and rewrote passages that used a term before defining it or assumed knowledge the reader may not have. Each technical term is now defined where it first appears, for example the rotary base, NTK-aware scaling, QK-normalization and the attenuation ∣φ(θj)∣|\varphi(\theta_j)|. Condensed sentences are split into shorter ones, and vague or unusual words are replaced with concrete descriptions of what is meant.

Q1. Thank you for pointing this out. The estimate in this sentence comes from the head-level measurement of Section 4.3, applied to a QK-RoPE model. On held-out text, we record how far back each head attends and compute how much of its content value rotation would cancel (the attenuation ∣φ(θj)∣|\varphi(\theta_j)| of Section 3). The QK-RoPE rows of our head-level table are this estimate, computed without retraining. In our QK-RoPE models, 78%, 79% and 95% of heads at 370M, 760M and 1.3B respectively have a mean attention distance above 256 tokens. So most heads average over many positions, and rotating their values would remove 41-44% of the content they share. We rewrote the sentence to explain the measurement.

Revised text (Introduction, last paragraph)

“Which of the two effects dominates in a given model depends on how its heads distribute their attention. This can be assessed on an existing QK-RoPE model without retraining by measuring the attenuation ∣φ(θj)∣|\varphi(\theta_j)| associated with value rotation for each head, using held-out text. We perform this analysis for our QK-RoPE models in Section 4.3.”

AuthorsOpenReview ↗

Rebuttal (2/2)

Q2. QKVO-RoPE is cheap to use. It adds no parameters and makes a training step only 1.3-2.0% slower. From 370M upwards, it changes the training loss at 4K by at most 0.006 nats. Its main benefit is needle retrieval beyond the training length. With NTK-aware scaling (a standard test-time method for running a RoPE model beyond its training length) applied to both models, the value-rotated 760M and 1.3B models find 46.1% and 24.5% of needles at 16K, against 3.7% and 0% for QK-RoPE (see our response to Reviewer WZ2A).

Our analysis suggests two ways to use it in practice. The first is to rotate only part of each head’s dimensions, as DeepSeek-V4 does, so that an unrotated copy of the content is kept (Discussion). The second is to use a larger rotary base. A larger base helps QK-RoPE as well. It matters for value rotation because slower rotation cancels less of the content that a head averages (Section 3), and at base 10610^6 the value-rotated model still retrieves more needles than QK-RoPE (82.8 vs 72.8% at 4K).

Q3. Yes, the penalty is small. A training step is 1.3-2.0% slower at every model size. The two extra rotations are elementwise multiplications of the T×dT\times d values and head outputs by sines and cosines. These rotations take O(Td)O(Td) operations per head, while computing dense attention takes O(T2d)O(T^2 d), where TT is the sequence length and dd is the head dimension. Attention therefore dominates at T=4096T=4096.

Revised text (Section 4.2)

“The two extra rotations slow a training step by 1.3 to 2.0%. They are elementwise operations on the T×dT\times d values and head outputs and take O(Td)O(Td) operations per head, while computing dense attention takes O(T2d)O(T^2 d), where TT is the sequence length and dd is the head dimension.”

Q4. Yes. Two settings differ between our model sizes, and each lines up with one of the size-dependent results. We tested both, head dimension with a new 370M model pair and QK-normalization with a new 130M model pair.

  • Head dimension. Only our 760M models have 96-dimensional heads, and associative recall improves only at 760M. We tested head dimension directly with a new 370M model pair with 128-dimensional heads (full table in our response to Reviewer WZ2A). With larger heads, value rotation keeps its lower perplexity at every length.
  • QK-normalization. In the submission, QK-normalization (which normalizes queries and keys before their dot product) was used only in the 1.3B pair. This was also the only pair where the value-rotated model had higher perplexity than QK-RoPE beyond the training length.

To test the second explanation, we trained a new 130M pair with QK-normalization in both models. Without QK-normalization the value-rotated 130M model has lower PG-19 perplexity beyond the training length (ratio QKVO/QK 0.72 at 16K, where below 1 means the value-rotated model is better). With QK-normalization in both models, the ratio becomes 1.36, close to the 1.45 of the 1.3B pair. The 130M pair now shows the same pattern as the 1.3B pair, which supports the QK-normalization explanation.

Q5 and ethics concern. In the initial submission form, we declared AI assistance “to aid or polish writing”, but the AI-use statement in the paper said only that an AI tool was used “to check for grammar mistakes”. The statement in the paper was therefore incomplete, and we have corrected it. For the submission, we used Claude (Claude Opus 5.5, Anthropic) to revise the wording of text that we had written ourselves, including rephrasing and shortening passages.

For the revision, a native English-speaking domain expert went through the revised text with us, and we checked every revised passage for technical correctness and for clear, precise wording (see our response to Weakness 2 for examples). All research ideas, the analysis of value rotation, the derivations in Section 3 and Appendix A, the experimental design, the training and evaluation of every model, and all results and figures are our own work, and we take full responsibility for all content. The corrected AI-use statement in the paper is quoted below.

Revised AI-use statement (paper)

“We used an AI tool (Claude Opus 5.5, Anthropic) to revise and polish the wording of text we had written. We have reviewed all AI-assisted work and take responsibility for the final content of this work, including the text and artifacts produced with the aid of generative AI. We did not use generative AI tools to generate research ideas, design the models or experiments, or produce any results. All derivations, experiments, analyses and results are our own.”

We are glad to provide any further information that the reviewer, the Area Chair or the ethics reviewers may need.

Thank you for taking my comments into account and providing specific responses to questions. Would it be possible to see the updated version of the paper in its entirety?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

We appreciate the reviewers’ constructive comments and suggestions.

Following the reviewers’ suggestions, we ran new experiments on our matched model pairs. In each pair, a standard RoPE model (QK-RoPE) and a QKVO-RoPE model differ only in value rotation, which means that QKVO-RoPE also rotates the values by their position and rotates the head output back by the position of the query. New experiments confirm that the value-rotated models keep their retrieval advantage beyond the training length. The main changes are listed below.

1. Long-context needle retrieval controls (WZ2A, rFMn). These controls test whether the retrieval advantage of value rotation holds beyond the training length, under the conditions the reviewers requested. We ran three controls on all four pairs and tested every comparison with Fisher’s exact test and Bonferroni correction. First, we applied NTK-aware scaling to both models. NTK-aware scaling is a standard way to run a RoPE model on inputs longer than its training length. At test time, the rotary base is increased so that distances beyond 4K give rotation angles the model was exposed to during training. Second, we replaced the repeated passage around the needle with non-repeated web text from LongCrawl64. Third, we placed the needle more than 4K tokens before the question, so that the model must look further back than it does during training. These results are summarized in two new tables.

  1. Same-base control (WZ2A, rFMn). This control separates the effect of the rotary base from the effect of value rotation. The rotary base is the RoPE hyperparameter that sets how fast each pair of dimensions rotates with position, and a larger base means slower rotation. In addition to the 130M QKVO-RoPE model at base 10610^6 from the submission, we trained a 130M QK-RoPE model at base 10610^6 and retrained the 130M QKVO-RoPE model at base 10410^4 with the same setup. All three models use the same training data, number of tokens, hyperparameters, data order and seed.
  2. QK-normalization ablation (UDnG, WZ2A). This ablation tests why our 1.3B pair behaves differently from the smaller sized model pairs. QK-normalization normalizes the queries and keys before their dot product. We trained a new 130M pair with QK-normalization in both models.
  3. Run-to-run variation (rFMn). Our significance tests treat the test prompts as random but use one trained model per setting. So they do not show how much results change between training runs. Two of our 130M QKVO-RoPE models were used to estimate run-to-run variation. They have the same settings, except that they saw the data in a different order.
  4. Budget-matched XPos and NoPE (WZ2A). This comparison tests value rotation against other positional encodings trained on the same budget. We trained 370M XPos and NoPE models with the same training data, number of tokens (10.7B), hyperparameters, data order and seed as a 370M QK-RoPE and QKVO-RoPE pair.
  5. Training with a second seed (rFMn). This tests whether the differences between QK-RoPE and QKVO-RoPE are larger than the differences between two runs of the same model. We trained 370M QK-RoPE and QKVO-RoPE models with a second seed (123, in addition to 42). A seed changes both the parameter initialization and the data order.

Our answers to each point follow under each review. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.