- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the existing QKVO-RoPE construction affects content aggregation and retrieval. It represents value rotation as a lag-dependent phase readout and characterizes attenuation of shared value components through the attention-lag distribution’s characteristic function. Paired experiments across 130M–1.3B models evaluate frequency allocation, perplexity, retrieval, copying, and downstream performance, supplemented by rotation and frequency-rescaling ablations. Results include 8K needle-retrieval gains alongside task-dependent regressions and a reversal in length-extrapolation behavior at the largest scale.
Critical strengths
- The derivations distinguish attenuation of shared components from preservation of independent zero-mean components’ expected energy.
- Matched data, ordering, and hyperparameters across four model scales, together with nine rotation combinations, provide substantial diagnostic breadth.
- Contribution beyond an established operator: Quantitative frequency analysis and systematic reporting of benefits, regressions, and failed predictions provide useful mechanistic and empirical insight.
Critical weaknesses
- Large absolute phase does not imply unreadability when nearly constant lags can be compensated by a fixed projection. The analysis needs explicit phase-distribution and decoder assumptions, a distinction between sufficient and necessary conditions, and qualification of the continuous approximation for finite integer positions.
- Successful 8K retrieval uses repeated-paragraph backgrounds and source-to-query distances within the training range, while all 16K conditions fail. Without non-repetitive backgrounds and paired NTK-scaled retrieval controls, the evidence supports conditional interference robustness rather than general long-distance retrieval recommendations.
- The PG-19 diagnostics neither establish that attenuated content is harmful nor measure the relevant retrieval computation.
Questions for authors
- Would matched-data-order, same-base QK controls and factorial score/value rescaling support the proposed value-path attribution, given the confounds in the secondary interventions?
- Are independent paired training runs available to determine whether small task differences and scale-dependent reversals persist beyond checkpoint-conditional prompt-sampling uncertainty?
- See weaknesses.
5 comments
Rebuttal (2/3)
Q1. We trained the matched-data-order, same-base QK-RoPE control, and we completed the factorial score/value rescaling. The paper already had the rescaling table for the value-rotated model, and we added a QK-RoPE column to it.
Same-base QK control. In addition to the 130M QKVO-RoPE model at base from the submission, we trained a 130M QK-RoPE model at base and retrained the 130M QKVO-RoPE model at base with the same setup. The rotary base is the RoPE hyperparameter that sets how fast each pair of dimensions rotates with position. All three models use the same training data, number of tokens, hyperparameters, data order and seed.
- Effect of the base, with the data order matched. Raising the base from to improves the value-rotated model throughout. PG-19 perplexity at 4K drops from 28.23 to 26.69, and needle accuracy at 4K rises from 33.6% to 82.8%.
- Effect of value rotation at the same base. At base , QKVO-RoPE finds more needles than QK-RoPE, 95.8% vs 90.6% at 2K and 82.8 vs 72.8% at 4K, at nearly the same perplexity at 4K (26.69 vs 26.80).
- Head level attenuation. The attenuation that value rotation causes in the QK-RoPE model at base (mean 0.708) is almost the same as the attenuation measured in the value-rotated model (0.710).
So the larger base helps both models, and part of its gain comes through the score. We revised Section 4.7 and the Discussion accordingly.
Factorial score/value rescaling. Here, rescaling means changing the rotary frequencies at test time in the attention score, the value path, or both. Rescaling both paths together corresponds to NTK-aware scaling. The paper already has a table that rescales the score and the value path separately for the value-rotated model, covering all four cases (none, score only, value only, and both). To complete the comparison with the baseline, we added a QK-RoPE column to the table.
With the score rescaled in both models, QK-RoPE has the same or lower perplexity than QKVO-RoPE at every model size (e.g., 18.3 vs. 23.0 at 760M and 8K). Rescaling the value path as well brings QKVO-RoPE to nearly the same perplexity as QK-RoPE at 760M (18.4 vs. 18.3).
The perplexity advantage of the value-rotated model beyond the training length therefore disappears once the attention score is rescaled. This is consistent with Section 4.7, where we show that part of the gain comes through the score. The retrieval advantage, on the other hand, holds when both models use NTK-aware scaling (see our response to W2).
Rebuttal (3/3)
Q2. Our significance tests treat the test prompts as random and use one trained model per setting, so they do not include variation between training runs. One estimate of run-to-run variation is already available from two 130M QKVO-RoPE runs at base . They reach PG-19 perplexity 28.83 and 28.23 at 4K, and retrieve 38.4% and 33.6% of needles at 4K.
Differences within the 130M pair at the training length are similar, so the revised paper now describes them as inconclusive. The differences we base our conclusions on are much larger. For example, needle retrieval beyond the training length improves by 11-49 points at 760M and 1.3B.
Within-model variance versus QK/QKVO variance. Two additional training runs show whether the differences between QK-RoPE and QKVO-RoPE are larger than the differences between two runs of the same model. We trained 370M QK- and QKVO-RoPE models using two different seeds (42 and 123). A seed changes both the parameter initialization and data order. All other settings are the same for both seeds, and all models were trained on the same hardware.
370M QK-RoPE and QKVO-RoPE models with 64-dimensional heads, each trained with two seeds on 10.7B tokens of FineWeb-Edu at a 4K context with identical hyperparameters. The seeds (42 and 123) change only the initialization and the data order. PG-19 perplexity, lower is better. NTK means NTK-aware scaling at test time. The better model of each seed is in bold.
| seed | model | 1K | 2K | 4K | 8K | 16K | 8K NTK | 16K NTK |
|---|---|---|---|---|---|---|---|---|
| seed 42 | QK-RoPE | 23.14 | 21.80 | 21.05 | 32.25 | 196.7 | 21.19 | 24.12 |
| QKVO-RoPE | 22.58 | 21.39 | 20.81 | 24.88 | 61.56 | 20.86 | 23.83 | |
| seed 123 | QK-RoPE | 22.59 | 21.19 | 20.37 | 35.15 | 110.5 | 20.46 | 23.82 |
| QKVO-RoPE | 22.79 | 21.53 | 20.93 | 25.73 | 87.35 | 21.37 | 24.76 |
The second seed confirms our main perplexity result. At 8K without NTK-aware scaling, the value-rotated model has lower perplexity at both seeds (24.9 vs 32.3 and 25.7 vs 35.2), and this gap is more than twice the change caused by the seed itself, which is less than 3 points for either model. At 4K, the two encodings stay within one point of each other at both seeds, as the paper already describes for differences at the training length. The perplexity advantage beyond the training length without NTK-aware scaling therefore persists across independent training runs.
Revised text (Appendix G)
“They are therefore conditional on the trained checkpoints and do not include variation between training runs. … Differences within the 130M pair at the training length are of this size and should be read as inconclusive.”
Rebuttal (1/3)
We thank the reviewer for the comments, which helped us clarify the assumptions in Section 3.
W1. Whether a fixed projection can read the content depends on how much the lag (how far back the head attends) varies across inputs, not on how large it is. Our fixed-lag result is a special case where the lag does not vary. We have made three changes to Section 3.3.
The shared-content and zero-mean independent-content assumptions are stated in Section 3.3 and Appendix A.
Revised text (Section 3.3, query-dependent lag)
Revised text (Section 3.3, after the count JW)
W2. We ran two new controls, a non-repetitive background and paired NTK-scaled retrieval, on all four pairs (full numbers in our response to Reviewer WZ2A). In the paired control, both models get the same NTK-aware scaling. NTK-aware scaling raises the rotary base at test time, so that a RoPE model can handle inputs longer than it was trained on.
Value rotation therefore improves needle retrieval beyond the 4K training length when both models use NTK-aware scaling. On natural text at 8K, the value-rotated models find more needles than QK-RoPE at all four model sizes, significant at each size. On the repeated haystack at 16K, the value-rotated 760M and 1.3B models find 46.1% and 24.5% of needles, against 3.7% and 0% for QK-RoPE. These results address the concern that the gain reflects only robustness to interference on repeated backgrounds. The gain also holds on natural text and with the same NTK-aware scaling on both models, and it reaches needles up to 5000 tokens back.
W3. We agree that the PG-19 diagnostic alone cannot show that the attenuated content is harmful, and we now state this limitation in Section 4.3. The head-level measurement shows where the cancellation happens and how the trained models adapt, by moving output energy to the low-frequency pairs. Whether the cancellation helps or hurts is tested by the task results in the following sections of the paper, and retrieval itself by the needle tests in our response to Weakness 2. The estimate made on a QK-RoPE model also matches the trained value-rotated model. The attenuation that value rotation would cause in the QK-RoPE model at base 106 (mean ∣φ∣ 0.708) is almost the same as the attenuation measured in the value-rotated model (0.710).