- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether position interventions in RoPE models actually remove position information or merely disrupt a previously trained readout. It compares frozen and re-fit position probes, analyzes redundancy and attention-head contributions, and separately evaluates behavioral effects on position-biased retrieval.
Critical strengths
- This paper is transparent about failed controls and evidential limitations.
Critical weaknesses
-
This manuscript reads more like an experiment report to me. Its highly templated and repetitive presentation makes it difficult to follow.
-
The large frozen / re-fit discrepancy in Gemma is presented as a main finding, but its simplest explanation is not examined. Since out-of-sample is highly sensitive to shifts in prediction offset and scale, the result may simply reflect trivial probe miscalibration. Without intercept-only and affine recalibration baselines, it remains unclear whether full probe refitting is necessary, limiting the interpretation and novelty of Contribution 1.
-
For Contribution 3, the only interpretable behavioral evidence comes from Llama on a small sample. The matched control also changes behavior, and overall QA performance does not clearly improve. The Qwen and Gemma audits provide no independent confirmation. Therefore, I do not find the behavioral evidence convincing.
Questions for authors
See Weaknesses.
Thank you for the comments on presentation, recalibration, and behavioral evidence.
[W1. Presentation.]
We have revised the Introduction and Contributions around the sink-value readout comparison and controlled behavioral contrast. The four-level structure remains, with primary results in Secs. 4 and 7, detailed procedures and secondary audits in the appendices, and interpretation and limitations grouped in Sec. 8.
[W2. Intercept and affine recalibration.]
We fitted the recalibration and full-refit baselines on 128 documents and evaluated them on 128 separate documents. For Gemma sink-value blocking, reference, intercept-corrected, affine-corrected, and full-refit R2 are −16.954, 0.596, 0.655, and 0.820. Thus, an offset shift explains much of the extreme frozen failure.
With clean probes fitted at the evaluation context length, full refitting still exceeds affine recalibration by 0.161 [0.152,0.170] in Gemma and 0.145 [0.137,0.153] in Llama base models. These nominal 95% intervals use 20,000 paired document-bootstrap resamples conditional on fitted probes. The advantage persists across three Gemma fitting cohorts sharing the same 128 evaluation documents. Contribution 1 now separates the substantial calibration effect from this remaining refit advantage (Sec. 4, Table 2; Appendix E).
[W3. Behavioral evidence, active control, and generalization.]
We evaluated Llama-Instruct and Qwen-Instruct on the same 256 held-out QA questions, with 128 separate fitting questions. Treatment fixes all 32 basis coefficients to their prompt means at prompt positions only; its control preserves a separately fitted scalar position readout while matching support and realized displacement.
Treatment-minus-control dip changes are −0.0242 [−0.0312,−0.0177] in Llama and −0.0190 [−0.0284,−0.0098] in Qwen (97.5% modelwise paired question-bootstrap intervals, Bonferroni-adjusted for the two primary contrasts). Both runs pass technical and specified control checks (Sec. 7, Table 4).
Llama's control remains active despite meeting the equivalence band. Qwen's treatment-identity 95% interval includes zero ([−0.0330,+0.0253]). Neither treatment meets the minimum-effect rule, which requires a treatment–identity 95% interval entirely below −0.065. The accuracy comparisons establish neither a corresponding dip effect nor an overall gain (Appendix G.1, Table 24).
The prompt-only Qwen comparison extends the treatment-decoder-null contrast to a second model. It uses different data and intervention support from the earlier sequence-wide audit, whose halted verdict is unchanged (Appendix G.8). Independent behavioral confirmation in Gemma remains absent.