- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies non-isometric positional transformations through their deformation rank and structure. Building on PaTH, PRISM introduces independent, mixed, and product expansions. Experiments on synthetic recall, language modeling, and hybrid architectures suggest that moderate expansion can improve information extraction and some long-context tasks, while higher expansion orders do not consistently help.
Critical strengths
-
A meaningful design question. Distinguishing the number of deformation modes from their organization offers a useful perspective on positional mechanisms.
-
Substantial gains on selected tasks. At 760M parameters and 50B training tokens, PRI-I improves the information-extraction average from 51.78 to 61.71 and RULER at 16K from 12.28 to 24.95 over PaTH.
-
Useful internal controls. The paper compares expansion structures, orders, and head expansion under matched backbones, data, and training-token budgets.
Critical weaknesses
-
Baseline coverage is insufficient. Comparisons focus on standard RoPE and PaTH, without representative long-context extensions or alternative non-isometric positional mechanisms. Failure of standard RoPE beyond its training length does not establish competitiveness against appropriately configured alternatives.
-
The source of improvement is not fully isolated. Increasing PRI-I's order also changes subspace partitioning and per-direction deformation strength. Effective-rank measurements show that additional modes are active, but do not establish whether rank, strength, or structural separation drives the gains.
-
The empirical conclusion is task-dependent. Gains do not extend consistently to longer-context BABILONG, and head expansion performs better on some metrics. With orders mainly limited to 1, 2, and 4 and a single training seed for LM experiments, the evidence supports a promising configuration rather than a general optimal-rank principle.
I lean toward 2/4 because experimental support for competitiveness and the proposed explanation remains incomplete. If stronger baselines and targeted controls confirm the main gains and their explanation, I would be inclined to raise the recommendation to 3/4 (Weak acceptance).
Questions for authors
-
What RoPE configuration is used beyond the training length? Does it include positional scaling or another long-context adaptation?
-
What existing evidence separates the effects of rank expansion from deformation strength and subspace partitioning?
-
How should the claimed benefits be qualified given the BABILONG results and the cases where head expansion performs better?
4 comments
Q2. What existing evidence separates the effects of rank expansion from deformation strength and subspace partitioning?
We address this through two complementary comparisons: measurements of the actual update strength and a control that matches the number and size of positional subspaces.
For strength, we have extended the effective-rank analysis in Appendix A.6 by measuring the actual update , including the learned gates and the composed transition for PRI-P (Figure 8, p. 22; definitions and setup on p. 20, lines 1054–1073). On the same input for the 340M models, the leading-mode strength of PRI-I () is close to PaTH's: 0.625 versus 0.609. A clear second mode appears with strength 0.338. The total update norm also increases, from 0.609 to 0.722. Thus, the change includes an additional active mode, not just a larger leading mode.
The other configurations help clarify this distinction. PRI-I with uses more modes but has a lower mean update norm than (0.690 versus 0.722; p. 20, lines 1074–1079). PRI-P has the largest update norm, but its information-extraction average is lower than PRI-I's in the corresponding 340M experiments (p. 21, lines 1130–1133). These fixed-input observations show that mode count and update magnitude do not simply increase together, and that the largest update norm does not correspond to the best retrieval result.
For subspace count, Section 4.4 compares PRI-I () with PaTH-head. Both have rank-one positional components, each acting on dimensions, with the same total attention width. PRI-I combines the modes within each head, while PaTH-head places them in separate heads (p. 9, lines 450–458). Under the same 760M/25B setting, their information-extraction averages are 49.9 and 40.2, respectively, compared with 42.1 for PaTH. Their SQuAD scores are 37.6 and 13.9 (Table 5, p. 9; discussion at lines 468–478). Simply adding independent subspaces therefore does not reproduce PRI-I's information-extraction gains under this budget. Combining multiple modes before a shared softmax, instead of forming separate attention distributions, may help explain this difference.
Taken together, the leading-mode strength is similar, while simply adding heads does not give the same retrieval gains. Rank expansion in PRI-I introduces additional within-head modes, their associated subspaces, and their learned strengths together. These are linked parts of the expanded operator. Our conclusion concerns the joint benefit of adding and organizing these modes, rather than a pure rank effect with subspaces and all mode strengths held unchanged.
Q3. How should the claimed benefits be qualified given the BABILONG results and the cases where head expansion performs better?
We limit our claim to improvements in information retrieval and tracking under the tested configurations. We do not claim uniform gains on all long-context tasks or that rank expansion always performs better than head expansion. The revised discussion makes these task-dependent results clear (p. 7, lines 370–377).
We also keep the cases where PaTH or PaTH-head performs better. The BABILong results are mixed (Table 9, p. 29, line 1527), and the tasks also need fact integration and reasoning. Better retrieval alone does not guarantee a higher overall score. PaTH-head also scores higher on RULER 16K in the 25B-token experiment: 10.84 versus 6.71 for PRI-I (Table 11, p. 30, lines 1595–1596).
The subspace-expansion experiment uses 25B tokens, half the main training budget. It is intended to compare the ability patterns of the models at the same training stage (p. 9, lines 455–458; Appendix E.4, p. 29, lines 1558–1560). We use it as qualitative evidence under this budget, rather than to decide the final ranking after full convergence. This limit applies to both methods.
Following your suggestion, we have rewritten the task discussion and added the full task-level results in Appendix E (Tables 8–10, pp. 28–29). The NIAH-1, NIAH-2, and Multi-key-1 analyses show that better language-modeling perplexity does not always lead to better information tracking. PaTH can have lower perplexity under length extrapolation but weaker retrieval. We discuss this trade-off in Section 4.3 (p. 8, lines 394–407). The practical point is that PRISM can help when tracking is important, while PaTH can be a better choice when language-modeling quality under length extrapolation is the main concern. We describe as a useful balance in the tested settings, not a universally optimal rank.
About the random seed, the synthetic recall experiments already average five seeds (p. 6, lines 307–309). We have also started an additional seed-43 comparison of PaTH and PRI-I () for the 760M models, with a target of 50B training tokens. At the time of writing, training has reached about 20B tokens, so it may not finish before the rebuttal deadline. If our paper is accepted, we will include the comparison results from this additional random seed in the appendix.
Thank you for your response. I have decided to keep my original rating.
Thank you very much for your suggestions, especially your second and third questions. They helped us rewrite the analysis of our experimental results. We have added measurements of the actual update strength and more detailed analyses of individual tasks in RULER and LongBench-E. Below, we answer your three questions directly. The page and line numbers refer to the revised manuscript.
Q1. What RoPE configuration is used beyond the training length? Does it include positional scaling or another long-context adaptation?
The original baseline used standard RoPE. At 8K and 16K, it did not use positional scaling or any other long-context adaptation.
Following your suggestion, we have completed an additional RoPE+YaRN experiment with 760M parameters, 50B training tokens, and the same 4K training length as the main experiments. We use static YaRN with a scaling factor of 4 and seed 42. The results below follow the metrics in Tables 2 and 3. All scores except perplexity are percentages.
HellaSwag and ARC-c use normalized accuracy; the other reasoning tasks use standard accuracy.
BABILong averages QA1–QA5, and LongBench-E uses the equal-weight average across tasks.
YaRN gives better results on some measures, such as WikiText, SWDE, and RULER 4K, than PRI-I (r=2). Its information-extraction average is 42.84, while PaTH and PRI-I (r=2) achieve 51.78 and 61.71, respectively (Table 2, p. 7). Its long-context scores are also lower in this setting. Therefore, after adding a scaled RoPE baseline, we still observe the main information-extraction gains. We will add these YaRN results and its configuration to the revised tables. We will make clear that this is a comparison under the same 4K training length, rather than a comparison of all possible long-context training methods.
We chose PaTH as our main reference because its local rank-one update gives us a clear starting point for studying rank expansion and different structures (Section 3.2, p. 5, lines 237–245). RoPE and YaRN provide additional comparisons with rotation-based positional encoding.