acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Non-Isometric Positional Transformations through Rank and Structure

Abstract

Positional encoding is increasingly formulated through structured transformations of token interactions, where preserving near-isometric geometry is often preferred to avoid contraction or degeneracy. Inspired by state-transition design in linear attention, we revisit whether non-isometric deformation should instead be treated as a controllable source of positional capacity. We introduce positional update rank, , and propose PRISM, which separates expansion order from expansion structure through independent, mixed, and product expansions. Our analysis shows that equal expansion order can yield substantially different operators depending on structure. Experiments across model scales, long-context tasks, and hybrid architectures reveal a consistent moderate-rank regime: expanding beyond rank one is often beneficial, whereas further expansion is not uniformly helpful. These results suggest that non-isometric positional deformation is not merely distortion, but a structured capacity whose utility depends on both its extent and organization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Positional Encoding, Non-Isometric Transformations, Attention Mechanisms.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JEf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies non-isometric positional transformations through their deformation rank and structure. Building on PaTH, PRISM introduces independent, mixed, and product expansions. Experiments on synthetic recall, language modeling, and hybrid architectures suggest that moderate expansion can improve information extraction and some long-context tasks, while higher expansion orders do not consistently help.

Critical strengths
  1. A meaningful design question. Distinguishing the number of deformation modes from their organization offers a useful perspective on positional mechanisms.

  2. Substantial gains on selected tasks. At 760M parameters and 50B training tokens, PRI-I improves the information-extraction average from 51.78 to 61.71 and RULER at 16K from 12.28 to 24.95 over PaTH.

  3. Useful internal controls. The paper compares expansion structures, orders, and head expansion under matched backbones, data, and training-token budgets.

Critical weaknesses
  1. Baseline coverage is insufficient. Comparisons focus on standard RoPE and PaTH, without representative long-context extensions or alternative non-isometric positional mechanisms. Failure of standard RoPE beyond its training length does not establish competitiveness against appropriately configured alternatives.

  2. The source of improvement is not fully isolated. Increasing PRI-I's order also changes subspace partitioning and per-direction deformation strength. Effective-rank measurements show that additional modes are active, but do not establish whether rank, strength, or structural separation drives the gains.

  3. The empirical conclusion is task-dependent. Gains do not extend consistently to longer-context BABILONG, and head expansion performs better on some metrics. With orders mainly limited to 1, 2, and 4 and a single training seed for LM experiments, the evidence supports a promising configuration rather than a general optimal-rank principle.

I lean toward 2/4 because experimental support for competitiveness and the proposed explanation remains incomplete. If stronger baselines and targeted controls confirm the main gains and their explanation, I would be inclined to raise the recommendation to 3/4 (Weak acceptance).

Questions for authors
  1. What RoPE configuration is used beyond the training length? Does it include positional scaling or another long-context adaptation?

  2. What existing evidence separates the effects of rank expansion from deformation strength and subspace partitioning?

  3. How should the claimed benefits be qualified given the BABILONG results and the cases where head expansion performs better?

4 comments

Thank you very much for your suggestions, especially your second and third questions. They helped us rewrite the analysis of our experimental results. We have added measurements of the actual update strength and more detailed analyses of individual tasks in RULER and LongBench-E. Below, we answer your three questions directly. The page and line numbers refer to the revised manuscript.

Q1. What RoPE configuration is used beyond the training length? Does it include positional scaling or another long-context adaptation?

The original baseline used standard RoPE. At 8K and 16K, it did not use positional scaling or any other long-context adaptation.

Following your suggestion, we have completed an additional RoPE+YaRN experiment with 760M parameters, 50B training tokens, and the same 4K training length as the main experiments. We use static YaRN with a scaling factor of 4 and seed 42. The results below follow the metrics in Tables 2 and 3. All scores except perplexity are percentages.

ModelWiki. PPLLMB. PPLLMB. accPIQAHella.Wino.ARC-eARC-cReasoning Avg.SWDESQuADFDAIE Avg.
RoPE+YaRN19.1918.2241.0169.2148.6254.8565.9532.1751.9761.8413.3453.3642.84

HellaSwag and ARC-c use normalized accuracy; the other reasoning tasks use standard accuracy.

ModelRULER 4KRULER 8KRULER 16KBABILong 0KBABILong 4KBABILong 8KBABILong 16KLongBench-E 0–4KLongBench-E 4–8KLongBench-E 8–16K
RoPE+YaRN41.691.020.0045.409.130.020.0017.236.192.72

BABILong averages QA1–QA5, and LongBench-E uses the equal-weight average across tasks.

YaRN gives better results on some measures, such as WikiText, SWDE, and RULER 4K, than PRI-I (r=2r=2). Its information-extraction average is 42.84, while PaTH and PRI-I (r=2r=2) achieve 51.78 and 61.71, respectively (Table 2, p. 7). Its long-context scores are also lower in this setting. Therefore, after adding a scaled RoPE baseline, we still observe the main information-extraction gains. We will add these YaRN results and its configuration to the revised tables. We will make clear that this is a comparison under the same 4K training length, rather than a comparison of all possible long-context training methods.

We chose PaTH as our main reference because its local rank-one update gives us a clear starting point for studying rank expansion and different structures (Section 3.2, p. 5, lines 237–245). RoPE and YaRN provide additional comparisons with rotation-based positional encoding.

Q2. What existing evidence separates the effects of rank expansion from deformation strength and subspace partitioning?

We address this through two complementary comparisons: measurements of the actual update strength and a control that matches the number and size of positional subspaces.

For strength, we have extended the effective-rank analysis in Appendix A.6 by measuring the actual update At=I−PtA_t=I-P_t, including the learned gates and the composed transition for PRI-P (Figure 8, p. 22; definitions and setup on p. 20, lines 1054–1073). On the same input for the 340M models, the leading-mode strength of PRI-I (r=2r=2) is close to PaTH's: 0.625 versus 0.609. A clear second mode appears with strength 0.338. The total update norm also increases, from 0.609 to 0.722. Thus, the change includes an additional active mode, not just a larger leading mode.

The other configurations help clarify this distinction. PRI-I with r=4r=4 uses more modes but has a lower mean update norm than r=2r=2 (0.690 versus 0.722; p. 20, lines 1074–1079). PRI-P has the largest update norm, but its information-extraction average is lower than PRI-I's in the corresponding 340M experiments (p. 21, lines 1130–1133). These fixed-input observations show that mode count and update magnitude do not simply increase together, and that the largest update norm does not correspond to the best retrieval result.

For subspace count, Section 4.4 compares PRI-I (r=2r=2) with PaTH-head. Both have 2H2H rank-one positional components, each acting on D/2D/2 dimensions, with the same total attention width. PRI-I combines the modes within each head, while PaTH-head places them in separate heads (p. 9, lines 450–458). Under the same 760M/25B setting, their information-extraction averages are 49.9 and 40.2, respectively, compared with 42.1 for PaTH. Their SQuAD scores are 37.6 and 13.9 (Table 5, p. 9; discussion at lines 468–478). Simply adding independent subspaces therefore does not reproduce PRI-I's information-extraction gains under this budget. Combining multiple modes before a shared softmax, instead of forming separate attention distributions, may help explain this difference.

Taken together, the leading-mode strength is similar, while simply adding heads does not give the same retrieval gains. Rank expansion in PRI-I introduces additional within-head modes, their associated subspaces, and their learned strengths together. These are linked parts of the expanded operator. Our conclusion concerns the joint benefit of adding and organizing these modes, rather than a pure rank effect with subspaces and all mode strengths held unchanged.

Q3. How should the claimed benefits be qualified given the BABILONG results and the cases where head expansion performs better?

We limit our claim to improvements in information retrieval and tracking under the tested configurations. We do not claim uniform gains on all long-context tasks or that rank expansion always performs better than head expansion. The revised discussion makes these task-dependent results clear (p. 7, lines 370–377).

We also keep the cases where PaTH or PaTH-head performs better. The BABILong results are mixed (Table 9, p. 29, line 1527), and the tasks also need fact integration and reasoning. Better retrieval alone does not guarantee a higher overall score. PaTH-head also scores higher on RULER 16K in the 25B-token experiment: 10.84 versus 6.71 for PRI-I (Table 11, p. 30, lines 1595–1596).

The subspace-expansion experiment uses 25B tokens, half the main training budget. It is intended to compare the ability patterns of the models at the same training stage (p. 9, lines 455–458; Appendix E.4, p. 29, lines 1558–1560). We use it as qualitative evidence under this budget, rather than to decide the final ranking after full convergence. This limit applies to both methods.

Following your suggestion, we have rewritten the task discussion and added the full task-level results in Appendix E (Tables 8–10, pp. 28–29). The NIAH-1, NIAH-2, and Multi-key-1 analyses show that better language-modeling perplexity does not always lead to better information tracking. PaTH can have lower perplexity under length extrapolation but weaker retrieval. We discuss this trade-off in Section 4.3 (p. 8, lines 394–407). The practical point is that PRISM can help when tracking is important, while PaTH can be a better choice when language-modeling quality under length extrapolation is the main concern. We describe r=2r=2 as a useful balance in the tested settings, not a universally optimal rank.

About the random seed, the synthetic recall experiments already average five seeds (p. 6, lines 307–309). We have also started an additional seed-43 comparison of PaTH and PRI-I (r=2r=2) for the 760M models, with a target of 50B training tokens. At the time of writing, training has reached about 20B tokens, so it may not finish before the rebuttal deadline. If our paper is accepted, we will include the comparison results from this additional random seed in the appendix.

Thank you for your response. I have decided to keep my original rating.

Reviewer_R1xpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a family of methods for positional encoding that leverage the compositional, non-isometric, and data-dependent transition structure of PaTH (Yang et al., 2025)*, replacing the rank-one transition update Pt=I−βwtwt⊤P_t = I - \beta w_t w_t^\top with three rank-r variants: PRI-I, PRI-M, PRI-P. The authors hypothesize that increasing the number of deformation modes and structuring how they interact provide useful capacity for the model. Their construction hinges on positional update rank, or the number of singular modes of PtP_t that depart from isometry. The paper includes comprehensive experimentation across synthetic retrieval tasks and long-context benchmarks, demonstrating that rank expansion at r=2 can improve performance.

*Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transformations. In Proceedings of NeurIPS, 2025b.

Critical strengths

Strong and comprehensive empirical results: all experiments are conducted across multiple model scales, architectures, and relevant benchmarks. The ablations against an additional attention head are particularly compelling.

  • Efficient implementation: I appreciate the appendix sections showing how the PRISM variants reduce to extensions of PaTH’s already efficient UT formulation (Yang et al., 2025). Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transformations. In Proceedings of NeurIPS, 2025b.
Critical weaknesses

The contribution is an incremental modification of PaTH, and the justification for increasing transition rank is insufficiently developed. -PaTH’s combined transition/encoding matrix: ∏i(I−βwiwi⊤)\prod_i (I - \beta w_i w_i^\top) can already deform multiple directions despite each token introducing only a rank-one update. It’s unclear what additional expressivity increasing the rank of each individual transition provides, theoretically or empirically. The experiments confirm this, as increasing rank beyond 2 seems to degrade performance.

-The Eckart-Young argument only shows that higher-rank operators can approximate higher-rank target deformations, but does not explain why this should benefit attention or improve downstream performance as a whole.

-The construction of PRI-I, PRI-M, and PRI-P seems to be motivated by analogous linear attention architectures, but again, lacks any characterization for why these structures should work. Any sort of expressivity result (as in PaTH) or intuition would strengthen the paper.

-While the experiments demonstrate that increasing positional update rank from r=1 to r=2 can improve performance, this does not explain how or why this should happen with respect to the underlying mechanism. For example, for any orthogonal Q. E(QP) = E(P), so r_\text{pos} (P) = r_\text{pos} (QP). If Q rotates the space by some angle, we may gain some expressivity which positional update rank is blind to.

-Many of the main benchmark empirical results are quite incremental, and the authors cite evaluation over a single random seed. The gains could very well be noise.

Presentation and experimental limitations

-I’m reasonably confident the majority of this paper is AI-generated. The writing can be significantly improved and is extremely difficult to parse.

-The taxonomy is unreasonably complex. For example, there seems to be redundancy in the definitions of expansion order, positional update rank, and non-isometric deformation modes. Understanding the basic terminology should not be an exercise for the reader.

-This paper contains insufficient exposition of previous work. For example, no explicit notation is defined, and I had to reread previous papers to understand what was being communicated in the main text.

-I find the connection between positional transformations, accumulated products, and deformation rank quite unsubstantiated. The burden should not be on the reader to understand the motivation independently

-Several crucial details are left ambiguous or deferred to the appendix. Including but not limited to:

-The diagonal matrix D in PRI-I

-The construction of the learned mixing matrix M because R_t is never defined

-All experimental evals are conducted across a single seed.

Questions for authors

I have no questions that could materially affect my assessment.

4 comments

Thank you for your comments, especially the questions about local rank and operator structure. They helped us clarify what we want to study beyond PaTH. We have revised the motivation and task analysis, added a notation table, and examined how the trained models actually use the additional modes.

Q1. Whether PaTH's rank-1 updates are already sufficient

We agree that multiple rank-1 transitions can form a rich global transformation. We state this explicitly in Section 2.1 (p. 3, lines 141–145).

What we want to study is the capacity available at each token step. A PaTH transition has at most one independent non-isometric deformation mode. Training can change its direction and strength, but the rank-one form keeps this local restriction. Allowing more modes gives the model additional ways to transform information within the same step.

The new analysis shows that these extra modes are used after training. For the 340M models on a shared 192-token input, PRI-I with r=2r=2 has a mean leading update singular value of 0.625, close to PaTH's 0.609, and a second value of 0.338 (Appendix A.6, p. 20, lines 1069–1076). The model keeps a leading mode of similar strength while using another substantial mode.

Q2. How the Eckart–Young argument connects to downstream performance

We use Eckart–Young to explain a capacity limitation. When a target metric deformation has several independent components, an operator with insufficient deformation rank cannot represent all of them. This argument appears in Section 3.1 (p. 4, lines 190–199).

The reason we expect additional modes to help is that different positional relations may compete for the same limited directions. More modes may give local interactions, long-range dependencies, and structural patterns more room to be represented separately. We explain this in Why rank expansion may help (p. 4, lines 201–210).

We use MQRAR-N to examine the tracking ability directly, then check its effects in language models. The theorem explains why more capacity may be useful. The experiments examine whether the models can use it.

Q3. Why we chose PRI-I, PRI-M, and PRI-P

Our motivation comes from the study of state transitions in linear attention. Those transitions control how earlier information is carried through memory. Our positional transitions control how earlier tokens contribute to later query–key interactions. Section 2.2 explains this connection and their different computational roles (pp. 3–4, lines 148–171).

We chose three structures to study different ways of using the additional modes. PRI-I keeps updates in separate coordinate blocks. PRI-M lets these directions interact through an input-dependent mixing matrix. PRI-P applies independently parameterized full-dimensional updates in sequence. Their constructions are given in Section 3.2 (pp. 5–6, lines 247–279).

Appendix A.2–A.5 analyzes their rank and spectral properties. The trained models also behave differently. At r=2r=2, the 340M models have average effective ranks of 1.92, 1.51, and 1.83 for PRI-I, PRI-M, and PRI-P on the shared held-out sequence (p. 19, lines 1014–1022).

The new update-strength analysis makes another distinction clear. More modes do not necessarily mean a stronger update. PRI-P has the largest measured update norm, but its information-extraction score is below PRI-I's in the corresponding 340M experiments (p. 20, lines 1074–1079; p. 21, lines 1130–1133). We need to consider how many modes are used, their strengths, and their structure.

Q4. The orthogonal QQ example

We agree that E(QP)=E(P)E(QP)=E(P) for an orthogonal QQ. We think this example also needs to be considered inside the actual model.

Replacing each local transition PtP_t with QPtQP_t changes the accumulated operator from

\quad\text{to}\quad (QP_i)(QP_{i-1})\cdots(QP_{j+1}).$$ These products generally differ, so the attention scores can change while the local positional update rank stays the same. If we insert $Q$ into a trained model, we change a structure that its queries, keys, and transitions have already adapted to. This may disturb the learned attention patterns. If we include $Q$ from the start, the model can adapt to that structure and learn different deformation patterns. This is the role of structure in our argument. We do not use positional update rank as a complete measure of positional expressivity. Its scope is stated on p. 3, lines 136–140, and p. 4, lines 183–188. We study rank and structure together because the rank count alone cannot explain the whole operator.

Q5. Small downstream gains and the decline beyond r=2r=2

Our main goal is to understand which abilities improve and what they cost. We do not expect every benchmark to improve in the same way.

Some changes on information-intensive tasks are substantial. For the 760M models, PRI-I with r=2r=2 increases the information-extraction average from 51.78 to 61.71 and SQuAD-completion from 21.88 to 45.24 (Table 2, p. 7). RULER 16K increases from 12.28 to 24.86 (Table 3, p. 8).

The r=4r=4 results show why we analyze the tasks separately. At 16K, NIAH-1 improves from PaTH's 3.80 to 99.20, while NIAH-2 falls from 56.60 to 26.60 (Table 8, p. 28). Length-extrapolation perplexity also becomes higher.

For the settings we tested, PRI-I with r=2r=2 gives a better balance for tracking-intensive tasks. PaTH can remain preferable when lower perplexity under length extrapolation is the main priority. We discuss this choice on p. 8, lines 394–407. Tables 8–10 provide the full task-level results, including the tasks that do not improve.

We also recognize that small score differences from a single training seed need to be treated cautiously.

Q6. Writing quality and AI-assisted content

We have made the main definitions easier to find and added Table 7 to collect the notation and terminology (p. 15). Section 3.1 now points readers to this table (p. 4, lines 180–181).

The paper includes an AI Use Statement that explains how AI tools were used.

Q7. Expansion order, positional update rank, and deformation modes

We use these terms for different things. The expansion order rr is the configured budget. The positional update rank

rpos(P)=rank⁡(P⊤P−I)r_{\mathrm{pos}}(P)=\operatorname{rank}(P^\top P-I)

is the actual number of independent non-isometric modes. Deformation modes are the geometric objects being counted.

The actual rank can be smaller than the budget, depending on the learned operator. These definitions appear in Section 3.1 (p. 4, lines 177–188) and Section 3.2 (p. 5, lines 243–245). We also summarize their relationship directly below Table 7 (p. 15, lines 800–801).

Q8. Previous work and notation

We have written the positional attention formulation and the RoPE and PaTH operators explicitly in Section 2.1 (p. 2, lines 99–107). Section 2.2 explains the relevant state-transition work in linear attention and its connection to our setting (pp. 3–4, lines 148–171).

Table 7 collects the symbols needed to follow the method, including local transitions, accumulated operators, deformation rank, and the matrices used by PRI-I and PRI-M. Readers can check these definitions within the paper.

Q9. Positional transformations, accumulated products, and deformation rank

We have made this connection explicit in the revised manuscript.

The attention score uses the positional operator through qi⊤Pijkjq_i^\top P_{ij}k_j. In PaTH and PRISM, PijP_{ij} is formed by multiplying local transitions (p. 2, lines 99–107). We use E(P)=P⊤P−IE(P)=P^\top P-I to describe the metric deformation of an operator and its rank to count the independent non-isometric modes.

For two consecutive operators,

\qquad r_{\mathrm{pos}}(AB) \leq r_{\mathrm{pos}}(A)+r_{\mathrm{pos}}(B).$$ These relations are given on p. 4, lines 212–215. They show how local deformations contribute to the accumulated result and why their interactions matter. This is why we study both the local rank budget and the way the modes are organized. More local modes provide more capacity at each step. Their structure affects how that capacity is combined across positions. We test the resulting behavior in Sections 4.1 and 4.3. **Q10. The diagonal matrix $D_t$ in PRI-I** Here $D_t$ is simply $$D_t=\beta_t I_r\in\mathbb R^{r\times r}.$$ The definition is explicit in Section 3.2 (p. 5, line 257) and Table 7 (p. 15, lines 784–785). It places the same scalar gate on the diagonal, giving $$P_t^{\mathrm{PRI-I}} =I-W_tD_tW_t^\top =I-\beta_tW_tW_t^\top.$$ **Q11. The construction of $M_t$ and $R_t$** We generate the mixing matrix from the current token representation: $$Z_t=\operatorname{Reshape}_{r\times r}(W_mh_t), \qquad R_t=\operatorname{L2Norm}(\lvert Z_t\rvert), \qquad M_t=\beta_tR_tR_t^\top.$$ We first project $h_t$ to $r^2$ values and reshape them into a matrix. We then take the element-wise absolute value and apply row-wise L2 normalization. Section 3.2 gives these steps on p. 5, lines 260–269. For unit-normalized rows, the Gram matrix has diagonal entries of one. Its off-diagonal entries describe similarities between the modes. If the rows are mutually orthogonal, $M_t=\beta_t I_r$, which recovers PRI-I. Appendix A.3 states this special case on p. 18, lines 924–925. Table 7 also summarizes the construction (p. 15, lines 789–793). **Q12. Random seeds** The main language-model experiments use one training seed. Training these models from scratch is costly, which limits the number of repeated runs. For MQRAR-N, where repeated runs are much less costly, we tested five seeds, 42–46, and report their average in Section 4.1 and Figure 3. This setting is stated explicitly on p. 6, lines 307–309. The single-seed limitation applies to the main language-model runs, rather than every experiment in the paper.

Thank you for your comments. I will maintain my rating.

Reviewer_joicOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies non-isometric positional transformations through positional update rank and proposes PRISM, with independent, mixed, and product expansion structures. The experiments investigate how expansion order and structure affect language modeling, retrieval, and long-context performance. The main observation is that moderate expansion can be beneficial, while further expansion does not consistently improve performance.

Critical strengths
  • Separating expansion order from structure provides a useful perspective on positional encoding.

  • Experiments cover multiple model scales, retrieval tasks, long-context benchmarks, and hybrid architectures.

  • PRI-I shows substantial information-extraction gains, and the head-expansion control strengthens the evaluation.

Critical weaknesses
  • Strengthen the evidence for robustness. The authors explicitly acknowledge that the language-model experiments use a single random seed. Repeating a representative comparison, such as PaTH versus PRI-I with expansion order 2 in the 340M setting, would help establish the reliability of the improvements. Figure 3 already averages five seeds; adding error bars would also be helpful.

  • Although PRI-I improves several retrieval and long-context scores, Figure 4 shows that PaTH maintains better perplexity at sufficiently long positions. The paper acknowledges extrapolation instability, but a short practical discussion of when to prefer PaTH or PRISM would make the findings more useful.

  • Table 3 mainly reports aggregate scores. A compact breakdown of RULER and LongBench-E by task would help readers understand whether the improvements are concentrated in retrieval or also extend to other capabilities.

Questions for authors

NA

2 comments

Thank you very much for your suggestions, especially your second and third points. They helped us rewrite the task analysis in the paper. In the revised version, we have added error bars for MQRAR-N, explained the trade-off between information tracking and language modeling, and added results and analysis for individual tasks in RULER and LongBench-E.

Q1. Multiple seeds and error bars for MQRAR-N

We have added error bars to Figure 3 on page 6 to show the variation across five random seeds on MQRAR-N.

Repeating the pretraining runs requires substantial compute. Since the gains in information extraction at 340M are relatively small, we chose to repeat the 760M comparison, where the gains are clearer (Table 2, page 7).

We have already started training PaTH and PRI-I(r=2) with seed 43 under the 760M/50B setting. As we write this response, both runs have reached about 20B tokens. Training is slow due to low MFU, so they may not finish before the rebuttal deadline. If the paper is accepted, we will include the completed seed 43 results in the final version.

Q2. When to prefer PaTH or PRISM

The analysis of NIAH-1, NIAH-2, and Multi-key-1 in RULER shows that PaTH can still struggle to find information in a long context, even though it has lower perplexity at long token positions. This suggests that information tracking can limit performance on these tasks. We have added this discussion on page 8, lines 394–407, and in Figure 4.

The contrast between r=2 and r=4 also shows that a higher expansion order does not always give better results. In our tested settings, we suggest considering PRI-I(r=2) for tasks that rely strongly on information tracking. PaTH is a better choice when the main goal is low perplexity at very long positions.

Q3. Results for individual tasks in RULER and LongBench-E

Following your suggestion, we rewrote the results discussion to focus more on information tracking and long-context tasks. This should help readers quickly see the focus of our work.

We also added results for individual tasks in RULER and LongBench-E: Table 8 on page 28 reports all 13 RULER subtasks, and Table 10 on page 29 reports all 11 evaluated LongBench-E tasks.

Looking at these tasks separately makes the roles of expansion order and structure clearer. For example, Table 4 on page 9 shows that, at the same expansion order r=2, PRI-I leads on NIAH-2 and TREC, PRI-M on VT and LCC, and PRI-P on NIAH-1 and GovReport. This shows that the structure matters even when the expansion order is the same.

The results also show gains in question answering, code tasks, and classification.

Reviewer_joicOpenReview ↗

Final Decision

Thank you for the author's response; my issue has been resolved, and I have decided to keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.