acceptodds
Under review as a conference paper at ICLR 2027

Rotate the Loop: Blockwise Conformality for Looped Transformers

Abstract

Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose **blockwise conformal** injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
looped models, test-time generalization, transformers, stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8W7iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Blockwise Conformal Injections to improve information preservation and test-time scaling in looped transformers.

Looped transformers repeatedly reuse the same block, but existing methods can cause hidden states to collapse toward dominant directions, making previously stored information harder to recover.

The authors divide hidden states into groups and apply 2D rotations with group-specific decay rates. This helps preserve information across recurrent steps and can be implemented efficiently using RoPE-style operations.

The method improves A5 state-tracking accuracy from 69.2% to 89.4% and achieves approximately 17% lower perplexity and 20% higher LAMBADA accuracy than the diagonal baseline. Importantly, performance continues improving with additional recurrent steps beyond those seen during training.

Critical strengths
  1. Clear motivation: The paper identifies an important limitation of looped transformers and clearly explains why hidden-state information can degrade across recurrent steps.

  2. Strong theoretical foundation: The geometric analysis provides a clear justification for using rotations and grouped decay rates to preserve information during recurrent computation.

  3. Efficient implementation: The method uses simple 2D rotations compatible with existing RoPE kernels, requiring only a modification to the injection module.

  4. Convincing experiments: The ablations isolate the contributions of grouping and rotation, while the results demonstrate improvements in state tracking, language modeling, and test-time depth scaling.

Critical weaknesses
  1. Limited model scale: Experiments only cover language models from 60M to 370M parameters. Evaluating at the 1B–3B+ scale would help establish whether the improvements generalize to larger models.

  2. Sensitivity to group count: Performance varies significantly with the number of groups G, with some configurations underperforming the baseline. The paper provides limited explanation for why G=2 performs best or how the optimal value should scale with model width.

  3. Missing reasoning evaluations: Although looped transformers are motivated by multi-step latent reasoning, the experiments do not evaluate downstream reasoning tasks. Such evaluations would strengthen the practical relevance of the results.

  4. Unclear inference-compute tradeoff: Performance continues improving with additional recurrent steps, but the marginal gains diminish. A quality-versus-compute analysis would help quantify the practical benefits of deeper inference.

Questions for authors
  1. How should the optimal group count scale with model width? The experiments identify G=2 as optimal for the synthetic model at d=512 and the preliminary language-model sweep at d=640. However, the larger models at d=768 and d=1024 retain G=2 without a corresponding sweep. Is there a theoretical or empirical reason to expect the optimal number of groups to remain approximately constant as width increases, rather than scaling proportionally with dimensionality?

  2. How should the Toeplitz-defect results be interpreted for blockwise conformal operators? Table 5 reports defects near machine precision for the G=2 models. However, blockwise conformality guarantees angular preservation within each group, whereas differing group decay rates do not generally preserve global conformality. Do the learned decay rates converge to nearly identical values, or is the reported defect evaluated separately within groups? Reporting the learned group-specific decay rates would help clarify this relationship.

  3. Does the theoretical guarantee translate to the full nonlinear transformer? The theoretical analysis and Table 5 examine the transition operator in isolation, but the actual recurrent dynamics also involve nonlinear transformer blocks. Could the authors measure information retention or angular collapse in the actual hidden-state trajectories to demonstrate that the proposed mechanism explains the observed performance gains?

3 comments

Limited model scale: Experiments only cover language models from 60M to 370M parameters. Evaluating at the 1B–3B+ scale would help establish whether the improvements generalize to larger models.

We thank the reviewer for this remark. Respectfully, we disagree. The sizes we report are the ones this comparison is made at and practical for current scopes of related work. Solve the Loop, Parcae, Towards Looped models done right evaluates its language models at scales comparable or equal to 140M, 370M, and 770M [1] [2] [3]. Our models’ scale fall inside the appropriate ranges. We see this as an encouraging sign for larger scales, and we list a larger-scale study as future work.

[1] Fein-Ashley J, Rashidinejad P. Solve the Loop: Attractor Models for Language and Reasoning. arXiv:2605.12466, 2026.

[2] Prairie H, Novack Z, Berg-Kirkpatrick T, Fu DY. Parcae: Scaling Laws for Stable Looped Language Models. arXiv:2604.12946, 2026.

[3] Huang B, Shi C, Chen J, Wen S, Liu Z, Xing E, Ma X. Towards Looped Models Done Right, Part II: Rethinking at Fixed Points. arXiv:2610.06833, 2026.

Sensitivity to group count: Performance varies significantly with the number of groups G, with some configurations underperforming the baseline. The paper provides limited explanation for why G=2 performs best or how the optimal value should scale with model width.

We thank the reviewer for this question. We would like to explain how GG acts.

The blockwise injection does two things at once. Inside a group, it keeps the geometry exact, so writes stay comparable over many steps. Across groups, it allows different lifetimes. The number of groups GG decides how the width dd is shared between these two purposes, since every group holds d/Gd/G coordinates. More groups give more lifetimes, but each group becomes smaller, and a small group leaves fewer dimensions in which to store and read a signal. With G=2G=2, each group keeps half of the width, and there are still two lifetimes. In our sweeps this was the best trade-off on state tracking and on language modelling. Large GG cuts the width into many small groups, which is why some of those settings fall below the baseline.

Missing reasoning evaluations: Although looped transformers are motivated by multi-step latent reasoning, the experiments do not evaluate downstream reasoning tasks. Such evaluations would strengthen the practical relevance of the results.

We thank the reviewer for this suggestion. The revision adds deterministic graph connectivity, following Merrill et al. [1] (Appendix I). The task is designed to separate what recurrent networks can do from what transformers can do, because solving it requires carrying a partial result through many sequential steps, which is the kind of multi-step computation a looped model is meant for. On graphs longer than those seen in training, the looped transformer lands between the two families, and the conformal injection moves it toward the recurrent network.

https://freeimage.host/i/o30vseV

Bibliography

[1] Merrill W, Jiang H, Li Y, Lin A, Sabharwal A. Why Are Linear RNNs More Parallelizable? PMLR 306, 2026. arXiv:2603.03612.

Unclear inference-compute tradeoff: Performance continues improving with additional recurrent steps, but the marginal gains diminish. A quality-versus-compute analysis would help quantify the practical benefits of deeper inference.

We thank the reviewer for this practical question. We count compute as layer applications per token: the prelude, the coda, and TT applications of the shared block.

SizeOurs, T=8T=8Diagonal, T=48T=48Layer applications
60M19.0822.4522 vs 102
140M16.5619.4720 vs 100
370M12.6614.9040 vs 200

The table reports perplexity, lower is better.

https://freeimage.host/i/o3VyMLN

At T=8T=8 our model is already better than the diagonal at T=48T=48, with about five times fewer layer applications. Past T=8T=8 our model gains little, and the diagonal does not improve. A trip count of about 88 is therefore the practical operating point. The revision adds this comparison in Appendix H.

How should the optimal group count scale with model width? The experiments identify G=2 as optimal for the synthetic model at d=512 and the preliminary language-model sweep at d=640. However, the larger models at d=768 and d=1024 retain G=2 without a corresponding sweep. Is there a theoretical or empirical reason to expect the optimal number of groups to remain approximately constant as width increases, rather than scaling proportionally with dimensionality?

We thank the reviewer for this question. The argument in our previous answer does not ask GG to grow with dd. What matters is that each group keeps enough dimensions to hold a signal, and that there are at least two lifetimes. At a larger width, the two groups of G=2G=2 only become larger, so the trade-off that made G=2G=2 best should not turn against it. We swept GG at d=512d=512 and d=640d=640 and kept G=2G=2 as can be seen from Table 8.

How should the Toeplitz-defect results be interpreted for blockwise conformal operators? Table 5 reports defects near machine precision for the G=2 models. However, blockwise conformality guarantees angular preservation within each group, whereas differing group decay rates do not generally preserve global conformality. Do the learned decay rates converge to nearly identical values, or is the reported defect evaluated separately within groups? Reporting the learned group-specific decay rates would help clarify this relationship.

We thank the reviewer for this precise question. Each group gg has a learned scale cg∈(0,1)c_g\in(0,1), and a write in that group is multiplied by cgc_g at every recurrent step. That scale is the group’s decay rate.

Inside a group the operator is exactly conformal, so the defect computed per group is at machine precision. On the whole state, two groups with different rates are not conformal, and the defect is small but nonzero, well below the free diagonal’s. The conformal rows of Table 5 correspond to the per-group value, and the revision labels them accordingly (Table 5 and Appendix B).

Does the theoretical guarantee translate to the full nonlinear transformer? The theoretical analysis and Table 5 examine the transition operator in isolation, but the actual recurrent dynamics also involve nonlinear transformer blocks. Could the authors measure information retention or angular collapse in the actual hidden-state trajectories to demonstrate that the proposed mechanism explains the observed performance gains?

We thank the reviewer for mentioning this point. We would like to provide the measurements and results for nonlinear block analysis below and in the revised version of the paper (Section 5.5 and Appendix G).

At the last token, a small change of the state passes through AA and then through the block. One loop step therefore has the Jacobian

J=(I+JB)A,J = (I + J_{B})A,

where JBJ_B is the Jacobian of the block (attention and MLP) with respect to its input. Attention is causal, so no other position depends on the last token’s state, and the product of these Jacobians over steps is exactly how a write made at one depth reaches a later depth, including the nonlinearity and the input-dependent addition. Without the block, JB=0J_B=0 and the product reduces to AkA^k, the object of Section 3.

Let ξ(M)=log⁡(σmax⁡(M)/σmin⁡(M))\xi(M)=\log\big(\sigma_{\max}(M)/\sigma_{\min}(M)\big) measure how far a linear map is from readable, so that ξ(M)=0\xi(M)=0 exactly when MM is conformal (Proposition 3.2). From depth ss to depth tt,

ξ(Jt−1⋯Js)≤∑r=st−1[ξ(I+JB,r)+ξ(Ar)],\xi\big(J_{t-1}\cdots J_{s}\big) \le \sum_{r=s}^{t-1}\Big[\xi\big(I+J_{B,r}\big) + \xi\big(A_{r}\big)\Big],

because the largest singular value of a product is at most the product of the largest ones, and the smallest is at least the product of the smallest ones.

The bound separates what the block does from what the transition operator does. The first term belongs to the block and is present for every choice of AA. The second term is the operator’s own share, and it is added again at every recurrent step. For a conformal operator this share is exactly zero, ξ(cQ)=0\xi(cQ)=0. For the blockwise operator it is ξ(A(G))=log⁡(cmax⁡/cmin⁡)\xi(A^{(G)})=\log(c_{\max}/c_{\min}), the spread between the group rates, and it is zero inside each group. For a free diagonal it is the spread between all per-coordinate rates, the largest of the three. So readability of AA does carry into the nonlinear recurrence in this sense.

To see the difference of how the state changes across architectures taking into account the nonlinear operations, we take the gradient of the log probability of the correct answer with respect to the state at each depth ss. It tells how much the final answer depends on each dimension of the state at that depth. We sort the dimensions by how fast AA shrinks them, split them into four equal quarters, and compare the slowest quarter with the fastest:

η(s)=log⁡usage of the slowest quarter at depth susage of the fastest quarter at depth s.\eta(s) = \log\frac{\text{usage of the slowest quarter at depth } s}{\text{usage of the fastest quarter at depth } s}.

A positive η\eta means the answer leans on the coordinates that AA keeps the longest, so AA’s preference has made it through the block. An η\eta near zero means that no part of the state is privileged. We chose quarters because they are coarse enough to be stable across words and seeds, and fine enough to show the ordering that AA creates, with one number per depth.

https://freeimage.host/i/o3GV0gV

The figure compares each trained loop (left) with what AA alone would do over the same number of steps (right).

The ordering follows how far AA is from conformal. Through the loop, η\eta is largest for the free diagonal, smaller for G=4G=4, and closer to zero for G=2G=2, for G=1G=1 and an orthogonal (Cayley) operator η\eta is predictably almost zero. Under AA alone the order is the same. So the preference a non-conformal AA creates among coordinates is what is acquired by the architectures and passed to the next recurrent steps. This way, the performance and readability of states are dependent on the relation between slow and fast part of the hidden representations.

Reviewer_t8yXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how the state transition operator between recurrent steps in Looped Transformers affects long-term information retention and the effectiveness of additional computational depth at test time. The authors propose Blockwise Conformal Injections, which divide the hidden state into multiple groups, preserve angular geometry within each group through two-dimensional rotations, and allow different groups to have independent decay rates. The method is evaluated on the A5 and S5 finite-group state-tracking tasks, as well as language models with 60M, 140M, and 370M parameters, showing promising results. The ablation studies also clearly distinguish the contributions of different components. The paper provides an intuitive geometric explanation for the motivation behind the proposed method.

Critical strengths
  1. The paper addresses an important problem in looped models: whether information computed at earlier recurrent steps can still be effectively utilized as recurrent depth increases. This is directly relevant to the test-time scaling capability of Looped Transformers. Starting from the spectral structure of the transition operator, the authors connect the preservation of state directions with different information decay timescales and propose a blockwise rotation structure accordingly. This approach has a clear theoretical intuition and provides a valuable perspective for understanding information degradation in recurrent architectures.

  2. The proposed method achieves significant improvements on both state-tracking and language modeling tasks by modifying only the transition operator between recurrent steps. In particular, the last-token accuracy on A5 at T=32 increases from 0.692 to 0.894. The paper also includes extensive generalization experiments and ablation studies that further support the effectiveness of the proposed approach.

Critical weaknesses

The proposed method achieves its best experimental performance at G=2, but the conformality guarantee in Proposition 3.6 holds only within individual groups. When the hidden state spans multiple groups, different decay rates continuously change their relative contributions, potentially affecting the overall direction and geometry of the state. If the learned decay rates of the two groups are very similar, the role of multiple timescales needs to be reconsidered. If they differ substantially, the authors should explain why the overall geometric metric remains close to zero. The current analysis does not sufficiently establish a clear causal connection between the observed performance improvements and blockwise conformality.

Questions for authors

Could the authors further clarify to what extent the geometric properties of the transition operator A are preserved after passing through the complete nonlinear Transformer block? For example, analyzing the actual hidden-state dynamics of the trained model, the effective recurrent Jacobian, or the retention of information written at earlier recurrent steps would help better explain the source of the observed performance improvements.

2 comments

The proposed method achieves its best experimental performance at G=2, but the conformality guarantee in Proposition 3.6 holds only within individual groups. When the hidden state spans multiple groups, different decay rates continuously change their relative contributions, potentially affecting the overall direction and geometry of the state. If the learned decay rates of the two groups are very similar, the role of multiple timescales needs to be reconsidered. If they differ substantially, the authors should explain why the overall geometric metric remains close to zero. The current analysis does not sufficiently establish a clear causal connection between the observed performance improvements and blockwise conformality.

We thank the reviewer for this precise question. We would like to address it with measured values

Each group gg has a learned scale cg∈(0,1)c_g \in (0,1). A write in group gg is multiplied by cgc_g at every recurrent step, so cgc_g is the group’s decay rate. On A5 we observe the non-degenerate cases of model (that has learned the task properly with the working training protocol described in the paper) they clearly differ (0.530.53 and 0.480.48). This way, the slower group carries most of what the answer depends on, as the relative factor (cg/ch)t(c_g/c_h)^t in Proposition 3.6 predicts. Inside each group, no direction is preferred, which is the per-group guarantee. On the whole state, two groups with different rates are not exactly conformal, and the deviation grows with the ratio between the rates. In trained models the two group rates are much closer to each other than the per-coordinate rates of the free diagonal, so the whole-state Toeplitz defect is small but nonzero, and well below the diagonal’s. The conformal rows of Table 5 correspond to the per-group value, and the revision labels them accordingly (Table 5 and Appendix B).

Could the authors further clarify to what extent the geometric properties of the transition operator A are preserved after passing through the complete nonlinear Transformer block? For example, analyzing the actual hidden-state dynamics of the trained model, the effective recurrent Jacobian, or the retention of information written at earlier recurrent steps would help better explain the source of the observed performance improvements.

We thank the reviewer for mentioning this point. We would like to provide the measurements and results for nonlinear block analysis below and in the revised version of the paper (Section 5.5 and Appendix G).

To see the difference of how the state changes across architectures taking into account the nonlinear operations, we take the gradient of the log probability of the correct answer with respect to the state at each depth ss. It tells how much the final answer depends on each dimension of the state at that depth. We sort the dimensions by how fast AA shrinks them, split them into four equal quarters, and compare the slowest quarter with the fastest:

η(s)=log⁡usage of the slowest quarter at depth susage of the fastest quarter at depth s.\eta(s) = \log\frac{\text{usage of the slowest quarter at depth } s}{\text{usage of the fastest quarter at depth } s}.

A positive η\eta means the answer leans on the coordinates that AA keeps the longest, so AA’s preference has made it through the block. An η\eta near zero means that no part of the state is privileged. We chose quarters because they are coarse enough to be stable across words and seeds, and fine enough to show the ordering that AA creates, with one number per depth.

https://freeimage.host/i/o3GV0gV

The figure compares each trained loop (left) with what AA alone would do over the same number of steps (right).

The ordering follows how far AA is from conformal. Through the loop, η\eta is largest for the free diagonal, smaller for G=4G=4, and closer to zero for G=2G=2, for G=1G=1 and an orthogonal (Cayley) operator η\eta is predictably almost zero. Under AA alone the order is the same. So the preference a non-conformal AA creates among coordinates is what is acquired by the architectures and passed to the next recurrent steps. This way, the performance and readability of states are dependent on the relation between slow and fast part of the hidden representations.

Finally, we noticed that the rating changed from 3 to 2 before our response was posted, without an accompanying comment. We would be grateful if the reviewer could tell us which concern led to this change, so that we can address it directly during the remaining discussion period.

Reviewer_YHUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors argue that making the transition operator of a looped transformer (blockwise) conformal makes reads to the residual stream interpretable across recurrent steps. This permits looped transformers to make better use of their depthwise representations in theory, an idea which is tested empirically in both language modeling and state-tracking setups. This line of research synthesizes work on looped transformers and conformal linear RNNs and provides both a theoretical account of how conformality can helpfully constrain residual representations and empirical evidence that doing so improves performance.

Critical strengths

The headline results are quite interesting. Authors provide an interesting synthesis of lines of existing research which have decent theoretical motivation for why they ought to improve the representational capacity of looped transformers, a topic very much in vogue. They also provide good empirical evidence that making these changes does have meaningful improvements in both general language modeling and state-tracking, which (in the setup studied here) is known to be adversarially-hard for fixed-depth transformer language models. The performance improvements over baseline seem large enough that the results are impactful beyond statistical significance.

Critical weaknesses

The main results are undercontextualized in a way which makes them a bit hard to interpret. For instance, the authors note that G=2 seems to be a peak in performance, but there is not much discussion of why G=2 is an optimal number of contiguous groups to partition the transition operator into, either theoretically or empirically. Likewise, there is not really a discussion of why "high" G should do worse than diagonal representations, something which happens in several cases in, eg, Table 1; this gives me some pause over whether the authors' provided account for why their intervention works is what is actually happening.

Questions for authors

I'm a little confused on how the readability/conformality results carry over form S3 to S4. First, is "readability" as defined in D3.1 a new term? I understand how it is the "same" as conformality, but it's unclear what the term means beyond that; is there a reason to not just call it conformal? Second, can the authors explain how the property of readability is maintained in the general recurrence relation? The property, as described, is one which is relevant under matrix powers (eg A^k is readable) and Figure 2 presents, I believe, an illustration of just a single write; but the recurrence relation in eq. (5) establishes the depth-wise recurrence as being mediated by a non-linearity B, among other factors. Just because A is readable, why does this automatically make B(A...) readable as well? The relevant composition that the authors must care about is not A^k but B^k(A...), and it strikes me as very plausible that the nonlinearity here matters, to say nothing of an input-dependent addition. If it is the case that the property still holds, can the authors elaborate on why? If not, can the authors explain why the property of readability is responsible for the empirical results they find?

4 comments

The main results are undercontextualized in a way which makes them a bit hard to interpret. For instance, the authors note that G=2 seems to be a peak in performance, but there is not much discussion of why G=2 is an optimal number of contiguous groups to partition the transition operator into, either theoretically or empirically. Likewise, there is not really a discussion of why “high” G should do worse than diagonal representations, something which happens in several cases in, eg, Table 1; this gives me some pause over whether the authors’ provided account for why their intervention works is what is actually happening.

We thank the reviewer for this productive remark. We agree that the choice of GG deserves more context than the submission gave it, and we would like to explain how we see it and how it connects to the main idea of the paper.

The paper starts from one observation. A looped transformer profits from extra recurrent steps only if what it writes on one step can still be read on later steps. The blockwise injection serves two purposes for that. Inside a group, it keeps the geometry exact, so writes made at different steps stay comparable with each other. Across groups, it allows different lifetimes, so some information can fade quickly while other information is kept for longer.

The number of groups GG decides how the width dd is shared between these two purposes. Every group holds d/Gd/G coordinates (dimensions). More groups give more lifetimes, but each group becomes smaller, and a small group leaves the model fewer dimensions in which to store and read a signal. With G=2G=2, each group keeps half of the width, and there are still two lifetimes, one for information that should fade and one for information that should stay. In our sweeps at our scale, this was the best trade-off on state tracking and on language modelling. When GG is large, the width is cut into many small groups, and the model can fall below the diagonal baseline, which is what Table 1 shows for several large values of GG. We do not claim that G=2G=2 is optimal in general; in our experiments at this scale it is the count that works best empirically (Section 5.1).

We would also like to show that the benefit is not limited to synthetic state tracking. In our revision we add deterministic graph connectivity, following Merrill et al. [1] (Appendix I). This task is designed to separate what recurrent networks can do from what transformers can do, because solving it requires carrying a partial result through many sequential steps. On graphs longer than those seen in training, the looped transformer lands between the two families of transformer and RNN respectively, and the conformal injection moves its capbilities toward the recurrent network with respect to the sense of length generalization for algorithmic reasoning task. This way, keeping writes readable is what lets the loop behave more like a recurrent computation

https://freeimage.host/i/o30vseV

Bibliography

[1] Merrill W, Jiang H, Li Y, Lin A, Sabharwal A. Why Are Linear RNNs More Parallelizable? PMLR 306, 2026. arXiv:2603.03612.

I’m a little confused on how the readability/conformality results carry over form S3 to S4. First, is “readability” as defined in D3.1 a new term? I understand how it is the “same” as conformality, but it’s unclear what the term means beyond that; is there a reason to not just call it conformal?

We thank the reviewer for this careful question. The two words describe the same property from two sides, and we kept both on purpose.

“Readable” is the requirement, stated in the language of the problem. A looped model applies one shared core again and again, and with the usual injection, something written on an early step gradually stops being recoverable on later steps. We asked a concrete question: what must the injection between steps satisfy so that every write can still be read later, by the same reader? Writing that requirement down gives Definition 3.1: the inner products between any two writes stay comparable, up to one common scale.

“Conformal” is the answer, stated in the language of linear algebra. Proposition 3.2 shows that the operators meeting this requirement are exactly the conformal ones, a positive scalar times an orthogonal matrix. Readability is therefore not a second property on top of conformality. It is the reason we look at conformality in the first place, and it explains which conformal property matters for a looped model. We use “readable” when we talk about the motivation and “conformal” when we talk about the operator class.

Second, can the authors explain how the property of readability is maintained in the general recurrence relation? The property, as described, is one which is relevant under matrix powers (eg A^k is readable) and Figure 2 presents, I believe, an illustration of just a single write; but the recurrence relation in eq. (5) establishes the depth-wise recurrence as being mediated by a non-linearity B, among other factors. Just because A is readable, why does this automatically make B(A…) readable as well? The relevant composition that the authors must care about is not A^k but B^k(A…), and it strikes me as very plausible that the nonlinearity here matters, to say nothing of an input-dependent addition. If it is the case that the property still holds, can the authors elaborate on why? If not, can the authors explain why the property of readability is responsible for the empirical results they find?

We thank the reviewer for mentioning this point. We would like to provide the measurements and results for nonlinear block analysis below and in the revised version of the paper (Section 5.5 and Appendix G).

To see the difference of how the state changes across architectures taking into account the nonlinear operations, we take the gradient of the log probability of the correct answer with respect to the state at each depth ss. It tells how much the final answer depends on each dimension of the state at that depth. We sort the dimensions by how fast AA shrinks them, split them into four equal quarters, and compare the slowest quarter with the fastest:

η(s)=log⁡usage of the slowest quarter at depth susage of the fastest quarter at depth s.\eta(s) = \log\frac{\text{usage of the slowest quarter at depth } s}{\text{usage of the fastest quarter at depth } s}.

A positive η\eta means the answer leans on the coordinates that AA keeps the longest, so AA’s preference has made it through the block. An η\eta near zero means that no part of the state is privileged. We chose quarters because they are coarse enough to be stable across words and seeds, and fine enough to show the ordering that AA creates, with one number per depth.

https://freeimage.host/i/o3GV0gV

The figure compares each trained loop (left) with what AA alone would do over the same number of steps (right).

The ordering follows how far AA is from conformal. Through the loop, η\eta is largest for the free diagonal, smaller for G=4G=4, and closer to zero for G=2G=2, for G=1G=1 and an orthogonal (Cayley) operator η\eta is predictably almost zero. Under AA alone the order is the same. So the preference a non-conformal AA creates among coordinates is what is acquired by the architectures and passed to the next recurrent steps. This way, the performance and readability of states are dependent on the relation between slow and fast part of the hidden representations

To the first point, I agree that you don’t over-claim G=2 to be some universal optimum, but I am more concerned with whether your theory predicts that some G should be better than others or explains why G=2 happens to be optimal in this case; since this is central to the benefit the proposed modification provides, it would be very useful to include this.

To the second point, these still seem like they could just be the same term, but i don’t feel strongly enough about it to press it.

To the third point, if i understand correctly it means that authors agree with me on the fact that conformality is not preserved through nonlinearities or additional layers by design, and may only be contingently approximateable with sufficiently high G; I think this poses a problem for the motivation of the work, since if true then none of the theory results which rely on this still hold without strong qualification. Likewise, the fact that optimal performance occurs at G values which are not the most conformal is hard to square with the central claims.

In light of these points, I maintain my score.

Comments on the paper

1 comment

We thank all reviewers for the time and effort they put into reading our paper and for the questions they raised. They helped us make the paper clearer and more complete.

We have answered each reviewer in the comments, and we have uploaded a revised PDF.

In the revision:

  • Section 5.5 analyses how nonlinearity affects the notion of readability of operators (YHUd, t8yX, 8W7i). The ordering of coordinates that the transition operator sets is conserved after the nonlinearity being applied. Figure 3 shows this through the loop and under the operator alone. Appendix G (new) gives the setup and Proposition G.1 on the operator’s share; its proof is in Appendix C.
  • Section 5.1 explains how GG trades the size of each group, d/Gd/G, against the number of lifetimes (YHUd, 8W7i).
  • Table 5 now labels the conformal rows as per-group values, and Section 5.4 and Appendix B explain the difference between the per-group and the whole-state Toeplitz defect for blockwise operators (t8yX, 8W7i).
  • Appendix I (new) adds sorted deterministic graph connectivity on algorithmic reasoning, following Merrill et al. (2026), with a pointer in Section 5.1 (YHUd, 8W7i).
  • Appendix H (new) compares perplexity and LAMBADA against layer applications per token (Table 9 and Figure 5), with a pointer in Section 5.3 (8W7i).

We are glad to clarify anything further, and we would be happy to continue the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.