- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Blockwise Conformal Injections to improve information preservation and test-time scaling in looped transformers.
Looped transformers repeatedly reuse the same block, but existing methods can cause hidden states to collapse toward dominant directions, making previously stored information harder to recover.
The authors divide hidden states into groups and apply 2D rotations with group-specific decay rates. This helps preserve information across recurrent steps and can be implemented efficiently using RoPE-style operations.
The method improves A5 state-tracking accuracy from 69.2% to 89.4% and achieves approximately 17% lower perplexity and 20% higher LAMBADA accuracy than the diagonal baseline. Importantly, performance continues improving with additional recurrent steps beyond those seen during training.
Critical strengths
-
Clear motivation: The paper identifies an important limitation of looped transformers and clearly explains why hidden-state information can degrade across recurrent steps.
-
Strong theoretical foundation: The geometric analysis provides a clear justification for using rotations and grouped decay rates to preserve information during recurrent computation.
-
Efficient implementation: The method uses simple 2D rotations compatible with existing RoPE kernels, requiring only a modification to the injection module.
-
Convincing experiments: The ablations isolate the contributions of grouping and rotation, while the results demonstrate improvements in state tracking, language modeling, and test-time depth scaling.
Critical weaknesses
-
Limited model scale: Experiments only cover language models from 60M to 370M parameters. Evaluating at the 1B–3B+ scale would help establish whether the improvements generalize to larger models.
-
Sensitivity to group count: Performance varies significantly with the number of groups G, with some configurations underperforming the baseline. The paper provides limited explanation for why G=2 performs best or how the optimal value should scale with model width.
-
Missing reasoning evaluations: Although looped transformers are motivated by multi-step latent reasoning, the experiments do not evaluate downstream reasoning tasks. Such evaluations would strengthen the practical relevance of the results.
-
Unclear inference-compute tradeoff: Performance continues improving with additional recurrent steps, but the marginal gains diminish. A quality-versus-compute analysis would help quantify the practical benefits of deeper inference.
Questions for authors
-
How should the optimal group count scale with model width? The experiments identify G=2 as optimal for the synthetic model at d=512 and the preliminary language-model sweep at d=640. However, the larger models at d=768 and d=1024 retain G=2 without a corresponding sweep. Is there a theoretical or empirical reason to expect the optimal number of groups to remain approximately constant as width increases, rather than scaling proportionally with dimensionality?
-
How should the Toeplitz-defect results be interpreted for blockwise conformal operators? Table 5 reports defects near machine precision for the G=2 models. However, blockwise conformality guarantees angular preservation within each group, whereas differing group decay rates do not generally preserve global conformality. Do the learned decay rates converge to nearly identical values, or is the reported defect evaluated separately within groups? Reporting the learned group-specific decay rates would help clarify this relationship.
-
Does the theoretical guarantee translate to the full nonlinear transformer? The theoretical analysis and Table 5 examine the transition operator in isolation, but the actual recurrent dynamics also involve nonlinear transformer blocks. Could the authors measure information retention or angular collapse in the actual hidden-state trajectories to demonstrate that the proposed mechanism explains the observed performance gains?
3 comments
Missing reasoning evaluations: Although looped transformers are motivated by multi-step latent reasoning, the experiments do not evaluate downstream reasoning tasks. Such evaluations would strengthen the practical relevance of the results.
We thank the reviewer for this suggestion. The revision adds deterministic graph connectivity, following Merrill et al. [1] (Appendix I). The task is designed to separate what recurrent networks can do from what transformers can do, because solving it requires carrying a partial result through many sequential steps, which is the kind of multi-step computation a looped model is meant for. On graphs longer than those seen in training, the looped transformer lands between the two families, and the conformal injection moves it toward the recurrent network.
https://freeimage.host/i/o30vseV
Bibliography
[1] Merrill W, Jiang H, Li Y, Lin A, Sabharwal A. Why Are Linear RNNs More Parallelizable? PMLR 306, 2026. arXiv:2603.03612.
Unclear inference-compute tradeoff: Performance continues improving with additional recurrent steps, but the marginal gains diminish. A quality-versus-compute analysis would help quantify the practical benefits of deeper inference.
We thank the reviewer for this practical question. We count compute as layer applications per token: the prelude, the coda, and applications of the shared block.
| Size | Ours, | Diagonal, | Layer applications |
|---|---|---|---|
| 60M | 19.08 | 22.45 | 22 vs 102 |
| 140M | 16.56 | 19.47 | 20 vs 100 |
| 370M | 12.66 | 14.90 | 40 vs 200 |
The table reports perplexity, lower is better.
https://freeimage.host/i/o3VyMLN
At our model is already better than the diagonal at , with about five times fewer layer applications. Past our model gains little, and the diagonal does not improve. A trip count of about is therefore the practical operating point. The revision adds this comparison in Appendix H.
How should the optimal group count scale with model width? The experiments identify G=2 as optimal for the synthetic model at d=512 and the preliminary language-model sweep at d=640. However, the larger models at d=768 and d=1024 retain G=2 without a corresponding sweep. Is there a theoretical or empirical reason to expect the optimal number of groups to remain approximately constant as width increases, rather than scaling proportionally with dimensionality?
We thank the reviewer for this question. The argument in our previous answer does not ask to grow with . What matters is that each group keeps enough dimensions to hold a signal, and that there are at least two lifetimes. At a larger width, the two groups of only become larger, so the trade-off that made best should not turn against it. We swept at and and kept as can be seen from Table 8.
How should the Toeplitz-defect results be interpreted for blockwise conformal operators? Table 5 reports defects near machine precision for the G=2 models. However, blockwise conformality guarantees angular preservation within each group, whereas differing group decay rates do not generally preserve global conformality. Do the learned decay rates converge to nearly identical values, or is the reported defect evaluated separately within groups? Reporting the learned group-specific decay rates would help clarify this relationship.
We thank the reviewer for this precise question. Each group has a learned scale , and a write in that group is multiplied by at every recurrent step. That scale is the group’s decay rate.
Inside a group the operator is exactly conformal, so the defect computed per group is at machine precision. On the whole state, two groups with different rates are not conformal, and the defect is small but nonzero, well below the free diagonal’s. The conformal rows of Table 5 correspond to the per-group value, and the revision labels them accordingly (Table 5 and Appendix B).
Does the theoretical guarantee translate to the full nonlinear transformer? The theoretical analysis and Table 5 examine the transition operator in isolation, but the actual recurrent dynamics also involve nonlinear transformer blocks. Could the authors measure information retention or angular collapse in the actual hidden-state trajectories to demonstrate that the proposed mechanism explains the observed performance gains?
We thank the reviewer for mentioning this point. We would like to provide the measurements and results for nonlinear block analysis below and in the revised version of the paper (Section 5.5 and Appendix G).
At the last token, a small change of the state passes through and then through the block. One loop step therefore has the Jacobian
where is the Jacobian of the block (attention and MLP) with respect to its input. Attention is causal, so no other position depends on the last token’s state, and the product of these Jacobians over steps is exactly how a write made at one depth reaches a later depth, including the nonlinearity and the input-dependent addition. Without the block, and the product reduces to , the object of Section 3.
Let measure how far a linear map is from readable, so that exactly when is conformal (Proposition 3.2). From depth to depth ,
because the largest singular value of a product is at most the product of the largest ones, and the smallest is at least the product of the smallest ones.
The bound separates what the block does from what the transition operator does. The first term belongs to the block and is present for every choice of . The second term is the operator’s own share, and it is added again at every recurrent step. For a conformal operator this share is exactly zero, . For the blockwise operator it is , the spread between the group rates, and it is zero inside each group. For a free diagonal it is the spread between all per-coordinate rates, the largest of the three. So readability of does carry into the nonlinear recurrence in this sense.
To see the difference of how the state changes across architectures taking into account the nonlinear operations, we take the gradient of the log probability of the correct answer with respect to the state at each depth . It tells how much the final answer depends on each dimension of the state at that depth. We sort the dimensions by how fast shrinks them, split them into four equal quarters, and compare the slowest quarter with the fastest:
A positive means the answer leans on the coordinates that keeps the longest, so ’s preference has made it through the block. An near zero means that no part of the state is privileged. We chose quarters because they are coarse enough to be stable across words and seeds, and fine enough to show the ordering that creates, with one number per depth.
https://freeimage.host/i/o3GV0gV
The figure compares each trained loop (left) with what alone would do over the same number of steps (right).
The ordering follows how far is from conformal. Through the loop, is largest for the free diagonal, smaller for , and closer to zero for , for and an orthogonal (Cayley) operator is predictably almost zero. Under alone the order is the same. So the preference a non-conformal creates among coordinates is what is acquired by the architectures and passed to the next recurrent steps. This way, the performance and readability of states are dependent on the relation between slow and fast part of the hidden representations.
We thank the reviewer for this remark. Respectfully, we disagree. The sizes we report are the ones this comparison is made at and practical for current scopes of related work. Solve the Loop, Parcae, Towards Looped models done right evaluates its language models at scales comparable or equal to 140M, 370M, and 770M [1] [2] [3]. Our models’ scale fall inside the appropriate ranges. We see this as an encouraging sign for larger scales, and we list a larger-scale study as future work.
[1] Fein-Ashley J, Rashidinejad P. Solve the Loop: Attractor Models for Language and Reasoning. arXiv:2605.12466, 2026.
[2] Prairie H, Novack Z, Berg-Kirkpatrick T, Fu DY. Parcae: Scaling Laws for Stable Looped Language Models. arXiv:2604.12946, 2026.
[3] Huang B, Shi C, Chen J, Wen S, Liu Z, Xing E, Ma X. Towards Looped Models Done Right, Part II: Rethinking at Fixed Points. arXiv:2610.06833, 2026.
We thank the reviewer for this question. We would like to explain how G acts.
The blockwise injection does two things at once. Inside a group, it keeps the geometry exact, so writes stay comparable over many steps. Across groups, it allows different lifetimes. The number of groups G decides how the width d is shared between these two purposes, since every group holds d/G coordinates. More groups give more lifetimes, but each group becomes smaller, and a small group leaves fewer dimensions in which to store and read a signal. With G=2, each group keeps half of the width, and there are still two lifetimes. In our sweeps this was the best trade-off on state tracking and on language modelling. Large G cuts the width into many small groups, which is why some of those settings fall below the baseline.