acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Refinement Moves Candidate-Quality Readability Earlier and Stabilizes Its Coordinates

Abstract

A looped language model applies the same layer stack several times to a token position, so “physical layer 8” is not one event but one per pass. This makes a question well-posed that a feed-forward stack cannot ask: when during recurrent refinement does a property of the computation become linearly readable at a given layer, and is it carried by the same coordinates throughout? We answer it for candidate-quality information in a frozen 2.6B looped transformer (Ouro), with small bias-free antisymmetric taps selected from a fixed domain-subset sweep and evaluated on a task-disjoint five-domain candidate-ranking target under an audited pairwise protocol. Two evaluation modes separate two questions. Locus-local refit asks whether the information is readable at a loop/layer cell: held-out macro top-1 at physical layers 8 and 16 is weak on the first pass (0.410, 0.440; chance 0.282) and statistically tied with the established late-layer basis by loops 3–4 (0.633, 0.628), with loop 4 − loop 1 contrasts of +0.19 to +0.21 at every tested layer. Frozen cross-loop transplant asks whether it is carried by functionally stable readout coordinates: taps fitted at loop 1 lose 0.09–0.23 when applied unchanged at loop 4, while taps fitted at loops 2–3 transfer at parity (|Δ| ≤ 0.024). This indicates a first-pass coordinate change followed by stable frozen-transfer behavior. Both signatures replicate on three further Ouro checkpoints, and the archived RLTT taps transfer frozen across checkpoints within 0.028 of locally refit taps; the depth trend replicates qualitatively in an out-of-family depth-recurrent model (Huginn, +0.107 from step 1 to step 8), where no comparable immediate first-pass transfer discontinuity is detected. A non-looped control shows that a same-class readout exists without recurrence, so we claim a property of where and when the signal is readable in looped models, not that looping creates it. All results are readout-only; no intervention is performed.

Primary area:
interpretability and explainable AI
Keywords:
looped transformers, recurrent refinement, hidden-state readouts, linear probes, representation dynamics, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rmyMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how candidate-quality information evolves across recurrent iterations in looped language models. Using linear probes on frozen Ouro-RLTT hidden states, the authors show that candidate-quality information becomes increasingly readable across loops, with early physical layers achieving performance comparable to late-layer baselines by loops 3–4. They further find that probes trained on the first loop transfer poorly to later loops, whereas probes trained from loop 2 onward remain effective, suggesting an initial change followed by stable readout directions. These findings are validated across multiple Ouro checkpoints, with partial replication on Huginn and a non-looped Transformer control.

Critical strengths

The distinction between local probe readability and frozen cross-loop transfer is a potentially interesting approach to studying representation dynamics in looped transformers. However, I do not find this contribution sufficiently novel or substantial to outweigh the concerns below.

Critical weaknesses
  1. This paper is completely an AI slop in my opinion

    • The paper is pretty hard to read, starting from the first sentence in the abstract (so “physical layer 8” is not one event but one per pass). While reading the abstract, I have no idea what "physical layer 8" is.
    • There are so many clear AI generated signals without any human intervention. E.g., unreadable draft and "SHA256SUMS" everywhere in the appendix.
  2. After I dig into some related works, I noticed that a large part of this paper is exactly the same as a paper cited by the author (arXiv preprint 2607.18553), and surprisingly they cited it ~10 times all through the paper. More specifically, the Figure 1 in this paper is the same as Figure 2 in this arXiv paper. Much of the data matches the arXiv preprint. The author even mentioned in Line 450 that The taps, CoreContent target, late-layer basis, and correction history come from a broader programme on frozen Ouro-RLTT (Kirin, 2026a;b). So it either violates double-blind (if the authors are the same, which is highly likely) or plagiarism. Therefore, I recommend that this submission be desk-rejected.

Questions for authors

See Weaknesses

Reviewer_wsT4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when candidate-quality information becomes linearly readable during the repeated passes of a looped language model and whether a readout learned at one pass continues to work at later passes. The main experiments use frozen hidden states from a 2.6B-parameter Ouro-RLTT model and train small bias-free, antisymmetric linear probes ("taps") to rank candidates from the CoreContent v2 dataset. The paper distinguishes two evaluations. In a locus-local refit, a new probe is trained at every layer/pass cell to measure whether candidate quality is linearly readable there. In a frozen cross-loop transplant, a probe trained at an earlier pass is applied unchanged at a later pass of the same physical layer to test whether the same readout direction remains useful.

Critical strengths

The distinction between local readability and frozen cross-loop transfer is useful. A separate probe trained at each layer/pass cell cannot show whether the same readout remains valid across passes. Evaluating both local refits and frozen transplants therefore addresses two genuinely different questions and is a sensible way to study recurrent representations.

Critical weaknesses

In my opinion, the paper is very difficult to understand and is not sufficiently self-contained. The main idea is fairly simple, but it is buried under a large amount of bespoke terminology, procedural detail, correction history, and references to a broader prior program. Central objects such as CoreContent v2, the construction of candidate groups and quality labels, the historical "late-layer basis," and the practical meaning of the learned direction are not introduced with enough concrete examples. Terms such as "tap," "candidate readout surface," "rotation," "parity," and "operational proto-introspection" accumulate faster than the underlying concepts are clarified. The manuscript also repeatedly alternates between headline claims, caveats, historical results, configuration audits, and claims about what was pre-specified. Despite careful reading, I found it hard to reconstruct a clean chain from the scientific question to the experiment and then to the conclusion.

The contribution is already narrow and descriptive; if the experimental object and interpretation cannot be understood without substantial familiarity with the authors' earlier work, it is difficult to evaluate the significance of the result.

Further, some of the main interpretations are stronger than the evidence supports.

First, the paper calls cells "at parity" when their point estimate is within an externally chosen 0.03 margin of the layer-24/pass-4 reference and the ordinary difference interval includes zero. Failure to find a significant difference is not evidence of equivalence, and several reported intervals extend outside the stated parity margin. A proper equivalence or non-inferiority analysis, together with sensitivity to the selected 0.03 threshold, is needed to support this language.

Second, degradation of a frozen linear probe relative to a separately selected local refit shows that one fitted readout is not equally effective at the target representation, but it does not uniquely demonstrate a coordinate "rotation" or a distinct coordinate system. Changes in feature scale, covariance, separability, or the non-uniqueness of probe solutions can also affect frozen transfer. I think this concern is relevant because the main probe intentionally preserves raw magnitude, while configurations are selected over training subsets and learning rates. The authors appropriately describe rotation as operational in some places, but the mechanistic language and headline interpretation remain too strong.

Questions for authors
  1. Can the authors give a self-contained description of CoreContent v2, including representative source tasks, how alternative candidates are generated, how the designated-best label is obtained in each domain, group-size and label distributions, and several concrete examples?

  2. Does the pass-1 transfer discontinuity persist when all cells use one fixed training-domain subset and hyperparameter configuration, and when hidden states or pair differences are normalized or whitened? Can the authors report feature-norm/covariance changes and a direct alignment analysis across passes? Such evidence would help distinguish a changed coordinate system from scale, conditioning, or probe-selection effects.

Reviewer_ci62OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to answer a well-posed question that when does a property of the computation become linearly readable at a give layer, during recurrect refinement, and whether is it carried by the same coordinate throuput. This paper investigate this problem using a fronzen 2.6B looped transformer. Based on thatm two evaluation modes are proposed. There are some intersting findings for the paper such that looping is not hte cause of making signal readable.

Critical strengths

(1) This paper investigates an interesting questions specifically targeting the loop-transofrmer readability via local-local refit and fronzen cross-loop transplant. By that, the paper find out recrrent depth correlate with linear readability. For example, From loop 1 to 2, there is a large change in frozen readout transfer. (2) The design is clear, and there is stastically supported empirical findings. (3) The acknowledgement of limitations is appropriate.

Critical weaknesses

(1) The operational parity threshold does not establish statistical equivalence, as some reported confidence intervals extend beyond the specified margi (for example layer 8 at loop 3) (2) The experimental results show interesting patterns, but the underlying reasons for these changes are not sufficiently explained. Further discussion or analysis of the mechanisms behind these observations would strengthen the contribution. (3) Most experiments are conducted on the Ouro model family, while the results on Huginn show somewhat different patterns. (4) Potential applications should be provided based on the understandings of the findings.

Questions for authors

(1) Authors acknowledge the cross-family resutls weaknen the universality of the findings, but a more broader mechanistic interpretation is encouraged. (2) The readability of the paper should be further increased. (3) Potential applications should be provided based on the understandings of the findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.