- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether recurrent architectures (Ouro-2.6B and Huginn-0125, with Qwen3.6-27B as baseline) exhibit the same functional workspace organisation as standard transformers, using distance-controlled Jacobian lens analyses and eleven behavioural experiment families, and finds that functional workspace organisation is preserved under recurrence despite being partially masked by readout limitations.
Critical strengths
None
Critical weaknesses
- Lack of a clear research problem and effective contributions. After reading the Abstract, I did not obtain any substantive information about the proposed method or the core insight of the work. The author appears to merely describe a scenario involving a Jacobian Transformer and a set of specific experimental observations. Beyond that, it remains unclear what scientific problem the author intends to solve or what novel approach is being proposed. Furthermore, the wording throughout the Abstract is highly obscure and convoluted, making it very difficult for readers to follow the author's intended meaning.
- Weak motivation and unclear logical flow in the Introduction. The author raises the question "Does the same functional organisation arise when depth is implemented by re-applying the same weights?" However, it is unclear whether this question is sufficiently important, and the author does not review to what extent this question has already been addressed in existing work. No analysis or motivation is constructed around this question. The Introduction then abruptly moves on to describing two checkpoints the author examined, followed by an account of reproducing baseline methods. The overall narrative reads more like an undergraduate course lab report than a coherent research paper, leaving the reader confused about what the author actually aims to convey.
Questions for authors
This paper reads like an undergraduate course lab report and does not belong at ICLR. Course assignments should not be submitted to ICLR in the first place; doing so not only violates the spirit of the conference's submission policy but also severely wastes reviewers' time and resources.
2 comments
Thank you for the response and clarification. I reread the revised abstract and found it clearer than the original. I appreciate the authors’ efforts. My current understanding is that the paper investigates what the Jacobian lens reveals about looped Transformers and how these findings differ from those for standard Transformers.
My first concern is that this question remains too broad. It reads more like an exploratory experiment than a well-defined research question. Almost any observations from these experiments could be presented as an answer. For a paper at a leading research conference, I would expect a more specific question and a clear explanation of why it matters. What makes this question worth studying, and how does it relate to prior work? Has it already been investigated?
The abstract states that the study compares two looped models with three standard Transformers. I do not find this range of models sufficient to support the conclusions being drawn. The authors report three findings, but in my view, these findings lack adequate theoretical justification and sufficiently comprehensive experimental evidence.
Finally, it is unclear how these observations translate into a substantive research contribution. Do they motivate a new method or help resolve a concrete problem? As presented, the paper reads largely as an experimental report, without a clear account of its scientific contribution. This is why I previously described it as resembling an undergraduate course lab report.
I will maintain my score, this is my final decision.
We have re-organised the paper to improve its clarity. Section, figure and table numbers are those of the revised paper.
W1. Research problem, approach and contributions.
Research problem. A looped transformer applies one block of layers several times in a forward pass. The output does not show what a later iteration does with the state it receives. In standard transformers, the Jacobian lens of Gurnee et al. (2026) reads the concepts held in an intermediate state as words, and edits along its directions change the model's answer. The introduction now states the research question explicitly, in italics: "what does the Jacobian lens show in a looped transformer, and where does the picture differ from a standard one?" (Section 1).
Approach. We keep the lens estimator of Gurnee et al. and add what a looped model requires:
The experiments form two groups, reading (Section 4) and writing (Section 5), and each test is introduced with an example. In "Fact: The language spoken in the country where the Amazon River ends is", the intermediate concept (the bridge) is Brazil and the prompt never states it. The reading test asks whether the lens recovers Brazil. The writing test moves Brazil onto Mexico and succeeds when the answer becomes Spanish.
Contributions. The abstract and the introduction list three:
On the wording, every term is defined where it first appears (Section 2 onwards).
W2. Motivation and earlier work.
Why the question matters. Looped transformers are an important research direction for language models: Ouro and Huginn are trained to continue their computation in the hidden state over several iterations before they answer. This reasoning cannot be inspected as written reasoning can. The Jacobian lens turns it into words, which is a first step towards overseeing what such models compute before they answer. The output does not show whether a later iteration carries the work of the one before it further or computes it again. Our third contribution answers this for Ouro: each loop rebuilds the state it is handed.
How far the question has been addressed (Section 1, paragraphs 2 and 3; Section 7). For standard transformers, Gurnee et al. (2026) read an intermediate state in words and test by edits whether the model uses what is read. To our knowledge, their lens has not been applied to looped models. Whether a looped model uses what can be read from its state, and what an iteration does with the state it is handed, remained untested.
On the remark under "Questions for authors".
The paper is not coursework. It is an interpretability study of two released looped language models, compared with three standard transformers. If the reviewer has further questions, we would be glad to discuss them.