acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Latent Reasoning: How Test-Time Compute Tracks Circuit Depth

Abstract

A recurrent-depth language model spends test-time compute by iterating a weight-shared block times, reasoning in latent space rather than emitting tokens. We show that this depth axis obeys a mechanistic scaling law: latent accuracy is the cumulative distribution function of a task's circuit depth, discounted by a per-step reliability that grows with model size . The law makes three falsifiable predictions: (i) accuracy saturates in , (ii) the iterations required grow linearly with reasoning depth, and (iii) the value of additional computation is proportional to the serial depth it advances. We verify these predictions on a compute-optimal ladder trained from scratch (M–M non-embedding parameters) across three reasoning domains. Fitting the law on smaller models predicts the entire depth-accuracy curve of the held-out largest model (), with the deeper of two depths requiring more iterations in of untied comparisons. The released Huginn-B follows the same saturating law beyond our ladder on GSM8K, ProsQA, and SVAMP (), while GSM8K accuracy decays geometrically with gold-solution depth (). Continuous-thought positions provide substantially less accuracy gain per unit of compute than recurrent depth, and Coconut does not out-scale CoT on public ProsQA. Optimizing the law yields an inference-time analogue of the Chinchilla rule for allocating compute between model size and reasoning depth. Across these settings, test-time compute scales when it advances the serial depth required by the task, establishing circuit depth as the relevant unit of latent computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, recurrent-depth transformers, test-time scaling, scaling laws, circuit depth, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SESyOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how performance of depth recurrent models changes with model size and number of times the recurrent block is iterated, fitting scaling laws. The initial setup is quite clean, the authors follow the Huginn architecture, and study 3 tasks modular arithmetic, deduction and permutation. The authors motivate well that it is actually circuit depth that is a dial for difficulty for the number of iterations required to solve a problem, and that the circuit depth is a distribution in the data. The authors note different models have different accuracies and gains from iterations and combine this with the circuit depth observation to motivate a closed form. The authors train models at different sizes on a Chinchilla optimal number of tokens. The authors study models from 19 to 121 million parameters, showing experiments where the law extrapolates the 121M parameter models. The authors also use the larger Huginn model to show their experiments generalise and also vary the number of iterations of the recurrent block by fine-tuning their models. The authors also compare with Coconut, showing that this gives a much smaller return that recurrent depth for solving state tracking tasks. A neat finding is that the accuracy over iteration distribution is often aligned with the difficulty distribution of the benchmark, although the definition of difficulty is defined as the number of lines in the solution. The authors also use a branching task to show that their circuit depth observation is correct, and it is not just the size of the problem that impacts number of iterations required.

Critical strengths
  1. This is a well motivated work studying inference scaling laws. It could be highlighted a little more how this paper is not a regular scaling law, emphasising more all the merits the paper does have.
  2. The function form is well motivated and generalises to larger models.

  3. The majority of the paper is well written and explained.
Critical weaknesses
  1. The related works section “Scaling laws for depth-recurrent models.” Is missing several related works which study scaling laws for recurrent models, for example [1]. There is also prior work studying recurrent models for arithmetic [2]
  2. The last section of the paper “How should inference budget be split between size and depth” is a little rushed and could do with more work and space.
  3. There is a lot of results presented in the text, it would be nice to have these in figures and tables so they’re easy to pick out.


The scoring system is very coarse I am very close to awarding a 4. Cleaning up the “How should inference budget be split between size and depth” is most likely to move my score. I will also take into account the other reviews also in my updated score, so this is not a guarantee of score movement. Score moved 3 -> 4 during rebuttal

[1] Prairie, H., Novack, Z., Berg-Kirkpatrick, T. and Fu, D.Y., 2026. Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946. [2] McLeish, S., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B.R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A. and Goldstein, T., 2024. Transformers can do arithmetic with the right embeddings. Advances in Neural Information Processing Systems, 37, pp.108012-108041.

Questions for authors
  1. In Table 1 the largest prediction gap is at T=1, is this due to the functional form caring too much about the larger T values? Perhaps this could suggest an incorrectness in the functional form?
  2. Why doesn’t the number of layers in the prelude and coda need to be taken into account in the functional form? E.g. a model with a huge prelude and coda could see very different uses recurrent block
2 comments
Reviewer_SESyOpenReview ↗

Rebuttal Response

Thank you for your prompt response. It addresses my concerns. I believe the paper addresses a gap in the literature well and should therefore be accepted.

Thank you for your positive assessment and for recognizing the contribution of our work. We sincerely appreciate your time and consideration.

Reviewer_iHmxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a study of the relationship between the performance of looping LLMs (in their variant with a prelude and a coda) and the structure of the problems they are asked to solve. The authors identify a functional form connecting accuracy to the expected depth of a circuit that can solve the task, the number of iterations or of latent tokens, and number of parameters of the model, and validate it through controlled experiments on mostly artificial datasets.

Critical strengths
  • The core results are an important verification of the intuition that transformer blocks implement elementary computation circuits and that dedicated ‘looping’ training can teach models to compose them.

  • The comparison of the relative effectiveness of different methods for realising composition is practically relevant, as it highlights a gap between alternative approaches to leverage inference-time computation.

Critical weaknesses
  • The considered tasks are all based on repeated deterministic application of elementary computation. They do not consider for instance tasks involving search, for which [Zhu et al. 2026] have shown that latent thought models can have a considerable advantage. It would be important to qualify the claims in this respect.

[Zhu et al. 2026] Zhu, H., Hao, S., Hu, Z., Jiao, J., Russell, S. and Tian, Y., 2026, April. Emergence of superposition: Unveiling the training dynamics of chain of continuous thought. In International Conference on Learning Representations (Vol. 2026, pp. 62257-62288).

  • The writing style is typical of AI (Claude?), with highly compressed sentences that are cryptic and need multiple reading passes. The last sentence in the abstract, just to mention one, is obscure before reading the paper. Terms are used before being introduced (e.g. ‘serial depth’ already in the abstract, ‘globality barrier’). The paper would benefit from a heavy-handed pass to make it more readable.

  • The assumptions on the way the problem is modelled are not clearly spelled out. Are operations expected to have all the same probability of success? Is this probability independent of where an operation is in the input? Is there any assumption on the algebraic properties of operations (e.g. associativity, commutativity)?

  • Several important aspects (model architecture, task construction) are relegated to the appendix, which reviewers are not expected to read.

Questions for authors
  • ‘Accuracy is a circuit-depth CDF’ in Proposition 1: how about the p(N)dp(N)^d term?
  • Claim 2 ‘A natural interpretation is that a less reliable executor must re-derive failed steps’: is there anything supporting this conjecture?
2 comments
Reviewer_iHmxOpenReview ↗

Acknowledgment of rebuttal.

Thank you for your rebuttal. These changes and clarifications are sufficient to justify an increase in rating from 2 to 3.

Thank you for your thoughtful feedback and for reconsidering your assessment in light of our clarifications. We sincerely appreciate your time and consideration. We would be grateful for any further questions, concerns, or suggestions that could help us further strengthen the paper. We remain available and would be happy to provide additional clarifications or make further revisions wherever helpful. Thank you again for your constructive feedback and support.

Reviewer_rYS2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how test-time accuracy scales with the number of recurrent iterations T in looped transformers. The authors propose a simple model where each iteration advances serial computation depth by a rate alpha and each step succeeds with probability p(N). Under this model, accuracy as a function of log T follows the CDF of the task's circuit depth scaled by a size-dependent reliability ceiling, and required iterations grow linearly with task depth. They test these predictions on a compute-matched ladder of recurrent models from 19M to 121M parameters on synthetic multi-step tasks, and also evaluate the pretrained Huginn-3.5B checkpoint on GSM8K, SVAMP, and ProsQA. Finally, they compare recurrent depth against Coconut-style continuous thoughts and chain-of-thought under matched compute.

Critical strengths

(1) Prior test-time scaling laws focus on token generation, and it has been unclear how accuracy scales when a model spends compute by looping hidden states in depth. This paper addresses that gap with a clean, testable model and a well-controlled experimental ladder. By pretraining six recurrent models with compute-matched token budgets and controlling for operations that collapse the state in mod-10 arithmetic, the authors isolate true dependency depth from surface task parameters. Across three synthetic task families, accuracy consistently saturates in log T and the onset shifts later as the effective dependency chain grows.

(2) A common limitation of synthetic scaling studies is that the fitted relationship might only reflect the toy generators used for training. Here, separating the scaling curve into a depth-coverage onset and a per-step reliability ceiling yields concrete predictions that transfer to a pretrained 3.5B model the authors did not train. On GSM8K, Huginn-3.5B's plateau accuracy drops geometrically from about 0.57 at depth 2 to 0.15 at depth 7, while few-shot ProsQA and SVAMP show the sharp turn-on around T=8 predicted by depth coverage. The comparison on Pythia showing Coconut falling behind CoT as model size grows from 410M to 2.8B is also a useful practical result.

Critical weaknesses

(1) The central claim of the paper is that test-time compute tracks serial circuit depth rather than total problem length, but the experiments do not convincingly separate the two. On the three linear-chain tasks, circuit depth and operation count grow together, so the entire distinction rests on the branching TreeTrack task in Sec 5.2. Looking at Fig 5b, only height 1 is solved, while heights 2, 3, and 4 stay low and roughly flat across the sweep without the depth-ordered onsets or plateaus claimed in the text. Prior theory shows that logarithmic depth is enough for parallelizable tree-like evaluation such as iterated composition and graph connectivity [1, 2], and these same models solve 5-step mod-10 chains on StateTrack from 44M up. So the flat curves on h=2,3,4 look like a fine-tuning failure on nested parentheses rather than a depth bottleneck.

(2) The empirical onset metric T*(K) mixes depth coverage with ceiling reliability, which makes the paper's quantitative slope and resolution-rate claims hard to interpret. Sec 5.2 defines T*(K) using an absolute 0.5 accuracy threshold, whereas the theory and the GSM8K analysis in Appendix K define onset relative to each curve's own plateau. Under a sigmoid curve, a lower plateau shifts the absolute 0.5 crossing to the right even when the midpoint T0 does not change. Because larger K and smaller N both lower the plateau,eg on the smaller ladder models, an absolute threshold makes harder depths and smaller models look like they turn on later simply because their ceiling is lower, which confounds the size trend in Table 5 and section 3.

ref: [1] Sanford, Hsu & Telgarsky, "Transformers, parallel computation, and logarithmic depth", ICML 2024. [2] Merrill & Sabharwal, "A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers", arXiv:2503.03961, 2025.

Questions for authors

(1) Did Table 5, Fig 2b, and Table 6 use an absolute 0.5 accuracy threshold or half of each curve's own plateau to define T*(K)? If they used absolute 0.5, how do the depth slopes and the trend across model size N look when recomputed under a relative half-plateau definition?

(2)On TreeTrack, what are the per-height onsets and plateaus for h=2, 3, and 4 across all six ladder sizes, rather than only the single model plotted in Fig 5b? Is there any model size where heights 2 and 3 clearly seperate in onset?

(3)In Sec 5.2, how is the "per unit of budget" gain ratio of 1/8 to 1/4 for continuous thoughts defined, and what is that ratio when compared at matched FLOPs rather than step count? Also, recent theoretical work argues that continuous thoughts mainly help by holding a superposition of search frontiers on branching problems [3], so could deterministic linear chains like StateTrack and LogicTrack simply be an unfavorable task structure for Coconut?

ref: [3] Zhu, Hao, Hu, Jiao, Russell & Tian, "Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought", arXiv:2505.12514, 2025.

1 comment
AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period ending in a few days

Dear Reviewer rYS2,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we explained why the original TreeTrack result was inconclusive (multiplication modulo 10 collapses many subtrees) and replaced it with TreeTrack-B. TreeTrack-B compares balanced trees with left-deep chains of matched operation count under both post-order and level-by-level supervision, and the recurrent budget follows the number of supervised operations. We also recomputed the onset analysis with plateau-aware definitions, under which onset is positively associated with depth in all 54 curves, and we withdrew the claim that the onset slope decreases with model size. Finally, we separated equal-step from equal-FLOP comparisons with continuous thoughts and reported the matched-FLOP frontier. All of these changes appear in the revised paper.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.