acceptodds
Under review as a conference paper at ICLR 2027

Tiny Recursive Models Underuse their Depth

Abstract

Tiny recursive models (TRMs) solve hard reasoning benchmarks by repeatedly applying a small weight-tied network to a pair of carried states, and improvements to this family have so far come from iterating more: extra recursion cycles, extra supervision steps, more effective depth. However, a failure case emerges: the recursion reaches a fixed point quickly. Successive iterations change the carried states less and less, so much of the depth budget is spent after the recursion has effectively reached its fixed point, and the added depth is ineffective. We show that a larger residual stream alleviates this failure case. Adopting Virtual Width Networks (VWN), we widen the state the recursion carries several-fold while leaving backbone compute unchanged, so each iteration has room to write something the previous one did not and the fixed point becomes harder to reach. Widening also makes the depth TRMs already run useful: held-out sweeps show the wide model's margin over the narrow one growing with the number of supervision steps allowed at evaluation when many candidate answers are scored, while the narrow model's candidate pool stops improving early. We validate on ARC-AGI-1, ARC-AGI-2, and Sudoku-Extreme with 7M-parameter models under matched configurations. The resulting VWN+TRM improves ARC-AGI-1 pass@2 by at least points over both TRM baselines we trained (the upstream and the depth-matched configuration), improves ARC-AGI-2 pass@2 by points over the matched baseline ( points, or relative, over the published TRM result), and adds points of exact accuracy on Sudoku-Extreme.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recursion, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p1JLOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that tiny recursive models (TRMs) underuse their recursion depth because their narrow carried states saturate quickly. It adapts Virtual Width Networks to widen these states 5× at nearly constant compute (+5.5% parameters). With 7M-parameter models, this improves ARC-AGI-1 pass@2 by 7.1 points, ARC-AGI-2 by 3.8, and Sudoku-Extreme by 11.2.

Critical strengths
  1. Simple, cheap, and consistent gains. Widening the carried states adds only 5.5% parameters, no extra depth, and is function-equivalent to TRM at init, yet improves all three benchmarks at every pass@k and nearly every checkpoint.
  2. Useful diagnosis of the TRM family. Showing that 4× more depth does not help and that halting collapses to 1–2 of 16 steps is a meaningful finding for a line of work that has mostly scaled iteration.
  3. Transparent evaluation. The non-standard cumulative-ensemble pass@k is clearly specified, top-2 scores are independently re-derived, and the failure to reproduce TRM's ARC-AGI-2 number is reported openly.
Critical weaknesses
  1. The central mechanism is asserted, not measured. The fixed-point / saturated-state explanation is never tested directly (e.g., iteration-to-iteration state change or effective rank of z_H, z_L), and the paper's own data cut against it: the wide model halts earlier (1.19 vs. 1.31 steps on ARC-1), and in Fig. 7 its pass@100 rises only 0.75 points from budget 4 to 16 while the pass@1 margin shrinks. The title claim that width makes depth useful is therefore not supported.

  2. The gain is not cleanly attributed to width. There is no parameter-matched baseline that simply widens D, no sweep over n, and no static-vs-dynamic GHC ablation, so the improvement may come from learned GHC routing rather than state capacity. VWN is also tested only at H6/L9, and all results are single-seed (the ARC-AGI-2 gap is 16 vs. 9 grids out of 172).

Questions for authors
  1. Can you report FLOPs, memory, and wall-clock for both models? "Nearly constant compute" is currently unquantified, and the runs used different GPU counts.
  2. Your ARC-AGI-2 TRM baseline (5.28) is below the published 7.8. What explains the gap, and does VWN still help over a baseline that matches the published number?
  3. Halting-head accuracy on ARC drops from 74.3% to 63.1%, and q is used to rank candidates. How much does this affect pass@k?
  4. Can you report standard single-checkpoint pass@2 as the main ARC metric, rather than the cumulative 10-checkpoint ensemble?
  5. Both ablations in Table 7 are partial runs. Can you complete them?
  6. What does this paper add beyond VWN (Seed et al., 2025) and the memory–depth trade-off in Sapunov (2026)?
1 comment

Thank you for the review. Short answers below. Table 5 is in App. D and Table 7 in App. G.

W1 (mechanism). 4x more depth gives TRM no pass@1/2 gain (Sec. 6.1, Fig. 5). Its pool is flat after 4 of 16 steps (Fig. 1), and one wide step beats every narrow budget (Table 5). We will add per-step state change and effective rank of both carried states in the final version.

W1 (halting). Steps used (Table 3) is a training statistic. The head halts once the answer is right. Fewer steps means the wide model gets there sooner, as more capacity per step predicts (Sec. 6.2). Held-out usefulness of later steps is tested with halting disabled in Sec. 6.3.

W1 (Fig. 7). Table 5: from budget 4 to 16 TRM's pass@100 gains nothing, the wide model gains 0.75. From 1 to 16 the gains are 5.50 vs. 3.62. The pass@1 margin dips only from budget 1 to 4 as TRM catches up, then grows again (7.00, 7.24, 7.62).

W2 (attribution). VWN helps at two depths, H6/L9 on ARC (Table 1) and H3/L6 on Sudoku (Table 2). A parameter-matched TRM would only move D from 512 to about 526, since parameters grow as D^2. We will add that baseline, a sweep over n, static vs. dynamic GHC and more seeds in the final version.

Q1 (compute). Backbone FLOPs are unchanged. Attention and FFN run at D = 512, same sequence length. The overhead is one small batched contraction per sublayer (Sec. 4, Cost). We will report FLOPs, memory and wall-clock on identical hardware in the final version.

Q2 (ARC-AGI-2 baseline). Our baseline is the deep H6/L9 config, not the upstream H3/L4 (Sec. 5.1). At H3/L4 we match the published ARC-AGI-1 number, and VWN+TRM (9.03) beats the published 7.8 (Table 1). We will add an H3/L4 pair on ARC-AGI-2 in the final version.

Q3 (halting head). q is only the tie-breaker after vote count (Sec. 3, App. A). If anything a weaker q hurts VWN+TRM, yet its largest ARC-AGI-1 gain is at k = 1 (+9.0, Table 1). We will add a vote-only ranking for both models in the final version.

Q4 (single checkpoint). It is in Table 5 at budget 16, 52.50 vs. 46.75 pass@2 on ARC-AGI-1 (+5.75). Table 1 follows the upstream evaluator (Sec. 3, App. A). We will report single-checkpoint pass@2 next to it for both ARC benchmarks in the final version.

Q5 (partial ablations). Yes, we will train both to the full schedule for the final version. At matched evaluations pseudo-tokens are 1.4 pass@2 below GHC and 1.5 above TRM (Table 7).

Q6 (prior work). VWN was shown in feed-forward LM pretraining. We bring it to weight-tied recursion. Sapunov adds memory tokens on Sudoku. We widen the residual stream itself and test on ARC (Sec. 2). New: depth alone does not help TRM (Sec. 6.1), while state width decides how much is used (Sec. 6.3).

Reviewer_QUmoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the performance of Tiny Recursive Models (TRMs) is limited by the width of their recurrent hidden states rather than the amount of recursive computation. The authors argue that TRMs rapidly approach a fixed point, making additional recursion steps increasingly ineffective. To alleviate this issue, they incorporate Virtual Width Networks (VWN) into TRM, expanding the persistent hidden states while keeping the backbone computation at its original width through generalized hyper-connections.

The proposed VWN+TRM is evaluated on ARC-AGI-1, ARC-AGI-2, and Sudoku-Extreme. The authors report consistent improvements over their matched TRM baselines, including approximately 7.1 percentage points in ARC-AGI-1 pass@2, 3.8 points in ARC-AGI-2 pass@2, and 11.2 points in Sudoku-Extreme exact accuracy, with approximately 5.5% additional parameters. Additional analyses investigate recursion depth, supervision-step budgets, and alternative capacity-expansion strategies.

The main claim is that increasing the capacity of the recurrent state can improve reasoning performance and make additional recursive computation more useful.

Critical strengths

1. Consistent empirical improvements across multiple reasoning benchmarks.

The proposed method achieves improvements over matched TRM baselines on ARC-AGI-1, ARC-AGI-2, and Sudoku-Extreme, with only a modest increase in parameter count. The gains are especially notable on ARC-AGI-1 and Sudoku-Extreme. The use of matched backbone configurations and consistent evaluation procedures makes these results meaningful as empirical evidence that widening the recurrent state can be beneficial.

2. An interesting empirical investigation of the relationship between recurrent-state width and iterative computation.

The paper goes beyond reporting benchmark improvements by examining recursion depth, supervision-step budgets, and candidate generation. In particular, the observation that increasing depth alone provides limited benefits for the baseline TRM, while widening improves performance, raises an interesting question about how recurrent models allocate their computational capacity. Although the proposed explanation is not fully established, this empirical observation is potentially useful for designing more efficient recursive architectures.

Critical weaknesses

1. Limited methodological novelty and insufficient isolation of the source of improvement.

The proposed method primarily integrates Virtual Width Networks, an existing architectural technique, into the TRM framework. The main architectural components, including virtual-width residual states and generalized hyper-connections, are inherited from prior work. While applying these components to recurrent reasoning models is a reasonable extension, the paper does not introduce a substantially new mechanism or establish why this particular integration is necessary.

More importantly, the current experiments do not sufficiently disentangle the effects of increased state dimensionality, dynamic hyper-connections, additional parameters, and modified residual mixing. The pseudo-token comparison is insufficient for this purpose, particularly because it is based on an incomplete training run and changes the sequence length and computational structure. A more controlled ablation would be necessary to establish whether the reported gains specifically result from increased persistent-state capacity or from the additional architectural flexibility introduced by VWN.

This substantially weakens the paper's methodological contribution.

2. The central fixed-point and depth-utilization hypothesis is not convincingly validated.

The paper's primary motivation is that TRMs rapidly converge toward fixed points because their recurrent states lack sufficient capacity, and that widening these states delays convergence and enables more effective use of recursive depth. However, the experimental evidence is largely indirect.

In particular, the paper does not directly measure convergence of recurrent states, state-update magnitudes, representational dimensionality, or the relationship between these quantities and prediction accuracy. The saturation of pass@k and the early halting behavior do not, by themselves, establish that the underlying recurrent dynamics have reached fixed points, let alone that limited state capacity is the cause.

Furthermore, the claimed benefit of additional supervision steps is mainly observed at large candidate budgets. In the held-out step-budget analysis (Figure 7 and Appendix D), the improvements at pass@1 and pass@2 are small or saturate early even for VWN+TRM, whereas larger candidate budgets show more noticeable gains. This suggests that additional iterations may primarily improve candidate diversity rather than progressively refine the model's predictions. These two explanations have different implications for the paper's central claim.

Finally, all main experiments use a single random seed, and the ARC results rely on cumulative voting across ten evaluation checkpoints. Although this protocol is applied consistently, it complicates the interpretation of performance improvements as evidence of stronger iterative reasoning. The absence of multi-seed results and more direct mechanistic measurements makes the central conclusion insufficiently supported.

Overall, the paper demonstrates useful empirical improvements, but the evidence is not sufficient to substantiate its main mechanistic explanation.

Questions for authors

Q1. [Mechanistic clarification] Can the authors provide direct evidence that the baseline TRM reaches a fixed point earlier than VWN+TRM? For example, how do normalized state-update magnitudes, distances between consecutive recurrent states, or effective representation ranks evolve across recursion cycles and supervision steps? Can these measurements be related to prediction correctness? Such evidence would be important for distinguishing actual fixed-point convergence from saturation of the evaluation metrics.

Q2. [Critical ablation] How much of the improvement can be attributed specifically to increased recurrent-state width rather than the additional parameters or the dynamic generalized hyper-connections? Could the authors compare against parameter-matched TRM variants, static versus dynamic GHC, and different virtual-width ratios under otherwise identical training conditions? These comparisons would help establish whether recurrent-state capacity is indeed the primary limiting factor.

Q3. [Depth-utilization clarification] In Figure 7 and Appendix D, additional supervision steps provide relatively limited gains at pass@1 and pass@2, even for the widened model, while the benefits are more pronounced at large k. Does this indicate that the additional steps primarily increase candidate diversity rather than improve iterative refinement of a single prediction? Can the authors quantify answer changes, corrections of initially incorrect predictions, and newly discovered correct candidates across supervision steps?

Q4. [Experimental reliability] Can the authors report results across multiple random seeds and separate single-checkpoint performance from cumulative cross-checkpoint ensemble performance? This is particularly relevant for ARC-AGI-2, where the absolute number of solved tasks is small and the reproduced TRM baseline is below the previously published result. Understanding the variance and the effect of checkpoint aggregation would materially affect confidence in the reported improvements.

1 comment

Thank you for the detailed feedback. Short answers below. Table 5 is in App. D and Table 7 in App. G.

W1 (novelty). Prior gains in this family all come from iterating more (Sec. 2). We show this depth is underused (Sec. 6.1, 6.2) and that the width of the carried state is the lever. VWN was only used in feed-forward LM pretraining, never with 60 weight-tied reuses per step (Sec. 4).

Q1 (fixed point). Attractor behavior of TRM-style models is documented by Huang et al. (2026). We see the consequence in Fig. 1: from budget 4 to 16 TRM moves by at most 0.5 points at any k. We will add per-step update norms and effective rank, related to correctness, in the final version.

Q2 (isolating width). Extra parameters are an unlikely cause. Spending the +5.5% (App. B) on the backbone only moves D from 512 to about 526. Our state is 5x wider. We will add a parameter-matched TRM, a sweep over n, and static vs. dynamic GHC in the final version.

Q2 (pseudo-tokens). This run adds state without GHC and still beats TRM by 1.5 pass@2 (Table 7). So extra state helps on its own. We will train it to the full schedule for the final version.

Q3 (diversity vs. refinement). As Sec. 6.3 says, top-1 settles in the first step for both models. Past four steps only the wide pool keeps improving (Fig. 7a), which a fixed-point recursion could not do. The final version will add per-step counts of answer changes, corrections and new correct candidates.

Q4 (seeds, voting). Table 5 at budget 16 is single-checkpoint: 52.50 vs. 46.75 pass@2 on ARC-AGI-1. So the gain is not produced by cumulative voting. We will add more seeds and single-checkpoint ARC-AGI-2 numbers in the final version.

Q4 (ARC-AGI-2 baseline). Our baseline is the deep H6/L9 config, not the upstream H3/L4 (Sec. 5.1). At H3/L4 we match the published ARC-AGI-1 number, and VWN+TRM (9.03) beats the published 7.8 (Table 1). It solves 16 grids vs. 9 (App. A). We will add an H3/L4 pair on ARC-AGI-2 in the final version.

We hope this clarifies the contribution and would appreciate it if you reconsider the score.

Reviewer_3XpjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether improving the persistent state representations of Tiny Recursive Models can make recurrent computation more effective. It adapts Virtual Width Networks to TRM, expanding the answer and latent states while retaining the original attention and FFN widths, with dynamic generalized hyper-connections controlling state mixing. With approximately 5.5% additional dense-network parameters, the method reports single-seed gains over depth-matched baselines on ARC-AGI-1, ARC-AGI-2, and Sudoku-Extreme. ARC pass@2 improves by 7.1 and 3.8 percentage points under cumulative ten-checkpoint voting, while Sudoku exact accuracy improves by 11.2 points. A separate single-checkpoint ARC-AGI-1 sweep shows that one wide supervision step outperforms all tested narrow-model step budgets, with additional iterations providing some further benefits at large candidate cutoffs. I understand the main contribution as evidence that redesigning persistent states and residual connections is a promising direction for TRM. The experiments do not yet isolate whether the gains require increased state width or primarily reflect improved connectivity and optimization.

Critical strengths
  • A worthwhile direction for improving TRMs. The paper looks beyond adding more iterations and asks whether better persistent-state representations and connections can make recurrent computation more effective. I find this a useful direction, even if the specific capacity-based explanation remains open.

  • A practical architectural adaptation. Applying VWN to both carried states keeps the attention and FFN widths and sequence length unchanged, while adding only about 5.5% to the reported dense-network parameter count. This offers an appealing alternative to simply enlarging the backbone, though its actual runtime and memory costs still need measurement.

  • Meaningful gains across the evaluated tasks. The reported improvements over depth-matched baselines are substantial on both ARC benchmarks and Sudoku-Extreme. The additional upstream TRM baseline on ARC-AGI-1 also makes the comparison more convincing. These are single-seed results, but the size and consistency of the gains make the approach worth taking seriously.

  • An informative evaluation of iteration budgets. The single-checkpoint sweep goes beyond headline ensemble scores: one wide supervision step outperforms every tested narrow-model step budget across all six candidate cutoffs. This is an interesting finding in its own right and gives a concrete starting point for studying how state design affects the value of additional computation.

Critical weaknesses
  • The experiments do not isolate what makes VWN effective. State width, dynamic mixing, and normalization change together, with no same-width GHC or gated-residual control. The partial pseudo-token and LoopViT-style runs do not resolve this. The results support the full VWN modification, but leave open whether wider states are necessary or better connections could recover most of the gains.

  • The fixed-point explanation goes beyond the evidence. Performance saturation is treated as evidence that narrow states reach fixed points because their capacity is exhausted, but neither state convergence nor capacity usage is measured directly. Some wording also conflicts with the results: wide-model pass@1000 falls from 68.50% at four steps to 68.25% at sixteen, despite the claim of continued late-step improvement.

  • The empirical robustness needs stronger support. All main results use one seed, and checkpoint trajectories cannot substitute for independent runs. The ARC-AGI-2 baseline falls below the published TRM result, and the Sudoku attention baseline is also weaker than previously reported. These issues do not erase the matched gains, but make their expected size and reliability harder to judge.

  • The practical efficiency advantage is not yet established. Keeping backbone shapes unchanged does not account for the added mixing operations, wider activations, or memory traffic. Different GPU allocations prevent a clean runtime comparison, and the headline ARC scores require cumulative voting across ten checkpoints. The paper provides a promising accuracy result, but still needs matched measurements of total inference cost, throughput, and memory to establish the resource tradeoff.

Questions for authors

I would prioritize these questions for the authors to continuously improve your work:

  1. How much of the gain requires wider states, and how much comes from better residual connections? A same-width dynamic GHC control would be particularly helpful, alongside a wide-state variant with static mixing. These comparisons could distinguish the benefit of increased state capacity from the benefit of more flexible connectivity. If both matter, can you characterize their separate contributions and interaction?

  2. What direct evidence supports the premature fixed-point explanation? Using the existing checkpoints, could you examine normalized state changes across iterations, separately for correctly and incorrectly solved examples? It would also help to connect these measurements to the step-budget results: the wide model is already substantially better after one step, while pass@2 is flat beyond four steps and pass@1000 slightly declines. What do these observations suggest about better early computation versus more useful late computation?

  3. How stable are the main gains, and what explains the weaker baseline reproductions? Could you investigate the gap from the published ARC-AGI-2 and attention-based Sudoku baselines, and repeat at least one representative matched comparison with additional seeds? Task-level paired uncertainty would also help interpret the smaller step-budget differences, although it would not replace seed replication. I would be interested in whether the main conclusions survive these checks even if the exact improvement changes.

  4. Does the early-step advantage translate into a practical accuracy-cost improvement? Could you profile the matched models on the same hardware, reporting inference latency and peak memory at several step budgets? The finding that one wide step beats all tested narrow budgets seems especially promising here. Extending the single-checkpoint ARC evaluation to the main comparisons would also clarify how much of the benefit is available without ten-checkpoint voting.

The first two would matter most for me to potentially reconsider my assessment.

1 comment

Thank you for the careful review. Our answers follow Q1 to Q4, which also cover the four weaknesses. Table 5 is in App. D and Table 7 in App. G.

Q1 (width vs. connections). In Seed et al. (2025) the gain grows log-linearly with width (Sec. 2). Our pseudo-token run adds state without GHC and still beats TRM by 1.5 pass@2 (Sec. 6.4, Table 7). We will run the 2x2 (n = 8 or 40, static or dynamic GHC) for the final version.

Q2 (fixed point). From budget 4 to 16 TRM moves by at most 0.5 points at any k (Fig. 1, Table 5). Its halting also settles at 1.31 of 16 steps (Table 3). We will add per-step state-change curves, split by solved and unsolved tasks, in the final version.

Q2 (pass@1000). That 0.25 is one task of 400 and matches repeat-evaluation noise (App. D). Our claim is at pass@100 (Table 5): the wide model goes 62.25, 67.00, 67.50, 67.75, TRM stalls at 59.0. We will make the Sec. 6.3 sentence say pass@100.

Q2 (early vs. late). Both, see Table 5. One wide step (45.12 pass@1) beats every narrow budget. Budget 1 to 16 adds 5.50 at pass@100 and 4.75 at pass@1000 for the wide model, 3.62 and 2.62 for TRM. Top-1 settles early for both models and later steps add to the pool, as Sec. 6.3 says.

Q3 (seeds). Our two separately trained ARC-AGI-1 TRM baselines differ by 0.6 pass@2 (Table 1). VWN+TRM is 7.1 above the better one. It leads at every k on ARC and on Sudoku (Tables 1, 2). We will add more seeds and task-level paired bootstrap intervals in the final version.

Q3 (baselines). Our ARC-AGI-2 baseline is the deep H6/L9 config, not the upstream H3/L4 (Sec. 5.1). At H3/L4 the same pipeline reproduces the published ARC-AGI-1 number, 44.6 pass@2 (Table 1, Sec. 6.1). VWN+TRM also beats published TRM: 9.03 (Table 1) vs. 7.8, and 80.70 (Table 2) vs. 74.7 on Sudoku.

Q4 (cost). One wide step beats 16 narrow steps at every k (Table 5), i.e. 120 vs. 1920 blocks. Attention and FFN still run at D = 512 with the same sequence length (Sec. 4, Cost). We will profile latency and peak memory at several budgets on the same hardware in the final version.

Q4 (single checkpoint). Table 5 at budget 16 is single-checkpoint: 52.50 vs. 46.75 pass@2. So the ARC-AGI-1 gain does not come from ten-checkpoint voting. We will add single-checkpoint numbers for ARC-AGI-2 in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.