acceptodds
Under review as a conference paper at ICLR 2027

Random Recursive Models

Abstract

Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. However, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of learned layers and performs recursive steps by sampling one layer independently with replacement for each example and step. This enables flexible layer reuse while retaining the parameter efficiency of recurrence. We evaluate RRM on challenging reasoning tasks, where it matches or exceeds the baselines, often with 50–75% fewer parameters. RRM can vary its depth at inference, including beyond that seen during training, without retraining or adding parameters, improving tasks that benefit from deeper iterative computation. RRM also supports Monte Carlo inference and probabilistic test-time scaling, both of which improve performance without retraining. These insights may open new directions in neural network architecture design.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
recursive models, recurrent depth, parameter sharing, iterative reasoning, test-time scaling, random networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QsbqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Random Recursive Models (RRM). RRM has a pool of L layers, and at each step, samples one layer from this pool to use, up to T samples. This is like recursive depth / looped language models but with random parameter sharing. The authors provide experiments on a variety of tasks and benchmarks, showing that RRM outperforms comparable baselines.

Critical strengths
  1. RRM outperforms comparable baselines, demonstrating the approach works well
  2. The authors provide results on a variety of tasks
Critical weaknesses

The primary weakness is that the evaluation suite is somewhat limited. Although the authors evaluate on a number of datasets with a variety of tasks, the scale is still relatively small. For example, the image classification results are restricted to CIFAR-10, and logic puzzles are used for language modeling (my understanding based on the title of subsection 5.2).

If the authors could expand the experimental section and show comparable results for RRM at greater scale, it would greatly strengthen the work. My suggestions would be to i) expand the image classification results to ImageNet-1k and ii) show results for scaled up language modeling. The language modeling experimental suite can be minimal. I would be very curious how RRM does on language modeling (perplexity plots), especially since their approach does provide the flexibility of more parameters relative to a 1-layer looped tranformer. If the authors could further evaluate the RRM pretrained on language modeling on other benchmarks (like question answering or math word problems), that would be quite interesting too.

Questions for authors
  1. Why do the authors think this approach works? I find it surprising that randomly sampling layers works so well.
  2. How similar are the learned weights of each layer?
  3. Given a trained RRM, is there an optimal ordering of the layers, or does uniformly sampling at inference work the best too?

If the answer is in the paper and I missed it, please point out where.

Reviewer_asDoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RRM learns a small pool of layers and applies a uniformly random one at each step, so depth can grow without new parameters and can be changed at inference. Across CIFAR-10, Sudoku/Zebra, Maze-Hard, Sudoku-Extreme, and ARC-AGI-1, it roughly matches dense models with fewer parameters and is competitive with recursive models. Swapping TRM's fixed layer order for random reuse raises Sudoku-Extreme from 87.63% to 89.17% in Table 4 from the paper. Extra inference depth helps Maze-Hard and Sudoku-Extreme but not some other tasks like CIFAR-10, Zebra, or ARC.

Critical strengths
  1. The paper provides a simple idea tested broadly, with limits shown. Results span vision, logic, and puzzle reasoning, and the paper reports where extra depth plateaus or hurts.

  2. It offers a clean controlled result. TRM+RRM changes only the layer order and gains 1.5 points, which is stronger evidence than the parameter-count comparisons.

Critical weaknesses
  1. Randomness and efficiency are not isolated. The CIFAR RRM beats the dense model only with 16 trajectories (128 vs. 8 block applications), Sudoku/Zebra RRM uses 16 vs. 8 applications, and the one-layer control changes capacity. A fixed cyclic schedule with the same layers, applications, and budget is needed, along with measured latency, memory, and training cost.

  2. The paper shows small gains lack robustness evidence. The gains are 0.077 points (CIFAR) and 0.19 points (ARC), and the error bars come from three evaluation seeds of one checkpoint, not from training seeds. Maze-Hard and Sudoku-Extreme also use the test split for periodic checkpoint evaluation in A.2, with no stated selection rule.

Questions for authors
  1. How does fixed cyclic reuse compare at matched layers, applications, and budget? Does the TRM+RRM gain hold across training seeds?
  2. How were checkpoints and depths chosen for Maze-Hard and Sudoku-Extreme?
  3. What are the measured training cost and latency for RRM, Monte Carlo RRM, and fixed-order baselines?
  4. Does the 44.79% ARC checkpoint keep its accuracy at the smaller budget? I find that Table 8's 16× reduction uses the 43.79% one.
Reviewer_oaULOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Random Recursive Model structure with the chief instantiations being transformer models. The primary distinguishing feature of these models being the fact that instead of sequential repeated application as is done in common recursive models, these models randomly sample from a pretrained pool of K layers. The recursion is performed for T layers. This decoupling of the number of trained layers and computational steps results in the ability of increased computation while still retaining relatively small parameter size.

The primary testing methodology here is to replace the backbone of existing models with thus random recursive setup to compare performance across the different backbones. The test is done on various data benchmarks such as CIFAR-10, Zebra as well as heavy reasoning tasks such as ARC-AGI and Maze-Hard. In tasks such as CIFAR-10, sudoku and zebra, the method was able to achieve comparable or better performance with 50-75% fewer parameters. Increasing test time depth which is a chief feature of the work, increases performance on maze-hard and sudoku extreme but is the same or degrades in CIFAR and Zebra Extreme and ARC AGI.

Critical strengths
  1. The model is simple and easy to implement, can be seamlessly inserted into existing model architectures.
  2. There is a clean and significant reduction in the number of parameters (50-75%) while maintaining the same and or better performance. That is a practical advantage that is clearly visible.
  3. Increase test time scaling is possible without the need to increase model size. Another advantage although the benefits of this point are not as consistent across the benchmarks.
  4. Further increase in performance on the sudoku extreme benchmarks are obtained by incorporating the Probabilistic test-time scaling approach.
Critical weaknesses
  1. The main claim of the paper seems to be that random order is what allows us to achieve the increase in performance. In benchmarks that are used, should there also not be a baseline where the sequential recursion is used, while other parameters are the same as the RRM model? That will isolate the effect of randomization of recursion more cleanly.
  2. In the case of table 1, while it is true that there is parameter decrease, while still maintaining comparable performance, we see that the closest performance number is achieved with T=16. Does this not indicate that significantly more compute is needed to get comparable performance? While this does not take away from the parameter savings, some clarification of this would be good.
  3. The evaluations are stated to have been done over three seeds. This seems to be quite a low number and may not be statistically rigorous. Also some baseline results such as HRM in table 3 and Dense in table 2 seem to have only one run evaluated.
Questions for authors
  1. Can you clarify what is meant by "RRM results are reported as mean ± sample standard deviation over three evaluation seeds for a fixed checkpoint.", what is the checkpoint being referred to here?
  2. Are there any tests done on your part on how the increase in L affects performance? Is it similar to more standard recursive schemes.
  3. As a more conceptual question, since this layers are being sampled randomly, do you have any insight on the fact that they might converge to a very similar functions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.