acceptodds
Under review as a conference paper at ICLR 2027

Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning

25 views

Abstract

Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computation by dynamically deciding when to stop looping. Motivated by the recent success of looped transformers in language modeling and reasoning, we investigate whether dynamic looping can similarly benefit sequential decision-making. We provide a complexity-theoretic motivation for this approach by showing that there exist Markov decision processes in which a state-adaptive policy achieves the optimal return with asymptotically less expected computation than any optimal fixed-runtime policy. To learn compute-adaptive policies in practice, we introduce Looped Actor, a transformer-based policy that repeatedly refines a latent representation toward a fixed point using a shared computational block. This allows the model to allocate computation adaptively by varying the number of loops based on the current state. We evaluate Looped Actor on 22 tasks across six environments, ranging from combinatorial puzzles to robotic manipulation and spanning online and offline reinforcement learning (RL) with discrete and continuous actions. Looped Actor matches or exceeds the performance of an untied baseline with 16 more parameters, with the largest gains in environments where action selection requires substantial multistep planning. For the Boxoban environment, we find that the computation allocation is structured: the number of loops increases with the number of remaining pushes and future optimal pushes become increasingly predictable from the latent state over successive loops. Together, these results highlight actor looping as a simple and efficient way to equip RL agents with adaptive computation and improve their planning capabilities.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reasoning models, looped transformer, multistep planning, adaptive computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_E8suOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Looped Actor, a reinforcement learning policy that uses a shared Transformer block for iterative reasoning. Instead of running a fixed number of layers for every state, the model decides when to stop based on the KL divergence between consecutive action distributions. The idea is to spend more computation on difficult decisions and less on easy ones, without increasing the number of parameters.

The authors also provide a theoretical result showing that adaptive computation can be more efficient than fixed computation in certain MDPs. Experiments are conducted on 22 tasks across six environments, covering both online and offline RL. Looped Actor performs particularly well on planning-heavy tasks such as Boxoban and Rush Hour, while achieving comparable performance to a much larger Transformer on OGBench.

Critical strengths
  1. Strong results on tasks that require multi-step planning.The Boxoban results are quite impressive. In Figure 1, Looped Actor reaches around 80% success rate, compared to roughly 40% for the much larger Iso-FLOPs baseline. The improvement is also clear on Rush Hour. What makes this result interesting is that Looped Actor uses only about 0.5M parameters, compared to 8M for Iso-FLOPs. This suggests that reusing the same parameters for additional computation can be more useful than simply stacking more layers, at least for these planning tasks. I think this is the strongest result in the paper.

  2. The authors make a good effort to understand what happens during looping. I particularly like the analysis in Figures 3 and 4. The probing results show that future optimal box pushes become easier to decode as the model performs more loops. The number of loops also increases with the remaining planning difficulty in successfully solved episodes. These results are interesting because they give some insight into what the additional computation is doing, rather than just showing another performance improvement. The fixed-runtime comparison in Figure 6 is also useful, as it shows that adaptive stopping can save computation without noticeably hurting performance.

Critical weaknesses
  1. The methodological novelty is somewhat limited. My main concern is that the proposed method seems to rely heavily on existing ideas. The looped architecture is adapted from FPRM, while the stopping criterion based on the KL divergence between consecutive output distributions has also been explored before. The main contribution appears to be bringing these ideas together in an RL policy. I think this is a reasonable direction, and the experimental results are useful. However, it is not entirely clear what new technical insight is introduced beyond applying an existing looped reasoning architecture to action selection. The theoretical result is interesting, but it is based on a specially constructed family of one-step MDPs. I am not sure how much it tells us about the behavior of Looped Actor in the actual planning environments.

  2. The benefit of adaptive computation could be better established. Figure 6 shows that using a fixed 16 loops gives almost the same performance as adaptive looping. This suggests that the main performance improvement comes from repeated computation, while adaptive stopping mainly helps reduce the computational cost. I do think saving computation is valuable. However, since adaptive computation is one of the main motivations of the paper, I would like to see a more direct comparison of performance and computational cost. For example, how does Looped Actor compare with fixed-loop models under the same average inference budget? How sensitive are the results to the stopping threshold? Another point is that the improvements on OGBench are relatively small compared to those on Boxoban and Rush Hour. The overall success rate is 69.7% versus 68.5% for Iso-FLOPs, and Looped Actor is not better on every individual task. This makes me wonder how broadly the benefits of looping carry over beyond the two planning-heavy environments.

Questions for authors
  1. Can the authors provide a more direct comparison between adaptive and fixed computation under matched inference budgets? Figure 6 is helpful, but I think a performance-versus-compute plot would make the benefit of adaptive stopping much clearer. It would also be useful to report actual inference latency, not just loop counts or FLOPs.

  2. How much of the improvement comes from the looped architecture itself, rather than the specific FPRM design? The comparison with DRC is useful, but DRC is modified to remove its memory across environment steps. Could the authors explain how this affects the comparison? A stronger comparison with other recurrent or adaptive-depth policies would help clarify the contribution.

  3. Do the probing results necessarily indicate multi-step planning? Figure 3 shows that future actions become easier to decode after more loops, which is interesting. However, it is still unclear whether the model is actually constructing a plan or simply learning better features for action prediction. Is there any additional evidence that could help distinguish these explanations?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer E8su Part 1/2

We thank the reviewer for appreciating the merits of our paper and their suggestions for improvement. Below, we address the concerns raised by the reviewer.

  • ''The methodological novelty is somewhat limited.'' We thank the reviewer for this comment and take it as an opportunity to rephrase our core contributions more precisely. While it is true that we leverage the architectural core of FPRM as an instantiation of our looped policy, our main contributions are distinct from that. In particular, we provide (i) a separation result showing that state-adaptive computation can be unboundedly more efficient than fixed runtime at optimal return, a question Ghugare et al. (2026) leave open; (ii) RL-specific design choices, such as halting on the action distribution with a single threshold across discrete and continuous actions, and training without deep supervision, since intermediate-loop policies never interact with the environment; and (iii) the empirical findings connecting reasoning with planning in complex RL environments. We have revised the Introduction and Section 4 to make our contributions more apparent.
  • ''The theoretical result is interesting, but it is based on a specially constructed family of one-step MDPs.'' Theorem 3.1 is a separation result: the one-step construction isolates the effect of adaptivity from credit assignment and horizon, and the same bounds hold per step for multi-step episodes with i.i.d. states. We do not claim that Looped Actor is compute-optimal, but the mechanism behind the theorem, i.e., frequent easy states and rare hard ones, is visible in our experiments (Figures 4 and 5).
  • ''The benefit of adaptive computation could be better established.'' We thank the reviewer for this comment. Although we reported success rates for fixed-runtime looped policies using 4, 8, and 16 loops on Boxoban, we did not include a compute-matched fixed-runtime baseline. To address this, we have updated Figure 6 to include a fixed-runtime policy using 10 loops, closely matching Looped Actor's average of 9.8 loops. This baseline achieves a success rate of 74.9±4.8%74.9 \pm 4.8\%, compared with 81.5±5.7%81.5 \pm 5.7\% for Looped Actor. Following the reviewer's suggestion, we also measured the wall-clock time per decision for the trained Boxoban checkpoints. Looped Actor takes approximately 1.951.95 ms per decision, compared with 2.572.57 ms for the fixed-runtime variant using 16 loops. These results provide further evidence for the benefit of allocating computation adaptively.
  • ''The improvements on OGBench are relatively small'' We acknowledge that Looped Actor does not outperform Iso-FLOPs model on OGBench. The results show, however, that our looped policy can reach parity with a 16$\times$ larger model rather than superiority. The parameter-matched comparison isolates the effect of looping: 69.7% vs. 33.0% for Iso-Parameters.
  • ''How much of the improvement comes from the looped architecture itself, rather than the specific FPRM design?'' We would like to clarify that the Iso-FLOPs baseline uses the same FPRM architecture but with untied weights. The substantial performance gap on Boxoban, where Looped Actor achieves an 81.5%81.5\% success rate compared with 42.8%42.8\% for the untied baseline, provides evidence that the gains stem from looping rather than the FPRM architecture alone. Nevertheless, we acknowledge that FPRM provides a particularly effective architectural foundation. To assess the role of architecture, we therefore include an additional Looped Actor variant based on TRM. Figure 2 compares the FPRM- and TRM-based variants on Rush Hour. While both outperform the Iso-params baseline, the FPRM-based variant and DRC achieve substantially higher success rates than the TRM-based variant.
  • ''The comparison with DRC is useful, but DRC is modified to remove its memory across environment steps. Could the authors explain how this affects the comparison?'' We removed the memory carried across environment steps in the original DRC architecture to ensure a fair comparison with Looped Actor and the Iso baselines, which do not use such memory. Cross-step memory allows recurrent agents to amortize planning across steps (Taufeeque et al., 2024; Anokhin et al., 2026), which is an orthogonal axis. Since Boxoban is fully observable, the stateless variant isolates computation within a single step.
AuthorsOpenReview ↗

Reply to Reviewer E8su Part 2/2

  • ''Do the probing results necessarily indicate multi-step planning?'' We thank the reviewer for raising this key point, and agree that the original analysis could not separate planning from broadly better features. We therefore have expanded the analysis, training two separate probes with different supervision targets: a "solution" probe to predict the move-optimal solution, and a "plan" probe to predict the agent's actual future push. We fix the future box push at j=2j = 2, which lies an average of 5 environment steps ahead of the agent. When the agent's push coincides with the optimal move, the probes behave nearly identically. When the agent's action deviates from the optimal solution, the solution probe barely improves at predicting the optimal push, rising in accuracy from 16% to 26% after 16 loops. Instead, the solution probe increasingly predicts the agent's suboptimal push, rising from 13% to 45%, although it was never trained on the agent's actions. This effect is seen most strongly between j=1j = 1 and j=4j = 4, corresponding to an average of 3-8 environment steps, after which the difference between probe behavior subsides. We thus interpret looping to increasingly develop the agent's own intended plan along a short horizon, in addition to building generic features of the board useful across a longer horizon.

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask the reviewer to update their score accordingly.

Thank you for the detailed response and additional experiments. The compute-matched comparison and wall-clock measurements provide stronger evidence for the benefits of adaptive computation. I also appreciate the expanded probing analysis, which offers more convincing evidence that looping helps the policy develop representations of its own future actions. The clarifications regarding FPRM and DRC are also helpful.

Some concerns about methodological novelty and the generality of the theoretical results remain. Nevertheless, the rebuttal has addressed most of my major concerns and strengthened the empirical support for the paper. I am therefore willing to raise my rating from 2 to 3.

AuthorsOpenReview ↗

Thanking the Reviewer

We thank the reviewer for appreciating our rebuttal and revised version and for raising their score.

Reviewer_ETdjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Looped Actor, a modification of prior looped transformer work (FPRM) that dynamically computes the number of loop counts (based on a convergence threshold) and is used to solve toy benchmarks (all with small action spaces) such as Boxoboan, RushHour, and OGBench. The authors compared with baselines such as Iso-Parameter transformers and Iso-FLOPs baselines and show that Looped Actor improves performance while being efficient. The authors also conducted a few analysis such as using probing to find knowledge of plans; and also analyzing whether loop counts correlates to task difficulty.

Critical strengths
  • The writing is overall clear and easy to follow. The overall idea of Looped Actor is intuitive.

  • The authors included multiple baselines such as Iso-Parameter, Iso-FLOPs, and DRC on three toy-like benchmarks (Boxoban, RushHour, and OGBench) to validate the effectiveness of Looped Actor.

  • The authors conducted a few interesting analysis, such as probing to analyze what plans are leaned by the latent vector; and whether there is correlation between loop counts with task difficulty.

Critical weaknesses
  1. The paper motivates looped transformers through the need for adaptive test-time compute (L38–39, 43–46, 62–63). However, existing reasoning LLMs already support adaptive compute by generating more reasoning tokens for harder questions and fewer for simpler ones. Looped transformers instead scale compute through model depth rather than tokens, but it is unclear/this work has not sufficiently justify why depth-based scaling is preferable than scaling tokens; or whether it can be beneficially combined with existing approaches (i.e., scaling tokens).

  2. I believe the paper lacks several important baselines. Since Looped Actor can be viewed as a test-time scaling method, the authors should compare against (1) an iso-parameter transformer that scales test-time compute through reasoning token generation, similar to traditional reasoning models; and (2) the original FPRM, since the proposed method is largely an adaptation of FPRM (Section 4). These comparisons are important for understanding the benefits of depth-based scaling over token-based scaling, and whether the proposed modifications to FPRM are actually beneficial.

  3. There seems to be no ablation studies in the main paper. While Figure 6 compares adaptive versus fixed looping, the paper lacks ablations on key design choices, particularly the modified halting criterion and training strategy compared to the original FPRM.

Questions for authors
  • Pleases bold the highest numbers in Table 1.

  • In Figure 4, it seems that all failed attempts have significantly higher loop counts than successful ones. What is the implication of this? Why would this happen?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer ETdj

We thank the reviewer for appreciating the merits of our paper and their suggestions for improvement. Below, we address the concerns raised by the reviewer.

  • ''It is unclear/this work has not sufficiently justify why depth-based scaling is preferable than scaling tokens; or whether it can be beneficially combined with existing approaches'' To assess whether latent chain-of-thought reasoning can compete with looped reasoning in our RL setting, we trained a Coconut-style baseline based on the Iso-param model. This baseline generates up to 16 latent reasoning tokens before acting and uses the same halting criterion as Looped Actor. On Boxoban, it yielded no improvement over the Iso-param model. On OGBench Cube double, it improved mean performance from 15.2%15.2\% to 22.7%22.7\% across five seeds, but the standard deviation increased substantially, from 7.67.6 to 17.117.1 percentage points, suggesting greater training instability. Its performance also remained substantially below that of Looped Actor, which achieved 67.7%67.7\% with a standard deviation of 4.64.6 percentage points. We have added these results together with a discussion in the Appendix Section C.2.
  • ''I believe the paper lacks several important baselines.'' Besides the continuous latent chain-of-thought baseline we discussed in the above reply, we added another looped reasoning baseline based on TRM with matched parameters and FLOPs, which Looped Actor outperforms on Rush Hour (Figure 2).
  • ''There seems to be no ablation studies in the main paper. [...] particularly the modified halting criterion and training strategy compared to the original FPRM.'' We thank the reviewer for this comment. We added ablations of both design choices relative to the original FPRM (Section 5.4): halting on the relative residual of the latent state instead of the action distribution, and training with deep supervision instead of backpropagation through the executed loops. We can see that while the different halting choices require different calibrations for the looped policy to converge within the 16 loops, it yields no measurable difference in performance. Using deep supervision, however, adds an additional layer of complexity, while either performing on par or significantly worse than looped actor without deep supervision. In addition, we added a sensitivity analysis of the halting threshold τ\tau (Figure 7) and a fixed-runtime baseline that matches the average computation of Looped Actor (Figure 6).
  • ''Bold the highest numbers in Table 1'' Done, thank you for the suggestion.
  • ''In Figure 4, it seems that all failed attempts have significantly higher loop counts than successful ones. What is the implication of this?'' Thank you for highlighting this observation. Since halting is triggered by convergence of the action distribution, a high loop count indicates that the policy has not yet settled on an action distribution. The gap persists even at the same number of remaining pushes (about 13 vs. 7-8 loops at 4-6 remaining pushes). We interpret this as evidence that loop count reflects the agent's own uncertainty, beyond what this measure of state difficulty captures. Loop count can therefore serve as an intrinsic signal of competence, for example to trigger a fallback such as search. This interpretation is consistent with findings in the looped reasoning literature that models can act as their own verifiers: non-convergent reasoning trajectories tend to correspond to tasks the model cannot solve.

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask the reviewer to update their score accordingly.

Reviewer_ETdjOpenReview ↗

Response

Thank you for the added results. I believe my only remaining main concern is the CoT baseline comparison.

You mentioned in the setup "this baseline generates up to 16 latent reasoning tokens". I have two concerns. (1) Is there a particular reason why 16 is chosen here? For modern reasoning models (assuming with KV cache) it is quite normal and fast for CoT to be on the order to at least a few hundreds and commonly thousands of tokens [1]. In comparison 16 latent reasoning tokens seems trivial. (2) In the revised paper, I did not find explanation of how this variant setup and trained. This makes it hard to access whether it is comparable to modern RLVR training pipeline for reasoning models (e.g., [2-3]), which is basically the my main concern. Can you elaborate?


References

[1] https://openai.com/index/learning-to-reason-with-llms/

[2] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[3] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.

AuthorsOpenReview ↗

Reply to Reviewer ETdj

We thank the reviewer for reading our response and the continued engagement in the rebuttal, we truly appreciate that. We are happy to clarify the two remaining concerns.

  1. We set 16 as an upper limit on the number of latent thought tokens, leaving the model to determine how many it needs. Please note that our CoT variant never reaches this limit, even during training, with its output stabilizing after only 4–6 latent thought tokens on average. This suggests that increasing the limit would not improve performance. This behavior is also consistent with the original Coconut paper [1], which uses only 6 latent tokens to solve tasks such as GSM8K and ProsQA. These latent reasoning sequences are substantially shorter than typical token-level CoT sequences, since Coconut-style reasoning can operate in superposition [2].

  2. Indeed, we adopt RLVR to train our CoT variant, i.e., we never train on the explicit chain-of-thought and only reward the final action output.

We sincerely hope this has clarified all remaining concerns. Thank you again for your engagement in this rebuttal.

[1] S Hao, S Sukhbaatar, DJ Su, X Li, Z Hu, J Weston, Y Tian, "Training large language models to reason in a continuous latent space". COLM 2025.

[2] H. Zhu, S. Hao, Z. Hu, J. Jiao, S. J. Russell, Y. Tian, "Reasoning by superposition: A theoretical perspective on chain of continuous thought". NeurIPS 2025.

Reviewer_2C3vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Inspired by looped reasoning models, particularly the Fixed Point Reasoning Model (FPRM), this paper introduces Looped Actor, which enables adaptive, recurrent planning in policies. Through a computation-theoretic analysis, the authors establish that there exists an MDP in which state-adaptive policies achieve the optimal return with less computation. Empirical evaluations show that Looped Actor achieves comparable or better performance than fixed-runtime policies, demonstrating its computational efficiency.

Critical strengths

The paper brings adaptive computation to agent policies. The derivation of the optimal computation cost is sound, and the approach is intuitive and easy to follow. The empirical evaluation is sound and solid overall.

Critical weaknesses

My main reservation concerns the scope of the methodological contribution. The approach builds closely on FPRM and an established output-stability halting criterion, so its contribution lies primarily in adapting and systematically evaluating these ideas in RL. Nevertheless, the adaptation is well motivated, and the empirical study provides useful evidence of its effectiveness.

Questions for authors

The multistep planning probes suggest that additional loops progressively refine the latent representations. Have you explored running the trained checkpoint beyond the maximum loop count used during training? I would be interested in any observations on depth extrapolation and test-time scaling, as investigated in Ouro [1].

[1] Zhu et al., Scaling Latent Reasoning via Looped Language Models.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer 2C3v

We start by thanking the reviewer for appreciating the merits of our paper and the very interesting question on test-time scaling. Below, we address the concerns raised by the reviewer.

  • ''My main reservation concerns the scope of the methodological contribution.'' We thank the reviewer for this comment and take it as an opportunity to rephrase our core contributions more precisely. While it is true that we leverage the architectural core of FPRM as an instantiation of our looped policy, our main contributions are distinct from that. In particular, we provide (i) a separation result showing that state-adaptive computation can be unboundedly more efficient than fixed runtime at optimal return, a question Ghugare et al.\ (2026) leave open; (ii) RL-specific design choices, such as halting on the action distribution with a single threshold across discrete and continuous actions, and training without deep supervision, since intermediate-loop policies never interact with the environment; and (iii) the empirical findings connecting reasoning with planning in complex RL environments. We have revised the Introduction and Section 4 to make our contributions more apparent.
  • ''Have you explored running the trained checkpoint beyond the maximum loop count used during training?'' Thank you for this suggestion. Since halting is triggered by convergence of the action distribution, increasing the loop cap affects only decisions that have not converged within 16 loops. To assess whether additional computation at test time improves performance, we evaluated Looped Actor on Rush Hour with caps of 24 and 32 loops, exceeding the training cap of 16. We find that this can yield modest improvements of up to 2.0 percentage points. However, these gains occur only for looped policies for which most trajectories converge during training. With a very small halting threshold τ\tau, few trajectories converge during training, and extending the loop cap at test time substantially degrades performance. We have added these results together with a discussion in the Appendix Section C.1. We thank the reviewer for drawing our attention to it.

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask the reviewer to update their score accordingly.

Thank you for your response and clarification. I will keep my positive score.

AuthorsOpenReview ↗

Thanking the Reviewer

We thank the reviewer for your positive assessment of our paper and for appreciating our rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to all the reviewers

We would like to thank all three reviewers for their thorough reading of our article and for describing our approach as ''intuitive and easy to follow'' (2C3v, ETdj) and ''well motivated'' (2C3v). We are also glad the reviewers appreciated our experimental results and analyses, describing the empirical evaluation as ''sound and solid overall'' (2C3v), the Boxoban results as ''quite impressive'' (E8su), and noting that ''the authors make a good effort to understand what happens during looping'' (E8su).

The constructive suggestions allowed us to improve the paper, which we hope the reviewers will find to satisfactorily address the raised concerns. We uploaded a revised version of the paper incorporating the reviewers' suggestions. All changes are highlighted in blue color.

We answer the points raised by each of the reviewers individually in our comments below. Please note that all the references to page numbers, sections, figures, tables, equation numbers and references, refer to those in the revised version. We hope that based on the answers to their questions and the improvements made to the revised version, the reviewers will increase their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.