- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Looped Actor, a reinforcement learning policy that uses a shared Transformer block for iterative reasoning. Instead of running a fixed number of layers for every state, the model decides when to stop based on the KL divergence between consecutive action distributions. The idea is to spend more computation on difficult decisions and less on easy ones, without increasing the number of parameters.
The authors also provide a theoretical result showing that adaptive computation can be more efficient than fixed computation in certain MDPs. Experiments are conducted on 22 tasks across six environments, covering both online and offline RL. Looped Actor performs particularly well on planning-heavy tasks such as Boxoban and Rush Hour, while achieving comparable performance to a much larger Transformer on OGBench.
Critical strengths
-
Strong results on tasks that require multi-step planning.The Boxoban results are quite impressive. In Figure 1, Looped Actor reaches around 80% success rate, compared to roughly 40% for the much larger Iso-FLOPs baseline. The improvement is also clear on Rush Hour. What makes this result interesting is that Looped Actor uses only about 0.5M parameters, compared to 8M for Iso-FLOPs. This suggests that reusing the same parameters for additional computation can be more useful than simply stacking more layers, at least for these planning tasks. I think this is the strongest result in the paper.
-
The authors make a good effort to understand what happens during looping. I particularly like the analysis in Figures 3 and 4. The probing results show that future optimal box pushes become easier to decode as the model performs more loops. The number of loops also increases with the remaining planning difficulty in successfully solved episodes. These results are interesting because they give some insight into what the additional computation is doing, rather than just showing another performance improvement. The fixed-runtime comparison in Figure 6 is also useful, as it shows that adaptive stopping can save computation without noticeably hurting performance.
Critical weaknesses
-
The methodological novelty is somewhat limited. My main concern is that the proposed method seems to rely heavily on existing ideas. The looped architecture is adapted from FPRM, while the stopping criterion based on the KL divergence between consecutive output distributions has also been explored before. The main contribution appears to be bringing these ideas together in an RL policy. I think this is a reasonable direction, and the experimental results are useful. However, it is not entirely clear what new technical insight is introduced beyond applying an existing looped reasoning architecture to action selection. The theoretical result is interesting, but it is based on a specially constructed family of one-step MDPs. I am not sure how much it tells us about the behavior of Looped Actor in the actual planning environments.
-
The benefit of adaptive computation could be better established. Figure 6 shows that using a fixed 16 loops gives almost the same performance as adaptive looping. This suggests that the main performance improvement comes from repeated computation, while adaptive stopping mainly helps reduce the computational cost. I do think saving computation is valuable. However, since adaptive computation is one of the main motivations of the paper, I would like to see a more direct comparison of performance and computational cost. For example, how does Looped Actor compare with fixed-loop models under the same average inference budget? How sensitive are the results to the stopping threshold? Another point is that the improvements on OGBench are relatively small compared to those on Boxoban and Rush Hour. The overall success rate is 69.7% versus 68.5% for Iso-FLOPs, and Looped Actor is not better on every individual task. This makes me wonder how broadly the benefits of looping carry over beyond the two planning-heavy environments.
Questions for authors
-
Can the authors provide a more direct comparison between adaptive and fixed computation under matched inference budgets? Figure 6 is helpful, but I think a performance-versus-compute plot would make the benefit of adaptive stopping much clearer. It would also be useful to report actual inference latency, not just loop counts or FLOPs.
-
How much of the improvement comes from the looped architecture itself, rather than the specific FPRM design? The comparison with DRC is useful, but DRC is modified to remove its memory across environment steps. Could the authors explain how this affects the comparison? A stronger comparison with other recurrent or adaptive-depth policies would help clarify the contribution.
-
Do the probing results necessarily indicate multi-step planning? Figure 3 shows that future actions become easier to decode after more loops, which is interesting. However, it is still unclear whether the model is actually constructing a plan or simply learning better features for action prediction. Is there any additional evidence that could help distinguish these explanations?
4 comments
Reply to Reviewer E8su Part 2/2
- ''Do the probing results necessarily indicate multi-step planning?'' We thank the reviewer for raising this key point, and agree that the original analysis could not separate planning from broadly better features. We therefore have expanded the analysis, training two separate probes with different supervision targets: a "solution" probe to predict the move-optimal solution, and a "plan" probe to predict the agent's actual future push. We fix the future box push at , which lies an average of 5 environment steps ahead of the agent. When the agent's push coincides with the optimal move, the probes behave nearly identically. When the agent's action deviates from the optimal solution, the solution probe barely improves at predicting the optimal push, rising in accuracy from 16% to 26% after 16 loops. Instead, the solution probe increasingly predicts the agent's suboptimal push, rising from 13% to 45%, although it was never trained on the agent's actions. This effect is seen most strongly between and , corresponding to an average of 3-8 environment steps, after which the difference between probe behavior subsides. We thus interpret looping to increasingly develop the agent's own intended plan along a short horizon, in addition to building generic features of the board useful across a longer horizon.
We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask the reviewer to update their score accordingly.
Thank you for the detailed response and additional experiments. The compute-matched comparison and wall-clock measurements provide stronger evidence for the benefits of adaptive computation. I also appreciate the expanded probing analysis, which offers more convincing evidence that looping helps the policy develop representations of its own future actions. The clarifications regarding FPRM and DRC are also helpful.
Some concerns about methodological novelty and the generality of the theoretical results remain. Nevertheless, the rebuttal has addressed most of my major concerns and strengthened the empirical support for the paper. I am therefore willing to raise my rating from 2 to 3.
Thanking the Reviewer
We thank the reviewer for appreciating our rebuttal and revised version and for raising their score.
Reply to Reviewer E8su Part 1/2
We thank the reviewer for appreciating the merits of our paper and their suggestions for improvement. Below, we address the concerns raised by the reviewer.