acceptodds
Under review as a conference paper at ICLR 2027

Recursive Successor Models

14 views

Abstract

Planning with world models enables improvement beyond the dataset, but compounding prediction errors hinder long-horizon planning. To prevent one-step compounding error, we instead model longer-term future state distributions, known as successor measures. We introduce Recursive Successor Models, which combine (1) a stable supervised training objective that avoids bootstrapping with (2) a recursive sampling procedure for planning that composes short-horizon distributions into long-horizon distributions. We demonstrate the capabilities of RSM in simulated benchmarks and real-world robot experiments by showcasing improvements in policy performance, planning latency, and training convergence speed across long-horizon and visual tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Robotics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WyuYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a new offline reinforcement learning algorithm that can learn a policy that is capable of composing trajectories/skills in the offline dataset. Specifically, the authors propose to learn successor measure from offline dataset using supervised learning by sampling future states from the dataset trajectory. With this recursive successor model (RSM) and a learned policy using say imitation learning, the proposed method is capable of selecting best actions based on imagined future states and their rewards.

Critical strengths

The paper is generally well written and easy to follow.

The method is also motivated well and the intuition is reasonable.

The proposed method looks new to me.

Experiments on both simulated environments and real robots demonstrate the effectiveness of the proposed method.

Critical weaknesses

Learning compositions of skills/subtrajectories from offline datasets does not seem new. Many existing methods like diffusion policy (and their improvement) are already capable of learning this purely offline.

My concern is that the proposed method needs to sample many actions and future states during test/inference. If the environment/MDP is highly stochastic, i.e., the transition function has a high randomness over next state (say 50-50 for two drastically different next states), its estimated Q-values can be noisy and executing its chosen action in this stochastic environment can produce very different outcomes, and therefore has highly variable episode returns. It seems that the simulated environments used are mostly very deterministic.

Questions for authors

How is the proposed method related to this related work on predicting distributions of future states/rewards?

Wiltzer H, Farebrother J, Gretton A, Tang Y, Barreto A, Dabney W, Bellemare MG, Rowland M. A distributional analogue to the successor representation. ICML. 2024

2 comments

Thank you for your thoughtful review and questions. We appreciate the acknowledgment of the quality of writing and motivation, novelty of our method, and our effective experimental validation.

Learning compositions of skills/subtrajectories from offline datasets does not seem new. Many existing methods like diffusion policy (and their improvement) are already capable of learning this purely offline.

It is true that randomly composing offline dataset behaviors can be accomplished by simple imitation (e.g. Diffusion Policy), and many methods are targeted to accomplish improving this composition, each with their own capabilities and drawbacks.

To clarify, we do not claim to be the first to develop offline composition of behaviors (i.e. stitching) for improvement. Rather, our contribution is a new method to accomplish policy improvement with an eye towards long-horizon tasks where TD-based and WM-based methods can break down. Our novelty lies in achieving composition without using temporal-difference learning or excessive 1-step forward predictions, paving the way for better long-horizon planning methods (as validated by our experimental suites).

My concern is that the proposed method needs to sample many actions and future states during test/inference. If the environment/MDP is highly stochastic, i.e., the transition function has a high randomness over next state (say 50-50 for two drastically different next states), its estimated Q-values can be noisy and executing its chosen action in this stochastic environment can produce very different outcomes, and therefore has highly variable episode returns. It seems that the simulated environments used are mostly very deterministic.

Very stochastic environments indeed makes value estimation more difficult, including with RSM. We would like to highlight that while the simulated environments in our work are mostly deterministic, our real robotic experiments show RSM’s promise in handling real-world stochasticity in practical setups.

We also understand your concern regarding the inference cost of sampling many times to handle stochasticity. However, our model’s sampling is parallelizable and thus scales well in terms of latency (as shown in the table below). On the non-parallelizable axis—the number of consecutive recursions—RSM scales more favorably than standard world models by simply requiring fewer recursions for a given horizon.

Sample Count (N)Relative Latency (w.r.t N=1)
11.00
161.33
641.91
2562.87

How is the proposed method related to this related work on predicting distributions of future states/rewards? Wiltzer H, Farebrother J, Gretton A, Tang Y, Barreto A, Dabney W, Bellemare MG, Rowland M. A distributional analogue to the successor representation. ICML. 2024

Thank you for introducing us to this work, it is very related to RSM and we have added it to the related work section on successor measures in the new revision.

Distributional Successor Measure focuses on learning the occupancy distribution of a target policy for robust zero-shot policy evaluation. The authors parameterize the model with a weighted ensemble of gamma models (instantiated with GANs), trained with distributional TD Bellman backup. RSM-ZS, the RSM application most closely related to this work, instead focuses on learning the occupancy distribution of an offline dataset, then performing zero-shot policy improvement for arbitrary downstream rewards through planning. We parameterize the model using a flow-matching model trained with Monte Carlo supervised learning and composed recursively. As we have shown throughout the paper, this is more stable and effective than training with TD.

I would like to thank authors' response and explanation. I encourage the author improve the manuscript with these additional explanation, which would strengthen the paper. I will keep my positive rating.

Reviewer_csLgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Recursive Successor Models (RSM) for offline long-horizon robotic planning. RSM decouples training and recursive composition: it uses Monte-Carlo supervision during training to avoid bootstrap instability of prior successor models, and recursively combines short-horizon successor distributions at inference for long-horizon prediction. The authors validate RSM on simulation benchmarks and real Franka robot manipulation tasks, showing improved policy performance and better convergence than world-model and TD-based successor baselines.

Critical strengths
  1. The core decoupling idea is neat and targets a known pain point: TD bootstrap causes unstable training for successor models, while pure MC methods lose stitching ability. Moving the recursive composition to inference nicely balances both.
  2. Empirical validation is solid, including visual offline RL benchmarks and real-world robot hardware experiments, with proper ablation studies on key hyperparameters.
  3. The framework supports multiple downstream use cases, including standard planning, subgoal generation and zero-shot reward adaptation.
Critical weaknesses
  1. The method relies on an external proposal policy to generate candidate actions, so its final performance is bounded by the quality of this proposal policy. It only performs one‑step best‑of‑N selection and cannot iteratively improve the policy. This reveals a critical implicit assumption: the proposal policy must already cover reasonable near‑optimal action candidates. If the offline dataset is heavily sub‑optimal and the proposal policy fails to produce promising candidates, RSM cannot recover by itself. Restricting policy improvement to single‑step ranking also means RSM cannot perform multi‑step policy iteration; its upper performance ceiling is inherently constrained by the behavioral cloning prior rather than being able to discover genuinely novel optimal behaviors.
  2. Recursive composition still accumulates error with more hops, and the method is sensitive to hyperparameters such as β and sampling count, which requires non‑trivial grid search. The theoretical derivation guarantees unbiased estimation under ideal model conditions, yet this ideal assumption breaks down under model approximation errors in practice. There is no principled analytical rule to select β; all hyperparameters are tuned via validation‑set grid search. Such heavy grid‑search dependency raises reproducibility concerns: performance may degrade substantially when transferred to unseen environments without exhaustive hyper‑parameter sweeps. Additionally, ablation results only demonstrate performance trends on one specific task, and the paper lacks cross‑environment robustness analysis for these key hyper‑parameters.
Questions for authors

The theoretical derivation shows recursive β‑composition yields unbiased successor‑measure estimates assuming perfect generative models. However, neural successor models inevitably suffer from approximation error in practice. Could the authors elaborate on how model approximation error propagates through successive recursive hops, and whether there exists any theoretical or empirical bound to characterize such error accumulation? How should practitioners balance the trade‑off between choosing a smaller β (requiring more recursive hops) versus a larger β (fewer hops but harder generative modelling objective) under non‑ideal, real‑world model approximation?How severely does the performance deteriorate when the offline dataset is heavily suboptimal, and what are the failure modes in such regimes?Have you explored practical strategies to mitigate the high inference compute overhead incurred by recursive sampling chains?

3 comments
AuthorsOpenReview ↗

Response, part 1 - one-step improvement and suboptimal data

We appreciate your thorough and constructive review and questions. Thank you for highlighting our core idea, the solid empirical validation and the versatility of our framework.

The method relies on an external proposal policy to generate candidate actions, so its final performance is bounded by the quality of this proposal policy. It only performs one‑step best‑of‑N selection and cannot iteratively improve the policy. This reveals a critical implicit assumption: the proposal policy must already cover reasonable near‑optimal action candidates. If the offline dataset is heavily sub‑optimal and the proposal policy fails to produce promising candidates, RSM cannot recover by itself. Restricting policy improvement to single‑step ranking also means RSM cannot perform multi‑step policy iteration; its upper performance ceiling is inherently constrained by the behavioral cloning prior rather than being able to discover genuinely novel optimal behaviors.

One-step improvement is competitive in many realistic suboptimal datasets. We agree that one-step improvement can constrain the performance RSM can obtain when learning from offline datasets. Nevertheless, several previous works report that in realistic suboptimal datasets, one-step improvement actually yields similar performance to multi-step improvement, suggesting one-step improvement is enough in many cases [1, 2, 3]. Our empirical results on RISE, which contains very suboptimal data such as failures and undirected play, also support this claim. We show significant improvement over the proposal policy as well as over IDQL which performs multi-step policy improvement. Extending RSM to multi-step policy improvement is an interesting avenue for future work.

[1] Brandfonbrener, David, et al. "Offline rl without off-policy evaluation." Advances in neural information processing systems 34 (2021): 4933-4946.

[2] Eysenbach, Benjamin, et al. "A connection between one-step RL and critic regularization in reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[3] Goo, Wonjoon, and Scott Niekum. "You only evaluate once: a simple baseline algorithm for offline rl." Conference on Robot Learning. PMLR, 2022.

How severely does the performance deteriorate when the offline dataset is heavily suboptimal, and what are the failure modes in such regimes?

Performance and failure modes with heavily suboptimal datasets. We expect RSM in its current formulation to be less effective when trained on heavily suboptimal datasets, as we mentioned in the “Limitations” paragraph in the Discussion Section. Specifically, our current method assumes suboptimal trajectories have useful sub-trajectories, that can be composed to achieve higher returns than the original trajectories (see illustration in Figure 1). For example, a pick-and-place trajectory where the robot reaches to the object but fails to grasp would be considered a failure, despite the reach being a useful sub-trajectory.

Heavily suboptimal data is not common. We observe that suboptimal data, particularly in robotics, often comes from failed human demonstrations or failed on-policy robot rollouts (i.e. from classical planners or learned policies) [4, 5], and this data matches our assumption. Additionally our RISE data was generated by humans, further validating that our method applies to realistic suboptimal data settings.

[4] Oopsie Data Workshop. https://workshop.oopsie-data.com/index.html

[5] Huang, Kevin, et al. “Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning”. Arxiv, 2025.

AuthorsOpenReview ↗

Response - part 2, hyperparameter generalization, sensitivity, and scaling analysis, tuning cost

Recursive composition still accumulates error with more hops, and the method is sensitive to hyperparameters such as β\beta and sampling count, which requires non-trivial grid search.

Recursive composition accumulates error whether during training (TD learning) or at test time (world models, RSM). RSM's advantage over single-step TD learning and world modeling is that it requires significantly fewer compositions for a given prediction horizon.

To study sensitivity to β\beta, we evaluated fixing β\beta across tasks versus tuning it per-task on OCBench (Fig. 10b) and RISE (new Table 17). Both benchmarks show robustness across a wide range of β\beta values, reducing tuning cost. RSM also improves with more action candidates and chains per action, scaling more favorably than competing methods (Figs. 4b, 10a, 10c). Thus, sampling count can simply be chosen based on the latency budget.

RISE Table 1

β\betaSuccess over 7 tasksAverage hops
0.9026.1 ± 1.511.0
0.9326.6 ± 1.47.5
0.9527.1 ± 1.45.2
0.9727.3 ± 1.43.1
0.9828.3 ± 1.52.0
0.99 (single hop)25.0 ± 1.31.0

Such heavy grid-search dependency raises reproducibility concerns: performance may degrade substantially when transferred to unseen environments without exhaustive hyper-parameter sweeps.

To test whether β\beta tuning transfers to unseen environments, we evaluate all splits of the 7 RISE tasks into kk tuning tasks and 7−k7-k test tasks. For each kk, we average test success across all splits. Performance remains comparable to per-task tuning, indicating that β\beta generalizes reasonably well to unseen tasks (Table 17).

RISE Table 2

Num tuning tasksNum test tasksRSM
6131.2
5231.0
4330.6
3429.8
2529.6
1629.2
tuned on each task-32.4

There is no principled analytical rule to select β\beta; all hyperparameters are tuned via validation-set grid search.

How should practitioners balance the trade-off between choosing a smaller β\beta (requiring more recursive hops) versus a larger β\beta (fewer hops but harder generative modelling objective) under non-ideal, real-world model approximation?

Although we used grid search, we now propose a cheaper, one-directional search procedure.

Trade-off. Decreasing β\beta shortens each hop, reducing long-range prediction error, but requires more hops, increasing compounding error and inference latency. We therefore prefer the largest β\beta that provides sufficient stitching.

Procedure. We track policy improvement over behavior cloning, Δ(β)=SR(β)−SRBC\Delta(\beta) = \mathrm{SR}(\beta) - \mathrm{SR}_{\mathrm{BC}}. Starting at the highest β\beta, we decrease it until Δ(β)\Delta(\beta) no longer improves, then select the preceding β\beta. Since RSM is trained conditional on β\beta, no retraining is required. On the OCBench Chamber task family, this selects the best β\beta in our sweep. Details are in Appendix H.2.1.

Additionally, ablation results only demonstrate performance trends on one specific task, and the paper lacks cross-environment robustness analysis for these key hyper-parameters.

We expanded our ablation and test-time scaling analysis to all 4 OCBench Chamber tasks (Figs. 4, 10). RSM scales more favorably than all baselines with increasing action candidates (Fig. 10a), Monte Carlo chains (Fig. 10c), and inference latency (Fig. 4b). Cross-environment robustness to β\beta is addressed above.

RSM shows better scaling properties than all baselines, with performance continuing to improve with larger number of action candidates (Fig. 10a), larger number of Monte Carlo chains (Fig. 10c) and inference latency budget (Fig. 4b). Cross‑environment robustness analysis for the key hyperparameter β\beta is detailed in the above responses.

AuthorsOpenReview ↗

Response, part 3 - theoretical analysis of compounding error, mitigating compute overhead

The theoretical derivation guarantees unbiased estimation under ideal model conditions, yet this ideal assumption breaks down under model approximation errors in practice.

Could the authors elaborate on how model approximation error propagates through successive recursive hops, and whether there exists any theoretical or empirical bound to characterize such error accumulation?

We analyze how prediction error compounds when recursively sampling (hopping) from a learned β\beta-discounted successor model d^β\hat d_\beta to approximate the true γ\gamma-discounted successor distribution dγd_\gamma, where 0<β≤γ<10<\beta\le\gamma<1. Since both are distributions over future states, we measure their difference using total variation, TV(μ,ν)=12∫∣μ(x)−ν(x)∣ dx\mathrm{TV}(\mu,\nu)=\tfrac12\int |\mu(x)-\nu(x)|\,dx. Let d^γ\hat d_\gamma denote the approximate distribution produced by this recursive sampling procedure. We show that the total variation distance between these distributions is bounded by

TV(d^γ(sf∣s0),dγ(sf∣s0))≤εβE[N]=εβγ(1−β)β(1−γ).\mathrm{TV}\big(\hat d_\gamma(s_f \mid s_0), d_\gamma(s_f \mid s_0)\big) \le \varepsilon_\beta \mathbb{E}[N] = \varepsilon_\beta \frac{\gamma(1-\beta)}{\beta(1-\gamma)}.

Here, εβ\varepsilon_\beta bounds the prediction error of a single hop (a hop is a call to our trained β\beta successor model generating a successor state), and E[N]\mathbb{E}[N] is the expected number of hops needed to sample from the γ\gamma-discounted successor distribution. The proof uses the triangle inequality to show that each hop adds at most εβ\varepsilon_\beta error (Eq. 19, Appendix E), then sums this bound over the number of hops. The resulting bound decreases as β\beta increases, reflecting reduced compounding error from fewer recursive predictions. We additionally construct an example MDP showing that the bound is asymptotically tight as εβ→0\varepsilon_\beta\to 0. The full proof is provided in Appendix E.

Have you explored practical strategies to mitigate the high inference compute overhead incurred by recursive sampling chains?

A practical strategy is to use the same set of random seeds across action candidates, so their values are compared under shared sampling randomness. This can reduce the variance of estimated value differences, enabling reliable action selection with fewer imagined rollouts per action candidate. This is an established variance-reduction technique in Monte-Carlo methods, also known as common random numbers (CRN).

Also, both the flow matching policy and flow matching successor model can have compute significantly reduced by caching visual tokens during the first velocity field forward pass, to be used for all remaining N−1N-1 forward passes in flow matching sample generation (since visual conditioning in the conditional velocity field does not change for subsequent passes). Our real world hardware experiments demonstrate that the RSM is practical for online planning given real world latency requirements.

Reviewer_R97oOpenReview ↗
Rating
4
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Recursive Successor Models, a novel Reinforcement Learning method for overcoming compounding prediction errors in long-horizong planning. As in Successor Measures based approaches, RSM aims to predict future state-distributions of agent's actions rather than predicting the exact sequence of states. In contrast to traditional temporal difference methods, RSM does not bootstrap its learning process using its own predictions but instead only learns short-horizon predictions of short term futures, thus using only actual observations as supervision. Long-horizon composition only happens at inference time through a recursive sampling procedure. This combats instability during training and works because longer-horizon hops are being sampled rather than singular steps. The authors present successful results of the method on several benchmarks.

Critical strengths

The paper presents a novel idea and method to stabilize training objectives and separate long-horizon composition of states for planning to test time. The idea is novel and unique and has potential to inspire new research and impact the way the field thinks about model-based long-term value estimation and planning.

Critical weaknesses

Personally, I do not have background in this subfield to judge whether the evaluation is thorough enough, tests on the relevant recent benchmarks and compares to competitive state of the art. I will leave this judgement to other reviewers.

Questions for authors
  • Why is long horizon planning evaluated on OCBench but subgoal generation evaluated on OGBench?

Other, small remarks:

  • Figure 1 caption should mention what non-filled circles represent
  • Figure 4 caption should mention the benchmark that is being evaluated
1 comment

Thank you for your kind review acknowledging the novelty of our approach. We are glad to hear you believe it has the potential to inspire new research and help shape how the field approaches long-horizon planning.

Why is long horizon planning evaluated on OCBench but subgoal generation evaluated on OGBench?

OCBench task horizons are much longer than OGBench tasks. Within OCBench we select the longest-horizon tasks requiring several thousands of actions to complete, while the longest manipulation task in OGBench requires less than one thousand. Another difference between the two benchmarks is that OGBench tasks are goal-conditioned while OCBench tasks are not, making OGBench a better fit for testing hierarchical subgoal generation approaches.

Figure 1 caption should mention what non-filled circles represent. Figure 4 caption should mention the benchmark that is being evaluated

We edited Figure 1 itself and the caption to ensure it is clearer for readers. In particular, we decided to depict a single representative sample from each planning method, and to highlight the number of black arrows (mentioned in the caption now) to show how compositionality and model calls interact with each other across the 3 different paradigms shown. Originally, we wanted to show the aggregated samples across many rollouts, and the hollow circles had represented “intermediate hops” from RSM before it is counted in the gamma-discounted distribution. We removed this to prevent confusion and to highlight the point of the figure (i.e., how RSM is in between SMs and WMs when it comes to recursion and composition).

We have updated the Figure 4 caption to include the name of the benchmark (OCBench).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.