- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a new offline reinforcement learning algorithm that can learn a policy that is capable of composing trajectories/skills in the offline dataset. Specifically, the authors propose to learn successor measure from offline dataset using supervised learning by sampling future states from the dataset trajectory. With this recursive successor model (RSM) and a learned policy using say imitation learning, the proposed method is capable of selecting best actions based on imagined future states and their rewards.
Critical strengths
The paper is generally well written and easy to follow.
The method is also motivated well and the intuition is reasonable.
The proposed method looks new to me.
Experiments on both simulated environments and real robots demonstrate the effectiveness of the proposed method.
Critical weaknesses
Learning compositions of skills/subtrajectories from offline datasets does not seem new. Many existing methods like diffusion policy (and their improvement) are already capable of learning this purely offline.
My concern is that the proposed method needs to sample many actions and future states during test/inference. If the environment/MDP is highly stochastic, i.e., the transition function has a high randomness over next state (say 50-50 for two drastically different next states), its estimated Q-values can be noisy and executing its chosen action in this stochastic environment can produce very different outcomes, and therefore has highly variable episode returns. It seems that the simulated environments used are mostly very deterministic.
Questions for authors
How is the proposed method related to this related work on predicting distributions of future states/rewards?
Wiltzer H, Farebrother J, Gretton A, Tang Y, Barreto A, Dabney W, Bellemare MG, Rowland M. A distributional analogue to the successor representation. ICML. 2024
2 comments
I would like to thank authors' response and explanation. I encourage the author improve the manuscript with these additional explanation, which would strengthen the paper. I will keep my positive rating.
Thank you for your thoughtful review and questions. We appreciate the acknowledgment of the quality of writing and motivation, novelty of our method, and our effective experimental validation.
It is true that randomly composing offline dataset behaviors can be accomplished by simple imitation (e.g. Diffusion Policy), and many methods are targeted to accomplish improving this composition, each with their own capabilities and drawbacks.
To clarify, we do not claim to be the first to develop offline composition of behaviors (i.e. stitching) for improvement. Rather, our contribution is a new method to accomplish policy improvement with an eye towards long-horizon tasks where TD-based and WM-based methods can break down. Our novelty lies in achieving composition without using temporal-difference learning or excessive 1-step forward predictions, paving the way for better long-horizon planning methods (as validated by our experimental suites).
Very stochastic environments indeed makes value estimation more difficult, including with RSM. We would like to highlight that while the simulated environments in our work are mostly deterministic, our real robotic experiments show RSM’s promise in handling real-world stochasticity in practical setups.
We also understand your concern regarding the inference cost of sampling many times to handle stochasticity. However, our model’s sampling is parallelizable and thus scales well in terms of latency (as shown in the table below). On the non-parallelizable axis—the number of consecutive recursions—RSM scales more favorably than standard world models by simply requiring fewer recursions for a given horizon.
Thank you for introducing us to this work, it is very related to RSM and we have added it to the related work section on successor measures in the new revision.
Distributional Successor Measure focuses on learning the occupancy distribution of a target policy for robust zero-shot policy evaluation. The authors parameterize the model with a weighted ensemble of gamma models (instantiated with GANs), trained with distributional TD Bellman backup. RSM-ZS, the RSM application most closely related to this work, instead focuses on learning the occupancy distribution of an offline dataset, then performing zero-shot policy improvement for arbitrary downstream rewards through planning. We parameterize the model using a flow-matching model trained with Monte Carlo supervised learning and composed recursively. As we have shown throughout the paper, this is more stable and effective than training with TD.