- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to address the exploitation of model errors in model-based reinforcement learning by restricting policy improvement to the support of the learned model while maintaining optimistic planning for data collection. They consider the common setting where a learned model predicts future states and rewards and the policy is trained on those imagined trajectories. Since the model is learned online from limited data, there can be model error in the predictions that accumulate over imagined trajectories and reinforce suboptimal behavior in the policy. The authors propose ARGO which builds on the TWISTER architecture and chooses actions to execute in the real environment optimistically and uses support-regularized local policy improvement for conservative actor training. More specifically, they learn the model with an ensemble of predictive heads and estimate uncertainty of outputs by measuring the disagreement between ensemble members and also learn a conditional density model to estimate the likelihood of historical actions being taken. They use these when computing the optimistic planning scores for data collection and the conservative actor target, which reweighs the replay density using uncertainty penalised predicted returns. The method is evaluated on a number of DeepMind Control Suite tasks in the vision domain and show performance improvements over strong baselines, namely DreamerV3, TD-MPC2, and TWISTER.
Critical strengths
- The proposed method achieves strong performance (apart from on Reacher Hard) relative to well-established baselines on standard benchmark tasks. This supports the utility of the algorithm for future research and practical applications.
- The algorithm idea of separating the optimistic data collection from conservative actor learning is elegant and well-motivated and it makes sense why this would work well.
Critical weaknesses
- While the method appears principled, the theoretical guarantees are limited. For instance, Proposition 1 only shows that the target policy maximizes the KL regularised surrogate and not that it necessarily improves the current actor. Additionally, the ensemble disagreement is not a calibrated confidence interval so the optimistic and conservative scores are not guaranteed bounds on the true returns.
- The ablations given do not fully isolate the contributions of the main algorithm. In particular, the effectiveness of support regularized policy improvement and the uncertainty dependent temperature are not studied.
Questions for authors
- How much of the improvement comes from support regularized policy improvement rather than optimistic planning? An ablation experiment separating these would be very informative.
- Also can the uncertainty dependent temperature influence be ablated as well? It is currently unclear how much improvement this is responsible for in the method.
- It is argued that the conditional density model p_\omega helps to encourage actions supported in the replay, however could this also harm exploration? For instance if a rare good action has low density in the replay buffer and high model uncertainty, it seems this would be suppressed even though this is the kind of action that should be explored.
- Is there any intuition for why ARGO not perform as well on Reacher Hard? Repeating the ablations here would be helpful to see
5 comments
Response 2 for Reviewer fd5m
Q3. It is argued that the conditional density model p_\omega helps to encourage actions supported in the replay, however could this also harm exploration? ...
Response: We thank the reviewer for raising this very good point. The case the reviewer describes is exactly what ARGO's design is meant to handle: exploration and policy improvement are separated, and the density is used only for policy improvement. Exploration is done by the planner during data collection, which adds a bonus for model uncertainty and does not use the density. The density enters policy improvement because that step relies on the learned model and not on the real environment.
Take the reviewer's example of a good action that is already in replay but rare, with high model uncertainty. Its weight in the SRLPI target is its density multiplied by an exponential of its predicted advantage, so a low density reduces the weight without removing it. The high uncertainty reduces it further, on purpose, because a high predicted return for an action seen only a few times is not yet reliable. But the action is not lost: the planner's uncertainty bonus makes it more likely to be tried again, and the actor's original objective still favors it. Each repetition raises its density and lowers the uncertainty, so its weight grows and good actions are repeated more and more.
This also makes exploration itself more reliable: the planner's candidates are sampled around the actor, so an actor that follows unreliable predictions would explore around poor actions. The new ablation is consistent with this: with the planner but without SRLPI, the agent ends at 749.8 on Quadruped Walk, below both the baseline (810.2) and the full method (922.9) (Appendix D, Figure 5).
Q4. Is there any intuition for why ARGO not perform as well on Reacher Hard? Repeating the ablations here would be helpful to see
Response: We thank the reviewer for raising this good point. The lower score on Reacher Hard comes from the TWISTER backbone and not from ARGO. TWISTER itself reaches 565.8 on this task, and ARGO raises it to 634.4. Reacher Hard has a very small target, so its return is sensitive to small errors in locating the fingertip and the target. A possible reason for TWISTER's lower score is that its Transformer world model, which is designed to capture long-horizon structure, is less precise at this fine scale. In the updated draft we also implement ARGO on a second backbone, DreamerV3 (revised Table 1). There the gap disappears: DreamerV3 reaches 856.6 on Reacher Hard and ARGO raises it to 897.8, the second-highest score among all methods. ARGO improves both backbones on this task, by 68.6 and 41.2 points.
References
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018.
- Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016.
- Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-Supervised Exploration via Disagreement. ICML 2019.
- Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via Self-Supervised World Models. ICML 2020.
- Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL. ICLR 2024.
- Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. NeurIPS 2020.
Looking Forward to Your Feedback
Dear Reviewer fd5m,
Thank you again for your thoughtful review. Your summary captured the central idea of the paper precisely. The ablations you asked for are now in the updated draft (Appendix D, Figure 5), and they support the design as you described it: most of the improvement comes from SRLPI, the optimistic planner mainly speeds up early exploration, and the uncertainty-dependent temperature is more reliable than a fixed one. The strong performance you noted also holds on a second backbone, DreamerV3 (revised Table 1), where the gap on Reacher Hard disappears: ARGO reaches 897.8 on this task.
The discussion period closes in two days, on November 18. If anything remains unclear, we would be glad to address it in the remaining time.
Best regards, The Authors of ARGO
Response 1 for Reviewer fd5m
We sincerely thank the reviewer for their constructive suggestions and questions. We address each point below. Changes in the updated draft are marked in blue.
Response: We thank the reviewer for raising both insightful points. SRLPI is added to the actor's original objective, which remains the source of improvement over the current actor; the SRLPI term pulls the actor toward actions seen in replay that have a higher predicted advantage. A better action that is still rare in replay starts with a small weight in the target. The actor's original objective and the planner still favor it, so it is executed again, its density rises, and its weight grows. Good actions are thus reinforced step by step as real experience confirms them. As with standard actor-critic updates (Mnih et al., 2016; Haarnoja et al., 2018), the improvement in return is shown empirically: ARGO raises the mean return over its backbone by 7.3% on TWISTER and by 9.2% on DreamerV3 (revised Table 1).
As in prior work that uses disagreement for exploration or for conservative penalties (Pathak et al., 2019; Sekar et al., 2020; Yu et al., 2020; Wang et al., 2024), we use ensemble disagreement only as a relative signal, to rank candidate actions and to set the temperature. The ablations indicate that the signal is useful in this role: setting the penalty to zero lowers the final return on Quadruped Run from 869.5 to 817.0 (Figure 3b and Table 7). More reliable uncertainty quantification remains a shared challenge and a meaningful direction for future research.
Response: We thank the reviewer for this suggestion. The updated draft adds ablations of ARGO on Quadruped Walk that study both components (Appendix D, Figure 5). Removing SRLPI while keeping the planner lowers the final return from 922.9 to 749.8. For the temperature, holding it constant at 0.5 or at 1.3 gives 792.1 and 748.8: a fixed temperature can match the uncertainty-dependent one in some runs, but it is less reliable. Our answers to Q1 and Q2 give the details.
Questions
Response: We thank the reviewer for this question. The new ablation separates the two parts (Appendix D, Figure 5), and most of the improvement comes from SRLPI: with SRLPI removed and the planner kept, ARGO ends at 749.8 on Quadruped Walk, against 922.9 for the full method.
The optimistic planner mainly helps early in training, where it speeds up exploration. The planner-only variant is ahead of the baseline during the first 300k steps, and on Quadruped Run the variant without the planner improves more slowly early in training (Figure 3a). It is not the main source of policy improvement, however: without SRLPI the planner-only variant ends below the baseline (749.8 against 810.2). A possible reason is that optimistic collection biases the data used to train the world model toward uncertain actions, and SRLPI is what turns this experience into reliable policy improvement.
Response: We thank the reviewer for this question. We ran this ablation with everything else unchanged (Appendix D, Figure 5). A fixed temperature can match the uncertainty-dependent one, but it is less reliable: fixing it at 0.5 or at 1.3 gives final returns of 792.1 and 748.8, against 922.9 for the uncertainty-dependent temperature, and for each constant one of the three runs ends far below the others.
Two observations explain why a good fixed constant is hard to pick.
First, the temperature changes during training. On Quadruped Walk the uncertainty-dependent temperature averages 0.8 over the first 100k steps and rises to about 1.4, because the scores it divides and the ensemble's disagreement about them both grow as returns grow. Keeping the temperature at 0.5 therefore makes the target more and more concentrated.
Second, the good temperature differs across tasks. With the same hyperparameters, the uncertainty-dependent temperature settles at 0.71 on Walker Walk, 1.14 on Quadruped Run, 1.34 on Acrobot Swingup and 1.38 on Quadruped Walk, following how strongly the ensemble disagrees on each task. A fixed temperature would thus be one more hyperparameter to tune per task, whereas the uncertainty-dependent rule uses one setting for all tasks.