acceptodds
Under review as a conference paper at ICLR 2027

ARGO: Asymmetric Replay Grounding with Optimistic Planning

Abstract

Model-based reinforcement learning can improve sample efficiency by reusing learned dynamics, but policy optimization can exploit inaccurate predictions for actions poorly represented in real experience. We introduce ARGO, an online framework that combines replay-grounded actor learning with optimistic data collection. Its central component, support-regularized local policy improvement (SRLPI), derives an auxiliary actor target from a KL-regularized objective at replay states. A learned replay-action density anchors the target, while ensemble disagreement guides conservative action ranking and controls the strength of regularization toward that density. The actor learns from this target alongside its base objective. During collection, an actor-centered planner combines predicted return with ensemble disagreement to select among direct actor samples and local perturbations. Executed transitions enter replay and inform subsequent model and actor updates. Experiments on visual continuous-control tasks show that ARGO improves sample efficiency over strong world-model baselines, with ablations examining the roles of collection and actor learning.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fd5mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the exploitation of model errors in model-based reinforcement learning by restricting policy improvement to the support of the learned model while maintaining optimistic planning for data collection. They consider the common setting where a learned model predicts future states and rewards and the policy is trained on those imagined trajectories. Since the model is learned online from limited data, there can be model error in the predictions that accumulate over imagined trajectories and reinforce suboptimal behavior in the policy. The authors propose ARGO which builds on the TWISTER architecture and chooses actions to execute in the real environment optimistically and uses support-regularized local policy improvement for conservative actor training. More specifically, they learn the model with an ensemble of predictive heads and estimate uncertainty of outputs by measuring the disagreement between ensemble members and also learn a conditional density model to estimate the likelihood of historical actions being taken. They use these when computing the optimistic planning scores for data collection and the conservative actor target, which reweighs the replay density using uncertainty penalised predicted returns. The method is evaluated on a number of DeepMind Control Suite tasks in the vision domain and show performance improvements over strong baselines, namely DreamerV3, TD-MPC2, and TWISTER.

Critical strengths
  • The proposed method achieves strong performance (apart from on Reacher Hard) relative to well-established baselines on standard benchmark tasks. This supports the utility of the algorithm for future research and practical applications.
  • The algorithm idea of separating the optimistic data collection from conservative actor learning is elegant and well-motivated and it makes sense why this would work well.
Critical weaknesses
  • While the method appears principled, the theoretical guarantees are limited. For instance, Proposition 1 only shows that the target policy maximizes the KL regularised surrogate and not that it necessarily improves the current actor. Additionally, the ensemble disagreement is not a calibrated confidence interval so the optimistic and conservative scores are not guaranteed bounds on the true returns.
  • The ablations given do not fully isolate the contributions of the main algorithm. In particular, the effectiveness of support regularized policy improvement and the uncertainty dependent temperature are not studied.
Questions for authors
  • How much of the improvement comes from support regularized policy improvement rather than optimistic planning? An ablation experiment separating these would be very informative.
  • Also can the uncertainty dependent temperature influence be ablated as well? It is currently unclear how much improvement this is responsible for in the method.
  • It is argued that the conditional density model p_\omega helps to encourage actions supported in the replay, however could this also harm exploration? For instance if a rare good action has low density in the replay buffer and high model uncertainty, it seems this would be suppressed even though this is the kind of action that should be explored.
  • Is there any intuition for why ARGO not perform as well on Reacher Hard? Repeating the ablations here would be helpful to see
5 comments
AuthorsOpenReview ↗

Response 1 for Reviewer fd5m

We sincerely thank the reviewer for their constructive suggestions and questions. We address each point below. Changes in the updated draft are marked in blue.

W1. While the method appears principled, the theoretical guarantees are limited...

Response: We thank the reviewer for raising both insightful points. SRLPI is added to the actor's original objective, which remains the source of improvement over the current actor; the SRLPI term pulls the actor toward actions seen in replay that have a higher predicted advantage. A better action that is still rare in replay starts with a small weight in the target. The actor's original objective and the planner still favor it, so it is executed again, its density rises, and its weight grows. Good actions are thus reinforced step by step as real experience confirms them. As with standard actor-critic updates (Mnih et al., 2016; Haarnoja et al., 2018), the improvement in return is shown empirically: ARGO raises the mean return over its backbone by 7.3% on TWISTER and by 9.2% on DreamerV3 (revised Table 1).

As in prior work that uses disagreement for exploration or for conservative penalties (Pathak et al., 2019; Sekar et al., 2020; Yu et al., 2020; Wang et al., 2024), we use ensemble disagreement only as a relative signal, to rank candidate actions and to set the temperature. The ablations indicate that the signal is useful in this role: setting the penalty to zero lowers the final return on Quadruped Run from 869.5 to 817.0 (Figure 3b and Table 7). More reliable uncertainty quantification remains a shared challenge and a meaningful direction for future research.

W2. The ablations given do not fully isolate the contributions of the main algorithm. In particular, the effectiveness of support regularized policy improvement and the uncertainty dependent temperature are not studied.

Response: We thank the reviewer for this suggestion. The updated draft adds ablations of ARGO on Quadruped Walk that study both components (Appendix D, Figure 5). Removing SRLPI while keeping the planner lowers the final return from 922.9 to 749.8. For the temperature, holding it constant at 0.5 or at 1.3 gives 792.1 and 748.8: a fixed temperature can match the uncertainty-dependent one in some runs, but it is less reliable. Our answers to Q1 and Q2 give the details.

Questions

Q1. How much of the improvement comes from support regularized policy improvement rather than optimistic planning? An ablation experiment separating these would be very informative.

Response: We thank the reviewer for this question. The new ablation separates the two parts (Appendix D, Figure 5), and most of the improvement comes from SRLPI: with SRLPI removed and the planner kept, ARGO ends at 749.8 on Quadruped Walk, against 922.9 for the full method.

The optimistic planner mainly helps early in training, where it speeds up exploration. The planner-only variant is ahead of the baseline during the first 300k steps, and on Quadruped Run the variant without the planner improves more slowly early in training (Figure 3a). It is not the main source of policy improvement, however: without SRLPI the planner-only variant ends below the baseline (749.8 against 810.2). A possible reason is that optimistic collection biases the data used to train the world model toward uncertain actions, and SRLPI is what turns this experience into reliable policy improvement.

Q2. Also can the uncertainty dependent temperature influence be ablated as well? It is currently unclear how much improvement this is responsible for in the method.

Response: We thank the reviewer for this question. We ran this ablation with everything else unchanged (Appendix D, Figure 5). A fixed temperature can match the uncertainty-dependent one, but it is less reliable: fixing it at 0.5 or at 1.3 gives final returns of 792.1 and 748.8, against 922.9 for the uncertainty-dependent temperature, and for each constant one of the three runs ends far below the others.

Two observations explain why a good fixed constant is hard to pick.

First, the temperature changes during training. On Quadruped Walk the uncertainty-dependent temperature averages 0.8 over the first 100k steps and rises to about 1.4, because the scores it divides and the ensemble's disagreement about them both grow as returns grow. Keeping the temperature at 0.5 therefore makes the target more and more concentrated.

Second, the good temperature differs across tasks. With the same hyperparameters, the uncertainty-dependent temperature settles at 0.71 on Walker Walk, 1.14 on Quadruped Run, 1.34 on Acrobot Swingup and 1.38 on Quadruped Walk, following how strongly the ensemble disagrees on each task. A fixed temperature would thus be one more hyperparameter to tune per task, whereas the uncertainty-dependent rule uses one setting for all tasks.

AuthorsOpenReview ↗

Response 2 for Reviewer fd5m

Q3. It is argued that the conditional density model p_\omega helps to encourage actions supported in the replay, however could this also harm exploration? ...

Response: We thank the reviewer for raising this very good point. The case the reviewer describes is exactly what ARGO's design is meant to handle: exploration and policy improvement are separated, and the density is used only for policy improvement. Exploration is done by the planner during data collection, which adds a bonus for model uncertainty and does not use the density. The density enters policy improvement because that step relies on the learned model and not on the real environment.

Take the reviewer's example of a good action that is already in replay but rare, with high model uncertainty. Its weight in the SRLPI target is its density multiplied by an exponential of its predicted advantage, so a low density reduces the weight without removing it. The high uncertainty reduces it further, on purpose, because a high predicted return for an action seen only a few times is not yet reliable. But the action is not lost: the planner's uncertainty bonus makes it more likely to be tried again, and the actor's original objective still favors it. Each repetition raises its density and lowers the uncertainty, so its weight grows and good actions are repeated more and more.

This also makes exploration itself more reliable: the planner's candidates are sampled around the actor, so an actor that follows unreliable predictions would explore around poor actions. The new ablation is consistent with this: with the planner but without SRLPI, the agent ends at 749.8 on Quadruped Walk, below both the baseline (810.2) and the full method (922.9) (Appendix D, Figure 5).

Q4. Is there any intuition for why ARGO not perform as well on Reacher Hard? Repeating the ablations here would be helpful to see

Response: We thank the reviewer for raising this good point. The lower score on Reacher Hard comes from the TWISTER backbone and not from ARGO. TWISTER itself reaches 565.8 on this task, and ARGO raises it to 634.4. Reacher Hard has a very small target, so its return is sensitive to small errors in locating the fingertip and the target. A possible reason for TWISTER's lower score is that its Transformer world model, which is designed to capture long-horizon structure, is less precise at this fine scale. In the updated draft we also implement ARGO on a second backbone, DreamerV3 (revised Table 1). There the gap disappears: DreamerV3 reaches 856.6 on Reacher Hard and ARGO raises it to 897.8, the second-highest score among all methods. ARGO improves both backbones on this task, by 68.6 and 41.2 points.

References

  • Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018.
  • Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016.
  • Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-Supervised Exploration via Disagreement. ICML 2019.
  • Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via Self-Supervised World Models. ICML 2020.
  • Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL. ICLR 2024.
  • Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. NeurIPS 2020.
AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer fd5m,

Thank you again for your thoughtful review. Your summary captured the central idea of the paper precisely. The ablations you asked for are now in the updated draft (Appendix D, Figure 5), and they support the design as you described it: most of the improvement comes from SRLPI, the optimistic planner mainly speeds up early exploration, and the uncertainty-dependent temperature is more reliable than a fixed one. The strong performance you noted also holds on a second backbone, DreamerV3 (revised Table 1), where the gap on Reacher Hard disappears: ARGO reaches 897.8 on this task.

The discussion period closes in two days, on November 18. If anything remains unclear, we would be glad to address it in the remaining time.

Best regards, The Authors of ARGO

Reviewer_hN1mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The underlying motivation is to do a better re-use of env. interactions, especially in model based reinforcement learning. the authors use model disagreement as defined in paathak et al where multiple models predict returns and the std. dev. is used as model disagreeement score.

the avg. model estimated returns (based on actions proposed by actor) + disagreement is used to select the action taken in the env. to collect data.

the actor is trained using the the original actor loss + conservative reward tiled action distribution, where: action distribution is learnt from the data buffer and conservative because the reward mean est. returns - disagreement.

Critical strengths

Useful performance gains. ARGO improves aggregate return and sample efficiency across 13 visual-control tasks, including a 12.5% increase in final mean return over its TWISTER backbone.

No planner needed at evaluation. The improvements carry over to the learned actor, rather than depending on additional planning at test time.

A mathematically motivated actor target. The auxiliary target follows from a KL-regularized objective that balances conservative predicted value against staying close to recorded behavior.

Critical weaknesses
  1. Lack of novelty, where the main contribution is not clearly isolated. the authors are hones about it in Sec. 2, the disagreement-based exploration, optimistic collection with conservative model are existing ideas.

  2. The effect of older replay behavior is unclear. The KL derivation explains the target, but does not establish an improvement in the actual environment return. Replay contains earlier, suboptimal behavior, so I'm concerned that the reference could sometimes hinder an improving actor. This is a possible failure mode. I'd like a more direct test of when replay grounding helps and when it becomes restrictive.

Questions for authors
  1. Can you isolate the replay reference? Please keep the same candidate-sampling mixture, and change only the target reference to the replay density to the current actor, with the appropriate importance weights.

  2. Is adaptive temperature necessary? Would a fixed temperature do just as well with everything else unchanged?

  3. Does replay age matter? How are older versus newer transitions sampled for density training? A comparison against a density trained only on recent experience, with the other training data left unchanged, would help address the concern about outdated behavior.

  4. Is SRLPI tied to TWISTER/Dreamer-style training, or can it augment model-based actor–critics using Q-based policy improvement? Testing a second method would help establish generality.

4 comments
AuthorsOpenReview ↗

Response 1 for Reviewer hN1m

We sincerely thank the reviewer for their constructive suggestions and questions. We address each point below. Changes in the updated draft are marked in blue.

W1. Lack of novelty, where the main contribution is not clearly isolated. the authors are hones about it in Sec. 2, the disagreement-based exploration, optimistic collection with conservative model are existing ideas.

Response: We agree that disagreement-based exploration and optimistic collection with a conservative model, as in COPlanner (Wang et al., 2024), are established directions for using model uncertainty in model-based RL, and the related works section discusses and cites them. We take one step further and ask how the agent's past real experience can help. A learned model can be confidently wrong about actions the agent has never tried, and a policy trained inside the model then moves toward those actions. Lowering the predicted return of uncertain actions helps less when the predictions for all actions at a state are unreliable. SRLPI addresses this at the source: an action that was rarely taken has little real evidence behind its predicted return, so it receives little weight in the actor target (Eq. 1). To test this against the closest prior method, the updated draft adds a comparison with COPlanner on its own DreamerV3 backbone across all 13 tasks (revised Table 1). ARGO has a higher mean return than COPlanner on 12 of the 13 tasks (784.7 against 728.3). A new ablation on Quadruped Walk also shows that removing SRLPI while keeping the planner lowers the final return from 922.9 to 749.8 (Appendix D, Figure 5).

W2. The effect of older replay behavior is unclear...

Response: As the reviewer notes, past experience also contains suboptimal actions, so how often an action was taken does not necessarily correlate with its quality. This is why the target weights replay actions by their predicted advantage, computed from ensemble rollouts with a penalty on their disagreement to limit return overestimation (Janner et al., 2019; Yu et al., 2020). A good action that is still rare in replay is not suppressed. It has a high advantage and a low density, so it starts with a small weight in the target. The actor's original objective and the planner still favor it, so it is executed again, its density rises, and its weight grows. Good actions are thus reinforced step by step as real experience confirms them. As with standard actor-critic updates (Mnih et al., 2016; Haarnoja et al., 2018), the improvement in return is then shown empirically.

We added ablations on Quadruped Walk that test this directly (Appendix D, Figure 5). Replay grounding is restrictive early in training, when replay holds little good behavior, and helps afterwards. The table gives the mean evaluation return over the 100k steps ending at each point.

Environment stepsFull ARGOWithout SRLPI
300k132218
400k164279
600k367397
700k555471
1M872750

Without SRLPI the agent learns faster in the first half of training but ends at 749.8, against 922.9 for ARGO. This early cost is why ARGO pairs SRLPI with an optimistic planner, which is designed to bring new actions into replay sooner (Section 3.2). On Quadruped Run, the variant without the planner improves more slowly early in training (Figure 3a).

Questions

Q1. Can you isolate the replay reference? Please keep the same candidate-sampling mixture, and change only the target reference to the replay density to the current actor, with the appropriate importance weights.

Response: We thank the reviewer for this suggestion. We added the ablation the reviewer describes: the candidate mixture is unchanged (α=0.2\alpha = 0.2), only the reference in the target is switched to the current actor, and the importance weights are computed for that target. On Quadruped Walk it ends at 790.9, against 922.9 for ARGO (Appendix D, Figure 5).

AuthorsOpenReview ↗

Response 2 for Reviewer hN1m

Q2. Is adaptive temperature necessary? Would a fixed temperature do just as well with everything else unchanged?

Response: We thank the reviewer for this insightful question. We ran this ablation with everything else unchanged (Appendix D, Figure 5). A fixed temperature can match the adaptive one, but it is less reliable: fixing it at 0.5 or at 1.3 gives final returns of 792.1 and 748.8, against 922.9 for the adaptive temperature, and for each constant one of the three runs ends far below the others.

Two observations explain why a good fixed constant is hard to pick.

First, the temperature changes during training. On Quadruped Walk the adaptive temperature averages 0.8 over the first 100k steps and rises to about 1.4, because the scores it divides and the ensemble's disagreement about them both grow as returns grow. Keeping the temperature at 0.5 therefore makes the target more and more concentrated.

Second, the good temperature differs across tasks. With the same hyperparameters, the adaptive temperature settles at 0.71 on Walker Walk, 1.14 on Quadruped Run, 1.34 on Acrobot Swingup and 1.38 on Quadruped Walk, following how strongly the ensemble disagrees on each task. A fixed temperature would thus be one more hyperparameter to tune per task, whereas the adaptive rule uses one setting for all tasks.

Q3. Does replay age matter? How are older versus newer transitions sampled for density training? A comparison against a density trained only on recent experience, with the other training data left unchanged, would help address the concern about outdated behavior.

Response: We thank the reviewer for this insightful suggestion. In ARGO the density is trained on batches sampled uniformly from the whole replay, so older and newer transitions are treated alike. We ran the comparison the reviewer suggests: the density is trained only on the most recent 20% of the data, and all other training data are unchanged (Appendix D, Figure 5). This variant learns more slowly in the second half of training and ends at 861.3, against 922.9 for ARGO. Restricting the density to recent experience does not help, so we see no sign that older behavior in replay holds the actor back on this task. This is consistent with the design discussed in our answer to W2: the target weights replay actions by their predicted advantage, so older behavior stays in the reference but is reinforced only where it is still better.

Q4. Is SRLPI tied to TWISTER/Dreamer-style training, or can it augment model-based actor–critics using Q-based policy improvement? Testing a second method would help establish generality.

Response: We thank the reviewer for this question. SRLPI is not tied to TWISTER: the updated draft adds ARGO on DreamerV3, a recurrent world model, and evaluates it on all 13 tasks (revised Table 1). ARGO-Dreamer reaches a mean return of 784.7, against 718.5 for DreamerV3 and 728.3 for COPlanner. SRLPI is a separate term added to the training process, so it does not replace the agent's own policy-improvement rule. What it needs is a learned model that can roll out candidate actions for a few steps and a critic to bootstrap those rollouts, which Q-based model-based actor–critics also have. Extending the mechanism to these and other model-based RL settings is a promising direction.

References

  • Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018.
  • Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.
  • Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016.
  • Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL. ICLR 2024.
  • Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. NeurIPS 2020.
AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer hN1m,

Thank you again for your careful review and insightful questions. We would like to briefly follow up on our response. We ran the ablations you suggested on Quadruped Walk (Appendix D, Figure 5): switching the target reference to the current actor with the candidate mixture unchanged, fixing the temperature, and training the density only on recent experience. Each ends below full ARGO. We also removed SRLPI to show when replay grounding is restrictive and when it helps.

The discussion period closes in two days, on November 18. We would greatly appreciate hearing whether our response has addressed your concerns, and whether anything still prevents you from reassessing your score; we would be glad to address it in the remaining time. We look forward to the discussion.

Best regards, The Authors of ARGO

AuthorsOpenReview ↗

Look forward to your reply

Dear Reviewer hN1m,

We would like to thank you for your time and insightful questions. As the discussion period closes tomorrow, November 18, we would like to follow up on our response and the updated draft of the paper. We would be grateful to receive your comments on whether we have addressed your concerns and would be happy to answer any further questions.

Best regards, The Authors of ARGO

Reviewer_C2GmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model-based reinforcement learning and proposes an online MBRL framework, ARGO: Asymmetric Replay Grounding with Optimistic Planning. The ARGO framework includes an optimistic planner driven by ensemble disagreement and an auxiliary actor loss that anchors the actor to replay-supported actions. Evaluated with TWISTER as the base architecture, it outperforms TWISTER on DMC Vision tasks. On the same TWISTER architecture, it also performs better than COPlanner on three selected tasks.

Critical strengths
  • [S1] The proofs provided in the Appendix have no obvious errors; the theory provides justification for the SRLPI target and the corresponding loss.
  • [S2] The authors evaluate ARGO on 13 DMC vision tasks. The results in their current experimental setup show improvement over TWISTER. ARGO also performs better than COPlanner-TWISTER in the three-task comparison.
Critical weaknesses
  • [W1] Baseline Selection: The paper reports SAC, DreamerV3, TD-MPC2 and TWISTER as baselines. Many algorithms, including COPlanner, have been proposed and have improved performance compared to the baseline DreamerV3. Hence, the comparison with baseline DreamerV3 adds minimal value as a fair comparison with DreamerV3 requires that ARGO build on the DreamerV3 framework.

  • [W2] Limited experimental evidence to establish improvement: The paper evaluates performance on 13 DMC Vision environments with SAC, DreamerV3, TD-MPC2, and TWISTER as baselines. However, COPlanner, a key baseline, is only evaluated on three tasks.

  • [W3] Inconsistencies in the evaluation: The performance of these baselines is reported from official repositories instead of running them again, but there are several inconsistencies in the reported values. For example, the DreamerV3 paper reports 5 seeds per task, while only 3 seeds are included here. TWISTER and ARGO are run by the authors, but TWISTER is run for three seeds while ARGO for five seeds.

Questions for authors
  • [Q1] Comparison with COPlanner: The paper provides a comparison with COPlanner on only three environments despite COPlanner being the most relevant baseline along with TWISTER. How were those three tasks selected? The key table for this paper is the comparison between algorithms under the same base architecture; hence, a complete Table 2 is more insightful than Table 1.

  • [Q2] As described in Section 3, the ARGO framework is not limited to the TWISTER architecture. Can the authors provide detailed justification on why they chose TWISTER as their base architecture instead of the DreamerV3 architecture? Comparing DreamerV3, COPlanner-DreamerV3 and an ARGO-DreamerV3 variant seems like a natural choice of experiment, given that COPlanner is already built on DreamerV3 and DreamerV3 performs better than TWISTER in Table 1.

  • [Q3] Can the authors provide a comparison of model sizes used in the various algorithms? The values reported for DreamerV3 are based on a 12M model. A table of wall-clock times for all baselines would be informative; this is currently only briefly mentioned on page 20.

  • [Q4] Additional results on the state-based DMC benchmark would make the case for better performance stronger. Can the algorithm be easily extended to the discrete-action case (such as Atari)?

8 comments
AuthorsOpenReview ↗

Response 1 for Reviewer C2Gm

We sincerely thank the reviewer for their constructive suggestions and questions. We address each point below. Changes in the updated draft are marked in blue.

W1. Baseline Selection: The paper reports SAC, DreamerV3, TD-MPC2 and TWISTER as baselines...

W2. Limited experimental evidence to establish improvement: The paper evaluates performance on 13 DMC Vision environments with SAC, DreamerV3, TD-MPC2, and TWISTER as baselines. However, COPlanner, a key baseline, is only evaluated on three tasks.

Response (W1 and W2): DreamerV3 uses a recurrent world model, whereas TWISTER, the backbone of our submission, is Transformer-based. We included DreamerV3 to compare ARGO with a strong agent built on a different world-model architecture. We thank the reviewer for the suggestion to make this comparison on a shared backbone, which strengthened the paper. We implemented ARGO on DreamerV3 (ARGO-Dreamer) and evaluated it against DreamerV3 and COPlanner on all 13 tasks with five seeds per method, which also extends the COPlanner comparison from three tasks to the full benchmark. ARGO-Dreamer reaches a mean return of 784.7, against 718.5 for DreamerV3 and 728.3 for COPlanner, which is 9.2% and 7.7% higher, respectively. Its mean is higher than DreamerV3's on 12 of the 13 tasks and higher than COPlanner's on 12. Per-task results are in the revised Table 1 and learning curves in Figure 2.

W3. Inconsistencies in the evaluation: The performance of these baselines is reported from official repositories instead of running them again...

Response: We thank the reviewer for pointing this out. We updated Table 1 with five-seed results for both backbones: DreamerV3 and TWISTER are now reported with five seeds each. With these results, ARGO-TWISTER reaches a mean return of 785.0 against 731.6 for TWISTER (7.3% higher) and matches or exceeds TWISTER on all 13 tasks, and ARGO-Dreamer reaches 784.7 against 718.5 for DreamerV3 (9.2% higher).

Questions

Q1. Comparison with COPlanner: The paper provides a comparison with COPlanner on only three environments...

Response: The three tasks cover a hard-exploration swing-up task (Acrobot Swingup) and two locomotion tasks, one of which has the largest action space among the 13 tasks (Quadruped Run). We limited the TWISTER-backbone comparison to three tasks because of its cost: COPlanner's lookahead multiplies the number of model steps, and each step runs TWISTER's Transformer sequence model, so training for 1M frames takes 51.8 h, against 10.0 h for TWISTER and 21.4 h for ARGO-TWISTER. The complete comparison is now made on DreamerV3, COPlanner's own backbone. Across all 13 tasks, ARGO-Dreamer has a 7.7% higher mean return than COPlanner and the higher mean on 12 tasks (please also refer to our answer to W1 and W2). The three-task comparison remains as Table 2.

Q2. As described in Section 3, the ARGO framework is not limited to the TWISTER architecture...

Q3. Can the authors provide a comparison of model sizes used in the various algorithms? ...

Response (Q2 and Q3): We built on TWISTER first because it is a recent Transformer-based world model, and we wanted to test whether ARGO improves an agent of that kind, including on tasks where it underperforms recurrent agents. On Cheetah Run, for example, TWISTER reaches 694.0 and DreamerV3 826.8, and ARGO-TWISTER reaches 830.9. ARGO operates on the model states of the backbone and is not tied to TWISTER: the DreamerV3 results in our answer to W1 and W2 show a similar gain on a recurrent backbone.

The table below gives model size and training cost for each agent, measured on a single RTX 5090 GPU.

AgentTrainable parametersTraining speed (frames/s)Time for 1M framesGPU memory
DreamerV317.9M90.73.1 h8.8 GB
ARGO-Dreamer25.4M82.63.4 h8.8 GB
COPlanner30.6M31.88.7 h8.8 GB
TWISTER43.7M27.710.0 h5.4 GB
ARGO-TWISTER52.9M13.021.4 h5.5 GB
COPlanner–TWISTER60.6M5.451.8 h7.8 GB

On DreamerV3, ARGO adds 7.5M parameters and about 10% training time (3.4 h against 3.1 h), while COPlanner adds 12.7M parameters and takes 8.7 h. COPlanner is slower because it chooses every imagined action by planning. At each of the 15 steps of an imagined rollout, it rolls four candidate actions forward for four steps under the model ensemble and keeps the candidate with the best uncertainty-penalized return. SRLPI leaves the imagined rollouts unchanged and evaluates candidate actions once per update, at 64 replay states. The table is added as Table 5 in Appendix F.

AuthorsOpenReview ↗

Response 2 for Reviewer C2Gm

Q4. Can the algorithm be easily extended to the discrete-action case (such as Atari)?

Response: ARGO extends naturally to discrete actions. With a finite action set, the SRLPI target in Eq. (1) can be computed exactly by summing over actions, so the sampled approximation of Section 3.1 is not needed and the replay-action density becomes a categorical model.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer C2Gm,

Thank you again for your careful review and constructive suggestions. We would like to briefly follow up on our response. Following your suggestion, we implemented ARGO on DreamerV3 and compared it with DreamerV3 and COPlanner on all 13 tasks with five seeds per method (revised Table 1). On this backbone, ARGO's mean return is 9.2% higher than DreamerV3's and 7.7% higher than COPlanner's. We also updated Table 1 with five-seed results for both backbones and added a table of model sizes and training costs (Table 5).

The discussion period closes in two days, on November 18. We would greatly appreciate hearing whether our response has addressed your concerns and whether anything still prevents you from reassessing your score; we would be glad to address it in the remaining time. We look forward to the discussion.

Best regards, The Authors of ARGO

Comments on the paper

2 comments

Thank you for your specific comments. We have added ABM, MOTO and BREMEN to the related work and revised the limitations.

Response (1 and 2):

We agree that Proposition 1 is the MPO/AWR closed form, that AWR's reference is already the replay mixture, and that ABM already uses a learned behavior model as the reference and reweights candidate actions sampled from it. SRLPI builds on this structure. What remains new is the setting and one finding, which is our core message:

In world-model agents that learn online from scratch and train their actor in imagination, the actor's model-based improvement should be anchored to the actions the agent has actually executed (a learned replay-action density), not to its current policy.

ABM, MOTO and BREMEN regularize toward a dataset that is fixed, collected offline first, or gathered in a few large batches; ARGO regularizes toward a replay buffer that the agent builds online from scratch and that changes at every step. World-model agents such as DreamerV3 and TWISTER learn online from scratch, and their actor objective never refers to executed actions. We add a replay reference there and compare it with a current-actor reference. Unlike AWR, which reweights only the actions that were executed, SRLPI weights sampled candidate actions that were never executed, scoring them with the model and the replay density (Section 3.1).

The main contributing component is this replay reference. Changing only the reference to the current actor, as we discussed in Q1, or removing SRLPI, removes the final-return gain over DreamerV3 on Quadruped Walk (three runs; Appendix D, Figure 5).

The other two mechanisms address two problems of a replay reference. Frequent actions in replay can be poor, so the target also ranks candidates by model-predicted return, with an ensemble-disagreement penalty so that model error does not drive the ranking. The reference is also restrictive early in training, when replay holds few good actions, so an optimistic planner is used to bring new actions into replay sooner. Removing either one lowers the mean final return on Quadruped Run and Cheetah Run.

Response (3 and 4):

Thank you for both suggestions. We agree with your comments on adaptive temperature: the uncertainty-dependent temperature is primarily proposed as an implementation choice. We have also revised the limitations to state that SRLPI is untested on agents that use Q-function-based policy improvement.

We hope this answers your (1) and (2), and we would be glad to answer any further questions

Dear Reviewer hN1m,

Thank you for the detailed second round of comments and for raising Soundness and Informativeness. As the discussion period closes today, we would be grateful to hear whether these answers are satisfactory or what is still missing, and we would be glad to clarify anything in the remaining time.

Best regards, The Authors of ARGO

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.