- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper builds on BMPC to propose EfficientTDMPC that makes planning less prone to model errors. It averages return estimates across rollout depths and dynamics models, and adds an uncertainty penalty when refreshing policy targets. On HumanoidBench and DMC, the resulting method reaches BMPC’s final aggregate performance with 57% fewer environment steps, while keeping training time roughly the same.
Critical strengths
-
The method is straightforward and easy to understand. Both multi-horizon and multi-dynamics aggregation make intuitive sense to me.
-
More importantly, the results and ablations show that these simple changes work well in practice. This is what matters most to me here, especially after seeing many recent methods struggle to make clear gains over Mrs. Q.
Critical weaknesses
- I would like to see a deeper analysis of why multi-horizon and multi-dynamics aggregation work. The cross-scoring analysis is a useful start, but I still do not have a clear picture of what drives the gains from each. For example, does multi-horizon aggregation help mainly by reducing estimation errors, or by changing which action sequences the planner prefers? The empirical results are convincing, and I think a clearer understanding of the mechanisms would add an important missing piece.
Questions for authors
See Weaknesses.
2 comments
Thanks for the rebuttal and the clearification, I decide to keep my score.
rebuttal
Thank you for your review, apologies for the delayed response.
The effect of the dynamics aggregation is simple, we have multiple estimates of the same quantity and we average them for a lower error estimate. The cross scoring analysis in Figure 11 shows that the dynamics aggregation reduces estimation error and our ablations in Figure 4 show a significant increase in sample efficiency. The horizon aggregation is indeed more complex because it is a truely different objective that is being estimated and optimized. The contribution has two properties as you mentioned, 1) It changes the true objective, and 2) that objective can be estimated with less error.
We hypothesize that the sample-efficiency gain comes from lower estimation error. Firstly, the measurements we added in Appendix C.2 and the table below show that the objective is estimated with lower error, so we expect this to have an effect. Secondly, without estimation errors there is no reason to expect our horizon aggregated objective to choose better action sequences than a terminal value bootstrap, since the terminal value bootstrap is per definition the value of the full sequence, which is what we aim to maximize. Our horizon aggregated objective is only useful because it can be estimated with less error, but it does represent how good an action sequence is as well as a terminal value bootstrap.
A deeper experimental analysis of the multi-horizon objective is outside the scope of this paper. The table below shows how horizon aggregation reduces value estimation error during planning. More details are in Appendix C.2.
We hope this clarifies your concern and we hope you will consider raising your score.