- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes BRVL, an action selection method that combines prior methods such as relative return-based action selection, decision-aware learning, and split-conformal calibration to replace incumbent actions if the predicted return improvements derived from world-model features are sufficiently large when compared to those of the incumbent actions. The authors show performance improvements over prior work across both DreamerV3 and TD-MPC2 in the DMC, Meta-World, and ManiSkill2 benchmarks.
Critical strengths
The authors demonstrate improvements in performance across two modern model-based RL methods and three benchmark suites (DMC, Meta-World, and ManiSkill2). When compared to cases without any of their additions, the BRVL method achieves up to a 20 percentage point higher success rate in Meta-World and up to a 35 percentage point increased success rate in ManiSkill2. This indicates that the combination of these methods does provide generalized performance across different agent algorithms and environment types.
The inclusion of ablations for BRVL that show the relative improvements and contributions of each component to the overall performance of the method are very useful. They isolate which parts are most important and demonstrate how each contributes to the overall performance.
Critical weaknesses
The primary weakness of this paper is the limited novelty of the technical contributions. The paper combines a number of existing ideas like relative returns (advantage estimates), an antisymmetric head, decision-aware learning, and split-conformal calibration. None of these components are novel, and there is an argument to be made that adding them together is a novel application, but from my understanding these components were 1. already used for similar purposes previously (meaning there isn’t novelty in their utilization or in resolving another underlying problem) and 2. in several cases, were already used in similar RL and decision-making benchmark settings. Under those two conditions the novelty of application seems limited, and their combination appears to be an incremental improvement rather than a substantial novel contribution.
There does seem to be some evidence of improved performance being achieved through combining the methods into BRVL, but without empirical insights into why the combination works better, this seems to be demonstrating A + B + C leads to better performance, rather than something impactful for the RL or world model community. Overall, further discussion of the novelty of combining the methods, not just their performance relative to the ablated baselines from which these methods inherited, is necessary to improve this paper.
I think the writing in this paper is very poor. In just the first page there are numerous cases where sentences fail to build a coherent argument. The connections between ideas throughout this paper are poorly developed at worst and require substantial reader effort to draw connections properly. For example the first three sentences of second paragraph in the introduction list prior work and citations in a very formulaic and enumerated way without making it clear their relationship or relevance to the paper’s central contributions and themes. It should be made clear to the reader through the flow of the paper, not left to the reader to read between the lines.
This same patterns is repeatedly used throughout the paper and makes a substantial revision necessary to establish a coherent narrative flow and make the technical contributions more digestible. Similarly in the results section the authors frequently restate the results from their tables with very little additional insight or explanation.
Scientific writing requires more than presenting technically correct statements and writing out experimental results in prose. This paper does quite a poor job at this fundamental science communication task. There is some interesting content here, but as written I don't think this paper is ready for acceptance.
Questions for authors
Unless I am missing something, prior work already motivated why each individual component that makes up this approach is useful for addressing individual issues. What about the combined effect of BRVL is novel? Some greater intuition and additions to the paper to describe why the combination of these methods produces better performance, and whether that benefit is simply additive. What scientific insight are we to gain from this work?
BRVL uses additional compute resources to generate the 2400 branched trajectories from the simulator per task and checkpoint. Could comparable gains be made by simply using these additional trajectories directly for further training of the agent or the world model? It would seem this is the most relevant missing algorithmic baseline.