- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new auxiliary objective for ensemble-based offline RL. It claims that ensemble-based offline RL methods such as EDAC suffer from excessive ensemble disagreement. Specifically, even on in-distribution dataset actions, the variance across Q-ensemble members exceeds the variation of Q-values across actions at the same state. To address this, the paper proposes EVA, a regularizer that penalizes differences between the Q-values of ensemble members on in-distribution state-action pairs. EVA is added on top of EDAC's objective. The paper reports the highest average score compared against state-of-the-art model-free and scalable offline RL methods on D4RL MuJoCo, AntMaze, and OGBench.
Critical strengths
- The proposed objective is simple, easy to add to existing ensemble-based offline RL methods, and adds negligible overhead.
- The OGBench results are impressive. Adding an ensemble critic with EVA to FQL substantially improves performance on non-saturated tasks, suggesting that ensemble critics are underexplored in scalable offline RL.
Critical weaknesses
- It seems that the central claim is only partially supported. Figure 4 shows that, without EVA, ensemble disagreement is similar on dataset and random actions, which supports the claim that EDAC's disagreement does not separate ID from OOD actions. However, the paper's main claim, that ensemble variance exceeds the intrinsic action-value variance at the same state and thereby makes the policy chase noise, is shown only for a single state.
- The OGBench evaluation covers only the default task of three environments. Results on all five tasks per environment or on additional environments would better support the claim of broad applicability.
- There are some citation errors in Related Work. Double Q-learning is attributed to Mnih et al. (2013), which is the DQN paper. The correct references should be van Hasselt et al. (2016) [1]. Decision Transformer is attributed to Janner et al. (2021), which is the Trajectory Transformer paper. Decision Transformer is Chen et al. (2021) [2].
[1] van Hasselt et al., Deep Reinforcement Learning with Double Q-learning, AAAI 2016
[2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021
Questions for authors
- Figure 1 qualitatively compares ensemble disagreement with the variation of Q-values along the line between the dataset and policy actions, but only for a single state. Can the authors quantify both quantities (e.g., averaged over many states and tasks), with and without EVA, to support the claim that ensemble variance systematically exceeds the intrinsic action-value variance?
- Since all critics share in EDAC the same TD target under min aggregation, why do they disagree so substantially on the dataset action in Figure 1? Is this mainly due to EDAC's gradient-diversity term?
Thank you for the detailed response and the additional OGBench experiments. Most of my concerns have been addressed, and Figure 2 is a good addition. Therefore, I will raise my score from 2 to 3. One suggestion: rerunning humanoidmaze with different settings seems unnecessary, as a slightly lower score would not weaken the main claims. I encourage keeping the experimental setup consistent across tasks.