- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PS-DAP, a model-based offline reinforcement learning method that selects dynamics models using predictive-signature diversity. Candidate models are filtered by validation prediction loss and compared through multi-step state and reward predictions on shared anchor inputs. A max–min diversity criterion selects the rollout ensemble, while a diversity regularizer further separates model predictions during refinement. The selected models generate synthetic experience for an adaptive policy built on the MAPLE framework.
Critical strengths
Shared anchor inputs provide a clear and practical basis for comparing dynamics models. Evaluating models on identical states and action sequences removes differences in policy-induced inputs from the diversity calculation. The resulting selection mechanism is modular and shows improvements over the reported MAPLE baseline on several D4RL datasets.
Critical weaknesses
-
The baseline comparison is substantially outdated and does not support the claims of state-of-the-art performance. Table 1 primarily compares against methods from 2018–2021, including SAC, BEAR, CQL, MOPO, and MAPLE. Important recent model-based offline RL methods are missing, including ADMPO-OFF [1], which addresses compounding prediction errors and model uncertainty; LEQ [2], which improves model-based value estimation through lower expectile regression; and MAC [3], which uses action chunks to support longer model rollouts. These methods address central challenges motivating PS-DAP and would provide more informative comparisons under compatible evaluation settings. Although the paper additionally evaluates MoDAP (2024), Appendix B explicitly acknowledges that its reproduction falls below the originally reported performance. This transparency is appreciated, but comparisons against that reproduction cannot establish superiority over the original method. The empirical positioning therefore remains incomplete.
-
The connection between predictive diversity and reliable uncertainty estimation is insufficiently established. Maximizing separation among model predictions does not necessarily produce plausible alternative dynamics or better calibrated epistemic uncertainty. The distance in Eq. (2) averages discrepancies over offline anchors, while the max–min criterion operates over model pairs; this is not a worst-case guarantee over states or policy-induced distribution shifts. Moreover, the diversity regularizer explicitly encourages disagreement, which could amplify prediction errors rather than meaningful uncertainty. Initial validation-loss filtering and an MLE term provide some control, but Algorithm 3 does not explicitly reapply the validation-loss constraint after model refinement. The paper needs direct measurements of prediction accuracy, uncertainty–error correspondence, and behavior outside dataset support to justify its central interpretation.
-
The experiments do not adequately isolate the proposed mechanism or substantiate the robustness claims. Evaluation is restricted to three D4RL locomotion environments, while the main diversity-loss and subset-selection ablations are limited to walker2d-medium-replay. The MAPLE comparison also changes ensemble configuration: MAPLE trains 20 models and selects 14, whereas PS-DAP trains 10 and selects 6. Consequently, the main comparison does not isolate predictive-signature selection from ensemble-size effects. The random-selection ablation is useful, but broader comparisons with matched ensemble sizes and refinement schedules are needed. In addition, the reported results are not uniformly robust: PS-DAP underperforms MAPLE on halfcheetah-random, and Appendix D acknowledges substantial fluctuations on hopper-random. Broader controlled experiments, including explicit distribution-shift tests, are necessary before concluding that the method improves robustness in poorly covered or out-of-support regions.
[1] Haoxin Lin, Yu-Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia, Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning. ICLR, 2025.
[2] Kwanyoung Park and Youngwoon Lee. Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning. ICLR, 2025.
[3] Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable Offline Model-Based RL with Action Chunks. arXiv:2512.08108, 2025.
Questions for authors
-
How does PS-DAP compare with ADMPO-OFF, LEQ, and MAC under matched evaluation and tuning protocols?
-
Does diversity regularization improve the correspondence between ensemble disagreement and actual prediction error, especially on policy-induced states outside dataset support?
-
Do the gains persist when ensemble sizes and model-refinement schedules are matched, and when the component ablations are repeated across multiple environments?