acceptodds
Under review as a conference paper at ICLR 2027

Predictive Signature Diversity for Model-Based Offline Reinforcement Learning

Abstract

Offline reinforcement learning (offline RL) aims to learn policies from static datasets, but suffers from distributional shift when evaluating out-of-distribution actions. Model-based approaches mitigate this issue using learned dynamics models, yet remain highly sensitive to model uncertainty, especially in low-coverage regions. Existing approaches fail to adequately capture such uncertainty, particularly in terms of worst-case discrepancies between models. We propose a predictive, policy-independent framework for model selection based on predictive signature diversity. Each model is represented by its multi-step predictions on shared anchor inputs, enabling comparison of behaviors in a common space. We periodically select a compact subset of accurate models by maximizing worst-case separation under this representation, yielding a dynamically updated ensemble for rollout generation and policy learning. Our approach decouples diversity estimation from policy rollouts and directly captures model disagreement in uncertain regions. We provide empirical results demonstrating improved robustness over a broad range of baselines, including model-free, model-based, ensemble-based, and mutual-information-based methods.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Model-Based Reinforcement Learning, Dynamics Ensembles, Uncertainty Estimation, Distribution Shift, Ensemble Diversity, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dcLvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PS-DAP, a model-based offline reinforcement learning method that selects dynamics models using predictive-signature diversity. Candidate models are filtered by validation prediction loss and compared through multi-step state and reward predictions on shared anchor inputs. A max–min diversity criterion selects the rollout ensemble, while a diversity regularizer further separates model predictions during refinement. The selected models generate synthetic experience for an adaptive policy built on the MAPLE framework.

Critical strengths

Shared anchor inputs provide a clear and practical basis for comparing dynamics models. Evaluating models on identical states and action sequences removes differences in policy-induced inputs from the diversity calculation. The resulting selection mechanism is modular and shows improvements over the reported MAPLE baseline on several D4RL datasets.

Critical weaknesses
  • The baseline comparison is substantially outdated and does not support the claims of state-of-the-art performance. Table 1 primarily compares against methods from 2018–2021, including SAC, BEAR, CQL, MOPO, and MAPLE. Important recent model-based offline RL methods are missing, including ADMPO-OFF [1], which addresses compounding prediction errors and model uncertainty; LEQ [2], which improves model-based value estimation through lower expectile regression; and MAC [3], which uses action chunks to support longer model rollouts. These methods address central challenges motivating PS-DAP and would provide more informative comparisons under compatible evaluation settings. Although the paper additionally evaluates MoDAP (2024), Appendix B explicitly acknowledges that its reproduction falls below the originally reported performance. This transparency is appreciated, but comparisons against that reproduction cannot establish superiority over the original method. The empirical positioning therefore remains incomplete.

  • The connection between predictive diversity and reliable uncertainty estimation is insufficiently established. Maximizing separation among model predictions does not necessarily produce plausible alternative dynamics or better calibrated epistemic uncertainty. The distance in Eq. (2) averages discrepancies over offline anchors, while the max–min criterion operates over model pairs; this is not a worst-case guarantee over states or policy-induced distribution shifts. Moreover, the diversity regularizer explicitly encourages disagreement, which could amplify prediction errors rather than meaningful uncertainty. Initial validation-loss filtering and an MLE term provide some control, but Algorithm 3 does not explicitly reapply the validation-loss constraint after model refinement. The paper needs direct measurements of prediction accuracy, uncertainty–error correspondence, and behavior outside dataset support to justify its central interpretation.

  • The experiments do not adequately isolate the proposed mechanism or substantiate the robustness claims. Evaluation is restricted to three D4RL locomotion environments, while the main diversity-loss and subset-selection ablations are limited to walker2d-medium-replay. The MAPLE comparison also changes ensemble configuration: MAPLE trains 20 models and selects 14, whereas PS-DAP trains 10 and selects 6. Consequently, the main comparison does not isolate predictive-signature selection from ensemble-size effects. The random-selection ablation is useful, but broader comparisons with matched ensemble sizes and refinement schedules are needed. In addition, the reported results are not uniformly robust: PS-DAP underperforms MAPLE on halfcheetah-random, and Appendix D acknowledges substantial fluctuations on hopper-random. Broader controlled experiments, including explicit distribution-shift tests, are necessary before concluding that the method improves robustness in poorly covered or out-of-support regions.

[1] Haoxin Lin, Yu-Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia, Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning. ICLR, 2025.

[2] Kwanyoung Park and Youngwoon Lee. Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning. ICLR, 2025.

[3] Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable Offline Model-Based RL with Action Chunks. arXiv:2512.08108, 2025.

Questions for authors
  • How does PS-DAP compare with ADMPO-OFF, LEQ, and MAC under matched evaluation and tuning protocols?

  • Does diversity regularization improve the correspondence between ensemble disagreement and actual prediction error, especially on policy-induced states outside dataset support?

  • Do the gains persist when ensemble sizes and model-refinement schedules are matched, and when the component ablations are repeated across multiple environments?

Reviewer_rvyBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes PS-DAP, a model-based offline RL method that compares dynamics models through multi-step state/reward predictions on shared offline anchors. It filters models by validation error, selects a subset with a max-min predictive diveristy, and refines selected models with a diversity regularizer. These models generate rollouts for a history-conditioned SAC policy. Evaluation covers 12 D4RL datasets demonstrate promising improvement over selected baselines.

Critical strengths
  1. Shared anchors allow direct comparison of model predictions without relying on trajectories from the current policy. The max-min criterion discourages redundant ensemble members and can be used without changing the policy optimizer.

  2. The experiments show promising performance gains over the evaluated baselines, with ablations suggesting that both diversity refinement and model selection contribute.

Critical weaknesses
  1. The max-min diversity criterion is heuristic. The paper provides no theoretical result on proposed predictive separation leading to calibrated uncertainty, reduced model bias, or improved policy return. Encouraging disagreement may amplify model error, and validation filtering followed by a soft likelihood penalty does not guarantee that models remain accurate during refinement. I would like a theoretical justification for the claimed robustness benefit.

  2. The authors acknowledge sensitivity to several hyperparameters, but the reported ablations and tuning guidance are limited, particularly for the model-refinement and reselection schedules. Results are also averaged over only three random seeds, leaving uncertainty about how consistently the reported gains hold across initializations.

Questions for authors
  1. Can the authors provide a theoretical justification for the improved performance? For example, under what assumptions would the proposed diversity-driven model refinement and ensemble reselection improve policy performance?
  2. On Walker2d environment with dataset generated by medium-replay policy, MAPLE appears comparable to PS-DAP during training in Fig. 6(c), but Table 1 reports 68.7±7.668.7\pm7.6 for MAPLE versus 95.3±7.995.3\pm7.9 for PS-DAP. Could you clarify how these table scores are computed from the learning curves?
  3. Could the authors evaluate main methods over more random seeds to assess sensitivity to initialization?
Reviewer_qfPJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Model-based offline reinforcement learning (RL) fits dynamics models on a static dataset to generate synthetic trajectories for policy training. However, these models often suffer from inaccuracy in regions with low dataset coverage. While ensembling can help capture epistemic uncertainty, selecting models based solely on prediction accuracy often yields redundant models that fail to capture diverse plausible dynamics. Furthermore, existing diversity-seeking methods rely on policy-induced trajectories, tying diversity estimation to the current policy. To address this, the authors propose PS-DAP, a framework that selects a set of accurate yet diverse dynamics models independently of the policy. Specifically, PS-DAP: (1) train an ensemble of dynamics models using maximum likelihood (2) measure the difference between the models using fixed, policy-independent predictive signatures (3) selects a subset of k models that have the maximum values their minimum pariwise distance arg⁡max⁡min⁡i,jd(i,j)\arg \max \min_{i,j} d(i,j).

Critical strengths

Well-motivated and convincing algorithm: The paper is well motivated, clearly identifying the limitations of existing ensemble-based approaches in model-based offline RL. The proposed algorithm is intuitive and convincing, combining prediction-loss-based filtering with diversity-aware model selection. In particular, measuring model diversity through predictive signatures on shared inputs provides a straightforward and policy-independent way to compare dynamics models.

Critical weaknesses

Lack of theoretical justification for diversity objective: Diversity does not necessarily guarantee robustness. Maximizing discrepancies between predicted trajectories does not ensure that the selected models capture plausible dynamics or improve worst-case policy performance. Specifically, the max–min objective optimizes pairwise diversity rather than bounding worst-case prediction error or policy return. A theoretical foundation connecting predictive signature diversity to worst-case performance bounds would significantly strengthen the paper.

Limited experimental scope: The empirical evaluation is restricted to only three MuJoCo environments and lacks targeted diagnostics. Covering the pracitical scenarios would enhance the paper. The evaluation would be much stronger with direct experiments demonstrating that predictive signature diversity leads to better uncertainty calibration, higher model accuracy in out-of-distribution regions, or enhanced robustness under dynamics shifts.

Questions for authors

See the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.