acceptodds
Under review as a conference paper at ICLR 2027

Offline Distributional World Model Value Iteration

Abstract

Learned world models enable scalable planning in reinforcement learning but remain challenging in the offline setting, where planning can exploit erroneous rewards and transitions induced by model approximation error. While prior work has established distributional value iteration under known dynamics, little is known about distributional planning over learned world models in partially observable environments. We address this gap by deriving Wasserstein bounds that characterize how world-model approximation error affects latent return distributions and by introducing a support-dependent reward shift with distributional convergence guarantees under stated assumptions. Building on this analysis, we propose Offline Distributional World model Value Iteration (ODWVI), a practical algorithm for support-constrained distributional planning in learned latent world models. Across three offline planning benchmarks, ODWVI achieves the highest mean return among the evaluated methods in the main benchmark configurations. Support ablations further show that performance degrades as planning is extended to increasingly weakly supported actions, highlighting the importance of behavior-policy support when planning with imperfect learned world models.

Primary area:
reinforcement learning
Keywords:
distributional reinforcement learning, offline reinforcement learning, world models, model-based reinforcement learning, POMDPs, distributional value iteration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qc6LOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Offline Distributional World Model Value Iteration (ODWVI) for distributional value iteration in learned world models under offline partial observability. The method combines behavior-support constraints, a support-dependent reward shift, and OOD regularization to reduce the exploitation of model errors on weakly supported actions. The paper also provides a Wasserstein-based analysis of model-error propagation and convergence, and evaluates the method in three ViZDoom environments.

Critical strengths

The paper addresses the amplification of model errors through Bellman backups in offline world-model planning, which is a meaningful research problem.

Critical weaknesses
  1. The paper contains numerous overtly defensive statements that appear intended to preempt potential reviewer criticisms. These statements substantially detract from the paper's scholarly tone and professionalism.

  2. Key ablation studies are missing, making it difficult to assess the individual contributions of the distributional critic and the support-dependent left shift.

  3. The main experiments provide insufficient baseline coverage, with some strong baselines reported in only one environment. Furthermore, the evaluation is limited to three ViZDoom tasks, which is too narrow to support broader conclusions.

Questions for authors
  1. Could the authors clarify the meaning and purpose of the overtly defensive statements in lines 227–230, 243–247, 288–290, 422–423, 430–431, and 475–478?

  2. Could the authors provide additional experiments addressing Weaknesses 2 and 3 to establish the method's broader applicability and isolate the contributions of the relevant components?

I would raise my score if the authors satisfactorily address these two key questions and resolve my concerns. Thank you for considering them.

5 comments

Thank you for your careful review and specific feedback. We appreciate the opportunity to improve the paper’s presentation and provide clearer evidence about the contributions of its components.

  • Presentation and tone: The passages you identified were intended to distinguish theoretical guarantees, implementation choices, and empirical limitations. We recognize that their wording is unnecessarily defensive and interrupts the presentation. We will revise all identified passages to state these distinctions directly and concisely, while more clearly acknowledging the prior work underlying our approach.
  • Component ablations: We have completed ablations on our Deadly Corridor dataset across k=1,2,4,8, removing the left shift, removing OOD regularization, and comparing scalar and distributional critics with left shift disabled in both. Removing OOD regularization substantially reduces mean returns, and distributional learning improves mean returns at several support sizes. The left shift provides little additional benefit in this environment. We will incorporate these results and revise the discussion to reflect the differing contributions of these components.
  • Baseline coverage and applicability: We agree that broader evaluation would strengthen the study. Our baselines serve complementary purposes: JOWA provides a closely related world-model-based comparison; ODWVI-CQL shares our frozen world model to compare value-learning objectives; Dreamer Offline evaluates another world-model-based learning approach; and BC tests whether value learning improves upon imitation. We selected ViZDoom to study visual partial observability while retaining simulator access for evaluating policies trained entirely offline. Our computational budget limited additional baseline adaptation and evaluation: on Deadly Corridor, JOWA required more than 4x the total run wall time of ODWVI-CQL across five seeds on NVIDIA H200 GPUs, while yielding similar observed mean returns. These resource constraints explain our experimental scope but do not establish broader applicability. We will keep our conclusions scoped to the evaluated settings.

We hope these additions address the component-isolation concern and make the remaining limitations clearer.

Thank you for your response and clarifications.

However, I believe the paper has substantial issues with its organization and presentation that require more than minor wording changes. Since ICLR allows manuscript revisions during rebuttal, I would have expected these concerns to be addressed directly in a revised manuscript rather than through promises of future changes.

As the manuscript remains unchanged, I will maintain my current rating. I encourage the authors to submit a revised version with the changes highlighted.

We have uploaded a substantially revised manuscript, and we appreciate the opportunity to address your concerns directly. In particular:

Organization and presentation: We revised the Introduction and reorganized the theoretical exposition, methodology, and results to establish a clearer scientific narrative. Section 6 now presents the algorithm through dedicated subsections and a new architectural overview (Figure 1). We also removed or rewrote the defensive passages identified in your review, presenting the relevant scientific qualifications more directly and consolidating the principal limitations in Section 8.

Component ablations: Section 7.3 and Figure 2 now report experiments examining the support-dependent left shift, OOD regularization, and categorical versus scalar critics. The complete paired statistical comparisons are provided in Appendix F. Removing OOD regularization significantly reduces performance across all four evaluated support sizes, while the categorical configuration significantly outperforms the scalar configuration at (k=1) after Holm correction.

Experimental scope: We have clarified the roles of the evaluated baselines and explicitly acknowledged the remaining limitations in environment and baseline coverage. Our conclusions remain restricted to the evaluated settings.

We have also provided a detailed summary of the revisions and their locations in our new top-level comment.

We are grateful for your feedback, which motivated substantial improvements to the manuscript's organization, clarity, and experimental validation. We hope the revised version addresses your principal concerns and welcome any further feedback.

Reviewer_j3p8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically characterizes how model error affects distributional Q learning in partially observable settings and proposes a support-dependent reward-shaping term for safer planning under these settings.

Critical strengths

The theoretical results proposed in the paper could be useful for developing model-based RL algorithms for POMDPs.

Critical weaknesses

The experiments section is particularly weak. It's not clear from Table 1 that the results are statistically significant. There also seems to be a disconnect between Secs 4 and 5. Its not immediately clear to me why support-constrained learning is motivated by the preceding section.

Questions for authors
  1. Could you specify how many seeds were used clearly in the main text? The appendix mentions using a particular seed 360749. Could you clarify why this specific seed was used?

  2. Please add references to more recent work on distributional Q learning in the Related Work section, eg. Value Flows (Dong et., al., 2026)

  3. From Sec 7.4: "They neither isolate the effects of distributional parameterization, the left shift, and OOD regularization nor directly measure model exploitation or categorical return-law accuracy. ".. could you add ablation studies that would help to separate out these effects?

  4. Please include statistical significance tests to your existing analysis in Table 1.

4 comments

Thank you for your constructive feedback and for recognizing the potential usefulness of our theoretical results. Your suggestions will help us strengthen the experimental evidence and explain the connection between the analysis and algorithm more clearly.

  • Seeds: We used five distinct offline training seeds, separate from the data-collection seed (360749) and a shared evaluation seed. The collection seed was chosen arbitrarily for reproducibility; it has no special methodological significance. We will clearly distinguish these roles and state the number of training seeds in the main text.
  • Connection between Sections 4 and 5: Section 4 characterizes how reward and transition-model errors affect distributional Bellman backups. This motivates limiting reliance on potentially unreliable imagined outcomes. Section 5 uses behavior support as a practical proxy for identifying actions on which to construct backups and regularize values. The theory does not establish that behavior likelihood bounds model error. We will update the paper to explain this motivation and distinction more clearly.
  • Component ablations: We have completed ablations on our Deadly Corridor dataset across k=1,2,4,8, removing the left shift, removing OOD regularization, and comparing scalar and distributional critics with left shift disabled in both. Removing OOD regularization substantially reduces mean returns, while distributional learning improves mean returns at several support sizes. The left shift provides little additional benefit in this environment. We will add these findings to the paper.
  • Statistical significance: We have completed five-seed statistical comparisons for Table 1. Some comparisons favor ODWVI at an unadjusted p<0.05 threshold, while others remain inconclusive. We will include these analyses and discuss their implications for the performance claims.
  • Related work: Thank you for pointing us to Value Flows (Dong et al., 2026). We will incorporate this work and strengthen our discussion of recent distributional RL methods and their relationship to ODWVI.

We have uploaded the revised manuscript incorporating the changes described in our earlier response.

Specifically, Section 7.1 clarifies the five-seed training and evaluation protocol; Section 2 incorporates Value Flows and expands the related literature; Sections 4–5 clarify the connection between model-approximation error and support-dependent planning; and Section 7.3 and Figure 2 present the requested component ablations.

We have also added complete statistical comparisons in Appendix F, including mean differences, confidence intervals, and Holm-adjusted (p)-values. These analyses identify statistically significant improvements in selected benchmark comparisons while distinguishing results that remain inconclusive.

We sincerely appreciate your specific suggestions, which substantially strengthened both the experimental evidence and the presentation of our theoretical contribution. We welcome any further questions or comments on the revised manuscript.

I thank the authors for their response and revisions. While the response addresses some of my concerns, the results in Table 6 are not indicative that ODWVI performs that well. This is in addition to the already weak empirical analysis on a limited number of tasks. I therefore retain my score.

Thank you for reviewing our revision. We recognize that the benchmark comparisons do not demonstrate consistent statistical superiority across all settings.

We have posted an additional clarification regarding the limited availability of established offline POMDP benchmarks with comparable world-model results, which necessitated constructing datasets and adapting baselines. This motivated us to construct our own datasets and adapt relevant baselines under a common evaluation protocol. We have also updated the Abstract, Introduction, and Section 7.1 to communicate this motivation more clearly.

We acknowledge that broader empirical validation would strengthen the work and appreciate your continued consideration of its contributions.

Reviewer_rZ2yOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies distributional planning in learned latent world models under partial observability in the offline setting. On three ViZDoom tasks, ODWVI with k = 1 obtains the highest mean return among the evaluated methods, and performance decreases as k grows, collapsing at k = 128 on Deadly Corridor.

Critical strengths

The problem is relevant: how model approximation error propagates into return distributions when planning in learned latent world models, and how to limit model exploitation in the offline setting. The paper consistently distinguishes idealized operators from the implemented algorithm, states that the assumptions’ constants are not estimated, and avoids overclaiming in the discussion.

Critical weaknesses

The best configuration (k = 1) is close to behavior cloning, and its gains over BC are not mechanistically explained.

Insufficient experimental scope. All experiments use three ViZDoom tasks with data from a single DreamerV3 run, so results cannot be compared with the offline RL literature.

The method part cites very little prior work, and several design components are presented without acknowledging their close counterparts, which makes the contribution boundaries unclear.

Questions for authors

In what practical situations would one want to plan with a learned world model offline instead of simply imitating the dataset?

The paper emphasizes modeling the full return distribution, but the final policy only uses its average. What does the agent gain, in practice, from knowing the full distribution?

The differences between ODWVI and the baselines are often smaller than the reported standard deviations. How confident are the authors that ODWVI is actually better, rather than the result being due to random variation across training runs?

3 comments

Thank you for taking the time to provide this thoughtful and constructive feedback. We appreciate your suggestions, which will help us improve the clarity of the paper and strengthen its empirical support.

  • Why learn values rather than imitate? Under partial observability, agents need a representation of interaction history that captures information about the underlying state. A world model provides a way to learn such a representation. In our experiments, both BC and ODWVI use representations from the same learned world model. BC learns to reproduce logged action choices, whereas ODWVI uses predicted rewards and transitions to learn values through imagined backups. This is useful when logged behavior is suboptimal and further online interactions are costly or constrained, as in safety-critical applications such as healthcare.
  • BC and k=1: Greedy BC selects the highest-probability action under the learned behavior policy. ODWVI restricts Bellman backups to the top-k behavior-supported actions but executes greedily over all learned action values during evaluation. Our evaluation logs show action disagreement between ODWVI and greedy BC, confirming that their executed policies differ. We will clarify this evidence in the paper.
  • Distributional learning: Distributional targets can improve learning even with mean-greedy execution, as in C51 (Bellemare et al., 2017). We have completed new ablations including a scalar-versus-distributional ablation (with left shift disabled in both) showing higher mean returns for the distributional variant at several support sizes on our Deadly Corridor dataset. We will add these results to the paper.
  • Statistical significance: We have completed comparisons across five training seeds. Some comparisons provide statistically significant evidence favoring ODWVI, while others (including ODWVI versus BC on our Deadly Corridor dataset) remain inconclusive. We will add these analyses and discuss their implications for our performance claims.
  • Experimental scope and attribution: We constructed offline ViZDoom datasets to study distributional planning in learned world models under visual partial observability and trained the evaluated baselines on these datasets. We acknowledge that custom datasets and limited task and baseline coverage restrict broader comparisons. We evaluated JOWA on Deadly Corridor and included ODWVI-CQL across our evaluated settings. Sharing the world model makes ODWVI-CQL a controlled comparison of learning objectives and reduces training costs, although it does not substitute for broader comparisons with JOWA. Our computational budget limited additional baseline adaptation and tuning. We will strengthen method-level citations to acknowledge the prior work underlying our design and clarify our contributions.

Thank you again for your thoughtful feedback. We have uploaded the revised manuscript, including additional experimental evidence and methodological clarifications addressing your questions.

In particular, Section 6.4 and Appendix E clarify the distinction between ODWVI at (k=1) and greedy behavior cloning. Although Bellman backups are restricted to the highest-probability behavior action, execution remains critic-greedy over the full action space.

We have also added empirical measurements in Appendix F comparing the two policies at the same latent beliefs along ODWVI's evaluation trajectories. Across five training seeds and 100 evaluation episodes per seed, episode-averaged action disagreement ranges from 29.82% to 70.86% across the five settings. These results establish that the learned execution policy does not simply reproduce greedy behavior cloning.

To investigate the practical contribution of distributional learning, Section 7.3 and Figure 2 now include categorical-versus-scalar critic ablations, with the left shift disabled in both configurations. The categorical configuration achieves higher observed mean returns across the evaluated support sizes, with the (k=1) comparison remaining significant after Holm correction.

Finally, Sections 6–7 and Appendix F provide stronger methodological attribution, clarify the experimental comparisons, and report statistical significance tests with confidence intervals and multiple-comparison correction.

We appreciate your feedback, which helped us strengthen the methodological explanation and empirical evidence. We welcome further discussion of any remaining concerns.

Thank you again for your thoughtful feedback. We wanted to briefly follow up now that the revised manuscript has been available for review and the discussion period concludes today.

We have clarified the distinction between ODWVI and behavior cloning, added empirical action-disagreement measurements and categorical-versus-scalar critic ablations, and expanded the methodological attribution and statistical analysis.

We would appreciate any further feedback you may have, particularly regarding whether these additions address your original questions and concerns.

Thank you for your time and consideration.

Comments on the paper

3 comments

Area Chair and Reviewers,

Thank you for your thoughtful and constructive feedback and for the opportunity to substantially improve our manuscript. We have now uploaded a revised version of ODWVI incorporating the changes discussed in our earlier responses.

To facilitate your review, we summarize the principal revisions and their locations below.

  1. Theoretical analysis and motivation (Sections 4–5; Appendix A). We substantially revised the exposition of our simulation lemmas, Wasserstein error bounds, and conditional convergence results. We clarified the distinction between convergence of distributional planning within a learned world model and the accuracy of its return distributions relative to the true environment. We also strengthened the motivation connecting model-approximation error to support-dependent planning.

  2. Algorithm presentation (Section 6; Figure 1). We reorganized the methodology around the two training phases, with dedicated explanations of behavior-supported distributional Bellman backups, OOD value regularization, and policy extraction. We added a new architectural overview (Figure 1) and clarified how ODWVI at (k=1) differs from greedy behavior cloning. We also added empirical action-disagreement measurements in Appendix F, showing episode-averaged disagreement rates of 29.82%–70.86% across the five evaluated settings.

  3. Component ablations (Section 7.3; Figure 2; Appendix F). We added experiments evaluating the support-dependent left shift, OOD value regularization, and categorical versus scalar critics across multiple support sizes. Removing OOD regularization significantly reduces performance at all four evaluated support sizes after Holm correction. The categorical critic achieves significantly higher mean return than the scalar configuration at (k=1), while removing the left shift produces no statistically significant differences in these experiments.

  4. Statistical analysis (Section 7; Appendix F). We added formal statistical comparisons across five training seeds, reporting mean differences, confidence intervals, and Holm-adjusted (p)-values. The revised manuscript distinguishes statistically supported improvements from comparisons that remain inconclusive, with complete results provided in the appendix.

  5. Introduction, related work, organization, and presentation (Sections 1–2 and throughout). We substantially revised the Introduction to sharpen the problem motivation, articulate the central research question, and establish a clearer scientific narrative connecting the theoretical analysis, support-dependent planning, practical algorithm, and empirical evaluation. We expanded the related work, strengthened attribution to prior methods, and clarified our contributions relative to existing research. Importantly, we substantially reorganized and condensed the manuscript, removing or rewriting the defensive commentary identified in the reviews. We now present the theoretical assumptions and methodological distinctions more directly, with the principal limitations consolidated in Section 8.

We recognize that broader experimental evaluation remains an important limitation, particularly the restriction to three ViZDoom environments and the evaluation of JOWA on Deadly Corridor only. We have maintained conclusions appropriately scoped to the evaluated settings.

We are sincerely grateful for your detailed and valuable feedback. Your comments provided an important opportunity to strengthen the theoretical exposition, empirical validation, scientific narrative, and overall presentation of our work. We believe the revised manuscript is considerably stronger as a result, and we appreciate the time and care you have devoted to reviewing it.

Thank you again for your time and consideration. We welcome any additional questions or feedback on the revised manuscript.

Dear Reviewers,

Less than two days remain before the discussion period ends on November 18 (AoE). If you have not yet replied to the author rebuttal, please do so as soon as possible, indicating which of your concerns are resolved and which remain.

Best regards, Area Chair

Area Chair and Reviewers,

We would like to clarify an important consideration regarding our experimental scope, which we have now emphasized in the revised manuscript.

While offline RL and POMDP benchmarks exist, established offline POMDP benchmarks with directly comparable world-model results remain limited. We therefore constructed our own offline ViZDoom datasets using single-frame observations (i.e., no frame stacking), preserving partial observability and requiring the recurrent world model to infer task-relevant history.

JOWA, our closest related baseline, was originally designed for multi-game Atari learning with frame-stacked observations. Methods such as MOPO, MOReL, and COMBO primarily evaluate on state-based continuous-control benchmarks. Adapting these methods to our partially observable, discrete-action setting requires substantial implementation and computational effort. We therefore selected BC, Dreamer Offline, JOWA, and ODWVI-CQL as complementary comparisons.

ODWVI-CQL was motivated by JOWA’s use of CQL-style regularization. It trains a categorical critic with a CQL objective using the same frozen world model as ODWVI, providing a more directly comparable CQL-based baseline while controlling for differences in world-model architecture.

We have clarified this motivation in the Abstract, Introduction, and Section 7.1.

We recognize that broader evaluation remains valuable, but hope this context helps explain our experimental choices and the challenges of establishing comparable benchmarks.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.