acceptodds
Withdrawn from ICLR 2027

Better Samples, Wrong Odds: When Prediction Rewards Distort World-Model Probabilities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

70 views

Abstract

Prediction rewards post-train world models to produce better samples; we study when they preserve the probabilities a planner needs across futures and when they distort them. A prediction reward, which we call a point reward, compares one generated future with one observed outcome. Under such a reward a model can score below the truth's own risk only with wrong probabilities, which gives a checkable diagnostic: for any metric, the excess gain—the part of the gain that lies below the truth's own risk—lower-bounds the Wasserstein-1 distance to the truth. In PushT, where all futures can be enumerated, about 72% of the gain from point-reward training is excess gain. On RLVR-World's RT-1 video world model, point rewards lower pixel error but give a worse held-out joint energy score than an energy-score reward from the same base, data, and budget in all five seeds, by 2.1% of Base's score at 200 updates and 7.2% at an exploratory 800. The distortion reaches decisions when action rankings depend on how probability is split: in an image-conditioned MiniGrid task, it costs 10.3 points of success, and continuing with a proper score recovers most of it. Proper scoring alone is not sufficient: the update rule also matters. Under idealized GRPO standardization, energy-score and point-reward updates coincide for binary outcomes, while unbiased leave-one-out credits keep the truth a fixed point. We therefore recommend reporting a proper score beside sample metrics and training proper rewards with unbiased credits.

Primary area:
reinforcement learning
Keywords:
world models, proper scoring rules, energy score, reinforcement learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D6qQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how post-training world models with point prediction rewards improves sample quality while distorting probabilities over valid futures. It introduces an excess-gain metric that bounds the Wasserstein-1 distance to the ground truth when point risk drops below the truth's intrinsic uncertainty. The authors prove that GRPO group standardization biases energy-score gradient updates, collapsing to point-reward updates in the binary case. Evaluations across PushT, RT-1, FrozenLake, MiniGrid, and synthetic domains demonstrate conditional decision degradation, show that proper-score fine-tuning can recover performance, and advocate reporting distributional metrics alongside sample fidelity.

Critical strengths
  • Concrete, checkable distortion metric: The excess-gain lower bound transforms an apparent improvement below intrinsic risk into an actionable indicator of distributional mismatch, supported by direct probability evaluations in PushT.
  • Clear update-rule bias analysis: The paper demonstrates the equivalence between standardized energy-score updates and point rewards for binary outcomes and provides an actionable leave-one-out (LOO) credit assignment alternative.
Critical weaknesses
  • Constrained decision evidence and setup sensitivity: Task degradation is demonstrated primarily in small, controlled environments. In MiniGrid, observable harm requires freezing the backbone and training only the prediction head, failing to separate cleanly in end-to-end alternatives.
  • Incomplete video model validation: The RT-1 study uses only 256 training windows and omits closed-loop control evaluations; none of the variants beats the base model on joint energy score, and larger gains at extended steps are confounded by drift from Base.
Questions for authors
  • Q1: Beyond the binary equivalence and LOO derivations for energy scores, what is the central conceptual advance over existing analyses of GRPO miscalibration (e.g., Bereket & Leskovec, 2025)?
  • Q2: Why does decision harm appear in MiniGrid only when the CNN backbone is frozen, but fails to separate cleanly in trainable-backbone configurations?
  • Q3: In the 800-update RT-1 experiments, can the performance deficit of ES-GRPO relative to ES-LOO be disentangled from its greater divergence from the base checkpoint?
Reviewer_o8HNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether prediction-reward post-training improves world-model samples while distorting their probabilities. It introduces an excess-gain diagnostic: point-risk improvement below the true distribution's own risk lower-bounds Wasserstein distance from the truth. Experiments examine probability distortion, its effect on fixed planners, and recovery through proper-score or likelihood continuation. The third contribution concerns optimization: GRPO normalization can bias energy-score gradients, with exact equivalence between energy-score and point-reward updates for binary outcomes under idealized standardization. Unbiased leave-one-out credits preserve the intended energy-score gradient. Evidence includes enumerable simulators, controlled RT-1 video experiments, and planning tasks with both positive and negligible harm.

Critical strengths
  1. The paper goes beyond contrasting sample metrics: it reports probability diagnostics, repeated seeds, matched-divergence comparisons, and equal-budget continuation controls. For example, the reported MiniGrid setup improves pixel error while losing 10.3 percentage points of success, and proper-score continuation recovers most of that loss. It also reports settings with little planning harm. These controls and boundaries provide practical knowledge about when improved prediction metrics can mislead planning.

  2. The binary equivalence under idealized GRPO standardization makes the loss of the proper-score correction explicit. More specifically, pairwise credit dependence makes even mean-only centering biased, unlike the constant gradient scaling in Bereket & Leskovec's individual-reward setting. The appropriate leave-one-out construction preserves the expected energy-score gradient. This is a concrete estimator issue worth understanding when applying proper scores to generated futures.

Critical weaknesses
  1. A distance reward against an observed future is a prediction objective optimized through sampled policy gradients. Increasing model rollouts estimates this objective more accurately but supplies no additional environment observations. Its preference for concentrated predictions is therefore closely related to the established distinction between point and distributional prediction, rather than a newly identified RL-specific problem. The excess-gain diagnostic and world-model experiments have value, but it remains unclear how much additional knowledge this part establishes beyond applying mature scoring-rule theory to post-training. This concern does not imply that one observation per condition makes distribution learning impossible.

  2. Bereket & Leskovec (2025) cited in the paper demonstrates miscalibration under proper rewards, analyzes standard-deviation normalization, and tests RLOO and removal of that normalization. This paper adds energy-score coupling and exact binary equivalence, which I find interesting, but these are specific extensions of an existing problem. The mean-only distinction is also not associated with a substantial task advantage for LOO in the reported FrozenLake results: ES-mean and ES-LOO each lose about 0.1 points, while ES-GRPO loses 2.3–2.5 points (Table 11). This limits the demonstrated practical importance of that additional theoretical result.

Questions for authors
  1. What new empirical knowledge does the first part establish beyond the standard distinction between point and distributional prediction? Please identify the existing results that most directly support this added value, and distinguish outcome of the objective from outcome of the RL update rule.

  2. Regarding Bereket & Leskovec (2025), please clarify the specific added theoretical result and practical lesson of the energy-score analysis. Specifically, how should the practical importance of mean-only centering bias be interpreted when ES-mean and ES-LOO have nearly identical FrozenLake task performance?

Reviewer_VXyeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analysis the problem of collapse in world model.

Critical strengths

None

Critical weaknesses

Weaknesses

W1. Clarity. The paper is hard to read. It introduces many new terms (point reward, truth’s own risk, excess gain, Point→X, -GRPO/-LOO, etc.), nearly every sentence is packed with numbers and pointers to the appendix, and six experimental systems are each covered in a paragraph or two with no clear role in the argument. The writing reads as heavily LLM-polished, as the AI-use statement acknowledges, and the main storyline gets lost.

W2. Limited novelty, and the setting is not specific to world models. That a distance-based (non-proper) reward collapses the predicted distribution is well known from the point-forecast literature (Gneiting, 2011). It has also been observed as blurring or collapse in video prediction, in trajectory forecasting, and in RL post-training of LLMs (Kirk et al., 2024; GX-Chen et al., 2026; Bereket & Leskovec, 2025). Lemma 1 and Proposition 1 apply to any conditional generative model, so the world-model framing looks like a choice of application rather than a new insight.

W3. Practical importance is unclear. All of the decision-level evidence comes from toy or constructed tasks (FrozenLake, a head-only MiniGrid model, a two-channel task). The only realistic world model, RT-1, has no planner attached. The authors’ own results show little or no harm in PushT, Blackjack, MountainCar and CliffWalking, which suggests the problem may rarely matter in practice.

W4. Weak and fragile results. On RT-1, no post-trained arm beats Base, including MLE, and the main gap is small (2.1%). The larger 7.2% comes from a post-hoc, exploratory 800-update run. In PushT the effect reverses on poorly fitted roots, which is arguably the realistic regime. The MiniGrid result comes from one selected configuration; two other variants did not separate the objectives.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.