- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how post-training world models with point prediction rewards improves sample quality while distorting probabilities over valid futures. It introduces an excess-gain metric that bounds the Wasserstein-1 distance to the ground truth when point risk drops below the truth's intrinsic uncertainty. The authors prove that GRPO group standardization biases energy-score gradient updates, collapsing to point-reward updates in the binary case. Evaluations across PushT, RT-1, FrozenLake, MiniGrid, and synthetic domains demonstrate conditional decision degradation, show that proper-score fine-tuning can recover performance, and advocate reporting distributional metrics alongside sample fidelity.
Critical strengths
- Concrete, checkable distortion metric: The excess-gain lower bound transforms an apparent improvement below intrinsic risk into an actionable indicator of distributional mismatch, supported by direct probability evaluations in PushT.
- Clear update-rule bias analysis: The paper demonstrates the equivalence between standardized energy-score updates and point rewards for binary outcomes and provides an actionable leave-one-out (LOO) credit assignment alternative.
Critical weaknesses
- Constrained decision evidence and setup sensitivity: Task degradation is demonstrated primarily in small, controlled environments. In MiniGrid, observable harm requires freezing the backbone and training only the prediction head, failing to separate cleanly in end-to-end alternatives.
- Incomplete video model validation: The RT-1 study uses only 256 training windows and omits closed-loop control evaluations; none of the variants beats the base model on joint energy score, and larger gains at extended steps are confounded by drift from Base.
Questions for authors
- Q1: Beyond the binary equivalence and LOO derivations for energy scores, what is the central conceptual advance over existing analyses of GRPO miscalibration (e.g., Bereket & Leskovec, 2025)?
- Q2: Why does decision harm appear in MiniGrid only when the CNN backbone is frozen, but fails to separate cleanly in trainable-backbone configurations?
- Q3: In the 800-update RT-1 experiments, can the performance deficit of ES-GRPO relative to ES-LOO be disentangled from its greater divergence from the base checkpoint?