acceptodds
Withdrawn from ICLR 2027

MetaReach: Meta-Reachability Group Relative Policy Optimization for Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conditional video world models predict future visual states under given actions, but post-training video world models with group relative policy optimization (GRPO) remains challenging. Existing methods freeze the decoder and compute rewards against ground-truth frames, introducing an irreducible residual that distorts candidate ranking because the decoder's reachable space is limited (target-set mismatch). They also broadcast a single sequence-level advantage to all token blocks, ignoring the causal structure of autoregressive generation (long-horizon credit misallocation). We propose MetaReach, a bi-level meta-learning framework that jointly addresses both issues. The inner loop adapts the decoder to reconstruct frames within the world model's output distribution, and the outer loop updates the world model under the adapted decoder. This reachability-calibrated meta-learning (RCM) eliminates target-set mismatch at its source. We further introduce autoregressive credit assignment (ACA), which assigns a position-specific advantage to each token block according to its future horizon. Gradient projection preserves the original objective. Experiments on RT-1, RoboNet, and VP2-RoboSuite on a 15-frame prediction horizon show that MetaReach consistently outperforms GRPO-based baselines and recent long-horizon post-training methods, reducing LPIPS by 7.7% and MSE by 13.3% over RLVR-World on RT-1.

Primary area:
optimization
Keywords:
Video World Models, Group Relative Policy Optimization, Meta-Learning, Reachability Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dwe6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetaReach post-trains action-conditioned video world models through decoder adaptation and world-model updates. Reachability-calibrated rewards compare predictions with reconstructed targets, while autoregressive credit assignment supplies block-wise future returns. A gradient projection combines calibrated and raw objectives. Reported results improve short-horizon visual prediction quality.

Critical strengths
  1. The method jointly addresses reward-target construction and temporal credit assignment, with component ablations supporting both.
  2. On RT-1, MetaReach reduces LPIPS by 7.7% and MSE by 13.3% relative to RLVR-World.
Critical weaknesses
  1. The cost–benefit comparison is incomplete. Each iteration includes three decoder-adaptation steps, 16 candidate rollouts, and gradients for both raw and calibrated objectives. Training time, GPU-hours, and equal-compute comparisons are absent, making the value of the additional computation unclear.

  2. Long-horizon and control benefits remain unvalidated.

    • RT-1 and RoboNet predictions cover only 15 frames; VP2-RoboSuite uses eight. This provides limited support for the central long-horizon credit-assignment claims.
    • Evaluation relies on appearance metrics without downstream planning or control success rates. Better LPIPS/MSE does not establish useful action-conditioned dynamics, particularly given the acknowledged preference of these rewards for static or blurry predictions.
Questions for authors
  1. What are training wall-clock time and GPU-hours relative to the baselines? Do gains persist under equal compute?
  2. Does ACA improve substantially longer autoregressive rollouts, and how does its advantage change with horizon?
  3. Do the prediction gains improve downstream planning or control success under the same controller and interaction budget?
Reviewer_KjSROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MetaReach proposes a post-training method for discrete-token, autoregressive, action-conditioned video world models. It addresses two issues: comparing decoded predictions with ground-truth frames outside the decoder's reachable space, and assigning the same sequence-level advantage to every token block despite their different effects on later predictions.

The method has three components. RCM adapts the decoder for Ninner=3N_{inner}=3 steps using the reconstruction loss ∥Dϕ(E(xt))−xt∥2\|D_\phi(E(x_t))-x_t\|^2 (Eq. 4), retaining the decoder parameters across iterations. It then replaces the ground-truth reward reference with the reachable reconstruction x~t=Dϕ′(E(xt))\tilde x_t=D_{\phi'}(E(x_t)); candidates are decoded by the same adapted decoder (Eqs. 5–6). ACA replaces the shared sequence-level advantage with discounted reward-to-go, using γ=0.95\gamma=0.95 and normalization across the K=16K=16 candidates at each timestep (Eq. 10). A gradient projection adjusts the calibrated-reward gradient g1g_1 toward the raw-reward gradient g0g_0 to satisfy ⟨g,g0⟩≥∥g0∥2\langle g,g_0\rangle\geq\|g_0\|^2 (Eqs. 12–13). Algorithm 1 returns both the updated world model and the adapted decoder.

Tables 1–3 report 15-frame prediction results on RT-1 and RoboNet. On RT-1, the full method improves LPIPS by 7.7% and MSE by 13.3% over RLVR-World and obtains the best result on all five reported metrics. The ablations remove RCM or ACA, vary the number of inner steps and examine per-frame prediction quality. Fig. 3 reports results on held-out scenes. VP2-RoboSuite is described in the setup and included in the claims in the abstract and conclusion, but its results are not reported. The paper refers both to five paired seeds and to averaging the last five checkpoints, without reporting variability.

Critical strengths
  1. The interaction between the decoder and post-training is worth studying. Rewards are computed on decoded frames, so decoder error can affect the supervision received by a token-based world model. Temporal credit assignment is also relevant, since an early token block conditions later frames. MetaReach combines these two concerns without introducing a learned value network. The discussion of appearance-based rewards favoring static or blurry predictions is a useful acknowledgment of a remaining limitation (p9 L480–482).

  2. The RT-1 improvements are consistent across metrics. Table 1 reports LPIPS of 0.20856, compared with 0.21506 for WorldCompass and 0.22591 for RLVR-World, together with improvements in MSE, PSNR, SSIM and final-frame LPIPS. Removing either RCM or ACA worsens the reported results. The inner-step and per-frame studies in Tables 2–3 also examine questions relevant to the proposed components. These results motivate a closer look at the method, although the concerns below make the contribution of each component difficult to assess.

Critical weaknesses
  1. The RCM explanation and the decoder ablation are not convincing yet.

    The introduction attributes ranking distortion to an "irreducible fixed residual for all candidates" (p1 L50–52). An additive residual shared by all candidates cannot change their ranking or the normalized advantages in Eq. 3: the group mean shifts by the same amount, and the standard deviation stays unchanged. For squared error,

    ∥x^i−x∥2=∥x^i−x~∥2+∥x~−x∥2+2⟨x^i−x~,x~−x⟩\|\hat x_i-x\|^2=\|\hat x_i-\tilde x\|^2+\|\tilde x-x\|^2+2\langle\hat x_i-\tilde x,\tilde x-x\rangle.

    The middle term is constant across candidates. The cross term can change their ranking relative to calibrated squared error, but the paper does not examine its effect. LPIPS requires a separate analysis because it is nonlinear. Sec. 4.1 promises pairwise and top-1 reward accuracy (p7 L369–371), yet neither result is given and the reference ranking is unspecified. These measurements are needed to support the claim that RCM corrects ranking distortion. In particular, when the raw and calibrated rewards prefer different candidates, it remains unclear which choice produces the better rollout.

    The description of the inner loop also does not match Eq. 4. The abstract says that the decoder adapts to the world model's output distribution, whereas Eq. 4 reconstructs ground-truth frames from their own codes. It contains no world-model samples or parameters, and the outer loop holds the adapted decoder fixed (p5 L233–256). As written, the algorithm alternates supervised decoder adaptation with policy optimization. The authors need to explain where the dependence on the world model enters, or revise the "bi-level meta-learning" description to match the stated objective.

    More importantly, the experiments do not separate reward calibration from improved decoding. Decoder parameters are carried across training iterations, Algorithm 1 returns the adapted decoder, and decoded predictions are fed back into subsequent rollout steps. The decoder used for each method's evaluation is not clearly specified. If MetaReach uses the adapted decoder and baselines use the original one, an improvement in reconstruction could account for some of the gains attributed to better world-model training. A common-decoder comparison and a decoder-only adaptation baseline would help resolve this.

    The existing ablations make this question particularly relevant. On RT-1, "w/o RCM" gives LPIPS 0.21487 (Table 1), static calibration with Ninner=0N_{inner}=0 gives 0.21750, and three inner steps give 0.20856 (Table 2). If the other settings match, changing the reward target alone worsens LPIPS by 0.00263, while enabling adaptation improves it by 0.00894. The improvement therefore appears only after decoder training is introduced. Before attributing that change to better reward comparisons, the paper needs to establish how much decoder adaptation achieves on its own and whether these rows use the same ACA and evaluation settings.

  2. Several results and experimental details need to be reconciled.

    The abstract and conclusion claim experiments on RT-1, RoboNet and VP2-RoboSuite over a 15-frame horizon, but no VP2 results are presented. Sec. 4.1 also specifies a different VP2 protocol: predicting eight frames from two input frames. The reported coverage of the evaluation should match the experiments actually shown.

    There is a consequential discrepancy in the ACA ablation. For RT-1 without ACA, Table 1 gives final-frame LPIPS 0.23418 and Table 3 gives 0.245. The full-model values agree to rounding, at 0.22631 and 0.226. The discrepancy changes the measured size of the ACA effect and needs to be resolved before Table 3 is used to support the claimed horizon-dependent benefit.

    Table 1 refers to five paired seeds, while Sec. 4.1 says the results average the last five checkpoints. It is unclear whether checkpoint averaging was performed within each seed, or whether the two descriptions refer to different evaluations. Neither table provides uncertainty estimates. This is important on RoboNet, where the full model improves LPIPS over WorldCompass by only 0.00041, about 0.52%, and both single-component variants are worse than WorldCompass. Paired intervals would make these differences much easier to assess.

    Finally, the gradient projection and the relevant baseline implementations need more detail. The projection constraint concerns first-order descent for an idealized gradient-descent step; it does not establish the same guarantee for the actual AdamW updates. A g1g_1-only ablation would show what the calibrated reward contributes without the projected raw-reward direction. For ACA, the paper cites Shao et al. but does not specify which form of GRPO the "Seq-GRPO" baseline implements. Its discounting and normalization should be compared explicitly with the process-supervised formulation. The adaptations of Self Forcing and WorldCompass to discrete-token prediction, and the split used for the unseen RoboNet scenes, are also insufficiently described.

I recommend rejection in the current form. The RT-1 gains are encouraging, but their attribution to RCM is unresolved, and missing results and inconsistent ablation numbers weaken the evidence. The decoder controls and direct ranking analysis are the most important points to address.

Questions for authors
  1. Which decoder is used during rollout and evaluation for MetaReach, its ablations and each baseline? Could you compare the post-trained world model with the original decoder, and the original world model with a decoder trained by Eq. 4 alone? Please distinguish changing the decoder used for rollout conditioning from changing only the final decoding.

  2. Where are the pairwise/top-1 reward-accuracy results announced in Sec. 4.1, and what is their reference ranking? How often do raw and calibrated rewards select different candidates? When they disagree, which selection leads to better ground-truth rollout quality?

  3. What exactly differs between "w/o RCM" in Table 1 and Ninner=0N_{inner}=0 in Table 2? Do they use the same ACA and evaluation settings? Also, why does final-frame LPIPS for w/o ACA change from 0.23418 in Table 1 to 0.245 in Table 3?

  4. Please provide the VP2-RoboSuite results and clarify the prediction horizon. How are the five paired seeds and last-five-checkpoint averages combined? Please report uncertainty for the main paired comparisons, especially on RoboNet.

  5. Does the implemented inner-loop loss depend on world-model predictions or parameters in a way omitted from Eq. 4? What is the exact Seq-GRPO configuration, and how were Self Forcing and WorldCompass adapted? A g1g_1-only comparison would help assess the gradient projection; please also describe the discount-factor choice and the held-out scene split.

Two smaller reporting suggestions: include the extra training cost of decoder adaptation and the second gradient computation, and add WorldCompass to Fig. 3 so that the held-out comparison includes the strongest baseline in Table 1.

Reviewer_N2SROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MetaReach, a post-training framework for autoregressive video world models. The method combines Reachability-Calibrated Meta-learning (RCM), Autoregressive Credit Assignment (ACA), and a gradient-projection update. RCM adapts the visual decoder using a reconstruction objective and replaces ground-truth reward references with reconstructions produced by the adapted decoder. ACA assigns each frame's token block an advantage based on discounted future rewards, normalized across sampled trajectories at the same temporal position. Experiments report improved prediction metrics on RT-1 and RoboNet, together with component ablations and selected qualitative examples. Although VP2-RoboSuite is included in the stated evaluation scope, its results are absent from the supplied manuscript.

Critical strengths
  1. The reported RT-1 results provide a preliminary positive empirical signal. Within the author's own evaluation protocol, MetaReach improves mean LPIPS and MSE relative to RLVR-World. These improvements should be acknowledged.
Critical weaknesses
  1. The motivation for Reachability-Calibrated Meta-learning is not convincing, and the necessity of the meta-learning formulation remains unclear.

    The proposed RCM appears much closer to alternating optimization of two modules than to a clearly motivated meta-learning framework. The decoder is adapted using a standard reconstruction objective, after which the adapted decoder is treated as fixed while updating the world model. The paper does not clearly identify what is actually meta-learned, such as an initialization, adaptation rule, or other quantity optimized for post-adaptation performance. As currently formulated, introducing an "inner loop" for decoder adaptation and an "outer loop" for world-model optimization does not appear substantially different from separately and alternately optimizing the decoder and the world model.

    More importantly, the paper does not establish why a meta-learning formulation is necessary for addressing target-set mismatch. Several simpler alternatives seem equally plausible, such as jointly optimizing the decoder and world model, fine-tuning the decoder before standard GRPO, or using reconstructed targets with a fixed decoder. Since the experiments do not compare against these alternatives, it is difficult to attribute the reported gains specifically to the proposed formulation rather than to ordinary decoder adaptation.

  2. The empirical evidence is not strong enough to establish the effectiveness of the proposed method. The reported gains over the strongest baselines are modest, particularly on RoboNet, where MetaReach only marginally improves over WorldCompass in LPIPS and MSE. Moreover, although the paper claims evaluation on VP2-RoboSuite, the main quantitative results only cover RT-1 and RoboNet, leaving the claim in Lines 28-29 incomplete.

    More importantly, the evaluation focuses almost entirely on prediction-quality metrics such as LPIPS, MSE, PSNR, and SSIM. In Lines 411-412, the authors further suggest that the qualitative improvements are important for downstream control, yet no planning success rate, policy performance, or other control-oriented evaluation is provided to support this claim. This is a significant limitation for a world-model paper, since improvements in visual prediction metrics do not necessarily imply improvements in decision making. Closely related works cited in the paper, including iVideoGPT, RLVR-World, and WorldCompass, complement perceptual prediction metrics with evaluations that directly measure action prediction quality or policy performance, such as action-related prediction metrics, planning success rates, or policy evaluation scores. Without comparable downstream evaluations, it remains unclear whether MetaReach actually improves the usefulness of the learned world model for decision making, rather than only improving reconstruction-oriented visual metrics.

Questions for authors
  1. What is actually meta-learned in RCM? The inner loop only fine-tunes the decoder with a standard reconstruction loss, while the outer loop freezes the adapted decoder and updates the world model. How does this differ from standard alternating optimization, and why is a meta-learning formulation necessary?

  2. Where are the VP2-RoboSuite results?

  3. Please provide uncertainty estimates for the main results across the five random seeds reported in Table 1. This would help assess the statistical reliability of the improvements and determine the appropriate number of significant digits for the reported metrics.

  4. Could the authors provide more qualitative comparisons across different scenes and benchmarks to better assess the actual generation quality and failure cases of MetaReach relative to the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.