acceptodds
Withdrawn from ICLR 2027

WAVE: World-Model-Augmented Value Learning for Action-Chunk Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action chunking provides a temporal abstraction for long-horizon reinforcement learning, but a chunk critic must infer what an entire H-step action sequence will accomplish from reward and temporal-difference supervision alone. A world model can predict these consequences, but substituting its predicted endpoints into the Bellman backup lets model error change the preferred chunk: we construct an example in which a small endpoint error yields a suboptimal policy despite exact rewards and an exactly solved Bellman equation. With observed endpoints, by contrast, the Bellman fixed point is optimal for any world model, so model error can affect only what the critic represents. We therefore introduce WAVE (World-Model-Augmented Value Learning), which conditions the critic on a causal world-model posterior through a zero-initialized residual adapter and trains its features to predict the model's current latent state and the H-step consequence of the same chunk, while every Bellman target uses real rewards and observed endpoints. On two OGBench manipulation tasks with three seeds each, WAVE improves on Q-Chunking, raising the area under the success curve over 1M online interactions from 0.832 to 0.929 on Cube and from 0.922 to 0.956 on Scene and reaching 70% and 80% success with 76–85% fewer real interactions on average; substituting predicted endpoints from the same world model instead yields near-zero success. Ablations attribute most of the auxiliary objective's gain to predicting future rather than current latent states.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, action chunking, world models, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_axGZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes WAVE, which adds the latent feature of the history trained with the auxillary world model loss on the Q function. Policy is extracted by BC rejection sampling. WAVE improves the performance upon QC in offline-to-online RL in Scene and Cube environments in OGBench.

Critical strengths

S1. Novelty of the methodology.

  • Incorporating history-conditioned latent into the critic is an interesting approach to improving value estimation.
  • However, the motivation for using a history-conditioned RSSM representation is not sufficiently justified, and the experiments do not clearly establish why this particular design is necessary.
Critical weaknesses

W1. Insufficient comparisons for the claim.

  • The key difference between WAVE and QC is that WAVE incorporates a history-conditioned latent representation learned through an auxiliary prediction loss. The authors argue that world-model representations should be used as inputs to the critic rather than for constructing value targets.
  • However, the current comparison between WAVE and QC does not sufficiently support this claim. In particular, it is unclear whether the improvement comes from incorporating historical information into the critic, learning predictive representations through the auxiliary world-model loss, or their combination.
  • To disentangle these factors, I encourage the authors to include the following ablations:
  1. QC with an RSSM encoder: Use only the RSSM latent state (z_t) and conditioning variable (c) as critic inputs, without the current observation (o_t).
  2. QC with observation history: Replace the RSSM latent representation with a history of observations (e.g., stacked observations) as critic inputs, to determine whether the improvement comes from history conditioning or the learned world-model representation.

W2. Limited experimental evaluation.

  • WAVE is compared only against QC, despite the existence of several relevant offline RL baselines, including FQL [1], ReBRAC [2], and FINO [3].
  • Furthermore, the experiments are conducted on only two tasks (scene-play-singletask-task5-v0 and cube-triple-singletask-task5-v0). This evaluation is too limited to show whether WAVE consistently outperforms existing methods.
  • I strongly encourage the authors to extend the evaluation to a broader set of tasks, ideally covering the 25 visual OGBench tasks evaluated in QC, and to include comparisons with additional baselines. Results reported in the QC and FINO papers could serve as useful references.

[1] Seohong Park et al., Flow Q-Learning, ICML 2025

[2] Denis Tarasov et al., Revisiting the Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2023

[3] Yongjae Shin et al., Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning, ICLR 2026

Questions for authors

Q1. Adding comparisons (W1)

  • Can the authors provide additional ablation studies on QC with an RSSM encoder and QC with an observation history?
  • It will help to disentangle the contributions of history conditioning / RSSM representation from the WAVE critic formulation.

Q2. More evaluation (W2)

  • Can the authors expand the experimental evaluation to include more OGBench tasks and additional baselines, such as FQL, ReBRAC, and FINO?
  • It is difficult to assess the generality of WAVE's improvements in this setting.
Reviewer_5D7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WAVE, a world-model-augmented approach to improving the sample efficiency of action-chunk reinforcement learning. WAVE enhances a pretrained Q-Chunking critic through a causal world-model posterior and an auxiliary dynamics consistency objective that predicts future latent states, while retaining real transitions for Bellman targets. The authors also provide a theoretical analysis of the risks of using model-predicted endpoints for value bootstrapping. Experiments on two OGBench manipulation tasks demonstrate improved online sample efficiency over Q-Chunking and several other baselines.

Critical strengths
  1. Strong improvements in online sample efficiency. WAVE substantially improves the learning efficiency of Q-Chunking on both evaluated tasks. The comparisons with multiple baselines, together with component and data-scale ablations, provide useful empirical evidence for the proposed approach.
  2. A well-motivated integration of world models and value learning. The distinction between using world-model predictions for representation learning and directly incorporating them into Bellman targets is meaningful. The theoretical analysis and Direct-WM experiments provide complementary support for this design choice.
Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for horizon alignment. Predictive representation learning has been extensively explored in prior work, including SPR, TD-MPC, and SALE. Although WAVE adapts this idea to action-chunk critics, its H-step future prediction achieves nearly identical AUC200k to one-step prediction (0.735 vs. 0.736 in Table 2). This weakens the evidence that horizon-aligned supervision is a distinctive or necessary component of the method.
  2. Limited experimental generality. The evaluation includes only two simulated manipulation tasks with low-dimensional state observations and three seeds. No visual-input or real-world robotic experiments are provided. This makes it difficult to assess whether the reported improvements generalize to more challenging robotic learning settings, particularly given the additional complexity of training and maintaining a world model.
Questions for authors

See Weaknesses.

Reviewer_dd6eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

WAVE augments an action-chunking critic with representations from a separately trained recurrent state-space model (RSSM). The world model provides recurrent state representations and auxiliary predictive targets to improve the critic’s representations. Both models are updated during training using their respective objectives, while critic gradients are prevented from updating the world model. The method is evaluated for offline-to-online sample efficiency on two OGBench manipulation tasks.

Critical strengths

None

Critical weaknesses
  1. The limited experimental coverage and number of seeds constrain the empirical conclusions.

The evaluation covers only cube-triple task5 and scene task5. Although the results are promising, experiments on these two tasks provide limited evidence of generalization across environments with different dynamics, reward structures, and exploration requirements. Evaluation with additional action-chunking learners would also help establish whether the benefits extend beyond the chosen backbone.

Although statistical significance tests are reported, results from only three independent seeds provide limited evidence about run-to-run reliability. Additional seeds and confidence intervals for performance differences would strengthen the empirical conclusions.

  1. The benefits of separating the world model and critic are not sufficiently isolated.

The component ablations provide useful evidence that world-model-derived representations benefit the critic. However, they do not establish whether training the RSSM separately is preferable to sharing representations or jointly training the world model and critic.

The sensitivity of the method to RSSM architecture and training choices also remains unclear. Without these comparisons, it is difficult to determine whether the improvements reflect a broadly useful design principle or depend on the particular configuration used in the experiments.

Questions for authors
  1. Could the authors evaluate WAVE on additional tasks with different dynamics and reward structures, and with other action-chunking learners?

  2. Could the authors compare the separate-model design with shared-representation or jointly trained alternatives, controlling for model capacity and training budget where possible? How sensitive are the results to RSSM architecture and training hyperparameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.