- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes WAVE, which adds the latent feature of the history trained with the auxillary world model loss on the Q function. Policy is extracted by BC rejection sampling. WAVE improves the performance upon QC in offline-to-online RL in Scene and Cube environments in OGBench.
Critical strengths
S1. Novelty of the methodology.
- Incorporating history-conditioned latent into the critic is an interesting approach to improving value estimation.
- However, the motivation for using a history-conditioned RSSM representation is not sufficiently justified, and the experiments do not clearly establish why this particular design is necessary.
Critical weaknesses
W1. Insufficient comparisons for the claim.
- The key difference between WAVE and QC is that WAVE incorporates a history-conditioned latent representation learned through an auxiliary prediction loss. The authors argue that world-model representations should be used as inputs to the critic rather than for constructing value targets.
- However, the current comparison between WAVE and QC does not sufficiently support this claim. In particular, it is unclear whether the improvement comes from incorporating historical information into the critic, learning predictive representations through the auxiliary world-model loss, or their combination.
- To disentangle these factors, I encourage the authors to include the following ablations:
- QC with an RSSM encoder: Use only the RSSM latent state (z_t) and conditioning variable (c) as critic inputs, without the current observation (o_t).
- QC with observation history: Replace the RSSM latent representation with a history of observations (e.g., stacked observations) as critic inputs, to determine whether the improvement comes from history conditioning or the learned world-model representation.
W2. Limited experimental evaluation.
- WAVE is compared only against QC, despite the existence of several relevant offline RL baselines, including FQL [1], ReBRAC [2], and FINO [3].
- Furthermore, the experiments are conducted on only two tasks (
scene-play-singletask-task5-v0andcube-triple-singletask-task5-v0). This evaluation is too limited to show whether WAVE consistently outperforms existing methods. - I strongly encourage the authors to extend the evaluation to a broader set of tasks, ideally covering the 25 visual OGBench tasks evaluated in QC, and to include comparisons with additional baselines. Results reported in the QC and FINO papers could serve as useful references.
[1] Seohong Park et al., Flow Q-Learning, ICML 2025
[2] Denis Tarasov et al., Revisiting the Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2023
[3] Yongjae Shin et al., Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning, ICLR 2026
Questions for authors
Q1. Adding comparisons (W1)
- Can the authors provide additional ablation studies on QC with an RSSM encoder and QC with an observation history?
- It will help to disentangle the contributions of history conditioning / RSSM representation from the WAVE critic formulation.
Q2. More evaluation (W2)
- Can the authors expand the experimental evaluation to include more OGBench tasks and additional baselines, such as FQL, ReBRAC, and FINO?
- It is difficult to assess the generality of WAVE's improvements in this setting.