acceptodds
Under review as a conference paper at ICLR 2027

WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

Abstract

Reinforcement learning (RL) can improve Vision-Language-Action (VLA) policies beyond supervised fine-tuning, but critic-based methods depend on value estimates that must account for partial observability. Critics operating on current-observation features can miss temporal information relevant to task progress, while simply adding observation history does not reliably improve policy performance in our experiments. We propose the World Critic Model (WCM), a history-conditioned critic that combines value estimation with action-conditioned latent prediction. Built on a lightweight LeJEPA-based architecture, WCM encodes a short observation history and a language instruction into a shared temporal representation, from which it estimates returns and predicts the next observation latent under the executed action. The prediction objective complements return supervision by encouraging the representation to retain transition-relevant information, without requiring pixel reconstruction or imagined rollouts for value estimation. WCM integrates into both on-policy and off-policy RL pipelines and supports diverse VLA backbones. Experiments spanning 149 tasks across four simulation benchmarks demonstrate improvements in policy performance and generalization, with ablations supporting the benefit of combining history with latent prediction. Evaluations on seven real-world manipulation tasks further demonstrate the effectiveness of WCM in off-policy VLA reinforcement learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action models, Reinforcement Learning, Robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kHe7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduce World Critic Model (WCM), a critic model for VLA reinforcement learning. Besides the typical input of state action pair of the current timestamp for a critic model, the proposed model additionaly takes observation history, language instruction, and predicts not only the value, but also the future latent representation. The proposed method was tested using both on-policy and off-policy settings on various simulation and real-world benchmarks, and shows improvement over baseline critics even with distribution shifts and temporally ambiguous settings.

Critical strengths

The paper is easy to follow, the experiments cover various of settings that support the conclusion on the history conditioning in VLA-RL critics. These experiments provide emprical evidence for people who want to build critic using history under different settings (models and envs).

Critical weaknesses

This paper mainly focus on the history length when designing a critic model for VLA-RL, which is a limited subset of a so called World Critic Model design space. Though the history length is an important factor, more fundamental problem according to my understanding is the effect of future prediction, which turns a critic model into a world critic model conceptually.

I appreciate the indirect evidence in table 4 shows the importance of all these designs e.g. SIGReg, future prediction and so on, it would bring more insights if the authors could show the success rate of some of the variants in table 4. The new experiments would also support the second contribution better.

Questions for authors

No

Reviewer_ruUTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a world critic model as a history-conditioned critic that combines value estimation with action-conditioned latent prediction. Built on LeJEPA, WCM encodes short observation history and language instructions into a shared temporal representation to estimate returns and predict the next observation latent under the executed action. The critic model integrates into both on-policy and off-policy RL pipelines and supports multi VLA backbones.

Critical strengths
  1. The paper provides a comprehensive evaluation spanning many tasks across four simulation benchmarks, three distinct VLA backbones, and seven real-world robotic manipulation tasks. The proposed critic model demonstrates consistent improvements over the evaluated baseline critics, particularly under out-of-distribution conditions and challenging manipulation scenarios.
  2. The paper is well-motivated. The method is presented in a clear and well-structured manner, with intuitive explanations of the architecture and training objectives, making the overall approach easy to understand and follow.
Critical weaknesses
  1. Limited methodological novelty. The core idea of jointly learning value estimation and latent dynamics prediction through a shared representation has been extensively explored in prior model-based RL works, such as SPR, DeepMDP and JOWA, ... The method primarily combines existing techniques, within the VLA-RL setting. Overall, the proposed approach appears rather conventional and lacks a compelling or particularly novel methodological insight.
  2. Although the paper's ablation studies demonstrate performance improvements from adding the prediction objective, they do not sufficiently prove that these gains arise from improved temporal dynamics understanding rather than the additional self-supervised training signal.
  3. The proposed method uses next-latent prediction only to regularize the critic representation, without incorporating predicted dynamics into value estimation or planning. However, the paper does not clearly justify this design choice or investigate whether directly leveraging the learned world model for value learning or planning could provide additional benefits. Comparisons with model-based alternatives would help clarify the advantages and limitations of the proposed approach.
Questions for authors
  1. Why is next-latent prediction used only as an auxiliary objective rather than directly for value estimation or planning?
  2. What advantages does the WCM design offer over existing model-based approaches?
Reviewer_zeoYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new critic model for VLA reinforcement learning to replace the critics in existing RL algorithms. Alongside value prediction, the critic uses observation history and the executed action to predict the next observation’s latent representation as an auxiliary training objective. The method is evaluated on four simulation benchmarks and seven real-world tasks, with results demonstrating its effectiveness.

Critical strengths
  1. Extensive experiments. The paper evaluates the proposed method on four simulation benchmarks and seven real-world tasks, covering multiple VLA backbones and both on-policy and off-policy RL settings. These results demonstrate its effectiveness across different tasks and training settings.

  2. Ablation studies support the proposed components. The paper includes ablations on observation history length, the next-latent prediction objective, action conditioning, and SIGReg. These experiments help demonstrate the contributions of the individual components to the overall method.

Critical weaknesses
  1. The methodological novelty appears limited. The main technical components, including history-conditioned representations, action-conditioned latent prediction, and SIGReg regularization, have been explored in prior work such as PBL, SPR, and LeWorldModel. The contribution therefore appears to lie primarily in adapting and combining these techniques for VLA critic learning. The paper could more clearly identify the technical advances beyond this integration.

  2. The explanation of why the method works could be clearer. Although the paper provides a reasonable motivation and ablation results, the connection between the auxiliary prediction objective and more accurate value estimates remains insufficiently established. A more direct analysis would help explain the mechanism behind the improvements.

Questions for authors
  1. Prediction horizon. Since VLA policies often execute action chunks, how does WCM’s one-step prediction align with the executed action chunk? It would be helpful to include an ablation with different prediction horizons to examine their effects on value estimation and policy performance.

  2. Figure readability (presentation suggestion). Could the text in the figures be enlarged? In particular, the labels in the two bar charts within the “Real-World RL Training” part of Figure 1 are barely readable.

Comments on the paper

1 comment

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.