- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduce World Critic Model (WCM), a critic model for VLA reinforcement learning. Besides the typical input of state action pair of the current timestamp for a critic model, the proposed model additionaly takes observation history, language instruction, and predicts not only the value, but also the future latent representation. The proposed method was tested using both on-policy and off-policy settings on various simulation and real-world benchmarks, and shows improvement over baseline critics even with distribution shifts and temporally ambiguous settings.
Critical strengths
The paper is easy to follow, the experiments cover various of settings that support the conclusion on the history conditioning in VLA-RL critics. These experiments provide emprical evidence for people who want to build critic using history under different settings (models and envs).
Critical weaknesses
This paper mainly focus on the history length when designing a critic model for VLA-RL, which is a limited subset of a so called World Critic Model design space. Though the history length is an important factor, more fundamental problem according to my understanding is the effect of future prediction, which turns a critic model into a world critic model conceptually.
I appreciate the indirect evidence in table 4 shows the importance of all these designs e.g. SIGReg, future prediction and so on, it would bring more insights if the authors could show the success rate of some of the variants in table 4. The new experiments would also support the second contribution better.
Questions for authors
No
Dear Reviewers,
I would like to kindly remind everyone to review the authors' responses and provide your feedback.
If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.
Thank you all very much for your time and effort! And, have a nice weekend. :)
Best regards,
AC