acceptodds
Under review as a conference paper at ICLR 2027

RoboCritic: A Grounded Reward Model to Assess and Guide Robot Policies

Abstract

Reliable feedback is critical for reinforcement learning-based robot policy improvement and deployment-time guidance. Terminal rewards summarize completed rollouts, while progress estimates based on temporal cues can diverge from the observed state during stalls or regressions. Scalar scores also leave the underlying physical conditions and appropriate next steps implicit. We introduce RoboCritic, a vision-language reward model that can be queried as observations arrive to assess execution and provide grounded guidance. Given a task instruction and one or more camera streams, RoboCritic maintains bounded causal memory with persistent task context and the initial observation, a sliding visual window, and a history of generated feedback. This design retains execution context without storing the full observation history. At each query, it jointly outputs a progress score, a grounded state description, and actionable next-step or corrective feedback. The final query recovers terminal outcome assessment. We train RoboCritic on RPF-1M, extending RoboReward trajectories with over one million aligned process-feedback records for joint supervision. The frozen critic supplies progress-difference rewards for reinforcement learning and language instructions to a frozen vision-language-action policy. Adding state descriptions and guidance as both supervision and history reduces progress-estimation MAE from 0.441 to 0.404. On two real-world manipulation tasks, RoboCritic achieves higher success than the compared baselines in reward-guided policy learning and language guidance of a shared frozen policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Reward Modeling, Vision-Language Models, Process Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7YmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboCritic, a vision-language model that jointly predicts task progress, describes the observed execution state, and provides actionable guidance for robot policies. The model maintains bounded visual and feedback history to support ongoing assessment. The authors construct RPF-1M by adding aligned process annotations to existing robot trajectories. The same trained critic supplies rewards for policy learning and language guidance during execution. Evaluation includes reward prediction, feedback quality, ablations, and two real-world manipulation tasks, with improvements over the reported baselines.

Critical strengths

The paper addresses a meaningful problem through a coherent formulation. Robot policies benefit from feedback that explains the current physical situation and suggests an appropriate continuation. Combining progress assessment, state descriptions, and corrective guidance is well motivated, particularly when execution deviates from the intended plan. Supporting both policy learning and execution guidance with the same critic makes the formulation practically useful.

The dataset and experiments provide positive evidence for the approach. The aligned process-feedback annotations are a potentially useful contribution, and the evaluation covers several aspects of the proposed system. The ablations support the usefulness of richer feedback and the real-robot experiments demonstrate downstream applicability. Overall, these results make the approach promising.

Critical weaknesses

The distinctive contribution of joint modeling remains insufficiently established. Progress modeling, semantic state understanding, language-based correction, and historical context all have substantial precedent. Their integration through aligned supervision and a shared feedback interface is reasonable, but the paper provides limited insight into what is gained specifically by coupling these capabilities. The schema ablation supports the complete feedback configuration, but changes both supervision and historical context. Consequently, the connection between the proposed joint design and the downstream benefits remains unclear, limiting the significance of the contribution.

The empirical evidence is stronger for progress prediction than for reliable assessment and guidance during failures and recovery. The failure-state evaluation provides useful evidence about prediction quality, but does not directly establish improved failure detection or recovery during execution. The real-robot results are encouraging, although the limited task coverage and lack of reported variability across training runs make their robustness difficult to assess. These limitations leave uncertainty about whether the approach consistently delivers the behavioral advantages that motivate the work.

Questions for authors
  • What do the authors consider the main new insight established by jointly modeling progress, state, and guidance? The schema ablations demonstrate a benefit from the overall configuration; how should that benefit be interpreted in relation to additional supervision, richer history, and improved downstream behavior?

  • Do the downstream gains primarily arise from better guidance during ordinary task progression, or from detecting and correcting failures? An analysis of the existing evaluation trials separating these situations would materially strengthen my understanding of whether the results support the central motivation.

  • Could the authors provide actual failure and recovery demonstrations beyond the schematic example in Appendix C.7, showing when RoboCritic’s feedback successfully guides recovery and when it fails?

Reviewer_GwdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboCritic, a model that jointly predicts a discrete task-progress score, a description of the current state, and textual next-step or corrective feedback. Initialized from Qwen3-VL-4B, RoboCritic is trained on the proposed RPF-1M dataset. A multi-view RoboReward teacher generates progress supervision, followed by temporal smoothing and terminal-consistency filtering, while Seed-Pro 2.0 generates state descriptions and feedback targets. RoboCritic achieves better overall progress-prediction performance than the evaluated baselines on RPF-1M. On two real-world manipulation tasks, its progress-score differences provide effective rewards for online RL using RL-Token, and its textual feedback provides guidance to a frozen VLA while preserving the original task instruction.

Critical strengths

The paper is in general well-written and easy to follow. The overall technical approach is described thoroughly, with details such as prompt templates and the data generation protocol. Through evaluation and ablations are included in the paper.

Critical weaknesses

The important weakness of the paper is its limited novelty. Generating grounded reasoning along with progress prediction has been explored in the literature, e.g., SOLE-R1, ProcVLM, ProgressLM, R^2VLM, as is described in the related work section. The technical approaches presented in this paper do not differ significantly from existing work.

Secondly, it is hard to assess the statistical significance of the results. For example, in Table 2 and 3, RoboCritic and PrefixCritic MAEs are quite close (similar for the ablations in Table 5(a)). It is unclear whether there is substantial difference in performance between the two, or this is simply due to stochasticity in training and evaluation.

Questions for authors
  1. Why is RoboCritic, though distilled from a RoboReward model, outperforming RoboReward significantly in the experiments? Is it because the reward teacher uses multiple views or the filtering process in data generation? What are the reward prediction results (MAE and VOC) for the reward teacher RoboReward model? Is RoboCritic outperforming the reward teacher?
  2. Since the state description and feedback prediction are predicted after the progress prediction, how are they helping to make progress prediction better? Is it through history?
Reviewer_EHqmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RoboCritic, a visionlanguage grounded reward model for robot manipulation. Different from prior scalaronly progress reward models, RoboCritic jointly outputs ordinal progress score (15), structured state description, and actionable corrective guidance. The authors build the RPF1M dataset, derived from existing robot trajectories augmented with dense processlevel annotations including progress scores, state tags and textual feedback. RoboCritic employs bounded causal streaming memory: persistent task/initialobservation context, sliding visual window, and a queue of its own past feedback records. Evaluations are conducted on offline metric benchmarks and two realworld robotic tasks (pickandplace, insertion). Results show improved progress estimation quality, and better realrobot success compared with Robometer / RoboReward baselines.

Critical strengths

1)The motivation is clear: the same scalar score can correspond to very different physical states. Grounding the score in an explicit state description, and allowing the score to drop, addresses a real failure mode of time-based and terminal-only rewards. 2)The evaluation is broad: process scoring, terminal scoring, state and guidance quality, real-robot RL, and language guidance. The gains on the failure subset are substantial (VOC 0.816 vs. 0.552 for Time Proxy). 3)RPF-1M is a useful resource, especially because it includes failed trajectories with process labels. The debouncing and terminal-consistency design is sensible. 4)Bounded streaming memory keeps inference cost constant. The writing is careful about which comparisons cannot isolate individual factors.

Critical weaknesses
  1. Limited statistical power. There are 20 trials per cell. Per task, the gains over Robometer are not significant (Fisher p≈0.11 and 0.34; pooled p≈0.043). Language guidance at 34/40 vs. 30/40 for GPT-5 mini gives p≈0.40.
  2. Unexplained gain over human reward. RoboCritic (75%/55%) outperforms the human-labeled reward (50%/45%) without explanation. A terminal-only reward from the same model is also missing as a baseline.
  3. Label quality and same-source bias. Training labels come from a teacher model and an LLM and are not human-verified. On the test set, the auto-labels have MAE 0.37 against human labels, close to the model's 0.404. The terminal benchmark and the language references share the training pipeline, which disadvantages zero-shot baselines.
  4. Train–test mismatches. Training uses ground-truth history, while inference uses self-generated history (exposure bias). Labels are made at 0.5 Hz, but RL queries the model at 2 Hz. The difference reward in Eq. 5 treats score levels as equally spaced, while Table 7 explicitly says they are not.
  5. Narrow real-world scope. The real-robot study uses one arm (Piper), two tasks and vision only. Language guidance is tested only on the policy trained with RoboCritic's own reward.
Questions for authors

Q1. Why does RoboCritic outperform the human-labeled reward? How dense and how frequent was the human reward? Q2. Can you report more trials or seeds, and a terminal-only reward from RoboCritic? Q3.How large is the failure subset, and how does PrefixCritic perform on it? Q4.Does training with self-generated history (e.g., scheduled sampling) reduce error propagation? Q5.Does the summed difference reward behave like potential-based shaping in practice, and how noisy is it at 2 Hz?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.