- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends the RoboReward-style VLM reward model—originally based on a simple 1–5 scale for progress scoring—into four paradigms: progress scoring, video QA-based scoring, and pairwise comparison versions of both. It introduces POISE, a method that treats pairwise labels as partial orders and employs isotonic regression to adjust pointwise scores, thereby eliminating conflicts where scores contradict pairwise preferences in the training data. The resulting Qwen3-VL-4B model performs on par with GPT-5-mini while achieving higher consistency between scores and pairwise preferences.
Critical strengths
- Existing robot reward models output only point-wise scores, whereas processes such as RLHF, DPO, and Bradley-Terry require pairwise preferences. Since the four paradigms are built upon the same set of video samples, cross-paradigm consistency can be directly measured.
- It requires no additional queries to a teacher model and is supported by a clear projection-based rationale. Operating during the training phase, it complements TrustJudge, which functions during inference. It achieves significantly higher consistency compared to RoboReward-4B and demonstrates improved Spearman correlation with human judgments.
Critical weaknesses
- The improvement over vanilla distillation is marginal, and the results do not report data across multiple random seeds.
- On its original Score-A task, RoboReward-8B actually outperforms the corresponding model from this paper on Score-A and essentially matches the paper's best performance.
Questions for authors
- Add a matching baseline: the original S1∪P1 followed by SFT and then GRPO, while reporting results across multiple random seeds.
- The RoboReward baseline was trained only on Score-A; how were the training and experimental results for the other supervised paradigms in this paper obtained?
Thanks for the responses. Unfortunately, I think my first concern was not really addressed directly. Just because the reward model better agrees with human judgement doesn't mean that it will lead to policy improvement in an RL retraining loop. I think demonstrating this would make the paper much stronger. All things considered, I will stick with my original score. Best of luck to the authors!