acceptodds
Under review as a conference paper at ICLR 2027

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

Abstract

Reward models are a core bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision-based feedback, which cannot be fully satisfied by manually crafted reward functions or task-specific human annotations. Existing open-source VLM reward judges such as RoboReward adopt a simple 1–5 trajectory progress scoring scheme, lacking pairwise preferences essential for modern RLHF, DPO and Bradley-Terry frameworks, while also failing to optimize agents' video scene understanding directly. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inherent inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance—an issue that aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with the Preference-Ordered Isotonic Score Editing (POISE) module. We construct a unified four-paradigm dataset containing trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise preference counterparts (Pair-A, Pair-B). We find pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. The POISE module rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE provably reduces training-corpus score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge still retains 20.46% reversal conflicts on the same training corpus. Evaluated on our multi-paradigm benchmark, the Qwen3-VL-4B model trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding both RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Further integrating TrustJudge aggregation during inference boosts our model's overall score to 78.57%, surpassing the proprietary GPT-5-mini teacher model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, embodied AI, vision-language models, reward modeling, preference learning, reinforcement learning, isotonic regression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E3WKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the RoboReward-style VLM reward model—originally based on a simple 1–5 scale for progress scoring—into four paradigms: progress scoring, video QA-based scoring, and pairwise comparison versions of both. It introduces POISE, a method that treats pairwise labels as partial orders and employs isotonic regression to adjust pointwise scores, thereby eliminating conflicts where scores contradict pairwise preferences in the training data. The resulting Qwen3-VL-4B model performs on par with GPT-5-mini while achieving higher consistency between scores and pairwise preferences.

Critical strengths
  1. Existing robot reward models output only point-wise scores, whereas processes such as RLHF, DPO, and Bradley-Terry require pairwise preferences. Since the four paradigms are built upon the same set of video samples, cross-paradigm consistency can be directly measured.
  2. It requires no additional queries to a teacher model and is supported by a clear projection-based rationale. Operating during the training phase, it complements TrustJudge, which functions during inference. It achieves significantly higher consistency compared to RoboReward-4B and demonstrates improved Spearman correlation with human judgments.
Critical weaknesses
  1. The improvement over vanilla distillation is marginal, and the results do not report data across multiple random seeds.
  2. On its original Score-A task, RoboReward-8B actually outperforms the corresponding model from this paper on Score-A and essentially matches the paper's best performance.
Questions for authors
  1. Add a matching baseline: the original S1∪P1 followed by SFT and then GRPO, while reporting results across multiple random seeds.
  2. The RoboReward baseline was trained only on Score-A; how were the training and experimental results for the other supervised paradigms in this paper obtained?
Reviewer_XC3eOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a multi-paradigm reward modeling method for robot manipulation, adding video-QA scoring and pairwise preference supervision. The main contribution is POISE, a module that treats pairwise labels as a partial order and edits pointwise reward scores via isotonic regression so that the training corpus contains no score-pair reversal conflicts. The authors train Qwen3-VL models on the corrected scores, report performance with marginally improved score-pair consistency, and demonstrate reward-guided trajectory optimization in small scale.

Critical strengths
  • The paper identifies a real phenomenon that pointwise reward scores and pairwise preferences from the same judge may conflict and the proposed fix is conceptually clean.
  • Framing pairwise labels as a target partial order and applying isotonic projection is a simple, linear-time, theoretically grounded correction.
Critical weaknesses

1. The motivating gap and the baseline set ignore important prior work that natively outputs both pointwise and pairwise scores

  • The abstract motivates the entire framework by claiming existing open-source VLM reward judges "lack pairwise preferences essential for modern RLHF, DPO and Bradley-Terry frameworks." This is not accurate: Robometer [1] can natively produce preference labels as well as dense reward values for a trajectory, so the claimed capability gap that justifies the four-paradigm construction does not hold as stated.
  • Robometer supports pairwise comparison and pointwise reward simultaneously, making it the most relevant point of comparison for a "multi-paradigm" reward model - yet it is neither cited nor compared against. The evaluation only compares against RoboReward (single-paradigm), so the paper never tests its method against the closest existing multi-paradigm judge.
  • Beyond a baseline comparison, Robometer is also a natural testbed for POISE itself: if Robometer exhibits pointwise–pairwise mismatch, applying POISE on top of it would demonstrate that the contribution generalizes beyond the authors' own distillation corpus.

2. The central empirical claims are not objectively established.

  • The performance improvement is very marginal: 0.87 points Overall over raw-distillation SFT (77.96 vs. 77.09), and in many individual metrics the proposed model does not perform best as shown in Table 1. No seeds, no standard errors, and no significance testing are reported; at these margins the differences across baselines may well be within the error range and statistically insignificant.
  • The marginal accuracy gains also raise a question the paper does not confront: do they imply that pointwise–pairwise inconsistency is not actually a consequential problem for reward quality in the first place? The large Cons. gains do not answer this, since POISE eliminates reversals in the training labels by construction.
  • The downstream PAIBench-G study, which carries the claim that the reward model "provides winning guidance for robot policy optimization," lacks basic experiment and result details: it is unclear how the subset of tasks was chosen, and no statistics beyond "judged by 3 annotators with majority vote" and an approximate 69% win rate over 68 comparisons are given.
  • Theorem 1 assumes the pairwise-induced order is exactly consistent with the golden scores, which is usually modeled to be untrue in reward learning. It is more common to use Bradley Terry model to sample preference label on a pair with a probability that corresponds to the Boltzmann rationality model. Why does this paper make a hard assumption that pairwise induced orders have no uncertainty?
  • There is also a conceptual question about whether the correction matters where it applies: PAVA pools trajectories in the same chain block to a shared mean, meaning the edited items already had similar scores. If the end goal of reward learning is better policy learning or trajectory optimization, such small score differences are unlikely to matter downstream. PAIBench-G study might be able to address the advantage of the proposed method in downstream policy, but why would the model underperform on Table 1 but work on PAIBench-G study?

3. Presentation leaves core components of the method underspecified.

  • For the video-QA paradigms, it is not clearly stated what exactly is being scored or compared. The task definition for Score-B/Pair-B needs to be made explicit in section 3.1 rather than inferred from prompt templates in the appendix.
  • For Pair-A and Pair-B, the output of the comparison is unclear from the main text - is it a value or something else? A clear worked example would help the reader understand the difference between the A and B paradigms.

Minor Comments

  • The term "label-local strategies" is used to dismiss TrustJudge and majority voting before it is defined; terms should be defined before use.

[1] Liang et al., Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons, Robotics: Science and Systems (RSS) 2026

Questions for authors
  • Please compare against Robometer, which performs pairwise comparisons and pointwise reward simultaneously, and additionally evaluate Robometer combined with POISE: Robometer may also exhibit pointwise–pairwise mismatch, and this would test whether your correction generalizes beyond your own setup.
  • Why is the performance improvement so marginal, and why does the model not perform best on several sub-metrics? Does this imply pointwise–pairwise inconsistency is not a significant source of error? Please run repeated experiments with different seeds and report standard errors, as the current differences may be statistically insignificant.
  • Within a PAVA chain block, pooled trajectories already have similar raw scores. If the end goal is policy optimization, why do these small score differences need correcting at all? Can you show downstream sensitivity to the magnitude of corrected score changes?
  • For the PAIBench-G evaluation: how was the subset of tasks chosen, and can you provide the full quantitative protocol and statistics?
  • What exactly is scored/compared in the video-QA paradigms, and what is the output format of the Pair-A/Pair-B comparisons? A concrete example in the main text would clarify the distinction between the A and B axes.
Reviewer_Tq6sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a generalist robot reward model based on the Qwen3-VL-4B architecture. The key is training with four different losses, point-wise scores for task progress, point-wise score for video QA, and pairwise preference ordering for each of these. The point-wise labels are algorithmically corrected to respect pairwise ordering, which leads to a prediction quality guarantee in Theorem 1. The model compares favorably to a variety of baselines on standard robotics reward prediction datasets.

Critical strengths

The main contribution claims are convincing and interesting. Pairwise ordering does seem like an important source of supervision for generalist reward models, since pointwise quantifications of reward or progress can depend on the task, and may not be comparable across tasks or scenes. The performance comparison seems to be quite exhaustive and shows that the proposed model performs very well. The writing and organization is quite clear.

Critical weaknesses

While this is a strong paper, there are a few areas for improvement. In decreasing order of importance:

  • The only substantive test of a reward model in robotics is how well does it improve the robot policy through on-robot RL? This test is not performed in this paper, so in that sense we do not know whether the proposed method will be impactful for robots. The video prediction task is not closed loop and does not include real robot hardware, so it is not a suitable proxy for robot RL

  • The other concern is that the paper is quite dense with jargon specific to the niche of robot reward model learning. The authors should try to use terms more generally understandable throughout the community, or provide definitions as appropriate

Questions for authors

Can you show real robot experiments with RL policy improvement?

1 comment

Thanks for the responses. Unfortunately, I think my first concern was not really addressed directly. Just because the reward model better agrees with human judgement doesn't mean that it will lead to policy improvement in an RL retraining loop. I think demonstrating this would make the paper much stronger. All things considered, I will stick with my original score. Best of luck to the authors!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.