acceptodds
Under review as a conference paper at ICLR 2027

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Abstract

Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce DeltaRubric, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. \dr operates in two steps: acting first as a , the model generates a neutral, instance-specific verification checklist. Transitioning into a , it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate \dr as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains, improving base-model overall accuracy on VL-RewardBench by (4B) and (8B) points and consistently surpassing no-rubric baselines. Beyond accuracy, DeltaRubric outperforms compute-matched inference scaling (verbose CoT and best-of-), preference learning (DPO), and automated prompt optimization while generating fewer tokens, generalizes to a different backbone InternVL3 and to scalar rewards, and as a frozen reward model improves downstream policy training. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reward Modeling, Rubrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k9WFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeltaRubric, a generative multimodal reward modeling framework that decomposes pairwise preference evaluation into two stages: Disagreement Planning and Checklist Verification. Given an image, a question, and two candidate responses, the Planner generates an instance-specific checklist targeting factual disagreements, while the Verifier evaluates these criteria against the visual evidence to predict the preferred response. The authors jointly optimize both roles using multi-role reinforcement learning with separately normalized advantages. Experiments on Qwen3-VL-4B/8B demonstrate improvements over no-rubric, static-rubric, and alternative reward-modeling baselines. Additional experiments investigate reward formulations, generalization across architectures, inference costs, and downstream policy optimization.

Critical strengths
  1. The manuscript is easy to understand, and the methodology is clearly presented.
  2. This work includes extensive experiments.
Critical weaknesses
  1. Limited innovation. I find the approach of collectively identifying discrepancies and verifying them one by one to be lacking in innovation.
  2. This work is restricted to pairwise verification and fails entirely in pointwise verification scenarios, making its scope very limited.
  3. The proposed reward formulations appear mathematically equivalent to simpler rewards under the stated advantage normalization. A central contribution is the design of separate rewards for the Planner and Verifier, intended to encourage checklists that correct the model's initial judgment. However, according to Equations (1)–(3), both reward modifications appear to be eliminated by group-wise advantage normalization. For the Planner, the relative reward subtracts the correctness of the no-rubric baseline, which is constant across candidate checklists for the same input. Subtracting this constant does not change the group-normalized advantages. Therefore, the proposed relative reward should produce the same advantages as the absolute correctness reward. Similarly, the Verifier's guidance bonus depends on whether the no-rubric baseline is correct. Since this baseline is shared within each group and the final accuracy is binary, the bonus effectively applies a common positive scaling factor to the correctness reward. This factor also disappears after group-wise standardization. Surprisingly, Table 3 and Appendix Table 8 report nontrivial performance differences between these mathematically equivalent formulations.
Questions for authors

please see weakness.

2 comments

We thank the reviewer for the helpful feedback and for noting that the manuscript is easy to understand, the methodology is clearly presented with extensive experiments. We address the concerns bellow in detail and will incorporate all feedback into the final version.

W1. Novelty.

We agree the high-level recipe is simple, and we see that as a strength. The contribution is in three specifics beyond just find differences, then verify. (1) What the rubric is conditioned on. Rubric-based reward modeling has mostly been developed for text, where criteria can be stated at the task or query level. In multimodal judging the deciding facts are instance-level visual details, so our checklist is generated per instance. This choice is what drives the gain: a static rubric executed by the same RL-trained judge performs worse. (2) How it is learned. A single model learns both roles from preference labels alone, with no rubric annotations and no external rubric generator. (3) What we learn from it. Structured verification is cheaper than free-form reasoning while more accurate (1.10× tokens, yet +4.3 over best-of-3 at 3.28×, Table 15), and the verifier executes rather than pattern-matches the checklist (negating one decisive item drops Overall 80.1→64.4, Table 17). And this is especially helpful for multimodal reward modeling.

W2. Pointwise use.

We would like to clarify that the method does, however, already provide per-response rewards in two ways. (1) Policy training (Table 6). GRPO needs a reward for every sampled response. Each of the G=8 responses to a prompt is compared against an anchor response from the same group, with randomized A/B order; GRPO normalizes rewards within the group, these relative comparisons are sufficient. The resulting policy improves over the one trained with the no-rubric reward model on three benchmarks AMBER, MMHal and MMStar. (2) Pointwise scoring (Table 14). A Bradley–Terry scalar head scores each response independently; conditioning it on DeltaRubric checklists raises it from 68.6 to 72.2 Overall.

W3. Reward formulations.

Thanks for the suggestions. When all N checklists are valid, R_rel = R_abs − 1(z_0=z*), the relative reward and absolute reward differ a within-group constant, so the two give identical advantages. They differ through the filter: a rejected checklist is judged without a checklist, so its verdict is z_0 and its relative reward is 0, whereas in the absolute ablation a rejected checklist is scored 0 directly. The two then differ by 1(z_0=z*) v, where v is the valid-checklist indicator, not constant once a group contains a rejected candidate and the unguided verdict is correct. On those prompts the absolute reward gives positive advantage to valid checklists that changed nothing. We add this clarification into paper Appendix A.3.

For the Verifier, we agree with this analysis. z_0 is shared across the M trajectories of a prompt, so the Eq. 2 bonus rescales a group's rewards by (1+\lambda) and per-group standardization is invariant to it; Table 8 is consistent with this: the four runs show no trend in lambda. And consistently every run lands within 77.8–80.1 Overall and at least 5.8 above the no-rubric judge. We revise the paper to simplify Eq.2 as R_verify = 1(z_j = z*), and Table 8 is presented as an insensitivity analysis over the runs.

Thank you for your response. Having reviewed the questions raised by other reviewers—as well as my own—alongside your replies, and considering your stated motivation and contributions, I do not believe the submission meets the standards for a top-tier conference. I am not the type of reviewer who resorts to "silent" or vague feedback; therefore, I am explicitly stating my recommendation as "reject," with the hope that you will work to improve the quality of the manuscript.

Reviewer_N4PkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DeltaRubric, a multimodal reward modeling framework that decomposes preference evaluation into checklist generation (Planner) and checklist-based verification (Verifier), jointly optimized through reinforcement learning. The method achieves improvements across several multimodal reward benchmarks and shows promising transfer to downstream policy training. However, the reliability of the proxy reward and the stability of the reported gains require further clarification.

Critical strengths
  1. The Planner–Verifier design provides a clear motivation for addressing superficial text-based shortcuts in multimodal preference evaluation. The generated checklists encourage the model to focus on visual evidence that distinguishes candidate responses.
  2. The paper includes meaningful experiments beyond benchmark accuracy. The checklist-negation experiment, compute-matched comparisons, and cross-architecture evaluation provide useful evidence for the proposed mechanism.
  3. The improvements transfer to downstream policy training. For example, the DeltaRubric reward model improves AMBER accuracy from 88.7 to 90.9 and MMHal-Bench scores from 3.56 to 4.14.
Critical weaknesses
  1. The reliability of the proxy reward deserves further attention. The Planner is optimized using a lightweight verifier rather than the full Verifier. Although Appendix A.12 reports a Pearson correlation of 0.92 across 24 evaluation points, this only demonstrates agreement in overall trends, not necessarily reliable sample-level reward judgments. The authors could compare the two reward signals at the sample level and evaluate whether using the full Verifier changes Planner training and final performance.

  2. The robustness of the reported improvements needs clearer evidence. The main results show promising gains, but the tables do not clearly report variability across independent training runs. In addition, the rule-based checklist filter cannot guarantee that generated checklists are factually accurate or genuinely useful. The authors could report multi-seed results with uncertainty estimates and evaluate the impact of incorrect or misleading checklists on final judgments.

Questions for authors

Please see the weaknesses.

2 comments

We thank the reviewer for the valuable assessment and for recognizing we provide a clear motivation, with meaningful experiments beyond benchmark accuracy, and the improvements transfer to downstream policy training. Below we address the concerns in detail and will incoporate all feedback into the final version.

W1. Reliability of the proxy reward

We conduct additional experiments to train the Planner with the full Verifier scoring candidates instead of lightweight verfier. We report the evaluation results as follows. Final performance is similar while the full verifier costs 2.6× more generation. This demonstrates the reliability of the proxy reward.

MethodOverallMacroGen. cost/step
Lightweight Verifier80.173.51.0×
Full Verifier80.473.72.6×

W2. Multi-seed results

We train three seeds with the no-rubric baseline and DeltaRubric with the 8B model. We present the results in the following table. As show, Deltarubric works robustly and largely outperforms the no-rubric baseline with non-overlapping bands.

MethodVL-RewardBench OverallVL-RewardBench MacroMM-RewardBench Overall
No-rubric71.9 ± 0.663.3 ± 0.668.6 ± 0.5
DeltaRubric78.9 ± 1.171.9 ± 1.472.9 ± 0.4

Thank you for the detailed responses and additional experiments. These additional results have strengthened the empirical evidence and improved the clarity of the paper. I consider my concerns to be sufficiently addressed and will maintain my original rating.

Reviewer_EV7ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeltaRubric, which divides multimodal reward modeling from a single-step judgment into a two-step process. First, the reward MLLM acts as a Planner, analyzing the factual disagreements between two candidate responses and generating an instance-specific verification checklist. Then, the same model acts as a Verifier, executing the checklist item by item against the image and producing the final judgment. The two roles are jointly optimized via multi-role RL. The experiments on VL-RewardBench and Multimodal RewardBench demonstrating the effectiveness of the proposed framework.

Critical strengths
  1. The proposed two-stage Plan–Verify approach to reward modeling is reasonable and well motivated.

  2. The experimental results show substantial and consistent improvements over both the no-rubrics variant and the base model on two benchmarks.

  3. The extensive and thorough experiments in the appendix provide further evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. A key limitation is that the main experiments compare DeltaRubric only against the base model and a no-rubric variant, and do not compare it with prior baseline methods that also use rubrics or with other approaches to optimizing multimodal reward modeling. This limits the evidence supporting its advantages over existing approaches.

  2. The benefit of relative reward in Multi-Role RL is unclear. If I(Z0=Z∗)I(Z_0=Z^*) is constant within each GRPO group, group-wise centering should cancel this term. The authors should clarify the advantage computation and explain why Table 3 nevertheless reports a substantial performance gap between relative and absolute reward.

  3. In Appendix A.2, the training comparison does not control for compute. Table 7 in Appendix A.2 reports 5.0 hours for DeltaRubric versus 3.4 hours for the baseline when training the 4B model. DeltaRubric uses two-stage rollouts and two reward signals, so the reported gains do not isolate the method’s benefit under a matched training budget.

Questions for authors

In the Problem Formulation, DeltaRubric requires a pair of responses as input. How is it implemented when assigning a reward to a single response, and how is it subsequently used for within-group scoring in GRPO?

3 comments

We thank the reviewer for the constructive review and for recognizing the approach reasonable and well motivated, showing substantial and consistent improvements, extensive and thorough experiments. We address the concerns below in detail and will incorporate all feedback into final version.

W1. Comparison with prior rubric-based and reward modeling methods.

We compare with prior rubric-based and reward modeling methods, evaluating on the VL-RewardBench. We gather them into the following table. The results validate the advantages over existing approaches.

CategoryMethodOverallMacro
Prior multimodal RMMM-RLHF-Reward-7B50.251.0
LLaVA-Critic-8B41.250.7
Rubric-basedStatic rubric75.366.8
Other RM optimizationNo-rubric judge (same RL recipe)72.063.4
DPO68.262.8
Bradley–Terry scalar RM68.661.3
GEPA (Planner+Verifier prompts)70.364.7
No-rubric + verbose CoT (2.81× tokens)74.366.6
No-rubric + best-of-3 vote (3.28× tokens)75.868.0
OursDeltaRubric (1.10× tokens)80.173.5

W2. Benefit of the relative reward.

Thanks for the comments. When all N checklists are valid, R_rel = R_abs − 1(z_0=z*), the relative reward and absolute reward differ a within-group constant, so the two give identical advantages. They differ through the filter: a rejected checklist is judged without a checklist, so its verdict is z_0 and its relative reward is 0, whereas in the absolute ablation a rejected checklist is scored 0 directly. The two then differ by 1(z_0=z*) v, where v is the valid-checklist indicator, not constant once a group contains a rejected candidate and the unguided verdict is correct. On those prompts the absolute reward gives positive advantage to valid checklists that changed nothing. We add this clarification into paper Appendix A.3.

W3. Training comparison under a matched budget.

We conduct additional experiments with the 4B model with the no-rubric baseline training more steps to match the training budget of DeltaRubric. As shown, DeltaRubric still outperforms under the matched training budget.

MethodWall-clockOverallMacro
No-rubric5.0 h74.064.5
DeltaRubric5.0 h77.569.7

Q1. Rewarding a single response; within-group scoring in GRPO

In policy training (Table 6), for each prompt the policy samples G=8 responses and one serves as the anchor; every other response is compared against the anchor by the reward model with randomized A/B order. GRPO then standardizes these rewards within the group of G responses, so only the ranking induced by the anchor comparisons is needed. For settings that need a standalone per-response score, the Bradley–Terry head conditioned on DeltaRubric checklists also works (Table 14).

Thank you for the further explanation and additional experiments. I have 2 follow-up questions:

  1. Could you compare DeltaRubric with closely related methods that also optimize rubric generation during training, such as Rubric-ARM [1] and Proxy-GRM [2]? These comparisons would help establish how DeltaRubric performs against other learned-rubric approaches.

  2. Could you report how the fraction of Planner rollout groups with a correct no-checklist verdict and both accepted and rejected checklists changes over training? This would help clarify how often the two rewards produce different GRPO advantages and explain the +2.5pp gain reported in Table 3.

[1] Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training, https://arxiv.org/abs/2602.01511

[2] Rationale matters: Learning transferable rubrics via proxy-guided critique for VLM reward models, https://arxiv.org/abs/2603.16600

Comparison with Rubric-ARM and Proxy-GRM

We compare DeltaRubric with Rubric-ARM and Proxy-GRM, training with the 4B model and evaluating on VL-RewardBench. We show the results below. All rubric methods improve over the no-rubric baseline, while DeltaRubric achieves better performance than Rubric-ARM and Proxy-GRM, demonstrating the effectiveness of our approach.

MethodOverallMacro
No rubric73.263.5
Rubric-ARM76.068.2
Proxy-GRM75.567.6
DeltaRubric77.569.7

Fraction of rollout groups

We report in following table the fraction of planner rollout groups with a correct no-checklist verdict and mixed accepted/rejected checklists over training. The fraction decreases, as training proceeds the Planner learns to emits checklists that pass the filter, so fewer groups contain filtered candidates.

Training step% groups with correct z0 and mixed accepted/rejected checklists
020%
608%
1203%

Thank you for the follow-ups, and if your concerns have been addressed, we would be sincerely grateful if you could raise the score. Thank you again for helping us improve our work.

Comments on the paper

1 comment

Dear ACs and Reviewers,

We sincerely thank all reviewers for their constructive feedback and thoughtful suggestions, which help further strengthen our work. We greatly appreciate the reviewers' recognition of the following strengths:

  • The approach is reasonable and well motivated, and provides a clear motivation for addressing superficial text-based shortcuts in multimodal preference evaluation (EV7R, N4Pk)
  • Substantial and consistent improvements over both the no-rubrics variant and the base model on two benchmarks (EV7R)
  • Extensive and thorough experiments (EV7R, k9WF)
  • Meaningful experiments beyond benchmark accuracy: the checklist-negation experiment, compute-matched comparisons, and cross-architecture evaluation provide useful evidence for the proposed mechanism (N4Pk)
  • The improvements transfer to downstream policy training (N4Pk)
  • The manuscript is easy to understand and clear to follow, and the methodology is clearly presented (N4Pk, k9WF)

During the rebuttal period, we conducted additional experiments and provided detailed clarifications to address all reviewers' concerns. We will include the additional results in the final version. For convenience, we summarize the main updates below:

  • Multi-seed results
  • Additional experiment on reliability of the proxy reward
  • Additional experiment with the no-rubric baseline trained under a matched training budget
  • Comparison with prior rubric-based and reward modeling methods
  • Clarifications on reward formulations

We would like to sincerely thank all reviewers again for the helpful suggestions that notably enhance the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.