- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DeltaRubric, a generative multimodal reward modeling framework that decomposes pairwise preference evaluation into two stages: Disagreement Planning and Checklist Verification. Given an image, a question, and two candidate responses, the Planner generates an instance-specific checklist targeting factual disagreements, while the Verifier evaluates these criteria against the visual evidence to predict the preferred response. The authors jointly optimize both roles using multi-role reinforcement learning with separately normalized advantages. Experiments on Qwen3-VL-4B/8B demonstrate improvements over no-rubric, static-rubric, and alternative reward-modeling baselines. Additional experiments investigate reward formulations, generalization across architectures, inference costs, and downstream policy optimization.
Critical strengths
- The manuscript is easy to understand, and the methodology is clearly presented.
- This work includes extensive experiments.
Critical weaknesses
- Limited innovation. I find the approach of collectively identifying discrepancies and verifying them one by one to be lacking in innovation.
- This work is restricted to pairwise verification and fails entirely in pointwise verification scenarios, making its scope very limited.
- The proposed reward formulations appear mathematically equivalent to simpler rewards under the stated advantage normalization. A central contribution is the design of separate rewards for the Planner and Verifier, intended to encourage checklists that correct the model's initial judgment. However, according to Equations (1)–(3), both reward modifications appear to be eliminated by group-wise advantage normalization. For the Planner, the relative reward subtracts the correctness of the no-rubric baseline, which is constant across candidate checklists for the same input. Subtracting this constant does not change the group-normalized advantages. Therefore, the proposed relative reward should produce the same advantages as the absolute correctness reward. Similarly, the Verifier's guidance bonus depends on whether the no-rubric baseline is correct. Since this baseline is shared within each group and the final accuracy is binary, the bonus effectively applies a common positive scaling factor to the correctness reward. This factor also disappears after group-wise standardization. Surprisingly, Table 3 and Appendix Table 8 report nontrivial performance differences between these mathematically equivalent formulations.
Questions for authors
please see weakness.
2 comments
Thank you for your response. Having reviewed the questions raised by other reviewers—as well as my own—alongside your replies, and considering your stated motivation and contributions, I do not believe the submission meets the standards for a top-tier conference. I am not the type of reviewer who resorts to "silent" or vague feedback; therefore, I am explicitly stating my recommendation as "reject," with the hope that you will work to improve the quality of the manuscript.
We thank the reviewer for the helpful feedback and for noting that the manuscript is easy to understand, the methodology is clearly presented with extensive experiments. We address the concerns bellow in detail and will incorporate all feedback into the final version.
We agree the high-level recipe is simple, and we see that as a strength. The contribution is in three specifics beyond just find differences, then verify. (1) What the rubric is conditioned on. Rubric-based reward modeling has mostly been developed for text, where criteria can be stated at the task or query level. In multimodal judging the deciding facts are instance-level visual details, so our checklist is generated per instance. This choice is what drives the gain: a static rubric executed by the same RL-trained judge performs worse. (2) How it is learned. A single model learns both roles from preference labels alone, with no rubric annotations and no external rubric generator. (3) What we learn from it. Structured verification is cheaper than free-form reasoning while more accurate (1.10× tokens, yet +4.3 over best-of-3 at 3.28×, Table 15), and the verifier executes rather than pattern-matches the checklist (negating one decisive item drops Overall 80.1→64.4, Table 17). And this is especially helpful for multimodal reward modeling.
We would like to clarify that the method does, however, already provide per-response rewards in two ways. (1) Policy training (Table 6). GRPO needs a reward for every sampled response. Each of the G=8 responses to a prompt is compared against an anchor response from the same group, with randomized A/B order; GRPO normalizes rewards within the group, these relative comparisons are sufficient. The resulting policy improves over the one trained with the no-rubric reward model on three benchmarks AMBER, MMHal and MMStar. (2) Pointwise scoring (Table 14). A Bradley–Terry scalar head scores each response independently; conditioning it on DeltaRubric checklists raises it from 68.6 to 72.2 Overall.
Thanks for the suggestions. When all N checklists are valid, R_rel = R_abs − 1(z_0=z*), the relative reward and absolute reward differ a within-group constant, so the two give identical advantages. They differ through the filter: a rejected checklist is judged without a checklist, so its verdict is z_0 and its relative reward is 0, whereas in the absolute ablation a rejected checklist is scored 0 directly. The two then differ by 1(z_0=z*) v, where v is the valid-checklist indicator, not constant once a group contains a rejected candidate and the unguided verdict is correct. On those prompts the absolute reward gives positive advantage to valid checklists that changed nothing. We add this clarification into paper Appendix A.3.
For the Verifier, we agree with this analysis. z_0 is shared across the M trajectories of a prompt, so the Eq. 2 bonus rescales a group's rewards by (1+\lambda) and per-group standardization is invariant to it; Table 8 is consistent with this: the four runs show no trend in lambda. And consistently every run lands within 77.8–80.1 Overall and at least 5.8 above the no-rubric judge. We revise the paper to simplify Eq.2 as R_verify = 1(z_j = z*), and Table 8 is presented as an insensitivity analysis over the runs.