acceptodds
Under review as a conference paper at ICLR 2027

Relative-Margin Reward for Candidate-set Ranking in Text-to-Image Generation

Abstract

Text-to-image models generate multiple candidates per prompt, and reward models are used to select among or rank them. Although benchmarks with human preferences over multiple images exist, reward model evaluation still centers largely on pairwise accuracy. Whether this metric adequately reflects a model’s ability to recover human rankings over entire candidate sets remains underexplored. We introduce Rank-Bench, a benchmark that recasts reward-model evaluation as tie-aware ranking over candidate sets, comprising 902 rankings over 6,236 images across 11 diagnostic dimensions. Rank-Bench reveals that comparable pairwise accuracy does not imply comparable set-level ranking agreement. We propose MarginRM, a relative-margin reward model trained to predict criterion-level signed ordinal margins and explanations under prompt-specific rubrics. At inference, it scores each candidate relative to a shared anchor under a common rubric, yielding a ranking over the full candidate set. Under all-pairs evaluation on Rank-Bench, MarginRM achieves 61.07% stable accuracy and a Kendall's of 0.519, compared with 54.47% and 0.487 for the strongest baseline. Single-anchor inference yields a of 0.485 using only 27.93% of the all-pairs comparison budget. As the reward for GRPO post-training of FLUX.1-dev, it improves GenEval from 63.72 to 72.61 and T2I-CompBench from 49.22 to 54.38, while producing rankings through shared-anchor comparisons without exhaustive pairwise evaluation.

Primary area:
generative models
Keywords:
Image generation, Reward model, Human preference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zdrNOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Rank-Bench which measures rank-preference recovery of image reward model. It also proposes MarginRM, a novel multi-aspect margin-based reward model that achieves high performance on the proposed Rank-Bench. The author also demonstrated that such reward model can be useful in RL.

Critical strengths
  1. This paper addresses the fundamental limitation of existing pair-based reward modeling and reward model evaluations, providing valuable insight to the community.
  2. MarginRM show considerable performance gains compared with existing reward models both in preference recovery and RL Post-training.
Critical weaknesses

Overall, the paper is sound and MarginRM's contribution is clear. My main concern is on the soundness of post-training experiment

  1. The post training experiment lacks compute analysis and Appendix C.3 is insufficient. Specifically, the proposed model seems larger than some of the baselines like HPSv2, the author should discuss the potential overhead of using Qwen3.5-9B based model. Does the model converge given the fixed training step, or will training longer improve the model further? If latter is the case, the author should provide compute-matched results by allowing RL run with cheaper reward models to traing longer to match the compute cost measured by wall-clock time
  2. The original flow-GRPO use SD3.5-M and FLUX.1-dev as the base model, this paper uses only FLUX.1-dev. It would be better to show results across different model families for post training experiment. One particular confounding factor that needs to be isolated is that in building MarginRM, some images are generated with FLUX.1-dev, so the reward model may work better on this specific model. Showing results on SD3.5 will demonstrate if MarginRM generalize beyond the set of models in its training data, or can be generalized.
Questions for authors

See weakness.

  1. What is the number of steps/epochs of RL post training and what is the compute overhead of MarginRM compared with baselines
  2. Does MarginRM's effectiveness in post-training generalized beyond the list of models in A.2 (SDXL, SD3, FLUX.1-dev, Z-Image, Nano Banana, Qwen-Image, GPT-Image, and Banana-Pro) ? Experiments on one other models like SD3.5-M,or Sana will clarify this problem.
  3. What is the breakdown of T2I CompBench score? Specifically, T2I comp bench have detailed rubrics on color, shape, texture, it would be interesting to show how it aligns with rubics of MarginRM.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zdrN (Weaknesses 1 & Questions 1)

We sincerely thank you for your thoughtful review and constructive feedback. To make our response easy to navigate, we have merged overlapping points from your Weaknesses (W1, W2) and Questions (Q1-Q3) and organized our point-by-point replies in a [Qi & Wj] →\to [Ai] format below.

\quad

[Q1 & W1]: (1.) Complete RL post-training steps and hyperparameters, (2.) convergence verification for lower-cost reward models (HPSv2, PickScore), and (3.) wall-clock compute overhead of the Qwen3.5-9B-based reward model.

[A1]: Regarding Appendix C.3 and post-training compute analysis: (1) we train MarginRM for 390 epochs (780 steps) with complete hyperparameters detailed below; (2) all baselines are evaluated at their converged optima, as training lower-cost RMs longer leads to reward over-optimization and performance degradation; and (3) thanks to its O(K)O(K) shared-anchor design, our Qwen3.5-9B-based MarginRM requires only 64.6s of reward model scoring per epoch (30.2% of total runtime), achieving 2.7×2.7\times faster epoch wall-clock time than existing Qwen3.5-9B-based pairwise reward models.

  1. RL Training Steps & Hyperparameters (Appendix C.3): For MarginRM, Flow-GRPO post-training runs for 390 epochs (780 optimizer steps), with complete training configurations detailed below: | Sampling | | Optimizer | | Training & LoRA | | |:---|:---:|:---|:---:|:---|:---:| | Parameter | Value | Parameter | Value | Parameter | Value | | Rollouts per prompt (KK) | 24 | Optimizer | AdamW | PPO clip range (ϵ\epsilon) | 10−410^{-4} | | Batches per iteration | 48 | Learning rate | 3×10−43\times10^{-4} | Ratio clip range | [0.9999, 1.0001] | | Image resolution | 512 × 512 | Adam β1\beta_1 | 0.9 | Advantage clip range | [-5, 5] | | Sampling steps | 6 | Adam β2\beta_2 | 0.999 | LoRA rank (rr) | 64 | | Timestep fraction | 0.99 | Adam ϵ\epsilon | 10−810^{-8} | LoRA α\alpha | 128 | | Guidance scale | 3.5 | Weight decay | 10−410^{-4} | Total epochs (steps) | 390 (780) |

  2. Convergence vs. Training Longer: Rather than stopping at an arbitrary fixed step, we evaluate each reward model at its converged optimum—selected as the checkpoint achieving the highest PickScore after the training reward plateaus (step 320 for HPSv2 at 22.82; step 440 for PickScoreat 23.93). Training cheaper scalar RMs (HPSv2, PickScore) beyond this peak degrades performance across metrics due to reward over-optimization rather than improving it: | Reward Model | Checkpoint Step | GenEval ↑ | T2I-CompBench ↑ | CLIP ↑ | PickScore ↑ | | :---: | :---: | :---: | :---: | :---: | :---: | | HPSv2 | 240 | 56.65 | 42.38 | 31.25 | 22.52 | | | 280 | 54.86 | 42.16 | 31.29 | 22.72 | | | 320 (Optimum) | 56.93 | 42.14 | 31.45 | 22.82 | | | 360 | 52.94 | 42.18 | 31.42 | 21.89 | | | 400 | 50.81 | 42.36 | 31.48 | 21.61 | | PickScore | 360 | 57.31 | 43.26 | 33.55 | 23.52 | | | 400 | 56.29 | 43.94 | 33.49 | 23.72 | | | 440 (Optimum) | 56.96 | 44.12 | 33.48 | 23.93 | | | 480 | 56.55 | 44.29 | 33.42 | 23.89 | | | 520 | 56.63 | 43.91 | 33.48 | 23.61 |

  3. Compute Overhead ( MarginRM vs. Qwen3.5-9B-based Baselines): Thanks to its O(K)O(K) shared-anchor formulation (K−1K-1 calls), MarginRM remains by far the fastest method even while evaluating 2×2\times more rollout images per epoch (1,152 vs. 512–576). Despite handling double the visual throughput, MarginRM reduces total RM calls per epoch from 1,792–2,304 down to 1,104, cutting per-epoch RM scoring time by 3.1×–4.9×3.1\times\text{–}4.9\times and overall epoch wall-clock time by 2.7×2.7\times compared to O(K2)O(K^2) pairwise UnifiedReward-Think and UnifiedReward-Flex baselines: | Metric | UnifiedReward-Think | UnifiedReward-Flex | MarginRM (Ours) | | :--- | :---: | :---: | :---: | | Prompts per epoch | 64 | 64 | 48 | | Rollouts per prompt (KK) | 8 | 9 | 24 | | Images per epoch | 512 | 576 | 1,152 | | RM calls per prompt | 28 | 36 | 23 | | RM calls per epoch | 1,792 | 2,304 | 1,104 | | Mean latency per prompt | 9.19 s | 9.00 s | 4.46 s (2.0×2.0\times faster) | | Mean total RM time / epoch | 202.8 s | 314.6 s | 64.6 s (3.1×3.1\times–4.9×4.9\times faster) | | Mean total wall-clock time / epoch | 588.2 s | 576.0 s | 214.0 s (2.7×2.7\times faster) |

AuthorsOpenReview ↗

Response to Reviewer zdrN (Weaknesses 2 & Questions 2-3)

[Q2 & W2]: Cross-family generalization of MarginRM in RL post-training to unseen base generators (SD3.5-Medium) not included in the MarginRM training set (Appendix A.2).

[A2]: MarginRM generalizes effectively to unseen base generators outside Appendix A.2, ruling out training-source bias toward FLUX.1-dev. As shown by our new Flow-GRPO post-training experiment on SD3.5-Medium (excluded from Appendix A.2), MarginRM achieves substantial gains over the base model and consistently outperforms UnifiedReward-Flex across all four benchmarks:

Backbone / Post-Training MethodGenEval ↑T2I-CompBench ↑CLIP ↑PickScore ↑
FLUX.1-dev (Base)63.7249.2235.2722.61
↳ + UnifiedReward-Flex68.9151.2236.4923.38
↳ + MarginRM (Ours)72.6154.3838.4423.36
SD3.5-Medium (Base, Unseen in A.2)65.7949.9539.3322.53
↳ + UnifiedReward-Flex69.2750.0938.8222.94
↳ + MarginRM (Ours)72.7052.9039.9122.98

\quad

[Q3]: Per-category breakdown of T2I-CompBench.

[A3]: Below we report the per-category breakdown on T2I-CompBench.

MethodColorShapeTexture2D Spatial3D SpatialNumeracyNon-SpatialComplexOverall
FLUX.1-dev (Base)0.7090.5840.6050.2850.4290.6290.3060.3910.492
w/ HPSv20.5390.5430.5010.2410.3550.5610.2940.3300.421
w/ HPSv30.6270.5440.5550.2550.3750.6190.3020.3490.453
w/ PickScore0.6130.5430.5780.2450.3380.5700.2990.3430.441
w/ UnifiedReward0.7690.5380.6450.2980.4130.6520.3110.3880.502
w/ UnifiedReward-Think0.7910.5290.6620.3040.4270.6680.2940.3960.509
w/ UnifiedReward-Flex0.7900.5590.6720.3060.3960.6710.3040.3940.512
w/ MarginRM (Ours)0.7760.5630.7180.3200.4380.6840.3070.5440.544

While holistic scalar RMs (HPSv2/v3, PickScore) degrade compositional fidelity below the base model (0.492 →\to 0.421–0.453) by over-optimizing global aesthetics, MarginRM explicitly decomposes prompts into criterion-level margins covering attribute binding (color/shape/texture), spatial layout (2D/3D), object counting (numeracy), and multi-attribute interactions (complex) (Appendix B.1). This targeted credit assignment yields the highest post-training scores on Shape (0.563), Texture (0.718), 2D/3D Spatial (0.320 / 0.438), Numeracy (0.684), and Complex (0.544).

Thank the authors for the response. My concerns have been addressed.

AuthorsOpenReview ↗

Replying to Reviewer zdrN

Thank you for your positive feedback and for taking the time to review our response. We are glad that our response has addressed your concerns. We sincerely appreciate your support and constructive comments, which helped us improve the paper. Thank you again for your constructive feedback and valuable suggestions.

Reviewer_1hqxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to evaluate and train reward models for ranking text-to-image results. The authors argue that existing reward models fall short in ranking a full set of candidate images: pointwise models score each image separately with hidden criteria, pairwise models compare two images but their criteria can change from pair to pair, and rubric-based models use fixed criteria but have no shared visual reference. They also note that reward models are mostly evaluated with pairwise accuracy, not full-set ranking. To address this, they first introduce Rank-Bench, a benchmark of human rankings over candidate image sets across several diagnostic dimensions, where ties are allowed. Then they propose MarginRM, a reward model trained on existing preference data with rubrics and margins generated by a large teacher model. MarginRM compares each image with an anchor image using a prompt-specific rubric, and predicts a signed margin with an explanation for each criterion. Finally, they use MarginRM as the reward for GRPO post-training of FLUX.1-dev. MarginRM outperforms the strongest baseline on Rank-Bench and improves FLUX.1-dev on compositional generation benchmarks.

Critical strengths
  1. Rank-Bench provides a more realistic evaluation of reward models by focusing on ranking multiple candidates for the same prompt rather than evaluating isolated pairs. The annotation process is also well designed, with trained annotators, multiple annotations per pair, quality checks, and explicit ties. The diagnostic dimensions are another strength, as they help identify where different models perform well or struggle.

  2. MarginRM offers an efficient alternative to all-pairs comparisons by comparing each candidate against a single shared anchor. Even with a randomly selected anchor, it achieves ranking performance comparable to the strongest baseline while requiring far fewer comparisons. This makes the approach practical for applications such as best-of-N selection and RL training.

Critical weaknesses
  1. The main ranking results do not clearly show the benefit of using a shared anchor (Table 1). For a fair comparison with the baselines, MarginRM uses all-pairs win-rate aggregation, so the ranking improvements seem to come mostly from better pairwise judgments rather than the shared anchor itself. When the shared anchor is used, the ranking performance is comparable to the strongest baseline, but not better. This suggests that the anchor mainly improves efficiency, while the paper presents it as a way to improve ranking by providing a shared reference frame. Also, since the anchor sensitivity analysis shows considerable variation, the results using a single random anchor should include variance across different anchors.

  2. Some important ablations are missing. The paper claims that both the rubric and the shared visual anchor are important, but it does not test their individual contributions. For example, it would be useful to evaluate rubric-based scoring without an anchor and anchor-based comparison without a rubric. Another missing comparison is using the large teacher model directly as a judge. Since MarginRM is trained using the teacher's rubrics and margins, it is difficult to tell how much of the improvement comes from the proposed method and how much comes from distilling knowledge from a much larger model.

Questions for authors

1- Is the single-anchor result in Table 3 based on one random anchor or an average over several? Please report the mean and variance across different random anchors. 2- Can you add ablations that remove each component: rubric-based scoring without a shared anchor, and anchor-based comparison without a rubric? This would show whether both are needed, as claimed in the introduction. 3- How does the teacher model (Qwen3.5-397B-A17B) perform on Rank-Bench when used directly as a judge with the same rubric and prompt? This would clarify how much of the gain comes from the proposed formulation versus distillation from a larger model.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1hqx (Weaknesses 1 & Questions 1)

We sincerely thank you for your thoughtful review and constructive feedback. Since your Weaknesses (W1–W2) and Questions (Q1–Q3) closely correspond to one another, we have consolidated overlapping points and organized our responses in a unified [Qi & Wj] →\to [Ai] format below for ease of reference.

\quad

[Q1 & W1]: Clarifying (1) the relationship between all-pairs (Table 1) and shared-anchor (Table 3) evaluations, (2) the ranking performance comparison between MarginRM and the strongest baseline, and (3) the mean and variance across random anchors in Table 1.

[A1]: We clarify that Table 1 serves as a compute-matched O(K2)O(K^2) control while Table 3 evaluates our native O(K)O(K) shared-anchor ranking, where MarginRM consistently outperforms UnifiedReward-Flex across matched budgets (and surpasses full-budget UnifiedReward-Flex at ≥2\ge 2 anchors) with low variance across multiple random anchor draws. Specifically:

  1. Shared-anchor ranking (Table 3) is our primary test for validating MarginRM’s native formulation, whereas all-pairs aggregation (Table 1) serves strictly as a compute-matched O(K2)O(K^2) control against pairwise baseline reward models to isolate per-pair reward quality from the comparison budget.
  2. The shared-anchor mechanism enables a flexible efficiency–performance trade-off that outperforms UnifiedReward-Flex both across and within budgets: compared against exhaustive 100%-budget UnifiedReward-Flex (τb=0.487\tau_b = 0.487), MarginRM already matches it at just 27.9% budget (1 anchor) and surpasses it at 55.9%–83.8% budgets (2–3 anchors). Moreover, when both models are evaluated under the same shared-anchor setting (see table below), UnifiedReward-Flex collapses (τb\tau_b drops from 0.487 to 0.331 at 1 anchor) because binary judgments cannot resolve the relative ordering among multiple candidates that all beat (or lose to) the same anchor.
  3. The shared-anchor results in Table 3 are averaged over multiple independent random anchor draws rather than a single run. Below we report the mean and variance (computed on unscaled [0,1][0, 1] scores, with variances in subscripts) for both MarginRM and UnifiedReward-Flex under matched comparison budgets of 1, 2, and 3 anchors:
MethodStrategyStable Acc. ↑Acc. w/ Tie ↑Acc. w/o Tie ↑Kendall's τb\tau_b ↑
UnifiedReward-Flex1 anchor33.12(1.28e-5)33.12_{\text{(1.28e-5)}}44.65(1.63e-5)44.65_{\text{(1.63e-5)}}43.07(1.69e-5)43.07_{\text{(1.69e-5)}}0.331(3.98e-5)0.331_{\text{(3.98e-5)}}
2 anchors44.56(2.29e-5)44.56_{\text{(2.29e-5)}}54.96(2.06e-5)54.96_{\text{(2.06e-5)}}60.64(1.48e-5)60.64_{\text{(1.48e-5)}}0.439(1.68e-5)0.439_{\text{(1.68e-5)}}
3 anchors50.47(1.28e-5)50.47_{\text{(1.28e-5)}}58.92(1.17e-5)58.92_{\text{(1.17e-5)}}67.63(1.43e-5)67.63_{\text{(1.43e-5)}}0.459(3.98e-5)0.459_{\text{(3.98e-5)}}
All pairs54.4762.0575.180.487
MarginRM (Ours)1 anchor55.19(1.12e-5)55.19_{\text{(1.12e-5)}}62.15(3.37e-6)62.15_{\text{(3.37e-6)}}75.11(5.57e-6)75.11_{\text{(5.57e-6)}}0.485(2.52e-5)0.485_{\text{(2.52e-5)}}
2 anchors58.84(3.77e-6)58.84_{\text{(3.77e-6)}}63.36(1.88e-6)63.36_{\text{(1.88e-6)}}76.02(2.91e-6)76.02_{\text{(2.91e-6)}}0.495(1.09e-5)0.495_{\text{(1.09e-5)}}
3 anchors61.56(1.97e-6)\mathbf{61.56}_{\text{(1.97e-6)}}64.37(1.33e-6)\mathbf{64.37}_{\text{(1.33e-6)}}77.83(2.11e-6)\mathbf{77.83}_{\text{(2.11e-6)}}0.511(8.89e-6)0.511_{\text{(8.89e-6)}}
All pairs61.0764.3477.700.519\mathbf{0.519}

Across all metrics, MarginRM exhibits extremely low variance (consistently lower than UnifiedReward-Flex), and increasing anchors from 1 to 3 monotonically reduces variance while steadily improving mean ranking performance.

AuthorsOpenReview ↗

Response to Reviewer 1hqx (Weaknesses 2 & Questions 2-3)

[Q2 & W2]: Ablating individual components: anchor-based comparison without a rubric and rubric-based scoring without a shared anchor.

[A2]: We have added the requested ablations under the exact same Qwen3.5-9B backbone and training pairs (table below), confirming that neither the anchor nor the rubric alone is sufficient and that both are required for strong ranking performance. Specifically, we compare four variants:

  • Anchor-only (w/o rubric): compares image pairs using only binary labels without rubrics or margins;
  • Rubric-only (w/o anchor): scores single images on rubrics without an anchor reference (degenerating into a pointwise model, so swap-order Stable Acc. is N/A);
  • Margin-only w/ Anchor: combines the anchor reference and rubric-level numeric margins, removing textual rationales;
  • MarginRM (Ours): the full formulation with anchor, rubric margins, and rationales. | Model / Variant | MMRB2 ↑ | GenAI-Bench ↑ | Stable Acc. ↑ | Acc. w/ Tie ↑ | Acc. w/o Tie ↑ | Kendall's τb\tau_b ↑ | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Anchor-only | 64.6 | 71.1 | 58.78 | 62.39 | 75.59 | 0.473 | Rubric-only | 60.3 | 67.5 | - | 59.72 | 67.85 | 0.438 | | Margin-only w/ Anchor| 70.1 | 71.7 | 58.96 | 63.46 | 76.73 | 0.508 | | MarginRM | 75.2 | 76.1 | 61.07 | 64.34 | 77.70 | 0.519 |

Neither Anchor-only (τb=0.473\tau_b = 0.473) nor Rubric-only (\tau_b& = 0.438) alone is sufficient—lacking either fine-grained margin resolution or an anchor to calibrate cross-image score scales. Combining both in **Margin-only w/ Anchor image** immediately boosts ranking (\tau_b: 0.473 \to 0.508$, MMRB2: 64.6→70.164.6 \to 70.1), while adding rationale supervision in MarginRM further elevates performance across all benchmarks (τb\tau_b = 0.519, MMRB2 = 75.2, GenAI-Bench = 76.1).

\quad

[Q3 & W2]: Evaluating the teacher model (Qwen3.5-397B-A17B) directly as a judge on Rank-Bench to disentangle formulation gains from knowledge distillation.

[A3]: When evaluated directly on Rank-Bench—either with standard pairwise prompting (UnifiedReward 2.0) or with the exact same rubric (Margin Rubric)—Qwen3.5-397B-A17B trails our 9B MarginRM across all metrics, confirming that gains come from our formulation rather than teacher distillation.

Model / VariantMMRB2 ↑GenAI-Bench ↑Stable Acc. ↑Acc. w/ Tie ↑Acc. w/o Tie ↑Kendall's τb\tau_b ↑
Qwen3.5-397B (Standard)71.668.056.4259.6672.060.451
Qwen3.5-397B (Margin Rubric)68.153.524.6950.6758.640.284
MarginRM75.276.161.0764.3477.700.519

Why MarginRM Surpasses the Teacher: Prompting the untrained 397B teacher to directly emit multi-criterion margins at test time suffers from severe score miscalibration (τb=0.284\tau_b = 0.284). In our pipeline, teacher outputs are strictly filtered to match human ground-truth labels, so the teacher only decomposes human preferences into structured rubrics rather than supplying stronger labels.

Thank you for the clarification and additional experiments. I appreciate the detailed response, and my questions and concerns has been addressed.

AuthorsOpenReview ↗

Replying to Reviewer 1hqx

Thank you for your positive feedback and for taking the time to review our response. We are glad that our clarifications and additional experiments have addressed your questions and concerns. Your comments helped us improve the clarity of the paper. Thank you again for your constructive feedback and valuable suggestions.

Reviewer_buwHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward modeling for text-to-image generation when several images share one prompt and must be ranked as a set. The authors argue that high pairwise preference accuracy does not guarantee agreement with a human consensus order over the full candidate set.

They introduce Rank-Bench, a tie-aware benchmark with human pairwise labels on eight images per prompt and prompt-level consensus rankings derived from preference graphs. They propose MarginRM, which evaluates candidates with a prompt-specific rubric and a shared anchor image, predicting criterion-level signed margins and explanations and aggregating them into anchor-relative scores. The model is fine-tuned on HPDv3 pairwise data with largely teacher-generated rubric and margin supervision.

Experiments report improved performance over strong baselines on MMRB2, GenAI-Bench, and Rank-Bench, including Kendall tau-b and stable accuracy under swapped image order. Rank-Bench rankings in the main table use all-pairs comparison with win-rate aggregation. A separate analysis examines lower comparison cost with shared-anchor inference. The authors also apply MarginRM as the reward in Flow-GRPO post-training of FLUX.1-dev and report gains on GenEval and T2I-CompBench relative to other reward models in their setup.

Critical strengths

The paper’s most consequential strength is the explicit treatment of candidate-set ranking as a first-class evaluation target, backed by Rank-Bench and metrics that go beyond isolated pairwise accuracy. The benchmark construction (dense human pairwise labels, tie handling, and consensus rankings) and the empirical gap between pairwise accuracy and Kendall tau-b, including cases where similar pairwise scores yield different set-level agreement, address a practical question for best-of-N selection and multi-sample RL: whether a reward model recovers a coherent order over all images for the same prompt.

A second strength is the coherence of the MarginRM design with that goal: a prompt-level rubric fixes what is being compared, a shared anchor fixes the visual reference, and criterion-level margins provide a structured signal that can be aggregated into comparable scores within a set.

Critical weaknesses

The paper argues that reliable candidate-set ranking requires a fixed rubric and a shared anchor, so each image receives a directly comparable score without exhaustive pairwise comparison or win-rate aggregation. The headline Rank-Bench results in Table 1, however, do not evaluate that procedure. Appendix A.4 states that MarginRM’s set-level ranking is still built from all unordered pairs and win-rate aggregation, the same construction used for pairwise baselines. Shared-anchor sorting appears only in a separate efficiency experiment. Because Kendall tau-b and the central comparison to UnifiedReward-Flex in the main table therefore reflect pairwise win-rate behavior rather than the proposed native ranking rule, the primary evidence does not test the mechanism the paper presents as the solution to set-level inconsistency.

The reward model’s training signal is overwhelmingly synthetic. Rubrics, per-criterion margins, and explanations are produced by a large teacher model from HPDv3 pairwise labels, with retention based largely on agreement of the aggregated margin direction with the human binary preference. The strongest ranking evaluation additionally relies on Rank-Bench, which the authors construct and define, including consensus steps that extend beyond raw votes. This combination is consequential because it becomes difficult to separate gains from the proposed rubric–anchor–margin interface from distillation of a stronger annotator and from evaluation on an in-house benchmark aligned with the method’s assumptions.

Questions for authors

Table 1 ranks MarginRM with all-pairs win-rate aggregation, while the method is defined for shared-anchor scores. Which of these do you treat as the primary test that MarginRM improves agreement with human consensus on Rank-Bench?

Under the shared-anchor sorting protocol in Table 3, does MarginRM still outperform UnifiedReward-Flex on mean Kendall τb at the same comparison budget?

What overlap exists between HPDv3 SFT prompts and the prompts in MMRB2, GenAI-Bench, and Rank-Bench?

What evidence shows that gains are due to the rubric–anchor–margin design rather than distillation from the large margin teacher?

Were teacher-generated criterion margins checked against human judgments beyond enforcing agreement of the aggregated direction with the binary preference label?

2 comments
AuthorsOpenReview ↗

Response to Reviewer buwH (Weaknesses 1 & Questions 1-3)

We sincerely thank you for your insightful and constructive feedback. To make our response easy to navigate and avoid redundancy, we have grouped overlapping Weaknesses (W) and Questions (Q) and organized our point-by-point responses in a [Qi & Wj] →\to [Ai] format below.

\quad

[Q1 & W1]: Primary evaluation protocol for Rank-Bench: Shared-anchor ranking (Table 3) vs. all-pairs win-rate aggregation (Table 1).

[A1]: We clarify that Shared-anchor ranking is our primary test for validating MarginRM’s native formulation, whereas all-pairs aggregation serves strictly as a compute-matched control against pairwise baselines reward models. Specifically, the two evaluations serve complementary purposes:

  1. Primary Test (Table 3, Shared-Anchor): Evaluates MarginRM in its intended O(K)O(K) deployment regime—ranking candidates directly via relative margins against shared anchor without exhaustive pairwise aggregation. Importantly, with just a single anchor (K−1K-1 comparisons), MarginRM achieves τb=0.485\tau_b = 0.485, already matching the exhaustive O(K2)O(K^2) all-pairs performance of UnifiedReward-Flex (0.4870.487).
  2. Compute-Matched Control (Table 1, All-Pairs): Because standard pairwise reward models suffer from severe tie and transitivity degradation when inducing a total order from shared anchors, evaluating all models under the identical O(K2)O(K^2) win-rate protocol in Table 1 isolates per-pair reward quality from comparison budget (see [A2] for a detailed comparison under strict anchor budgets).

\quad

[Q2 & W1]: Head-to-head comparison between MarginRM and UnifiedReward-Flex under the shared-anchor protocol at matched comparison budgets.

[A2]: Yes. Under the shared-anchor protocol at identical comparison budgets (1, 2, or 3 anchors), MarginRM consistently outperforms UnifiedReward-Flex on mean Kendall’s τb\tau_b and all accuracy metrics (see expanded Table below).

MethodStrategiesStable Acc. ↑Acc. w/ Tie ↑Acc. w/o Tie ↑Kendall's τb\tau_b ↑
UnifiedReward-Flex1 anchor33.1244.6543.070.331
2 anchor44.5654.9660.640.439
3 anchor50.4758.9267.630.459
MarginRM1 anchor55.1962.1575.110.485
2 anchor58.8463.3676.020.495
3 anchor61.5664.3777.830.511

With limited comparison information (e.g., 1-anchor), UnifiedReward-Flex suffer from coarse score resolution and frequent ties, resulting in a low Kendall’s τb\tau_b. While adding more anchors gradually mitigates this degradation (0.331→0.4590.331 \to 0.459), MarginRM produces fine-grained relative margins that maintain strong, stable ranking performance even at the lowest comparison budget (τb=0.485\tau_b = 0.485 at 1-anchor), and continues to improve as more anchors are added.

\quad

[Q3]: Prompt overlap between the HPDv3 training set and evaluation benchmarks (MMRB2, GenAI-Bench, and Rank-Bench).

[A3]: Exact-match prompt overlap between the HPDv3 SFT training set and all three evaluation benchmarks is negligible (0.00% on MMRB2 and Rank-Bench, and 0.25% on GenAI-Bench), ruling out data contamination. Specifically, checking exact string matches between all 532,507 HPDv3 SFT training pairs (154,968 unique prompts) and the evaluation prompts in each benchmark yields:

BenchmarkTotal PromptsUnique PromptsExact OverlapOverlap Rate (%)
MMRB21,00079100.00%
GenAI-Bench1,7341,20430.25%
Rank-Bench90290200.00%
AuthorsOpenReview ↗

Response to Reviewer buwH (Weaknesses 2 & Questions 4-5)

[Q4 & W2]: Is the model's performance attributable to the rubric–anchor–margin design, or to the distillation of a teacher model?

[A4]: MarginRM’s gains stem from the rubric–anchor–margin formulation rather than teacher distillation, as verified by three newly added experiments from both component ablation and teacher comparison perspectives:

  1. Expanded Component Ablation: We added new ablations under the exact same 9B backbone and training pairs to thoroughly isolate the effectiveness of the rubric–anchor–margin formulation:
    • Anchor-only (Binary SFT): compares image pairs using only binary labels without rubrics or margins;
    • Rubric-only (Pointwise): scores single images on rubrics without an anchor reference (degenerating into a pointwise model, so swap-order Stable Acc. is N/A);
    • Margin-only w/ Anchor: combines the anchor reference and rubric-level numeric margins, removing textual rationales;
    • MarginRM (Ours): the full formulation with anchor, rubric margins, and rationales. | Model / Variant | MMRB2 ↑ | GenAI-Bench ↑ | Stable Acc. ↑ | Acc. w/ Tie ↑ | Acc. w/o Tie ↑ | Kendall's τb\tau_b ↑ | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Anchor-only | 64.6 | 71.1 | 58.78 | 62.39 | 75.59 | 0.473 | Rubric-only | 60.3 | 67.5 | - | 59.72 | 67.85 | 0.438 | | Margin-only w/ Anchor| 70.1 | 71.7 | 58.96 | 63.46 | 76.73 | 0.508 | | MarginRM | 75.2 | 76.1 | 61.07 | 64.34 | 77.70 | 0.519 |

Neither Anchor-only (τb=0.473\tau_b = 0.473) nor Rubric-only (τb=0.438\tau_b = 0.438) alone is sufficient—lacking either fine-grained margin resolution or an anchor to calibrate cross-image score scales. Combining both in Margin-only w/ Anchor immediately boosts ranking (τb:0.473→0.508\tau_b: 0.473 \to 0.508, MMRB2: 64.6→70.164.6 \to 70.1), while adding rationale supervision in MarginRM further elevates performance across all benchmarks (τb=0.519\tau_b = \mathbf{0.519}, MMRB2 = 75.2\mathbf{75.2}, GenAI-Bench = 76.1\mathbf{76.1}).

  1. Direct Teacher Comparison: We newly evaluated the 397B teacher model (Qwen3.5-397B-A17B) across all benchmarks under two settings—Standard (direct preference) and Margin Rubric (prompted to output rubric margins)—and found that our 9B student substantially outperforms the 397B teacher in both modes: | Model / Variant | MMRB2 ↑ | GenAI-Bench ↑ | Stable Acc. ↑ | Acc. w/ Tie ↑ | Acc. w/o Tie ↑ | Kendall's τb\tau_b ↑ | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Qwen3.5-397B (Standard) | 71.6 | 68.0 | 56.42 | 59.66 | 72.06 | 0.451 | | Qwen3.5-397B (Margin Rubric) | 68.1 | 53.5 | 24.69 | 50.67 | 58.64 | 0.284 | | MarginRM (9B) | 75.2 | 76.1 | 61.07 | 64.34 | 77.70 | 0.519 |

Notably, prompting the untrained 397B teacher to directly emit rubric margins at test time severely degrades ranking (τb=0.284\tau_b = 0.284). During SFT data construction (Appendix B.2), teacher outputs are strictly conditioned on and filtered against human ground-truth labels—meaning the teacher acts solely as a human-anchored rationale decomposer rather than a stronger preference verifier.

\quad

[Q5 & W2]: Were teacher-generated criterion margins checked against human judgments?

[A5]: Beyond full-dataset sign filtering against binary human labels , we directly validated criterion-level margins during SFT data construction via iterative human-in-the-loop rubric calibration and a 200-pair blind human audit.

  1. Human-in-the-Loop Scale Calibration: Annotators iteratively inspected pilot samples to refine the rubric prompts (Appendix B.2), explicitly anchoring 0 (comparable), ±1 (minor difference), and ±2 (substantial difference) to concrete visual evidence to prevent score inflation or hallucinated sub-scores.
  2. 200-Pair Blind Human Audit: Prior to full-scale synthesis, human reviewers blindly audited 200 randomly sampled pairs and confirmed a >75% human approval rate on the criterion-level margin magnitudes and rationales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.