- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Rank-Bench which measures rank-preference recovery of image reward model. It also proposes MarginRM, a novel multi-aspect margin-based reward model that achieves high performance on the proposed Rank-Bench. The author also demonstrated that such reward model can be useful in RL.
Critical strengths
- This paper addresses the fundamental limitation of existing pair-based reward modeling and reward model evaluations, providing valuable insight to the community.
- MarginRM show considerable performance gains compared with existing reward models both in preference recovery and RL Post-training.
Critical weaknesses
Overall, the paper is sound and MarginRM's contribution is clear. My main concern is on the soundness of post-training experiment
- The post training experiment lacks compute analysis and Appendix C.3 is insufficient. Specifically, the proposed model seems larger than some of the baselines like HPSv2, the author should discuss the potential overhead of using Qwen3.5-9B based model. Does the model converge given the fixed training step, or will training longer improve the model further? If latter is the case, the author should provide compute-matched results by allowing RL run with cheaper reward models to traing longer to match the compute cost measured by wall-clock time
- The original flow-GRPO use SD3.5-M and FLUX.1-dev as the base model, this paper uses only FLUX.1-dev. It would be better to show results across different model families for post training experiment. One particular confounding factor that needs to be isolated is that in building MarginRM, some images are generated with FLUX.1-dev, so the reward model may work better on this specific model. Showing results on SD3.5 will demonstrate if MarginRM generalize beyond the set of models in its training data, or can be generalized.
Questions for authors
See weakness.
- What is the number of steps/epochs of RL post training and what is the compute overhead of MarginRM compared with baselines
- Does MarginRM's effectiveness in post-training generalized beyond the list of models in A.2 (SDXL, SD3, FLUX.1-dev, Z-Image, Nano Banana, Qwen-Image, GPT-Image, and Banana-Pro) ? Experiments on one other models like SD3.5-M,or Sana will clarify this problem.
- What is the breakdown of T2I CompBench score? Specifically, T2I comp bench have detailed rubrics on color, shape, texture, it would be interesting to show how it aligns with rubics of MarginRM.
4 comments
Response to Reviewer zdrN (Weaknesses 2 & Questions 2-3)
[Q2 & W2]: Cross-family generalization of MarginRM in RL post-training to unseen base generators (SD3.5-Medium) not included in the MarginRM training set (Appendix A.2).
[A2]: MarginRM generalizes effectively to unseen base generators outside Appendix A.2, ruling out training-source bias toward FLUX.1-dev. As shown by our new Flow-GRPO post-training experiment on SD3.5-Medium (excluded from Appendix A.2), MarginRM achieves substantial gains over the base model and consistently outperforms UnifiedReward-Flex across all four benchmarks:
| Backbone / Post-Training Method | GenEval ↑ | T2I-CompBench ↑ | CLIP ↑ | PickScore ↑ |
|---|---|---|---|---|
FLUX.1-dev (Base) | 63.72 | 49.22 | 35.27 | 22.61 |
| ↳ + UnifiedReward-Flex | 68.91 | 51.22 | 36.49 | 23.38 |
| ↳ + MarginRM (Ours) | 72.61 | 54.38 | 38.44 | 23.36 |
SD3.5-Medium (Base, Unseen in A.2) | 65.79 | 49.95 | 39.33 | 22.53 |
| ↳ + UnifiedReward-Flex | 69.27 | 50.09 | 38.82 | 22.94 |
| ↳ + MarginRM (Ours) | 72.70 | 52.90 | 39.91 | 22.98 |
[Q3]: Per-category breakdown of T2I-CompBench.
[A3]: Below we report the per-category breakdown on T2I-CompBench.
| Method | Color | Shape | Texture | 2D Spatial | 3D Spatial | Numeracy | Non-Spatial | Complex | Overall |
|---|---|---|---|---|---|---|---|---|---|
| FLUX.1-dev (Base) | 0.709 | 0.584 | 0.605 | 0.285 | 0.429 | 0.629 | 0.306 | 0.391 | 0.492 |
| w/ HPSv2 | 0.539 | 0.543 | 0.501 | 0.241 | 0.355 | 0.561 | 0.294 | 0.330 | 0.421 |
| w/ HPSv3 | 0.627 | 0.544 | 0.555 | 0.255 | 0.375 | 0.619 | 0.302 | 0.349 | 0.453 |
| w/ PickScore | 0.613 | 0.543 | 0.578 | 0.245 | 0.338 | 0.570 | 0.299 | 0.343 | 0.441 |
| w/ UnifiedReward | 0.769 | 0.538 | 0.645 | 0.298 | 0.413 | 0.652 | 0.311 | 0.388 | 0.502 |
| w/ UnifiedReward-Think | 0.791 | 0.529 | 0.662 | 0.304 | 0.427 | 0.668 | 0.294 | 0.396 | 0.509 |
| w/ UnifiedReward-Flex | 0.790 | 0.559 | 0.672 | 0.306 | 0.396 | 0.671 | 0.304 | 0.394 | 0.512 |
| w/ MarginRM (Ours) | 0.776 | 0.563 | 0.718 | 0.320 | 0.438 | 0.684 | 0.307 | 0.544 | 0.544 |
While holistic scalar RMs (HPSv2/v3, PickScore) degrade compositional fidelity below the base model (0.492 0.421–0.453) by over-optimizing global aesthetics, MarginRM explicitly decomposes prompts into criterion-level margins covering attribute binding (color/shape/texture), spatial layout (2D/3D), object counting (numeracy), and multi-attribute interactions (complex) (Appendix B.1). This targeted credit assignment yields the highest post-training scores on Shape (0.563), Texture (0.718), 2D/3D Spatial (0.320 / 0.438), Numeracy (0.684), and Complex (0.544).
Thank the authors for the response. My concerns have been addressed.
Replying to Reviewer zdrN
Thank you for your positive feedback and for taking the time to review our response. We are glad that our response has addressed your concerns. We sincerely appreciate your support and constructive comments, which helped us improve the paper. Thank you again for your constructive feedback and valuable suggestions.
Response to Reviewer zdrN (Weaknesses 1 & Questions 1)
We sincerely thank you for your thoughtful review and constructive feedback. To make our response easy to navigate, we have merged overlapping points from your Weaknesses (W1, W2) and Questions (Q1-Q3) and organized our point-by-point replies in a [Qi & Wj] → [Ai] format below.
[Q1 & W1]: (1.) Complete RL post-training steps and hyperparameters, (2.) convergence verification for lower-cost reward models (HPSv2, PickScore), and (3.) wall-clock compute overhead of the Qwen3.5-9B-based reward model.
[A1]: Regarding
Appendix C.3and post-training compute analysis: (1) we trainMarginRMfor 390 epochs (780 steps) with complete hyperparameters detailed below; (2) all baselines are evaluated at their converged optima, as training lower-cost RMs longer leads to reward over-optimization and performance degradation; and (3) thanks to its O(K) shared-anchor design, our Qwen3.5-9B-basedMarginRMrequires only 64.6s of reward model scoring per epoch (30.2% of total runtime), achieving 2.7× faster epoch wall-clock time than existing Qwen3.5-9B-based pairwise reward models.RL Training Steps & Hyperparameters (
Appendix C.3): ForMarginRM, Flow-GRPO post-training runs for 390 epochs (780 optimizer steps), with complete training configurations detailed below: | Sampling | | Optimizer | | Training & LoRA | | |:---|:---:|:---|:---:|:---|:---:| | Parameter | Value | Parameter | Value | Parameter | Value | | Rollouts per prompt (K) | 24 | Optimizer | AdamW | PPO clip range (ϵ) | 10−4 | | Batches per iteration | 48 | Learning rate | 3×10−4 | Ratio clip range | [0.9999, 1.0001] | | Image resolution | 512 × 512 | Adam β1 | 0.9 | Advantage clip range | [-5, 5] | | Sampling steps | 6 | Adam β2 | 0.999 | LoRA rank (r) | 64 | | Timestep fraction | 0.99 | Adam ϵ | 10−8 | LoRA α | 128 | | Guidance scale | 3.5 | Weight decay | 10−4 | Total epochs (steps) | 390 (780) |Convergence vs. Training Longer: Rather than stopping at an arbitrary fixed step, we evaluate each reward model at its converged optimum—selected as the checkpoint achieving the highest PickScore after the training reward plateaus (step 320 for
HPSv2at 22.82; step 440 forPickScoreat 23.93). Training cheaper scalar RMs (HPSv2,PickScore) beyond this peak degrades performance across metrics due to reward over-optimization rather than improving it: | Reward Model | Checkpoint Step | GenEval ↑ | T2I-CompBench ↑ | CLIP ↑ | PickScore ↑ | | :---: | :---: | :---: | :---: | :---: | :---: | | HPSv2 | 240 | 56.65 | 42.38 | 31.25 | 22.52 | | | 280 | 54.86 | 42.16 | 31.29 | 22.72 | | | 320 (Optimum) | 56.93 | 42.14 | 31.45 | 22.82 | | | 360 | 52.94 | 42.18 | 31.42 | 21.89 | | | 400 | 50.81 | 42.36 | 31.48 | 21.61 | | PickScore | 360 | 57.31 | 43.26 | 33.55 | 23.52 | | | 400 | 56.29 | 43.94 | 33.49 | 23.72 | | | 440 (Optimum) | 56.96 | 44.12 | 33.48 | 23.93 | | | 480 | 56.55 | 44.29 | 33.42 | 23.89 | | | 520 | 56.63 | 43.91 | 33.48 | 23.61 |Compute Overhead (
MarginRMvs.Qwen3.5-9B-based Baselines): Thanks to its O(K) shared-anchor formulation (K−1 calls),MarginRMremains by far the fastest method even while evaluating 2× more rollout images per epoch (1,152 vs. 512–576). Despite handling double the visual throughput,MarginRMreduces total RM calls per epoch from 1,792–2,304 down to 1,104, cutting per-epoch RM scoring time by 3.1×–4.9× and overall epoch wall-clock time by 2.7× compared to O(K2) pairwiseUnifiedReward-ThinkandUnifiedReward-Flexbaselines: | Metric | UnifiedReward-Think | UnifiedReward-Flex | MarginRM (Ours) | | :--- | :---: | :---: | :---: | | Prompts per epoch | 64 | 64 | 48 | | Rollouts per prompt (K) | 8 | 9 | 24 | | Images per epoch | 512 | 576 | 1,152 | | RM calls per prompt | 28 | 36 | 23 | | RM calls per epoch | 1,792 | 2,304 | 1,104 | | Mean latency per prompt | 9.19 s | 9.00 s | 4.46 s (2.0× faster) | | Mean total RM time / epoch | 202.8 s | 314.6 s | 64.6 s (3.1×–4.9× faster) | | Mean total wall-clock time / epoch | 588.2 s | 576.0 s | 214.0 s (2.7× faster) |