acceptodds
Withdrawn from ICLR 2027

Pair2Point: Reliable as Pairwise, Efficient as Pointwise Rewards for High-Quality Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent multimodal reward models (RMs), especially those built on powerful vision-language models, have shown great promise in aligning vision models with human preferences. However, they face two main challenges. First, their training images often come from earlier generations of text-to-image models and no longer match the quality of current generators. Second, it is difficult to combine the reliable judgments of pairwise comparison with the efficiency of pointwise scoring. To address these challenges, we propose Pair2Point. We build three resources with recent generators for training and evaluation, including P2P-CoT-16K, P2P-Image-70K, and P2P-Bench. On top of these resources, a two-stage framework first trains a pairwise model with structured comparison labels from Gemini 3.1 Pro, so that it evaluates two images along relevant dimensions and explains its judgment; it then distills this model into a pointwise model through multi-opponent aggregation. Experiments show that Pair2Point achieves the highest accuracy among the evaluated dedicated reward models across multiple reward benchmarks. When used for down-stream reinforcement learning, it consistently improves bothtext–image alignment and visual quality. The resources and model weights will be released publicly after review.

Primary area:
generative models
Keywords:
Reward Model, Text-to-Image Generation, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MnmGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Pair2Point is a VLM-based text-to-image reward model. It first trains a pairwise judge on Gemini-annotated, dimension-level comparisons, then distills it into a pointwise scorer by averaging the judge's scores over same-prompt opponents, and uses it as the reward for Flow-GRPO.

Critical strengths
  • Useful data contribution. Existing preference data mostly comes from older generators. The three resources built on recent models (P2P-CoT-16K, P2P-Image-70K, P2P-Bench) fill this gap, and P2P-Bench with unanimous six-annotator labels is a nice benchmark for modern reward models if released.
Critical weaknesses
  • Why bother with the pairwise teacher? Gemini already provides per-image, dimension-level scores and weights for every pair (Fig. 2). The simplest baseline is to turn these into one score per image with the same weighted rule as Eq. (7) and regress a pointwise model on it directly. Why train a pairwise teacher first and then distill it again into a pointwise student? This is the most important ablation for the paper's central design, and it is missing. The regression-head row in Table 3 doesn't answer it, since it uses a BT loss on pairwise winners only and drops the dimension-level labels.

  • Missing comparisons with the closest baselines. JRM, Z-Reward and RewardDance share the same motivation (reliable judgment during training, cheap pointwise scoring at inference) and are discussed in the paper (lines 75–77, 141–143), but none of them appears in Table 1 or Table 2.

  • Messy citations. Flow-GRPO is cited as Shao et al., 2024 (lines 151, 316), which is DeepSeekMath, the original GRPO paper. The actual Flow-GRPO paper isn't in the references at all. Similarly, "direct preference optimization" for diffusion and flow models (line 151) cites the LLM DPO paper instead of Diffusion-DPO. Core components the paper builds on, such as Qwen3-VL, FLUX.1-dev, Qwen-Image and LoRA, are not cited either.

Questions for authors
  • Can you add a baseline that converts Gemini's dimension-level scores into a per-image score with a rule like Eq. (7) and regresses a pointwise model directly, without the pairwise teacher? How does it compare with Pair2Point on Table 1 and in Flow-GRPO?

  • Can you compare against JRM, Z-Reward and RewardDance on both preference accuracy and downstream RL?

Reviewer_WewDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a reward model training strategy for post-training text-to-image diffusion models. Specifically, the contributions made by the paper are as follows:

  1. A high-quality preference dataset consisting of images from more recent models. These are labeled using Gemini 3.1 Pro and are also reviewed with human intervention where necessary.

  2. Training the reward model to initially be trained on pairwise reward labeling, again using the Gemini 3.1 Pro as supervision. The key novelty in the paper is to then adapt it to a point-wise training strategy. Here, it's done with a multi-opponent training setup, where the labels are essentially generated by contrasting a sample against multiple other samples for the same prompt.

Results on both reward modeling benchmarks and some downstream benchmarks (e.g. GenEval or UniGenBenchreward model scores like HPS) indicate that this reward model can be an effective tool in post-training text-to-image models.

Critical strengths

Overall, I think the method pursued in the paper is quite logical, sound, and intuitive.

The paper is also well-presented and sufficiently easy to follow.

The experimental results also indicate the promise of the idea.

Critical weaknesses

Broadly, I would say that I can see a few issues with the paper:

  1. While the title itself highlights the efficiency aspect of pointwise reward models, there's no result which really quantifies it. While I don't think it's particularly critical, it would still be good to have concrete results showing benefits of pointwise RMs.

  2. I think the most critical issue is the fact that most of the method could really be considered some form of distilling Gemini 3.1. As most multimodal benchmarks might indicate, Gemini 3.1 is definitely a much stronger model than any of the other comparisons (e.g. Qwen/HPS), and much of the gains seen in the paper could in theory be attributed to the use of Gemini vs the specifics of the Pair2Point method.

  3. Again while not the most critical issue, the improvements on the benchmarks does seem quite limited (say everything except UniGenBench with Flux shows minimal improvements vs other RMs).

Questions for authors
  1. The most critical question is regarding the effect of the Gemini. One idea I could think of is having a baseline pointwise regression head directly trained on the Gemini aggregated outputs, ignoring all the other steps, might be a good starting point, in addition to anything else the authors might have in their mind.

  2. Providing more contextualization to show whether or how significant the improvements are, especially compared to the other baselines.

Reviewer_1393OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between older preference datasets and contemporary text-to-image generators, together with the cost of deploying pairwise reasoning-based reward models. It introduces P2P-CoT-16K for structured comparison training, P2P-Image-70K for pointwise distillation, and P2P-Bench for evaluation against unanimous human preferences. A Qwen3-VL-based pairwise teacher learns dimension-level explanations, scores, weights, and preference verdicts from human-screened Gemini 3.1 Pro annotations. Its judgments over multiple same-prompt opponents are aggregated into targets for a pointwise student, which produces dimension-level scores and weights without long explanations. Pair2Point achieves improved preference prediction accuracy across three benchmarks. When used as the reward for Flow-GRPO fine-tuning of FLUX.1-dev and Qwen-Image-2512, it improves text–image alignment and visual preference scores over the base generators.

Critical strengths
  1. Clear motivation and useful contributions to preference data and judging. The paper introduces updated datasets covering recent text-to-image generators, with human screening of Gemini-generated training annotations and unanimous human preferences for P2P-Bench. This annotation and verification effort supports data quality. The proposed Pair2Point framework then transfers structured pairwise judgments into efficient pointwise scores through multi-opponent aggregation, forming a clear and coherent approach to reward modeling.

  2. The learned reward translates into improved generation. Using Pair2Point as the reward for Flow-GRPO improves text–image alignment and visual preference scores on both FLUX.1-dev and Qwen-Image-2512. This connects better preference prediction with downstream generation improvements, providing a coherent validation from reward modeling to practical use.

Critical weaknesses
  1. Limited separation between scores despite differences in visual quality. Based on my visual impression of the second example in Figure 1, the GPT-Image-2 output looks noticeably better than the Qwen-Image-2512 output, yet their Pair2Point scores are very close (7.88 vs. 7.81). Although this is a subjective comparison and the ranking is consistent with my preference, it raises a question about how well score differences reflect perceived quality differences.

  2. The choice of Gemini 3.1 Pro as the annotation model needs stronger justification. Were other strong models, such as GPT-based judges, evaluated before selecting Gemini 3.1 Pro? A comparison of their agreement with independently collected human annotations would provide a clearer empirical basis for this choice.

  3. Lack of novelty. The full framework benefits from structured annotations and additional distillation data, whereas the regression-head baseline uses only the 16K preference pairs. Comparisons under matched data and supervision would help establish whether the gains come from the proposed method or the additional training signals.

  4. Downstream improvements are modest and not consistently evident visually. Several Pair2Point results in Table 2 are close to UnifiedReward. The examples in Figure 4 also do not show a consistently clear visual improvement over competing rewards.

Questions for authors
  1. How should the small score gap between the Qwen-Image-2512 and GPT-Image-2 outputs in Figure 1 be interpreted? Do reward margins correlate with human-perceived quality differences?

  2. What motivated selecting Gemini 3.1 Pro for annotation? Were alternatives, such as GPT-based judges, compared against independent human annotations?

  3. How can the authors more convincingly demonstrate the advantage of the proposed reward over existing alternatives? The results in Table 2 and the visual examples in Figure 4 do not show a clear, substantial improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.