- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TRM is a generative pointwise reward model for image generation and editing that first writes a case-specific Yes/No checklist, checks each item, and then outputs a 0–10 score. It is trained by SFT on about 48K Gemini-annotated traces, followed by PD-GRPO with pairwise preferences, and is used as the reward for RL algorithms likFlow-GRPO.
Critical strengths
-
The checklist paradigm is reasonable and general. Deciding what to check for each case before scoring is a sensible way to make a reward model's judgment explicit, and the same procedure transfers naturally across tasks (generation and editing here, and potentially video or other modalities).
-
Broad coverage. One reward model handles both generation and editing, and it is used for Flow-GRPO on eight generators, including the already strong SenseNova-U1.5.
Critical weaknesses
-
Train-test overlap. TRM's training data includes Edit-Compass, RISE-Bench and EvalMuse. But EditReward-Compass is built from Edit-Compass, and MMRB2 samples its prompts from RISE-Bench and EvalMuse, while both EditReward-Compass and MMRB2 are used for evaluation.
-
Marginal gain from PD-GRPO. PD-GRPO adds only 1.1 / 2.1 points over SFT (Table 2), and 0.7 / 1.1 under the tie-aware protocol (Table 10).
-
Checklist quality is never evaluated. The checklist is the core of the paper, but its coverage, redundancy and the accuracy of the per-item yes/no judgments are never checked against Gemini or humans.
Questions for authors
-
Which splits of Edit-Compass, RISE-Bench and EvalMuse are used for training? Can you de-duplicate the training data against EditReward-Compass and MMRB2 and re-report Tables 2 and 3?
-
How good are the generated checklists? Can you report their coverage and redundancy against Gemini or human-written checklists, and the agreement between the per-item yes/no judgments and human labels?
-
Can you show concrete cases that SFT ranks wrongly and PD-GRPO fixes, and report multiple runs to confirm the gain?
-
With the same prompt and seed on the eval set, which checklist items get better or worse after RL, and what is the trend?
-
TRM samples at temperature 0.7 during PD-GRPO. How much do the checklist and score vary across samples for the same image, and does PD-GRPO reduce this variance? Which temperature is used when TRM serves as the reward in Flow-GRPO?
9 comments
Response to Weakness 2: Thank you for your suggestion. PD-GRPO’s gains are achieved on top of the strong TRM(SFT) baseline. TRM(SFT) already achieves state-of-the-art performance among open-source models on multiple reward-model benchmarks, and PD-GRPO further improves its preference prediction capability, with more pronounced gains on image editing benchmarks. Below, we explain the performance of the SFT baseline, the optimization objective of PD-GRPO, and its empirical gains.
- We use high-quality evaluation data to provide direct supervision for checklist generation, criterion-level judgments, and final score prediction, enabling the general-purpose backbone to learn the full Think Before You Score reward-modeling paradigm through structured SFT. As shown in Tables 2 and 3, TRM(SFT) achieves state-of-the-art performance among open-source models on multiple reward-model benchmarks for text-to-image generation and image editing, providing a strong starting point for subsequent PD-GRPO optimization.
- PD-GRPO introduces pairwise preference supervision to help independently assigned image scores reflect subtle quality differences between candidates. For example, two edited images may both complete the requested modification, while one blends more naturally with the original scene. Pairwise preferences provide a direct training signal for such differences.
- Starting from the strong TRM(SFT) baseline, PD-GRPO achieves further gains in reward-model evaluation for both text-to-image generation and image editing. In Table 2, TRM(SFT) already achieves accuracies of 70.1% and 65.8% on the two reward-model benchmarks for image generation, and PD-GRPO further improves these results by 1.1 and 2.1 percentage points, respectively. The gains are more pronounced for image editing: in Table 3, preference prediction accuracy improves by 5.2 and 3.5 percentage points on MMRB2 Edit and EditReward-ERB 2-path, respectively.
Response to Question 2: Thank you for your suggestion. To assess checklist quality, we randomly sampled 100 examples and evaluated TRM-generated checklists against human annotations in terms of coverage, redundancy, and agreement on criterion-level judgments. The checklists achieved 95% coverage, with redundancy found in 3 of the 100 checklists. Agreement between TRM’s criterion-level judgments and human annotations reached 94.82%. During data construction, human experts established and refined the annotation procedure, while a verifier based on Qwen3.5-397B-A17B further identified missing requirements and removed redundant or ambiguously worded checklist items to improve the quality of structured supervision.
Response to Question 3: Thank you for your suggestion. As shown in the case figures, TRM(SFT) produces rankings inconsistent with human preferences, whereas TRM(RL) produces rankings aligned with human preferences after PD-GRPO training. These cases illustrate how PD-GRPO corrects SFT’s preference prediction errors. Across all evaluation benchmarks, we evaluate both models using identical images and prompts, with the decoding temperature uniformly set to 0 to ensure stable and consistent evaluation.
Response to Question 4: Thank you for your question. We first analyze the improvements in checklist quality and criterion-level judgment accuracy following PD-GRPO training, and then examine how these improvements are reflected in reward-model benchmark results across the corresponding evaluation dimensions.
- During evaluation, we compare TRM(SFT) and TRM(RL) using identical inputs and set the decoding temperature to 0 for both models to maintain evaluation stability. After PD-GRPO training, the generated checklist items become more specific and easier to verify, and the criterion-level judgments become more accurate. For example, in image editing, the checklist explicitly checks whether the target object's specified attributes have been modified as requested and whether content outside the edited region has been preserved.
- These improvements to the checklist items are also reflected in higher scores on corresponding evaluation dimensions, such as instruction following and source consistency. In Table 2, TRM's preference prediction accuracy improves on both GenAI-T2I and MMRB2-T2I. In Table 3, instruction following, visual consistency, and overall evaluation on EditScore-ERB all improve, with instruction following increasing from 0.750 to 0.786 and visual consistency from 0.639 to 0.674. IA and VC on EditReward-Compass, as well as preference prediction accuracy on MMRB2 Edit and EditReward-ERB 2-path, also improve.
Response to Question 5: Thank you for your question. Below, we report score variability during PD-GRPO training and clarify the decoding temperature used for downstream reward evaluation.
-
Score variability for the same image and the effect of PD-GRPO. We compare score variability at the beginning and final step of PD-GRPO training using repeated samples for the same image. The average within-group variance of normalized scores decreases from 0.22 to 0.09, with an overall downward trend during training, indicating more stable scores for the same image after training. From an algorithmic perspective, GRPO updates the policy based on within-group relative advantages, increasing the probability of sampling outputs with higher rewards. This encourages more concentrated scores, consistent with the observed decrease in variance.
-
Decoding temperature in downstream Flow-GRPO. In downstream Flow-GRPO, we set TRM's decoding temperature to 0 to improve the stability of reward scores.
Think Before You Score: Thinking Reward Model for Visual Generation
Response to Weakness 1 and Question 1: Thank you for your question. Below, we explain our data construction rationale, discuss similar practices in prior work, and present downstream RL results to validate TRM's effectiveness in practice
We select source images and editing instructions from public benchmarks because they cover representative editing tasks and provide source-image–instruction pairs with clear editing objectives. For each task category, we refer to model rankings on relevant benchmarks to select editing models with different capability levels and generate candidate outputs for each fixed source-image–instruction pair. These candidates cover different quality levels and failure modes, providing fine-grained supervision for TRM's error detection and scoring. To further validate the effectiveness of our method, we conducted a training-data de-duplication experiment. Specifically, we checked the SFT training data against EditReward-Compass and MMRB2. We removed training examples that shared both the source image and editing instruction with evaluation examples, even when their edited images differed. We then retrained TRM(SFT) from the same base model, keeping all other training and evaluation settings unchanged. The results are reported below.
After de-duplication, performance on EditScore-ERB remains close to the original results, with absolute changes of at most 0.007 across IF, VC, and Overall. On EditReward-Compass, IA and VC decrease modestly by 0.012 and 0.015, respectively. These limited changes suggest that TRM(SFT)'s performance is not primarily dependent on the identified training–evaluation overlaps. Our method explicitly trains the model to formulate case-adaptive evaluation criteria and assess each candidate against them, providing a structured evaluation procedure that can be applied across different editing instructions and visual content. The largely preserved performance after de-duplication supports the effectiveness of this rubric-guided framework in learning transferable evaluation capabilities. Together with the downstream reinforcement learning improvements in Tables 4 and 5, these findings provide complementary evidence that TRM generalizes as an evaluator and produces useful reward signals for optimizing image generation and editing models.
Similar data construction strategies have been used in reward modeling. EditReward collects source images and editing instructions from benchmarks such as GEdit-Bench and ImgEdit-Bench and generates candidates using six editing models [1]. RationalRewards and CheckDistill also use EditReward-Data to train their image editing reward models [2, 3]. ADIEE uses source images from the Emu-Edit test set, creates new global editing and style editing instructions for these images, and generates candidates to train an editing scorer [4]. In language and reasoning tasks, UltraFeedback and C2RM collect tasks from benchmarks such as TruthfulQA, SciEval, and LogiQA, generate candidate responses using multiple models, and construct reward supervision from these responses [5, 6].
To further validate TRM's effectiveness in practice, we evaluate its effectiveness as a reward model for reinforcement learning across three base image generation models and five image editing model configurations. As shown in Tables 4 and 5, TRM-guided reinforcement learning improves FLUX.1-dev's TIIF-Long score from 74.82 to 81.38 and BAGEL's ImgEdit score from 3.37 to 3.91. BAGEL's overall scores on GEdit-Bench-EN and GEdit-Bench-CN also improve from 6.95 and 6.98 to 7.57 and 7.64, respectively.These results further demonstrate that TRM provides effective reward signals for different image generation and editing models, improving downstream performance.
References
[1] Wu et al. EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing. ICLR 2026.
[2] Wang et al. RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time. COLM 2026.
[3] Cai et al. CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing. arXiv:2610.26740, 2026.
[4] Chen et al. ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation. ICCV 2025.
[5] Cui et al. UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML 2024.
[6] He et al. Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning. EMNLP 2025.