acceptodds
Withdrawn from ICLR 2027

Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reward model, image generation, image editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zg2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes two specialized reward modeling pipelines. FIRM-Edit uses a difference-first procedure to evaluate editing execution and source consistency, while FIRM-Gen uses a plan-then-score procedure for complex text-to-image prompts. The paper further proposes CME and QMA reward formulations to mitigate reward hacking during RL.

Critical strengths

The paper covers reward data construction, human evaluation, reward model training, benchmarking, and downstream RL in a relatively complete system. The specialized critics consistently improve performance on both editing and generation tasks. The discussion of the interaction between reward formulation and reward hacking is also practically relevant for RL-based image generation and editing.

Critical weaknesses
  1. The principles behind CME and QMA are still largely heuristic, and the evaluation covers only a limited set of reward-hacking behaviors. The paper mainly studies copying the source image and generating low-quality black silhouettes. Other realistic failure modes include modifying an easy region while ignoring the main instruction and so on. A systematic adversarial reward-hacking suite would help determine whether CME and QMA generalize beyond the specific shortcuts studied in the paper. Comparisons with weighted sums, multiplicative formulations, constrained optimization, and Pareto-based approaches would also be useful.

  2. The generalization of the critic to reference-based editing is not fully established. Although FIRM-Edit receives both source and edited images, the training distribution and generator diversity may limit generalization. The authors should test unseen editing models, local edits, multi-step edits, identity preservation, text editing, and different image resolutions.

  3. The data construction pipeline relies heavily on MLLM-generated difference descriptions, scoring plans, and training labels. FIRM-Bench contains only 807 samples, and the paper does not sufficiently report inter-annotator agreement or how teacher errors propagate into the final critic.

Questions for authors

See Critical Weaknesses

Reviewer_egyLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose FIRM, a framework with two data-construction pipelines, FIRM-Edit and FIRM-Gen:

  • Editing labels come from a "difference-first" pipeline: an MLLM describes the differences between the source and edited images, then scores Execution and Consistency (1–5).
  • Generation labels come from a "plan-then-score" pipeline: Qwen3-32B writes a prompt-specific checklist that guides Qwen3-VL-235B.

The resulting datasets, FIRM-Edit-370K and FIRM-Gen-293K, are used to train Qwen3-VL-8B into the reward models FIRM-Edit-8B and FIRM-Gen-8B. The critics are validated by MAE on FIRM-Bench, which comprises 807 samples (557 for FIRM-Bench-Edit and 250 for FIRM-Bench-Gen). FIRM-Edit-8B outperforms open-source MLLMs (the Qwen3-VL and InternVL3.5 families) and GPT-5 on FIRM-Bench.

The paper also proposes reward formulations for image editing and image generation:

  • Consistency-Modulated Execution (CME) for image editing: R=Exec⋅(0.6+0.4⋅Consistency)R=\text{Exec}\cdot(0.6+0.4\cdot\text{Consistency}).
  • Quality-Modulated Alignment (QMA) for image generation: R=IF⋅(0.4+0.6⋅Quality)R=\text{IF}\cdot(0.4+0.6\cdot\text{Quality}).

With RL, Qwen-Image-Edit-2509 improves on GEdit (7.54 → 7.84) and ImgEdit (4.35 → 4.42), and SD3.5-Medium improves on GenEval, DPG, TIIF and UniGenBench++. In both cases, the gains exceed those obtained with Qwen3-VL-8B/32B critics (Tab. 3, 4).

Critical strengths

The paper provides a complete pipeline: dataset generation, reward models, a benchmark and the labeling prompts (App. G, H). Swapping critics under a fixed RL setup shows that FIRM outperforms both Qwen3-VL-8B and the larger Qwen3-VL-32B (Tab. 3, 4).

Critical weaknesses
  • Limited novelty and missing comparisons. MLLM-as-judge evaluation and editing reward models [1, 2, 3], checklist-based scoring [4, 5] and multiplicative reward gating are already established. However, the paper does not compare against these methods on FIRM-Bench or use them as RL baselines. Its main contribution is the reward-hacking analysis (Tab. 5, App. F). The paper also lacks ablations of the difference descriptions and checklists, and evaluates the critics only with MAE, without annotator agreement. Moreover, FIRM-Gen-8B underperforms Qwen3-VL-235B and GPT-5 on Gen-Hard (0.57 vs. 0.52/0.53). The gains may therefore come from distilling a larger teacher on in-domain data rather than from the proposed method.
  • Under-specified RL evidence. QMA's "Quality" term is never defined, and FIRM-Gen-8B's scoring prompt outputs only an alignment rating (App. H), so the generation reward behind Tab. 4 is not fully specified. App. B implies that the Qwen3-VL baselines differ only in the critic, but this is not stated explicitly. In addition, several margins over the Qwen3-VL critics are small (ImgEdit: +0.07 vs. +0.01; GenEval: 0.77 vs. 0.76), making it difficult to attribute the improvements specifically to FIRM.

[1] M. Ku, D. Jiang, C. Wei, X. Yue, and W. Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. In ACL, 2024.

[2] X. Luo, J. Wang, C. Wu, S. Xiao, X. Jiang, D. Lian, J. Zhang, D. Liu, et al. EditScore: Unlocking online RL for image editing via high-fidelity reward modeling. In ICLR, 2026.

[3] K. Wu, S. Jiang, M. Ku, P. Nie, M. Liu, and W. Chen. EditReward: A human-aligned reward model for instruction-guided image editing. In ICLR, 2026.

[4] Y. Hu, B. Liu, J. Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023.

[5] J. Cho, Y. Hu, R. Garg, P. Anderson, R. Krishna, J. Baldridge, M. Bansal, J. Pont-Tuset, and S. Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In ICLR, 2024.

Questions for authors
  1. [Clarification] How do difference-first and plan-then-score differ from prior MLLM-as-judge [1, 2, 3] and checklist-based [4, 5] methods? What is the FIRM-Bench MAE with and without the difference descriptions and checklists?
  2. [Clarification] Could you evaluate EditScore and EditReward on FIRM-Bench-Edit, with additional metrics such as rank correlation and pairwise accuracy? What is the agreement between human annotators?
  3. [Clarification] Which model generates QMA's "Quality" score?
Reviewer_3EiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FIRM, a framework for reward modeling and RL in image editing and text-to-image generation. It has two annotation pipelines ("difference-first" for editing, "plan-then-score" for generation), two fine-tuned Qwen3-VL-8B reward models, a new FIRM-Bench, and two reward aggregation strategies, CME and QMA. Experiments look at both agreement with human scores and downstream gains.

Critical strengths
  1. It tackles a real bottleneck. Unreliable reward signals hurt RL for image generation and editing, so checking both reward-model accuracy and downstream policy performance is worthwhile.

  2. Separating "observe the visual difference" from "evaluate it given the instruction" is a sensible way to avoid missed changes and to judge execution and consistency better.

  3. The work provides useful resources and broad downstream evaluation. The two scoring datasets, specialized reward models, and human-annotated benchmark could benefit subsequent research.

Critical weaknesses
  1. Novelty needs clearer positioning against close prior work. Several ingredients in §§2.1–2.3 already have substantial precedent. [1] already covers automated annotation → specialized generative reward modeling → benchmarking → online RL, and it also evaluates a Qwen3-VL-8B reward model. [2] generates prompt-specific criteria and uses criterion-level judgments as rewards. [3] combines task-decomposed evaluation, teacher supervision, evaluator distillation, score rebalancing, and human meta-evaluation. The most distinctive parts seem to be the dual-image difference-first annotation and the asymmetric CME formulation.

  2. The experiments don't isolate what the annotation pipelines contribute. Tables 1–2 show that task-specific fine-tuning helps, but not that difference-first or plan-then-score annotation is responsible. There's no comparison with the same source examples, teacher, student backbone, dataset size, and training budget, changing only the annotation procedure.

  3. Reward-model comparisons and metrics are incomplete. Tables 1–2 mostly compare FIRM with general-purpose VLMs. Comparisons with EditScore and other relevant reward models, plus evaluation on external benchmarks, would help.

[1] EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling. [2] RubricRL: Simple Generalizable Rewards for Text-to-Image Generation. [3] Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark.

Questions for authors
  1. What specific capability or mechanism separates FIRM-Gen's plan-then-score pipeline from RubricRL [2] and T2I-Eval [3]? How does FIRM-Edit differ from an EditScore-style evaluator with explicit difference descriptions added?

  2. Can you compare direct scoring with the staged pipelines under matched teachers, source data, student backbones, and budgets? Do human audits confirm that staged annotation gives more accurate labels?

  3. What model and scoring procedure produce the quality term in QMA? How many independent annotators label each benchmark example, how are disagreements resolved, and what are the uncertainty estimates for the main results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.