acceptodds
Under review as a conference paper at ICLR 2027

VisualRefineBench: Can Unified Multimodal Models Reliably Refine Their Own Images?

Abstract

Unified multimodal models (UMMs) can inspect and revise their own images at test time, but their gains in final image quality remain unclear. We introduce **VisualRefineBench**, combining end-to-end evaluation with three diagnostic tasks across 800 examples. Refine-E2E measures initial-to-delivered improvement and records the best image score during refinement. Under fixed inputs, Reflection tests error diagnosis, Decision tests action choice, and Editing measures net gains from instructed edits. With a four-image-call budget, **all six evaluated UMMs deliver lower mean quality than their corresponding Oracle best-of-N baselines**. Diagnostics reveal missed errors, poor action choices, and damage to correct content that often offsets repair gains. Replacing the Controller with a model stronger in Reflection improves mean final quality by **about 17 points** across four generator–editor combinations. These gains primarily reflect higher-quality candidates produced during refinement, while reliable final-image selection remains a challenge. These findings identify opportunities to improve UMM self-refinement and show how diagnostic evaluation can guide component choices that improve end-to-end outcomes.

Primary area:
datasets and benchmarks
Keywords:
test-time scaling, image self-refinement, unified multimodal models, image generation, image editing, reflection, decision-making, multimodal benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4WcXOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VisualRefineBench looks at whether UMMs really produce better images when they self-refine, given a hard cap on how many image calls they can make with N=4. It runs Refine-E2E and compares initial, best intermediate and final outputs against the Oracle Best-of-N upper bound alongside three fixed-input diagnostic tests, and we evaluated six different UMMs. Only a handful see modest improvements, and none hit their Oracle-BoN ceiling. Quite often, the best intermediate output beats the final one by a wide margin. Swapping in a more capable controller boosts final scores by roughly 17 points across four generator-editor setups, but picking the best result at the end still struggles.

Critical strengths

Diagnostics are not an afterthought. Reflection, decision-making, and editing each pull apart separate stages that get mixed together in an end-to-end pipeline. SignedEditScore works well for editing tasks because it directly penalizes harm to content that was already correct; tracking only the net change in quality would miss this kind of failure entirely.

The gap between the best intermediate result and the final output, paired with our controller swap experiments, explains why self-refinement falls short of Oracle-BoN. The models can generate stronger candidate images yet still end up with weaker final outputs. The real bottleneck lies in planning and selecting outputs, not the editing module itself. This is a concrete takeaway rather than just another leaderboard observation.

Critical weaknesses

Our main quantitative results rely on an automated judge. For this benchmark paper, we need human validation of ReflectionScore and SignedEditScore on a held-out subset, plus verification that the roughly 17-point gain from swapping the controller does not depend on this specific judge. Without these checks, we cannot firmly trust absolute claims about how far scores fall below the Oracle bound.

There is also a subtle tension when we frame the work as UMM self-refinement while using a separate orchestrated stack of controller, generator and editor. We should state plainly for every end-to-end setup whether these three roles run from one single checkpoint controlled by prompts, or from separate external models. If we skip this clarification, readers may misinterpret our negative results to mean unified world models are inherently bad at refinement, when the real limitation could come from how we structured the whole system.

Questions for authors
  1. Human–Judge agreement on Reflection and SignedEditScore (n≈50–100)? Does Controller replacement still win under human labels?
  2. Default Refine-E2E: same UMM for all roles, or mixed specialists?
  3. Do Final−Init conclusions flip at N=2 or N=8? What if delivery is “max Judge score so far” instead of the model’s STOP choice?
  4. Related work gap: I think some paper like Unicorn and SRUM are related, would you mind to discuss them in your related part?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4WcX (1/4): W1 & Q1

We thank the reviewer for the careful and constructive review, and for noting that the diagnostics are not afterthoughts and that the Best−Final gap together with the Controller replacement is a concrete finding.

W1 & Q1: human agreement and whether the Controller gain is Judge-specific.

We fully agree that a benchmark built on an automatic Judge should be validated against human judgments. We answer in two parts.

Part 1. Human agreement on ReflectionScore and SignedEditScore

Following your suggestion, we sampled 50 editing pairs and 75 Reflection reports from the paper's runs, and additionally 194 E2E images from 50 random tasks. The Judge and two state-of-the-art models from other families, Claude Opus 5.5 Extra-High and GPT-6 Astra Ultra, labeled every rubric item with the Judge's instructions. An experienced human researcher, who did not know which model produced each label or which condition an image came from, resolved every disagreement (for E2E, on 20 of the 50 tasks) and spot-checked a random sample of unanimous labels (the human's verdict is used wherever given). For ReflectionScore, the researcher also checked 60 reports drawn at random from all 24 Reflection models, defect by defect, blind to the Judge's score.

Agreement with human-verified labelsJudge
SignedEditScore (50 editing pairs)Spearman 0.80; mean difference −0.05 [−0.17, 0.07]
Editing, rubric level (50 pairs)κ = 0.82 [0.74, 0.89]; 96% agreement
E2E, rubric level (20 tasks, 1,168 labels)κ = 0.79 [0.74, 0.84]; image-score Spearman 0.91
ReflectionScore (60 random reports)65% exact [52, 77]; human mean 0.32 vs. 0.17

Editing and E2E. The Judge agrees closely with the human-verified labels. On E2E it scores images 6.3 points higher on average while ordering them consistently.

Reflection. Agreement is more limited.

  • On the 60 random reports (weighted to the full set's Judge-score mix), the human gives the same 0/1/2 score as the Judge on 65% [52, 77] of reports. Where they differ, the human is mostly more lenient (higher on 30% of reports, lower on 5%): mean 0.32 vs. 0.17 (difference 0.15 [0.07, 0.23]).
  • The gap is shared by all three models: they agree well with one another (weighted κ = 0.71–0.84), and on 20 further reports all three scored alike, the human scored 11 the same, 7 higher, and 2 lower.
  • Part of the gap reflects how demanding the task is: each report must be checked against every annotated defect, and the line between a partly and a fully correct diagnosis is fine. Because the rubric is defined per defect, the human applied it defect by defect; an earlier holistic pass over whole reports, on a different set of reports, was even more lenient.
  • Absolute ReflectionScores are thus stricter than a human's and should be read with caution, but the Reflection ranking is consistent across Judges (Kendall τ = 0.91 under a non-Qwen Judge, with an identical top three).

Part 2. The Controller gain under human labels, and the "points below Oracle" claims

B is the U1.5-Preview baseline; C replaces B's Controller with Qwen3.8-27B.

EvaluatorController gain C − BB Final − Init
Qwen3.8 Judge (paper; same 20 tasks)17.0−0.5
Human-verified labels (20 tasks)14.3 [7.7, 20.9]−1.1 [−4.3, 1.8]
Claude / GPT as evaluators (50 tasks)15.2 / 11.9−0.3 / +1.6
GPT-6.1 as Judge (198 / 200 tasks)15.3 [12.8, 17.8]−1.3 [−2.6, 0.0]

Under every evaluator, replacing the Controller improves the delivered image by 12–17 points, while the UMM's own self-refinement leaves it essentially unchanged (B's Final − Init stays within about ±1.6 points of zero). The paper's Judge gives a somewhat larger gain than the human-verified labels (17.0 vs. 14.3; difference 2.7 [−2.2, 7.5]), but the gain stays large under every evaluator.

You also asked whether these "points below Oracle" figures can be trusted. Under human-verified labels, B's delivered image is still clearly below Oracle-BoN, by 8.1 points [2.9, 13.6], so the direction holds; the Judge's 13.4, however, overstates the gap. This is expected, because Oracle-BoN is picked by the same Judge that scores it, which inflates its score (a winner's curse). We will therefore present these gaps as Judge-based upper estimates; our answer to Q3 quantifies this further.

To keep this reply easy to read, we do not repeat our other reliability checks here: our responses to Reviewer SKH9 (W1) and Reviewer scYB (W3) cover the full re-scoring of all outputs by a non-Qwen Judge, including a breakdown by model family, and repeated scoring by the same Judge. You are welcome to take a look if they help with your concern.

AuthorsOpenReview ↗

Response to Reviewer 4WcX (2/4): W2 & Q2

W2 & Q2: which checkpoint plays each role.

Thank you for pointing this out; we will make it explicit in the paper. In the default Refine-E2E setting (Table 2), all three roles, Controller, Generator, and Editor, are the same UMM checkpoint, prompted for each role. The three roles correspond to the abilities self-refinement needs (diagnosing and deciding, generating, editing), so the default setting tests the UMM itself. The same split also lets us swap one role at a time and see which ability limits the loop; external models appear only in this component-replacement study (Sec. 3.4).

For the replacements, we chose models that are clearly stronger in one role and can be dropped in without changing the protocol:

  • Controller → Qwen3.8-27B, the strongest Reflection model among the evaluated VLMs (46.5, vs. 3.75 for U1.5-Preview).
  • Generator / Editor → the released SenseNova-U1.5-8B-MoT, from the same family as the U1.5-Preview baseline. It is much stronger at generation and editing (initial images 6.2 points higher in Table 2; SignedEditScore +28.0 vs. −14.1) but similar in Decision, so swapping it in mainly changes those two abilities.
SettingControllerGeneratorEditor
Default, each of 6 UMMs (Table 2)same UMMsame UMMsame UMM
B (baseline of Sec. 3.4)U1.5-PreviewU1.5-PreviewU1.5-Preview
CQwen3.8-27BU1.5-PreviewU1.5-Preview
GU1.5-PreviewSenseNova-U1.5U1.5-Preview
EU1.5-PreviewU1.5-PreviewSenseNova-U1.5
CGQwen3.8-27BSenseNova-U1.5U1.5-Preview
CEQwen3.8-27BU1.5-PreviewSenseNova-U1.5
GEU1.5-PreviewSenseNova-U1.5SenseNova-U1.5
CGEQwen3.8-27BSenseNova-U1.5SenseNova-U1.5

We agree that the orchestration matters. We will phrase the negative result as "these UMMs, used as their own Controller, Generator, and Editor under this protocol, do not reliably improve their images" rather than "unified models cannot refine", and state that effects of the scaffolding cannot be fully excluded. Two observations limit how much the scaffolding alone can explain: the same scaffolding with a stronger Controller gains about 17 points (C vs. B), and the UMMs already struggle on the stand-alone diagnosis tasks (Reflection, Decision), which do not involve the loop.

AuthorsOpenReview ↗

Response to Reviewer 4WcX (3/4): Q3

Q3: budget N and the delivery rule.

Thank you for these two questions. Both probe how robust our main comparison is, so we answer them in turn.

Part 1. Results at N = 2 and N = 8

We re-ran U1.5, B (U1.5-Preview), and C at N = 2 and N = 8 on the first 50 tasks; the N = 4 results are from the paper. Each comparison is paired within a single run, which reduces the influence of drift in the Judge or hardware. At N = 8, we let C's Controller see up to 8 images instead of 4.

N = 2N = 4 (paper)N = 8
Final − Init
U1.5+4.4 [1.9, 7.3]+4.2 [0.5, 8.3]+7.1 [3.1, 11.0]
B−2.1 [−4.6, 0.4]+0.6 [−2.4, 3.7]+4.8 [1.7, 8.3]
C+12.1 [8.5, 15.9]+12.4 [7.1, 17.7]+21.5 [16.3, 26.8]
Final − Oracle-BoN (same budget)
U1.5−2.0 [−4.2, 0.4]−5.9 [−9.7, −2.2]−9.3 [−13.4, −5.0]
B−8.3 [−11.3, −5.5]−7.7 [−11.5, −4.0]−11.7 [−14.8, −8.8]
C+6.1 [2.7, 9.6]+5.1 [0.8, 10.0]+6.5 [1.6, 11.5]

Self-refinement does gain more with a larger budget. B's Final − Init, close to zero at N = 4, becomes clearly positive at N = 8 (+4.8 [1.7, 8.3]), and U1.5 and C gain more as well. What stays the same is the comparison with budget-matched Best-of-N: the two UMMs never beat Oracle-BoN at any budget, while the replaced Controller stays above it, with C's Final rising from 70.5 to 73.8 to 80.2. We would not read too much into how wide the gap gets at N = 8, since Oracle-BoN takes the best Judge score over more candidates and its selection bias grows with N (see Part 2).

Part 2. Delivering the "max Judge score so far"

This rule is exactly what the "Best" column of Table 2 reports. With the paper's Judge, Best roughly matches Oracle-BoN for U1 and U1.5 (+0.5 and +0.7, both CIs include 0) and falls below it for the other four UMMs (−2.7 to −5.6); only the configurations with the replaced Controller exceed Oracle-BoN. The catch is that Best and Oracle-BoN are both chosen by the same Judge that scores them, so their scores run high. Two checks show this:

  • Human-verified labels (B, 20 tasks): Best − Final = 3.7 [1.2, 6.5] (Judge: 5.6); Oracle-BoN − Final = 8.1 [2.9, 13.6] (Judge: 13.4).
  • Scoring the paper's choices with GPT-6.1: when the BoN image picked by the paper's Judge is scored by GPT-6.1 instead, Final − BoN is −0.9 [−2.7, 1.0] for U1.5, −5.9 [−7.7, −4.0] for B, and +9.7 [7.4, 12.1] for C, i.e., on par for U1.5, below for B, and above for C.

Delivering the best-so-far state would thus recover part of what the model loses through its own STOP choice (about 4 points for B under human labels). It is, however, no longer the model's own delivery: the selection relies on the evaluation Judge, which sees the full task rubrics that the model never sees, so we treat it as a privileged upper bound. In the paper, we will present Best and Oracle-BoN as Judge-selected upper estimates and report these checks alongside them. For the models, tasks, and budgets we tested, the qualitative conclusion is unchanged: self-refinement does not beat budget-matched sampling unless the Controller is replaced.

If helpful, our response to Reviewer scYB (W5) discusses the inference cost behind these budget-matched comparisons.

Reviewer_SKH9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark for measuring the self-refinement capabilities for image generation tasks in unified multimodal models (UMMs). This benchmark consists of generation, reflection, decision, and editing stages to evaluate both the end-to-end pipeline and to analyze intermediate failure modes. It shows that while native self-refinement in UMMs improves over the initial generation, the improvement is inconsistent and lags behind best-of-N generation. It also shows that some VLMs demonstrate better image understanding ability, and swapping them in for the reflection/decision stages greatly improves results.

Critical strengths

This paper is easy to follow, well-motivated, and well-executed. I find the overall pipeline sound, as it allows sufficient flexibility for the model to edit, roll back, or terminate, while keeping it structured such that the individual components (initial generation, reflection, decision, and editing) can be evaluated. The findings that iterative improvement does not help as much compared to a best-of-N approach are also interesting, and the score breakdown of each component helps future research in an actionable direction.

Critical weaknesses

The model-based evaluation uses Qwen3.8-Flash-Next. This might potentially lead to bias in evaluation. The Qwen series of models might be trained on a similar data corpus and have the same failure modes -- for example, errors not spotted by the "reflection" stage by a Qwen 3.8 might also be missed by Qwen3.8-Flash-Next in the evaluation stage. This could bias the evaluation to favor Qwen models, which are indeed performing the best in the current setup. Having some human annotations to establish correlation or testing on a different family of open-sourced models could mitigate this.

Questions for authors

Different image generation/editing models have different prompt distributions -- how are these aligned?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SKH9 (1/2): W1

We thank the reviewer for finding the paper easy to follow, well-motivated, and well-executed, and the overall pipeline sound. We address the weakness and the question below.

W1: potential bias of the Qwen-based Judge.

The concern is well founded, since Qwen models lead several of our results and could share blind spots with a Qwen Judge. A family bias is hard to rule out completely, as any second Judge has preferences of its own, so we checked from two independent sides: human verification and a full re-scoring by a non-Qwen Judge. We found at most a small family-dependent shift, and the main findings, including the Qwen lead in Reflection and the Controller gain, hold with a non-Qwen Judge.

(1) Under human verification, the Qwen Judge shows no detectable preference for Qwen models.

We sampled outputs from the paper's runs (194 E2E images, 50 editing pairs, and 75 Reflection reports). Besides the Judge, two state-of-the-art models from other families, Claude Opus 5.5 Extra-High and GPT-6 Astra Ultra, labeled every rubric item, and an experienced human researcher, blind to which model produced each label, resolved every disagreement (for E2E, on 20 of the 50 tasks).

  • Where the three models disagree, the human sides with the Qwen Judge at a rate between those of the two non-Qwen models (E2E: 58% vs. 65% for Claude and 47% for GPT; editing: 53% vs. 59% and 49%).
  • The Judge does not significantly inflate the gain of the Qwen-based Controller: relative to the human-verified labels, it adds 2.7 points [−2.2, 7.5] to the 14.3-point gain of C, whose Controller is Qwen3.8-27B.
  • For the scenario you describe, a Qwen model's blind spot shared by the Qwen Judge: on reports from Qwen3.8 and Gemma-4 for the same images, the Judge over-scores both by about the same amount relative to Claude and GPT (differences −0.005 [−0.09, 0.07] and −0.011 [−0.11, 0.08], on a 0–1 scale).

(2) Re-scoring everything with a non-Qwen Judge keeps the rankings largely intact, including the Qwen lead.

We replayed the exact recorded Judge inputs (prompt, images, response schema) with GPT-6.1 as Judge on 4,629 E2E images and all 10,897 Reflection reports. We chose GPT-6.1 because it handles the long multi-image rubric prompts reliably; none of the evaluated systems is GPT-based.

Qwen3.8 Judge (paper)GPT-6.1 Judge
Ranking of 6 UMMs / 13 configurations (Kendall τ)—0.87 / 0.95
Ranking of 24 Reflection models (Kendall τ)—0.91
Reflection top 3Qwen3.8-27B, Qwen3.6-35B-A3B, Qwen3.5-9Bidentical
Controller gain C − B17.0 [14.5, 19.8]15.3 [12.8, 17.8]

GPT-6.1 is stricter overall (−7.8 points on E2E) but agrees closely with the Qwen Judge per image (Spearman 0.90). It lowers Reflection scores for all models, mostly in proportion to the original score; beyond that, reports from Qwen-family and Qwen-based models drop by about 2 more points (out of 100) than those from Gemma, GLM, and Mistral. This difference is small, leaves the Reflection top three unchanged, and cannot by itself tell a Qwen-Judge preference apart from a GPT preference for a different report style; we will report it.

Qwen-based generators do not stand out either: BAGEL (built on Qwen2.5-7B) and OmniGen2 (built on Qwen2.5-VL-3B) rank at the bottom of Refine-E2E.

We will add these results, with the per-family breakdown, to the appendix.

In case it is useful, our response to Reviewer scYB (W3) covers further reliability checks: how closely the Judge agrees with the human-verified labels on each rubric item, how the main conclusions hold up under each alternative evaluator, and how stable repeated scoring by the same Judge is.

AuthorsOpenReview ↗

Response to Reviewer SKH9 (2/2): Q1

Q1: alignment of prompts across models.

Thank you for this question. How the prompts of different models are aligned does matter for a fair comparison. We align them by giving every model the same task text, unchanged; only the chat template, image placeholders, and sampling parameters follow each model's official inference code (Tables C2–C4). We chose this on purpose: self-refinement asks whether a model can understand and repair a request by itself, and rewriting prompts for each model would mix external prompt optimization into the result.

This choice is unlikely to bias our main conclusions, because they compare protocols within the same model (Final vs. Init, self-refinement vs. Oracle-BoN), and both sides of each comparison receive the same task text. It could affect comparisons across models, so we tested that on the initial generation.

(1) With each model's official input format, the ranking keeps its main structure; only BAGEL changes significantly.

The official inference pipelines of all six UMMs take the raw user prompt (Ovis-U1 and OmniGen2 add their own fixed prefix or system prompt internally). We regenerated the initial images for the first 50 E2E tasks without our benchmark wrapper, i.e., in each model's official input format, and scored both versions with the same Judge:

ModelBenchmark templateOfficial (raw) promptDifferenceSeed-only shift†
U156.553.8−2.7 [−6.9, +1.4]3.6
U1.564.762.2−2.5 [−6.9, +2.0]0.9
U1.5-Preview61.257.3−3.9 [−8.0, +0.3]1.3
Ovis-U122.824.2+1.5 [−1.0, +3.9]2.0
BAGEL19.123.8+4.7 [+2.2, +7.2]2.3
OmniGen223.421.3−2.0 [−5.0, +1.3]2.5

†Largest change of the 50-task mean when only the seed changes (benchmark template kept; Best-of-N candidates 1–3 vs. candidate 0), shown for scale. U1 has 49 scored tasks. The official-prompt images were generated on our current hardware, so the differences may also include hardware effects.

The top three and the roughly 30-point gap between the two groups stay the same, and the only rank swaps are among the three lowest models, which lie within 3 points of each other (Kendall τ = 0.73). Individual scores do move by a few points: BAGEL gains 4.7 without our wrapper, the only significant change, while the three SenseNova models lose 2.5–3.9, a consistent direction even though their 95% intervals include zero.

(2) Editor input limits affect only two editors, by about 1 point.

Editors with a 512-token text limit (Table C4) could truncate long requests. Counting the instruction text with each editor's own tokenizer over the 200 editing requests (template tokens excluded, so these are lower bounds), only FLUX.1-Kontext and FLUX.2-dev are affected (5.5% of requests partly truncated); all other editors are at 0.5% or below. Removing the affected tasks changes their scores by about 1 point (−2.55 → −1.61 and −18.89 → −17.83).

We will report these differences in the appendix and note that a single prompt protocol cannot guarantee that every model sees prompts close to its own training distribution.

We hope this addresses your question and the concern about Judge bias, and we would be glad to discuss further.

Thank you for the response. I think these results strengthen the paper. I have no further questions and tend towards acceptance.

Thank you again for your positive assessment of our work. We added a few new results near the end of the discussion, summarized in the updated General Response, and you are welcome to take a look if you are interested.

Reviewer_scYBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VisualRefineBench, a benchmark comprising 800 tasks that cover end-to-end image self-refinement with three diagnostic tasks: Reflection, Decision, and Editing. The authors evaluate six unified multimodal models and find that their self-refinement capabilities are limited that these models often fail to select the highest-quality image from the generation process. Component-replacement experiments further show that a more capable controller can significantly improve the final image quality.

Critical strengths
  1. This paper has a good motivation which combines end-to-end evaluation with three distinct diagnostic tasks, enabling a systematic analysis of model deficiencies in many areas.
  2. The paper highlights the difficulty existing models face in reliably selecting the optimal result which is notable.
  3. Experiments involving the replacement of individual components analyze the impact of various modules on final quality. The significant performance gains observed when replacing the Controller provide valuable empirical evidence for improving self-optimization systems.
Critical weaknesses
  1. The end-to-end experiments evaluate only six unified multimodal models, omitting other relevant models such as Emu3.5, Lumina-DiMOO, and UniWorld-V1. Furthermore, more recent models like FLUX.2-klein 4B/9B models were not included in the evaluation.
  2. The evaluation design lacks novelty. The fundamental concepts behind Reflection, Decision, and Editing overlap with existing literature. The approach of considering both editing success rate and content preservation has already been explored in [1][2][3]. The paper needs to articulate more clearly its unique contributions relative to existing evaluation methodologies.
  3. The reliability of the automated evaluation has not been sufficiently validated. The paper relies primarily on model-based scorers and lacks a systematic human evaluation.
  4. The analysis of the component-replacement experiments is inadequate. Replacing the Controller may alters multiple factors which makes it impossible to isolate the specific source of any performance improvement.
  5. The experiments only match the number of image operations performed, but do not control for total inference overhead, which may undermine the fairness of the comparison.

[1] Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing. CVPR 2025.

[2] Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing. ICML 2025.

[3] GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing. arXiv 2025.

Questions for authors

Refer to Critical Weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer scYB (1/5): W1

We thank the reviewer for the careful review and for recognizing the motivation of combining end-to-end evaluation with diagnostic tasks, the difficulty models face in reliably selecting the best result, and the value of the Controller-replacement experiments. We address each weakness below.

W1: end-to-end coverage.

We have added Emu3.5, Lumina-DiMOO, and UniWorld-V1 to Refine-E2E under exactly the paper's protocol: the frozen benchmark, all 200 tasks, and budget N = 4, with each model on its official inference backend behind our adapter and the same Judge model deployed with the paper's command (re-scoring the paper's saved outputs with this deployment closely reproduces the original scores).

ModelInitFinalBestOracle-BoNFinal−InitBest−BoNFinal−BoN
UniWorld-V118.019.023.726.7+1.0 [−0.3, +2.3]−3.0 [−4.1, −2.0]−7.6 [−8.9, −6.5]
Lumina-DiMOO*22.022.022.029.20.0−7.2 [−8.3, −6.1]−7.2 [−8.3, −6.1]
Emu3.547.144.950.656.8−2.2 [−3.7, −0.7]−6.2 [−7.7, −4.9]−11.9 [−13.6, −10.2]
Six UMMs in Table 2 (range, recomputed)21–6423–6925–7528–75−0.1 to +4.7−5.6 to +0.7−10.3 to −5.7

*Lumina-DiMOO almost never produces a well-formed Controller decision: already at the first decision, which sees a single image, its output starts a JSON object but degenerates into repeated keys and copied prompt text, so its trajectories end there and deliver the initial image. Shortening its fixed decoding length from 512 to 256 or 384 tokens does not help (on 30 tasks, 1 of 60 outputs parses, and it copies the prompt).

Differences are paired over 198–199 tasks (189–199 for the six original UMMs); 95% bootstrap CIs.

All three models are consistent with Table 2. None shows a clear gain in delivered quality (UniWorld-V1: +1.0, CI includes 0; Emu3.5: −2.2; Lumina-DiMOO cannot issue a valid refinement action), their Judge-selected best states stay below budget-matched Best-of-N, and they deliver 7–12 points less than Oracle-BoN, in line with the six original UMMs. Across all nine UMMs, only three show clear delivered gains and all nine deliver less than Oracle-BoN; we will add these rows to Table 2.

FLUX.2-klein 4B/9B are image generation and editing models: they produce images but no text, so they cannot diagnose defects or choose actions to run self-refinement on their own. They are, however, already evaluated in the paper as editors (Editing task, App. Table C15, 18 models):

  • FLUX.2-klein-9B: SignedEditScore −10.3 [−18.9, −2.0] (4th of 18);
  • FLUX.2-klein-4B: SignedEditScore −16.1 [−24.1, −8.0] (7th of 18).

Both rank in the upper half of the 18 editors, yet, like most editors, their repairs on average do not outweigh the content they damage.

AuthorsOpenReview ↗

Response to Reviewer scYB (2/5): W2

W2: novelty of the evaluation design.

We agree that weighing modification against preservation is an established principle, and our Editing score follows it. However, [1]–[3] score a single edit: AugCLIP [1] and BPM [2] are metrics for one edited image (CLIP-based and region-aware, respectively), and GIE-Bench [3] is a single-turn benchmark that leaves multi-step refinement to future work. VisualRefineBench instead asks whether a UMM that inspects and revises its own images delivers a better final image, and where the process breaks. This lets us answer three questions that a single-edit evaluation cannot, with the end-to-end task and the controlled diagnostics providing complementary evidence. The table gives an overview; details follow.

[1] AugCLIP[2] BPM[3] GIE-BenchOursWhat we find
Scores one edit for modification vs. preservation✓✓✓✓ (Editing)Same principle as [1]–[3]
Separates "no better image" from "better image not delivered"———✓U1/U1.5 reach +11 along the way but deliver only +4–5
Tests whether diagnostic scores carry over to end-to-end gains———✓Strongest-Reflection Controller: +17.0 delivered
Compares self-refinement with resampling at a matched image-operation budget———✓All 6 UMMs 5.6–10.3 below Oracle-BoN; replaced Controller above it

(— = not evaluated in that paper.)

(1) Does the model fail to produce a better image, or fail to deliver it?

Comparing the delivered image (Final) with the best image in the trajectory (Best) separates the two. SenseNova-U1 and U1.5 reach images more than 11 points above their initial ones but deliver gains of only 4–5 points (U1.5-Preview: +5.6 vs. +0.1), so part of the failure lies in choosing what to deliver.

(2) Do diagnostic scores translate into end-to-end gains?

Yes, to different degrees. As Controller, Qwen3.8-27B, far stronger than U1.5-Preview on Reflection (46.5 vs. 3.75), raises Final by 17.0 points, mostly through better candidates; an Editor that repairs more and damages less in fixed-input Editing adds 4.5; a stronger Generator's 6.1-point head start shrinks to 3.3. No model's Decision balanced accuracy is reliably above a constant-action reference, pointing to action choice as a remaining bottleneck.

(3) What does self-refinement add over spending the same budget on resampling?

Oracle-BoN spends the same four image operations on independent samples and keeps the best by Judge score. All six UMMs deliver 5.6–10.3 points less, so their self-refinement currently adds less than resampling with good selection; with the stronger Controller, all four configurations exceed it by 4.9–11.7 points.

We will add this discussion to the related-work section.

  • [1] Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing. CVPR 2025.
  • [2] Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing. ICML 2025.
  • [3] GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing. arXiv 2025.
AuthorsOpenReview ↗

Response to Reviewer scYB (3/5): W3

W3: reliability of the automated evaluation.

We agree. The submission checked the Judge only indirectly (cross-run consistency, App. C.6.5), and our Limitations acknowledge that this cannot replace human validation of the Judge. We therefore tested three questions.

Questions and setup.

(1) How accurate is the Judge on individual rubric items?

  • Data. All items are sampled from the paper's recorded outputs: 194 E2E images from 50 random tasks (initial, delivered, and Judge-selected images), 50 editing pairs, and 75 Reflection reports.
  • Model labeling. The Judge (Qwen3.8-Flash-Next) and two state-of-the-art models from other families, Claude Opus 5.5 Extra-High and GPT-6 Astra Ultra, each labeled every rubric item with the Judge instructions used in the paper.
  • Human verification. Hand-labeling every item is costly (about 16 rubric items per E2E image, over 4,000 images), so we focused human effort where labels are uncertain. An experienced human researcher resolved every label on which the three models disagreed (on 20 of the 50 E2E tasks) and spot-checked a random sample of unanimous labels; for Reflection, the researcher also checked 60 random reports from all 24 models defect by defect. The researcher was blind to which model produced each label and to each image's condition. The resulting human-verified labels take the human's verdict wherever the human checked a label, and the models' consensus elsewhere.

(2) Do the main conclusions change if the Judge is replaced?

  • We recomputed them with the human-verified labels, with Claude and GPT as evaluators on the same sample, and with GPT-6.1 as Judge on the full set (the exact recorded Judge inputs for 4,629 E2E images and all 10,897 Reflection reports).

(3) Does the same Judge fluctuate when re-scoring?

  • The same Judge re-scored the same 100 E2E images and 100 Reflection reports three times.

Results.

(1) The Judge is close to frontier models and human judgment on E2E and editing, less so on Reflection.

  • Agreement among models. The Judge agrees with Claude and GPT at rubric-level κ = 0.74 and 0.70 on E2E and 0.71 for both on editing, close to the Claude–GPT agreement (κ = 0.80).
  • Human check. Where the models disagree, the human sides with the Judge at a rate between those of Claude and GPT (E2E: 58% vs. 65% / 47%; editing: 53% vs. 59% / 49%). Spot checks confirmed most unanimous labels (E2E 27/33, editing 14/14).
TaskJudge vs. human-verified labels
E2E (20 tasks, 1,168 rubric labels)κ = 0.79; image-score Spearman 0.91
Editing (50 pairs)κ = 0.82; SignedEditScore Spearman 0.80
Reflection (60 random reports)65% exact [52, 77]; human mean 0.32 vs. 0.17

On E2E, the Judge scores images 6.3 points higher than the human-verified labels on average, while ordering them consistently. On Reflection, the three models agree well with one another (weighted κ = 0.71–0.84), but the human is more lenient, scoring higher than the Judge on 30% of the random reports (weighted to the full set) and lower on 5%. Reflection rankings are nonetheless consistent across Judges.

(2) The main conclusions do not change under a replaced Judge.

B is the U1.5-Preview baseline; C replaces B's Controller with Qwen3.8-27B.

EvaluatorController gain C − BB Final − Init
Qwen3.8 Judge (paper; same 20 tasks)17.0−0.5
Human-verified labels (20 tasks)14.3 [7.7, 20.9]−1.1 [−4.3, 1.8]
Claude / GPT as evaluators (50 tasks)15.2 / 11.9−0.3 / +1.6
GPT-6.1 as Judge (198 / 200 tasks)15.3 [12.8, 17.8]−1.3 [−2.6, 0.0]

Under every evaluator, replacing the Controller improves the delivered image by 12–17 points, while the UMM's own self-refinement leaves it essentially unchanged. GPT-6.1 is stricter (−7.8 points on E2E) but agrees closely per image (Spearman 0.90). Its rankings closely match the paper's (Kendall τ = 0.87 over 6 UMMs, 0.95 over 13 configurations, and 0.91 over 24 Reflection models), and whether Final − Init changes significantly agrees in 12 of 13 configurations (the exception is BAGEL's small gain, +1.4, which becomes +0.8 [−0.2, 1.9]). Only Judge-selected upper bounds shrink: under human-verified labels, Oracle-BoN exceeds B's delivered image by 8.1 [2.9, 13.6] points rather than 13.4; we will revise those statements.

(3) Under repeated scoring, mean scores are stable, while individual images still vary.

Across three rounds, mean E2E scores shift by only −0.7 to −1.5 points relative to the paper's scores, but individual images vary by about 4 points; both match the cross-run variation in App. C.6.5. Reflection scores match the paper's in 92–93% of reports (91–94% between rounds).

We will add all of this to a new appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful and constructive reviews. We summarize the main outcomes of the discussion below. The new results are in the revised paper (in blue, mostly Appendix E) and in each reviewer's thread.

Where things stand. After the rebuttal, SKH9 and 4WcX both said they lean toward acceptance. scYB's follow-up focuses on two remaining concerns: the added value of the diagnostics and the cost of configuration C.

What the reviewers agree on. All three see value in combining end-to-end refinement with diagnostic tasks. scYB says it enables "a systematic analysis of model deficiencies," SKH9 calls the pipeline "sound," and 4WcX notes that the diagnostics "are not afterthoughts." The Controller result was described as "a concrete finding, not just a leaderboard" (4WcX) and as "valuable empirical evidence for improving self-optimization systems" (scYB). In the second round, scYB also agreed that reporting both the best image reached and the delivered image is useful, and appreciated that the revision now limits C's advantage over Oracle-BoN to matched image operations.

Additional evidence provided during the rebuttal.

ConcernRaised byNew evidence
Reliability of the automatic JudgescYB, SKH9, 4WcXHuman verification on 50 editing pairs (κ = 0.82), 20 E2E tasks (κ = 0.79), and 60 Reflection reports. A non-Qwen Judge (GPT-6.1 Sol) keeps the rankings (Kendall τ = 0.87–0.95).
Is the ~17-point Controller gain Judge-specific?4WcX, SKH912–17 points under every evaluator tested, including 14.3 [7.7, 20.9] under human-verified labels.
Bias toward Qwen modelsSKH9Under GPT-6.1 Sol, the Reflection top three are unchanged.
End-to-end coveragescYBEmu3.5, Lumina-DiMOO, and UniWorld-V1 added; none shows a clear delivered gain.
Budget N, roles, prompts4WcX, SKH9At N = 2, 4, and 8, the tested UMMs never beat budget-matched Oracle-BoN. With each model's official input format, the ranking keeps its main structure.

Responses to the remaining concerns.

  • Added value of the diagnostics. We ran three more Controllers. Across all five, those with higher Reflection scores tend to produce larger candidate gains (Spearman ρ = 0.8; Appendix E.6). We do not claim that Reflection alone explains the Controller gain, since repair plans and action choices may also play a part, and the paper says so.
  • Cost of C. Appendix E.5 compares per-task time with Controller calls and selection included. Our latest W5 reply adds Best-of-N with a deployable selector: C at N = 2 scores higher than it at N = 2 in comparable time, and higher than it at N = 8 in less time; we will add this to Appendix E.5. This question concerns only C; the main negative result does not depend on it.

What we revised. Judge-selected quantities tend to run high. Under human-verified labels, Oracle-BoN beats the baseline's delivered image by 8.1 points, not 13.4. The paper therefore treats Best and Oracle-BoN as Judge-based upper estimates, and limits the negative result to these UMMs, each acting as its own Controller, Generator, and Editor under our protocol.

Overall, we believe the evidence supports the paper's main message: the tested native UMMs gain little from refining their own images and fall below Judge-selected Oracle-BoN at matched image-operation budgets, while among the component replacements we tested, replacing the Controller gives the largest quality gain. We thank the reviewers and the AC for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.