- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VisualRefineBench looks at whether UMMs really produce better images when they self-refine, given a hard cap on how many image calls they can make with N=4. It runs Refine-E2E and compares initial, best intermediate and final outputs against the Oracle Best-of-N upper bound alongside three fixed-input diagnostic tests, and we evaluated six different UMMs. Only a handful see modest improvements, and none hit their Oracle-BoN ceiling. Quite often, the best intermediate output beats the final one by a wide margin. Swapping in a more capable controller boosts final scores by roughly 17 points across four generator-editor setups, but picking the best result at the end still struggles.
Critical strengths
Diagnostics are not an afterthought. Reflection, decision-making, and editing each pull apart separate stages that get mixed together in an end-to-end pipeline. SignedEditScore works well for editing tasks because it directly penalizes harm to content that was already correct; tracking only the net change in quality would miss this kind of failure entirely.
The gap between the best intermediate result and the final output, paired with our controller swap experiments, explains why self-refinement falls short of Oracle-BoN. The models can generate stronger candidate images yet still end up with weaker final outputs. The real bottleneck lies in planning and selecting outputs, not the editing module itself. This is a concrete takeaway rather than just another leaderboard observation.
Critical weaknesses
Our main quantitative results rely on an automated judge. For this benchmark paper, we need human validation of ReflectionScore and SignedEditScore on a held-out subset, plus verification that the roughly 17-point gain from swapping the controller does not depend on this specific judge. Without these checks, we cannot firmly trust absolute claims about how far scores fall below the Oracle bound.
There is also a subtle tension when we frame the work as UMM self-refinement while using a separate orchestrated stack of controller, generator and editor. We should state plainly for every end-to-end setup whether these three roles run from one single checkpoint controlled by prompts, or from separate external models. If we skip this clarification, readers may misinterpret our negative results to mean unified world models are inherently bad at refinement, when the real limitation could come from how we structured the whole system.
Questions for authors
- Human–Judge agreement on Reflection and SignedEditScore (n≈50–100)? Does Controller replacement still win under human labels?
- Default Refine-E2E: same UMM for all roles, or mixed specialists?
- Do Final−Init conclusions flip at N=2 or N=8? What if delivery is “max Judge score so far” instead of the model’s STOP choice?
- Related work gap: I think some paper like Unicorn and SRUM are related, would you mind to discuss them in your related part?
6 comments
Response to Reviewer 4WcX (2/4): W2 & Q2
W2 & Q2: which checkpoint plays each role.
Thank you for pointing this out; we will make it explicit in the paper. In the default Refine-E2E setting (Table 2), all three roles, Controller, Generator, and Editor, are the same UMM checkpoint, prompted for each role. The three roles correspond to the abilities self-refinement needs (diagnosing and deciding, generating, editing), so the default setting tests the UMM itself. The same split also lets us swap one role at a time and see which ability limits the loop; external models appear only in this component-replacement study (Sec. 3.4).
For the replacements, we chose models that are clearly stronger in one role and can be dropped in without changing the protocol:
- Controller → Qwen3.8-27B, the strongest Reflection model among the evaluated VLMs (46.5, vs. 3.75 for U1.5-Preview).
- Generator / Editor → the released SenseNova-U1.5-8B-MoT, from the same family as the U1.5-Preview baseline. It is much stronger at generation and editing (initial images 6.2 points higher in Table 2; SignedEditScore +28.0 vs. −14.1) but similar in Decision, so swapping it in mainly changes those two abilities.
| Setting | Controller | Generator | Editor |
|---|---|---|---|
| Default, each of 6 UMMs (Table 2) | same UMM | same UMM | same UMM |
| B (baseline of Sec. 3.4) | U1.5-Preview | U1.5-Preview | U1.5-Preview |
| C | Qwen3.8-27B | U1.5-Preview | U1.5-Preview |
| G | U1.5-Preview | SenseNova-U1.5 | U1.5-Preview |
| E | U1.5-Preview | U1.5-Preview | SenseNova-U1.5 |
| CG | Qwen3.8-27B | SenseNova-U1.5 | U1.5-Preview |
| CE | Qwen3.8-27B | U1.5-Preview | SenseNova-U1.5 |
| GE | U1.5-Preview | SenseNova-U1.5 | SenseNova-U1.5 |
| CGE | Qwen3.8-27B | SenseNova-U1.5 | SenseNova-U1.5 |
We agree that the orchestration matters. We will phrase the negative result as "these UMMs, used as their own Controller, Generator, and Editor under this protocol, do not reliably improve their images" rather than "unified models cannot refine", and state that effects of the scaffolding cannot be fully excluded. Two observations limit how much the scaffolding alone can explain: the same scaffolding with a stronger Controller gains about 17 points (C vs. B), and the UMMs already struggle on the stand-alone diagnosis tasks (Reflection, Decision), which do not involve the loop.
Response to Reviewer 4WcX (3/4): Q3
Q3: budget N and the delivery rule.
Thank you for these two questions. Both probe how robust our main comparison is, so we answer them in turn.
Part 1. Results at N = 2 and N = 8
We re-ran U1.5, B (U1.5-Preview), and C at N = 2 and N = 8 on the first 50 tasks; the N = 4 results are from the paper. Each comparison is paired within a single run, which reduces the influence of drift in the Judge or hardware. At N = 8, we let C's Controller see up to 8 images instead of 4.
| N = 2 | N = 4 (paper) | N = 8 | |
|---|---|---|---|
| Final − Init | |||
| U1.5 | +4.4 [1.9, 7.3] | +4.2 [0.5, 8.3] | +7.1 [3.1, 11.0] |
| B | −2.1 [−4.6, 0.4] | +0.6 [−2.4, 3.7] | +4.8 [1.7, 8.3] |
| C | +12.1 [8.5, 15.9] | +12.4 [7.1, 17.7] | +21.5 [16.3, 26.8] |
| Final − Oracle-BoN (same budget) | |||
| U1.5 | −2.0 [−4.2, 0.4] | −5.9 [−9.7, −2.2] | −9.3 [−13.4, −5.0] |
| B | −8.3 [−11.3, −5.5] | −7.7 [−11.5, −4.0] | −11.7 [−14.8, −8.8] |
| C | +6.1 [2.7, 9.6] | +5.1 [0.8, 10.0] | +6.5 [1.6, 11.5] |
Self-refinement does gain more with a larger budget. B's Final − Init, close to zero at N = 4, becomes clearly positive at N = 8 (+4.8 [1.7, 8.3]), and U1.5 and C gain more as well. What stays the same is the comparison with budget-matched Best-of-N: the two UMMs never beat Oracle-BoN at any budget, while the replaced Controller stays above it, with C's Final rising from 70.5 to 73.8 to 80.2. We would not read too much into how wide the gap gets at N = 8, since Oracle-BoN takes the best Judge score over more candidates and its selection bias grows with N (see Part 2).
Part 2. Delivering the "max Judge score so far"
This rule is exactly what the "Best" column of Table 2 reports. With the paper's Judge, Best roughly matches Oracle-BoN for U1 and U1.5 (+0.5 and +0.7, both CIs include 0) and falls below it for the other four UMMs (−2.7 to −5.6); only the configurations with the replaced Controller exceed Oracle-BoN. The catch is that Best and Oracle-BoN are both chosen by the same Judge that scores them, so their scores run high. Two checks show this:
- Human-verified labels (B, 20 tasks): Best − Final = 3.7 [1.2, 6.5] (Judge: 5.6); Oracle-BoN − Final = 8.1 [2.9, 13.6] (Judge: 13.4).
- Scoring the paper's choices with GPT-6.1: when the BoN image picked by the paper's Judge is scored by GPT-6.1 instead, Final − BoN is −0.9 [−2.7, 1.0] for U1.5, −5.9 [−7.7, −4.0] for B, and +9.7 [7.4, 12.1] for C, i.e., on par for U1.5, below for B, and above for C.
Delivering the best-so-far state would thus recover part of what the model loses through its own STOP choice (about 4 points for B under human labels). It is, however, no longer the model's own delivery: the selection relies on the evaluation Judge, which sees the full task rubrics that the model never sees, so we treat it as a privileged upper bound. In the paper, we will present Best and Oracle-BoN as Judge-selected upper estimates and report these checks alongside them. For the models, tasks, and budgets we tested, the qualitative conclusion is unchanged: self-refinement does not beat budget-matched sampling unless the Controller is replaced.
If helpful, our response to Reviewer scYB (W5) discusses the inference cost behind these budget-matched comparisons.
Response to Reviewer 4WcX (1/4): W1 & Q1
We thank the reviewer for the careful and constructive review, and for noting that the diagnostics are not afterthoughts and that the Best−Final gap together with the Controller replacement is a concrete finding.
W1 & Q1: human agreement and whether the Controller gain is Judge-specific.
We fully agree that a benchmark built on an automatic Judge should be validated against human judgments. We answer in two parts.
Part 1. Human agreement on ReflectionScore and SignedEditScore
Following your suggestion, we sampled 50 editing pairs and 75 Reflection reports from the paper's runs, and additionally 194 E2E images from 50 random tasks. The Judge and two state-of-the-art models from other families, Claude Opus 5.5 Extra-High and GPT-6 Astra Ultra, labeled every rubric item with the Judge's instructions. An experienced human researcher, who did not know which model produced each label or which condition an image came from, resolved every disagreement (for E2E, on 20 of the 50 tasks) and spot-checked a random sample of unanimous labels (the human's verdict is used wherever given). For ReflectionScore, the researcher also checked 60 reports drawn at random from all 24 Reflection models, defect by defect, blind to the Judge's score.
Editing and E2E. The Judge agrees closely with the human-verified labels. On E2E it scores images 6.3 points higher on average while ordering them consistently.
Reflection. Agreement is more limited.
Part 2. The Controller gain under human labels, and the "points below Oracle" claims
B is the U1.5-Preview baseline; C replaces B's Controller with Qwen3.8-27B.
Under every evaluator, replacing the Controller improves the delivered image by 12–17 points, while the UMM's own self-refinement leaves it essentially unchanged (B's Final − Init stays within about ±1.6 points of zero). The paper's Judge gives a somewhat larger gain than the human-verified labels (17.0 vs. 14.3; difference 2.7 [−2.2, 7.5]), but the gain stays large under every evaluator.
You also asked whether these "points below Oracle" figures can be trusted. Under human-verified labels, B's delivered image is still clearly below Oracle-BoN, by 8.1 points [2.9, 13.6], so the direction holds; the Judge's 13.4, however, overstates the gap. This is expected, because Oracle-BoN is picked by the same Judge that scores it, which inflates its score (a winner's curse). We will therefore present these gaps as Judge-based upper estimates; our answer to Q3 quantifies this further.
To keep this reply easy to read, we do not repeat our other reliability checks here: our responses to Reviewer SKH9 (W1) and Reviewer scYB (W3) cover the full re-scoring of all outputs by a non-Qwen Judge, including a breakdown by model family, and repeated scoring by the same Judge. You are welcome to take a look if they help with your concern.