- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies visual abstention in unified multimodal models: recognizing when an image-editing request has no valid solution under its constraints and declining without generating an image. The authors introduce Draw-or-Decline (DoD), a benchmark containing 1,050 feasible–infeasible pairs across seven categories. Evaluations of eight existing models show that editing competence does not guarantee spontaneous refusal, while a simple reminder often increases both correct and false refusals. The proposed VisTA approach fine-tunes UniREdit-BAGEL using paired supervision for successful editing and explained abstention. Without an inference-time reminder, refusal success increases from 0.4% to 93.0%, with a false refusal rate of 0.8%, while editing success increases from 68.4% to 74.3%.
Critical strengths
-
The paper identifies a meaningful limitation of generative models. Producing a plausible image is insufficient when the original request cannot be satisfied. Evaluating editing success, refusal success, and false refusals together provides a useful framework for assessing whether a model can decide when generation is appropriate. Tables 1–3 clearly illustrate why either editing accuracy or refusal frequency alone would be inadequate.
-
The benchmark includes carefully controlled examples and verification. The six scene-based categories pair different instructions with the same image, while maze pairs retain the same instruction and change a single corridor cell. The latter design is particularly valuable because feasibility is verified by breadth-first search and cannot be inferred from instruction wording alone. Human review and the reported agreement between automated and human judgments further strengthen the evaluation.
-
The main improvement in refusal behavior is substantial. VisTA achieves a large increase in refusal success while keeping false refusals low. Its refusal success is at least 88.0% in every category without a reminder, suggesting that the improvement is not confined to a single task. The results provide convincing evidence that targeted training can substantially improve abstention on DoD.
Critical weaknesses
-
My main concern is that the aggregate editing improvement is primarily focused on maze-solving. Maze editing success increases from 51.3% to 88.0%, contributing approximately 5.24 percentage points to the overall improvement of about 5.9 points. Calculated from the reported category scores, the mean over the other six categories increases only from approximately 71.2% to 72.0%, with decreases in three categories. The improvement over the feasible-only control, 74.3% versus 72.0%, also comes from single training runs.
-
Generalization beyond the benchmark’s construction process remains uncertain. The six “real-world” categories use synthetic images, and training and evaluation employ related categories and instruction-construction procedures. The deduplication checks and two category-holdout experiments are valuable, but they do not establish robustness to natural photos.
-
The primary refusal metric only partially captures the formal definition of visual abstention. The formal definition requires both an explicit refusal and no generated image. However, Section 4.1 credits textual refusals even when an image follows, and the refusal verdict can receive credit despite an incorrect explanation.
Questions for authors
-
Could you provide a consolidated comparison of textual refusal, refusal without image generation, explanation correctness, and their joint success rate across the evaluated methods?
-
Could you evaluate natural photographs and independently authored instructions containing unseen types of infeasibility, together with closely matched feasible requests? For example, can the model reliably distinguish requests to modify an existing object from requests that creating it.
-
You have convincingly demonstrated that the model has learned to refuse. I would also like to verify whether the improvement in editing scores stems primarily from the specialized maze training. After accounting for differences in training setup and random fluctuations, does training on infeasible samples also help the model better execute edits that are actually feasible? For instance, by conducting evaluations on other types of image editing tasks.
4 comments
Reply to Reviewer oQJS (Part 2/2)
W3: The primary refusal metric only partially captures the formal definition of visual abstention. ... the refusal verdict can receive credit despite an incorrect explanation.
Q1: Could you provide a consolidated comparison of textual refusal, refusal without image generation, explanation correctness, and their joint success rate across the evaluated methods?
R3: We agree that the textual rule is weaker than the formal definition, and we will report all four measures in the revision. We adopted the textual rule because most UMMs always generate an image after their reasoning, even when the reasoning refuses (Sec. 4.1, Appx. B.2). The tables below cover the 1,050 infeasible DoD requests. Textual refusal is the D in the paper. Refusal without image additionally requires that no image is generated, as in Eq. (4). Correct explanation means the stated reason matches the verified cause. Joint success requires both refusal without image and a correct explanation.
Without a reminder:
| Method | Textual refusal | Refusal without image | Correct explanation | Joint success |
|---|---|---|---|---|
| BAGEL | 0.0% | 0.0% | 0.0% | 0.0% |
| Uni-CoT | 0.0% | 0.0% | 0.0% | 0.0% |
| BAGEL-Canvas | 0.0% | 0.0% | 0.0% | 0.0% |
| ThinkMorph | 1.7% | 0.1% | 1.5% | 0.0% |
| UniREdit-BAGEL | 0.4% | 0.0% | 0.4% | 0.0% |
| UniReason | 0.1% | 0.0% | 0.1% | 0.0% |
| InternVL-U | 0.2% | 0.0% | 0.2% | 0.0% |
| SenseNova-U1.0 | 0.5% | 0.0% | 0.5% | 0.0% |
| VisTA-BAGEL | 93.0% | 93.0% | 91.8% | 91.8% |
With the reminder:
| Method | Textual refusal | Refusal without image | Correct explanation | Joint success |
|---|---|---|---|---|
| BAGEL | 55.0% | 0.0% | 32.9% | 0.0% |
| Uni-CoT | 40.4% | 0.0% | 21.8% | 0.0% |
| BAGEL-Canvas | 16.0% | 0.0% | 1.2% | 0.0% |
| ThinkMorph | 57.5% | 0.4% | 45.4% | 0.3% |
| UniREdit-BAGEL | 5.2% | 0.0% | 4.9% | 0.0% |
| UniReason | 7.0% | 0.0% | 6.2% | 0.0% |
| InternVL-U | 28.1% | 0.0% | 15.1% | 0.0% |
| SenseNova-U1.0 | 16.9% | 0.0% | 13.6% | 0.0% |
| VisTA-BAGEL | 93.4% | 93.4% | 93.0% | 93.0% |
VisTA-BAGEL never generates an image after refusing, so its scores are the same under the textual and strict rules, and almost all of its refusals give the correct reason. The 8 UMMs behave differently. The reminder makes them state infeasibility more often, but they then generate an image anyway, and many of these refusals are not grounded in the image: for BAGEL, only about 60% give the correct reason.
We hope these responses address your concerns. Thank you again for your constructive feedback, and we would be happy to answer any further questions.
The authors have largely addressed my concerns. I will maintain my score and lean toward acceptance.
Thank you for your response and for leaning toward acceptance.
Reply to Reviewer oQJS (Part 1/2)
Thank you for the detailed review and for finding the refusal improvement convincing. We address each point below.
R1: Thank you for this careful analysis. We agree that the editing gain of VisTA-BAGEL on DoD comes mainly from maze. As shown in Appx. C.7, this gain comes from the maze training pairs: when maze is held out of training, maze editing stays close to the initialization. We therefore see the main benefit of VisTA as adding refusal while preserving editing ability, rather than improving editing itself.
To account for random fluctuation, we trained VisTA-BAGEL and the feasible-only control (Appx. C.5) with two additional seeds. The table reports the mean and standard deviation over three runs, without a reminder.
VisTA-BAGEL edits slightly better than the feasible-only control, but the gap (1.4% on both the six scene categories and overall) is within one standard deviation. Across three runs, we do not find clear evidence that infeasible-example training improves feasible editing. The main observed benefit is substantially stronger refusal.
We also evaluated on PhyEditBench [1], whose inputs are frames of real-world videos and whose feasible requests were written by its authors, independently of DoD. Of its 238 real-world instances, we use the 224 for which human annotators could write a valid infeasible counterpart on the same input image, giving 224 feasible-infeasible pairs.
On this out-of-domain benchmark, editing likewise shows no gain. We therefore see the main benefit of VisTA as adding refusal while preserving editing ability, rather than improving editing itself.
[1] PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing
R2: On PhyEditBench (see above), refusal carries over to real photographs and to infeasible requests written by human annotators: VisTA-BAGEL refuses most infeasible requests, while the baselines almost never refuse, and false refusals stay low.
To check whether the model separates creating an object from modifying an absent one, we classified each feasible DoD request in the six scene categories by whether it introduces an object not yet in the image. VisTA-BAGEL falsely refuses very few requests of either kind (0.5% and 0.4%), so on DoD it does not treat a newly requested object as a missing referent. On PhyEditBench, some of its false refusals come from requests that bring a new object into the scene. For “Drop a film strip from the top right so that it falls past the white pillow and lands on the bed surface”, the model reasons that “there is no film strip at the top right or elsewhere that could be set in motion”. Such creation requests are phrased as acting on the new object, and the model sometimes reads that object as an existing referent.