- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors build DistractEdit, a semi-synthetic dataset that was originated from web images and used Gemini and Nano Banana to edit the photos with added scene or similar-looking subjects. The authors executed 8 model editing methods on InternVL-3.5-8B and Blip2-OPT-2.7B showing that the effectiveness of the editing methods degrades with a held-out clean set of photos. They then propose SACO, a dual agent pipeline that regenerates the editing image as a clean and subject-centered picture before the edit, which the results show that it restores most of the lost image generality but hurts locality.
Critical strengths
- The motivation is relevant and real for the community, since real world editing images are rarely clean, and studies rarely vary the conditions of the editing images.
- The evaluation covers eight editors from different families, which is quite comprehensive, though only on two models.
- The breakdown of the evaluation scores makes the failure pattern informative. Specifically, reliability stays high while I-Gen drops and I-Loc rises.
- Full prompts and annotation criteria are included for with the paper along with cost and latency.
Critical weaknesses
- One of the biggest weakness I see is that the central finding of the paper that motivates the method after it is somewhat expected. In general, clutter and irrelevant backgrounds would make vision-driven methods harder to generalize, including editing. Also it is well known that knowledge editing can overfit the edit examples, which increases the level of expectation for the finding. Furthermore, the resulting metrics of the paper which showing high reliability and I-Gen collapse supports that expected overfitting behavior. ALso the paper adds no analysis that would make the finding non-obvious such as which image rehion the edit binds to or how far the edited representation moves from that of clean images.
- As a proposed solution motivated by the finding, the SACO method that the authors proposed is circular with respect to the way DistractEdit is generated. Specifically, the benchmark defines the problem as a cluttered editing image evaluation against a clean test image by injecting irrelevant things or similar confounders, but the proposed SACO method is practically reversing that benchmark data by editing them back into a clean and subject centered image. And this is exactly rewarded by I-Gen so that improvement is guaranteed by construction. Supporting evidence include the table 7, which shows that SACO does not help on Original inputs and it just recovers what the benchmark took away. Moreover, table 3 shows that cropping matches full SACO for MEND, WISE, and GRACE. Cost of the agentic approach is also a big limitation: the gemini models editing images would be extra operational cost.
- The paper states that degradation across scenerio “can be attributed to the visual distractions”, but then the attribution to distraction is not rigorously established. Specifically, there are multiple confounding factors that changes with the distraction: a) in terms of similarity between the editing and test images, the held-out test image is always clean, so it resembles the Original editing image more than the distracted ones. This is problematic because any change that moves the editing image away from the test image would lower I-Gen. b) The subject size is a major factor that is overlooked. The author's own words is "the subject remains small" when explaining SACO-DR, and the fact that cropping works well further suggest that the size need to be decoupled from the effects. c) There was no rigorous or quantitative subject fidelity check, with the only thing being annotators judging. The curation prompt also allows changes to pose, viewpoint and occlusion so the fidelity of the subject needs to be verified or controlled.
- There was no variance of confidence intervals of significance tests being reported. Given how small the differences are in some tables, this needs to be quantified.
- The benchmarking was only done on two models despite a number of KE methods.
Questions for authors
- Since editing methods are already known to overfit the edit examples, what other analysis may the authors perform to demonstrate the significance and novelty of the finding, and what are their results?
- How does SACO compare to a) a crop guided by cheap object detection on the entity name and b) an image generated from the entity name alone without the input image? And how does SACO perform on the real-photo sets where the clutter was not created by Nano Banana?
- If I-Gen is also measured on cluttered held-out images (e.g., by applying your curation pipeline to the test images), does the drop persist when the editing and test conditions match?
- What quantitative evidence shows the subject is preserved across conditions? For example: similarity on the subject region, face identity for celebrities, or OCR for logos.
- Can you add significance tests and confidence intervals to the reported numbers?
7 comments
SACO method that the authors proposed is circular, and the improvement is guaranteed by construction. How does SACO compare to a) a crop guided by cheap object detection and b) an image generated from the entity name alone without the input image? (c) And how does SACO perform on the real-photo sets (Continued)
We also add the experiment on the performance of SACO on the real-photo sets. We mix the two visual scenarios together to further demonstrate that SACO does not depend on the type of visual distraction, since the scenario of a real-world image is unknown in practice.
| Method | Metric | Raw Visual | Refined Visual (SACO) |
|---|---|---|---|
| MEND | Rel. | 0.90 | 0.94 |
| T-Gen. | 0.88 | 0.92 | |
| I-Gen. | 0.86 | 0.90 | |
| T-Loc. | 0.81 | 0.79 | |
| I-Loc. | 0.82 | 0.76 | |
| Avg. | 0.86 | 0.86 | |
| WISE | Rel. | 0.93 | 1.00 |
| T-Gen. | 0.83 | 0.90 | |
| I-Gen. | 0.65 | 0.85 | |
| T-Loc. | 0.96 | 1.00 | |
| I-Loc. | 0.96 | 0.82 | |
| Avg. | 0.87 | 0.91 | |
| GRACE | Rel. | 0.99 | 0.99 |
| T-Gen. | 0.85 | 0.84 | |
| I-Gen. | 0.52 | 0.63 | |
| T-Loc. | 0.93 | 0.91 | |
| I-Loc. | 0.93 | 0.92 | |
| Avg. | 0.84 | 0.86 | |
| BalancEdit | Rel. | 1.00 | 1.00 |
| T-Gen. | 0.99 | 0.97 | |
| I-Gen. | 0.54 | 0.75 | |
| T-Loc. | 0.90 | 0.94 | |
| I-Loc. | 0.99 | 0.92 | |
| Avg. | 0.89 | 0.92 |
From the above results, SACO improves the I-Gen. of all four editing methods and maintains or improves their average performance. Similar to the results on DistractEdit, the gains in I-Gen. come with a decrease in I-Loc., which indicates that the MMKE under visual distractions remains a challenge.
(a) The held-out test image is always clean, so it resembles the Original editing image more than the distracted ones. (b) The subject size is a major factor that is overlooked. (c) There was no rigorous or quantitative subject fidelity check
(a) Using clean images for I-Gen. is a deliberate design choice. If the test images also contained distractions, it would be unclear whether a failure comes from the distractions in the editing image or from other factors, such as the difficulty of retrieving the knowledge from a distracted test image. We agree that the distractions make the editing image less similar to the test image, which is a different real-world image of the subject. We also follow the reviewer’s suggestion and measure I-Gen. on cluttered hold-out images, and will provide the details in a follow-up response.
(b) We agree that the subject size changes with the distractions. However, visual distractions and subject size cannot always be separated. For example, when multiple competing subjects are presented, the target stands out less in the scene even if its pixel size is unchanged. Therefore, we view this an inherent challenge in the real world, and our curation framework provides a balance between the realness of images and controllability.
(c) Our curation allows limited changes in pose, viewpoint, and occlusion to keep the images natural, since forcing the subject to stay unchanged while the scene changes often yields unnatural images. We also follow the reviewer’s suggestion and measure the subject fidelity quantitatively, and will provide the details in a follow-up response.
Does the drop persist when the editing and test conditions match?
We add the following experiment as the reviewer suggested. We apply our curation pipeline to the test images under the Multi-subj scenario. To ensure that the generated images are not exactly the same as editing images, we provide the editing image as a reference and instruct the pipeline to design different scenes. We then measure the editing performance on these images under InternVL.
Complex Scene, InternVL-3.5 8B
| Method | Clean test image | Clutter test image |
|---|---|---|
| FT-L | 0.99 | 0.99 |
| IKE | 0.49 | 0.49 |
| MEND | 0.79 | 0.78 |
| LoRA | 0.63 | 0.65 |
| WISE | 0.71 | 0.76 |
| GRACE | 0.53 | 0.35 |
| BalancEdit | 0.82 | 0.89 |
Complex Scene, Blip2-OPT 2.7B
| Method | Clean test image | Clutter test image |
|---|---|---|
| FT-L | 0.51 | 0.54 |
| IKE | 1.00 | 1.00 |
| MEND | 0.95 | 0.96 |
| LoRA | 0.94 | 0.93 |
| WISE | 0.56 | 0.64 |
| GRACE | 0.79 | 0.71 |
| BalancEdit | 0.31 | 0.34 |
Multi-Subj, InternVL-3.5 8B
| Method | Clean test image | Clutter test image |
|---|---|---|
| FT-L | 0.93 | 0.97 |
| IKE | 0.49 | 0.49 |
| MEND | 0.72 | 0.79 |
| LoRA | 0.51 | 0.59 |
| WISE | 0.47 | 0.54 |
| GRACE | 0.46 | 0.35 |
| BalancEdit | 0.63 | 0.69 |
Multi-Subj, Blip2-OPT 2.7B
| Method | Clean test image | Clutter test image |
|---|---|---|
| FT-L | 0.45 | 0.46 |
| IKE | 0.99 | 0.99 |
| MEND | 0.93 | 0.96 |
| LoRA | 0.90 | 0.95 |
| WISE | 0.49 | 0.53 |
| GRACE | 0.71 | 0.70 |
| BalancEdit | 0.29 | 0.31 |
Does the drop persist when the editing and test conditions match? (Continued)
When the editing and test images are both under Multi-Subj, I-Gen. recovers only a small part of the drop for most methods compared with the clean test images, and it even decreases for GRACE. As a result, I-Gen. remains clearly below that of the edits made on the Original images for most methods, i.e., the drop persists when the editing and test conditions match. This indicates that the drop cannot be explained by the dissimilarity between the editing and test images alone, which is consistent with Sec. 3.3, where the edits made under visual distractions rely on the scene or on the distractions rather than on the subject.
What quantitative evidence shows the subject is preserved across conditions?
We thank the reviewer for the suggestion. We have added a quantitative analysis of subject fidelity in App. A.5 of the revised draft (Tab. 10; revisions in blue), using two automatic measures that are separate from the curation. First, a Gemini judge, which is given no information about the curation framework, compares the Original and the distracted image and judges whether they contain the same subject. Second, we compute the CLIP similarity on the subject region. We localize the subject with Gemini, segment it with SAM, and fill the remaining pixels with white. Unlike face identity or OCR, CLIP applies uniformly to all categories, including graphic logos without text and anime characters. As a reference, we compute the same similarity between the Original editing image and the I-Gen. image of each subject. The judge considers about 95% of the image pairs to contain the same subject under both scenarios, and the CLIP similarity approaches the reference, where the remaining gap partly reflects the allowed changes in lighting, pose, and viewpoint. These results show that the subject is largely preserved across scenarios.
There was no variance of confidence intervals of significance tests being reported.
We thank the reviewer for the suggestion. We have added the 95% confidence intervals of the main benchmark results in Tab. 1 to App. A.8 of the revised draft (Tab. 14; revisions in blue), and we regard two scores as significantly different when their confidence intervals do not overlap. Most of the declines in I-Gen. are significant, while the text-side metrics remain relatively more stable than I-Gen, and Multi-Subj yields a lower I-Gen. than Complex Scene for almost all methods. Our findings still hold when the confidence intervals are taken into account.
The benchmarking was only done on two models.
We agree that evaluating more VLMs would further strengthen the benchmark. We choose two backbones that differ substantially from each other: Blip2-OPT uses a Q-Former with a fixed number of visual tokens, while InternVL-3.5 is a more recent VLM with a different architecture and a larger language model. Adding a new backbone requires adapting and tuning all MMKE methods, which we unfortunately could not complete within the rebuttal period.
We thank the reviewer for the suggestion. We go beyond the standard MMKE metrics and study more detailed failure patterns of the edited models in a new Sec. 3.3 (details in App. A.7; revisions in blue). We analyze the failure patterns from two perspectives and summarize the analysis and the findings below.
First, we analyze which image region the edited knowledge binds to. We mask either the target subject or the visual distractions by using attention masks, and test whether the remaining part alone can still make the edited model recall the edit. In the follwing table, DS and SS denote the accuracy of recalling the edit when only the distractions or only the subject remain unmasked.
We find that:
Then, we analyze why this binding breaks image generalization. We use LoRA as a case study and examine whether the edit changes where the model looks in a new image, and whether a new image can still activate the edited knowledge. In the following table, the SAR change is the relative change of the attention on the subject after editing, and AR measures how strongly the new (I-Gen.) image activates the edit compared with the editing image.
We find that:
This suggests that the failure lies in how the knowledge is written into the model rather than in how the model reads the new image.
We also agree with the reviewer that some of our findings are consistent with intuition. However, by performing the same edit under different visual distractions while preserving the subject, DistractEdit provides a controlled testbed to quantify and study how visual distractions affect MMKE. Moreover, while existing MMKE studies mainly evaluate whether an edit succeeds or analyze the visual inputs to design better editors, our analysis examines the edited model itself, i.e., how the edited knowledge relies on the visual input and why it fails to generalize. We hope that our benchmark and these new findings can motivate future work in this direction.