acceptodds
Under review as a conference paper at ICLR 2027

Visuals Matter: Benchmarking and Enhancing Multi-Modal Knowledge Editing under Visual Distractions

Abstract

Multimodal knowledge editing (MMKE) has emerged as a critical approach for updating outdated knowledge in Vision-Language Models (VLMs). For MMKE, the knowledge must be grounded in visual cues that identify the target subject, so that the knowledge can transfer to any image of that subject while unrelated knowledge is preserved. However, real-world images often contain irrelevant visual elements, which we term as visual distractions. Existing MMKE benchmarks do not systematically examine their effects, leaving an open question: how do visual distractions affect MMKE? We introduce DistractEdit, a controlled, semi-synthetic benchmark that evaluates the same knowledge updates across increasingly complex visual scenarios: Isolated Entity, Contextual Background, and Multi-Subject Interference. By preserving the target subject’s visual details under different scenarios, DistractEdit enables controlled comparisons that isolate the effect of distractions on editing performance. Our evaluation across MMKE methods and VLMs reveals that visual distractions undermine editing performance, particularly the generalization of edited knowledge to unseen images, and the degradation degree varies across methods, models, and distraction types. We further propose SACO, an agent-based method that refines visual inputs by mitigating distractions before knowledge editing, following a See, Analyze, Clean, and Optimize strategy. Our results demonstrate that mitigating visual distractions improves the performance, highlighting visual refinement as a promising direction. We present SACO as a starting point and anticipate that DistractEdit and SACO will facilitate future research on MMKE under visual distractions.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Model, Knowledge Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ucKaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors build DistractEdit, a semi-synthetic dataset that was originated from web images and used Gemini and Nano Banana to edit the photos with added scene or similar-looking subjects. The authors executed 8 model editing methods on InternVL-3.5-8B and Blip2-OPT-2.7B showing that the effectiveness of the editing methods degrades with a held-out clean set of photos. They then propose SACO, a dual agent pipeline that regenerates the editing image as a clean and subject-centered picture before the edit, which the results show that it restores most of the lost image generality but hurts locality.

Critical strengths
  1. The motivation is relevant and real for the community, since real world editing images are rarely clean, and studies rarely vary the conditions of the editing images.
  2. The evaluation covers eight editors from different families, which is quite comprehensive, though only on two models.
  3. The breakdown of the evaluation scores makes the failure pattern informative. Specifically, reliability stays high while I-Gen drops and I-Loc rises.
  4. Full prompts and annotation criteria are included for with the paper along with cost and latency.
Critical weaknesses
  1. One of the biggest weakness I see is that the central finding of the paper that motivates the method after it is somewhat expected. In general, clutter and irrelevant backgrounds would make vision-driven methods harder to generalize, including editing. Also it is well known that knowledge editing can overfit the edit examples, which increases the level of expectation for the finding. Furthermore, the resulting metrics of the paper which showing high reliability and I-Gen collapse supports that expected overfitting behavior. ALso the paper adds no analysis that would make the finding non-obvious such as which image rehion the edit binds to or how far the edited representation moves from that of clean images.
  2. As a proposed solution motivated by the finding, the SACO method that the authors proposed is circular with respect to the way DistractEdit is generated. Specifically, the benchmark defines the problem as a cluttered editing image evaluation against a clean test image by injecting irrelevant things or similar confounders, but the proposed SACO method is practically reversing that benchmark data by editing them back into a clean and subject centered image. And this is exactly rewarded by I-Gen so that improvement is guaranteed by construction. Supporting evidence include the table 7, which shows that SACO does not help on Original inputs and it just recovers what the benchmark took away. Moreover, table 3 shows that cropping matches full SACO for MEND, WISE, and GRACE. Cost of the agentic approach is also a big limitation: the gemini models editing images would be extra operational cost.
  3. The paper states that degradation across scenerio “can be attributed to the visual distractions”, but then the attribution to distraction is not rigorously established. Specifically, there are multiple confounding factors that changes with the distraction: a) in terms of similarity between the editing and test images, the held-out test image is always clean, so it resembles the Original editing image more than the distracted ones. This is problematic because any change that moves the editing image away from the test image would lower I-Gen. b) The subject size is a major factor that is overlooked. The author's own words is "the subject remains small" when explaining SACO-DR, and the fact that cropping works well further suggest that the size need to be decoupled from the effects. c) There was no rigorous or quantitative subject fidelity check, with the only thing being annotators judging. The curation prompt also allows changes to pose, viewpoint and occlusion so the fidelity of the subject needs to be verified or controlled.
  4. There was no variance of confidence intervals of significance tests being reported. Given how small the differences are in some tables, this needs to be quantified.
  5. The benchmarking was only done on two models despite a number of KE methods.
Questions for authors
  1. Since editing methods are already known to overfit the edit examples, what other analysis may the authors perform to demonstrate the significance and novelty of the finding, and what are their results?
  2. How does SACO compare to a) a crop guided by cheap object detection on the entity name and b) an image generated from the entity name alone without the input image? And how does SACO perform on the real-photo sets where the clutter was not created by Nano Banana?
  3. If I-Gen is also measured on cluttered held-out images (e.g., by applying your curation pipeline to the test images), does the drop persist when the editing and test conditions match?
  4. What quantitative evidence shows the subject is preserved across conditions? For example: similarity on the subject region, face identity for celebrities, or OCR for logos.
  5. Can you add significance tests and confidence intervals to the reported numbers?
7 comments

Central finding is somewhat expected. No analysis that would make the finding non-obvious. Attribution to distraction is not rigorously established. what other analysis may the authors perform to demonstrate the significance and novelty of the finding?

We thank the reviewer for the suggestion. We go beyond the standard MMKE metrics and study more detailed failure patterns of the edited models in a new Sec. 3.3 (details in App. A.7; revisions in blue). We analyze the failure patterns from two perspectives and summarize the analysis and the findings below.

First, we analyze which image region the edited knowledge binds to. We mask either the target subject or the visual distractions by using attention masks, and test whether the remaining part alone can still make the edited model recall the edit. In the follwing table, DS and SS denote the accuracy of recalling the edit when only the distractions or only the subject remain unmasked.

EditsMethodRel.DSSS
Complex Scene (failed)WISE0.970.760.70
LoRA1.000.730.69
BalancEdit1.000.840.78
Multi-Subj (failed)WISE1.000.920.61
LoRA1.000.890.52
BalancEdit1.000.980.71
Multi-Subj (successful)WISE1.000.690.94
LoRA1.000.720.90
BalancEdit1.000.721.00

We find that:

  • Background visual elements under Complex Scene lead the edit to rely on the whole scene. Neither the subject nor the distractions alone can recall the edit.
  • Subject-level ambiguity makes the edit bind the knowledge to the visual distractions. Under Multi-Subj, the distractions alone can still recall the edit, while the subject alone recalls it much less.
  • The edits that do generalize are grounded in the subject. For these edits, the subject alone recalls the edit, whereas the distractions alone do not. This indicates that the failures are mainly induced by the distractions.

Then, we analyze why this binding breaks image generalization. We use LoRA as a case study and examine whether the edit changes where the model looks in a new image, and whether a new image can still activate the edited knowledge. In the following table, the SAR change is the relative change of the attention on the subject after editing, and AR measures how strongly the new (I-Gen.) image activates the edit compared with the editing image.

Edited withSAR change (editing image)SAR change (I-Gen. image)AR
Original+0.3%+1.8%0.90
Complex Scene+2.8%+0.7%0.63
Multi-Subj+6.4%+1.4%0.63

We find that:

  • Visual distraction barely changes where the model looks, but new images activate the edit less strongly. After editing, the attention on the subject in a new image stays almost unchanged. However, the new image activates the edit much less strongly when the edit is made under visual distractions.

This suggests that the failure lies in how the knowledge is written into the model rather than in how the model reads the new image.

We also agree with the reviewer that some of our findings are consistent with intuition. However, by performing the same edit under different visual distractions while preserving the subject, DistractEdit provides a controlled testbed to quantify and study how visual distractions affect MMKE. Moreover, while existing MMKE studies mainly evaluate whether an edit succeeds or analyze the visual inputs to design better editors, our analysis examines the edited model itself, i.e., how the edited knowledge relies on the visual input and why it fails to generalize. We hope that our benchmark and these new findings can motivate future work in this direction.

SACO method that the authors proposed is circular, and the improvement is guaranteed by construction. How does SACO compare to a) a crop guided by cheap object detection and b) an image generated from the entity name alone without the input image? (c) And how does SACO perform on the real-photo sets (Continued)

We also add the experiment on the performance of SACO on the real-photo sets. We mix the two visual scenarios together to further demonstrate that SACO does not depend on the type of visual distraction, since the scenario of a real-world image is unknown in practice.

MethodMetricRaw VisualRefined Visual (SACO)
MENDRel.0.900.94
T-Gen.0.880.92
I-Gen.0.860.90
T-Loc.0.810.79
I-Loc.0.820.76
Avg.0.860.86
WISERel.0.931.00
T-Gen.0.830.90
I-Gen.0.650.85
T-Loc.0.961.00
I-Loc.0.960.82
Avg.0.870.91
GRACERel.0.990.99
T-Gen.0.850.84
I-Gen.0.520.63
T-Loc.0.930.91
I-Loc.0.930.92
Avg.0.840.86
BalancEditRel.1.001.00
T-Gen.0.990.97
I-Gen.0.540.75
T-Loc.0.900.94
I-Loc.0.990.92
Avg.0.890.92

From the above results, SACO improves the I-Gen. of all four editing methods and maintains or improves their average performance. Similar to the results on DistractEdit, the gains in I-Gen. come with a decrease in I-Loc., which indicates that the MMKE under visual distractions remains a challenge.

(a) The held-out test image is always clean, so it resembles the Original editing image more than the distracted ones. (b) The subject size is a major factor that is overlooked. (c) There was no rigorous or quantitative subject fidelity check

(a) Using clean images for I-Gen. is a deliberate design choice. If the test images also contained distractions, it would be unclear whether a failure comes from the distractions in the editing image or from other factors, such as the difficulty of retrieving the knowledge from a distracted test image. We agree that the distractions make the editing image less similar to the test image, which is a different real-world image of the subject. We also follow the reviewer’s suggestion and measure I-Gen. on cluttered hold-out images, and will provide the details in a follow-up response. (b) We agree that the subject size changes with the distractions. However, visual distractions and subject size cannot always be separated. For example, when multiple competing subjects are presented, the target stands out less in the scene even if its pixel size is unchanged. Therefore, we view this an inherent challenge in the real world, and our curation framework provides a balance between the realness of images and controllability.
(c) Our curation allows limited changes in pose, viewpoint, and occlusion to keep the images natural, since forcing the subject to stay unchanged while the scene changes often yields unnatural images. We also follow the reviewer’s suggestion and measure the subject fidelity quantitatively, and will provide the details in a follow-up response.

Does the drop persist when the editing and test conditions match?

We add the following experiment as the reviewer suggested. We apply our curation pipeline to the test images under the Multi-subj scenario. To ensure that the generated images are not exactly the same as editing images, we provide the editing image as a reference and instruct the pipeline to design different scenes. We then measure the editing performance on these images under InternVL.

Complex Scene, InternVL-3.5 8B

MethodClean test imageClutter test image
FT-L0.990.99
IKE0.490.49
MEND0.790.78
LoRA0.630.65
WISE0.710.76
GRACE0.530.35
BalancEdit0.820.89

Complex Scene, Blip2-OPT 2.7B

MethodClean test imageClutter test image
FT-L0.510.54
IKE1.001.00
MEND0.950.96
LoRA0.940.93
WISE0.560.64
GRACE0.790.71
BalancEdit0.310.34

Multi-Subj, InternVL-3.5 8B

MethodClean test imageClutter test image
FT-L0.930.97
IKE0.490.49
MEND0.720.79
LoRA0.510.59
WISE0.470.54
GRACE0.460.35
BalancEdit0.630.69

Multi-Subj, Blip2-OPT 2.7B

MethodClean test imageClutter test image
FT-L0.450.46
IKE0.990.99
MEND0.930.96
LoRA0.900.95
WISE0.490.53
GRACE0.710.70
BalancEdit0.290.31

Does the drop persist when the editing and test conditions match? (Continued)

When the editing and test images are both under Multi-Subj, I-Gen. recovers only a small part of the drop for most methods compared with the clean test images, and it even decreases for GRACE. As a result, I-Gen. remains clearly below that of the edits made on the Original images for most methods, i.e., the drop persists when the editing and test conditions match. This indicates that the drop cannot be explained by the dissimilarity between the editing and test images alone, which is consistent with Sec. 3.3, where the edits made under visual distractions rely on the scene or on the distractions rather than on the subject.

What quantitative evidence shows the subject is preserved across conditions?

We thank the reviewer for the suggestion. We have added a quantitative analysis of subject fidelity in App. A.5 of the revised draft (Tab. 10; revisions in blue), using two automatic measures that are separate from the curation. First, a Gemini judge, which is given no information about the curation framework, compares the Original and the distracted image and judges whether they contain the same subject. Second, we compute the CLIP similarity on the subject region. We localize the subject with Gemini, segment it with SAM, and fill the remaining pixels with white. Unlike face identity or OCR, CLIP applies uniformly to all categories, including graphic logos without text and anime characters. As a reference, we compute the same similarity between the Original editing image and the I-Gen. image of each subject. The judge considers about 95% of the image pairs to contain the same subject under both scenarios, and the CLIP similarity approaches the reference, where the remaining gap partly reflects the allowed changes in lighting, pose, and viewpoint. These results show that the subject is largely preserved across scenarios.

There was no variance of confidence intervals of significance tests being reported.

We thank the reviewer for the suggestion. We have added the 95% confidence intervals of the main benchmark results in Tab. 1 to App. A.8 of the revised draft (Tab. 14; revisions in blue), and we regard two scores as significantly different when their confidence intervals do not overlap. Most of the declines in I-Gen. are significant, while the text-side metrics remain relatively more stable than I-Gen, and Multi-Subj yields a lower I-Gen. than Complex Scene for almost all methods. Our findings still hold when the confidence intervals are taken into account.

The benchmarking was only done on two models.

We agree that evaluating more VLMs would further strengthen the benchmark. We choose two backbones that differ substantially from each other: Blip2-OPT uses a Q-Former with a fixed number of visual tokens, while InternVL-3.5 is a more recent VLM with a different architecture and a larger language model. Adding a new backbone requires adapting and tuning all MMKE methods, which we unfortunately could not complete within the rebuttal period.

Reviewer_hCW4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how visual distractions affect multimodal knowledge editing. It introduces DistractEdit, a semi-synthetic benchmark with 174 knowledge seeds and three visual conditions: isolated entities, contextual backgrounds, and multi-subject interference. The benchmark evaluates the same knowledge updates across these conditions while keeping the held-out image for image generality fixed. Experiments with eight editing methods across two VLM backbones show that distractions can substantially reduce image generality despite high editing reliability. The paper also proposes SACO, an agent-based image refinement pipeline that identifies the target concept and removes distracting content before editing. Its evaluation includes component ablations, a cropping variant, and a small real-image consistency study.

Critical strengths
  1. The benchmark targets an important gap in multimodal knowledge editing. DistractEdit evaluates the same knowledge update under isolated-entity, contextual-background, and multi-subject conditions, while keeping the held-out image for evaluating image generality fixed. This paired design helps distinguish successful editing on the supplied image from successful transfer to another image of the same subject. The resulting evaluation perspective is valuable because conventional editing success metrics can conceal failures in visual grounding.

  2. The experiments reveal substantial and practically relevant failures of existing editors. The evaluation covers eight editing methods across two VLM backbones, with a smaller real-image consistency study providing additional support. For example, WISE’s image generality on InternVL decreases from 0.97 to 0.47 under multi-subject interference despite maintaining high editing reliability. SACO further demonstrates that input refinement can recover a substantial portion of this lost generality. These findings make the paper useful as both a diagnostic study and a starting point for improving editing robustness.

Critical weaknesses
  1. The central claim of isolating the effect of visual distractions is insufficiently validated. Generative image editing can change the target’s scale, texture, pose, or visibility alongside the intended background and distractor changes. The human checks establish recognizability and plausibility, but do not quantitatively establish preservation of the target’s visual features. Consequently, the observed degradation could partly reflect changes to the target itself rather than distractions alone. A foreground-preserving compositing control, together with measurements of target-region consistency, would substantially strengthen the causal interpretation. This matters particularly because controlled comparison is the benchmark’s central contribution.

  2. The additional value of the full SACO pipeline is not convincingly established. In Table 3, the cropping variant already matches or exceeds full SACO on WISE and GRACE and is close on MEND, although full SACO provides a substantial advantage on BalancEdit. Moreover, some generality gains come with large locality losses: for WISE on InternVL, image generality improves from 0.47 to 0.90 while image locality falls from 0.96 to 0.71. These results leave unclear when the more expensive generative pipeline is necessary and whether it improves the generality–locality trade-off rather than primarily expanding the scope of the edit. Comparisons at matched locality, alongside stronger simple preprocessing baselines, are needed to substantiate the method’s added value.

Questions for authors
  1. Validation of the benchmark’s controlled comparison: What evidence establishes that the target’s visual features remain sufficiently consistent across the three conditions, beyond human judgments of recognizability and plausibility? A target-region consistency analysis or an experiment that preserves the foreground pixels while changing the background would help determine whether the reported degradation can be attributed specifically to distractions.

  2. Clarification and validation of SACO’s benefits: After visual refinement, are locality metrics evaluated on exactly the same images and questions as in the raw-input setting? Given the competitive cropping results and the substantial locality losses for some editors, can the authors provide a comparison between full SACO and cropping at comparable locality levels? This would help establish whether generative refinement offers benefits beyond changing the generality–locality operating point.

4 comments

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. We address your comments and questions one by one.

The central claim of isolating the effect of visual distractions is insufficiently validated. Validation of the benchmark’s controlled comparison.

We thank the reviewer for the suggestion. Beyond the human checks, we have added a quantitative analysis of subject fidelity in App. A.5 of the revised draft (Tab. 10; revisions in blue). A Gemini judge, which is given no information about the curation framework, judges whether the Original and the distracted image contain the same subject, and we compute the CLIP similarity on the target region, where the subject is localized with Gemini and segmented with SAM. As a reference, we compute the same similarity between the Original editing image and the I-Gen. image, both of which belong to the Original scenario. The judge considers about 95% of the image pairs to contain the same subject, and the CLIP similarity approaches the reference, where the remaining gap partly reflects the limited changes in lighting, pose, and viewpoint that our curation allows. We allow these changes because preserving the foreground pixels while changing the scene often yields unnatural images, e.g., with inconsistent lighting and shadows. Moreover, our new analysis in Sec. 3.3 further supports that the degradation comes from the distractions. Under Multi-Subj, the distractions alone can still make the edited model recall the edits, which changes to the target itself cannot explain.

Clarification and validation of SACO’s benefits

First, the locality metrics are evaluated on exactly the same images and questions as in the raw-input setting, since SACO only replaces the image used for editing. SACO works by removing the visual distractions, so it refines the editing image to be closer to the Original scenario. As shown in Tab. 1, moving from clean images to images with visual distractions lowers I-Gen. and raises I-Loc. at the same time, since the edit becomes fragile and is rarely triggered by any image other than the editing image. With SACO, I-Gen. moves close to the level of the edits made on the clean Original images, while I-Loc. returns to about that level and T-Loc. remains largely unchanged. In other words, SACO restores the behavior of the MMKE under clean images, and the decrease in I-Loc. reflects the challenge of visual distractions revealed by DistractEdit rather than an issue of SACO itself. As for the unique benefits of the generative refinement, cropping can only keep a rectangular region, so it cannot remove the distractions that overlap with or lie next to the target. In contrast, generative refinement can remove the distractions anywhere in the image and enhance the target while keeping the image natural. We agree that cropping is already sufficient for most editors and is much cheaper. However, it is less robust on BalancEdit, cropping barely improves over the raw input, while full SACO improves I-Gen., T-Gen., and T-Loc. However, we agree with the reviewer and do think that cropping and generative refinement can be considered as complementary tools for the agent. Therefore, we present SACO as a first attempt and a starting point to mitigate the failures identified by DistractEdit, rather than as a complete solution, because the performance is far from perfect, and adaptively selecting tools is a promising direction.

Dear Reviewer,

Thank you once again for your commitment to reviewing our paper and assisting us in improving our work. We would like to remind you that the discussion window will close soon, and we eagerly await your feedback.

We have provided detailed explanations for each of your concerns. We would greatly appreciate it if you could review our responses and let us know if they fully or partially address your concerns. Any additional comments you may have would also be highly appreciated.

Best regards,

Authors

Thank authors for the response. They basically addressed my concerns, so I will maintain my positive score.

Dear Reviewer,

We are glad that our responses have addressed your concerns. As the discussion period is coming to an end, we would like to sincerely thank you for your time and effort. Your comments and insights have been greatly appreciated and have helped us improve the quality of our paper.

Best regards,

Authors

Reviewer_sHtBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how visual distractions affect multimodal knowledge editing (MMKE) in VLMs. It introduces DistractEdit, a benchmark with three scenarios—Isolated Entity, Contextual Background, and Multi-Subject Interference—that preserve target-subject details while varying distractions. Evaluations of eight editing methods on InternVL-3.5-8B and Blip2-OPT 2.7B show that distractions primarily impair image generality, while text generality and text locality remain relatively stable. Degradation patterns vary across methods, backbones, and distraction types.The paper also proposes SACO, an input-side visual refinement method combining knowledge perception and planning agents with an image editing tool to identify relevant cues, remove distractions, and enhance the target subject. Experiments and ablations demonstrate improved image generality across multiple editing methods.

Critical strengths
  1. A controlled visual-distraction benchmark with a clear causal motivation. DistractEdit constructs different visual scenarios for the same knowledge subject while attempting to preserve the subject’s visual details. This enables a relatively clear distinction between “editing is intrinsically difficult” and “editing degrades because of visual distractions.” This is more valuable for analysis than comparing performance across different real-world images alone.

  2. An input-side mitigation approach with practical value. SACO does not require redesigning or retraining each knowledge editor. Instead, it improves the inputs to existing editing methods through knowledge perception, distraction diagnosis, and image refinement. Its results across MEND, WISE, GRACE, and BalancEdit, together with the ablation studies and cost analysis, enhance its practical value.

Critical weaknesses
  1. The paper does not provide sufficiently detailed analysis of the performance of the different methods on the benchmark. The paper identifies some consistent patterns across these methods; for example, I-Gen. often decreases substantially, while I-Loc. sometimes increases instead. However, some metrics, such as the relatively widespread downward trend in T-Loc., lack further analysis. In addition, the different degradation patterns exhibited by different methods are not analyzed in sufficient depth.

  2. The paper does not sufficiently quantify whether SACO changes the key semantic details of the target subject, and lacks the necessary analysis and evidence regarding the stability of this method.

Questions for authors
  1. Did the three human annotators calculate any inter-annotator agreement metrics? Please report agreement statistics for each criterion, as well as the pass rates across different knowledge categories and visual scenarios.

  2. Were the images refined by SACO subjected to independent evaluations of subject preservation and semantic consistency? For example, are the target subject’s category, pose, texture, and key visual features preserved before and after editing?

  3. For the multiple editing methods mentioned in the experimental section, could the authors provide further introduction and analysis of their methodological characteristics? If possible, explaining the different degradation patterns by categorizing the methods would effectively improve the interpretability of this benchmark.

2 comments

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. We address your comments and questions one by one.

Report agreement statistics for each criterion, as well as the pass rates across different knowledge categories and visual scenarios.

We thank the reviewer for the suggestion. We have added the inter-annotator agreement and the pass rates to App. A.4 of the revised draft (Tab. 8 and Tab. 9; revisions in blue). We compute the agreement on all generated candidates before filtering, since the filtering itself removes candidates with large disagreements. The agreement is the highest for scenario compliance and the lowest for diversity. Overall, about 25% of the generated candidates pass the filtering. Multi-Subj has lower agreement and pass rates than Complex Scene across all criteria and categories, suggesting that it is more difficult to construct.

The paper does not sufficiently quantify whether SACO changes the key semantic details of the target subject. Were the images refined by SACO subjected to independent evaluations of subject preservation and semantic consistency?

Similar to our curation, we allow SACO to change visual factors other than the knowledge concept, such as the background, composition, and viewpoint. SACO is also designed to preserve the knowledge-relevant cues, where its planner explicitly preserves and strengthens the visual cues corresponding to the target knowledge, and the image editing tool only modifies the elements specified in the plan. To verify this quantitatively, we query the original unedited VLM with the same editing question on the clean image from the Original scenario and on the corresponding SACO-refined image from Multi-Subj, and measure the token-level agreement between the two answers. Since no model parameters are updated, consistent answers indicate that SACO preserves the identity of the subject without semantic drift.

MetricInternVLBlip
Acc.0.990.95

The agreement reaches 0.99 on InternVL-3.5 and 0.95 on Blip2-OPT, showing that the subject in the refined images is still recognized as the same entity.

The paper does not provide sufficiently detailed analysis of the performance of the different methods on the benchmark. For the multiple editing methods mentioned in the experimental section, could the authors provide further introduction and analysis of their methodological characteristics?

We group the methods by how the edit is stored and applied: weight-update methods (FT-L, LoRA, and MEND) directly modify the model parameters; memory-based methods (WISE, GRACE, and BalancEdit) store the edit in a separate memory that is activated only when the input matches the edit; IKE provides the new knowledge in context; and VisEdit edits the visual representations of the regions relevant to the prompt. The degradation patterns largely follow these groups. Memory-based methods show the largest I-Gen. drop and I-Loc. rise but the smallest change in T-Loc., likely because their activation is tied to the distracted editing input, so other images rarely trigger the edit. In contrast, IKE and VisEdit keep I-Gen. stable but show the largest drop in T-Loc., while weight-update methods lie in between, which strongly depends on the backbone. The backbone also matters for IKE, which performs much worse on InternVL-3.5 than on Blip2-OPT. We observe that, with simple in-context learning, the more recent InternVL-3.5 tends to reject the counterfactual knowledge given in context, as it considers the context to be incorrect. As for the drop in T-Loc., T-Loc. asks unrelated questions about the editing image itself. Since edits made under visual distractions rely on the whole image or visual distractions in the background rather than on the subject (Sec. 3.3), the failures can be caused by that they might more easily overwrite the unrelated knowledge about the same image.

Dear Reviewer,

Thank you once again for your commitment to reviewing our paper and assisting us in improving our work. We would like to remind you that the discussion window will close soon, and we eagerly await your feedback.

We have provided detailed explanations for each of your concerns. We would greatly appreciate it if you could review our responses and let us know if they fully or partially address your concerns. Any additional comments you may have would also be highly appreciated.

Best regards,

Authors

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We sincerely appreciate the time and effort you have put into reviewing our paper. We are glad to see that the reviewers found the problem of visual distractions in MMKE interesting and our controlled benchmark well-motivated. Your comments and suggestions have also been very helpful in improving the quality of our paper. Following them, we have uploaded an updated draft with the following revisions (revisions are colored in blue).

  1. We added a new Sec. 3.3 that analyzes why edits become fragile under visual distractions, going beyond the standard MMKE metrics (details in App. A.7). We analyze which region of the editing image the edited knowledge binds to, and use LoRA as a case study to examine whether the edit changes where the model looks in a new image and whether a new image can still activate the edit. We find that the edits made under visual distractions rely on the whole scene or on the distractions rather than on the subject, and that the failure lies in how the knowledge is written into the model rather than in how the model reads the new image.

  2. We added quantitative validations of the benchmark. We measure the subject fidelity across scenarios with a VLM judge and the CLIP similarity on the segmented subject (new App. A.5), and report the inter-annotator agreement for each criterion and the pass rates across knowledge categories and visual scenarios (updated App. A.4).

  3. We added the 95% confidence intervals of the main benchmark results, which confirm that our findings still hold (new App. A.8).

Additional results that are specific to individual questions are provided in the responses below.

Thank you again for your time. We look forward to hearing from you if you have any additional questions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.