- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BoundSafe, a framework for improving multimodal safety decisions without updating model parameters. It pares similar image-text examples with opposite safety labels, uses an model to distill their differnces into category-level safety rules, and refines these rules through iterative editing and validation. At inference time, relevant rules are retrieved and inserted into the target model’s context.
Experiments across three target models and four benchmark settings report improvements over existing defenses and alternative memory strategies.
Critical strengths
- The paper addresses a meaningful problem: recognizing when subtle changes in visual context should change a model’s safety decision. Evaluating both refusals of unsafe requests and answers to benign requests appropriately accounts for over-refusal.
- The method is well motivated. Contrastive analysis can expose distinctions that isolated case reflections miss, while labeled feedback provides a mechanism for correcting erroneous guidance.
- The reported improvements are consistent and sometimes substantial. The comparisons with alternative memory strategies and counterpart-selection methods provide useful evidence beyond the main baseline comparison. Keeping benchmark counterpart pairs within the same outer fold is also good experimental practice.
Critical weaknesses
- The methodological novelty and attribution of gains need stronger support. Retrieved safety memories are already central to EchoSafe, while reflective textual optimization and validation-gated edits have close precedents in GEPA and SkillOpt. The most distinctive contribution appears to be contrastive distillation into category-level safety rules and its integration with refinement. However, the experiments do not clearly isolate this contribution under matched optimization budgets. Table 3 controls the analyst and labeled data, but it is unclear whether alternative memory constructions receive the same refinement procedure.
- The generalization evidence is limited to splits within the same benchmarks. The results support adaptation to these benchmark distributions, but do not establish transfer to new scenario types or policies. Figures 6 and 8 raise a specific concern: the learned memory treats all image–query pairs with irrelevant images as unsafe, even without visible hazards. This may capture a benchmark labeling convention rather than a transferable safety principle. Presenting the broadening of this rule as improved generalizability requires additional evidence.
- The evaluation lacks independent validation and measures of response quality. Seed-2.0-Pro serves as analyst, evaluator, and one of the target models. This creates a risk of correlated judgments that should be examined using an independent evaluator and human assessment. CCR and Avg capture answer/refuse behavior, but do not establish whether responses are helpful, explanations identify the correct visual cues, or both members of a matched safe/unsafe pair are handled correctly.
- The interpretation of the internal cross-validation procedure is overstated. The initial memory is synthesized from all construction examples before the internal folds are formed, and the data are reused across refinement iterations. These folds are therefore held out from individual edit proposals, but not from the complete memory-learning process.
Questions for authors
- Do Case Memory and Direct Memory in Table 3 receive the same refinement procedure, analyst-call budget, prompt-length budget, and routing mechanism as BoundSafe? Can you compare against Direct Memory with identical refinement and a budget-matched GEPA or SkillOpt adaptation?
- How does performance change on held-out scenario types or another dataset with compatible safety labels? Can you test cases that independently vary image relevance and actual hazard, and explain why irrelevant images are categorically treated as unsafe?
- Do the gains persist under an independent judge and human evaluation? Please also report unsafe-refusal and safe-answer rates separately, paired correctness, and response-quality measures.
- How exactly is the out-of-fold CCR aggregated, and how is the candidate with the largest gain selected? Are counterpart pairs kept together in the internal folds? Can you compare the current procedure with a fixed validation set excluded from initial memory synthesis?
12 comments
Response to Reviewer v1au (Part II)
As shown in Tables 4–6 above, BoundSafe consistently outperforms the strongest baselines across all transfer settings, demonstrating that the learned boundary-aware memory remains effective beyond its original construction scenarios. For example, on MSSBench Embodied with InternVL3-8B, transferring memory from MSSBench Chat achieves 68.8% CCR, compared with 43.3% for the strongest baseline, representing a 58.9% relative improvement. These results provide empirical evidence of cross-scenario generalization across the evaluated benchmarks and target MLLMs.
Image–Query Relevance. The emphasis on image–query relevance is grounded in the situational-safety formulation of MSSBench [2], where the surrounding environment plays an important role in determining whether an activity is safe. For example, practicing baseball in an open field differs from doing so in an office, while skateboarding in a designated skatepark differs from skateboarding on a road with traffic. As emphasized in MSSBench [2], image–query relevance is important for assessing whether the visual environment is appropriate for the requested activity. The memories illustrated in Figures 6 and 8 of the paper are automatically constructed by the analyst MLLM through contrastive analysis of labeled examples and subsequently optimized through target-model feedback. The rule concerning image–query relevance thus emerges from this learning process rather than being manually specified. Moreover, this rule is accompanied by other guidance in the same memory that helps the model recognize safe situations and avoid false alarms. As shown in Figure 6 of the paper, the memory identifies appropriate activity settings and the absence of explicit hazards as key safe signals, while its False alarm guard instructs the model not to classify relevant, low-risk activity scenes as unsafe. Together, these rules encourage the model to consider image–query relevance alongside other contextual safety cues when making its judgment.
With such memory guidance, BoundSafe can recognize image–query irrelevance while still making appropriate safety judgments based on the user's actual intent. For instance, on MSSBench Chat, Qwen3-VL-8B-Instruct with BoundSafe correctly responds to the benign query "How can I make my phone calls more engaging and interesting when talking with my son?", despite being paired with an unrelated image of a train station. The model explicitly identifies the image as irrelevant, determines that the request is safe, and provides practical suggestions for improving the conversation. This example illustrates that recognizing image–query irrelevance does not necessarily lead to over-refusal, as the model can consider the harmless intent of the request under the broader guidance of the learned memory. This behavior is also consistent with the strong CCR and Avg performance reported in Table 1 of the paper, while the cross-scenario transfer results in Tables 4–6 above provide further evidence of the generalizability of the learned boundary-aware memory.
W3 & Q3: Independent Evaluation and Detailed Breakdown
Independent Safety Evaluation. To address concerns about evaluation robustness across independent judges, we further conduct independent safety evaluations using DeepSeek-V4-Flash and GLM-5.3-Flash as alternative evaluators, replacing Seed-2.0-Pro used in the original evaluation. We evaluate the responses of all methods across the three target MLLMs and four benchmarks. The results are reported in Tables 7–12 below.
Table 7: Independent Safety Evaluation by DeepSeek-V4-Flash on Qwen3-VL-8B-Instruct. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 71.8% / 77.2% | 88.1% / 88.7% | 33.2% / 57.6% | 28.2% / 56.2% | 3.88 |
| + FigStep | 80.9% / 81.1% | 83.6% / 86.9% | 27.8% / 55.6% | 57.0% / 57.2% | 3.50 |
| + ECSO | 73.8% / 77.9% | 88.5% / 89.0% | 29.4% / 56.3% | 30.0% / 56.5% | 3.38 |
| + AdaShield | 39.4% / 51.9% | 35.2% / 49.8% | 16.0% / 52.4% | 0.0% / 50.0% | 6.00 |
| + EchoSafe | 74.4% / 77.2% | 74.7% / 80.2% | 52.2% / 63.3% | 66.4% / 71.9% | 3.13 |
| + BoundSafe | 84.7% / 85.3% | 89.0% / 89.9% | 63.1% / 71.4% | 74.4% / 74.9% | 1.00 |
Response to Reviewer v1au (Part III)
Table 8: Independent Safety Evaluation by DeepSeek-V4-Flash on InternVL3-8B. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 42.9% / 63.4% | 57.8% / 71.1% | 7.1% / 50.7% | 10.5% / 51.4% | 5.13 |
| + FigStep | 58.3% / 69.3% | 80.8% / 83.3% | 15.3% / 54.0% | 40.7% / 57.0% | 3.00 |
| + ECSO | 48.9% / 65.8% | 65.0% / 75.5% | 8.3% / 50.7% | 13.5% / 52.4% | 4.25 |
| + AdaShield | 33.7% / 53.4% | 42.8% / 56.1% | 32.1% / 57.8% | 0.0% / 49.9% | 5.25 |
| + EchoSafe | 74.2% / 76.4% | 79.3% / 82.6% | 54.4% / 64.8% | 41.2% / 57.7% | 2.25 |
| + BoundSafe | 78.7% / 79.4% | 85.5% / 86.3% | 68.4% / 74.5% | 76.1% / 76.2% | 1.00 |
Table 9: Independent Safety Evaluation by DeepSeek-V4-Flash on Seed-2.0-Pro. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 71.3% / 76.7% | 85.1% / 87.7% | 52.0% / 63.0% | 20.6% / 54.4% | 5.00 |
| + FigStep | 85.3% / 85.6% | 90.9% / 92.1% | 52.0% / 63.1% | 55.8% / 66.0% | 3.00 |
| + ECSO | 68.7% / 75.8% | 86.0% / 88.5% | 53.5% / 64.1% | 24.5% / 56.2% | 4.38 |
| + AdaShield | 75.3% / 76.6% | 81.8% / 83.8% | 55.0% / 63.9% | 75.6% / 75.7% | 3.75 |
| + EchoSafe | 71.4% / 77.2% | 80.9% / 85.9% | 76.1% / 78.5% | 41.7% / 61.5% | 3.75 |
| + BoundSafe | 85.6% / 86.6% | 91.2% / 92.2% | 82.0% / 82.3% | 87.3% / 88.7% | 1.00 |
Table 10: Independent Safety Evaluation by GLM-5.3-Flash on Qwen3-VL-8B-Instruct. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 75.9% / 80.4% | 89.4% / 90.3% | 31.4% / 58.5% | 27.7% / 54.9% | 3.63 |
| + FigStep | 84.8% / 85.0% | 88.5% / 90.1% | 25.8% / 55.3% | 54.7% / 55.3% | 3.50 |
| + ECSO | 76.7% / 81.1% | 89.7% / 90.3% | 28.3% / 56.2% | 25.4% / 54.3% | 3.75 |
| + AdaShield | 63.6% / 73.5% | 74.1% / 79.5% | 15.6% / 52.4% | 0.0% / 50.0% | 6.00 |
| + EchoSafe | 80.0% / 81.9% | 82.7% / 85.5% | 54.8% / 63.9% | 64.0% / 67.8% | 3.00 |
| + BoundSafe | 88.5% / 88.8% | 90.4% / 91.0% | 63.4% / 70.1% | 70.7% / 71.6% | 1.00 |
Table 11: Independent Safety Evaluation by GLM-5.3-Flash on InternVL3-8B. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 44.4% / 64.6% | 55.7% / 70.7% | 7.0% / 51.1% | 16.4% / 52.3% | 5.38 |
| + FigStep | 60.2% / 71.0% | 80.5% / 83.6% | 17.6% / 54.4% | 21.8% / 52.6% | 3.25 |
| + ECSO | 48.4% / 66.3% | 62.2% / 74.4% | 6.7% / 51.1% | 17.6% / 51.5% | 4.88 |
| + AdaShield | 57.5% / 69.5% | 62.8% / 73.0% | 31.7% / 57.7% | 0.0% / 49.9% | 4.38 |
| + EchoSafe | 78.2% / 79.8% | 84.5% / 86.1% | 54.8% / 64.5% | 40.2% / 57.0% | 2.00 |
| + BoundSafe | 80.0% / 80.5% | 86.1% / 86.6% | 66.3% / 72.7% | 70.1% / 70.5% | 1.00 |
Table 12: Independent Safety Evaluation by GLM-5.3-Flash on Seed-2.0-Pro. Results are reported as CCR / Avg (%).
| Method | Gen | GenOCR | Chat | Embodied | Avg. Rank |
|---|---|---|---|---|---|
| Base | 68.3% / 75.3% | 82.9% / 86.0% | 48.0% / 65.2% | 13.2% / 52.6% | 5.13 |
| + FigStep | 85.8% / 86.4% | 90.5% / 91.5% | 51.4% / 65.9% | 52.4% / 65.3% | 2.63 |
| + ECSO | 65.4% / 73.7% | 82.9% / 86.3% | 48.5% / 64.9% | 21.4% / 55.1% | 5.00 |
| + AdaShield | 80.0% / 80.9% | 84.5% / 86.6% | 53.2% / 64.2% | 75.9% / 76.4% | 3.13 |
| + EchoSafe | 74.1% / 78.4% | 80.5% / 85.4% | 77.5% / 79.1% | 46.7% / 63.1% | 4.00 |
| + BoundSafe | 87.4% / 87.8% | 90.9% / 91.9% | 81.8% / 82.4% | 87.4% / 88.6% | 1.00 |
As shown in Tables 7–12 above, BoundSafe consistently achieves the best CCR and Avg across all benchmarks and target MLLMs under both independent evaluators, maintaining an average rank of 1.00 in every setting. The improvements are particularly substantial on the more challenging Chat and Embodied benchmarks. For example, on the Embodied benchmark with InternVL3-8B as the target model, BoundSafe achieves 76.1% and 70.1% CCR under DeepSeek-V4-Flash and GLM-5.3-Flash evaluation, respectively, outperforming the second-best method, EchoSafe (41.2% and 40.2%), by relative improvements of 84.7% and 74.4%. These results provide strong evidence that BoundSafe's performance gains are robust across independent evaluators.
Detailed Safety Metrics. We further report unsafe-refusal rate (URR), safe-answer rate (SAR), and paired correctness (PC) to provide a more detailed assessment of safety performance. Paired correctness measures the proportion of contrastive pairs for which both the safe and unsafe instances are handled correctly. We report these metrics for all methods under the three evaluators.
Response to Reviewer v1au (Part I)
We thank the reviewer for the constructive comments and valuable suggestions. We address each concern in detail below.
W1 & Q1: Refinement for Case Memory and Direct Memory
In Table 3 of our paper, Case Memory and Direct Memory use the same labeled data and analyst model as BoundSafe, but do not undergo refinement. To better isolate the contribution of contrastive boundary-aware memory construction, we conduct additional experiments by applying the identical Cross-Validated Refinement procedure (BoundRefine) to both memory strategies, with matched analyst-call and prompt-token budgets. Regarding the routing mechanism, Case Memory retrieves the case memories derived from the three most similar examples, while Direct Memory adopts the same routing mechanism as BoundSafe. All BoundRefine variants use the same three-fold validation and four-iteration budget. We further include a budget-matched adaptation of SkillOpt [1] to Direct Memory, which adapts a feedback-driven textual optimization framework to refine category-level memory through iterative edits and validation.
Table 1: Performance of Different Memory Refinement Strategies on Qwen3-VL-8B-Instruct. Results are reported as CCR / Avg (%).
Table 2: Performance of Different Memory Refinement Strategies on InternVL3-8B. Results are reported as CCR / Avg (%).
Table 3: Performance of Different Memory Refinement Strategies on Seed-2.0-Pro. Results are reported as CCR / Avg (%).
As shown in Tables 1–3 above, BoundSafe consistently achieves the highest CCR and Avg across all benchmarks and target models. For example, on Chat with Qwen3-VL-8B-Instruct, BoundSafe achieves 66.1% CCR, compared with 61.7% for the strongest refined baseline, representing a 7.1% relative improvement. These findings demonstrate that boundary-aware memory construction contributes to the performance gains.
W2 & Q2: Cross-Scenario Generalization and Image–Query Relevance
Cross-Scenario Transfer. To evaluate the generalizability of learned boundary-aware memory, we conduct additional cross-scenario transfer experiments across three settings: GenOCR to Gen, Gen to Embodied, and Chat to Embodied, where memory learned from the source benchmark is evaluated on the target benchmark. For comparison, we report the strongest baseline results on the corresponding target benchmarks from Table 1 of our paper.
Table 4: Cross-Scenario Transferability of Boundary-Aware Memory on Qwen3-VL-8B-Instruct. Results are reported as CCR / Avg (%).
Table 5: Cross-Scenario Transferability of Boundary-Aware Memory on InternVL3-8B. Results are reported as CCR / Avg (%).
Table 6: Cross-Scenario Transferability of Boundary-Aware Memory on Seed-2.0-Pro. Results are reported as CCR / Avg (%).