acceptodds
Under review as a conference paper at ICLR 2027

Hyperbolic Detoxification for Vision-Language Model Safety via Intent-Risk Calibration

Abstract

Improving the safety of vision–language models (VLMs) without compromising their utility on benign image–text inputs remains a fundamental challenge. A direct strategy is to first generate a draft response and then determine whether it should be rewritten based on its content. Although simple and effective, this response-first approach can be unreliable for ambiguous, gray-zone cases in which output-level signals alone are insufficient for robust safety decisions. To tackle the issue, we introduce Intent-Calibrated Hyperbolic Detoxification, a response-first safety framework that jointly assesses output safety and multimodal input risk. Our method first evaluates each draft using a response-level reward model: high-risk drafts are forcibly rewritten, whereas clearly safe drafts are returned unchanged. For drafts near the safety decision boundary, a hyperbolic encoder captures the hierarchical risk structure of the image–text input, and a learned multilayer-perceptron gate combines the resulting input-risk score with the response-level reward to determine whether rewriting is necessary. This design restricts input-driven intervention to genuinely ambiguous cases, thereby reducing unnecessary refusals and preserving benign responses. We evaluate the proposed method using attack success rate (ASR) and helpfulness rate (Helpful) on VLM safety benchmarks, together with accuracy on RealWorldQA. We additionally benchmark the underlying backbones and adapted safety baselines on MMMU and MMStar. Across the safety and RealWorldQA evaluations, we achieve a more favorable safety–utility trade-off than existing strong baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Model, Detoxification, Multimodal Safety, Hyperbolic Space

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZoXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a visual language model security control framework based on intent risk calibration, aiming to address the issues that existing security methods tend to overly reject normal requests and have difficulty accurately identifying ambiguous risks. The core idea of the paper is to first have the frozen VLM generate the initial response, and then use the Reward Model to evaluate the output security. Clearly safe responses are directly retained, obviously dangerous responses are rewritten, while for the gray area responses with uncertain security, the hyperbolic space is used to hierarchically model the image, text, and combined text-image risks, and through MLP, the input risks and output rewards are fused to determine whether intervention is needed. For the responses that need to be modified, the method learns the safe direction through the hidden state differences between safe and unsafe samples, and combines the reward-guided Hidden-State Steering to achieve safe rewriting. Experiments show that this method significantly reduces the attack success rate while improving the usefulness of the responses.

Critical strengths

The main strength of this paper lies in the selective safety intervention mechanism based on gray-zone decision-making. Different from traditional global safety intervention methods, this approach prioritizes the assessment of the safety of the response based on the output rewards, and only introduces multimodal input risks for joint decision-making when the judgment is uncertain, thereby reducing unnecessary modifications to the safe response. Generally speaking, there is usually a trade-off between model safety and usefulness, but this method improves both indicators simultaneously through targeted intervention. Compared with global intervention strategies, the gray-zone decision-making mechanism achieves a lower attack success rate and higher response usefulness, demonstrating the effectiveness and practical application value of this design.

Critical weaknesses

One of the key innovations proposed in the paper is based on hyperbolic space and it is claimed that it can effectively represent complex hierarchical structures. However, the actual method only divides multimodal risks into four predefined levels and measures the risk by the distance from the sample to the safety reference point. This modeling approach essentially relies on one-dimensional radial risk ranking and does not explicitly utilize the representation ability of hyperbolic space for complex branching hierarchical relationships. Moreover, the authors also admit in the appendix that Euclidean space can also satisfy these risk ranking constraints. This makes me question the necessity of hyperbolic space. Additionally, if the proposed risk structure does not require the special geometric properties of hyperbolic space, then using a more complex hyperbolic representation would introduce unnecessary overhead.

Questions for authors
  1. This article assumes a fixed risk ranking order: SS < SU < US < UU. Can the basis for this specific ranking be explained? If you quote someone else's argument, you should include the citation.
  2. The uncertain responses are identified based on the distance between the reward score of the Reward Model and the preset threshold, and the responses that are judged to have high confidence and are considered safe are directly retained. Could you explain the calibration degree of these reward scores? In particular, how many actually unsafe responses were wrongly classified as having high confidence and thus bypassed the subsequent joint security assessment of input and output? Or, how was the threshold determined? Please provide an explanation.
  3. This paper combines the Reward Model, hyperbolic risk encoder, gray area decision module, and hidden state-guided secure rewriting. It is necessary to analyze how much of the final performance improvement comes from more accurate identification of unsafe responses, and how much comes from more effective secure rewriting? Abandonment experiments should be supplemented to clearly address this part, or they should be added in the appendix.
  4. The model selected for the experiment is relatively outdated. I understand the author's work schedule for the article, but I think at least a set of models for 2026 should be added, such as qwen-3.5.
Reviewer_JSK7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a hierarchical framework to make the VLMs' output safe. It is purely inference-time, with no training requirement. It first scores a generated draft, preserves high-scoring responses, and rewrites low-scoring ones. For borderline drafts, a learned gate combines the output reward with image, text, and joint input-risk estimates obtained from hyperbolic embeddings. It uses a hidden-state steering method to re-write all unsafe responses.

Critical strengths
  1. The hierarchical safety routing framework makes re-writing happens only with clearly harmful responses, which saves overall inference cost.

  2. The experiments demonstrate the effectiveness of the proposed framework. Specifically, on LLaVA-1.5-7B and SPA-VL, the reported ASR decreases from 29.06% to 1.13%, while helpfulness increases from 63.21% to 91.89%.

  3. Ablation provides important understanding to the proposed framework. Table 3 compares output-only control, rule-based thresholds, a global gate, and the proposed gray-zone gate. Table 4 separately varies geometry and the hierarchy constraint, which helps distinguish their contributions.

Critical weaknesses
  1. The comparison with closely related methods is lacking. For example, ETA [1] already combines input and output assessment to selectively regenerate responses. HySAC [2] uses hyperbolic geometry to organize safe and unsafe multimodal content. ARGRE [3] already combines a PCA-derived safety direction, reward-gap-scaled steering, and gradient refinement.

[1] https://arxiv.org/abs/2410.06625

[2] https://arxiv.org/abs/2503.12127

[3] https://arxiv.org/abs/2510.01243

  1. The aggregate metrics make the safety-utility trade-off harder to interpret. Appendix F computes ASR over all evaluated prompts, including benign ones, while helpfulness also counts safe redirection on unsafe requests. These quantities depend on the prompt mixture and do not directly separate attack resistance from benign-answer quality.
Questions for authors

Please refer to the weaknesses.

Reviewer_r4Q6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of improving VLM safety while preserving helpful responses to benign multimodal inputs. It proposes Intent-Calibrated Hyperbolic Detoxification, which combines response-level safety assessment with input-risk modeling for ambiguous cases near the safety decision boundary. By selectively rewriting only high-risk or uncertain responses, the method reduces unnecessary refusals while maintaining safety. Experiments show a favorable safety–utility trade-off over strong existing baselines across multiple VLM benchmarks.

Critical strengths

S1. Favorable Balance between Safety and Helpfulness

  • Within the evaluated benchmarks and baseline comparisons, the proposed framework achieves a favorable balance between safety and helpfulness. However, this strength should be considered alongside the limitations discussed below.
Critical weaknesses

W1. Missing critical implementation details and questionable design choices

  • The paper primarily emphasizes the quantitative superiority of the resulting framework, while several implementation details that are critical not only for reproducibility but also for assessing the soundness and practicality of the method remain unclear.
  • First, for the hyperbolic multi-view risk encoding, the paper does not specify under what inference conditions, or from which tokens and layers, the pair-, text-, and image-level representations are extracted. If these require separate inference passes, the efficiency of the framework could be substantially affected; similarly, using benchmark-specific representations would raise concerns about practicality and generalizability.
  • Second, Eq. (4) appears limited in handling cases where the image and text are individually benign but become unsafe only when combined, as considered in benchmarks such as SIUO and HOLISAFE. The current design appears to aggregate image- and text-level risks largely through an OR-like formulation, which may not fully capture such compositional safety risks.
  • Third, the framework relies on multiple thresholds to distinguish safe, unsafe, and gray-zone cases. These thresholds directly affect both performance and practical usability, yet their exact values and selection procedures are not clearly reported. If substantial tuning is required, this would be a significant concern.

W2. Insufficient analysis of the multi-stage routing process and efficiency

  • The proposed framework consists of roughly four to five stages with non-trivial routing between them. Given this complexity, a detailed efficiency analysis and statistics on how frequently samples are routed through each stage are important for understanding the practical behavior of the framework. However, the paper provides only limited aggregate information, such as an approximately 1.3× inference-time overhead under a particular setting (line 471). Without more detailed routing and computational analyses, it is difficult to assess where the additional cost arises and how practical the method is across different operating conditions.
Questions for authors

I encourage the authors to carefully address the critical concerns raised above during the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.