acceptodds
Under review as a conference paper at ICLR 2027

Monitor More, Supervise Less: Cross-Modal Safety Monitoring for Omni-Models

Abstract

Omni-models can receive harmful requests through different modalities and their combinations. Yet safety monitors that detect such requests from internal representations often generalize poorly beyond the modalities on which they are trained. With modalities, a request can appear in up to nonempty modality configurations, making exhaustive safety supervision and continual updating costly. This raises a central question: Can we achieve effective cross-modal safety monitoring in a data-efficient way? We study this question by formulating three supervision regimes, which compare monitors trained with safety data from one individual modality (), every individual modality (), and every nonempty modality configuration (). Instantiating with text, image, and audio modalities, we evaluate all three regimes on two datasets across five omni-models. The performance differences between successive regimes define the modality gap and the combination gap. Empirically, the modality gap is generally larger, while much of the combination gap can be recovered with a simple strategy that retains safety supervision. To reduce the modality gap, we learn a modality-shared, safety-informed subspace while keeping safety supervision. Our method improves mean AUC by and points on the two datasets, closing 57.1% and 68.4% of the modality gap, respectively. Initialized only once, the learned subspace can be reused across new safety datasets, with strong performance in cross-dataset generalization and continual monitor updating.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety monitoring, omni-model safety, probing classifiers, cross-modal transfer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_D96dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at probe-based safety monitoring for omni-models and asks how much labeled safety data is needed across modality configurations. It defines three regimes by coverage. The authors build omni versions of WildGuardMix and Aegis 2.0 by rendering prompts into all seven configurations, and evaluate linear and MLP probes on five open omni-models. They find that the gap between O(1) and O(n) is usually larger than the gap between O(n) and O(2ⁿ), and that adding a single text+image+audio set to O(n) closes most of the latter. To address the modality gap under O(1), they learn a modality-shared, safety-informed subspace from unlabeled multimodal data plus text safety data, then train a text-only probe in that space. This gives around 3-4 AUC points on average and transfers across datasets

Critical strengths
  1. The O(1)/O(n)/O(2ⁿ) framing is clean and the modality-gap vs. combination-gap decomposition is a useful way to think about where supervision actually matters.
  2. The method is simple and practical. It needs no non-text safety labels, is learned once from general data, and the continual-updating experiment is a realistic deployment scenario.
Critical weaknesses
  1. The motivation of O(1) is that non-text safety data is expensive, yet the benchmark itself is generated automatically from text. In this setting O(2ⁿ) supervision is essentially free, so the obvious baseline is simply to render your text safety set into every configuration and train on that. The paper never addresses this.
  2. A simple per-modality mean-shift baseline is missing.
Questions for authors
  1. Given the datasets are generated from text automatically, why wouldn’t a practitioner just generate the full O(2ⁿ) training set? What does the O(1) method offer in that case?
  2. Does the learned subspace still help when the test-time rendering differs (different TTS voice, noise, real images, different fonts)?
  3. How do you set the decision threshold for non-text inputs in practice?
4 comments
AuthorsOpenReview ↗

Response 1/2

Thank you very much for your time and effort in providing valuable comments and for acknowledging the O(1)/O(n)/O(2n2^n) framing, the gap decomposition, and the simple and practical method. Our responses to your concerns are summarized as follows:

Q1 / W1. Cost comparison between O(1) and O(2n2^n)

A1. Thanks for raising the question! We would like to clarify that O(2n)O(2^n) supervision is not free in practice. We analyze three cost components: compute cost, API cost, and update cost.

(1) Representation-extraction compute cost scales as 2n−12^n-1. Extracting representations for all modality configurations is a major training cost. On a single 80GB A100, extraction for 10,000 prompts takes 2 GPU-hours for O(1)O(1) (text only) versus 30 for O(2n)O(2^n) (all seven configurations), as non-text inputs require more backbone tokens and modality encoders. In practice, a recent Anthropic report of scaling laws for safety probes [1] finds that difficult datasets require at least 1,000,000 examples to reach the error floor. Extrapolating to this scale gives 200 versus 3,000 GPU-hours per model per update, or $400 versus $6,000 at $2/A100 GPU-hour. Additional modalities, such as video, grow this cost exponentially.

Setting (per model per update, 1× A100 80GB, $2/GPU-hour)O(1)O(1) GPU-hoursO(1)O(1) costO(2n)O(2^n) GPU-hoursO(2n)O(2^n) cost
Our setup (10,000 prompts)2.0$430$60
Practical setting (1,000,000 prompts [1])200$4003,000$6,000

(2) API cost for multimodal data generation. Combined-modality examples require an LLM call to distribute semantic content across modalities, plus image rendering and/or TTS. Closed-source APIs typically produce higher-quality images and audio than open-source alternatives, but incur generation costs that scale as O(2n)O(2^n) across modality configurations.

(3) Update cost for new attack types. Safety data is not static as new attack types and taxonomies appear regularly. Each new text dataset would have to be re-converted to all configurations and re-extracted, so costs (1) and (2) recur with every update.

Our O(1)O(1) subspace is estimated once from general multimodal data (which is of the kind already used in omni-model training and evaluation) and reused across monitors and updates, so it incurs only a one-time compute cost.

[1] Wu, Y., & Wei, J. (2026). Scaling Laws for Safety Probes. arXiv preprint arXiv:2610.30678.

Q2. The learned subspace's robustness to differences in test-time rendering

A2. Thanks for this constructive suggestion! Following the Reviewer's suggestion, we test robustness to different TTS voices, added noise, real images, and different fonts. All results are AUC of our O(1)O(1)-subspace method with the subspace and probe kept fixed, i.e., estimated and trained on the original rendering only.

As shown in Figure 5 in the revised manuscript, voices and noise have little effect on AUC (Δ<0.02\Delta < 0.02), and fonts have almost none (Δ<0.005\Delta < 0.005). Replacing typographic images with real images causes the largest drop (Δ≈0.08\Delta \approx 0.08–0.100.10), reflecting an expected within-modality shift because the image component of subspace estimation data is solely from typographic images. We deliberately use typographic rendering to preserve semantic content and isolate the cross-modal gap (L205–211). For deployment, including real images in the general data used for subspace estimation may reduce this within-modality gap without additional safety labels. These results and analysis have been added to Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Response 2/2

Q3. How is the decision threshold set for non-text inputs?

A3. Our main metric is AUROC (AUC), which is threshold-free, so the main results do not depend on any threshold choice. For the complementary threshold-based metrics in the Appendix (Accuracy and F1), we use a fixed threshold of 0.5 for all configurations. Even with this simple choice, the gains carry over, as shown in Table 36 in the revised manuscript (mean accuracy over the seven test configurations). Our O(1)O(1) method improves accuracy over the original O(1)O(1) monitor in 9 of 10 settings. In practice, a practitioner can calibrate the threshold once on a small multimodal validation set and reuse it. For example, calibrating to a target false positive rate requires only benign examples, which can be drawn from general (non-safety) data in each modality.

W2. Missing a per-modality mean shift baseline.

A4. Following the Reviewer's suggestion, we add a per-modality mean-shift baseline: for a test input xx from configuration cc, we use x′=x−μc+μtextx' = x - \mu_c + \mu_{\text{text}}, where μc\mu_c is the mean representation of configuration cc. Note that we compute μc\mu_c from each test configuration's own training-set representations, so this baseline requires data from all 2n−12^n-1 configurations, which is more than our O(1)O(1) method has access to. Even so, it underperforms our O(1)O(1)-subspace method in all settings except Qwen2.5-Omni-7B, where the two are comparable. We note that for the linear probe, the mean shift leaves per-configuration AUC unchanged (identical to the original O(1)O(1) row in Table 1) as a per-configuration constant shift adds a constant to every score within that configuration and therefore does not alter the ranking. This comparison has been added to Appendix C.3.1 of the revised manuscript.

Qwen2.5-omni-3BQwen2.5-omni-7BQwen3-omni-30BPhi4-mmInteractiveOmni
Mean shift (O(2n)O(2^n)) - Linear Probe0.812±0.0070.874±0.0050.808±0.0130.758±0.0070.869±0.007
Ours (O(1)O(1)) - Linear Probe0.852±0.0020.874±0.0030.870±0.0020.798±0.0090.879±0.006
Mean shift (O(2n)O(2^n))- MLP Probe0.830±0.0070.881±0.0030.819±0.0100.757±0.0070.876±0.004
Ours (O(1)O(1)) - MLP Probe0.869±0.0010.879±0.0050.869±0.0020.823±0.0060.887±0.005

The rebuttal addresses my main concerns, particularly by clarifying the practical cost advantage of the O(1) approach and adding the requested mean-shift baseline and robustness evaluations. I find the responses satisfactory and will maintain my recommendation of acceptance.

Thank you for your response! We are pleased that your concerns have been satisfactorily addressed. We appreciate your time and valuable feedback, which have helped us improve the manuscript!

Reviewer_pPxVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies probe-based cross-modal safety monitoring in omni-modal models from the perspective of safety supervision coverage. The authors introduce three supervision settings: O(1)O(1), covering only a single modality; O(n)O(n), covering all individual modalities; and O(2n)O(2^n), covering all modality combinations. Based on these settings, they distinguish between the “modality gap” and the “combination gap.” Across five models and two datasets constructed through controlled modality transformations, they find that the modality gap is generally larger, while adding the full joint configuration to unimodal training data closes most of the combination gap. To improve generalization when safety supervision is available only for text, the authors combine general multimodal data with text safety data to learn a shared subspace that suppresses modality variation while preserving safety information. The method improves average AUC by 3.2 and 3.9 points on the two datasets, respectively, and demonstrates the feasibility of cross-dataset transfer and reusing the subspace when updating the monitor.

Critical strengths
  1. The problem is clearly defined.
  2. The experimental analysis is fairly systematic.
  3. The method is clearly designed and offers a degree of interpretability.
Critical weaknesses
  1. A potential issue with SmodalityS_{\mathrm{modality}} is that it can identify discrepancies between different modal representations of the same request, but cannot directly determine whether these discrepancies reflect irrelevant variation or useful safety information. For example, the text might say, “I want to jump from a high place,” while the image shows either a trampoline or a tall building. If these scene elements are discarded during the mapping, the safety classifier could be affected. How do the authors explain and analyze this issue?
  2. The paper demonstrates that the use of non-text safety supervision can be reduced. However, it does not sufficiently establish that this approach is more cost-effective than directly constructing a small amount of multimodal safety training data in a setting where safety labels can be directly inherited through synthetic transformations.
  3. Would increasing the amount of data used for estimation make the mapping more accurate? Are there any performance bottlenecks?
  4. Although the paper discusses MULTIGUARD, SafeSteer, and existing bimodal safety subspace methods, the experimental comparisons are largely limited to standard probes, PCA, and component ablations. I recommend conducting these comparisons with clearly specified supervision budgets and data access conditions.

This is a good piece of work, but I believe these experiments and discussion are needed to clarify its value. I would consider raising my score if my concerns are addressed.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response 1/2

Thank you very much for your time and effort in providing valuable comments and suggestions! We appreciate your recognition of the clearly defined problem, systematic analysis, and clearly designed method. Our responses to your concerns are summarized as follows:

W1. SmodalityS_{modality} cannot directly determine whether cross-modal discrepancies are safety related (e.g., trampoline vs. tall building).

A1. Thanks for this insightful question! We agree that because SmodalityS_{modality} contains no multimodal safety information (to keep safety supervision at O(1)O(1)), the classifier may struggle when harmfulness depends solely on the image. To test this, we select 100 harmful text+image prompts from WGM-Omni with neutral text and pair each with a benign-image replacement. We compare AUC on the original subset (100 harmful + 100 random benign) and the paired subset, where only the image determines the label.

ModelMonitorOriginal subset AUCPaired subset AUC
Qwen2.5-Omni-3BLinear0.781±0.0190.744±0.029
Qwen2.5-Omni-3BMLP0.812±0.0160.808±0.068
Phi4-mmLinear0.672±0.0170.577±0.005
Phi4-mmMLP0.728±0.0160.651±0.022

The AUC drop is modest on Qwen2.5-Omni-3B but larger on Phi4-mm, possibly reflecting differences in modality alignment. Better alignment through SmodalityS_{modality} may help transfer text safety signals from SsafetyS_{safety} to text+image inputs. The remaining gap highlights the trade-off between limiting safety supervision to text and incorporating direct image safety supervision. These results and analysis have been added to Appendix C.3.6.

W2. Cost-effectiveness comparison between text-only safety supervision and constructing a small amount of multimodal safety data.

A2. Thanks for the suggestion! Multimodal safety supervision incurs (i) potential API costs for high-quality speech and visual image generation (although our experiments use typographic image) and (ii) representation-extraction costs, which become substantial at scale (see A1(1) to Reviewer D96d). Both costs recur whenever new attack types appear and the monitor must be updated. Our method instead extracts representations once from paired general multimodal data, which is the kind already used in omni-model training and evaluation, and reuses them across monitors and updates.

Although the two approaches operate in different supervision regimes, to quantify this trade-off, we augment a text-trained MLP probe with kk multimodal safety examples. As shown below, this requires k>1,000k > 1{,}000 for Qwen3-Omni and k>2,000k > 2{,}000 for Phi4-mm (10% and 20% of the training set). Two practical points follow. First, the required kk differs across models, so it cannot be set in advance without a multimodal validation set. Second, although our method uses more general multimodal data, this cost is paid once and reused across monitors and updates. These results and analysis have been added to Appendix C.3.5.

Qwen3-Omni

# of multimodal data20010002000
Text+Multimodal0.852±0.0350.867±0.0040.872±0.003
Ours Text-only0.869±0.0020.869±0.0020.869±0.002

Phi4-mm

# of multimodal data100020004000
Text+Multimodal0.812±0.0050.821±0.0090.826±0.007
Ours Text-only0.823±0.0060.823±0.0060.823±0.006
AuthorsOpenReview ↗

Response 2/2

W3. How does the performance scale with the amount of estimation data?

A3. We vary the amount of general data used to estimate the subspace from 1% to 100% (50 to 5,000 prompts) and report mean AUC over the seven test configurations. As shown in Figure 11 in the revised manuscript, on Qwen2.5-Omni-3B, AUC is still rising at 100% data, so more general data is likely to help further. On Phi4-mm, performance saturates at around 60% data. Since general data is of the kind already used in omni-model training and evaluation, the bottleneck is not the estimation data amount. Intrinsic cross-modal alignment is one possible explanation for the bottleneck: a more strongly aligned model permits a more aligned subspace, while a weakly aligned model limits how far the subspace can go. The results and analysis have been added to Appendix C.3.6 of the revised manuscript.

W4. Comparisons with more methods such as MULTIGUARD and SafeSteer.

A4. Following the Reviewer's suggestion, we add comparisons with MULTIGUARD and SafeSteer under the O(1)O(1) setting with identical data access for all methods: 10k text safety data and 5k paired general data (NoRobots) in text/image/audio. The bimodal method CARE requires image safety data, which is incompatible with our O(1)O(1) setting. All methods use the same hyperparameter selection procedure and five seeds, and are evaluated by mean AUC over the seven test configurations. Since SafeSteer is a steering method rather than a detector, we use its SVD subspace as the projection and train monitors on it. Under identical data access, our method achieves the best mean AUC on all five models. This comparison has been added to Section 4.3 of the revised manuscript.

Qwen2.5-omni-3BQwen2.5-omni-7BQwen3-omniPhi4-mmInteractiveOmni
MULTIGUARD (MLP)0.792±0.0140.861±0.0100.826±0.0130.775±0.0140.879±0.004
SafeSteer-Linear0.810±0.0040.854±0.0020.824±0.0070.710±0.0070.856±0.001
SafeSteer-MLP0.831±0.0130.859±0.0030.830±0.0100.747±0.0150.864±0.007
Ours-Linear0.852±0.0020.874±0.0030.870±0.0020.798±0.0090.879±0.006
Ours-MLP0.869±0.0010.879±0.0050.869±0.0020.823±0.0060.887±0.005

Thanks!My concerns are addressed and I will raise my score.

Thanks for your response and for raising the score! We sincerely appreciate your constructive feedback, which helps us improve our manuscript!

Reviewer_4NcTOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates data-efficient cross-modal safety monitoring for omni-models. The authors formulate three supervision regimes based on modality-configuration coverage: Single Monitoring (O(1)), Compositional Monitoring (O(n)), and Exhaustive Monitoring (O(2^n)). By evaluating these regimes across different modality configurations, the paper identifies the modality gap and combination gap in cross-modal safety monitoring.

To improve generalization under text-only safety supervision, the authors propose a modality-shared, safety-informed subspace learning method. The method combines modality variation, semantic covariance, and safety-discriminative information to learn a shared representation space for lightweight safety probes. Experiments on five omni-models and two constructed datasets demonstrate improvements in cross-modal safety detection, as well as promising results in cross-dataset generalization and continual monitor updating.

Critical strengths

Important research problem. The paper addresses an important challenge in omni-model safety monitoring: reducing the need for modality-specific safety supervision while maintaining reliable detection across different input modalities and their combinations.

Simple and practical design. The proposed subspace learning approach is intuitive and lightweight. It provides a potentially practical way to improve cross-modal safety monitoring without modifying or fine-tuning the underlying omni-model.

Extensive experimental evaluation. The authors conduct experiments across five omni-models with different architectures and sizes, two safety datasets, and seven modality configurations. The evaluation also includes ablation studies, cross-dataset generalization, and continual monitor updating, providing encouraging evidence of the method's effectiveness.

Critical weaknesses

W1. The proposed idea is closely related to existing work, particularly OMNIGUARD (EMNLP 2025), but no direct comparison is provided.

The central idea of this paper is to identify or construct modality-shared internal representations for efficient cross-modal safety monitoring. However, a similar direction has already been explored in OMNIGUARD (EMNLP 2025), which also leverages modality-aligned internal representations and lightweight classifiers for safety moderation across multiple modalities.

Although the proposed method differs from OMNIGUARD by explicitly learning a safety-informed subspace through covariance-based optimization, the fundamental motivation and overall monitoring paradigm are closely related.

Unfortunately, the paper does not provide a direct comparison with OMNIGUARD or sufficiently demonstrate the advantages of its learned subspace over existing modality-aligned representation approaches.

Therefore, the methodological novelty and incremental contribution over closely related work remain unclear. A controlled comparison under matched backbone models, supervision budgets, and evaluation settings would help establish the unique contribution of the proposed approach.

W2. Generalization is the central contribution of this paper, but the current evaluation is insufficient to substantiate the broader claims.

The main value of the proposed method lies in its ability to generalize safety supervision across different modalities, modality combinations, and safety datasets. However, the evaluation relies primarily on synthetically constructed multimodal datasets derived from textual safety prompts.

Specifically, image inputs are generated by rendering text into images, while audio inputs are obtained through text-to-speech synthesis. Multimodal combinations are constructed by distributing textual content across different modalities.

Consequently, the experiments mainly evaluate the transfer of safety classification across different representations of textual information, rather than generalization to genuinely multimodal safety scenarios involving natural images, real-world audio, or complex interactions between modalities.

Although the paper reports encouraging cross-dataset generalization and continual updating results, both evaluation datasets are constructed using similar modality transformation procedures. Therefore, these results do not fully establish whether the learned subspace generalizes beyond the synthetic transformation setting.

Since generalization is the central contribution, evaluation on independent, realistic multimodal safety benchmarks is necessary to support the broader applicability of the proposed method.

W3. The experimental comparisons are insufficient to establish the necessity of the proposed safety-informed subspace learning objective.

The proposed method learns a linear projection using modality covariance, semantic covariance, and safety-discriminative information. This formulation is closely related to established discriminative subspace learning and domain-invariant representation alignment techniques.

The paper provides component ablations and comparisons with direct PCA, which are useful. However, these experiments do not fully establish whether the specific proposed objective is necessary or whether comparable improvements could be achieved through more conventional supervised dimensionality reduction or cross-modal alignment approaches.

In particular, when the learned projection is followed by a linear safety probe, the overall prediction function remains linear in the original hidden representation space. The improvement therefore comes from the learned inductive bias rather than increased model expressiveness.

More competitive comparisons with established representation alignment and discriminative subspace learning methods would help clarify the methodological contribution.

W4. The practical generalization and supervision-efficiency claims require further validation.

The paper emphasizes that the learned subspace can be initialized once and reused for subsequent safety datasets and continual monitor updating. This is a potentially useful property, and the reported experiments provide encouraging evidence.

However, the current evaluation primarily demonstrates reuse of a model-specific subspace across two related safety datasets. For different omni-models, the subspace must be learned separately. The method also requires aligned multimodal data to estimate modality variation, although these examples do not require safety labels.

Therefore, the claimed O(1) supervision efficiency should be interpreted specifically in terms of safety-labeled modality coverage, rather than the total data preparation or computational cost.

Furthermore, it remains unclear how the learned subspace behaves under more substantial distribution shifts, newly introduced modalities, or changes to the underlying omni-model. These scenarios would be valuable for establishing the practical benefits of the proposed reusable monitoring framework.

Questions for authors

Comparison with OMNIGUARD: OMNIGUARD (EMNLP 2025) has already explored leveraging modality-aligned internal representations for cross-modal safety monitoring. What are the key methodological differences and practical advantages of the proposed approach? Can the authors provide a direct comparison under matched backbones and safety supervision settings?

Realistic cross-modal generalization: Since the current image and audio inputs are largely derived from textual prompts, can the authors evaluate the method on independent multimodal safety benchmarks containing natural images, real-world audio, and genuinely multimodal harmful requests? Does the learned subspace maintain its effectiveness without re-estimation?

Necessity of the proposed objective: How does the proposed safety-informed subspace compare with conventional domain-invariant representation learning or discriminative subspace learning approaches? Can the authors provide stronger baselines to isolate the benefits of the specific covariance-based objective?

Cross-dataset generalization: Both constructed datasets share similar modality transformation procedures. How does the method generalize to datasets constructed through substantially different pipelines or containing different safety categories and attack patterns?

Reusability and scalability: The authors demonstrate that a learned subspace can be reused across safety datasets. Can the same subspace remain effective when the underlying omni-model is updated or when new input modalities are introduced? What additional data and computational costs would be required in these cases?

Practical monitoring effectiveness: Can the authors report performance on more realistic multimodal attacks and detection metrics at operationally relevant false-positive rates? This would help demonstrate whether the proposed method provides meaningful improvements in practical safety monitoring rather than only average classification performance.

3 comments
AuthorsOpenReview ↗

Response 1/3

We really appreciate your time and effort in providing valuable comments to help us improve our manuscript. We thank the reviewer for acknowledging the important research problem, simple and practical design, and extensive experimental evaluation. Our responses are as follows:

Q1 / W1. Methodological and experimental comparison with OMNIGUARD

A1. Following the Reviewer's suggestion, we add a direct comparison with OMNIGUARD (cited as MULTIGUARD in our submission, its EMNLP published title).

Key methodological difference. OMNIGUARD decides where to monitor: it selects the layer with the most aligned full-dimensional embeddings, then trains a separate classifier per modality without addressing cross-modal transfer. Our method learns what to monitor: a low-dimensional subspace that suppresses modality-specific covariance while preserving safety-domain and benign–harmful covariance, enabling a single text-trained probe to transfer to unseen modalities and their combinations. To our knowledge, we are also the first to formulate the combinatorial supervision-scaling problem across modality configurations and define and compare its supervision regimes.

Experimental comparison. We run OMNIGUARD under the same five omni-model backbones and text-only safety supervision, i.e., the O(1)O(1) setting in our paper. Our O(1)O(1) method achieves higher mean AUC than OMNIGUARD on all five models. This comparison has been added to Section 4.3 of the revised manuscript.

Qwen2.5-omni-3BQwen2.5-omni-7BQwen3-omniPhi4-mmInteractiveOmni
OMNIGUARD0.792±0.0140.861±0.0100.826±0.0130.775±0.0140.879±0.004
Ours-Linear0.852±0.0020.874±0.0030.870±0.0020.798±0.0090.879±0.006
Ours-MLP0.869±0.0010.879±0.0050.869±0.0020.823±0.0060.887±0.005

Q2 / W2. Cross-modality generalization to realistic multimodal data such as real audio and image.

A2. Thanks for the constructive suggestion! To test realistic inputs while isolating the cross-modal factor, we keep the dataset fixed and change only the rendering, so that cross-modal and cross-dataset effects are not entangled (independent benchmarks are addressed in A4).

As shown in Figure 5 in the revised manuscript, TTS voices and human voice have little effect on AUC (Δ<0.02\Delta < 0.02), and fonts have almost none (Δ<0.005\Delta < 0.005). Replacing typographic images with real images causes the largest drop (Δ≈0.08\Delta \approx 0.08–0.100.10), reflecting an expected within-modality shift because the image component of subspace estimation data is solely from typographic images. We deliberately use typographic rendering to preserve semantic content and isolate the cross-modal gap (L205–211). For deployment, including real images in the general data used for subspace estimation may reduce this within-modality gap without additional safety labels. These results and analysis have been added to Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Response 2/3

Q3 / W3. Comparison with more established representation alignment and discriminative subspace learning methods.

A3. Following the Reviewer's suggestion, we compare against two representative domain-invariant alignment methods, mean shift (first-order) and CORAL (second-order), and a conventional discriminative subspace method, LFDA. All baselines use the same text safety data, probe architectures, and hyperparameter-selection protocol as our method. Note that mean shift and CORAL require training-set statistics from every modality configuration, i.e., O(2n)O(2^n) data coverage, whereas LFDA and our method use only O(1)O(1). Despite this, our method outperforms all baselines in every setting except Qwen2.5-Omni-7B with the MLP probe, where mean shift is marginally higher (0.881 vs. 0.879). This comparison has been added to Appendix C.3.1 of the revised manuscript.

Qwen2.5-Omni-3BQwen2.5-Omni-7BQwen3-Omni-30BPhi4-mmInteractiveOmni
Mean shift (O(2n2^n)) - Linear0.812±0.0070.874±0.0050.808±0.0130.758±0.0070.869±0.007
CORAL (O(2n2^n)) - Linear0.832±0.0050.870±0.0010.836±0.0030.786±0.0060.864±0.004
LFDA (O(11)) - Linear0.801±0.0090.859±0.0050.790±0.0060.766±0.0170.855±0.005
Ours (O(1)) - Linear0.852±0.0020.874±0.0030.870±0.0020.798±0.0090.879±0.006
Mean shift (O(2n2^n))- MLP0.830±0.0070.881±0.0030.819±0.0100.757±0.0070.876±0.004
CORAL (O(2n2^n)) - MLP0.845±0.0080.877±0.0040.836±0.0060.790±0.0040.874±0.004
LFDA (O(1)) - MLP0.794±0.0190.857±0.0070.778±0.0110.762±0.0120.846±0.008
Ours (O(1)) - MLP0.869±0.0010.879±0.0050.869±0.0020.823±0.0060.887±0.005

Q4. Cross-dataset generalization to substantially different datasets

A4. Thanks for the constructive suggestion! We first note that our main contribution targets cross-modal generalization: the subspace is designed to transfer a text-trained monitor across modality configurations, whereas generalization across safety datasets is governed mainly by the probe architecture and its training data. We also state explicitly in the Limitations (L527–533) during submission that we do not test more complex transformations or modality combinations.

Nevertheless, we test two datasets that differ substantially from WGM-Omni in construction pipeline and attack pattern: a subset of MM-SafetyBench, which consists of stable diffusion generated images rather than rendered typographically, and ReNeLLM, a nested-prompt jailbreak attack absent from WildGuardMix, which we evaluate as text and as its conversion into image and audio. Each harmful set is paired with an equal number of benign Aegis-Omni examples in the same modality to compute AUC, with subspace and monitors kept fixed from WGM-Omni.

MM-SafetyBench (Text+Image)

ModelMonitorO(1)O(1)O(1)−OursO(1)-OursO(n)O(n)
Qwen2.5-Omni-3BLinear0.267 ± 0.0570.420 ± 0.0880.523 ± 0.122
Qwen2.5-Omni-3BMLP0.305 ± 0.0890.440 ± 0.1320.613 ± 0.058
Phi4-mmLinear0.811 ± 0.0680.842 ± 0.0150.918 ± 0.006
Phi4-mmMLP0.789 ± 0.0600.821 ± 0.0210.966 ± 0.010

ReNeLLM (Text, Image and Audio average)

ModelMonitorO(1)O(1)O(1)−OursO(1)-OursO(n)O(n)
Qwen2.5-Omni-3BLinear0.743 ± 0.0180.750 ± 0.0750.707 ± 0.027
Qwen2.5-Omni-3BMLP0.739 ± 0.0540.725 ± 0.0630.757 ± 0.018
Phi4-mmLinear0.673 ± 0.0260.779 ± 0.0060.659 ± 0.022
Phi4-mmMLP0.623 ± 0.0130.705 ± 0.0220.663 ± 0.007

Our O(1)O(1) subspace method outperforms the raw-space O(1)O(1) monitor in all settings except Qwen2.5-Omni-3B with the MLP probe on ReNeLLM. Notably, on ReNeLLM it exceeds O(n)O(n) in three of four settings. The results and analysis have been added to Appendix C.3.3 of the revised manuscript.

AuthorsOpenReview ↗

Response 3/3

Q5. Reusability and cost when the omni-model is updated or new modalities are introduced.

A5. When the omni-model is updated (e.g., finetuned), its internal representations change, so the subspace must be re-estimated. This requires no additional data: we re-extract representations for the same general text, image, and audio prompts, which in our setup takes about 5 GPU-hours on a single A100 (≈10\approx 10 USD at 2 USD/GPU-hour).

Introducing a new modality requires only general (non-safety) data in that modality, which is of the kind already used in omni-model training and evaluation. The additional cost is extracting representations for this one modality; assuming it costs twice as much as text, this is about 2 GPU-hours (≈4\approx 4 USD) in our setup. The subspace is then re-solved with the new modality included in SmodalityS_{\text{modality}}, which is a closed-form eigenproblem and negligible in cost. Safety supervision remains text-only in both cases.

Q6. Performance on more realistic multimodal attacks and detection metrics at operationally relevant false-positive rates?

A6. Consistent with our paper's focus on the combinatorial modality-scaling challenge, we compare the supervision regimes (O(1)O(1), O(1)O(1)-Ours, and O(n)O(n)) under this metric. We report MLP probe's TPR@10%FPR on WGM-Omni (mean over the seven test configurations), and additionally on MM-SafetyBench as a more realistic cross-dataset test with the subspace and monitors kept fixed from WGM-Omni. The threshold is set at 10% FPR on each benchmark’s benign set, per configuration.

TPR@10%FPR on WGM-Omni

ModelO(1)O(1)O(1)O(1)-OursO(n)O(n)
Qwen2.5-Omni-3B0.532±0.0200.622±0.0020.681±0.012
Qwen2.5-Omni-7B0.655±0.0190.655±0.0090.719±0.005
Qwen3-Omni0.582±0.0260.651±0.0080.725±0.007
Phi4-mm0.475±0.0240.549±0.0130.612±0.006
InteractiveOmni0.690±0.0100.683±0.0110.763±0.005
five-model mean0.5870.6320.700

TPR@10%FPR on MM-SafetyBench

ModelO(1)O(1)O(1)O(1)-OursO(n)O(n)
Qwen2.5-Omni-3B0.052±0.0360.108±0.0640.120±0.035
Phi4-mm0.682±0.1330.793±0.0440.922±0.029

On WGM-Omni, our O(1)O(1) method raises the 5-model mean TPR@10%FPR from 0.587 to 0.632, closing 40% of the gap to O(n)O(n). On MM-SafetyBench, it closes 82% of the gap on Qwen2.5-Omni-3B and 46% on Phi-4-mm, without re-estimating the subspace. We note that absolute TPR on Qwen2.5-Omni-3B is low for all regimes, including O(n)O(n), indicating that this transfer is difficult at the dataset level rather than specific to our method. The results and analysis have been added to Appendix C.3.7 of the revised manuscript.

W4. O(1)O(1) efficiency refers to safety-labeled modality coverage, not total data or compute cost. The reuse of the subspace is demonstrated on two related safety datasets.

A7. We agree that O(1)O(1) efficiency refers to safety-labeled modality coverage, and this is how the notation is defined in the paper. Section 3.1 (L168–L183) introduces the O(·) regimes as shorthand for the modality configuration coverage of safety training data. We would also like to clarify that WildGuardMix and Aegis are two distinct safety datasets that differ in origin and taxonomy. The Jaccard index between them is 0.71%, indicating negligible overlap, and the baseline’s cross-dataset AUC drops substantially. The transfer is therefore non-trivial.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.