acceptodds
Under review as a conference paper at ICLR 2027

Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

38 views

Abstract

Recent audio-video generators increasingly support joint conditioning on text, images, audio, and video. These capabilities also enable attacks that exploit cross-modal interactions or obscure harmful intent to bypass safeguards and induce harmful audio-video outputs. However, existing generation-safety benchmarks have not kept pace with these advances, providing limited coverage of multimodal input combinations and obscured attack intents. To address these gaps, we introduce Multi2AV-Safety, the first full-coverage red-team benchmark for multimodal-to-audio-video generation, comprising 11,024 attack instances across all 11 non-singleton T/I/A/V conditioning configurations, 4 attack-intent categories, and 5 harm categories. Our evaluation of recent state-of-the-art models, including four multimodal-conditioned audio-video generators and eight safety guards, reveals substantial vulnerabilities in both generation and safeguarding, with multimodal compositional risk and obscured attack-intent risk emerging as two complementary challenges. Guided by these findings, we introduce PerceptGuard, an omni-modal guard integrating compositional-risk and attack-intent supervision through structured risk perception learning. By jointly training rationale generation and safety classification, it learns shared risk representations that enable a safety head to make efficient predictions at inference without rationale decoding, while retaining the ability to generate explanations on demand. Across 34 safety benchmarks, PerceptGuard combines SOTA multimodal safety detection with highly competitive unimodal performance, strengthening input-side safeguards against multimodal attacks on omni models. In particular, it improves safeguarding against the above risks, achieving an overall recall of 86.06% on \bench and outperforming GuardReasoner-Omni by 14.56%.

Primary area:
datasets and benchmarks
Keywords:
Multimodal audio-video generation Safety, Safety Benchmarking, Omni-Modal Guardrails

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_C56qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a safety challenge in multimodal models, where harmful evidence may be obscured by benign context when multiple input modalities are combined. This can cause safety models to fail to detect harmful content. To address this problem, the authors introduce Multi2AV-Safety, a benchmark designed to evaluate model safety across different multimodal input combinations. The benchmark covers 11 input combinations, multiple attack types, and harm structures. The authors also conduct controlled experiments to examine how different modality combinations and additional benign context affect safety detection. Furthermore, the paper proposes PerceptGuard as a targeted solution. It introduces a Risk Relation Loss that explicitly constrains the risk relationships between individual modalities and their combined inputs. It helps the model better detect harmful content that emerges from multimodal combinations.

Critical strengths
  1. The proposed Multi2AV-Safety benchmark covers diverse input modality combinations, attack types, and harm structures, effectively addressing the limited multimodal input coverage of existing safety benchmarks. This benchmark has considerable practical value. Furthermore, the authors conduct controlled experiments to analyze how different modality combinations and the addition of benign context affect safety detection, providing direct empirical evidence for the identified safety challenges. The evaluation of 32,592 generated outputs by three independent experts, with high inter-annotator agreement, further strengthens the reliability of the experimental results.

  2. Beyond introducing the benchmark, the paper also proposes a targeted solution, PerceptGuard. In particular, the design of the Risk Relation Loss is thoughtful and well-motivated. By explicitly constraining the risk relationships between individual modalities and their combined inputs, it helps the model better recognize harm emerging from multimodal composition and prevents harmful evidence from being obscured by benign context. The subsequent ablation studies further demonstrate the effectiveness of PerceptGuard in improving safety detection.

Critical weaknesses
  1. Limited evaluation of false positive rates. The paper does not fully evaluate PerceptGuard's false positive rates across different multimodal input configurations. A practical safety guard should achieve both high detection rates and low false positive rates. Although the authors provide additional FPR experiments in the appendix, the evaluation is limited in scale and modality coverage, especially lacking four-modality benign inputs. Moreover, PerceptGuard still has a relatively high FPR of 42.52% on MMSafeAware Over-safety. I suggest adding more experiments to examine the trade-off between FPR and recall.

  2. Imbalanced benchmark distribution. The benchmark has an uneven data distribution, with few samples in some categories. This may affect the representativeness of the overall results. For example, bimodal inputs account for 80.3% of the benchmark, while four-modality inputs account for only 6.1%. The four-modality Temporal category contains only 25 samples, which may lead to less reliable statistical results. As a result, the overall metrics may not fully reflect model performance across complex multimodal combinations and attack types. Although the authors report results for individual groups, I suggest increasing the number of samples and making the sample sizes more consistent across categories to improve the statistical reliability of the results.

Questions for authors
  1. Have the authors evaluated the trade-off between recall and false positive rates under different decision thresholds? In particular, does PerceptGuard maintain its detection performance when the FPR is reduced?

  2. How does the uneven sample distribution across modality combinations and attack types affect the overall evaluation results? Would the main conclusions remain consistent if different categories were given similar weights?

5 comments
AuthorsOpenReview ↗

Response to Weakness1&Question1 [1/2]

We appreciate the reviewer’s positive assessment of the benchmark’s broad coverage, controlled experiments, and PerceptGuard’s risk-relation design. To address your concerns about false positives and sample imbalance, we added six targeted evaluations and analyses, reported in eight tables (R1–R8), together with ROC curves across eight tasks (Fig. 5) and revised the corresponding manuscript sections. Supporting results and brief analyses are provided in the replies below.

To expand benign coverage, we add 60 four-modality requests whose individual inputs and complete combinations are verified as benign. This brings the evaluation to 610 benign requests across all 11 multimodal input configurations. PerceptGuard achieves the lowest FPR among the compared models at 8.36% (Table R1).

Table R1. False-positive rates on 610 benign multimodal requests.

ModelFPR ↓
GuardReasoner-Omni-3B22.95%
Qwen3-Omni-30B9.84%
PerceptGuard (Ours)8.36%

To examine the recall–FPR trade-off, we select five request-level and three response-level tasks with relatively balanced benign and harmful classes from the 32-task evaluation suite. Keeping the trained model fixed, we vary the decision threshold from 0.1 to 0.9 and report recall, FPR, and F1 in Tables R2 and R3. We also provide the corresponding ROC curves in the revised main text (Section 4.2, p. 9, Fig. 5), with AUROC above 0.95 on all five request-level tasks.

Table R2. Request-level performance across decision thresholds.

Entries report recall / FPR / F1 (%).

ThresholdAEGIS 2.0XD-ViolenceWildGuardNemotron AudioUCF-Crime
0.196.34 / 28.23 / 87.3998.60 / 22.00 / 93.1188.86 / 7.20 / 89.8196.34 / 29.13 / 87.0896.43 / 20.00 / 88.52
0.295.19 / 23.62 / 88.3898.40 / 16.00 / 94.6287.93 / 6.46 / 89.7294.71 / 22.16 / 88.6594.29 / 10.67 / 91.67
0.394.71 / 20.70 / 89.1798.00 / 13.00 / 95.2486.47 / 6.03 / 89.1393.94 / 18.45 / 89.5892.86 / 10.00 / 91.23
0.493.74 / 19.12 / 89.2397.80 / 9.00 / 96.2685.01 / 5.71 / 88.4792.49 / 15.64 / 89.8691.43 / 8.67 / 91.10
0.5 (Default)92.88 / 16.42 / 89.7796.80 / 7.00 / 96.3283.42 / 5.71 / 87.5490.28 / 12.71 / 89.7691.43 / 8.00 / 91.43
0.691.72 / 13.72 / 90.1696.60 / 5.67 / 96.6082.36 / 5.50 / 87.0488.16 / 10.57 / 89.4189.29 / 6.67 / 90.91
0.790.09 / 11.25 / 90.2293.80 / 5.00 / 95.3380.50 / 5.40 / 85.9885.37 / 8.77 / 88.5286.43 / 4.67 / 90.30
0.887.39 / 9.34 / 89.4691.20 / 3.67 / 94.3178.78 / 4.97 / 85.1681.04 / 6.41 / 86.8981.43 / 3.33 / 88.03
0.983.06 / 7.20 / 87.7985.00 / 2.33 / 91.2075.20 / 4.34 / 83.2675.17 / 4.50 / 83.9868.57 / 2.00 / 80.33

Table R3. Response-level performance across decision thresholds.

Entries report recall / FPR / F1 (%).

ThresholdAEGIS 2.0Nemotron AudioHarmBench
0.197.72 / 34.61 / 83.3396.95 / 28.40 / 85.3697.44 / 27.36 / 84.58
0.296.95 / 24.34 / 87.0295.94 / 19.33 / 88.6395.97 / 22.19 / 86.18
0.395.69 / 20.29 / 88.0893.91 / 15.75 / 89.1693.04 / 19.15 / 86.10
0.493.40 / 16.95 / 88.3691.37 / 12.17 / 89.4492.67 / 17.93 / 86.50
0.5 (Default)91.12 / 13.13 / 88.8689.85 / 9.31 / 89.9690.84 / 16.41 / 86.26
0.688.07 / 10.50 / 88.4186.04 / 7.88 / 88.5187.55 / 14.29 / 85.51
0.786.04 / 8.35 / 88.2883.50 / 5.25 / 88.3286.08 / 13.68 / 84.99
0.881.73 / 5.73 / 87.0378.68 / 4.53 / 85.7581.68 / 11.25 / 83.68
0.974.11 / 3.58 / 83.3169.04 / 2.63 / 80.3571.79 / 9.12 / 78.56

PerceptGuard retains strong detection as the false-positive rate is reduced. At a threshold of 0.7, recall exceeds 80% on all eight tasks, with XD-Violence achieving 93.80% recall at 5.00% FPR. At a threshold of 0.9, AEGIS 2.0 request detection retains 83.06% recall, while FPR falls from 16.42% to 7.20%, more than halving the default FPR with a 1.98-percentage-point decrease in F1.

AuthorsOpenReview ↗

Response to Weakness1&Question1 [2/2]

MMSafeAware Over-safety further tests recognition of benign cross-modal context, where an individual modality may appear unsafe while the complete image–text request is benign. To contextualize performance on this challenging subset, we additionally evaluate three general-purpose multimodal models on the same 428 pairs.

Table R4. False positives on the 428 benign image–text pairs from MMSafeAware Over-safety.

ModelPredicted SafePredicted HarmfulFPR ↓
Claude-Opus-521021850.93%
gemini-4-argon25117741.36%
GPT-6-Sol19223655.14%
Shieldstral-1.0-3B18824056.07%
GR-Omni14328566.59%
SingGuard-4B13629268.22%
PerceptGuard (Ours)24618242.52%

PerceptGuard achieves 42.52% FPR, below the evaluated Claude and GPT models and close to Gemini’s 41.36%. It also outperforms all evaluated specialized guards, reducing FPR by 13.55 percentage points relative to the strongest specialized baseline, Shieldstral-1.0-3B. These comparisons highlight the difficulty of recognizing benign cross-modal context and PerceptGuard’s competitive performance on this subset.

We have incorporated the expanded benign evaluation and recall–FPR analysis, including ROC curves, into Section 4.2 (p. 9, Fig. 5). Detailed benign-input comparisons are provided in Appendix E.5 (p. 38, Tables E.5–E.6), with threshold-specific results in Appendix E.10 (p. 41, Tables E.13–E.14).

AuthorsOpenReview ↗

Response to Weakness2

To our knowledge, Multi2AV-Safety is the first safety benchmark for audio–video generation covering all 11 multimodal input combinations. Within each three- and four-modality configuration, we allocate 25 requests per feasible combination of attack intent and harmful-evidence assignment, balanced across the five harm categories. Direct attacks admit 15 such assignments with four modalities versus three with two, requiring five times as many requests for equal per-assignment coverage, alongside coordinated media generation and verification of individual and joint safety. In the future work, we will expand these subsets with additional independent, expert-validated samples to improve coverage and balance across modality orders.

To assess the effect of modality imbalance, we construct a 2,025-request evaluation set with 675 requests per modality order. We retain all 675 four-modality requests and select 675 bimodal and 675 trimodal requests through stratified sampling within input configurations, covering harm structures, attack intents, and harm categories. Aggregate scores assign equal weight to each evaluated modality order, preventing the larger bimodal group from dominating the comparison.

Table R5. Performance by modality order on the balanced evaluation set and comparison with the original full benchmark (%).

ModelMetricBimodalTrimodalFour-modalOrder-balancedOriginal Full
MiniMax-H3ASR ↓62.4875.5679.5672.5373.20
Seedance2ASR ↓25.6340.1536.0033.9334.02
LTX-2ASR ↓94.2291.5689.3391.7091.65
daVinci-MagiHumanASR ↓78.9878.46—78.7278.76
Qwen3-OmniRecall ↑65.3366.2252.5961.3861.35
GuardReasoner-OmniRecall ↑81.1972.1561.1971.5171.50
PerceptGuardRecall ↑74.8187.7095.8586.1286.06

Table R6. Performance by attack intent on the balanced evaluation set.

Parentheses report the original full-benchmark results (%).

ModelMetricDirectJailbreakTemporalAdversarial
MiniMax-H3ASR ↓83.30 (82.06)64.97 (62.19)43.66 (59.52)50.78 (52.59)
Seedance2ASR ↓27.73 (31.18)41.83 (34.75)49.33 (42.07)38.67 (44.72)
LTX-2ASR ↓95.64 (96.22)87.83 (83.47)74.67 (88.00)88.00 (88.00)
daVinci-MagiHumanASR ↓92.25 (88.49)61.93 (64.27)72.73 (70.77)90.00 (78.67)
Qwen3-OmniRecall ↑74.93 (73.82)39.33 (39.28)54.67 (48.83)54.67 (52.31)
GuardReasoner-OmniRecall ↑82.49 (85.93)59.33 (51.00)60.00 (52.14)52.89 (49.95)
PerceptGuardRecall ↑83.64 (85.39)87.00 (83.12)88.00 (91.72)95.56 (94.87)

The balanced evaluation preserves the main patterns underlying our four findings. Generators remain vulnerable under complex multimodal conditioning, with four-modality ASRs ranging from 36.00% to 89.33%. Existing safeguards also continue to struggle, with four-modality recall of 52.59% for Qwen3-Omni and 61.19% for GuardReasoner-Omni. Temporal attacks still elicit harmful outputs from individually benign inputs and evade both baseline guards, supporting the compositional-risk finding. Obscured intent likewise remains challenging. On Seedance2, Jailbreak, Temporal, and Adversarial attacks all achieve higher ASRs than Direct attacks, while both baseline guards detect these categories less reliably than Direct attacks.

PerceptGuard retains the highest order-balanced recall at 86.12%, closely matching its 86.06% full-benchmark result and exceeding GuardReasoner-Omni’s 71.51%. It also leads across all four attack categories in the balanced evaluation. These results support the consistency of the main conclusions under equal representation of modality orders.

We have summarized the balanced-subset findings in Section 4.2 (p. 10), clarified the coverage limitations and planned expansion in Section 5 (p. 10), and added the detailed evaluation in Appendix G.1 (p. 46, Tables G.1–G.2).

Reviewer_LGovOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the safety of multimodal-conditioned audio-video generation and introduces Multi2AV-Safety, a benchmark comprising 11,024 harmful requests across all 11 multimodal input combinations of text, image, audio, and video, covering four attack types and three harmful evidence structures. The authors evaluate four audio-video generation models and eight input-level safety guardrail baselines, using human annotations of the actual generated outputs to analyze the safety challenges arising from cross-modal composition, benign context dilution, and concealed attack intent.

To address these challenges, the authors propose PerceptGuard, a guardrail model built on Qwen2.5-Omni-3B that jointly optimizes safety classification, relational risk constraints between unimodal and joint multimodal inputs, explanation generation, and attack-type supervision. During inference, the model outputs safety decisions directly through its classification head without requiring explanation decoding. On Multi2AV-Safety, PerceptGuard achieves a harmful request detection rate of 86.06%, outperforming GuardReasoner-Omni by 14.56 percentage points.

Critical strengths

1.Practical relevance and comprehensive evaluation coverage: Generation safety conditioned on audio, reference video, and three- or four-modality inputs warrants systematic investigation. Beyond categorizing attack types, the paper explicitly annotates the distribution of harmful evidence across modalities, which helps uncover specific failure modes that would otherwise be obscured by aggregate metrics.

2.Substantial human evaluation of actual generated outputs: Rather than treating mere prompt compliance or topically related outputs as a successful attack, the authors require the generated content to actually manifest the core harmful semantics of the target scenario. Conducting three-annotator independent labeling across 32,592 generated outputs—along with reporting inter-annotator agreement—provides solid empirical grounding for the evaluation of existing models. The substantial effort undertaken by the authors is commendable.

3.Targeted and informative diagnostic experiments: The paired experiments that hold harmful inputs fixed while progressively adding benign modalities provide much stronger support for the observation that "benign context can dilute risk recognition" than simply comparing average scores across different modality counts. In addition, evaluating benign samples to assess over-refusal effectively complements the reporting of harmful request detection rates alone.

4.Well-motivated relational risk supervision: Explicitly encoding the risk relationship between individual unimodal inputs and the complete multimodal request as a training constraint is a distinctive design choice in the proposed method. Because the classification baseline also retains both unimodal and holistic labels, the ablation results for the relational loss provide credible evidence of its effectiveness beyond merely incorporating additional label supervision.

Critical weaknesses
  1. Lack of a validated, reusable scoring protocol for newly generated outputs: While human annotation is suitable for establishing reference labels on existing generations, evaluating new generative models yields fresh audio-video outputs to which pre-existing output labels cannot be directly applied. Although the authors experiment with three automated judges and majority voting, Table D.2 primarily reports inter-model agreement and predicted harmful proportions without reporting precision, recall, F1, or systematic error analyses against the human ground truth. Consequently, neither a reliable automated scoring pipeline nor the cross-team reproducibility of the human annotation protocol has been established. This limits both the affordability and score comparability of the benchmark for continuously evaluating future generative models (though this issue primarily affects generative model evaluation rather than input-level guardrail evaluation on fixed requests).

  2. Conflation of safety defenses with comprehension and generation failures in the ASR metric: The current Attack Success Rate (ASR) definition requires the output to be both harmful and aligned with the target scenario. Under this criterion, safety refusals, safe rewrites, modality neglect, failure to grasp cross-modal relationships, generation failures, and irrelevant outputs all manifest identically as failed attacks. Moreover, outputs that are harmful yet deviate from the specified target scenario are not adequately captured by this metric. Thus, a lower ASR does not necessarily indicate stronger safety defenses. Inconsistent evaluation coverage across different generative models further restricts direct comparisons of aggregate ASR scores.

  3. Insufficient isolation of individual training objectives and unclear boundaries of methodological novelty: The proposed method primarily combines multiple supervision objectives on an existing backbone, with the relational risk constraint serving as the most concrete architectural addition. The current cumulative ablations are insufficient to isolate the independent contributions and interaction effects of each objective; furthermore, because the explanation loss jointly supervises explanation text, safety judgments, and formatting tokens, it is unclear whether the observed gains truly stem from the explanation content itself. The lack of leave-one-out ablations from the full model, a judgment-only generation baseline, and comparisons with alternative relational objectives leaves the boundaries of the methodological contribution indistinct.

  4. Limited fine-grained supervision and generalization evidence: Compositional annotations are mainly constructed from text paired with a single media modality, which does not fully demonstrate how the model generalizes to complex three- and four-modality interactions. In addition, attack-type labels are assigned based on data sources and construction pipelines, and training samples for adversarial and temporal attacks are relatively scarce, making it necessary to rule out source- or template-based shortcut learning. Furthermore, Table A.3 omits the annotation pattern where "individual components appear harmful in isolation, yet the holistic request is benign due to context." Given that the model still suffers from a relatively high false-positive rate on hard benign samples, its capability for contextual disambiguation and risk de-escalation remains to be fully verified.

Questions for authors
  1. Leveraging the existing human labels as ground truth, could the authors report the precision, recall, and F1 scores of the three automated judges and their majority vote against human annotations, broken down by harm category and joint audio-visual harm? Furthermore, will the complete scoring pipeline be provided along with stability analyses across prompt variations and repeated API/inference calls?

  2. Could the authors separately annotate output harmfulness and target adherence, explicitly distinguishing among safety refusals, safe rewrites, irrelevant generations, cross-modal comprehension errors, and low-quality outputs? Additionally, could baseline generation capabilities be evaluated on matched benign scenarios, and could different generative models be compared on the shared subset of requests they all support?

  3. Could the authors supplement the study with leave-one-out ablations starting from the full model, and directly compare "explanation + judgment supervision" against "judgment-only supervision"? Please also clarify whether these comparisons use identical training data, training steps, and comparable compute budgets.

  4. What is the rationale for choosing equal loss weights, a margin of 0.2, and a tolerance of 0.05? Are the results stable across varying loss weights, hyperparameter configurations, and random seeds? Moreover, how do the authors explain the divergent effects of relational and explanation supervision across request-level versus response-level evaluation tasks?

I appreciate the comprehensive and thorough research methodology presented in this paper; it represents a well-executed benchmark study, and the authors have put forth a commendable amount of effort. However, several gaps, ambiguities, and limitations remain as outlined above, which could undermine the potential impact and scientific significance of this work and thus warrant careful resolution. If the authors can satisfactorily address my questions and concerns during the rebuttal, I would be very willing to raise my score.

10 comments
AuthorsOpenReview ↗

Response to Weakness1&Question1 [1/2]

We appreciate the reviewer’s positive assessment of the benchmark’s comprehensive coverage, rigorous evaluation, and PerceptGuard’s relational risk supervision. To address your concerns about evaluation reliability, ASR interpretation, component contributions, generalization, and training robustness, we added eleven targeted evaluations and analyses, reported in twelve tables (R1–R12) and revised the corresponding manuscript sections. Supporting results and brief analyses are provided in the replies below.

Prior work shows that consistent LLM judgments can still be incorrect [1], omni-modal models can incorrectly associate audio and visual evidence [2], and safety alignment may not transfer reliably across modalities [3]. These findings motivate task-specific validation of automated judges against expert annotations under explicit harm criteria.

For new generators, we apply the same annotation protocol to newly generated outputs. Following Appendix D, three trained safety experts independently review each clip against its target scenario and harm category, examining speech, non-speech sounds, visible actions, text, symbols, and their cross-modal and temporal relationships. A harmful label requires observable content that realizes the target scenario, excluding thematic similarity or assumed events beyond the clip. Final labels follow majority vote, and ASR is computed over evaluated requests, with refusals counted as failed attacks. Across 32,592 outputs, the protocol achieves 96.26% unanimous agreement and Fleiss’ �0��0�, demonstrating strong consistency among annotators trained on the same guidelines.

Using these human labels as ground truth, we report precision, recall, and F1 for the three automated judges and their majority vote on joint audio–visual harm (Table R1) and across the five harm categories (Table R2).

Table R1. Automated judge performance on joint audio–visual harm against human reference labels (%).

Evaluation MethodPrecisionRecallF1
Qwen3.5-Omni-Flash89.5169.2678.09
Gemini-3.8-Flash94.6948.2563.93
Doubao-Seed-2.0-Lite93.0360.3573.21
Majority Vote93.9660.9373.92

Table R2. Automated judge performance by harm category.
Entries report precision / recall / F1 (%).

Harm CategoryQwen3.5-Omni-FlashGemini-3.8-FlashDoubao-Seed-2.0-LiteMajority Vote
Hate & Discrimination89.28 / 69.84 / 78.3794.85 / 53.13 / 68.1192.58 / 58.80 / 71.9293.86 / 61.69 / 74.45
Politically Sensitive Content90.47 / 62.08 / 73.6495.94 / 32.25 / 48.2793.77 / 49.81 / 65.0695.14 / 48.61 / 64.35
Violence & Gore90.44 / 77.75 / 83.6194.94 / 52.67 / 67.7593.43 / 74.01 / 82.6094.04 / 72.37 / 81.79
Sexual Content & Nudity91.78 / 61.37 / 73.5696.58 / 53.54 / 68.8995.56 / 55.35 / 70.1096.36 / 56.36 / 71.12
Illegal Activities85.92 / 73.78 / 79.3991.38 / 49.11 / 63.8990.04 / 61.13 / 72.8290.97 / 63.72 / 74.94

The judges achieve high precision but substantially lower recall. For joint audio–visual harm, majority voting still misses 39.07% of human-labeled harmful outputs, with recall falling to 48.61% for politically sensitive content. Qualitative inspection identifies missed harm conveyed through audio–visual interactions and overlooked text or graphic symbols within video frames. These errors highlight the need to validate cross-modal understanding and adherence to shared harm criteria against human labels, beyond measuring inter-judge agreement.

References

[1] Vineet Kumar, Darshita Rathore, and Anindya Moitra. All Verdicts are Not Equal: Rethinking LLM Judge Reliability. AACL-IJCNLP 2026.

[2] Zixuan Dong, Jiafu Tang, Zhide Lei, Zhe Cao, Zijie Zhang, Yanghai Wang, Shihao Li, Xiaodong Wang, Baoyun Peng, and Jiaheng Liu. OmniHalluc-L: Counterfactual Benchmarking and Modality-Perturbation Calibration for Long-Form Omni Hallucination. EMNLP 2026.

[3] Yooju Shin, Chentao Ye, Weitong Ruan, and Rahul Gupta. UNISAF-Bench: Intention-Grounded Safety Evaluation for Unified Multimodal Generation Models. EMNLP 2026.

AuthorsOpenReview ↗

Response to Weakness1&Question1 [2/2]

To assess scoring stability, we evaluate Gemini-3.8-Flash in two tests, each using 1,200 audio–video outputs. We compare three semantically equivalent prompt formulations and separately make three repeated calls with the original prompt. Table R3 reports three-way prediction agreement and performance against human labels.

Table R3. Stability of Gemini-3.8-Flash under prompt paraphrases and repeated calls on 1,200 outputs (%).

Stability TestPrompt / RunPrecisionRecallF1Three-Way Agreement
Prompt paraphrasesOriginal prompt97.1035.0051.4597.08 (1,165/1,200)
Prompt paraphrasesEquivalent paraphrase 197.4034.8851.37—
Prompt paraphrasesEquivalent paraphrase 297.4135.0051.50—
Repeated callsRun 197.4435.8152.3797.58 (1,171/1,200)
Repeated callsRun 297.4235.5752.11—
Repeated callsRun 397.4335.6952.24—

Gemini-3.8-Flash achieves 97.08% agreement across prompt paraphrases and 97.58% across repeated calls, with F1 varying by only 0.13 and 0.26 percentage points, respectively. Recall nevertheless remains approximately 35%, showing that stable predictions can systematically miss harmful outputs. These results distinguish scoring stability from accuracy and support expert verification for reference evaluations.

To support consistent evaluation of new generations across research teams, we will release the complete scoring package, including harm definitions, annotation guidelines, worked examples and forms, voting and ASR aggregation scripts, and automated-judge prompts with preprocessing and inference configurations. We will also document model versions and preserve per-output judgments for auditing.

We have added a reference to the human annotation protocol in Section 3.2 (p. 5) and incorporated the reusable scoring procedure, automated-judge validation, and stability analyses into Appendix D (pp. 33–35, Tables D.3–D.5).

AuthorsOpenReview ↗

Response to Weakness2&Question2

We supplement the original human evaluation with fine-grained manual annotations of 7,956 generated outputs that did not meet the target-aligned attack-success criterion, and separately report 3,804 safety refusals. Table R4 summarizes all 11,760 unsuccessful requests, distinguishing safety responses, comprehension and generation failures, and harmful but off-target outputs to clarify the relationship between output harmfulness and target adherence.

Table R4. Fine-grained outcomes of unsuccessful attacks.

Outcome CategoryMiniMax-H3Seedance2LTX-2daVinci-MagiHumanOverall
Safety refusal0 (0.00%)3,804 (52.30%)0 (0.00%)0 (0.00%)3,804 (32.35%)
Safe rewrite862 (34.49%)1,883 (25.89%)37 (4.02%)39 (3.66%)2,821 (23.99%)
Irrelevant generation40 (1.60%)931 (12.80%)15 (1.63%)13 (1.22%)999 (8.49%)
Cross-modal comprehension error907 (36.29%)329 (4.52%)256 (27.83%)372 (34.86%)1,864 (15.85%)
Low-quality output410 (16.41%)31 (0.43%)493 (53.59%)572 (53.61%)1,506 (12.81%)
Harmful but off-target output280 (11.20%)296 (4.07%)119 (12.93%)71 (6.65%)766 (6.51%)
Total2,499 (100%)7,274 (100%)920 (100%)1,067 (100%)11,760 (100%)

Refusals and safe rewrites account for 78.18% of Seedance2’s unsuccessful requests, while comprehension errors and low-quality outputs account for the majority for the other three generators. The additional annotations also identify 766 harmful but off-target outputs, explicitly distinguishing broader output harmfulness from target-aligned attack success.

To ensure identical input coverage, we compare all four generators on the same 3,624 requests.

Table R5. Generator ASR on shared requests and the original evaluation sets (%).

GeneratorT+I (1,749)I+A (1,400)T+I+A (475)Shared Overall (3,624)Original Overall
LTX-287.1999.2188.2191.9791.65
Seedance236.4222.0048.6332.4534.02
MiniMax-H378.1668.3673.2673.7373.20
daVinci-MagiHuman81.4885.3674.9582.1278.76

Substantial generation vulnerabilities and the original ASR ordering persist under identical input coverage, with shared-set ASRs ranging from 32.45% to 91.97%. This comparison confirms the consistency of the observed patterns across matched evaluation inputs.

To assess generation capability under benign conditioning, we evaluate all four generators on the same 175 benign requests, comprising 75 T+I, 75 I+A, and 25 T+I+A inputs. Following CAST-Bench [1], we report five metrics covering perceptual quality and temporal consistency alongside output safety.

Table R6. Generation quality and output safety on 175 shared benign requests.

GeneratorV-Aes ↑A-PQ ↑SbC ↑BgC ↑Mot.P ↑Safe Outputs ↑
LTX-20.8066.3810.8670.9480.901100%
daVinci-MagiHuman0.7895.9170.8410.9210.839100%
Seedance20.9036.8150.9040.9690.981100%
MiniMax-H30.8976.7610.8910.9740.841100%

V-Aes denotes video aesthetics; A-PQ, audio production quality; SbC, subject consistency; BgC, background consistency; and Mot.P, motion plausibility.

All four generators achieve 100% safe outputs on the evaluated benign scenarios. The accompanying quality metrics provide a generation-capability reference for interpreting the harmful-request results, complementing the fine-grained outcome analysis and shared-request comparison.

We have clarified the ASR definition and annotation criteria in Section 3.2 (p. 5) and Appendix D (p. 34), and added the fine-grained outcome analysis, shared-request comparison, and benign-generation evaluation to Appendix C (pp. 31–32, Tables C.5–C.7).

Reference

[1] Yucong Liu, Qi Tian, Shuyuan Tu, Kaihang Pan, Weijie Kong, and Chao Pan. CAST-Bench: Tracing Every Sound to Its Source in Multi-Shot Audio-Video Generation. arXiv preprint arXiv:2611.05422, 2026.

Reviewer_fQLeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Multi2AV-Safety, a benchmark of 11,024 harmful requests for audio-video generators conditioned on text, images, audio, and video. It evaluates four generators and eight input-side guards, then proposes PerceptGuard, which learns from individual and joint risk, rationales, and attack intent. PerceptGuard reaches 86.06% recall on the new benchmark, compared with 71.50% for GR-Omni.

Critical strengths
  1. The benchmark asks a useful question: does harm appear in one input, or only when inputs are combined? Breaking down the results by modality, harm structure, and attack intent reveals failures that an overall attack success rate would hide.

  2. The output evaluation is thorough. Three experts independently rated 32,592 generated outputs, with Fleiss’ κ of 0.933. The paper also shows why human review matters: three automated judges unanimously agreed on only 62.29% of outputs.

  3. The ablation gives a clear account of PerceptGuard’s gains. Starting from the same base model, the authors add risk-relation, rationale, and intent supervision; benchmark recall rises from 71.43% to 76.24%, 82.18%, and 86.06%, respectively.

Critical weaknesses
  1. The two risk dimensions are not well disentangled. All 725 Temporal attacks are Composed Harm, and the other 725 Composed Harm examples are Jailbreak attacks. There are no Direct or Adversarial examples in that group. It is therefore hard to tell whether a result reflects the harm structure or the attack family.

  2. The main benchmark contains only harmful requests, so its headline recall does not show the cost of false refusals. The 550 benign controls are derived by removing harmful components from benchmark requests. On a separate set of 428 challenging benign image–text pairs, PerceptGuard still has a 42.52% false-positive rate. A recall–false-positive curve on a more representative mix of requests would help.

  3. Generalization to new attack families remains unclear. The authors separate training and evaluation instances, scenarios, and exact templates, which is good. However, both still draw on methods such as SceneSplit, PGJ, DiffZOO, and AdvWave. Testing on a held-out attack family would make the gains on obscured intent more convincing.

  4. PerceptGuard is never placed in front of the four generators, so the paper does not measure how many harmful outputs it actually prevents. A comparison with ConceptGuard on the shared text–image setting would also clarify what PerceptGuard’s detection method adds over earlier cross-modal work.

Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Weakness1

We appreciate the reviewer’s positive assessment of the benchmark’s diagnostic value, the thoroughness of our evaluation, and the ablations supporting PerceptGuard’s design. To address your four concerns, we added seven targeted evaluations and analyses, reported in seven tables (R1–R7), together with ROC curves across eight tasks (Fig. 5) and revised the corresponding manuscript sections. Supporting results and brief analyses are provided in the replies below.

To evaluate attack-family differences within the same harm structure, we compare the Temporal and Jailbreak subsets of Composed Harm, holding �0��0� fixed.

Table R1. Generator ASR and input-side guard recall (%) by attack intent within Composed Harm.
NN denotes the benchmark subset size.

Attack IntentNLTX-2 ASR ↓Seedance2 ASR ↓MiniMax-H3 ASR ↓daVinci ASR ↓SingGuard-4B Recall ↑Shieldstral-1.0-3B Recall ↑Qwen3-Omni Recall ↑GR-Omni Recall ↑PerceptGuard Recall ↑
Temporal72588.0042.0759.5270.7736.0019.5948.8352.1491.72
Jailbreak72579.3115.5966.2464.6233.7926.0732.2843.4554.90

From Table R1, temporal attacks yield higher ASRs on three of the four generators, while both families remain challenging for baseline guards. PerceptGuard improves recall over the strongest baseline, GR-Omni, by 39.58 and 11.45 percentage points on Temporal and Jailbreak attacks, respectively. These results highlight vulnerabilities in both generation and safeguarding, alongside substantial attack-family variation within the same harm structure.

The absence of Direct and Adversarial attacks from this subset follows from our construction protocol. Harm structure describes how harmful evidence is distributed across modalities, whereas attack intent describes how the harmful objective is expressed. Composed Harm requires all modality inputs to be benign in isolation while their combination conveys harm (�2��2�). Our Temporal construction satisfies this condition by splitting a harmful event into individually benign, ordered fragments. Compositional Jailbreak variants similarly convey harm through relationships among individually benign inputs. By contrast, our Direct construction retains explicit harmful content, while our Adversarial construction optimizes harmful text or perturbs harmful speech while preserving the target semantics. Both constructions retain at least one individually harmful input (�3��3�), placing these samples in Diluted or Full Harm. This construction-dependent relationship explains the observed subset composition.

We have clarified the definitions and their construction-dependent relationship in Section 3.1 (p. 4), added the within-Composed-Harm analysis in Section 3.3 (p. 6), and reported the detailed results in Appendix C (p. 31, Table C.3).

AuthorsOpenReview ↗

Response to Weakness2

PerceptGuard is trained on OmniRisk-Train, while Multi2AV-Safety and its benign controls are held out from training at both the instance and scenario levels. To broaden benign coverage, we add 60 four-modality requests whose inputs are benign both individually and jointly, expanding the evaluation to 610 benign requests across all 11 multimodal input configurations. PerceptGuard achieves the lowest FPR among the compared models at 8.36% (Table R2).

Table R2. False-positive rates on 610 benign multimodal requests.

ModelFPR ↓
GuardReasoner-Omni-3B22.95%
Qwen3-Omni-30B9.84%
PerceptGuard (Ours)8.36%

Beyond these controls, Table 4 compares PerceptGuard with existing guards across 32 request- and response-level safety tasks. To directly assess the recall–FPR trade-off, we vary the decision threshold and plot ROC curves for five request-level and three response-level tasks with relatively balanced benign and harmful classes (Section 4.2, p. 9, Fig. 5). PerceptGuard achieves AUROC above 0.95 on all five request-level tasks, with AUROCs of 0.9299–0.9601 on the response-level tasks (Table R3). These curves complement the fixed-threshold comparisons in Table 4 by explicitly showing harmful-content recall at different false-positive rates.

Table R3. PerceptGuard AUROC on mixed benign and harmful evaluation sets.

TaskBenchmarkSafeHarmfulAUROC ↑
RequestAEGIS 2.08891,0390.9537
RequestXD-Violence3005000.9826
RequestWildGuard9457540.9525
RequestNemotron Audio8891,0390.9529
RequestUCF-Crime1501400.9675
ResponseAEGIS 2.04193940.9580
ResponseNemotron Audio4193940.9601
ResponseHarmBench3292730.9299

The 428-pair MMSafeAware Over-safety subset further tests contextual disambiguation using image–text pairs in which an individual modality may appear unsafe even though the complete request is benign. To contextualize performance on this challenging subset, we additionally evaluate three general-purpose multimodal models on the same requests (Table R4). PerceptGuard achieves an FPR of 42.52%, below Claude (50.93%) and GPT (55.14%) and close to Gemini (41.36%). It also outperforms all evaluated specialized guards, reducing FPR by 13.55 percentage points relative to the strongest baseline, Shieldstral-1.0-3B. These results highlight the shared challenge of recognizing benign cross-modal context and PerceptGuard’s competitive performance on this subset.

Table R4. False positives on the 428 benign image–text pairs from MMSafeAware Over-safety.

ModelPredicted SafePredicted HarmfulFPR ↓
Claude-Opus-521021850.93%
gemini-4-argon25117741.36%
GPT-6-Sol19223655.14%
Shieldstral-1.0-3B18824056.07%
GR-Omni14328566.59%
SingGuard-4B13629268.22%
PerceptGuard (Ours)24618242.52%

We have added the expanded benign evaluation, additional model comparisons, and ROC analysis to Section 4.2 (p. 9, Fig. 5), with detailed results in Appendix E.5 (p. 38, Tables E.5–E.6) and Appendix E.10 (p. 41, Tables E.13–E.14).

AuthorsOpenReview ↗

Response to Weakness3

To assess whether the gains depend on supplementary attack methods used during training, we cumulatively remove five reproduced methods while retaining the GuardReasoner-Omni training subset. A1 removes AdvWave, and A2–A5 additionally remove MultiAudioJail, DiffZOO, PGJ, and SceneSplit, respectively. All variants retain the proposed training objectives and are evaluated on the complete Multi2AV-Safety benchmark.

Table R5. Generalization under cumulative exclusion of supplementary attack methods from training.
Entries report detection counts and recall (%). Column headers indicate evaluation subset sizes.

Model / Training VariantDirect (6,475)Jailbreak (2,849)Adversarial (975)Temporal (725)Overall (11,024)
PerceptGuard (Full)5,529 (85.39%)2,368 (83.12%)925 (94.87%)665 (91.72%)9,487 (86.06%)
A1: Remove AdvWave5,423 (83.75%)2,245 (78.80%)902 (92.51%)654 (90.21%)9,224 (83.67%)
A2: Additionally remove MultiAudioJail5,467 (84.43%)2,328 (81.71%)896 (91.90%)652 (89.93%)9,343 (84.75%)
A3: Additionally remove DiffZOO5,413 (83.60%)2,405 (84.42%)884 (90.67%)668 (92.14%)9,370 (85.00%)
A4: Additionally remove PGJ5,595 (86.41%)2,316 (81.29%)898 (92.10%)713 (98.34%)9,522 (86.38%)
A5: Additionally remove SceneSplit5,701 (88.05%)2,276 (79.89%)899 (92.21%)586 (80.83%)9,462 (85.83%)
Qwen3-Omni-30B4,780 (73.82%)1,119 (39.28%)510 (52.31%)354 (48.83%)6,763 (61.35%)
GuardReasoner-Omni-3B5,564 (85.93%)1,453 (51.00%)487 (49.95%)378 (52.14%)7,882 (71.50%)

After excluding all five supplementary methods, A5 outperforms both baseline guards across all four attack categories, achieving 85.83% overall recall versus 71.50% for GuardReasoner-Omni (Table R5). Without SceneSplit supplementation, it retains 80.83% Temporal recall, exceeding the strongest baseline by 28.69 percentage points. Relative to A5, the full model further improves recall across all three obscured-intent categories, raising Temporal recall to 91.72%.

These results support the transferability of PerceptGuard’s cross-modal risk and obscured-intent recognition, as well as the effectiveness of its attack-intent risk perception module. We also contribute OmniRisk-Train to support omni-modal safety research, providing broader coverage of recent obscured-intent attack methods.

We have incorporated these findings into Section 4.3 (p. 10), with the experimental setup and detailed results in Appendix F.5 (pp. 44–45, Table F.8).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for recognizing Multi2AV-Safety’s practical value as the first safety benchmark for multimodal-conditioned audio–video generation, its coverage of all 11 input configurations and evaluation by experts and three SOTA omni-modal judges, as well as PerceptGuard’s well-motivated, interpretable design.

We have addressed each concern and incorporated the corresponding analyses and clarifications into the revised manuscript. Reviewer LGov stated that the additional experiments met their expectations and indicated that they raise their score, highlighting the reproducibility and stability of our detection process and describing our interpretability arguments as “comparable to human-level reasoning.” Reviewer C56q also responded positively to our replies and maintained their positive rating.

To address Reviewer fQLe’s questions on risk characterization, benign-request handling, attack generalization, and practical protection, we clarified the two risk annotations and compared attack families within Composed Harm. Expanded benign evaluations and ROC analyses demonstrate that PerceptGuard maintains high harmful-content recall at reduced false-positive rates, complemented by additional comparisons on challenging benign inputs. Excluding all five supplementary attack methods from training preserves 85.83% overall recall, exceeding the strongest evaluated baseline by 14.33 percentage points and outperforming both baseline guards across all four attack categories. These results support strong generalization beyond these attack-specific training supplements. We also evaluated downstream protection and clarified the unavailability of implementation and evaluation resources for ConceptGuard. Under identical text–image inputs, PerceptGuard outperforms both state-of-the-art bimodal guards, Shieldstral-1.0-3B and Nemotron-Guard-3.5, with recall gains of 19.44–64.00 percentage points over the stronger baseline across all four attack-intent categories. These matched comparisons demonstrate the contribution of its detection approach under shared modality access. (Tables R1–R7.)

In response to Reviewer LGov’s requests to validate the evaluation protocol and clarify the methodological contributions, we documented the reusable scoring procedure and reported automated-judge precision, recall, and F1 against human labels, with breakdowns by harm category and for joint audio–visual harm. We also analyzed error patterns and tested Gemini’s stability under prompt variations and repeated calls. Fine-grained human annotations of 7,956 generated outputs, together with separate refusal counts, distinguish safety responses, comprehension and generation failures, and harmful but off-target outputs. Shared-request comparisons and benign-generation quality assessments further support the interpretation of ASR. Under matched training settings, we evaluated all eight objective combinations, including leave-one-out ablations, and added a judgment-only control. These comparisons clarify component contributions and interactions, with rationale content improving Multi2AV-Safety recall by 3.98 percentage points over judgment-only supervision. We further examined transfer to three- and four-modality inputs, dependence on attack sources and templates through method exclusions, and contextual disambiguation on challenging benign inputs. Finally, we explained the parameter choices, assessed sensitivity to loss weights, relation parameters, and random seeds, and clarified how shared supervision affects request and response judgments differently. (Tables R1–R12.)

Following Reviewer C56q’s suggestions to broaden evaluation coverage and assess the consistency of the findings, we added four-modality benign requests, expanding the benign evaluation to 610 requests across all 11 input configurations. PerceptGuard achieves 8.36% FPR on this expanded set. ROC and threshold analyses further demonstrate strong harmful-content detection as false positives are reduced. At a threshold of 0.7, recall remains above 80% across all eight tasks, including 93.80% recall at 5.00% FPR on XD-Violence. Additional comparisons on challenging benign inputs assess recognition of benign cross-modal context. We also clarified the fine-grained sampling design and outlined plans to expand the three- and four-modality subsets in the future work. To assess the effect of sample distribution on the overall evaluation, we evaluated a 2,025-request balanced subset, applied three alternative weighting schemes, and balanced modality orders within each attack family. The main benchmark findings and PerceptGuard’s overall advantage remain consistent across these analyses. (Tables R1–R8; Fig. 5.)

The individual replies provide the complete results and corresponding revision locations. We would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.