acceptodds
Under review as a conference paper at ICLR 2027

RuleSafe-VL: Benchmarking Rule-Conditioned Reasoning in Vision-Language Moderation

Abstract

Policy-grounded vision-language moderation can warrant different decisions for the same content depending on context. When evidence about outcome-changing conditions is missing, definitive judgments may rely on unsupported assumptions. How should policy application be formulated when conditions interact and the available evidence leaves multiple outcomes possible? We formalize this problem as a rule-conditioned reasoning task and introduce RuleSafe-VL. Derived from public platform policies through expert annotation and adjudication, its 93 atomic rules and 92 typed relations are linked to 2,166 expert-reviewed image–text cases across three high-risk domains. Four complementary tasks evaluate satisfied policy conditions, rule relations, decision states, and context-conditioned decisions. Across 10 frontier, open-source, and safety-oriented VLMs, the best rule-relation recovery and decision-state assessment scores reach 64.8 and 64.5 Macro-F1, respectively. Context-Pair Accuracy peaks at 35.4%, compared with an expert reference of 84.2%. Diagnostic analyses reveal modest gains in decision-state assessment from providing expert-annotated policy structure, alongside both missed policy-required decision changes and unnecessary changes when the correct outcome remains unchanged. These findings highlight challenges in determining when evidence warrants a moderation decision and how that decision should respond to policy-relevant context.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Policy-Grounded Content Moderation, Rule-Conditioned Reasoning, Benchmarking, Diagnostic Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6gCaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces RuleSafe-VL, a benchmark for evaluating how vision-language models apply explicit content-moderation policies when rules interact and relevant context may be missing. Its main contributions are a rule-conditioned task formulation and an annotated benchmark covering four capabilities: identifying satisfied policy conditions, recovering relationships between rules, determining whether evidence supports a unique decision, and adapting decisions when additional context is supplied.

The benchmark contains 2,166 image-text cases, linked to 93 atomic rules and 92 relations derived from public platform policies, across nudity/sexualized content, dangerous/harmful behavior, and graphic/injury-related content. Labels are established through independent human annotation and expert adjudication.

The study provides extensive experiments to evaluate 10 existing VLMs under zero-shot prompting.

Critical strengths

The benchmark links 2,166 image-text cases to explicit policy rules and relations, supported by independent annotation and expert adjudication. Reported agreement k=0.91, 0.82, and 0.86 across the case-level annotation tasks-provides evidence that annotators apply the guidelines consistently. Holding image-text content fixed while varying supplied context also creates a useful setting for testing whether decisions change appropriately. This combination gives the benchmark potential value for developing and comparing moderation systems, beyond documenting low performance on a collection of difficult examples.

Critical weaknesses

1, The study is lacking an overview figure to help readers can capture the overview of the work and pipeline.

  1. There are several internal inconsistencies in the benchmark reported results. a) Table 1 reports 918 decidable cases out of 2,166 (42.4%), whereas the categories with decidable paired evidence in Table 12 sum to 49.0%. (b) LLaVAGuard’s T2 score is 4.7 in Table 2, but 6.7 in Table 10
Questions for authors
  1. Could you provide an overview figure to help readers capture the workflow and pipeline easier?

  2. I would like recommend you to review and correct all the inconsistencies in the benchmark.

2 comments
AuthorsOpenReview ↗

Author Response to Weakness 1, Weakness 2

W1 & Q1. Overview of the Benchmark Construction and Evaluation Pipeline

Thank you for this suggestion. We have added Figure 2 in Page 3 to the revised manuscript to show the benchmark construction pipeline and the inputs and outputs of all four tasks.

Benchmark construction. The left panel presents policy construction and case construction. We collect 186 policies from Meta, YouTube, Reddit, and TikTok, filter them for conditions assessable from observable evidence or explicit context, and remove duplicates to obtain 24 policies. Expert annotation and adjudication establish 93 atomic rules and 92 typed relations. These policies guide the selection of 2,320 candidate image–text pairs, from which validation and annotation yield 2,166 cases. Experts annotate rule activation and decision states, then provide contextual completions and compliance labels for underdetermined cases. The left panel makes explicit that every gold label is traceable to specific policy requirements and case evidence through documented human annotation and adjudication.

Evaluation framework. The right panel illustrates four complementary tasks: Task 1 identifies evidence-supported atomic rules; Task 2 recovers relationships among policy rules; Task 3 assesses whether the available evidence determines a unique outcome; and Task 4 determines the appropriate outcome under supplied context. Each task includes a representative input, a guiding question, and its expected output, making the evaluation procedure directly accessible. Together, the panels show how RuleSafe-VL formalizes moderation as rule-conditioned reasoning under the joint constraints of policy, content, and context, and translates its requirements into independently annotated evaluation targets. Evaluating these targets alongside end-to-end predictions distinguishes final-outcome correctness from correctness on the associated diagnostic judgments—a distinction supported by §5.3. Figure 2 thus connects the formulation, benchmark construction, and evaluation objectives.

W2 & Q2. Numerical inconsistencies: LLaVAGuard’s Task 2 score (4.7 vs. 6.7) and decidable-case proportions (42.4% vs. 49.0%)

Thank you for identifying these discrepancies. The Task 2 score difference is a transcription error in Table 2. The decidable-case proportions use different denominators: the full benchmark in Table 1 and a modality-sufficiency diagnostic subset in Table 12.

Task 2 score: correction of Table 2. Regarding “4.7 in Table 2, but 6.7 in Table 10,” the correct Task 2 Macro-F1 for LLaVAGuard-7B is 6.7, consistent with the per-family results and their average in Appendix Table 10. We will correct the transcription error in Table 2 from 4.7 to 6.7. This changes its relative ordering with GuardReasoner-VL-3B, which scores 6.6, but preserves the conclusion that both safety-oriented models score substantially below the evaluated general-purpose VLMs on rule-relation recovery under our structured evaluation protocol.

Decidable proportions: full benchmark versus diagnostic subset. Regarding “42.4%” in Table 1 versus “49.0%” in Table 12, Table 1 reports 918 decidable cases out of 2,166, giving 918/2,166×100=42.38%918/2{,}166 \times 100 = 42.38\%, or 42.4% rounded to one decimal place. Table 12 alone uses a separately selected subset for modality-sufficiency analysis. We retained all 918 decidable cases and selected 956 of the 1,248 underdetermined cases through stratified random sampling by policy category, independently of model predictions. The resulting subset contains 1,874 cases, giving 918/1,874×100=48.99%918/1{,}874 \times 100 = 48.99\%. This matches the rounded sum of the four categories with decidable paired evidence: 38.9+4.9+2.7+2.5=49.0%38.9 + 4.9 + 2.7 + 2.5 = 49.0\%. The original image–text decision-state labels remain unchanged; text-only and image-only sufficiency were additionally annotated for this subset. Both the proportions and model accuracies in Table 12 refer to these 1,874 cases. We will document the subset construction and denominator explicitly and rename “Dataset Share” to “Share within the diagnostic subset (%)”.

Thank you very much for your additional information. They address all my concerns.

Reviewer_ZiBBOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RuleSafe-VL, a benchmark designed to evaluate whether vision-language models (VLMs) can perform structured content moderation in accordance with platform policies, rather than merely predicting whether content is compliant or in violation. The benchmark comprises 93 atomic rules, 92 types of rule relationships, and 2,166 image-text cases across three high-risk domains. It evaluates models’ abilities to identify applicable rules, determine relationships between rules, assess whether the available evidence is sufficient to make a decision, and revise decisions when additional context is provided.

Critical strengths

S1: The problem is practically relevant

Distinguishing moderation outcome prediction from evidence sufficiency assessment is important, as determining whether the available evidence supports a definitive decision is a key aspect of policy-based content moderation.

S2: The dataset provides a valuable benchmark for model evaluation and improvement

The dataset involves substantial annotation efforts, including independent annotation, calibration, and disagreement adjudication. The authors report pre-adjudication Cohen’s κ for rule activation, decision status, and contextual review. Categorizing conditions into content, context, and justification also supports fine-grained diagnosis.

Critical weaknesses

W1: Gap between motivation and practical applicability.

The connection between the paper’s motivation and real-world deployment remains unclear. In practice, moderation policies are generally predefined, and inputs—including content and contextual information—are expected to provide the evidence needed for decision-making, even if the technical alignment is imperfect. Models can make policy-conditioned decisions either by internalizing policy knowledge or by identifying atomic capabilities that can be handled by a rule engine. In this setting, treating insufficient evidence as indeterminate may reflect a business requirement rather than a general reasoning capability. The authors should clarify the practical need that RuleSafe-VL addresses beyond these existing workflows.

W2: Limited coverage and potential inaccuracies of the benchmark.

Prior work such as Hi-Guard already formulates vision-language content moderation as rule-conditioned reasoning and incorporates related capabilities into its training strategy. Although the failure modes identified by RuleSafe-VL may help guide model training, the current taxonomy appears incomplete, and these issues could potentially be addressed through either prompting or post-training. While the benchmark is useful for evaluating model performance, a more actionable approach would explicitly map each case to the relevant policy rules and the content supporting the moderation decision.

W3: Insufficiently specified operational definitions of rule relationships.

The paper introduces four types of rule relationships—Threshold, Override, Gating, and Independent—and states that each relationship must be supported by policy text. However, the main text does not sufficiently specify how these types are distinguished, how rule pairs are constructed for evaluation, or where the annotation boundaries for Independent lie. More explicit and reproducible operational criteria are needed to ensure consistent annotation.

W4: The Underdetermined label may conflate different sources of uncertainty.

The paper defines Underdetermined as a state in which multiple outcomes remain compatible with the current policy. While this formulation is clear, underdetermination in practical moderation may arise from missing image-text evidence, ambiguous policy language, incomplete rule structures, or disagreements among annotators. The paper needs to clarify how these sources are distinguished and which forms of uncertainty models are expected to identify, as opposed to ambiguities introduced by the benchmark construction itself.

W5: Inconsistencies in reported results and unclear justification for contextual decisions.

  • In the main text, Table 2 reports a Task 2 score of 4.7 for LLaVAGuard-7B, whereas Appendix Table 10 reports 6.7. This discrepancy needs to be clarified.
  • Table 1 reports that 918 out of 2,166 cases are decidable (42.38%), whereas the proportions for the joint image-text evidence status D in Table 12 sum to 49.0% (38.9% + 4.9% + 2.7% + 2.5%). The authors should explain this inconsistency.
  • Section 4.3 states that 1,248 underdetermined cases are expanded into 2,194 contextual variants, averaging fewer than two variants per case. However, Figure 6 reports 1,097 pairs expected to change the outcome and 1,190 pairs expected to preserve it. The relationship between these numbers requires clarification.
  • It remains unclear whether the added context actually resolves the missing conditions required by the policy. Figure 3 provides a concrete example: the relevant exception requires sensitive-content masking or an age restriction, denoted by X1. Case B is labeled Underdetermined because X1 is missing. However, after adding the context “general audience / educational purpose,” the outcome is marked as compliant. Based on the information shown, neither the intended audience nor the purpose directly establishes that masking or age restrictions are in place. If the satisfaction of X1 is implicitly assumed, this assumption should be explicitly stated; otherwise, the case appears to remain underdetermined. The authors should provide a clear justification for this decision.
Questions for authors

Q1: Practical applicability

How does RuleSafe-VL address practical challenges beyond conventional policy-based moderation? Does evidence-sufficiency assessment reflect real-world needs or merely predefined abstention requirements?

Q2: Rule-relation definitions

Could the authors clarify the operational criteria for distinguishing Threshold, Override, Gating, and Independent relationships, particularly when policy text is ambiguous?

Q3: Sources of underdetermination

How do the authors distinguish uncertainty caused by missing evidence, ambiguous policies, incomplete rule structures, and annotation disagreement? Which types are models expected to identify?

Q4: Evaluation reproducibility

Since results are averaged across two prompts, could the authors report prompt-level variability and provide the exact model versions, inference parameters, and prompts to support reproducibility?

Q5: Numerical inconsistencies

Could the authors reconcile the discrepancies between the main text and appendix and clarify the corresponding calculations?

14 comments
AuthorsOpenReview ↗

Author Response to Weakness 1 and Question 1

W1 & Q1. Practical Relevance and Motivation of Evidence-Sufficiency Assessment

Thank you for highlighting the connection between our motivation and practical moderation workflows.

Predefined policies and evidence sufficiency. Regarding policies being "generally predefined," two distinctions matter. First, a predefined policy does not guarantee that the available image–text evidence supports a unique outcome. In Figure 1, recognizing surgical imagery does not establish whether its presentation qualifies for the educational exception; unresolved context can change the policy outcome. Second, "predefined abstention requirements" specify how to handle insufficient evidence, while determining when they apply requires assessing the evidence against the policy. Underdetermined denotes an unresolved compliance outcome and is compatible with a predefined action such as requesting additional information.

Practical relevance and reasoning requirements. Evidence-sufficiency assessment determines whether a model's moderation recommendation is warranted. Hi-Guard [1], for example, reports an online deployment that produces classifications and structured evidence before human review. Overstating sufficiency can yield unsupported definitive recommendations; understating it can trigger unnecessary information requests or review. RuleSafe-VL evaluates these judgments under the joint constraints of policy, content, and context (§3.1, Eq. (2)). Regarding "a business requirement rather than a general reasoning capability," both apply: determining whether all evidence-consistent completions yield the same rule-governed outcome is a general reasoning problem under partial information. Task 3 instantiates this requirement by testing whether unresolved conditions can alter the moderation outcome. A "rule engine" can perform the logical computations; Tasks 3 and 4 evaluate the reliability of these judgments when assigned to a VLM. This scope does not assume that every moderation architecture delegates them to VLMs.

Contribution beyond existing workflows. RuleSafe-VL contributes "beyond these existing workflows" by evaluating the judgments required to execute them reliably. HateModerate [2] evaluates hateful/non-hateful predictions against individual Facebook policies, while EVADE-Bench [3] evaluates violation categories under explicit rules using exact and partial label matching. Correct classification alone does not establish whether a model recognizes an unresolved policy outcome or responds appropriately when missing context is supplied. Section 5.3 demonstrates this distinction: correct end-to-end predictions coexist with errors in separately evaluated condition identification or decision-state assessment. Task 3 directly tests evidence sufficiency. Task 4 tests contextual adaptation through controlled completions that hold policy and original content fixed and require decisions to change or remain unchanged. These diagnostic labels need not appear in operational outputs. During development, the resulting capability profiles identify weaknesses and provide criteria for assessing improvements. We will clarify this connection in the revision.

References.

[1] Li, A., et al. Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling. KDD, 2026, pp. 2278–2289. https://doi.org/10.1145/3770854.3783934

[2] Zheng, J., et al. HateModerate: Testing Hate Speech Detectors against Content Moderation Policies. Findings of NAACL, 2024, pp. 2691–2710. https://doi.org/10.18653/v1/2024.findings-naacl.172

[3] Xu, A., et al. EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection. SIGIR, 2026, pp. 3506–3513. https://doi.org/10.1145/3805712.3808579

AuthorsOpenReview ↗

Author Response to Weakness 2

W2. Benchmark Coverage, Rule Taxonomy, and Diagnostic Actionability

Thank you for these comments.

Distinction from prior work. Methods such as Hi-Guard incorporate policies into moderation models and improve decisions through training, demonstrating the practical relevance of rule-conditioned judgment. RuleSafe-VL asks which aspects of policy application remain unreliable. Classification, generalization, and explanation-quality evaluations alone do not separately localize errors in condition identification, rule relations, evidential sufficiency, and contextual adaptation. Section 5.3 demonstrates the diagnostic value of this separation: correct end-to-end predictions can coexist with errors in condition identification or decision-state assessment. RuleSafe-VL makes these capabilities independently testable, providing specific targets for improvement and criteria for verifying progress.

Coverage and taxonomy. The concern that "the current taxonomy appears incomplete" involves two questions. For domain coverage, we select nudity and sexualized content, dangerous and harmful behavior, and graphic and injury-related content because their policies involve visual evidence, interacting conditions, and contextual exceptions central to our evaluation. Decision-state and contextual-adaptation failures recur across all three families, establishing that the observed difficulties are not confined to one domain. Our objective is to identify these failures within the stated scope; establishing their existence does not require exhaustive domain coverage. For the rule taxonomy, the three atomic-condition types and four relation types are derived from public policies across widely used platforms—Meta, YouTube, Reddit, and TikTok—and reviewed by experts. Appendix D shows that all seven types recur across the three policy families, supporting their applicability within the evaluated scope. We will clarify this scope and the taxonomy's role in organizing recurring policy structures.

Case–rule–evidence mapping. Mapping "each case to the relevant policy rules and the content supporting the moderation decision" is integral to our construction and annotation procedure. Sections 4.1–4.2 describe policy decomposition into atomic rules and relations, followed by case alignment with target rule configurations. During Task 1 annotation, experts assess each atomic rule against the image–text evidence and record activated rule IDs with brief evidence-based justifications (Appendix B.3.1). Review and adjudication verify these annotations, linking policy requirements, individual cases, and supporting evidence. We will illustrate this mapping with a concrete annotation example in the revision.

Few-shot prompting on Task 3. To examine whether these difficulties "could potentially be addressed through … prompting," we added controlled 2-shot and 4-shot experiments on decision-state assessment. Each demonstration includes the complete task input, gold decision-state label, and required structured output. Under 4-shot prompting, Gemini 3.1's Macro-F1 increases from 0.3853 to 0.5003 and GPT-5.4's from 0.5168 to 0.5411, while Claude Opus 4.6's decreases slightly from 0.5857 to 0.5835. Other models show gains, declines, or reversals as demonstrations increase. Thus, the tested configurations produce model-dependent benefits without consistently resolving difficulties in determining whether the evidence supports a unique policy outcome.

Model0-shot2-shot4-shot
GPT-5.40.51680.53110.5411
Gemini 3.10.38530.46970.5003
Claude Opus 4.60.58570.57510.5835
Qwen2.5-VL-3B0.47010.50640.5211
Qwen2.5-VL-7B0.34590.25740.2638
Qwen3-VL-4B0.64450.67670.6369
Qwen3-VL-8B0.59200.62690.5507
InternVL3.5-8B0.60860.62690.6454
LLaVAGuard-7B0.48690.50840.5063
AuthorsOpenReview ↗

Author Response to Weakness 3 and Question 2

W3 & Q2. Operational Definitions and Annotation of Rule Relationships

Thank you for requesting clarification. We address the relation criteria, Independent boundaries, rule-pair construction, and ambiguity resolution below.

Operational distinctions. Regarding "how these types are distinguished," Figure 2 and Appendix B.2 distinguish the three directed relations by their effects on the target rule. Override replaces its default effect through an explicit exception or precedence provision. Threshold adjusts its decision threshold or restriction criteria while retaining restrictions on the permitted content. Gating specifies a prerequisite for a rule's applicability. In Figure 3, J1/J2 replace C1's default prohibition with permission when their respective conditions hold and are therefore labeled Override. J3 changes C1's blanket prohibition to restricted permission, retaining a sensitivity-screen or age-restriction requirement; this change in restriction criteria is labeled Threshold. X1 → J3 specifies the protective prerequisite for J3's applicability and is labeled Gating. Prerequisites can accompany either Override or Threshold: their distinction concerns the effect on the target rule, independently of whether a prerequisite exists.

Independent boundaries and scoring. Regarding "the annotation boundaries for Independent," a positive pair concerns different aspects of the same policy subject, with policy text supporting their separate applicability. Neither rule requires the other to hold or changes its conditions, criteria, or effects. In Figure 3, J1 and J2 address the same restricted-content category through separately specified permission paths. The positive evidence lies in these separate provisions, rather than merely the absence of language linking them. Shared subject matter or semantic similarity alone is insufficient; pairs without textual support for any defined relationship remain unconnected. Independent pairs are therefore actively identified, not assigned as a residual category. For scoring, their endpoints are treated as an unordered pair: either source–target ordering is accepted if the pair and type are correct. Gating, Threshold, and Override require the correct pair, type, and direction; reversed edges are scored as incorrect. We will make this convention explicit wherever the representation and matching criteria are described.

Rule-pair construction. Regarding "how rule pairs are constructed for evaluation," Appendix B.2 describes annotation following atomic-rule decomposition. Three experts independently examine the complete policy and its atomic rules, identify supported pairs, and record their endpoints, relation types, directions where applicable, and supporting passages. Discussion and adjudication establish the gold relation set. Task 2 then asks models to recover this set from the policy text and atomic rules, evaluating both pair identification and relation classification.

Ambiguity resolution. Regarding "when policy text is ambiguous," Appendix B.2 specifies adjudication against the policy's full context, conditions, exceptions, and qualifiers. The three experts provide competing interpretations and supporting rationales; a fourth expert applies the operational criteria to adjudicate relation existence and type. Fifteen candidate pairs involved disagreements: six concerned Independent existence, with four retained and two excluded; five concerned Gating existence, with four retained and one excluded; and four concerned Override versus Threshold, all retained with adjudicated types. Thus, 12 disputed relations were retained and three pairs were excluded because no policy-supported relation was established. We will report this breakdown and link the operational criteria directly to the supporting policy passages and examples.

Reviewer_jUf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes policy-grounded vision-language moderation as a rule-conditioned reasoning task and introduces RuleSafe-VL, a benchmark of 93 atomic rules and 92 typed relations (Threshold / Override / Gating / Independent) derived from public platform policies (Meta, YouTube, Reddit, TikTok) and instantiated in 2,166 expert-reviewed image–text cases across three high-risk domains (nudity/sexualized, dangerous/harmful, graphic/injury). Ten frontier, open-source, and safety-oriented VLMs are evaluated zero-shot.

Critical strengths
  1. A genuinely new task-level framing: casting moderation as rule-conditioned reasoning and making evidential sufficiency a first-class evaluation target is a real departure from prior safety benchmarks that score labels/severity or final outcomes.
  2. The benchmark construction is careful and credible: expert annotation with reported agreement, multi-stage adjudication, and a controlled context-pair design that varies only audience/purpose while holding image, text, and policy structure fixed give the dataset strong internal validity.
  3. Validation and failure localization are the paper's strongest evidence: the cross-check justifies the task decomposition itself, the rule-type, relation-type, policy-structure-intervention, and under/over-update analyses each isolate a distinct bottleneck, and claims stay honest and well-scoped with a human expert reference and broad model coverage.
Critical weaknesses
  1. The figures contradict the reported experiment: the legends of Figures 5 and 6 list models that never appear in Section 5.1 or Table 2 — Figure 6 shows 12 entries (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro/Flash, Claude 3 Sonnet/Opus, Qwen2.5-VL-72B/14B/7B, Llama3.2-VL-90B/11B/3B) while the paper evaluates GPT-5.4, Gemini 3.1, Claude Opus 4.6, Qwen3-VL-4B/8B, InternVL3.5-8B, etc., and Figure 5a has 10 entries although only 9 models are scored — so the numbers in the Section 5.4 prose (drops of up to 26.5 and 47.8 points; frontier models at 48.7–52.5) cannot be verified against these figures.
  2. Data contamination is not addressed: cases are drawn from COCO, OpenImages, and existing safety benchmarks — images frontier VLMs almost certainly saw during pretraining — which matters most for content recognition, so a benchmark paper measuring capability rather than familiarity needs a contamination analysis (e.g., provenance splits or memorization probes).
  3. Related work omits directly relevant policy-grounded moderation benchmarks: Section 2 does not cite HateModerate (Zheng et al., NAACL Findings 2024), which tests detectors against Facebook's 41 hate-speech policies; EVADE-Bench (Xu et al., 2025), a multimodal policy-rule benchmark for evasive content; KuaiMod (Lu et al., 2025); LLaVAShield (Huang et al., 2025), which uses policy-dimension labels in multi-turn moderation; or the classic context-dependence result of Pavlopoulos et al. (ACL 2020) that context flips toxicity labels — leaving the novelty claim under-differentiated from these lines of work.
  4. Task 2 is not a vision-language task: given the atomic rules and policy text, the model only identifies relations among rules and no image is involved, so relation recovery is a text-only capability that should either be justified explicitly as a policy-structure skill — acknowledging the modality imbalance — or complemented with a grounded variant.
  5. The gold "decision state" is under-specified: Eq. (4) defines Decidable iff |Y_P(q)| = 1, but the paper never states how Y_P(q) is derived from the policy graph (deterministic evaluation procedure vs. expert judgment) nor how annotators were instructed to determine it.
Questions for authors

The critical figure–table inconsistency undermines confidence in the reported numbers, and there are several major issues (contamination analysis, related-work coverage Task 2's modality, and the decision-state gold specification). See weakness for more details.

6 comments
AuthorsOpenReview ↗

Author Response to Weakness 1

W1. Clarification of the Alleged Figure–Table Inconsistencies

Thank you for examining the consistency of our experimental reporting. Regarding the concern about a “critical figure–table inconsistency,” several factual descriptions in the review differ from the submitted figures, including the model identities, the number of available results, and the locations of the cited values. We clarify these differences below with direct references to the figures and captions. We would appreciate the reviewer reconsidering this concern and its implications for the overall assessment and score in light of this evidence.

Model identities: Figure 6 matches §5.1 and Table 2. Contrary to the claim that “Figure 6 shows 12 entries” featuring GPT-4o, Gemini 1.5, Claude 3, and Llama3.2 models, Figure 6a contains the ten models specified in §5.1 and Table 2: GPT-5.4, Claude Opus 4.6, Gemini 3.1, InternVL3.5-8B, Qwen3-VL-4B/8B, Qwen2.5-VL-3B/7B, GuardReasoner-VL-3B, and LLaVAGuard-7B. Figure 6b contains the nine models evaluated on Task 4. The older-model list cited in the review is absent from the figure and therefore cannot establish a figure–table mismatch.

Model counts: Figure 5a displays ten rows and nine valid results. The claim that “Figure 5a has 10 entries although only 9 models are scored” overlooks the explicit N/A entry for GuardReasoner-VL-3B. That row contains no Task 1 result; the other nine rows contain the scores discussed in §5.4. Table 2’s caption explains this model’s exclusion from Task 1, and Figure 5’s caption defines N/A. The figure and text therefore report the same number of evaluated models.

Numerical traceability: the cited results are located in Figure 5. The declines of 26.5 and 47.8 percentage points refer to the Context–Content circle and Justification–Content diamond, respectively, in LLaVAGuard-7B’s row of Figure 5a. The range 48.7–52.5 is explicitly printed in Figure 5b’s Independent column: Gemini 3.1, 48.7; Claude Opus 4.6, 52.0; GPT-5.4, 52.5. These locations directly connect the statements in §5.4 to the corresponding models and quantities. The cited evidence thus does not support the alleged “critical figure–table inconsistency,” and we ask that this factual correction be reflected in the reviewer’s final assessment.

AuthorsOpenReview ↗

Author Response to Weakness 3

W3. Distinction from Prior Policy-Grounded Moderation Benchmarks

Thank you for identifying these relevant studies. We will add all five references to §2 and clarify how RuleSafe-VL extends their evaluation objectives.

From moderation outcomes to policy-application capabilities. Regarding the “novelty claim under-differentiated from these lines of work,” HateModerate [1] identifies failures against individual Facebook policies, EVADE-Bench [2] evaluates evasive-content detection under explicit rules, and KuaiMod [3] improves moderation through case-based training and feedback. These studies evaluate policy conformity and improvements in moderation performance. RuleSafe-VL examines a further question: whether successful predictions coincide with correct condition identification, relation understanding, and evidence-sufficiency assessment. Its rule-conditioned reasoning formulation (§3.1) makes these requirements independently testable, using three atomic-condition types and four relation types derived from selected policies of Meta, YouTube, Reddit, and TikTok (§3.2). Independently annotated targets enable joint analysis of these judgments and final outcomes. RuleSafe-VL abstracts these requirements into shared policy-application capabilities, grounded in three atomic-condition types and four relation types derived from the moderation policies of Meta, YouTube, Reddit, and TikTok (§3.2). Its rule-conditioned reasoning formulation (§3.1) evaluates these capabilities independently, such that the evaluation targets are the capabilities needed to apply policies, extending beyond performance organized by particular policy categories or moderation scenarios. Section 5.3 demonstrates the value of this joint analysis: correct end-to-end predictions can coexist with errors in separately evaluated condition identification or decision-state assessment. RuleSafe-VL therefore distinguishes outcome correctness from correctness on the associated diagnostic judgments.

Evidence sufficiency and contextual adaptation under a fixed policy. LLaVAShield [4] integrates multimodal, multi-turn information to assess user and assistant safety under specified policy dimensions. An outcome-relevant condition may remain unresolved after this information is integrated. Task 3 explicitly tests whether the evidence warrants a unique conclusion, assigning Underdetermined when plausible values of a missing condition yield different outcomes under the same policy. Pavlopoulos et al. [5] show that context can change human toxicity judgments and examine whether classifiers benefit from it. RuleSafe-VL evaluates whether contextual effects conform to a specified policy: Task 4 fixes the policy and original image–text content and supplies controlled context completions requiring either a change or preservation of the outcome. Together, these tasks assess when a conclusion is warranted and how it should respond to additional evidence under policy constraints. We will make these distinctions explicit in the revised related-work discussion.

References.

[1] Zheng, J., et al. HateModerate: Testing Hate Speech Detectors against Content Moderation Policies. Findings of NAACL, 2024, pp. 2691–2710. https://doi.org/10.18653/v1/2024.findings-naacl.172

[2] Xu, A., et al. EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection. SIGIR, 2026, pp. 3506–3513. https://doi.org/10.1145/3805712.3808579

[3] Lu, X., et al. VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform. KDD, 2025, pp. 4682–4693. https://doi.org/10.1145/3711896.3737273

[4] Huang, G., et al. LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models. CVPR, 2026, pp. 30130–30140.

[5] Pavlopoulos, J., et al. Toxicity Detection: Does Context Really Matter?. ACL, 2020, pp. 4296–4305. https://doi.org/10.18653/v1/2020.acl-main.396

AuthorsOpenReview ↗

Author Response to Weakness 4

W4. Justification of Task 2 as Policy-Structure Reasoning

Why policy-structure reasoning belongs in RuleSafe-VL. Regarding “Task 2 is not a vision-language task,” Task 2 takes policy text and atomic rules as inputs to evaluate policy-structure reasoning, a component of rule-conditioned moderation. In Figure 3, correctly identifying C1 and J1 as active is insufficient to determine the outcome: the model must also understand that J1 overrides C1, replacing the default prohibition with permission when the exception applies. Rule relationships specify how identified conditions jointly determine a decision. RuleSafe-VL evaluates both component capabilities and end-to-end moderation performance. Task 2 examines the same policies used in the case-level tasks, helping distinguish policy-structure errors from case-level evidence errors through separate evaluation.

Distinct evaluation objects and modality requirements. Regarding “modality imbalance,” Task 2 and the case-level tasks assess different objects. A single policy governs multiple image–text cases, each of which may activate different atomic rules within the same fixed policy structure. Task 2 evaluates relationships within this structure using textual inputs. Tasks 1, 3, and 4 require image–text evidence to evaluate how the policy applies to particular cases through condition identification, evidence-sufficiency assessment, and contextual adaptation. The modality difference thus follows from assessing the policy itself and its application to multimodal cases.

Role of a grounded variant. A “grounded variant” would assess whether policy relationships are correctly applied to a particular image–text case, combining relation understanding with case-specific evidence assessment. Tasks 1, 3, and 4 already evaluate complementary aspects of case-level policy application. A grounded relation-application task could add a further diagnostic, while Task 2 isolates understanding of the governing relationships independently of any particular case. We will clarify these evaluation targets, their input modalities, and their complementary roles in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank the reviewers for their detailed feedback. Across the discussion, we addressed five questions about RuleSafe-VL.

  1. Why evaluate policy-conditioned VLMs, and where does this evaluation matter in practice? Systems such as Hi-Guard already use VLMs to assess content under explicit moderation policies. At initial review, a model’s recommendation can affect whether a post is retained, removed, or referred for further review. On appeal, the original post may stay the same while the user supplies evidence relevant to a policy exception. RuleSafe-VL gives developers a way to test, before deployment, whether a model’s recommendations are warranted at both points in the workflow.

  2. What does RuleSafe-VL reveal beyond final-label accuracy, including when inputs are complete? We clarified the distinction between judging whether the available evidence is sufficient (Task 3) and applying a policy once the decisive context is supplied (Task 4). Gemini 3.1 illustrates the value of this distinction. It ranked second in condition identification and first in rule-relation recovery, yet correctly judged both members of only 28.5% of the completed context pairs. The separate evaluations reveal a gap between performance on individual policy judgments and performance when those judgments must be applied to complete cases.

  3. How are policy boundaries preserved and gold labels established? We clarified how annotators retain source-policy qualifiers and exceptions, distinguish the four relation types, and adjudicate disputed relations against the supporting text. Three of 15 disputed candidate relations were excluded for insufficient textual support. We also made the decision-state criterion explicit: experts label a case Underdetermined only when plausible values of a missing condition can change the outcome under the same policy. Task 2 isolates the textual policy structure, while Tasks 1, 3, and 4 assess its application to multimodal cases.

  4. Do the findings depend on image provenance or prompting? We conducted source-stratified analyses and additional 2-shot and 4-shot experiments. Difficulties in condition identification, evidence sufficiency, and contextual updating recurred across source datasets, although prior exposure to individual images cannot be ruled out. Few-shot effects varied by model: with four demonstrations, Gemini 3.1’s Task 3 Macro-F1 rose from 38.5 to 50.0, while Claude Opus 4.6 remained essentially unchanged. The additional evidence identifies where prompting helps and where the difficulty persists.

  5. Which reporting issues were corrected or clarified? We added a construction-and-evaluation overview to the revised manuscript showing how benchmark construction and expert annotation connect to the inputs and outputs of all four tasks. We verified the disputed figure entries: Figure 6a contains the ten models listed in §5.1, Figure 6b contains the nine evaluated on Task 4, and Figure 5a shows nine valid Task 1 results plus one explicit N/A. Separately, we identified a transcription error in Table 2: LLaVAGuard-7B’s Task 2 Macro-F1 should read 6.7. We reconciled the decidable-case proportions in Tables 1 and 12 by documenting their denominators: 918/2,166 for the full benchmark and 918/1,874 for the diagnostic subset. We also distinguished the 2,194 contextual variants from the 2,287 within-case pairs used for PairAcc, which counts a pair as correct only when both variants are judged correctly. We will incorporate the numerical correction, metric definitions, and full context descriptions in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.