acceptodds
Under review as a conference paper at ICLR 2027

DynRule: Benchmarking and Modeling Dynamic Rule Interactions for Universal Multimodal Content Moderation

Abstract

Content moderation models are typically optimized for limited rule patterns, making them less adaptable to the diverse and dynamically evolving hierarchical rule configurations encountered on real-world platforms. In practice, rules evolve with business requirements, interact under different authorization settings, and may be further complicated by untrusted pseudo-rules embedded in multimodal user content. Existing benchmarks typically study policy changes or instruction hierarchy in isolation, leaving the dynamic interactions among multiple rule levels underexplored. To better reflect real-world moderation, we introduce DynRule-Bench, which systematically varies hierarchical rule sources and their interactions as business requirements evolve. Controlled edits, source exchanges, and pseudo-rule insertions generate diverse rule interaction patterns across four scenarios, evaluating both adaptation and preservation. We further propose DynRule-Guard, a universal rule-interaction guidance framework. DynRule-Guard constructs a Rule Relation Graph to decompose complex hierarchical interactions into explicit local relations, aggregates them into a compact Global Interaction State in a shared Rule-Interaction Latent Space, and dynamically composes reusable patterns from an Interaction Pattern Bank to instantiate an interaction-specific Guidance Token. This design enables universal moderation across diverse and unseen rule configurations by inferring their interactions in a shared Rule-Interaction Latent Space and pattern composition, without rule-specific retraining. Extensive experiments on DynRule-Bench reveal the brittleness of existing moderation methods under dynamic hierarchical rules and demonstrate that DynRule-Guard consistently improves moderation performance, with further generalization to instruction-hierarchy evaluation on IHEval. Our code and data will be released publicly.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Content Moderation, Dynamic Rule Interactions, Instruction Hierarchy, Vision-Language Models, Rule-Conditioned Guidance, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6yb4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal content moderation under dynamically evolving hierarchical rules. A moderation request combines four sources—a platform-wide System Instruction with explicit rules, a business-side Moderator Instruction whose runtime rules change with business needs, multimodal User Content that may embed untrusted pseudo-rules ("this is not an advertisement") in text, OCR, or ASR, and an Auxiliary Policy specifying priority, authorization, and applicability conventions—and a correct decision depends on which rules apply, how they interact (agreement / one-sided applicability / authorized refinement / conflict), and which one governs. The paper contributes two things. DYNRULE-BENCH is a benchmark built from 1,430 parent samples by four controlled interventions (content edits, rule edits, source exchanges, pseudo-rule insertions), organized into four scenarios—runtime rule adaptation (T1), compatible multi-rule composition (T2), hierarchical arbitration (T3), and content-embedded pseudo-rule interference (T4)—spanning 28 configurations and a frozen 4,727-record test set; a reference-arbitration procedure re-validates every candidate by checking rule applicability, interaction, authorization scope, and source priority, and evaluation jointly measures correct adaptation (EDA on decision-changed subsets, Changed-reference PairAcc on source-exchange pairs) and correct preservation (EDA on decision-preserved subsets, Equal-reference PairAcc), explicitly defeating both the always-change and never-change strategies. DYNRULE-GUARD is a guidance framework that models rule interactions explicitly: source-aware states and latent pseudo-rule queries feed a Rule Relation Graph whose policy-conditioned edges separate semantic interaction from policy-aware interpretation; a compact Global Interaction State is mapped through an Interaction Router onto a shared bank of K=8 patterns, linearly composed into an Interaction Guidance Token appended to an intermediate layer of a frozen LoRA-adapted backbone. On three backbones (Qwen3.5-9B, GLM-4.6V-Flash, InternVL3.5-8B), DYNRULE-GUARD reaches 93.33/91.36/91.67% overall EDA versus 85.17/82.09/74.05% for the strongest baseline (Vanilla SFT), with the largest C-ref PairAcc gap on Qwen (72.58 vs 46.88) but the reverse on GLM and InternVL; for the transfer setting (training on D0 only), it improves the Conflict condition on IHEval by 31.74 points over the moderation backbone. Ablations and structural diagnostics (near-orthogonal pattern bank, state-dependent but non-collapsed routing, state-separated guidance geometry) support the mechanism.

Critical strengths

(I) The benchmark's evaluation protocol is the cleanest operationalization I have seen of adaptation-versus-preservation under rule dynamics, and it produces a genuinely useful result. The four-way metric design (Changed/Preserved EDA × Changed/Equal-reference PairAcc) closes both degenerate strategies by construction: a pair counts only if both members match their respective references, so neither always-flipping nor never-flipping can score. The strict source-exchange pairs—identical content, policy, and full rule texts, only the System/Moderator assignment swapped—separate authority effects from wording effects in a way that policy-change benchmarks (SafeEditBench, PolicyShiftBench) and instruction-hierarchy benchmarks (IHEval, Control Illusion) individually cannot. The measured consequence is stark and informative: defense methods score mostly 0-8% on Changed-reference pairs (with one notable exception, ISE at 45.16% on Qwen), revealing that most never actually arbitrate by authority; and the general VLMs' near-collapse on T4 (11.8-18.9%) quantifies how far pseudo-rule interference alone shifts decisions. This is consequential to my assessment because the benchmark contribution stands on its own: it exposes a real, previously unmeasured failure mode with a protocol others can reuse. (II) The method is mechanistically motivated and its gains are supported by evidence beyond aggregate accuracy. DYNRULE-GUARD's design maps directly onto the problem decomposition (policy-conditioned graph edges separating semantic interaction from policy interpretation; pseudo-rule queries for unannotated multimodal pseudo-rules; pattern composition for reuse across configurations), and the paper does not stop at the headline numbers: ablations isolate each component and auxiliary loss; the pattern-bank Gram matrix confirms near-orthogonality (off-diagonal 0.0034-0.0049); routing is state-dependent without collapsing (single-rule→P2, untrusted-content→P5/P7/P8) yet non-degenerate within states (t-SNE manifolds); and the IHEval zero-shot transfer (+31.74 Conflict over the backbone, +27.83 over Vanilla SFT) demonstrates that the learned interaction modeling generalizes beyond the constructed distribution to a text-only instruction-hierarchy benchmark the model was never trained on. The combination of in-benchmark gains, cross-benchmark transfer, and structural diagnostics makes the central claim—explicit interaction modeling beats implicit supervision—credible rather than merely consistent with the tables.

Critical weaknesses

All positive results live inside the paper's own construction pipeline, and no external moderation benchmark validates them. D1 and D2 are both descendants of the same 1,430 parents, generated by the same four intervention types, validated by the same reference-arbitration procedure, and balanced into the same 28-configuration grid. DYNRULE-GUARD is trained on D1 and evaluated on D2; its +8 to +18-point headline margins are therefore in-distribution by construction. The one external evaluation, IHEval, covers only text instruction-hierarchy (no multimodal content, no business-rule editing, no pseudo-rule insertion in OCR/ASR), so the claim that DYNRULE-GUARD is a "universal" moderation guidance framework rests, for its multimodal and dynamic-rule dimensions, entirely on the benchmark introduced in the same paper. Relatedly, the construction distribution is itself a design choice: the uniform scenario cells and the reference-arbitration conventions (e.g., System always wins unresolved conflicts) encode a specific—reasonable but contestable—model of how real platforms' rules behave; models trained and evaluated within this encoding may inherit its conventions rather than learn to read arbitrary auxiliary policies. An evaluation on any independently constructed moderation set with rule annotations (even a small one), or a held-out auxiliary-policy format differing from the training policy, would substantially change how far I am willing to extrapolate.

Questions for authors

(I) The paper's positive results are all measured on descendants of the same construction pipeline used for training. Can you evaluate DYNRULE-GUARD on at least one independently constructed moderation dataset with rule/policy annotations (e.g., a public policy-conditioned moderation set, or RuleSafe-VL adapted to your output schema), even if only for a subset of scenarios? If the interaction-modeling advantage shrinks outside your pipeline, the "universal" claim needs qualification. (II) Table 1 shows two cases where Vanilla SFT beats DYNRULE-GUARD on Changed-reference PairAcc (GLM: 59.68 vs 43.55; InternVL: 35.48 vs 20.97), which the text does not discuss. What explains this inversion on the smaller backbones, and given 65 C-ref pairs, is it statistically meaningful? Please report bootstrap intervals for the PairAcc columns. (III) Please add uncertainty quantification throughout: standard errors or bootstrap intervals for Table 1 (at minimum the Overall EDA and PairAcc columns), Table 3 ablations, and Table 19. Several conclusions (component deltas of 0.5-0.6 points; K=8 vs K=16) are within plausible resampling noise. (IV) What are the inference-time costs? The interaction modules (width d=256) add graph cross-attention, 16 pseudo-rule queries, and K-pattern composition per request, and the guidance token lengthens each forward pass—please report added latency/FLOPs relative to the Moderation Backbone, since deployment feasibility for moderation at scale depends on this.

7 comments

We sincerely thank the reviewer for the positive assessment of DynRule-Bench's adaptation–preservation evaluation protocol, its ability to reveal practically important failure modes, and the mechanistic motivation and extensive empirical validation of DynRule-Guard. We appreciate the constructive suggestions and have conducted additional experiments to address these concerns.

Q1. Generalization to External Moderation Benchmarks

We agree that evaluation on an independently constructed benchmark is essential to establish generalization beyond our data construction pipeline.

We additionally evaluate DynRule-Guard on PolicyShiftBench, an independent multimodal moderation benchmark that evaluates policy-conditioned decisions on images under dynamically changing policies, including held-out policy definitions. It reports F1 and Policy Shift Score (PSS), measuring classification quality and correct adaptation to policy changes, respectively.

MethodAdaptive F1Adaptive PSSShift F1Shift PSS
Moderation Backbone55.304.0450.990.00
Policy Prompt53.012.8648.630.00
Vanilla SFT37.391.3745.3616.29
Spotlighting55.143.5549.362.94
StruQ55.5712.2148.120.81
SecAlign53.804.9548.702.56
ISE33.330.0047.2511.54
ASIDE31.580.0036.360.00
V-Steer55.374.0450.720.00
PolicyShiftGuard-3B86.1575.7958.0650.54
PolicyShiftGuard-7B85.5673.2169.7869.89
DynRule-Guard (Zero-shot)70.0768.5060.2052.00
DynRule-Guard (Fine-tuned)86.9182.4080.2675.32

Zero-shot transfer. Without any PolicyShiftBench-specific training, DynRule-Guard achieves 60.20 F1 and 52.00 PSS on the Shift split, outperforming Vanilla SFT by 14.84 and 35.71 points, respectively. This provides direct evidence of generalization to independently constructed moderation policies beyond our benchmark's templates.

Task-specific adaptation. After fine-tuning on 3,000 training examples for three epochs, DynRule-Guard further achieves 80.26 F1 and 75.32 PSS on the Shift split, surpassing the specialized PolicyShiftGuard-7B baseline. These results demonstrate the transferability and adaptability of our framework, although differences in model capacity and training procedures should be considered.

We initially planned to evaluate our method on RuleSafe-VL. However, to the best of our knowledge, the benchmark has not yet been publicly released. We will conduct additional evaluations and provide further analysis once it becomes available.

Q2. Changed-reference PairAcc on GLM and InternVL

We thank the reviewer for identifying this inconsistency. Our investigation revealed an inter-cluster data synchronization issue that inadvertently omitted 201 Changed-reference training pairs from the DynRule-Guard training data for GLM and InternVL.

After restoring the missing training data and retraining the affected models, the results are:

BackboneMethodC-ref PairAccE-ref PairAccOverall PairAcc
Qwen3.5-9BVanilla SFT43.55 [32.26, 56.45]81.43 [71.43, 90.00]63.64 [55.30, 71.21]
Qwen3.5-9BDynRule-Guard72.58 [61.29, 83.87]98.57 [95.71, 100.00]86.36 [80.30, 91.67]
GLM-4.6V-FlashVanilla SFT59.68 [48.39, 70.97]48.57 [37.14, 60.00]53.79 [46.21, 62.12]
GLM-4.6V-FlashDynRule-Guard69.35 [58.06, 80.65]98.57 [95.71, 100.00]84.85 [78.03, 90.15]
InternVL3.5-8BVanilla SFT35.48 [24.19, 48.39]48.57 [37.14, 60.00]42.42 [34.09, 50.76]
InternVL3.5-8BDynRule-Guard75.81 [64.52, 85.48]98.57 [95.71, 100.00]87.88 [81.82, 93.18]

The results show that DynRule-Guard consistently outperforms Vanilla SFT on Changed-reference PairAcc across all three backbones, resolving the previously observed inversion.

We additionally report 95% bootstrap confidence intervals, obtained from 2,000 resamples with replacement, using the 2.5th and 97.5th percentiles.

Q3. Statistical Reliability of Experimental Results

We appreciate the reviewer's suggestion and have expanded our uncertainty analysis across Tables 1, 3, and 19.

Table 1 — Main Results. We report 95% bootstrap confidence intervals for Overall EDA and all PairAcc metrics using 2,000 resamples. For example, on Qwen3.5-9B, DynRule-Guard achieves 93.33 [91.96, 94.62] Overall EDA and 86.36 [80.30, 91.67] Overall PairAcc, compared with 84.92 [83.09, 86.72] and 63.64 [55.30, 71.21] for Vanilla SFT, respectively.

[Qwen3.5-9B]Overall EDAOverall PairAcc
General VLM65.59 [63.45, 67.54]42.42 [34.09, 50.76]
Moderation Backbone58.53 [55.73, 61.31]38.64 [30.30, 46.97]
Policy prompt60.73 [58.00, 63.24]37.12 [28.79, 45.45]
Vanilla SFT84.92 [83.09, 86.72]63.64 [55.30, 71.21]
Spotlighting58.21 [55.30, 60.93]36.36 [28.79, 44.70]
StruQ62.47 [59.95, 64.83]32.58 [25.00, 40.91]
SecAlign57.95 [55.10, 60.67]35.61 [27.27, 43.94]
ISE80.74 [78.73, 82.65]53.79 [45.45, 62.12]
ASIDE57.25 [55.05, 59.23]31.06 [23.48, 38.66]
V-Steer59.23 [56.35, 61.83]30.30 [22.73, 38.64]
DynRule-Guard93.33 [91.96, 94.62]86.36 [80.30, 91.67]

Table 3 — Component Ablations. We additionally repeat all component and objective ablations using three independent training seeds, reporting mean ± standard deviation.

VariantOverall EDAOverall PairAcc
Full93.28 ± 1.0386.17 ± 0.95
w/o Pseudo-Rule Queries88.95 ± 2.5881.63 ± 2.93
w/o Rule Interaction Graph86.59 ± 4.1976.52 ± 6.28
w/o Global Interaction Queries87.29 ± 2.7979.73 ± 2.72
w/o Pattern Bank & Router85.85 ± 4.5776.33 ± 5.33
w/o Anchor Loss88.22 ± 3.1578.98 ± 1.68
w/o Orthogonality Loss89.06 ± 2.8581.82 ± 1.24
w/o Balance Loss87.09 ± 3.1378.79 ± 1.07

Removing individual components or objectives reduces mean EDA by 4.22–7.43 points and mean PairAcc by 4.35–9.84 points. These repeated-run results provide stronger evidence for the contributions of the proposed components while explicitly accounting for training variability.

Table 19 — Pattern Capacity and Readout Layers. We report 95% bootstrap confidence intervals for all 16 combinations of (K\in{4,8,16,32}) and (\ell\in{4,8,16,24}), covering Overall EDA, Changed-reference PairAcc, and the newly added Overall PairAcc.

Overall EDA [95% CI]

K / layer481624
493.41 [92.02, 94.65]93.28 [91.90, 94.49]92.99 [91.58, 94.30]93.59 [92.29, 94.76]
892.94 [91.40, 94.33]93.33 [91.98, 94.59]92.89 [91.52, 94.11]93.25 [91.93, 94.51]
1693.38 [91.96, 94.63]93.33 [91.96, 94.59]92.45 [90.97, 93.73]92.32 [90.82, 93.71]
3293.75 [92.43, 94.99]93.59 [92.21, 94.89]94.06 [92.76, 95.23]93.80 [92.40, 95.07]

Changed-reference PairAcc [95% CI]

K / layer481624
470.97 [59.68, 82.26]64.52 [53.23, 75.81]43.55 [30.65, 56.45]67.74 [56.45, 79.03]
866.13 [53.23, 77.46]72.58 [61.29, 83.87]54.84 [41.94, 67.74]64.52 [53.23, 75.81]
1679.03 [69.35, 88.71]64.52 [53.23, 77.42]43.55 [32.26, 56.45]64.52 [53.23, 75.81]
3269.35 [58.06, 80.65]70.97 [59.68, 82.26]61.29 [48.39, 74.19]72.58 [61.29, 83.87]

Overall PairAcc [95% CI]

K / layer481624
485.61 [78.79, 90.91]83.33 [76.52, 89.39]71.97 [64.39, 78.79]84.85 [78.03, 90.15]
883.33 [76.52, 89.39]86.36 [80.30, 91.67]78.03 [71.21, 84.85]82.58 [75.76, 88.64]
1688.64 [83.33, 93.94]82.58 [75.76, 88.64]73.48 [65.91, 80.30]82.58 [75.76, 88.64]
3285.61 [79.55, 90.91]85.61 [79.55, 91.67]81.82 [75.00, 87.88]86.36 [80.30, 91.67]

The results indicate that K=8,l=8 provides a competitive performance–capacity trade-off, achieving 93.33 EDA, 72.58 Changed-reference PairAcc, and 86.36 Overall PairAcc. Although some larger configurations achieve higher individual scores, their confidence intervals overlap substantially. We therefore clarify that the default configuration represents a compact and balanced design rather than a statistically unique optimum.

Q4. Inference-time Computational Overhead

We additionally benchmark inference efficiency on a single NVIDIA H800 GPU using Qwen3.5-9B, comparing DynRule-Guard with the Moderation Backbone.

Batch SizeMethodMedian Latency (ms)Throughput (samples/s)FLOPs/batch
1Backbone813.741.22722.130 T
1DynRule-Guard1,027.920.93122.210 T
8Backbone1,471.285.591177.038 T
8DynRule-Guard1,835.804.361177.681 T

DynRule-Guard introduces only 0.346% additional parameters, approximately 0.363% additional FLOPs, and less than 90 MiB additional peak allocated GPU memory.

The median inference latency increases by 26.3% and 24.8% at batch sizes 1 and 8, respectively. This measurable latency overhead primarily reflects the additional online interaction-analysis operations rather than substantial increases in model size or arithmetic complexity.

These results quantify the practical trade-off: DynRule-Guard substantially strengthens dynamic hierarchical rule modeling with minimal parameter, FLOP, and memory overhead, at the cost of additional inference latency.

We thank the reviewer again for these valuable suggestions, which have helped us strengthen the external validation, statistical reliability, and efficiency analysis of our work.

Reviewer_1zAyOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dynamic hierarchical rule interactions in multimodal content moderation. Models must determine which rules apply, resolve compatible or conflicting rules from different sources, and avoid treating pseudo-rules embedded in user content as authoritative instructions. The authors introduce DynRule-Bench, covering four types of rule interactions, and DynRule-Guard, which explicitly models rule relations and composes reusable interaction patterns into moderation guidance. Experiments show that DynRule-Guard outperforms Vanilla SFT in overall accuracy across three vision-language models and performs well on the conflict setting of IHEval. These results support the value of explicit rule-interaction modeling for the tested hierarchical moderation tasks, while broader generalization to real-world settings remains to be established.

Critical strengths
  1. The benchmark targets an important problem that prior evaluations often study in isolation. DynRule-Bench combines multimodal content, dynamic rule changes, hierarchical conflicts, and pseudo-rules embedded in user content.
  2. The method shows consistent gains across multiple backbones. DynRule-Guard outperforms Vanilla SFT on Qwen, GLM, and InternVL, and achieves a substantial improvement on the conflict setting of IHEval. Results across multiple backbones and an external benchmark strengthen the paper’s central claim that explicitly modeling rule interactions can help with hierarchical conflicts.
Critical weaknesses

The benchmark does not yet establish generalization to genuinely unseen rules. Training and test data are split by source content, but both cover the same scenario categories and rule-construction procedures. The results therefore show performance on new examples within known templates, but do not rule out learning construction patterns or fully support generalization to novel rule systems. The data are also primarily edits of existing moderation samples; although the authors report partial human semantic review, they do not report inter-annotator agreement.

The method’s gains are confounded with additional supervision. DynRule-Guard uses auxiliary supervision for seven rule-interaction states and additional structured losses, whereas the main SFT baselines appear to train only on moderation outputs. Without a matched baseline that receives the same interaction labels but not the DynRule-Guard architecture, it is unclear whether the gains come from the architecture or from the additional annotations and supervision. This weakens the attribution of the improvement to the proposed method itself.

Questions for authors

see weaknesses

4 comments

We sincerely thank the reviewer for recognizing the significance of DynRule-Bench in addressing practical challenges in dynamic multimodal moderation, as well as the consistent improvements achieved by DynRule-Guard across multiple backbones. We appreciate the constructive feedback and provide additional experiments to address the concerns.

W1. Generalization to Unseen Rule Configurations

R1.1. Cross-benchmark generalization on IHEval.

Our existing zero-shot evaluation on IHEval demonstrates that the learned rule-interaction representations transfer beyond DynRule-Bench's construction pipeline. In particular, the substantial improvement on the Conflict setting provides evidence of generalizable source-priority reasoning rather than merely memorizing moderation-specific patterns.

R1.2. New evaluation on held-out interaction combinations.

We conduct an additional experiment to explicitly evaluate compositional generalization. Using Qwen3.5-9B as the shared backbone, we compare conditions where Conflict × Content Edit is either seen during training or entirely held out, while retaining Conflict × Rule Edit and Compatibility × Content Edit in both conditions.

All methods follow identical holdout constraints with matched training-data volumes and computational budgets. Evaluation uses the same 100 Changed-reference pairs (200 examples), with content IDs disjoint from training and validation.

MethodSeen EDASeen C-ref PairAccHeld-out EDAHeld-out C-ref PairAccPairAcc Drop
Vanilla SFT72.047.065.537.010.0 pp
ISE70.045.064.036.09.0 pp
DynRule-Guard86.073.082.067.06.0 pp

DynRule-Guard achieves 82.0% EDA and 67.0% C-ref PairAcc on the held-out combination, outperforming Vanilla SFT by 16.5 and 30.0 points, respectively, while exhibiting the smallest performance degradation. These results support its ability to generalize to unseen interaction combinations through a shared Rule-Interaction Latent Space and reusable pattern composition.

R1.3. External generalization on PolicyShiftBench.

We further evaluate DynRule-Guard on PolicyShiftBench, an independently constructed multimodal moderation benchmark whose policy definitions and evaluation scenarios differ substantially from our training data. It evaluates policy-conditioned image moderation, including adaptation to changing policies and generalization to held-out policy definitions.

MethodAdaptive F1Adaptive PSSShift F1Shift PSS
Moderation Backbone55.304.0450.990.00
Policy Prompt53.012.8648.630.00
Vanilla SFT37.391.3745.3616.29
Spotlighting55.143.5549.362.94
StruQ55.5712.2148.120.81
SecAlign53.804.9548.702.56
ISE33.330.0047.2511.54
ASIDE31.580.0036.360.00
V-Steer55.374.0450.720.00
PolicyShiftGuard-3B86.1575.7958.0650.54
PolicyShiftGuard-7B85.5673.2169.7869.89
DynRule-Guard (Zero-shot)70.0768.5060.2052.00
DynRule-Guard (Fine-tuned)86.9182.4080.2675.32

Without PolicyShiftBench-specific training, DynRule-Guard achieves 60.20 F1 and 52.00 PSS on the Shift split, outperforming Vanilla SFT by 14.84 and 35.71 points. After fine-tuning on 3,000 samples for three epochs, it further achieves 80.26 F1 and 75.32 PSS, surpassing the specialized PolicyShiftGuard-7B by 10.48 and 5.43 points.

Together, these results provide complementary evidence of generalization to unseen interaction combinations and independently constructed moderation policies, beyond the original benchmark's construction pipeline.

R1.4. Human semantic review.

Each reviewed sample group was jointly examined by two reviewers following the same annotation criteria, with no unresolved disagreements recorded. As the review was conducted jointly rather than through independent annotations, we do not claim an independent inter-annotator agreement. We will clarify this review procedure and its limitations in the revised manuscript.

W2. Distinguishing Architectural Contributions from Additional Supervision

We appreciate the reviewer's concern regarding the contribution of auxiliary interaction-state supervision.

R2.1. Existing ablation without Anchor supervision.

Our original Table 3 already shows that DynRule-Guard maintains substantial improvements even without the seven-class interaction-state supervision Anchor-loss.

R2.2. New supervision-matched baseline.

To further isolate the benefit of our architectural design, we introduce Vanilla SFT + Anchor, which adds a learnable token and the same seven-class interaction-state supervision, without the Rule Relation Graph, Global Interaction Queries, Pattern Bank, or Router.

MethodEDAPairAcc
Vanilla SFT85.1760.58
Vanilla SFT + Anchor86.6963.64
DynRule-Guard w/o Anchor92.9181.06
DynRule-Guard Full93.3386.36

Adding Anchor supervision improves Vanilla SFT by 1.52 EDA and 3.06 PairAcc points. DynRule-Guard still outperforms this supervision-enhanced baseline.

Moreover, without Anchor supervision, DynRule-Guard still surpasses Vanilla SFT by 7.74 EDA and 20.48 PairAcc points.

These findings demonstrate that auxiliary supervision alone cannot account for the improvements. The explicit modeling of hierarchical rule relations, global interaction aggregation, and dynamic pattern composition provides substantial additional benefits for moderation under evolving rule configurations.

We thank the reviewer again for the valuable suggestions, which have strengthened our evaluation of generalization and the empirical justification of the proposed architecture.

Thank you for your thoughtful evaluation and constructive feedback. We have incorporated the additional experiments and analyses into the revised manuscript, with all changes highlighted in blue.

We hope these revisions have addressed your main concerns and strengthened the credibility of our findings. If so, we would greatly appreciate your reconsideration of the rating and confidence. If any questions or concerns remain, we would welcome your follow-up comments and be happy to provide further clarification.

We sincerely appreciate your time and valuable suggestions, which have helped improve our work.

Reviewer_1zAyOpenReview ↗

Response to the authors

Thanks for the authors’ response. I’ll keep my rating but lower my confidence in it, good luck.

Reviewer_hRuSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal content moderation under changing rules from different instruction sources. It introduces DynRule-Bench, covering rule adaptation, compatible rule composition, hierarchical conflicts, and content-embedded pseudo-rules. It also proposes DynRule-Guard, which models interactions through pairwise attention and composes learned pattern vectors into a guidance token. The method reports substantial improvements over Vanilla SFT across three backbones and additional results on IHEval.

Critical strengths
  1. The problem is useful. Moderation models should follow the currently supplied policy, including authorized exceptions, rather than rely on a fixed classification boundary.

  2. The source-exchange paired evaluation is a useful control. Keeping content and rule texts fixed while exchanging their authority helps isolate source-priority effects. Requiring both predictions to be correct avoids rewarding arbitrary prediction changes.

  3. The reported improvements over Vanilla SFT are substantial across three backbones. The no-anchor ablation in Table 3 retains most of the EDA gain on Qwen, suggesting that the additional interaction-state supervision is not the sole source of improvement.

Critical weaknesses
  1. The evaluation counts and reported scores do not reconcile. Section 5 and Appendix C describe a frozen test set of 4,727 examples, but the scenario counts in Table 15 sum to 4,161. Its field, modality, and transition counts independently give the same smaller total. Table 15 also reports 65 changed-reference and 78 equal-reference pairs. However, the Qwen scores of 72.58% and 98.57% in Table 1 cannot result from integer correct counts over these denominators under ordinary rounding. The overall PairAcc of 86.36% is likewise incompatible with 143 pairs under Eq. 5. These could be reporting or dataset-version issues, but they currently leave the actual evaluation scope unclear and prevent verification of the results.

  2. The evidence for unseen-configuration generalization is not sufficiently targeted. Appendix A.4 specifies a split by original content identifier, which is appropriate. However, Appendix A.5 reports the same scenario–variant cells across training, validation, and testing. This supports generalization to new contents, but does not by itself establish generalization to unseen interaction structures or combinations. The paper does not clearly establish which aspects of test rule configurations are unseen during training. The IHEval results provide useful external evidence, but I am not convinced that they directly validate the proposed reusable-pattern mechanism for multimodal moderation.

  3. The benefit of the central architectural design remains unclear. In Table 3, removing the Rule Interaction Graph reduces EDA by 0.62 points and PairAcc by 0.76 points. Removing the Pattern Bank and Router leaves EDA unchanged and reduces PairAcc by 2.27 points. These results suggest some benefit, but provide limited evidence that reusable pattern composition explains the large improvement over SFT. A simpler guidance-token adapter with comparable capacity, training budget, and auxiliary supervision would be an important control. Repeated runs would also help assess the small component-level differences. The representation plots in Appendix D.2 show structure in the learned representations, but do not independently establish compositional reasoning, especially given the interaction-state training objective.

  4. The baseline adaptations need more implementation and validation details. Appendix C.1 appropriately specifies shared domain-adapted initialization and rule-adaptation data sources. However, method-specific optimization, SecAlign preference-pair construction, and checkpoint selection for the trainable baselines are not sufficiently described. Appendix C.5 provides a detailed PairAcc-focused selection rule for DynRule-Guard, while equivalent details for the baselines are unclear. Several methods perform substantially below Vanilla SFT and obtain zero PairAcc in Table 1. Without the missing details and parsing statistics, I cannot assess how much they reflect limitations of the methods under this task, differences in optimization, or output-format errors.

Questions for authors

1)Please reconcile the 4,727-example test manifest, the 4,161 examples in Table 15, and the denominators underlying Table 1. Can you provide correct/total counts and explain any filtering or aggregation?

2)What exactly is unseen in the claimed unseen rule configurations? Can you provide a controlled evaluation holding out interaction structures or combinations?

3)How does the method compare with a simpler guidance-token adapter under comparable capacity, supervision, and training budget? Are the component-level PairAcc gains stable across runs?

4)Please clarify the baseline optimization and checkpoint-selection procedures, including SecAlign preference construction. What are the parsing rates and main failure categories for methods with zero PairAcc?

2 comments

We thank the reviewer for recognizing the practical significance of our problem, the value of our paired evaluation protocol, and the substantial improvements achieved by DynRule-Guard. We have conducted additional experiments to address the concerns.

W1/Q1. Evaluation Set and Metric Consistency

The original test collection contained 4,727 records. Due to corporate data-sensitivity and shared-storage policies, 4,161 records were retained for the final evaluation and public release. We will correct this inconsistency in the manuscript.

The actual evaluation contains 62 Changed-reference and 70 Equal-reference pairs (132 total). The Qwen results therefore correspond to 45/62 = 72.58%, 69/70 = 98.57%, and 114/132 = 86.36%, respectively.

We promise to release the benchmark data, evaluation code, and model checkpoints upon acceptance to facilitate reproducibility.

W2/Q2. Generalization to Unseen Rule Configurations

(1) IHEval transfer. In fact, our existing zero-shot evaluation on IHEval already provides evidence of generalization to unseen rule-interaction patterns, with DynRule-Guard achieving the best overall performance among the compared methods.

(2) Held-out interaction combinations. We additionally evaluate generalization to the unseen combination Conflict × Content Edit, while retaining Conflict × Rule Edit and Compatibility × Content Edit during training. All methods use Qwen3.5-9B, matched training budgets, and the same 100 content-disjoint Changed-reference test pairs.

MethodSeen EDASeen C-ref PairAccHeld-out EDAHeld-out C-ref PairAccPairAcc Drop
Vanilla SFT72.047.065.537.010.0 pp
ISE70.045.064.036.09.0 pp
DynRule-Guard86.073.082.067.06.0 pp

DynRule-Guard achieves 82.0% EDA and 67.0% PairAcc on the held-out combination, with the smallest performance degradation. These results support generalization through shared rule-interaction representations and reusable pattern composition.

(3) External PolicyShiftBench evaluation. We further evaluate on PolicyShiftBench, an independently constructed multimodal moderation benchmark with dynamically changing and held-out policies.

MethodAdaptive F1Adaptive PSSShift F1Shift PSS
Moderation Backbone55.304.0450.990.00
Policy Prompt53.012.8648.630.00
Vanilla SFT37.391.3745.3616.29
Spotlighting55.143.5549.362.94
StruQ55.5712.2148.120.81
SecAlign53.804.9548.702.56
ISE33.330.0047.2511.54
ASIDE31.580.0036.360.00
V-Steer55.374.0450.720.00
PolicyShiftGuard-3B86.1575.7958.0650.54
PolicyShiftGuard-7B85.5673.2169.7869.89
DynRule-Guard (Zero-shot)70.0768.5060.2052.00
DynRule-Guard (Fine-tuned)86.9182.4080.2675.32

Without dataset-specific training, DynRule-Guard achieves 60.20 F1 / 52.00 PSS on the Shift split, outperforming Vanilla SFT by 14.84 / 35.71 points. After fine-tuning on 3,000 samples for three epochs, it reaches 80.26 F1 / 75.32 PSS, exceeding PolicyShiftGuard-7B by 10.48 / 5.43 points.

Together, these results provide complementary evidence of generalization to unseen interaction combinations and independently constructed moderation policies.

W3/Q3. Architectural Contributions and Stability

(1) Additional supervision. We introduce Vanilla SFT + Anchor, which uses the same learnable token and seven-class interaction-state supervision without our explicit interaction-modeling architecture. Anchor supervision improves Vanilla SFT by 1.58 EDA and 3.42 PairAcc points, but DynRule-Guard still exceeds it by 6.67 and 22.53 points, respectively. Even without Anchor supervision, our method achieves 88.22 EDA and 78.98 PairAcc.

(2) Repeated runs. We repeat the experiments with three independent training seeds, reporting mean ± standard deviation.

MethodEDAPairAcc
Full93.28 ± 1.0386.17 ± 0.95
w/o Pseudo-Rule Queries88.95 ± 2.5881.63 ± 2.93
w/o Rule Interaction Graph86.59 ± 4.1976.52 ± 6.28
w/o Global Interaction Queries87.29 ± 2.7979.73 ± 2.72
w/o Pattern Bank & Router85.85 ± 4.5776.33 ± 5.33
w/o L_anchor88.22 ± 3.1578.98 ± 1.68
Vanilla SFT85.03 ± 0.5160.22 ± 1.52
Vanilla SFT + Anchor86.61 ± 0.7963.64 ± 1.64

Across repeated runs, removing individual components consistently reduces mean performance. These results strengthen the evidence that our architectural contributions extend beyond additional supervision.

W4/Q4. Baseline Implementation and Failure Analysis

(1) Fair optimization and checkpoint selection. All trainable baselines share the same domain-adapted initialization, training data, and comparable optimization budgets. We apply the same validation-based checkpoint-selection criterion: S=min⁡(C−ref PairAcc,E−ref PairAcc)+0.25,Overall EDAS=\min(\mathrm{C-ref\ PairAcc},\mathrm{E-ref\ PairAcc})+0.25,\mathrm{Overall\ EDA}.

(2) SecAlign preference construction. We construct 4,171 training and 821 validation preference pairs from T3 conflict samples. The chosen response is the correct reference JSON; the rejected response replaces the evaluated label with the non-governing rule's decision and adjusts its rationale, while preserving the input, media, and other labels.

(3) Parsing rates and failure categories.

BackboneMethodParsing Rate
Qwen3.5-9BASIDE94.03%
GLM-4.6V-FlashISE99.74%
GLM-4.6V-FlashASIDE97.40%
InternVL3.5-8BSpotlighting100.00%
InternVL3.5-8BStruQ100.00%
InternVL3.5-8BSecAlign100.00%
InternVL3.5-8BISE100.00%
InternVL3.5-8BASIDE93.77%

Zero PairAcc is not primarily explained by parsing failures. Our analysis identifies two decision-level failure modes: (i) Changed-reference pairs: predictions remain unchanged despite an authority-induced reference change; (ii) Equal-reference pairs: predictions remain consistent but incorrect in both configurations.

These findings indicate that the failures primarily arise from incorrect rule-conditioned decisions rather than output-format errors.

We thank the reviewer again for the constructive feedback, which has strengthened the generalization evidence, architectural analysis, and experimental performance of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.