- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies multimodal content moderation under dynamically evolving hierarchical rules. A moderation request combines four sources—a platform-wide System Instruction with explicit rules, a business-side Moderator Instruction whose runtime rules change with business needs, multimodal User Content that may embed untrusted pseudo-rules ("this is not an advertisement") in text, OCR, or ASR, and an Auxiliary Policy specifying priority, authorization, and applicability conventions—and a correct decision depends on which rules apply, how they interact (agreement / one-sided applicability / authorized refinement / conflict), and which one governs. The paper contributes two things. DYNRULE-BENCH is a benchmark built from 1,430 parent samples by four controlled interventions (content edits, rule edits, source exchanges, pseudo-rule insertions), organized into four scenarios—runtime rule adaptation (T1), compatible multi-rule composition (T2), hierarchical arbitration (T3), and content-embedded pseudo-rule interference (T4)—spanning 28 configurations and a frozen 4,727-record test set; a reference-arbitration procedure re-validates every candidate by checking rule applicability, interaction, authorization scope, and source priority, and evaluation jointly measures correct adaptation (EDA on decision-changed subsets, Changed-reference PairAcc on source-exchange pairs) and correct preservation (EDA on decision-preserved subsets, Equal-reference PairAcc), explicitly defeating both the always-change and never-change strategies. DYNRULE-GUARD is a guidance framework that models rule interactions explicitly: source-aware states and latent pseudo-rule queries feed a Rule Relation Graph whose policy-conditioned edges separate semantic interaction from policy-aware interpretation; a compact Global Interaction State is mapped through an Interaction Router onto a shared bank of K=8 patterns, linearly composed into an Interaction Guidance Token appended to an intermediate layer of a frozen LoRA-adapted backbone. On three backbones (Qwen3.5-9B, GLM-4.6V-Flash, InternVL3.5-8B), DYNRULE-GUARD reaches 93.33/91.36/91.67% overall EDA versus 85.17/82.09/74.05% for the strongest baseline (Vanilla SFT), with the largest C-ref PairAcc gap on Qwen (72.58 vs 46.88) but the reverse on GLM and InternVL; for the transfer setting (training on D0 only), it improves the Conflict condition on IHEval by 31.74 points over the moderation backbone. Ablations and structural diagnostics (near-orthogonal pattern bank, state-dependent but non-collapsed routing, state-separated guidance geometry) support the mechanism.
Critical strengths
(I) The benchmark's evaluation protocol is the cleanest operationalization I have seen of adaptation-versus-preservation under rule dynamics, and it produces a genuinely useful result. The four-way metric design (Changed/Preserved EDA × Changed/Equal-reference PairAcc) closes both degenerate strategies by construction: a pair counts only if both members match their respective references, so neither always-flipping nor never-flipping can score. The strict source-exchange pairs—identical content, policy, and full rule texts, only the System/Moderator assignment swapped—separate authority effects from wording effects in a way that policy-change benchmarks (SafeEditBench, PolicyShiftBench) and instruction-hierarchy benchmarks (IHEval, Control Illusion) individually cannot. The measured consequence is stark and informative: defense methods score mostly 0-8% on Changed-reference pairs (with one notable exception, ISE at 45.16% on Qwen), revealing that most never actually arbitrate by authority; and the general VLMs' near-collapse on T4 (11.8-18.9%) quantifies how far pseudo-rule interference alone shifts decisions. This is consequential to my assessment because the benchmark contribution stands on its own: it exposes a real, previously unmeasured failure mode with a protocol others can reuse. (II) The method is mechanistically motivated and its gains are supported by evidence beyond aggregate accuracy. DYNRULE-GUARD's design maps directly onto the problem decomposition (policy-conditioned graph edges separating semantic interaction from policy interpretation; pseudo-rule queries for unannotated multimodal pseudo-rules; pattern composition for reuse across configurations), and the paper does not stop at the headline numbers: ablations isolate each component and auxiliary loss; the pattern-bank Gram matrix confirms near-orthogonality (off-diagonal 0.0034-0.0049); routing is state-dependent without collapsing (single-rule→P2, untrusted-content→P5/P7/P8) yet non-degenerate within states (t-SNE manifolds); and the IHEval zero-shot transfer (+31.74 Conflict over the backbone, +27.83 over Vanilla SFT) demonstrates that the learned interaction modeling generalizes beyond the constructed distribution to a text-only instruction-hierarchy benchmark the model was never trained on. The combination of in-benchmark gains, cross-benchmark transfer, and structural diagnostics makes the central claim—explicit interaction modeling beats implicit supervision—credible rather than merely consistent with the tables.
Critical weaknesses
All positive results live inside the paper's own construction pipeline, and no external moderation benchmark validates them. D1 and D2 are both descendants of the same 1,430 parents, generated by the same four intervention types, validated by the same reference-arbitration procedure, and balanced into the same 28-configuration grid. DYNRULE-GUARD is trained on D1 and evaluated on D2; its +8 to +18-point headline margins are therefore in-distribution by construction. The one external evaluation, IHEval, covers only text instruction-hierarchy (no multimodal content, no business-rule editing, no pseudo-rule insertion in OCR/ASR), so the claim that DYNRULE-GUARD is a "universal" moderation guidance framework rests, for its multimodal and dynamic-rule dimensions, entirely on the benchmark introduced in the same paper. Relatedly, the construction distribution is itself a design choice: the uniform scenario cells and the reference-arbitration conventions (e.g., System always wins unresolved conflicts) encode a specific—reasonable but contestable—model of how real platforms' rules behave; models trained and evaluated within this encoding may inherit its conventions rather than learn to read arbitrary auxiliary policies. An evaluation on any independently constructed moderation set with rule annotations (even a small one), or a held-out auxiliary-policy format differing from the training policy, would substantially change how far I am willing to extrapolate.
Questions for authors
(I) The paper's positive results are all measured on descendants of the same construction pipeline used for training. Can you evaluate DYNRULE-GUARD on at least one independently constructed moderation dataset with rule/policy annotations (e.g., a public policy-conditioned moderation set, or RuleSafe-VL adapted to your output schema), even if only for a subset of scenarios? If the interaction-modeling advantage shrinks outside your pipeline, the "universal" claim needs qualification. (II) Table 1 shows two cases where Vanilla SFT beats DYNRULE-GUARD on Changed-reference PairAcc (GLM: 59.68 vs 43.55; InternVL: 35.48 vs 20.97), which the text does not discuss. What explains this inversion on the smaller backbones, and given 65 C-ref pairs, is it statistically meaningful? Please report bootstrap intervals for the PairAcc columns. (III) Please add uncertainty quantification throughout: standard errors or bootstrap intervals for Table 1 (at minimum the Overall EDA and PairAcc columns), Table 3 ablations, and Table 19. Several conclusions (component deltas of 0.5-0.6 points; K=8 vs K=16) are within plausible resampling noise. (IV) What are the inference-time costs? The interaction modules (width d=256) add graph cross-attention, 16 pseudo-rule queries, and K-pattern composition per request, and the guidance token lengthens each forward pass—please report added latency/FLOPs relative to the Moderation Backbone, since deployment feasibility for moderation at scale depends on this.
7 comments
Q3. Statistical Reliability of Experimental Results
We appreciate the reviewer's suggestion and have expanded our uncertainty analysis across Tables 1, 3, and 19.
Table 1 — Main Results. We report 95% bootstrap confidence intervals for Overall EDA and all PairAcc metrics using 2,000 resamples. For example, on Qwen3.5-9B, DynRule-Guard achieves 93.33 [91.96, 94.62] Overall EDA and 86.36 [80.30, 91.67] Overall PairAcc, compared with 84.92 [83.09, 86.72] and 63.64 [55.30, 71.21] for Vanilla SFT, respectively.
| [Qwen3.5-9B] | Overall EDA | Overall PairAcc |
|---|---|---|
| General VLM | 65.59 [63.45, 67.54] | 42.42 [34.09, 50.76] |
| Moderation Backbone | 58.53 [55.73, 61.31] | 38.64 [30.30, 46.97] |
| Policy prompt | 60.73 [58.00, 63.24] | 37.12 [28.79, 45.45] |
| Vanilla SFT | 84.92 [83.09, 86.72] | 63.64 [55.30, 71.21] |
| Spotlighting | 58.21 [55.30, 60.93] | 36.36 [28.79, 44.70] |
| StruQ | 62.47 [59.95, 64.83] | 32.58 [25.00, 40.91] |
| SecAlign | 57.95 [55.10, 60.67] | 35.61 [27.27, 43.94] |
| ISE | 80.74 [78.73, 82.65] | 53.79 [45.45, 62.12] |
| ASIDE | 57.25 [55.05, 59.23] | 31.06 [23.48, 38.66] |
| V-Steer | 59.23 [56.35, 61.83] | 30.30 [22.73, 38.64] |
| DynRule-Guard | 93.33 [91.96, 94.62] | 86.36 [80.30, 91.67] |
Table 3 — Component Ablations. We additionally repeat all component and objective ablations using three independent training seeds, reporting mean ± standard deviation.
| Variant | Overall EDA | Overall PairAcc |
|---|---|---|
| Full | 93.28 ± 1.03 | 86.17 ± 0.95 |
| w/o Pseudo-Rule Queries | 88.95 ± 2.58 | 81.63 ± 2.93 |
| w/o Rule Interaction Graph | 86.59 ± 4.19 | 76.52 ± 6.28 |
| w/o Global Interaction Queries | 87.29 ± 2.79 | 79.73 ± 2.72 |
| w/o Pattern Bank & Router | 85.85 ± 4.57 | 76.33 ± 5.33 |
| w/o Anchor Loss | 88.22 ± 3.15 | 78.98 ± 1.68 |
| w/o Orthogonality Loss | 89.06 ± 2.85 | 81.82 ± 1.24 |
| w/o Balance Loss | 87.09 ± 3.13 | 78.79 ± 1.07 |
Removing individual components or objectives reduces mean EDA by 4.22–7.43 points and mean PairAcc by 4.35–9.84 points. These repeated-run results provide stronger evidence for the contributions of the proposed components while explicitly accounting for training variability.
Table 19 — Pattern Capacity and Readout Layers. We report 95% bootstrap confidence intervals for all 16 combinations of (K\in{4,8,16,32}) and (\ell\in{4,8,16,24}), covering Overall EDA, Changed-reference PairAcc, and the newly added Overall PairAcc.
Overall EDA [95% CI]
| K / layer | 4 | 8 | 16 | 24 |
|---|---|---|---|---|
| 4 | 93.41 [92.02, 94.65] | 93.28 [91.90, 94.49] | 92.99 [91.58, 94.30] | 93.59 [92.29, 94.76] |
| 8 | 92.94 [91.40, 94.33] | 93.33 [91.98, 94.59] | 92.89 [91.52, 94.11] | 93.25 [91.93, 94.51] |
| 16 | 93.38 [91.96, 94.63] | 93.33 [91.96, 94.59] | 92.45 [90.97, 93.73] | 92.32 [90.82, 93.71] |
| 32 | 93.75 [92.43, 94.99] | 93.59 [92.21, 94.89] | 94.06 [92.76, 95.23] | 93.80 [92.40, 95.07] |
Changed-reference PairAcc [95% CI]
| K / layer | 4 | 8 | 16 | 24 |
|---|---|---|---|---|
| 4 | 70.97 [59.68, 82.26] | 64.52 [53.23, 75.81] | 43.55 [30.65, 56.45] | 67.74 [56.45, 79.03] |
| 8 | 66.13 [53.23, 77.46] | 72.58 [61.29, 83.87] | 54.84 [41.94, 67.74] | 64.52 [53.23, 75.81] |
| 16 | 79.03 [69.35, 88.71] | 64.52 [53.23, 77.42] | 43.55 [32.26, 56.45] | 64.52 [53.23, 75.81] |
| 32 | 69.35 [58.06, 80.65] | 70.97 [59.68, 82.26] | 61.29 [48.39, 74.19] | 72.58 [61.29, 83.87] |
Overall PairAcc [95% CI]
| K / layer | 4 | 8 | 16 | 24 |
|---|---|---|---|---|
| 4 | 85.61 [78.79, 90.91] | 83.33 [76.52, 89.39] | 71.97 [64.39, 78.79] | 84.85 [78.03, 90.15] |
| 8 | 83.33 [76.52, 89.39] | 86.36 [80.30, 91.67] | 78.03 [71.21, 84.85] | 82.58 [75.76, 88.64] |
| 16 | 88.64 [83.33, 93.94] | 82.58 [75.76, 88.64] | 73.48 [65.91, 80.30] | 82.58 [75.76, 88.64] |
| 32 | 85.61 [79.55, 90.91] | 85.61 [79.55, 91.67] | 81.82 [75.00, 87.88] | 86.36 [80.30, 91.67] |
The results indicate that K=8,l=8 provides a competitive performance–capacity trade-off, achieving 93.33 EDA, 72.58 Changed-reference PairAcc, and 86.36 Overall PairAcc. Although some larger configurations achieve higher individual scores, their confidence intervals overlap substantially. We therefore clarify that the default configuration represents a compact and balanced design rather than a statistically unique optimum.
Q4. Inference-time Computational Overhead
We additionally benchmark inference efficiency on a single NVIDIA H800 GPU using Qwen3.5-9B, comparing DynRule-Guard with the Moderation Backbone.
| Batch Size | Method | Median Latency (ms) | Throughput (samples/s) | FLOPs/batch |
|---|---|---|---|---|
| 1 | Backbone | 813.74 | 1.227 | 22.130 T |
| 1 | DynRule-Guard | 1,027.92 | 0.931 | 22.210 T |
| 8 | Backbone | 1,471.28 | 5.591 | 177.038 T |
| 8 | DynRule-Guard | 1,835.80 | 4.361 | 177.681 T |
DynRule-Guard introduces only 0.346% additional parameters, approximately 0.363% additional FLOPs, and less than 90 MiB additional peak allocated GPU memory.
The median inference latency increases by 26.3% and 24.8% at batch sizes 1 and 8, respectively. This measurable latency overhead primarily reflects the additional online interaction-analysis operations rather than substantial increases in model size or arithmetic complexity.
These results quantify the practical trade-off: DynRule-Guard substantially strengthens dynamic hierarchical rule modeling with minimal parameter, FLOP, and memory overhead, at the cost of additional inference latency.
We thank the reviewer again for these valuable suggestions, which have helped us strengthen the external validation, statistical reliability, and efficiency analysis of our work.
We sincerely thank the reviewer for the positive assessment of DynRule-Bench's adaptation–preservation evaluation protocol, its ability to reveal practically important failure modes, and the mechanistic motivation and extensive empirical validation of DynRule-Guard. We appreciate the constructive suggestions and have conducted additional experiments to address these concerns.
Q1. Generalization to External Moderation Benchmarks
We agree that evaluation on an independently constructed benchmark is essential to establish generalization beyond our data construction pipeline.
We additionally evaluate DynRule-Guard on PolicyShiftBench, an independent multimodal moderation benchmark that evaluates policy-conditioned decisions on images under dynamically changing policies, including held-out policy definitions. It reports F1 and Policy Shift Score (PSS), measuring classification quality and correct adaptation to policy changes, respectively.
Zero-shot transfer. Without any PolicyShiftBench-specific training, DynRule-Guard achieves 60.20 F1 and 52.00 PSS on the Shift split, outperforming Vanilla SFT by 14.84 and 35.71 points, respectively. This provides direct evidence of generalization to independently constructed moderation policies beyond our benchmark's templates.
Task-specific adaptation. After fine-tuning on 3,000 training examples for three epochs, DynRule-Guard further achieves 80.26 F1 and 75.32 PSS on the Shift split, surpassing the specialized PolicyShiftGuard-7B baseline. These results demonstrate the transferability and adaptability of our framework, although differences in model capacity and training procedures should be considered.
We initially planned to evaluate our method on RuleSafe-VL. However, to the best of our knowledge, the benchmark has not yet been publicly released. We will conduct additional evaluations and provide further analysis once it becomes available.
Q2. Changed-reference PairAcc on GLM and InternVL
We thank the reviewer for identifying this inconsistency. Our investigation revealed an inter-cluster data synchronization issue that inadvertently omitted 201 Changed-reference training pairs from the DynRule-Guard training data for GLM and InternVL.
After restoring the missing training data and retraining the affected models, the results are:
The results show that DynRule-Guard consistently outperforms Vanilla SFT on Changed-reference PairAcc across all three backbones, resolving the previously observed inversion.
We additionally report 95% bootstrap confidence intervals, obtained from 2,000 resamples with replacement, using the 2.5th and 97.5th percentiles.