- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RegulaBench, a benchmark derived from real-world enforcement cases to evaluate multimodal models as advertising pre-regulators across three progressive tiers: violation detection, evidence-to-norm grounding, and regulatory fidelity. Although models align well with humans in detecting violations, their legal justifications diverge significantly by favoring specific content prohibitions over broader truthfulness norms, revealing fundamental fidelity challenges in scalable AI-assisted regulation.
Critical strengths
-
The paper moves beyond the limitations of conventional content moderation that relies solely on binary violation decisions, establishing a multi-tiered evaluation framework that spans violation detection, statutory grounding, and normative selection.
-
Rather than relying solely on static end-to-end scoring, the framework design incorporates multi-stage interventional experiments.
Critical weaknesses
-
The advertisement images in the benchmark are synthetically reconstructed via text-to-image models based on factual descriptions in administrative penalty decisions, rather than being authentic ad materials collected from actual law enforcement cases. These synthetic images inevitably carry visual artifacts and oversimplification tendencies typical of generative models, failing to reflect the highly intricate, covert, and multimodal nature of genuine illegal advertisements encountered in practice.
-
Actual administrative penalty rulings depend on comprehensive offline investigative transcripts, commercial context, and administrative discretion, whereas the evaluated model receives only an isolated image under a zero-shot setup. This fundamental information asymmetry suggests that disparities between model outputs and human statutory citations are far more likely driven by missing background context and statute retrieval failures, rather than an inherent model preference regarding normative abstraction levels as claimed by the authors.
-
The evaluation is limited to only 600 images, offering an insufficient number of complex instances that involve multiple intersecting regulations. Furthermore, evaluating fine-grained statutory memorization without RAG or hierarchical legal prompts causes the multi-statute match rate to drop near zero, making it impossible to disentangle knowledge recall deficiencies from genuine legal reasoning capabilities.
Questions for authors
see Critical Weaknesses
Considering that there is no rebuttal from the authors, I will keep my score unchanged.