acceptodds
Under review as a conference paper at ICLR 2027

Can AI Help Regulate AI? Benchmarking Multimodal Models as Advertising Regulators

Abstract

AI generation is scaling far faster than human regulation. Generative models enable regulated content to be produced at machine speed and near-zero marginal cost, with multimodal advertising generation providing a prominent example, while regulatory review remains constrained by scarce human attention. This growing asymmetry creates a **scalable regulation problem**: can AI help humans regulate AI-generated content at the scale at which it is produced? We introduce **RegulaBench**, a benchmark for evaluating multimodal models as advertising **pre-regulators**—scalable assistants that screen potentially unlawful content, organize legally relevant evidence, and provide grounded regulatory recommendations for subsequent human review. Built from real advertising enforcement cases, RegulaBench links generated advertising images to the factual findings and legal grounds recorded in corresponding administrative penalty decisions. It evaluates three progressively demanding capabilities: **violation detection**, whether models identify advertisements that trigger regulatory intervention; **evidence-to-norm grounding**, whether they connect case-specific evidence to the conditions governing applicable legal provisions; and **regulatory fidelity**, whether their selection and application of legal norms remain aligned with the structure of human regulatory decisions. Our experiments reveal a two-sided picture of AI-assisted regulation. On the one hand, across a substantial share of cases, model judgments are consistent with those of human regulators, suggesting that AI can help extend human regulatory capacity in screening potentially unlawful advertising at scale. On the other hand, this agreement becomes substantially weaker once we move beyond regulatory outcomes to examine how those outcomes are legally justified. In particular, across the provision groups examined, models show a directional tendency to rely more heavily on specific content prohibitions, whereas human enforcement decisions more often invoke broader truthfulness provisions. These findings reveal an important tension in scalable AI-assisted regulation: **AI can scale regulation without faithfully reproducing existing patterns of human regulation**. Such divergence raises consequential questions about how AI pre-regulators should be integrated into human regulatory systems.

Primary area:
datasets and benchmarks
Keywords:
AI-Assisted Regulation, Multimodal Large Language Models, Legal Reasoning, Regulatory Benchmarking, Advertising Compliance, Human–AI Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_XLdKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RegulaBench, a benchmark derived from real-world enforcement cases to evaluate multimodal models as advertising pre-regulators across three progressive tiers: violation detection, evidence-to-norm grounding, and regulatory fidelity. Although models align well with humans in detecting violations, their legal justifications diverge significantly by favoring specific content prohibitions over broader truthfulness norms, revealing fundamental fidelity challenges in scalable AI-assisted regulation.

Critical strengths
  1. The paper moves beyond the limitations of conventional content moderation that relies solely on binary violation decisions, establishing a multi-tiered evaluation framework that spans violation detection, statutory grounding, and normative selection.

  2. Rather than relying solely on static end-to-end scoring, the framework design incorporates multi-stage interventional experiments.

Critical weaknesses
  1. The advertisement images in the benchmark are synthetically reconstructed via text-to-image models based on factual descriptions in administrative penalty decisions, rather than being authentic ad materials collected from actual law enforcement cases. These synthetic images inevitably carry visual artifacts and oversimplification tendencies typical of generative models, failing to reflect the highly intricate, covert, and multimodal nature of genuine illegal advertisements encountered in practice.

  2. Actual administrative penalty rulings depend on comprehensive offline investigative transcripts, commercial context, and administrative discretion, whereas the evaluated model receives only an isolated image under a zero-shot setup. This fundamental information asymmetry suggests that disparities between model outputs and human statutory citations are far more likely driven by missing background context and statute retrieval failures, rather than an inherent model preference regarding normative abstraction levels as claimed by the authors.

  3. The evaluation is limited to only 600 images, offering an insufficient number of complex instances that involve multiple intersecting regulations. Furthermore, evaluating fine-grained statutory memorization without RAG or hierarchical legal prompts causes the multi-statute match rate to drop near zero, making it impossible to disentangle knowledge recall deficiencies from genuine legal reasoning capabilities.

Questions for authors

see Critical Weaknesses

Reviewer_2HKxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RegulaBench, a corpus of 6,011 generated advertising images derived from enforcement cases and legal provisions. Its experiments target 11 multimodal models on a 600-image set, comprising 400 case-derived positives and 200 human-reviewed negatives. Nine models have complete API coverage; the two Claude models have provisional results on 584 images each (Table 7). The evaluation separates violation detection, conditional law-name identification and provision matching, and comparison with the norm types represented in enforcement decisions. Two interventions add a description of visible image content or the correct law name. The reported results show that high violation recall can coexist with poor specificity and that complete provision coverage is low. The abstract and introduction further argue that models exhibit directional differences from human regulators in their selection of legal norms.

Critical strengths
  • S1. The evaluation asks a useful question beyond binary detection. Separating detection, law identification, and provision coverage exposes limitations that detection accuracy alone would conceal. For example, Table 1 reports 100% recall but only 11.5% specificity for Gemini-2.5-Pro, while Table 3 shows poor complete provision coverage across models. Reporting invalid outputs and explicitly distinguishing conditional provision coverage from exact matching are also useful choices. This provides a worthwhile basis for an evaluation resource, subject to the validity concerns below.
Critical weaknesses

W1. The validity of the reconstructed images and their legal reference labels is not established

Evidence. Sections 3.1 and 3.2 describe generating images from factual descriptions in enforcement decisions and retaining links to the source provisions (lines 145-161). Subsequent evaluations call the reference sets image-supported, but the manuscript does not explain how this support was independently verified for each retained law and provision. It provides no worked case-to-image-to-label example, annotator qualifications or agreement results, or adjudication procedure for deciding which source grounds survive reconstruction. The origin and sampling procedure for the 200 negative images are also insufficiently described.

This distinction is central to the task. Rendering an advertising claim does not itself establish its truth or falsity, and facts available to an enforcement authority may be absent from a reconstructed image. The paper acknowledges that models and regulators did not receive identical evidence (lines 137-138 and 481-483), but does not provide the validation needed to show that the retained labels remain answerable from the model's input. Calling them image-evident does not resolve this issue.

The missing documentation directly limits scrutiny of the benchmark. The reproducibility statement says that the appendix contains parsing, normalization, and adjudication procedures (lines 507-509), whereas the supplied appendix contains only Table 7 and its explanatory notes. The image-generation model, actual generation/evaluation prompts, and detailed label-validation protocol are not provided in this PDF.

Impact on recommendation. These omissions affect the central experimental target, rather than peripheral implementation details. Without a documented image-level legal audit, disagreement with the references cannot be reliably attributed to model failure rather than missing evidence or reconstruction-induced label changes. Establishing benchmark validity may require relabeling and reevaluation, which makes this a blocking concern.

W2. The reported analysis does not establish the central claim about directional norm-selection preferences

Evidence. The abstract and introduction claim a directional shift from broad truthfulness provisions toward specific content prohibitions (lines 33-36 and 92-100). However, Table 4 compares 4,646 model citations absent from the reference sets with 8,554 human reference-provision appearances. It excludes correct model citations. This comparison describes the composition of unmatched citations, not the distribution of all model norm selections or the direction of changes within matched cases. For example, the near absence of principles among unmatched citations cannot establish that models rarely select principles overall without also examining their matched citations.

The table reports a Rule share of 54.0% for unmatched model citations and 62.6% for human references, alongside 17.8% Mixed citations for models. These marginal distributions do not demonstrate the claimed directional shift. Indeed, Section 4.3 describes its finding as concentration rather than a uniform shift across norm types (lines 372-384). I could not find a case-level directional comparison or uncertainty estimate for the headline asymmetry. Moreover, among reliably paired law-article citations in cases with a valid affirmative judgment, an article is counted as wrong when its law-article pair is absent from the image's reference set (lines 358-370). The PDF does not document an applicability assessment of these unmatched grounds. The introduction itself recognizes that a mismatch may reflect an alternative permissible ground rather than a legal error (lines 59-66).

The interventions do not resolve these ambiguities. Section 1 promises factual context, relevant legal provisions, and their combination (lines 88-91). Section 4.4 instead supplies visible-scene descriptions or a law name, without provision text, and reports detection repair or provision recovery rather than norm-selection shifts. Finally, the unlawful-content and justification fields are explicitly unscored (line 170), so citation overlap does not establish the evidence-to-norm reasoning capability advertised in the abstract.

Impact on recommendation. The paper establishes descriptive citation mismatches, but its principal interpretive contribution requires distinguishing legal error, permissible alternative grounds, and norm preference under comparable evidence. The current analysis does not make those distinctions. Supporting that contribution requires a different analysis and stronger controls, or a substantial narrowing of the claims.

Questions for authors
  1. What independent image-level audit established that every retained reference law and provision is supported by the generated image alone? Please provide the annotation/adjudication protocol, agreement and exclusion results, and representative source-decision/image/label examples, together with the negative-image sampling procedure. Existing evidence of a rigorous audit would materially reduce W1.

  2. Is there a matched-case analysis of directional norm selection that includes all model citations, adjudicates alternative applicable grounds, and controls the evidence and legal information available to the decision-makers? Please provide the transition counts and uncertainty estimates, and clarify where the factual-context/provision-text conditions described in Section 1 are reported. Such results could materially change W2; Table 4 alone cannot do so.

1 comment

Considering that there is no rebuttal from the authors, I will keep my score unchanged.

Reviewer_nA5uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RegulaBench, a corpus of 6,011 generated advertising images: 2,549 case-derived images that reconstruct the facts of administrative penalty decisions published by China's SAMR, and 3,462 rule-derived images that instantiate statutory provisions. The experiments use a 600-image evaluation set centered on the case-derived track (l.150–151). It contains 400 case-derived positives, each linked to a set of image-supported legal references, and 200 human-reviewed images with no image-evident violation; the paper does not state which track these 200 come from. Given only the image, each of 11 multimodal models returns a violation judgment, the applicable laws, and specific provisions. The evaluation has three parts: (1) a staged baseline evaluation that scores violation detection, law-name identification, and provision grounding separately; (2) a comparison of the norm types (Rule, Standard, Principle, Mixed) of model citations outside the reference set with those of the reference provisions; and (3) two targeted interventions. The first adds a description of visible scene content to the 38 violation images that GPT-4o missed and re-queries all 11 models on them. The second applies to single-law images on which a model detected the violation but named neither the gold law nor any gold provision; there, the model is given the correct law name. Separately, the abstract states that models lean toward specific content prohibitions, while human decisions more often invoke broader truthfulness provisions. Main results: most models favor sensitivity over specificity (e.g., Gemini-2.5-Pro has 100% recall and 11.5% specificity), and GPT-4o has the highest accuracy (88.0%) and specificity (83.0%). Complete law-set matching drops sharply on multi-law images, to 0% for five models on two-law images. The best provision all-hit rate is 39.8% on single-provision images, falling to 5.5% and 2.1% on two- and three-provision images. Four Advertising Law articles account for 63.2% of the citations outside the reference set. Scene descriptions repair 22 of GPT-4o's 38 false negatives, and law-name cues rarely recover complete provisions (e.g., 1/68 for GPT-4o).

Critical strengths

S1. The staged evaluation exposes a real gap. Because the paper scores Violation Detection separately from Law Identification and Provision Grounding, it can show that agreement on the outcome may hide weak legal grounding. On two-law images, five models have 0% complete law-set matches, yet each names exactly one of the two reference laws in at least 77.4% of its retained cases (Table 2, l.296–299). The best provision all-hit rate falls from 39.8% on single-provision images to 2.1% on three-provision images (Section 4.2.2, l.321–323). This matters because it is the paper's clearest and most practically relevant finding: a benchmark that scores only binary judgments would overstate how useful these models are as pre-screening tools. The exact size of the gap depends on how the reference sets were built.

S2. Detection results are reported transparently. Table 7 gives TP/FN, TN/FP, invalid outputs, API coverage, and Wilson intervals. Invalid outputs stay in the denominators, models with incomplete API coverage are excluded from the Table 1 ranking (l.219–224), and the text states plainly that high recall does not imply reliable screening (l.245–252): Gemini-2.5-Pro flags all 400 violations but produces 177 false positives. This makes the detection numbers recomputable from the reported counts and supports the paper's practical point about false-positive burden. How to interpret those numbers still depends on how the 200 negatives were selected and reviewed.

Critical weaknesses

The headline norm-selection claim is not supported. The abstract (l.034–036) states that, across "the provision groups examined," models show a directional tendency toward specific content prohibitions, whereas human decisions more often invoke broader truthfulness provisions. The introduction (l.092–094) goes further, attributing to a "preliminary analysis" the finding that models more often shift from truthfulness provisions toward specific content prohibitions than the reverse. I see three problems. First, that analysis is not reported, and the provision groups are not identified. Second, Section 4.3 gives only marginal distributions (Table 4, apparently pooled across the 11 models) and describes its result as "concentration rather than a uniform shift toward one of the three norm types" (l.372). Table 4 does not support even the abstract's marginal claim. Among model wrong citations, the Rule share is 54.0% with Mixed included, below the reference share of 62.6%; it is about 65.7% vs. 62.6% with Mixed excluded; and if all Mixed citations were coded as Standard, the model Standard share would be higher instead (45.7% vs. 35.3%). Mixed appears only on the model side, plausibly because gold provisions include paragraphs and items (l.313–314) while model citations are compared at the article level (l.359); Article 9 alone accounts for about 83% of Mixed (l.376–377). Third, the paper restricts references to image-supported provisions (l.138, l.265) and acknowledges that models and regulators did not see the same evidence (l.138–139, l.482–483). However, the factual-context, legal-provision, and combined conditions promised at l.088–091 to separate preference from information deficits are not reported, and the two interventions that were run supply only visible-scene text or a law name and do not report norm type. A difference in norm type therefore cannot be separated from what a single image allows a model to establish. This is consequential because the paper presents the norm-selection analysis as its advance over citation-accuracy benchmarks (l.113–115, l.131–139).

Questions for authors
  1. Which analysis supports the directional claim in the abstract (l.034–036) and introduction (l.092–094), and which provisions are "the provision groups examined"? Can you report, per model, case-level paired counts of Standard→Rule and Rule→Standard shifts, the number of opportunities for each, and intervals clustered by source decision? Is the 8,554 in Table 4 pooled over reference-provision appearances in all 11 models' retained cases? If paired results cannot be provided, I believe the directional claim should be removed from the abstract and introduction, and Figure 1 (E3), which shows "Matched Case-Model Pairs" feeding "Direction and concentration," should be revised to match the analysis actually reported in Section 4.3.
  2. How was each image's image-supported reference set derived from the provisions cited in its source decision? Were any cited provisions excluded, and if so, how many of each norm type? Does the gold set include penalty-basis or discretionary provisions (e.g., the mitigation provisions of the Administrative Penalty Law)? Who annotated the references, and what was the inter-annotator agreement?
  3. Which generation track and which source cases do the 200 negatives come from, and what is the eligibility criterion at l.216–217? Did the human review of the negative images check for mandatory disclosures (e.g., those required by Advertising Law Art. 16(2) and Art. 18(2))?
  4. In every row of Table 2, Exact + Over-cite = 100%, so the column labelled over-citation (l.268) is 1 − Exact and also contains misses. For Qwen3-VL-Plus, Qwen3-VL-Flash, GPT-4o, and GPT-4o-mini, the single-law counts implied by this column (77, 72, 68, 73) equal their Table 6 targets, i.e., cases naming neither the gold law nor any gold provision (l.424–426), so for these four models the column consists entirely of misses. For DeepSeek-V4.1-Flash and Gemini-2.5-Flash, the 58.1% and 49.6% cited at l.293–294 include 13.7 and 16.2 points of misses (100% − 86.3%, 100% − 83.8%). Could you split the column into "gold law named, with additions" and "gold law missed," and report the 0/k…k/k distribution for multi-law images that l.270 says is preserved in the audit data? This bears on how much of the law-level gap in S1 reflects missing laws versus extra laws.
Reviewer_k3N5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AI systems can be used to assist humans in regulatory decisions, however the past benchmarks do not study the nuanced reasoning that goes in regulatory decisions, i.e. what norms (rules, standards, principles) are applied or whether AI models can do so, and have instead focused on the correctness of the outcome. The paper's goal is to present a bechmark to study norm-selection preference in AI systems. It uses advertising images as a case study for that, and presents a bechmark of advertising images from real Chinese administrative penalty decisions, with each image linked to the provisions cited in its source decisions. Models are then evaluated on violation detection, on identifying applicable laws and provisions, and on how the types of norm they cite compare with those cited by regulators.

Critical strengths

The paper certainly tackles one core problem in current models: i.e. testing for their (normative) reasoning in a challenging case of regulation, as opposed to just outcome matching or verification. I know of other emerging works on testing models for normative competence, but it focuses primarily or moral reasoning, so it is interesting to see the same thing for regulatory work as well.

The created bechmark derives from real-world Chinese advertising cases, as opposed to synthetically created. And the extensive experiments are done on a range of models.

Critical weaknesses

My main concern, however, is the paper's stated goals and its experiments (on my reading) do not match. The abstract frames the work as asking whether AI can help regulators keep pace with AI-generated content, and primarily motivates whether it can be used for specific regulatory tasks. However, the introduction positions primarily on the norm-selection preference: to study whether models systematically invoke different types of legal norm than humans when several grounds apply, with advertising as a test case. Even the bechmark name is different in abstract and the intro. The experiments, on my reading, addresses the abstract (detection, law and provision matching, and error repair), while the norm-selection preference (by far the most interesting one) as well as presented as a main contribution in the intro, is weakly supported. Below are the specific concerns:

  1. The dataset has regulator cited norms and decisions, and the model outputs are evaluated against it. However, for a task that naturally has many possible grounds available for evaluation, it is not clear if the regulators have cited all the possible grounds that apply to the case. It is possible the regulator cites just enough to justify the penalty or decision, not every applicable one, so a citation that is not in taht set can very well be a legitimate alternative rather than just wrong. However, both are treated as wrong. On my reading, this test agreement with regulators, as opposed to preference.

  2. Additionally, the norm-type analysis is error analysis. It compares the types of the model's out-of-reference citations with the types of all reference provisions. This is a valid analysis to study is there a systematic tendency of the model to cite. However, well-reasoned test for norm-preference selection would require something like a test case where one collects every provision that legally applies to the case (i.e. having legal experts annotate some examples with all legal provisions that apply to the image), and then comparing the actual regulator's preference as well as the model's.

Questions for authors

see above.

Overall, my summary is the paper has right motivation and an interesting problem, however the paper falls short of reasonably establishing the norm-selection preference that it sets out to. I can point to the authors to [1] that might be useful.

[1] Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs. In IASEAI'26.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.