- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper is a benchmarking paper that in the area of evaluating the regulatory compliance ability of LLMs. More specifically, the authors look at the areas of privacy, copyright protection and content safety. For privacy protection, they consider several regulatory jurisdictions. Six jurisdictions have been considered that include China, EU, US, Saudi Arabia, Turkey and Thailand. For copyright laws compliance, the focus is on reproduction oriented requests to models. Again, the six jurisdictions have been covered. For content safety, the scenarios considered are Knowledge Q&A, Value Denigration and Illegal Assistance. This has been specifically catered for three jurisdictions of Turkey, Saudi Arabia and Thailand. Using a base collection for these three areas (not exactly clear how many), it has been further augmented using (a) adversarial augmentation using 15 jailbreak methods and (b) 5 generalization enhancement methods. This leads to the total benchmark dataset of 37,433 cases with multilingual and multicultural splits. Using this comprehensive dataset, 24 LLMs have been evaluated as summarized in Table 32. This table shows that all tested models perform poorly on the proposed benchmark.
Critical strengths
(1) It is an ambitious benchmarking and evaluation effort across various compliance issues. The inclusion of multiple jurisdictions, languages and cultures is a real strength. The adversarial enhancements add value to the dataset.
(2) The evaluation has been thorough and the results, as summarized in the main paper tables as well as the appendix tables, are interesting.
(3) The localization aspect of the benchmark evaluation is interesting, as highlighted in Section 4.3. Given that most models are meaningful to users in their own jurisdiction, localized evaluation that is sensitive to the local language and culture is indeed very valuable.
Critical weaknesses
(1) The introduction and the narrative is not at all clear. What is the problem being addressed in the paper? What are the existing gaps identified in the works listed in Table 1 that the PrismEval benchmark overcomes? To what extent have the identified gaps have been plugged? This is not at all obvious to the reader. Given that this is a benchmarking paper, some quantitative assessment of the progress due to the proposed work should be documented.
(2) It is not clear why the six particular regulatory regimes of China, EU, US, Saudi Arabia, Turkey and Thailand have been chosen. The reasoning must be explained. Moreover, for culture, why Saudi Arabia, Turkey and Thailand have been chosen needs to be explained.
(3) The testing results summarized in Table 32 show that only small LLMs have been tested against the proposed benchmark. Those results clear show that these models need a lot of improvement with respect to regulatory compliance. However, it is not clear whether this generalizes to larger models? Does the violation rate drop as the models (in the same model family) become bigger? Also, if possible, it would be useful to know what is the performance of at least one frontier model. It is understandable that this may be difficult to evaluate due to cost or resource constraints.
(4) The writing is confusing and complex. The paper is difficult to read and understand.
Questions for authors
(1) It would be very useful to the the readers to concretely highlight the novel contribution of this benchmark, which the previous benchmarks are not able to capture.
(2) The performance behavior across models of different sizes of the same family will be very useful for the reader. Do large models have better performance on the benchmark?
(3) It is not clear to the reader how many of the 37,433 cases totally new and how many have been re-used or re-purposed from some other existing datasets. Please state clearly.
Dear Reviewer Gwg2,
Thank you for your constructive review. We clarified the dataset contribution, jurisdiction choices, and model-size comparisons.
W1: Unclear problem, gaps relative to Table 1, and quantitative progress.
PrismEval asks how models respond when protected information, protected expression, or locally applicable requirements change. Its three response targets determine construction and judgment: Privacy protects sensitive attributes while testing appropriate public-information answers; Copyright anchors reproduction assessment to protected works; Content Safety specifies the requirement and language context for each case (Sections 1 and 3.3–3.4).
The results show why these targets matter. For Qwen3-32B, moving from Direct to Reasoning Conditions raises the English protected-request score-0 rate from 5.00% to 25.10%, while public-information false refusal falls from 15.00% to 5.50% (Section 4.3; Appendix I.4). A single refusal score obscures this tradeoff.
Paired generalization raises Qwen3-30B-A3B's Privacy violation rate by 2.23 points and lowers Mistral-small-2509's by 4.08 points on the same retained seeds (Section 4.3/Table 2). Table 1 compares benchmark properties; Figures 3–4 and the paired analysis show the model differences exposed by PrismEval. These measure within-benchmark effects; equal-budget external comparisons remain separate experiments.
W2: Why these six jurisdictions and three cultural settings?
Section 3.1 explains the selection. China, the EU, and the US supply contrasting regulatory approaches and documented sources. Saudi Arabia, Türkiye, and Thailand add Arabic, Turkish, and Thai settings with distinct requirements and language-review materials.
Beyond translation, Dialectal Transformation applies local expression rules; Local Model Rewriting preserves protected fields and request intent. Several dozen legal and language reviewers worked over several months, with multiple professors overseeing the work and multi-reviewer adjudication resolving disagreements (Appendix D.1). Section 4.4 evaluates the complete language–jurisdiction settings.
W3: Larger models, within-family scaling, and frontier models.
Section 4.2 and Appendix I.5 now report within-family changes in violation rate. Columns: Privacy, Copyright, Value Denigration, Illegal Assistance. Negative changes favor larger checkpoints (percentage points).
Vicuna 13B minus 7B: -1.05, -0.14, -6.49, -4.84. Llama-2 13B minus 7B: +0.32, -0.54, +3.80, +3.10. Qwen3 32B minus 8B: -0.55, -0.29, +0.16, -0.95.
Vicuna improves in four fields, Qwen3 in three, and Llama-2 in one, showing family- and risk-dependent scaling.
The base Copyright diagnostic includes DeepSeek-v3.2 and Claude-sonnet-4: 6.32% and 5.12% of their responses exceed either source-matching threshold (Section 5.1; Appendix F.3). This supplies frontier-model evidence for base protected-text reproduction, a complementary setting to strengthened four-field evaluation.
W4: Confusing and complex writing.
We reorganized the explanation around the tested behavior, construction source, applied transformations, and observed failures. The three contributions now correspond to the resource, validated request variation, and model evaluation with optimization tradeoffs. Key evidence appears in Section 4.2 (model sizes), Section 4.3/Table 2 (Privacy controls and paired generalization), and Section 4.5/Table 5 (ranking retention), with complete profiles in Appendix I.
Q1: What does PrismEval capture beyond previous benchmarks?
Public-information controls expose false refusal; protected works anchor reproduction assessment; local requirements specify expected content behavior. Adversarial and reviewed linguistic variants test these targets across request forms. Opposing Privacy outcomes and model-dependent generalization effects support the dataset design (Sections 1, 3, and 4.3).
Q2: Do larger models perform better within a family?
Larger checkpoints improve different fields in different families (W3; Appendix I.5).
Q3: How much of the 37,433-case suite is new versus reused?
Appendix I.1 distinguishes requests, source materials, and variants. The suite contains 17,208 Chinese/English and 20,225 Turkish/Arabic/Thai cases. Requests are constructed from protected data, documents, local requirements, and templates, then expanded into seed-linked variants. Regulations, protected works, and attack procedures are inputs; external benchmarks support coverage mapping. The total counts prompt instances, including shared-seed variants. Exact cross-benchmark textual-overlap percentages have not been measured; construction provenance and parent-seed counts specify the resource's origin and composition.
We sincerely hope you could raise your score based on our clarification!