acceptodds
Under review as a conference paper at ICLR 2027

PrismEval: A Benchmark for Fine-Grained LLM Compliance Evaluation

Abstract

LLM compliance spans privacy, copyright, and content safety, and the behavior under evaluation changes with the applicable jurisdiction, language, cultural context, and request form. We introduce PrismEval, a fine-grained benchmark that constructs each dimension from its own source evidence: protected data and subject types for Privacy, protected works and reproduction requests for Copyright, and legal, policy, official, and cultural materials for Content Safety. PrismEval covers five languages and strengthens the resulting test cases through 15 jailbreak methods, five controlled transformations, Dialectal Transformation, and Local Model Rewriting. The complete Jailbreak Enhancement collection contains 32,238 cases, while the multilingual Privacy and Content Safety suite contains 37,433 cases. Expert audits report usable rates of 100% for Content Safety, 100% for Copyright, and 98% for Privacy; both local transformation procedures also produce response differences exceeding three times their within seed baselines. Across 24 models, mean violation rates range from 85.9% to 96.5% across the reported risks, while the lowest overall violation rate remains 78.6%. Five Test Case Optimization Procedures reduce evaluation volume by 40.17% while retaining the delivered model performance characterization. PrismEval exposes large, risk-specific compliance failures across dimensions, languages, and localized requirements.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmarking, Compliance Evaluation, Jailbreak Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Gwg2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper is a benchmarking paper that in the area of evaluating the regulatory compliance ability of LLMs. More specifically, the authors look at the areas of privacy, copyright protection and content safety. For privacy protection, they consider several regulatory jurisdictions. Six jurisdictions have been considered that include China, EU, US, Saudi Arabia, Turkey and Thailand. For copyright laws compliance, the focus is on reproduction oriented requests to models. Again, the six jurisdictions have been covered. For content safety, the scenarios considered are Knowledge Q&A, Value Denigration and Illegal Assistance. This has been specifically catered for three jurisdictions of Turkey, Saudi Arabia and Thailand. Using a base collection for these three areas (not exactly clear how many), it has been further augmented using (a) adversarial augmentation using 15 jailbreak methods and (b) 5 generalization enhancement methods. This leads to the total benchmark dataset of 37,433 cases with multilingual and multicultural splits. Using this comprehensive dataset, 24 LLMs have been evaluated as summarized in Table 32. This table shows that all tested models perform poorly on the proposed benchmark.

Critical strengths

(1) It is an ambitious benchmarking and evaluation effort across various compliance issues. The inclusion of multiple jurisdictions, languages and cultures is a real strength. The adversarial enhancements add value to the dataset.

(2) The evaluation has been thorough and the results, as summarized in the main paper tables as well as the appendix tables, are interesting.

(3) The localization aspect of the benchmark evaluation is interesting, as highlighted in Section 4.3. Given that most models are meaningful to users in their own jurisdiction, localized evaluation that is sensitive to the local language and culture is indeed very valuable.

Critical weaknesses

(1) The introduction and the narrative is not at all clear. What is the problem being addressed in the paper? What are the existing gaps identified in the works listed in Table 1 that the PrismEval benchmark overcomes? To what extent have the identified gaps have been plugged? This is not at all obvious to the reader. Given that this is a benchmarking paper, some quantitative assessment of the progress due to the proposed work should be documented.

(2) It is not clear why the six particular regulatory regimes of China, EU, US, Saudi Arabia, Turkey and Thailand have been chosen. The reasoning must be explained. Moreover, for culture, why Saudi Arabia, Turkey and Thailand have been chosen needs to be explained.

(3) The testing results summarized in Table 32 show that only small LLMs have been tested against the proposed benchmark. Those results clear show that these models need a lot of improvement with respect to regulatory compliance. However, it is not clear whether this generalizes to larger models? Does the violation rate drop as the models (in the same model family) become bigger? Also, if possible, it would be useful to know what is the performance of at least one frontier model. It is understandable that this may be difficult to evaluate due to cost or resource constraints.

(4) The writing is confusing and complex. The paper is difficult to read and understand.

Questions for authors

(1) It would be very useful to the the readers to concretely highlight the novel contribution of this benchmark, which the previous benchmarks are not able to capture.

(2) The performance behavior across models of different sizes of the same family will be very useful for the reader. Do large models have better performance on the benchmark?

(3) It is not clear to the reader how many of the 37,433 cases totally new and how many have been re-used or re-purposed from some other existing datasets. Please state clearly.

1 comment

Dear Reviewer Gwg2,

Thank you for your constructive review. We clarified the dataset contribution, jurisdiction choices, and model-size comparisons.

W1: Unclear problem, gaps relative to Table 1, and quantitative progress.

PrismEval asks how models respond when protected information, protected expression, or locally applicable requirements change. Its three response targets determine construction and judgment: Privacy protects sensitive attributes while testing appropriate public-information answers; Copyright anchors reproduction assessment to protected works; Content Safety specifies the requirement and language context for each case (Sections 1 and 3.3–3.4).

The results show why these targets matter. For Qwen3-32B, moving from Direct to Reasoning Conditions raises the English protected-request score-0 rate from 5.00% to 25.10%, while public-information false refusal falls from 15.00% to 5.50% (Section 4.3; Appendix I.4). A single refusal score obscures this tradeoff.

Paired generalization raises Qwen3-30B-A3B's Privacy violation rate by 2.23 points and lowers Mistral-small-2509's by 4.08 points on the same retained seeds (Section 4.3/Table 2). Table 1 compares benchmark properties; Figures 3–4 and the paired analysis show the model differences exposed by PrismEval. These measure within-benchmark effects; equal-budget external comparisons remain separate experiments.

W2: Why these six jurisdictions and three cultural settings?

Section 3.1 explains the selection. China, the EU, and the US supply contrasting regulatory approaches and documented sources. Saudi Arabia, Türkiye, and Thailand add Arabic, Turkish, and Thai settings with distinct requirements and language-review materials.

Beyond translation, Dialectal Transformation applies local expression rules; Local Model Rewriting preserves protected fields and request intent. Several dozen legal and language reviewers worked over several months, with multiple professors overseeing the work and multi-reviewer adjudication resolving disagreements (Appendix D.1). Section 4.4 evaluates the complete language–jurisdiction settings.

W3: Larger models, within-family scaling, and frontier models.

Section 4.2 and Appendix I.5 now report within-family changes in violation rate. Columns: Privacy, Copyright, Value Denigration, Illegal Assistance. Negative changes favor larger checkpoints (percentage points).

Vicuna 13B minus 7B: -1.05, -0.14, -6.49, -4.84. Llama-2 13B minus 7B: +0.32, -0.54, +3.80, +3.10. Qwen3 32B minus 8B: -0.55, -0.29, +0.16, -0.95.

Vicuna improves in four fields, Qwen3 in three, and Llama-2 in one, showing family- and risk-dependent scaling.

The base Copyright diagnostic includes DeepSeek-v3.2 and Claude-sonnet-4: 6.32% and 5.12% of their responses exceed either source-matching threshold (Section 5.1; Appendix F.3). This supplies frontier-model evidence for base protected-text reproduction, a complementary setting to strengthened four-field evaluation.

W4: Confusing and complex writing.

We reorganized the explanation around the tested behavior, construction source, applied transformations, and observed failures. The three contributions now correspond to the resource, validated request variation, and model evaluation with optimization tradeoffs. Key evidence appears in Section 4.2 (model sizes), Section 4.3/Table 2 (Privacy controls and paired generalization), and Section 4.5/Table 5 (ranking retention), with complete profiles in Appendix I.

Q1: What does PrismEval capture beyond previous benchmarks?

Public-information controls expose false refusal; protected works anchor reproduction assessment; local requirements specify expected content behavior. Adversarial and reviewed linguistic variants test these targets across request forms. Opposing Privacy outcomes and model-dependent generalization effects support the dataset design (Sections 1, 3, and 4.3).

Q2: Do larger models perform better within a family?

Larger checkpoints improve different fields in different families (W3; Appendix I.5).

Q3: How much of the 37,433-case suite is new versus reused?

Appendix I.1 distinguishes requests, source materials, and variants. The suite contains 17,208 Chinese/English and 20,225 Turkish/Arabic/Thai cases. Requests are constructed from protected data, documents, local requirements, and templates, then expanded into seed-linked variants. Regulations, protected works, and attack procedures are inputs; external benchmarks support coverage mapping. The total counts prompt instances, including shared-seed variants. Exact cross-benchmark textual-overlap percentages have not been measured; construction provenance and parent-seed counts specify the resource's origin and composition.

We sincerely hope you could raise your score based on our clarification!

Reviewer_PEsVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PrismEval, a benchmark for evaluating LLM compliance across Privacy, Copyright, and Content Safety. Each dimension is constructed from a different evidence source: protected data and public-information controls for Privacy, protected books and news for Copyright, and jurisdiction-specific legal, policy, official, and cultural materials for Content Safety. The benchmark applies 15 jailbreak methods, five controlled transformations, dialectal transformations, and local-model rewriting. The resulting collections contain 32,238 jailbreak-enhanced cases and 37,433 multilingual Privacy and Content Safety cases across Chinese, English, Turkish, Arabic, and Thai. Responses from 24 models are judged by a Qwen2.5-7B evaluator using dimension-specific guidelines. The paper reports very high violation rates across all four reported risk fields and studies five procedures for reducing the evaluation set.

Critical strengths
  1. The paper recognizes that privacy, copyright, and content-safety compliance cannot be evaluated with one generic refusal criterion. Privacy includes public-information controls, Copyright is grounded in protected source material and reproduction behavior, and Content Safety retains a jurisdiction-specific Rule Record. This dimension-specific construction is more informative than treating every refusal as safe behavior.

  2. PrismEval is broad in scale and includes several useful forms of variation. The supplementary material provides detailed source inventories, taxonomies, record fields, construction counts, transformation audits, and full model-level results. The benchmark also retains provenance from source requirements to Risk Items and response-level judgments. The authors separately audit 513 Privacy responses, 450 controlled transformations, and 200 localized transformations, which is a meaningful effort to check parts of the pipeline.

Critical weaknesses
  1. The main 24-model conclusions depend heavily on an automated evaluator whose calibration is not established clearly. The main experiment uses Qwen2.5-7B to judge every response. Appendix D.3 calls 400 Qwen3-32B responses a calibration set, but Table 25 reports automated Jailbreak Success Rates of 93%, 98%, 100%, and 95%. These values appear to be the evaluator's positive-decision rates, not its agreement with independent expert labels. The text says reviewers inspect the records, but it does not report expert labels, accuracy, precision, recall, a confusion matrix, or inter-annotator agreement for this set. Separate audits report 97.7% agreement for the foundational Privacy evaluator and 90.1%--93.5% agreement for localized Content Safety, but those results do not validate the strengthened four-field evaluation used for the headline numbers. Qwen2.5-7B is also among the evaluated model family, which creates a self-evaluation concern for its own responses. Given the reported violation rates of 85.9%--96.5%, even a moderate judge bias could materially change the findings.

  2. The legal and cultural validity of the benchmark remains difficult to assess. The supplementary material lists regulatory sources and reviewed files, but does not report the number or qualifications of the legal and language experts, how disagreements were resolved, or agreement on Rule Records and Expected Safe Behavior. The Content Safety labels combine laws, policies, official guidance, and cultural materials with different levels of authority. The paper acknowledges that these mappings are not legal determinations, yet repeatedly describes the resulting judgments as compliance violations. Localized evaluation is limited to three jurisdictions, omits Copyright, and evaluates only four models. These choices support a useful benchmark sample, but not the broader claim that the reported rates characterize multilingual or jurisdiction-level legal compliance.

  3. The benchmark contains many transformed versions of shared seed cases, but the statistical analysis treats the resulting judgments mainly as a large collection of cases. The paper does not report uncertainty that clusters by source item, Rule Record, transformation family, or model. This matters because thousands of prompts derived from the same protected item or seed request are not independent evidence. The expert audits are also small relative to the 32,238 and 37,433 strengthened collections. For example, the generalization audit checks 150 cases per dimension, while the localized audit checks 100 cases for each transformation procedure. High usability on these samples does not show that all transformation families preserve the intended boundary equally well.

  4. The claim that Test Case Optimization preserves the delivered model characterization is too strong. Four of the 25 comparisons in Table 3 differ from the random-sampling reference at p < 0.05, including Privacy under two deduplication methods and Copyright under embedding deduplication. The selected Copyright score changes by 4.32 percentage points in one case. These are not merely negligible deviations, and no multiple-comparison procedure or ranking-stability analysis is reported. The optimization methods may reduce cost, but the current evidence does not show that every reduced set preserves the same conclusions.

Questions for authors

See Weaknesses.

2 comments

Dear Reviewer PEsV,

Thank you for your careful review. We revised the evaluator, expert-review, uncertainty, and optimization descriptions.

W1: Automated evaluator calibration and possible self-evaluation bias.

Appendix D.3 now identifies the four percentages on 400 strengthened responses as automated positive-decision rates, not accuracy. Section 4.1 defines the results as operational failures against Expected Safe Behavior.

Qwen2.5-7B uses temperature 0.1, dimension-specific prompts, binary decisions, and rationales. Privacy distinguishes protected-attribute assistance from false refusal of public information; Copyright checks protected-source reproduction; Content Safety follows the local Risk Item.

Appendix D separates the 97.70% agreement on 513 foundational Privacy responses from the localized synthetic score-band check and the strengthened response-review sample. Strengthened-response accuracy, precision/recall, confusion matrices, annotator agreement, and evaluator-family bias require independent gold labels and a distinct judge.

W2: Legal and cultural validity, experts, disagreement resolution, and scope.

Section 3.1 and Appendix D.1 describe several dozen legal and language reviewers working over several months, with multiple professors overseeing the work. Legal review examines requirements and applicability; language review checks local expression and semantic preservation. Disagreements undergo multi-reviewer adjudication. Retained comments, tracked revisions, and reviewed documents record the accepted decisions. Pre-adjudication agreement on Rule Records and Expected Safe Behavior requires independent label pairs.

Appendix I.6 distinguishes source authority: statutes and regulations specify legal requirements; policies, standards, and official guidance specify governance expectations; cultural materials supply reviewed context. Cultural preferences therefore serve as contextual evidence, not automatically as statutory prohibitions.

Section 4.4 specifies four-model localized Privacy and Content Safety evaluation in Turkish/Türkiye, Arabic/Saudi Arabia, and Thai/Thailand settings. Copyright has separate collections of 10,020 base and 15,030 jailbreak cases.

W3: Shared-seed dependence and audit uncertainty.

Section 4.3 and Appendix I.2 add paired original-versus-generated comparisons for 7,876 Privacy, 429 Value Denigration, and 398 Illegal Assistance seeds across four models. Each seed receives equal weight: its mean variant failure indicator is compared with its original indicator. The 2,000-replicate bootstrap samples seeds within source-file strata while keeping variants and four-model labels together.

Qwen3-30B-A3B's Privacy rate increases by 2.23 percentage points, with a 95% interval of [1.48, 3.06]; Mistral-small-2509's decreases by 4.08 points, with [-5.10, -3.00]. Table 2 reports Privacy; Appendix I.2 reports all 12 comparisons. The intervals describe this subset. Clustering the complete headline experiment by source item, Rule Record, or transformation family requires corresponding identifiers and judgments.

Section 5.3 and Appendix I.6 add audit counts and Wilson intervals: [97.50%, 100.00%] for 150/150 usable Content Safety or Copyright cases and [94.29%, 99.32%] for 147/150 Privacy cases. Localized procedures have separate 100-case audits. Family-specific estimates require per-family audit labels and sampling weights.

W4: Does optimization preserve model characterization?

We replaced blanket preservation claims in the Abstract, contributions, Section 4.5, and Conclusion with measured score and ranking changes.

Table 4 retains all 25 baseline/selected comparisons, including Copyright's -4.32-point change under embedding deduplication. The z/p columns are nominal normal-reference statistics; Appendix F.2 specifies their formula and unadjusted status across 25 comparisons. Four nominal p values are below 0.05. Non-significance is not used to establish equivalence.

Table 5 directly evaluates rankings across four models, three filtering procedures, and three fields. Six of nine comparisons preserve the complete ordering; the remaining Value Denigration comparisons have correlations of 0.95, 0.80, and 0.80. Mean absolute score errors range from 0.28 to 1.29 points; the largest individual error is 2.17 points. Appendix I.3 supplies all 36 integer count pairs and uses average ranks for ties. These comparisons concern the recorded models that also contribute selection evidence.

The inventory falls from 72,408 to 43,325 cases (40.17%). The results quantify score and ranking changes at reduced evaluation size.

We sincerely hope you could raise your score based on our clarification!

Reviewer_PEsVOpenReview ↗

Official Comment

While the additional experiments strengthen the paper, my main concerns remain, so I will maintain my original score.

Reviewer_Mp68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrismEval, a fine‑grained compliance evaluation benchmark for large language models (LLMs) covering three dimensions: privacy, copyright, and content safety. Built upon 52 regulatory, policy and standard documents as well as protected and copyrighted materials from six countries or regions, the authors construct test samples grounded in concrete risk requirements. Samples are further augmented via 15 jailbreak techniques, 5 generalization transformations, dialect conversion and local‑model‑driven rewriting. The benchmark contains 32,238 jailbreak‑augmented samples plus 37,433 privacy‑ and content‑safety‑focused samples across five languages. Evaluations over 24 LLMs show violation rates from 85.9% to 96.5% across risk domains, and model rankings vary across evaluation dimensions. The authors also investigate five sample optimization methods that cut candidate evaluation samples by 40.17%.

Critical strengths
  • Focuses on multi‑lingual and multi‑regional LLM compliance evaluation with good real‑world practical value
  • Covers privacy, copyright and content safety with dimension‑specific data sources and assessment criteria
  • Privacy evaluation adopts control samples built from public information, supporting assessment for both privacy protection and over‑refusal problems
  • Incorporates jailbreak attacks, semantic transformations and localization testing to expose model vulnerabilities under diverse scenarios
  • Conducts large‑scale model evaluation and explores approaches for lowering evaluation cost
Critical weaknesses
  • Technical innovation mainly lies in integration of evaluation dimensions and dataset construction, with limited methodological novelty
  • Automated scoring depends on a single judge model, yet sufficient reliability validation for this judge model is missing
  • Uneven multilingual coverage; localization experiments are only conducted on four models; copyright dimension is not adequately addressed by the framework
  • Near‑saturated violation rates in certain risk categories may hurt the benchmark’s model discrimination capability
  • Referenced appendices are absent in current submission, impeding verification of data construction and experimental details
Questions for authors

The paper presents a compliance evaluation framework spanning privacy, copyright, and content safety, mapping concrete risks to regulatory requirements and adopting jailbreak attacks as well as localization‑aware transformations. It is a valuable design to use differentiated construction pipelines for different dimensions and introduce public‑information‑based control samples for privacy assessment. Experiments demonstrate that aggregated overall scores may conceal model weaknesses on particular risk facets.

Nevertheless, three major concerns need further clarification and reinforcement for the work to reach the bar of top‑tier venues: First, the standalone methodological contribution is ambiguous. Although the paper combines regulatory mapping, existing jailbreak techniques, semantic rewriting and sample selection, it emphasizes benchmark scale and final test results without clearly articulating the added value of each individual component. The authors are suggested to add ablation studies and perform head‑to‑head comparisons against existing benchmarks under identical models, evaluation budgets and scoring metrics, in terms of risk coverage, failure detection performance and model discriminative power.

Second, core results heavily hinge on the reliability of the automated judge model. This work uses Qwen2.5‑7B for compliance judgement and observes extremely high violation rates. It is critical to sufficiently eliminate misclassification risks and biases from over‑permissive scoring rules. The authors should report human‑judge alignment rates, false‑positive and false‑negative rates together with typical disagreement cases stratified per risk category and per language, and perform cross‑validation with multiple distinct judge models. Besides, qualitatively distinct failure modes including real privacy leakage, sensitive attribute inference and over‑refusal for public‑information queries ought to be analysed separately instead of being lumped together.

Third, conclusions from localization and efficiency experiments need stricter validation. The localization setup changes language and corresponding regulatory requirements simultaneously, making it hard to disentangle whether performance gaps come from language differences or requirement updates. Controlled single‑variable experiments are recommended: fix requirements while varying language, or fix language while varying requirements. For sample optimization, statistically significant differences are observed in partial comparisons, while non‑significant results cannot directly prove equivalence. Additional validations are required to verify the stability of model rankings, risk coverage and failure‑detection ability after sample reduction.

2 comments

Dear Reviewer Mp68,

Thank you for your constructive suggestions. We clarified the dataset contribution and added paired generalization, Privacy control, and ranking analyses.

W1: Limited methodological novelty.

Our contribution is a benchmark with distinct response targets: Privacy pairs 15,600 protected-attribute requests with 1,100 public-information controls; Copyright links 10,020 base and 15,030 jailbreak requests to protected works; localized Content Safety translates local requirements into reviewed Turkish, Arabic, and Thai cases. Sections 1 and 3 explain how these sources determine construction and judgment.

These distinctions expose opposing outcomes. For Qwen3-32B, the English protected-request score-0 rate rises from 5.00% to 25.10% from Direct to Reasoning Conditions, while public-information false refusal falls from 15.00% to 5.50% (Section 4.3; Appendix I.4).

W2: Single-judge reliability.

Section 4.1 specifies Qwen2.5-7B, temperature 0.1, dimension-specific prompts, binary decisions, and rationales. Appendix D distinguishes 97.70% agreement on 513 foundational Privacy responses, a localized synthetic score-band check, and positive-decision rates on 400 strengthened responses. The latter measure failure fractions, not accuracy. Strengthened-response gold labels and a distinct judge remain necessary for the requested confusion matrices and cross-judge validation.

W3: Multilingual coverage, four-model localization, and Copyright.

Copyright contributes 15,030 of 32,238 jailbreak cases and has a protected-source base evaluation covering extraction, continuation, completion, rewriting, and close reproduction. Appendix F.3 reports five-model source matching: DeepSeek-v3.2 and Claude-sonnet-4 exceed either matching threshold in 6.32% and 5.12% of base responses (Section 5.1).

The 24-model experiment covers all three dimensions; the four-model localized experiment covers Privacy and Content Safety in three language–jurisdiction settings (Section 4.4). The multilingual subset and three-dimensional jailbreak collection are identified separately.

W4: Discrimination under high violation rates.

Copyright and Value Denigration span 11.96 and 12.89 percentage points across models; Illegal Assistance has a standard deviation of 7.57 points. The lowest-rate model differs by field (Section 4.2; Figures 3–4). Paired generalization also distinguishes models: Privacy changes by +2.23 points for Qwen3-30B-A3B and -4.08 for Mistral-small-2509 (Table 2).

W5: Missing appendices.

The appendices were submitted as supplementary material. We will submit them with the main paper in one linked PDF; the current revision already combines both.

Q1: Component ablations and equal-budget benchmark comparisons.

We added original-versus-generated comparisons for 7,876 Privacy, 429 Value Denigration, and 398 Illegal Assistance seeds across four models. Variants are averaged within seeds; 2,000 seed-cluster bootstrap replicates quantify uncertainty. Qwen3-30B-A3B's Privacy increase is 2.23 points, with a 95% interval of [1.48, 3.06] (Section 4.3; Appendix I.2). Table 1 compares benchmark properties. Complete component ablations and equal-budget external benchmark runs remain separate experiments.

Q2: Human alignment, multiple judges, and separate Privacy failure modes.

Sections 3.3 and 4.3 and Appendix I.4 separate protected-request failures from public-information false refusal. Section 4.1 distinguishes disclosure, guessing, and inference; factual private disclosure requires a protected reference. Appendix D documents available checks. Per-language precision/recall, false-positive/negative rates, disagreement examples, and evaluator-family bias require independent strengthened-response labels and judgments.

Q3: Localization controls and preservation after optimization.

The localized experiment measures complete language-and-requirement settings. Appendix I.6 identifies 533 source-to-English preparation pairs for a fixed-requirement comparison; paired target responses are still required.

Section 4.5/Table 5 adds ranking retention: six of nine comparisons preserve the ordering, Spearman correlations range from 0.80 to 1.00, and mean absolute errors from 0.28 to 1.29 points. Table 4 retains all 25 score changes, including Copyright's -4.32 points; Appendix I.3 gives 36 integer count pairs. These results quantify size–score–ranking tradeoffs. Unseen-model failure retention and risk coverage require further validation.

We sincerely hope you could raise your score based on our clarification!

Thank you for the response and the additional experiments. The new analyses strengthen the paper, but my main concern about the reliability of the automated evaluation remains unresolved. I therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.