acceptodds
Under review as a conference paper at ICLR 2027

PolyAEGIS: Benchmarking Large Language Model Safety Across 100 Languages

Abstract

Large language models are increasingly deployed across linguistically diverse populations, yet safety evaluation remains concentrated in English and a small set of high-resource languages. We introduce a multilingual safety benchmark derived from the AEGIS unsafe-prompt taxonomy, translated into 100 typologically diverse languages spanning 16 language families, 26 scripts, and a balanced split of high- and low-resource languages. Source prompts are filtered from the AEGIS corpus across 21 safety-hazard categories, machine-translated using NLLB-200, and scored for translation quality using three complementary metrics (BERTScore-F1, COMET-Kiwi, chrF++), with a translation retained unless it simultaneously fails all three. The benchmark is constructed via multi-factor equal-allocation stratified sampling across safety category and prompt-length strata, crossed completely with all 100 target languages, yielding full coverage — 100/100 languages, 21/21 categories, 16/16 language families, and 26/26 scripts — across more than 9,000 prompt-translation pairs. We generate model responses using three open-source large language models spanning distinct architectures and parameter scales, and evaluate safety compliance using an LLM-based judging protocol combining closed- and open-source judges. We present the dataset construction methodology, translation-quality validation, and coverage analysis here, with full response-generation and safety-evaluation results to follow in the complete submission.

Primary area:
datasets and benchmarks
Keywords:
multilingual AI safety, large language model evaluation, safety benchmark, low-resource languages, machine translation quality, LLM-as-judge, cross-lingual robustness, harmful prompt detection, stratified sampling, red-teaming

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AVLbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces PolyAEGIS, a massively multilingual safety benchmark designed to evaluate LLMs across 100 languages. The benchmark consists of 10,362 harmful prompts sourced from AEGIS, translated via NLLB, and filtered through dual-metric quality gating. The authors evaluate six LLMs using a three-judge LLM consensus protocol and propose a four-class taxonomy distinguishing Full Refusal, Partial Leakage, Full Compliance, and Irrelevant/Unintelligible responses. The core contributions include revealing that "Partial Leakage" accounts for nearly half of unsafe outcomes, demonstrating that open-weight models are 1.6x more vulnerable than closed-source counterparts, and highlighting systemic safety degradation in low-resource languages.

Critical strengths
  1. The benchmark evaluates 100 languages spanning 17 language families and 26 scripts (Table 2), which significantly advances the field's coverage beyond English-centric safety evaluations and provides excellent experimental rigor.
  2. A leakage-aware evaluation framework is proposed to identify partial leakage as a distinct failure mode accounting for nearly half of unsafe responses.
  3. The evaluation on 61,536 responses reveals that partial leakage accounts for 6.32% of responses, providing impactful evidence that current binary safety metrics are insufficient for capturing nuanced model behaviors.
Critical weaknesses
  1. Relies on only one machine translation system (NLLB-200), which may introduce semantic drift particularly in low-resource languages despite quality filtering.
  2. Format problem: Tab. 2 is beyond the boundary. The abstract section uses too many dashes, which does not conform to the standard English norms. At the same time, some abbreviations need to be fully written out for the first time, and some experimental settings should be presented more clearly in the abstract and further refined. The "Conclusion" section of this paper is on page 11, which clearly exceeds the submission requirements of ICLR 2027.
  3. The lack of necessary graphical explanations (such as the dataset construction in the methodology section with two pipelines) results in poor readability.
  4. One limitation of this study is the small number of evaluated models, which does not include recent mainstream LLMs (e.g., Qwen 3.5+ series, GPT-5.5, Claude Opus 4.6+, Gemini 3+ series).
  5. The authors note an anomaly where low, resource languages show artificially higher mean BERTScore F1 due to repetitive vocabulary acting as a back-translation artifact, a complication conceptually similar to multilingual transfer issues identified in related works like MultiJail (Table 25; Sec. 4.6), which weakens the technical soundness of the overall quality-gating pipeline.
  6. The study currently lacks formal statistical significance testing or bootstrap confidence intervals for the observed cross-lingual harm score differences, reducing the statistical rigor of the overarching claims.
Questions for authors

What is the rationale for determining the quality bucket thresholds? That is Gold — BERTScore F1≥0.95, why not 0.93 or other value?

10 comments
AuthorsOpenReview ↗

Comment 01: Overview and Response Roadmap

Response to R3 (AVLb) — Overview

We thank the reviewer for a review that identified two issues we could not have closed without it: the dependence on a single translation system, and the absence of any uncertainty quantification. Both are now addressed with measurements rather than argument, and one of them overturned a headline claim of the submission.

Main new evidence:

  • Single MT system (W1): The benchmark was re-translated with two architecturally independent systems (MADLAD-400-3B, SeamlessM4T-v2) and re-judged. On 3,528 matched triples across 81 languages, changing the translation system moves ASR by 0.23–0.60 pp, while changing the judge moves the same number by 11.3–16.7×. Aggregate conclusions are translator-robust; per-item labels are not (3.8–4.2% flip under one judge, 23–26% under the other).
  • Model coverage (W4): The evaluated set grew from 6 to 12 models across 6 developers, including every family the reviewer named. The judge panel grew from 2 to 4 open-weight judges. The open-vs-closed headline does not replicate: on seven current generators the gap reverses under all four judges (−0.13 to −11.95 pp; ratios 0.72–0.97×).
  • BERTScore anomaly (W5): Our own stated explanation was wrong. The inversion is not a repetitive-vocabulary back-translation artifact but a composition effect of pooling two differently gated corpora. Within the only corpus containing both tiers the difference is 0.0022; with prompt content held fixed the effect reverses.
  • Statistical rigour (W6): Language-clustered bootstrap intervals are ≈6× wider than naive row resampling (13.78% [12.21, 15.51] vs [13.52, 14.05]). Leave-one-model-out moves the headline by at most 1.86 pp, but per-model spread is 17.71 pp. Four-judge Fleiss' κ is 0.288, rising to 0.461 only when Mistral-Large is removed.
  • Threshold rationale (Q1): We give the honest answer — the threshold met a sample-count budget, not a validity criterion — and then show with an independent arbiter (LaBSE, 1,178,928 rows) that no absolute threshold can be correct, because which metric selects better reverses in 20 of 72 languages.
  • Presentation (W2, W3): All items are accepted and will be fixed in a revised manuscript uploaded 17 November 2026; see Comment 07.

Claims withdrawn or narrowed: the repetitive-vocabulary explanation for the tier inversion; the open/closed safety gap; any per-item or small-cell claim presented as translator-independent; any safety ranking presented as judge-independent; and the pooled partial-leakage headline. We retain and report partial leakage as judge-dependent.

We address W1 (translation systems), W4 (model coverage), W5 (BERTScore anomaly), W6 (significance and intervals), Q1 (threshold rationale) and W2/W3 (presentation) in Comments 02–07. Completed results are distinguished throughout from pending human validation.

AuthorsOpenReview ↗

Comment 02: W1 — Reliance on a single machine translation system

Reviewer concern. Relying on NLLB-200 alone may introduce semantic drift, particularly in low-resource languages, despite quality filtering.

Response. We accept the criticism and treated it as a control to be run. The same prompts were re-translated with MADLAD-400-3B (T5-style, different web-scale corpus) and SeamlessM4T-v2 (multimodal, text path trained jointly with speech) — chosen to differ in training data and architecture, not merely in checkpoint. Unsupported languages were recorded as absent, never back-filled from NLLB.

Aggregate safety conclusions are translator-robust. On the v2 corpus, matched triples of (prompt, generator, budget) differing only in translator:

judgegeneratornlangsNLLBMADLADSeamlessspread
Qwen3.5-122BQwen3.5-27B3,528811.70%1.70%1.47%0.23 pp
Qwen3.5-122BGemma-4-31B3,528812.95%2.66%2.81%0.28 pp
Mistral-LargeQwen3.5-27B3,5298128.42%28.36%28.96%0.60 pp
Mistral-LargeGemma-4-31B3,5298133.30%33.24%32.98%0.31 pp

This is the stronger of the two corpora for the question: 81 languages spanning both resource tiers, selected by percentile rather than by NLLB's own round-trip score, so the circularity affecting our v1 comparison does not apply here. The translator moves the headline by at most 0.60 points; the judge moves it by 11.3–16.7×.

Per-item labels are not robust, and we now say so. Pairwise label agreement is 96.0–97.0% under the Qwen judge but only 73.6–76.7% under Mistral — so between 3% and 26% of individual items change safety label when only the translator changes, with flips near-symmetric in both directions. No system is systematically "safer".

One language where NLLB's own score is not credible. NLLB rates its Fula round-trips at median F1 1.0000; MADLAD and Seamless rate the same sentences at 0.53 and 0.49. Fula also has the benchmark's highest unintelligible-response rate (23.28%). A perfect round-trip score is therefore not evidence of adequacy.

Revision and limits. We will report the aggregate robustness as a measured control, attach the matched-pair agreement rate to any per-item or per-language claim, and stop implying per-language cells are translator-independent. Three systems agreeing could be three systems making the same error on overlapping web corpora; only native-speaker validation settles adequacy, and that remains pending (Comment 06).

AuthorsOpenReview ↗

Comment 03: W4 — Small number of models; no recent mainstream LLMs

Reviewer concern. The study evaluates few models and omits recent mainstream LLMs (Qwen 3.5+, GPT-5.5, Claude Opus 4.6+, Gemini 3+).

Response. We added a model from every family named, and the evaluated set grew from 6 to 12 models across 6 developers (Cohere, Mistral, Alibaba, Google, OpenAI, Anthropic), spanning open and closed weights and two capability tiers within two vendors. The v2 corpus now carries seven current generators, each scored by four independent judges on one decode contract:

generator (v2)Qwen3.5-122BQwen3.5-27Bgemma-3-27bMistral-Large4-judge majority
qwen3.5-27b1.75%5.21%8.49%28.82%3.91%
gpt-5.51.71%6.96%12.33%39.73%5.17%
gemma-4-31b-it3.04%9.17%12.50%33.51%7.34%
claude-opus-4-64.59%10.10%11.10%34.01%8.09%
gemini-3.8-flash6.26%13.48%18.01%40.85%11.47%
mistral-large-123b8.51%17.16%17.62%31.83%14.00%
claude-opus-5.55.67%15.95%25.54%58.78%14.33%

Three findings follow, and one of them is negative for us.

(a) The open/closed claim does not replicate. Mean ASR for 3 open versus 4 closed models reverses sign under every judge: −0.13 pp (0.97×) under Qwen3.5-122B, −1.10 pp (0.91×), −3.88 pp (0.77×), and −11.95 pp (0.72×) under Mistral-Large. The submission reported +6.25 pp in the opposite direction. We withdraw the open/closed headline. The submitted gap is better explained by model age than by weight availability: our two oldest models were both open and both among the least safe.

(b) The measurement-validity finding replicates on models chosen by the reviewer, not by us. Between the two original judges, Cohen's κ on the added models is 0.050–0.175 with ASR ratios of 6.4–26× — at or below the κ = 0.271 the submission reported. The result therefore cannot be dismissed as an artifact of our original model selection.

(c) Newer models are not obviously safer. Under Mistral-Large the 2026 frontier closed models sit above the open models; under Qwen3.5-122B the order differs again. We report no recency effect in either direction.

Revision and limits. Twelve models remain a convenience sample. We now state a selection criterion (developer × access grid with capability tiers) but apply it retrospectively, and we report that the grid is not balanced on release date. Adding models does not resolve judge validity: twelve models scored by instruments that disagree 6–26× yields twelve uncertain numbers rather than six.

Reviewer_MebeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyAEGIS, a multilingual safety benchmark based on AEGIS-derived prompts translated into 100 languages and filtered using BERTScore and COMETKiwi. Six generation models are evaluated by three LLM judges using four response categories: full refusal, partial leakage, full compliance, and irrelevant or unintelligible responses. The analysis highlights partial leakage and variation across languages, resource tiers, and models. The released prompts, responses, and annotations could support broader multilingual safety evaluation.

Critical strengths
  1. The benchmark broadens access to multilingual safety data. Coverage of 100 languages, together with responses and judge annotations, makes it a potentially useful resource for studying underrepresented languages.

  2. The taxonomy separates harmful leakage from full compliance and failed comprehension. This makes the evaluation more informative than a single attack-success rate and helps distinguish refusal failures from language-understanding failures.

Critical weaknesses
  1. The contribution is mainly broader coverage. The paper combines translation, quality filtering, and LLM judging, but does not clearly establish what the evaluation method adds beyond a larger multilingual benchmark.

  2. Filtering changes which requests are evaluated in each language. The reported safety gaps may therefore reflect differences in retained content as well as differences in model behavior.

Questions for authors
  1. How much does each translation-quality filter contribute? Comparisons of BERTScore-only, COMETKiwi-only, and joint filtering would help establish their effects on semantic fidelity and safety conclusions.

  2. Does majority voting improve judgment accuracy over individual judges, particularly for partial leakage? Please provide human-validated comparisons and distinguish label agreement from verdict parseability.

  3. How do judges distinguish actionable leakage within a refusal from harmless discussion? Does this distinction remain reliable across response languages and refusal styles?

12 comments
AuthorsOpenReview ↗

Comment 01: Overview and Response Roadmap

Response to R2 (Mebe) — Overview

We thank the reviewer for the constructive, technically focused review. We agree that the submitted manuscript did not clearly establish methodological novelty beyond broader language coverage, that filtering introduced selection bias, and that majority-vote safety judgments—especially partial leakage—were not adequately validated. We have audited the pipeline, completed additional experiments, corrected unsupported claims, and revised the contribution framing to reflect both positive and negative findings.

Main new evidence:

  • Methodological contribution (W1): We position the work as a 100-language safety resource plus a measurement-validity audit of the translate–filter–judge pipeline, rather than treating its established components as new. We also evaluated 10 guardrails on 3,875 harmful prompts across 98 languages (best harmful recall: 76.0%) and compared guard behavior on matched English prompts and their translations.
  • Filtering and selection bias (W2/Q1): Across 1,178,928 translations, the documented joint quality gate retains just 41 rows in 3 languages. The implemented fallback retains 18,246 rows in 22 languages, with 99.78% bypassing COMET. On judged responses, aggregate ASR is 13.76% ungated versus 14.30% with the implemented gate; this near-stability does not validate cross-language comparisons because prompt and language composition change.
  • Semantic fidelity (Q1): An independent LaBSE audit of 1.18M translations finds no universally superior filter: COMET-selected items score higher in 52 of 72 sufficiently sampled languages, while 20 show the reverse. We motivate per-language percentile thresholds; human adequacy and harm-preservation judgments remain pending.
  • Matched-content limitations (W2): The evaluated corpus contains 10,262 root–language pairs, only 1.94% of the possible grid. Its largest fully crossed subset has 2 languages × 342 roots. We therefore narrow per-language and resource-tier claims; a fully matched 100-language rebuild is proposed, not completed.
  • Judge reliability and voting (Q2): The submitted 97.05% figure measured verdict parseability, not consensus; three-judge exact label agreement was 53.66%. On 26,346 identical responses, four-judge Fleiss' κ is 0.288, increasing to 0.461 without Mistral-Large. Four-judge voting yields 3.4–10.9% unresolved 2–2 ties. Without human reference labels, voting's accuracy benefit is unknown.
  • Actionable leakage (Q3): Judge disagreement persists even on 10,604 English-source responses. A paired 5,036-judgment rubric experiment lowered Mistral's leakage labels on disputed cases from 99.0% to 10.8%, increasing raw agreement from 7.2% to 88.7%. But detection of agreed full-compliance cases fell substantially, so the new rubric is not yet deployable.

Claims withdrawn or narrowed: novelty of individual pipeline components; effectiveness of the documented joint gate; intrinsic language/tier rankings from unmatched prompts; majority-vote accuracy without human labels; and the pooled headline that partial leakage constitutes nearly half of unsafe outcomes. We retain partial leakage as an analytic category but report its judge-dependent prevalence.

We address W1 (methodological contribution), W2 (selection confounding), Q1a–b (filter ablation and fidelity), Q2a–b (agreement, voting, and human validation), and Q3a–b (actionable leakage and rubric sensitivity) in Comments 02–09. Completed results are distinguished throughout from proposed experiments and pending human validation.

AuthorsOpenReview ↗

Comment 02: W1 — Contribution beyond coverage

Reviewer concern. The translation–filter–judge pipeline combines established components; what is new beyond 100-language coverage?

Response. We agree that the components themselves are not new. MultiJail already considers invalid responses; XL-SafetyBench discusses comprehension masking; StrongREJECT studies graded harmfulness; and CultureGuard applies translation/quality assessment to AEGIS 2.0. We have revised the manuscript's contribution list to credit these precedents and to distinguish resource scale from methodological novelty.

The additional contribution is a measurement-validity audit of the assembled pipeline. On 1,178,928 translations, the documented joint quality gate retains only 41 rows in 3 languages; 99.78% of the implemented gate's survivors enter via a COMET-skipping fallback. On identical generated responses, open judges yield Cohen's κ = 0.050–0.175, ASRs differing 6.4–26×, and model-ranking inversions. A four-judge panel (26,346 responses) yields Fleiss' κ = 0.288, increasing to 0.461 when Mistral-Large is excluded. Thus filtering and judge selection can materially alter what the benchmark appears to measure.

We also added a ten-guardrail evaluation (3,875 unsafe prompts, 98 languages; best unsafe recall 76.0%, low-vs-high-resource strict-evasion difference +21.56 pp, 95% CI [17.98, 25.17]). In a separate matched English/source-translation test (18,880 rows), Aegis-2.0 recall changes 77.14% → 22.98% and CultureGuard 80.86% → 64.61%. These matched comparisons expose guard-specific multilingual sensitivity rather than merely increasing language count. We avoid treating the difference between guards as a formally identified causal decomposition.

Revision and limits. The contribution list now emphasizes (i) a 100-language resource, (ii) an applied, prior-grounded leakage taxonomy, (iii) the measurement-validity audit, and (iv) the scoped guardrail evaluation. We withdraw claims that the pipeline or taxonomy is newly proposed, and withdraw the pooled partial-leakage and open/closed-model headlines. We do not claim this audit resolves the unmatched-prompt confound or establishes judge accuracy; those limits are addressed in W2/Q2.

AuthorsOpenReview ↗

Comment 03: W2 — Prompt-selection confounding

Reviewer concern. Filtering changes which requests are evaluated per language; reported safety differences may be content-selection effects.

Response. We agree. The evaluated corpus contains 10,262 root–language pairs, spanning 5,285 roots and 100 languages—only 1.94% of the full 5,285 × 100 grid. Per-language prompt counts range from 46 to 734 (median 48). Only 14 languages share 10 prompts, and only 4 share 100. The largest fully crossed subset is 2 languages × 342 roots; consequently, the existing data cannot be re-sliced into a robust 100-language matched comparison.

The selection mechanism is substantial: on a balanced 5,000-row benign translation pool, the implemented gate reduces coverage from 100 to 17 languages and shifts the high-/low-resource composition from 48/52 to 0/100. On the full ungated translation pool, the documented gate retains only 41/1,178,928 rows, while the implemented fallback gate retains 18,246 (1.55%) across 22 languages. These are direct selection effects, not evidence of model behavior.

The aggregate ASR on judged responses changes comparatively little across gates with useful sample sizes: 13.76% ungated, 13.76% BERTScore-only, and 14.30% implemented gate. This robustness does not validate per-language or resource-tier rankings. The language-clustered 95% CI for the response-weighted resource-tier gap is [−0.42, +4.42] percentage points, including zero.

Revision and limits. We will explicitly label unmatched per-language/tier comparisons as descriptive, not causal language effects. A matched rebuild using 250 common roots × 100 languages would require 25,000 prompts, 175,000 generator responses, and 700,000 four-judge assessments; it is proposed, not completed. The balanced benign control is useful for filter testing but does not remove confounding from the unsafe evaluation.

Reviewer_fyepOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • The authors translate AEGIS 2.0 unsafe prompts into 100 languages with NLLB-3.3B. Translations are meant to pass a Gold-tier gate on BERTScore (via back-translation) and COMETKiwi
  • The set has 10,262 unique evaluated instances
  • 6 generator models and 3 LLM judges; responses are labelled with a 4-class taxonomy (Full Refusal / Partial Leakage / Full Compliance / Irrelevant-Unintelligible) and scored with ASR_full and ASR_interp
  • Reported results, all based on judge labels: 13.78% of responses unsafe; partial leakage ≈ 46% of unsafe responses; open-weight models 1.6× as unsafe as closed; low-resource languages less safe than high-resource
Critical strengths
  • I like the finer-grained response labels. Partial leakage is reported separately from full compliance and from comprehension failures, and the paper shows which languages fail mostly through leakage (Table 16)
  • The language set is broad: 17 families and 26 scripts. The authors say they will release the multi-judge annotations
  • Worst-case harm is reported alongside mean harm (Table 5)
  • The authors are upfront about construction problems: they report the duplicate removal and the per-model judge parse fallbacks (Table 38)
Critical weaknesses

W1: Limited novelty, overclaimed contributions

  • Multilingual safety benchmarking is a crowded area. The claim that "existing benchmarks remain limited in scale, linguistic diversity, and evaluation methodology" is overstated [1–5]
  • Neither the variation across languages nor the effect of resource level and typology (RQ1) is new. Low-resource languages are known to be less safe [6, 7], toxicity is known to rise as resource level falls [1], and attacks and defences have been compared across resource tiers [8]
  • Machine-translating an English safety benchmark and filtering the output with quality-estimation metrics has been done before [3]. CultureGuard [9] does this with the same Aegis 2.0 data, yet the paper lists it only as a row in Table 6 and never discusses it
  • The main contribution is the leakage-aware evaluation, but none of its parts is new, and the paper does not discuss the earlier work:
    • MultiJail [6] already has an invalid response class (≈ IU)
    • XL-SafetyBench [10] already argues that comprehension failures can mask unsafe behaviour, the motivation for ASR_interp
    • StrongREJECT [11] and severity-graded labels [4, 12] already go beyond binary ASR
  • What remains new is combining these at 100-language scale
  • Guardrail evaluation is claimed (§2.4; GR ✓ in Table 6), but no guardrail model is evaluated. The Cultural Robustness ✓ in Table 6 is not backed by any cultural experiment either

W2: Missing controls; headline comparisons are confounded

  • Only unsafe prompts are evaluated, so over-refusal and the safety/usefulness trade-off are not measured
    • A model that refuses everything would get a perfect score
    • Very low-resource inputs may themselves cause odd refusals
    • The authors build a pool of safe prompts during data construction (§3.1.1) but never use it
  • ASR_full counts IU responses as safe. ASR_interp helps only if the IU label really captures comprehension failure, and that is never checked
  • There are no English results, and ∆_safe is defined (Eq. 6) but never reported. Without an English reference point, the claim that safety degrades under language shift is not actually measured
  • The quality gate was not stratified by language or category (46–734 prompts per language), so each language is tested on a different set of prompts
    • The statement that languages "receive identical prompts" (§4.5) does not follow from how the benchmark was built
    • The 3.96× mean-harm ratio and the resource, script and family effects (§4.5, §4.7) may simply reflect which prompts each language ended up with
  • The resource-tier gap is 1.81 pp ASR (Table 24), with no confidence intervals
  • The open/closed "1.6×" rests on three models per side. The appendix notes that GPT-OSS-120B is safer than two of the closed models, yet the abstract still presents the gap as an open-vs-closed effect

W3: Translation quality is unverified, and the evaluation corpus is inconsistent

  • Translations are checked only with automatic metrics; no native speaker looked at them, unlike in comparable benchmarks [3, 13]. The "Translation Quality Analysis" (§4.6) claims more than these metrics can show
  • 19 of the 100 languages are not covered by COMETKiwi. Its model card lists the ~94 InfoXLM languages it supports and warns that scores for other languages are unreliable
    • The uncovered languages are Asturian, Cebuano, Fula, Ganda, Igbo, Lingala, Luo, Luxembourgish, Maltese, Maori, Northern Sotho, Nyanja, Occitan, Shona, Sorani Kurdish, Tajik, Wolof, Yoruba and Zulu
    • In the released prompt table these account for 5,813 of 10,000 prompts, and none of them has a COMET score (comet_supported = False)
    • So more than half of the benchmark was filtered on back-translation BERTScore alone. The paper describes this fallback as applying to "the small number of cases where a COMET score is unavailable" (§3.1.2)
    • These are also the languages with by far the most prompts (Luxembourgish, Maltese, Asturian, Occitan, …), probably because they only had to pass one threshold
    • This would also explain why low-resource languages have higher mean F1 than high-resource ones (Table 25) and make up most of the benchmark
  • Even for covered languages, COMETKiwi agrees only moderately with human judgments on English→X pairs [15]: Spearman 0.385 for en-ja, 0.592 for en-mr, and 0.409 for zero-shot en-yo (Yoruba is in PolyAEGIS)
  • Applying one absolute cut-off (COMET ≥ 0.92) to all 100 languages also assumes the scores mean the same thing across language pairs, which is not shown
  • Translating largely US-centric English prompts word for word ignores cultural and local context, and nothing is localized (contrast [10, 13])
  • Translated prompts are also known to elicit less toxicity than native ones [1], which threatens a benchmark built only from machine translation. The paper does not discuss this
  • I could not work out which data the headline numbers are computed on:
    • The benchmark is described as Gold-only, but Table 20 assigns only 36,352 of 61,536 responses to Gold prompts
    • The caption calls the non-Gold rows a diagnostic. Yet the four buckets add up to exactly 61,536, and their weighted ASR equals the headline 13.78%
    • Per-language mean F1 goes down to 0.73 (Table 22), and the high-resource tier averages 0.85 (Table 25). Neither is possible if every item passed F1 ≥ 0.95
  • The released dataset (linked in the paper) does not match the paper:
    • The dataset card describes a "category/language-stratified" 10,000-prompt subset. The paper says selection was not stratified and reports 10,262 prompts
    • The card names Qwen3.5 as a judge; the paper says Qwen2.5
    • The card lists 16 language families; the paper says 17
    • Released prompts that have a COMET score average 0.81 (range 0.12–0.92), well below the Gold cut-off of 0.92. BERTScore goes as low as −0.15, against a Gold requirement of ≥ 0.95
  • The authors note that round-trip F1 is inflated but do nothing about it. For example, Fula has F1 0.986 together with 23% IU (Table 22)
  • ASR goes up with translation quality (12.4 → 13.5 → 13.9%; Table 19). This sits poorly with the claim in §4.6 that poor translation adds to safety risk, which is based on a single category

W4: Judge validity is unverified

  • None of the judge labels are checked against native speakers, so the agreement analysis says little. LLM judges are known to agree poorly with human annotators on multilingual harm [2]. Comparable work validates its judge against native speakers [10]
  • Agreement between generation models, or how often judge outputs parse, does not tell us whether the judges agree with each other or with humans
  • Some headline numbers are mislabelled:
    • The abstract and introduction say the three judges agree on one label for only 36.8% of prompts. According to Table 39, 36.8% is how often the six generation models receive the same label
    • The "97.06% three-judge consensus rate" is actually the share of responses where all three judge outputs could be parsed
    • Inter-judge agreement (e.g., κ) is never reported
  • Mistral-Large is used both as a generator and as a judge, and the judge checkpoint is not given. High IU rates in low-resource languages could also come from judges not understanding the responses, which is not checked
  • The four labels are defined, but the following are missing:
    • the judge prompts
    • how the harm, refusal-strength and actionability scores are computed
    • definitions of RQS and HCAS (named in Table 14, not defined in Appendix B)
    • how ties are resolved when only two judges return a verdict, or when all three disagree

W5: Model selection and reproducibility

  • The models look arbitrary and somewhat dated (e.g., Aya-Expanse, Gemini 2.5 Flash, and Qwen2.5 as a judge). No selection criteria are given, yet the conclusions are stated for LLMs in general
  • Several details are missing: API snapshot identifiers, judge versions and sizes (Mistral, Qwen2.5), decoding settings, system prompts and query dates

W6: Presentation is confusing and haphazard

  • The paper walks through the history of the v1 and v2 pipelines, which is hard to follow. It would be clearer to describe one final pipeline (or simply build one) and add a pipeline figure
  • Almost all results discussed in §4 are in the appendix; the main text has only Tables 4–5
  • Some numbers are inconsistent or stale:
    • §2.4 says +3.9 pp and 4.64×; the abstract and §4 say 6.25 pp and 3.96×
    • The Occitan and Tajik mean-harm values in §4.5 differ from Table 5
    • It is unclear whether the v2 set difference was taken against the 5,376 pairs before deduplication or the 2,894 unique prompts
  • The prompt counts in the agreement tables do not add up:
    • Table 39 counts 5,285 prompts "shared across all six models" out of 10,262 instances, without explaining the drop
    • Table 40 lists a model pair that shares only 5,256 prompts, but prompts shared by all six models must also be shared by every pair
  • 36 responses are missing (Gemini 33, GPT-5.4 Mini 3; Table 31), with no explanation
  • Smaller issues:
    • Table 17's caption says the listed languages reach harm 1.0, but most of the listed maxima are below 1.0
    • Appendix A is one sentence long and has typos ("Litertaure", "If we summary…"). Appendix C.1 has a heading but no text
    • The note under Table 32 points to a discussion of an "analysis-pipeline bug" in Section 6, but Section 6 has no such discussion

W7: Unreliable bibliography

  • Several references are wrong:
    • a Table 6 benchmark is credited to an unrelated orthopaedics paper
    • one bibliography entry is reused for three different works
    • several citations point to the wrong paper
    • several entries have garbled author lists
  • The source dataset is cited only by its Hugging Face URL, not by the AEGIS 2.0 paper [14]
  • BERTScore, COMETKiwi, the Joshi resource classes, and most of the evaluated and judge models are not cited
  • Table 6 has wrong entries:
    • HarmBench is English-only but is marked as having unsafe-translation analysis
    • One column lumps "multimodal" and "multi-model" together
    • The "17 languages" listed for CulturalKaleidoscope could not be verified
  • There are duplicate entries (Deng 2023a/b) and wrong years or missing venues (XSafety, TituLLMs, MADLAD-400)

References

  • [1] Jain et al. PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models. arXiv:2405.09373, 2024
  • [2] de Wynter et al. RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? AAAI 2025
  • [3] Friedrich et al. LLMs Lost in Translation: M-ALERT Uncovers Cross-Linguistic Safety Inconsistencies. arXiv:2412.15035, 2024
  • [4] Ning et al. LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models. arXiv:2508.12733, 2025
  • [5] Kumar et al. PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages. arXiv:2504.04377, 2025
  • [6] Deng et al. Multilingual Jailbreak Challenges in Large Language Models. ICLR 2024
  • [7] Yong, Menghini & Bach. Low-Resource Languages Jailbreak GPT-4. arXiv:2310.02446, 2023
  • [8] Atil, Passonneau & Morstatter. Do Methods to Jailbreak and Defend LLMs Generalize Across Languages? arXiv:2511.00689, 2025
  • [9] Joshi et al. CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applications. IJCNLP-AACL 2025
  • [10] Choi et al. XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity. arXiv:2605.05662, 2026
  • [11] Souly et al. A StrongREJECT for Empty Jailbreaks. NeurIPS 2024
  • [12] Ji et al. PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference. ACL 2025
  • [13] Chua et al. Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps. arXiv:2507.05980, 2025
  • [14] Ghosh et al. Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. NAACL 2025
  • [15] Rei et al. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. WMT 2022

Additionally, Flagged by automated literature review as missing:

  • Krasnodebska et al. Safety of LLMs Beyond English: A Systematic Literature Review. EACL 2026: the obvious anchor for positioning the paper
  • Pattnayak & Chowdhuri. IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia. arXiv:2603.17915, 2026: a companion to IndicJR, which the paper cites. It reports cross-language agreement that can be compared with Table 41
  • Zhao et al. BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs. arXiv:2605.00689, 2026: argues against relying on machine-translated data, and evaluates guardrails
  • Shukla et al. Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms. arXiv:2602.07963, 2026: shows some harm types are distorted or lost in translation
  • Li et al. A Cross-Language Investigation into Jailbreak Attacks in Large Language Models. arXiv:2401.16765, 2024
  • Tasawong et al. SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures. arXiv:2512.05501, 2025
  • Le Jeune et al. Phare: A Safety Probe for Large Language Models. arXiv:2505.11365, 2025
Questions for authors
  1. Why is the constructed safe-prompt pool not used? Please report over-refusal on benign translated prompts per language and resource tier.
  2. Can you have native speakers check a stratified sample of (a) translations and (b) judge labels, especially IU, per resource tier?
  3. Which data are the headline results, Table 20, and Tables 22 and 25 computed on? Table 20's buckets sum to all 61,536 responses and reproduce the headline ASR, and Tables 22 and 25 report F1 below the Gold threshold.
  4. Do the per-language and resource-tier gaps hold on prompts shared across languages, or after reweighting by category? Please add confidence intervals, resampling over prompts.
  5. Did you run the English source prompts? If so, please report their ASRs and the ∆_safe values.
  6. What is the inter-judge agreement (e.g., Fleiss' κ), and what does each judge's label distribution look like? How are disagreements resolved when two judges disagree or all three differ? How much do results change without the Mistral judge?
  7. Please provide the judge prompts, the scoring rules for harm, refusal strength, actionability, RQS and HCAS, API snapshots, judge versions, decoding settings, system prompts and query dates.
  8. 19 languages (58% of released prompts) are outside COMETKiwi's coverage and have no COMET score. How exactly was the Gold gate applied to them, and how do the results change if these languages are excluded or gated differently?
  9. Which version of the data and which judges (Qwen2.5 or Qwen3.5) does each analysis use, and which version is released? Why do the released prompts not meet the Gold thresholds? Would a single pipeline reproduce the findings?
  10. Which guardrail models were evaluated? If none, please remove the guardrail claims and the GR ✓.
  11. How much does the open/closed gap depend on the chosen models, e.g., when leaving one model out, and with confidence intervals?
16 comments
AuthorsOpenReview ↗

Comment 01 — Overview and response roadmap

Response to R1 (fyep) — Overview

We thank the reviewer for a thorough, technically specific review. We agree that several claims in the submitted version were overstated, some analyses lacked necessary controls, and the manuscript and dataset documentation contained substantive inconsistencies. Rather than defend these errors, we audited the released artifacts, corrected the analysis, and ran new experiments. We will revise the paper around the findings actually supported by the evidence, including negative results.

Manuscript–codebase mismatch (W6/E5). Our audit identified a version-control failure: the submitted Overleaf results/ directory was last synchronized on 23 September 2026 using outputs from an earlier 23-language analysis, whereas the active repository was rerun on 26 September over 100 languages. Some submitted figures and descriptions therefore do not match the current analysis. We take responsibility. We have traced specific stale values, corrected identified defects, and are still reconciling all affected tables and figures against versioned outputs. Importantly, this synchronization failure does not explain separate problems such as the agreement-pivot bug, misdescribed Gold selection, or corrupted judge output. These are disclosed and addressed independently in Comment 02 and the responses below.

Main new evidence:

  • Judge reliability (W4/Q6): On 26,346 identical v2 responses scored by four judges, Fleiss' κ is 0.288, increasing to 0.461 when Mistral-Large is excluded. Pairwise Cohen's κ for Qwen3.5-122B vs Mistral is 0.123; label decisions, especially partial leakage, materially depend on the judge.
  • English controls (W2/Q5): On matched source roots for two generators, language-shift ASR changes by −0.08 to +2.57 pp under Qwen but +12.91 to +30.78 pp under Mistral. Restricting to unambiguous full_compliance reduces the shift to −0.84 to +3.04 pp across both judges.
  • Model-set sensitivity (W2/W5/Q11): The submitted 1.6× open/closed gap cannot be generalized. On seven current v2 generators (three open, four closed), the direction reverses under all four judges; the four-judge majority gives 8.42% open vs 9.77% closed. We withdraw the general access-type conclusion.
  • Guardrails (W1/Q10): Ten published guardrails were newly evaluated on 3,875 multilingual harmful prompts and a benign control. The best attains 76.0% harmful recall; strict evasion differs by +21.56 pp between high- and low-resource tiers. On matched English/translated v2 inputs, Aegis-2.0 recall falls 77.14% → 22.98%, compared with 80.86% → 64.61% for CultureGuard-v3. This is not cultural localization, and we remove that capability claim.
  • Dataset audit (W3/W6/Q3): We reconcile 10,362 generated prompt rows, 10,262 distinct evaluation instances, and 10,000 released prompts; the headline 13.78% ASR pooled four quality buckets, not Gold alone. An independent LaBSE audit covers 1,178,928 translations, but native-speaker validation (several languages) remains incomplete and waiting for Non Human Subject Resarch Determination results from IRB office.

Claims withdrawn or narrowed: the general open/closed safety claim; causal translation-quality effects; unsupported cultural robustness; per-language/typology rankings from non-matched prompts; the purported 97.06% “judge consensus” (actually parseability); and the erroneous 36.8% agreement claim (cross-generator agreement recalculates to 32.65%).

We first detail the W6/E5 manuscript–codebase synchronization failure, then respond in focused comments on W1+Q10; W2+Q5; W3+Q3; W4+Q6; W5+Q11; remaining W6 defects; W7; E1–E5; Q1; Q2, followed by a revision/limitations summary. We distinguish completed results from planned work and do not treat agreement among automated judges as human validity.

AuthorsOpenReview ↗

Comment 02 — W6/E5: manuscript–codebase synchronization and audit

Response to W6/E5 — Manuscript–Codebase Mismatch and Corrective Audit

We thank the reviewer for identifying discrepancies between the paper, dataset card, and experiments. Our audit confirmed a serious manuscript–analysis synchronization failure, and we take responsibility for it.

Root cause and scope. The submitted Overleaf project contained a results/ directory last synchronized on 23 September 2026 with outputs from an earlier 23-language analysis. The active analysis codebase was rerun on 26 September 2026 using all 100 languages, but the regenerated artifacts were not fully propagated to Overleaf before submission. Consequently, portions of the submitted paper retained old per-language figures while describing the 100-language evaluation. This explains some stale results, not all the errors reported by the reviewer; we distinguish independent analysis and methodology defects below.

Concrete findings and corrections:

  1. Stale values. Section 2.4 reports an open–closed gap of +3.9 pp and a language-harm spread of 4.64×. The recomputed 100-language descriptive values are +6.247 pp (open 16.90%, closed 10.65%) and 3.963×. Even these updated sample-specific figures do not support the original general open/closed claim, which we withdraw after new sensitivity tests (W2/W5/Q11).
  2. Separate agreement-code defect. The analysis pivoted on root_id rather than (root_id, language), silently reducing 10,262 evaluation instances to 5,285 roots. Corrected cross-generator agreement is 32.65%, not 36.8%. This bug is not caused by the Overleaf sync; corrected model intersections and distributions appear in our W6 response.
  3. Corpus and methods descriptions. The main 13.78% ASR uses 61,536 responses across four quality buckets, not Gold alone. v1 and v2 followed distinct selection gates. The paper also misstated the judge (Qwen2.5 instead of Qwen3.5-122B-A10B), v2 stratification, language-family count (17 instead of 16), and the scope of COMET missingness. These are documentation/methodology errors requiring separate corrections (W3/Q3), not just copied old plots.
  4. Corrupted judge artifact. One closed-judge output file broadcasts the same verdict across thousands of distinct rows. The published κ = 0.271 is therefore withdrawn until a validated recomputation; this is an independent output-writing defect (W4/Q6).

Corrective procedure. We are regenerating tables and figures from the active codebase, checking join keys, row counts, source populations, and model/judge configurations, and reconciling them with the manuscript and dataset card. Revised results will identify the underlying artifact and experimental population; unresolved figures will not be presented as verified. We will distinguish results already recomputed from analyses still requiring verification. The revision must acknowledge these failures visibly rather than silently substitute new numbers.

We recognize that synchronization, code correctness, and manuscript verification are authors' responsibilities. This explanation accounts for how some inconsistencies arose, not an excuse for their appearance in the submitted paper.

AuthorsOpenReview ↗

Comment 03 — W1 and Q10: novelty, guardrails, CultureGuard

Response to W1 + Q10 — Contributions, prior work, and guardrail evidence

We accept that the submitted novelty and capability claims were too broad. RQ1 is replication at larger language scale, not the discovery that low-resource languages can be less safe. We withdraw the statement that earlier work was generally “limited” in methodology and will explicitly credit the reviewer-identified precedents for IU/invalid outputs, comprehension-aware ASR, and graded severity labels (including MultiJail, XL-SafetyBench, and StrongREJECT). The defensible methodological contribution is the systematic integration of established techniques into a large-scale, multidimensional safety audit spanning 100 languages. Our analysis examines model behavior, language-resource tiers, translation quality, response severity, judge reliability, guardrail robustness, and evaluation artifacts, revealing measurement failures and confounding factors that isolated analyses may overlook. We are additionally conducting human validation of translation quality and model-response quality, including comparisons between human annotations and LLM-judge assessments to evaluate judge reliability across languages and resource tiers. These human-validation analyses are pending (hoping to be done before Nov 18th) and are not presented as completed findings. We will discuss CultureGuard substantively: it shares AEGIS 2.0 source material but studies culturally adapted safety data/guard training, whereas ours uses literal translations for generator and judge evaluation.

The submitted guardrail claim was unsupported; we have now run that experiment. Ten published guardrails—CREST, Qwen3Guard-Gen-8B, GuardReasoner, AprielGuard, Nemotron Safety Guard, LlamaGuard-3, LlamaGuard-Permissive, X-Guard, WildGuard, and ShieldGemma—were evaluated on 3,875 v1 multilingual harmful prompts (98 languages; 817 source roots; 19 categories) and 4,900 benign controls. Best harmful recall is 76.0% (95% CI [72.15, 79.60]), hence 24.0% strict evasion. Strict evasion rises 45.4% → 66.9% from high- to low-resource languages (+21.56 pp, 95% CI [17.98, 25.17]). Recall alone is not deployability: GuardReasoner flags 56.1% of benign controls.

A separate matched v2 experiment tested two NVIDIA guards on 3,820 English source prompts and their own translations, holding prompt identity and evaluation protocol fixed:

GuardEnglish harmful recallTranslated recallChange
Aegis-2.077.14%22.98%−54.16 pp
CultureGuard-v380.86%64.61%−16.26 pp

The benign false-positive rates also fall after translation (6.00% → 0.78% for Aegis; 12.00% → 6.56% for CultureGuard), consistent with reduced sensitivity rather than more conservative blocking. These results reveal substantial guard-dependent multilingual coverage differences but do not causally isolate translation effects or demonstrate cultural adaptation within our benchmark. Because this experiment evaluates the 3,820 v2 prompts spanning 89 languages, and CultureGuard's language coverage varies within this subset, we will report results stratified by language coverage rather than generalizing them to all 100 benchmark languages.

We therefore delete the Cultural Robustness checkmark and keep any Guardrail checkmark strictly scoped to the newly performed prompt-level detection study, explicitly distinguished from the submitted paper. Table 6 will also correct HarmBench's translation mark and split “multimodal” from “multi-model.” The principal new insight is the measurement-validity audit, not a claim that prior research overlooked multilingual failures.

Comments on the paper

2 comments
Area_Chair_qSXdOpenReview ↗

Please acknowledge author response

Dear Reviewers,

Please acknowledge the authors' response and adjust the scores if appropriate.

Best,

AC

AuthorsOpenReview ↗

Revised manuscript uploaded: Appendix B lists every corrected claim and reference

Dear Area Chair,

We have uploaded a revised manuscript (all changes marked in blue) and posted point-by-point responses to all three reviewers. We are writing because one item is properly yours rather than the reviewers': the ethics flag raised by Reviewer fyep.

On the flag. The substance of it was correct, and we apologise for it. Twelve bibliography entries were wrong, including one citing an unrelated orthopaedics article and one entry reused for three different works. Separately, seven claims contradicted our own tables or released data: most seriously, the benchmark was described as drawn exclusively from the highest quality tier under a conjunctive filter, when the filter as implemented is disjunctive and 40.9% of evaluated responses come from lower tiers. The immediate cause was that the manuscript described the pipeline we intended rather than the one the code ran, and was never reconciled against the artifacts; the reviewer's characterisation of that is accurate. Both findings were verifiable from the data we released. We have corrected every item: each reference is now checked against a primary record (arXiv page or ACL Anthology), and each contradicted claim is listed individually, with its correction, in Appendix B.

On the science. The revision also answers the substantive weaknesses: eight published guardrails evaluated on our own prompts (§A.4), two further translation systems (§A.3), a four-judge agreement panel (§A.1), an independent quality arbiter never used in gating (§A.8), root-clustered bootstrap confidence intervals (§A.11), and a full reproducibility appendix with prompts, model identifiers, decoding settings and run dates (§E). The main body now ends on page 9.

What the revision does not fix. Several results went against us and we report them as such: the open/closed gap reverses on current models, so that claim is withdrawn; our own harm rubric inverts on benign prompts, so we report no over-refusal rate from it; and the resource-tier contrast is confounded with prompt identity by construction, since of 5,285 source roots only one appears in both tiers. Human validation is specified but unrun waiting NHSRD/IRB approval (will hear back from IRB this Friday).

We would like to request you only that the ethics flag be assessed against the revised manuscript, since the specific defects it cites have been corrected and the corrections are individually checkable.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.