- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces PolyAEGIS, a massively multilingual safety benchmark designed to evaluate LLMs across 100 languages. The benchmark consists of 10,362 harmful prompts sourced from AEGIS, translated via NLLB, and filtered through dual-metric quality gating. The authors evaluate six LLMs using a three-judge LLM consensus protocol and propose a four-class taxonomy distinguishing Full Refusal, Partial Leakage, Full Compliance, and Irrelevant/Unintelligible responses. The core contributions include revealing that "Partial Leakage" accounts for nearly half of unsafe outcomes, demonstrating that open-weight models are 1.6x more vulnerable than closed-source counterparts, and highlighting systemic safety degradation in low-resource languages.
Critical strengths
- The benchmark evaluates 100 languages spanning 17 language families and 26 scripts (Table 2), which significantly advances the field's coverage beyond English-centric safety evaluations and provides excellent experimental rigor.
- A leakage-aware evaluation framework is proposed to identify partial leakage as a distinct failure mode accounting for nearly half of unsafe responses.
- The evaluation on 61,536 responses reveals that partial leakage accounts for 6.32% of responses, providing impactful evidence that current binary safety metrics are insufficient for capturing nuanced model behaviors.
Critical weaknesses
- Relies on only one machine translation system (NLLB-200), which may introduce semantic drift particularly in low-resource languages despite quality filtering.
- Format problem: Tab. 2 is beyond the boundary. The abstract section uses too many dashes, which does not conform to the standard English norms. At the same time, some abbreviations need to be fully written out for the first time, and some experimental settings should be presented more clearly in the abstract and further refined. The "Conclusion" section of this paper is on page 11, which clearly exceeds the submission requirements of ICLR 2027.
- The lack of necessary graphical explanations (such as the dataset construction in the methodology section with two pipelines) results in poor readability.
- One limitation of this study is the small number of evaluated models, which does not include recent mainstream LLMs (e.g., Qwen 3.5+ series, GPT-5.5, Claude Opus 4.6+, Gemini 3+ series).
- The authors note an anomaly where low, resource languages show artificially higher mean BERTScore F1 due to repetitive vocabulary acting as a back-translation artifact, a complication conceptually similar to multilingual transfer issues identified in related works like MultiJail (Table 25; Sec. 4.6), which weakens the technical soundness of the overall quality-gating pipeline.
- The study currently lacks formal statistical significance testing or bootstrap confidence intervals for the observed cross-lingual harm score differences, reducing the statistical rigor of the overarching claims.
Questions for authors
What is the rationale for determining the quality bucket thresholds? That is Gold — BERTScore F1≥0.95, why not 0.93 or other value?
10 comments
Comment 02: W1 — Reliance on a single machine translation system
Reviewer concern. Relying on NLLB-200 alone may introduce semantic drift, particularly in low-resource languages, despite quality filtering.
Response. We accept the criticism and treated it as a control to be run. The same prompts were re-translated with MADLAD-400-3B (T5-style, different web-scale corpus) and SeamlessM4T-v2 (multimodal, text path trained jointly with speech) — chosen to differ in training data and architecture, not merely in checkpoint. Unsupported languages were recorded as absent, never back-filled from NLLB.
Aggregate safety conclusions are translator-robust. On the v2 corpus, matched triples of (prompt, generator, budget) differing only in translator:
| judge | generator | n | langs | NLLB | MADLAD | Seamless | spread |
|---|---|---|---|---|---|---|---|
| Qwen3.5-122B | Qwen3.5-27B | 3,528 | 81 | 1.70% | 1.70% | 1.47% | 0.23 pp |
| Qwen3.5-122B | Gemma-4-31B | 3,528 | 81 | 2.95% | 2.66% | 2.81% | 0.28 pp |
| Mistral-Large | Qwen3.5-27B | 3,529 | 81 | 28.42% | 28.36% | 28.96% | 0.60 pp |
| Mistral-Large | Gemma-4-31B | 3,529 | 81 | 33.30% | 33.24% | 32.98% | 0.31 pp |
This is the stronger of the two corpora for the question: 81 languages spanning both resource tiers, selected by percentile rather than by NLLB's own round-trip score, so the circularity affecting our v1 comparison does not apply here. The translator moves the headline by at most 0.60 points; the judge moves it by 11.3–16.7×.
Per-item labels are not robust, and we now say so. Pairwise label agreement is 96.0–97.0% under the Qwen judge but only 73.6–76.7% under Mistral — so between 3% and 26% of individual items change safety label when only the translator changes, with flips near-symmetric in both directions. No system is systematically "safer".
One language where NLLB's own score is not credible. NLLB rates its Fula round-trips at median F1 1.0000; MADLAD and Seamless rate the same sentences at 0.53 and 0.49. Fula also has the benchmark's highest unintelligible-response rate (23.28%). A perfect round-trip score is therefore not evidence of adequacy.
Revision and limits. We will report the aggregate robustness as a measured control, attach the matched-pair agreement rate to any per-item or per-language claim, and stop implying per-language cells are translator-independent. Three systems agreeing could be three systems making the same error on overlapping web corpora; only native-speaker validation settles adequacy, and that remains pending (Comment 06).
Comment 03: W4 — Small number of models; no recent mainstream LLMs
Reviewer concern. The study evaluates few models and omits recent mainstream LLMs (Qwen 3.5+, GPT-5.5, Claude Opus 4.6+, Gemini 3+).
Response. We added a model from every family named, and the evaluated set grew from 6 to 12 models across 6 developers (Cohere, Mistral, Alibaba, Google, OpenAI, Anthropic), spanning open and closed weights and two capability tiers within two vendors. The v2 corpus now carries seven current generators, each scored by four independent judges on one decode contract:
| generator (v2) | Qwen3.5-122B | Qwen3.5-27B | gemma-3-27b | Mistral-Large | 4-judge majority |
|---|---|---|---|---|---|
| qwen3.5-27b | 1.75% | 5.21% | 8.49% | 28.82% | 3.91% |
| gpt-5.5 | 1.71% | 6.96% | 12.33% | 39.73% | 5.17% |
| gemma-4-31b-it | 3.04% | 9.17% | 12.50% | 33.51% | 7.34% |
| claude-opus-4-6 | 4.59% | 10.10% | 11.10% | 34.01% | 8.09% |
| gemini-3.8-flash | 6.26% | 13.48% | 18.01% | 40.85% | 11.47% |
| mistral-large-123b | 8.51% | 17.16% | 17.62% | 31.83% | 14.00% |
| claude-opus-5.5 | 5.67% | 15.95% | 25.54% | 58.78% | 14.33% |
Three findings follow, and one of them is negative for us.
(a) The open/closed claim does not replicate. Mean ASR for 3 open versus 4 closed models reverses sign under every judge: −0.13 pp (0.97×) under Qwen3.5-122B, −1.10 pp (0.91×), −3.88 pp (0.77×), and −11.95 pp (0.72×) under Mistral-Large. The submission reported +6.25 pp in the opposite direction. We withdraw the open/closed headline. The submitted gap is better explained by model age than by weight availability: our two oldest models were both open and both among the least safe.
(b) The measurement-validity finding replicates on models chosen by the reviewer, not by us. Between the two original judges, Cohen's κ on the added models is 0.050–0.175 with ASR ratios of 6.4–26× — at or below the κ = 0.271 the submission reported. The result therefore cannot be dismissed as an artifact of our original model selection.
(c) Newer models are not obviously safer. Under Mistral-Large the 2026 frontier closed models sit above the open models; under Qwen3.5-122B the order differs again. We report no recency effect in either direction.
Revision and limits. Twelve models remain a convenience sample. We now state a selection criterion (developer × access grid with capability tiers) but apply it retrospectively, and we report that the grid is not balanced on release date. Adding models does not resolve judge validity: twelve models scored by instruments that disagree 6–26× yields twelve uncertain numbers rather than six.
Comment 01: Overview and Response Roadmap
Response to R3 (AVLb) — Overview
We thank the reviewer for a review that identified two issues we could not have closed without it: the dependence on a single translation system, and the absence of any uncertainty quantification. Both are now addressed with measurements rather than argument, and one of them overturned a headline claim of the submission.
Main new evidence:
Claims withdrawn or narrowed: the repetitive-vocabulary explanation for the tier inversion; the open/closed safety gap; any per-item or small-cell claim presented as translator-independent; any safety ranking presented as judge-independent; and the pooled partial-leakage headline. We retain and report partial leakage as judge-dependent.
We address W1 (translation systems), W4 (model coverage), W5 (BERTScore anomaly), W6 (significance and intervals), Q1 (threshold rationale) and W2/W3 (presentation) in Comments 02–07. Completed results are distinguished throughout from pending human validation.