acceptodds
Under review as a conference paper at ICLR 2027

ĪCÁRŪS: Flying Over Safety Alignment to Jailbreak Large Language Models via Ancient Grammar

Abstract

Large language models are trained to refuse harmful requests, yet jailbreak attacks still bypass this safety alignment. We show that expressing a harmful request in an ancient language reliably bypasses alignment, and that the effect is carried by grammar rather than vocabulary, concentrated in its orthographic rules. The most effective grammar rules fragment harmful words into many tokens and shift the input embedding away from English, while barely changing the visual form of the prompt. Building on this, we propose Iterative Combinatorial Ancient Rule Search ĪCÁRŪS, a black-box attack that uses beam stack search over a pool of ancient grammar rules to find an effective combination for each request. Across five open- and closed-source models, ĪCÁRŪS outperforms strong baselines on StrongREJECT and attack success rate, and its plain-English output is directly usable. On a widely-used model, an ancient-language translation alone raises the attack success rate from 15% to 48%, and ĪCÁRŪS raises it as high as 85%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak, safety, ethical considerations, low resource language

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TvX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that ancient-language grammar rules, particularly orthographic rules, can increase jailbreak success rates by introducing token fragmentation and shifting input embeddings. Based on this observation, the authors propose ĪCÁRŪS, which employs Beam Stack Search to identify effective combinations of grammar rules. Experiments on five open-source and closed-source LLMs demonstrate relatively high attack success rates.

Critical strengths
  1. Interesting research perspective. The paper investigates LLM jailbreaks from the perspective of ancient-language grammar and analyzes the different effects of vocabulary and grammatical transformations.

  2. Systematic experimental analysis. The authors conduct ablation studies, token fragmentation analysis, and rule combination experiments, providing substantial empirical evidence.

  3. Practical applicability. The proposed method requires neither model parameters nor gradient access, and successful attacks produce directly usable English responses.

Critical weaknesses
  1. Insufficient mechanistic understanding. The paper primarily relies on correlation analyses to establish relationships between token fragmentation, embedding shifts, and jailbreak success. However, it does not adequately explain why these transformations weaken safety alignment while preserving the model's ability to understand harmful requests. The analysis remains largely at the input representation level.

  2. Limited methodological novelty. The use of special-character perturbations resembles existing Unicode-based obfuscation attacks. Moreover, Beam Stack Search essentially performs black-box optimization over combinations of predefined rules, without demonstrating clear technical distinctions from existing iterative jailbreak optimization methods.

  3. Insufficient evaluation on recent models. The commercial models evaluated mainly include GPT-3.5-turbo, GPT-4o, and Gemini 2.5 Flash. The absence of more recent commercial LLMs makes it difficult to assess the effectiveness of the proposed attack against more advanced safety alignment mechanisms.

  4. Limited significance of the empirical improvements. Numerous existing jailbreak methods have already achieved high attack success rates, making further ASR improvements alone less compelling. In particular, the standalone ĪCÁRŪS method does not consistently outperform existing baselines, while its strongest results rely on combinations with other jailbreak techniques.

Questions for authors
  1. Can the authors provide a deeper mechanistic explanation of why token fragmentation weakens safety judgments without substantially impairing semantic understanding? Could analyses of internal representations or safety-related activations help validate this hypothesis?

  2. What are the key technical innovations of ĪCÁRŪS compared with existing special-character-based attacks and iterative black-box jailbreak optimization methods?

  3. Can the authors extend the evaluation to more recent commercial LLMs and further distinguish the performance gains achieved by standalone ĪCÁRŪS from those obtained through combinations with existing jailbreak methods?

3 comments

We appreciate the thoughtful feedback on our paper, and we included additional comments to further improve our work.


[W1, Q1] Mechanism beyond correlation

Thank you for the comment. We further investigated why token fragmentation weakens safety alignment through controlled interventions and additional comprehension analysis.

Our interventions identify generic token fragmentation as a key mechanism behind ancient-language jailbreaks. We also find that moderate transformations increase ASR, while excessive transformations disrupt comprehension and reduce attack success.

We will clarify these findings in Section 3 and distinguish our behavioral evidence from direct analysis of internal safety activations.

[A for W1, Q1] Controlled interventions support token fragmentation as the key mechanism, while comprehension analysis shows that excessive transformations cause comprehension failure rather than safety refusal.


Details

I. Controlled interventions

We test token fragmentation by replacing ancient marks at matched token counts, varying random mark density, and removing combining marks.

Table A. Binary ASR (%) under matched token counts

ModelNoneAncientDotCaretHyphenAsteriskAcuteRandomZWSP
Llama-3152436333331262117
Qwen367141113132159

Generic marks match or outperform ancient marks, while ZWSP shows little improvement.

Table B. Token inflation, embedding shift, and ASR under random mark density

Densityρ\rhoδ\deltaASR
010.0015%
0.570.7746%
1130.8162%
2220.8461%
3310.8556%

ASR increases from 15% to 62%, then declines at higher fragmentation levels.

Table C. Effect of removing combining marks on Llama-3

MethodOriginal SRAfter stripping
ĪCÁRŪS0.830.13

Removing combining marks reduces StrongREJECT from 0.83 to 0.13, further supporting the role of orthographic perturbations.

II. Comprehension analysis

We additionally analyzed vocabulary and grammar interactions and classify responses as COMPLY (C), REFUSE (R), or INCOHERENT (I) using GPT-4o.

Table D. Vocabulary and grammar interactions on Llama-3

Family+Grammar+Vocab+BothINTER (log-odds)
Arabic0.340.220.00−6.31
Sanskrit0.330.220.01−5.34
Chinese0.130.220.20−1.17

Arabic and Sanskrit show strong negative interactions, with StrongREJECT dropping close to zero when both transformations are applied.

Table E. Response classification (%) under vocabulary and grammar transformations

FamilyVocab-only (C / R / I)Grammar-only (C / R / I)Both (C / R / I)
Arabic17 / 68 / 1532 / 60 / 82 / 2 / 96
Sanskrit18 / 74 / 812 / 37 / 511 / 4 / 95
Chinese19 / 78 / 311 / 89 / 018 / 77 / 5

Under combined transformations, 96% of Arabic and 95% of Sanskrit responses become incoherent, with only 2% and 4% classified as refusals. In contrast, Chinese responses remain largely coherent and predominantly refuse.

These results suggest that moderate transformations can increase ASR while preserving comprehension, whereas excessive transformations disrupt understanding and reduce attack success rather than restore safety alignment.


[W2, Q2] Novelty Beyond Existing Attacks

Thank you for the comments on methodological novelty. We clarify that our contribution is the analysis of a linguistic vulnerability and an attack method built on these findings, rather than a new search algorithm.

While prior methods compose transformations or iteratively refine prompts, we investigate which linguistic properties enable jailbreaks and why. We separate vocabulary and grammar, identify orthographic token fragmentation as a key mechanism, and analyze interactions between rules.

These findings guide ĪCÁRŪS, which combines grammar rules while keeping the request and response in plain English.

[A for W2, Q2] Our contribution is identifying a linguistic vulnerability and using these findings to design a rule combination attack, rather than introducing a new black-box search algorithm.

[W3, Q3.1] Evaluation on Recent Models

Thank you for the comment. We selected five target models (Llama-3, Qwen3, Gemini 2.5, GPT-3.5, GPT-4o) to facilitate comparisons with prior jailbreak studies and provide a reference for future evaluations.

Recent studies continue to evaluate the same model families, including GPT, Gemini, Llama, and Qwen, as shown below.

[A for W3, Q3.1] Our model selection enables direct comparison with prior jailbreak studies and provides a consistent reference for future research.


Details

I. Model selection in prior jailbreak studies

Table F. Representative jailbreak studies and target models

Jailbreak paperVenueTarget models
CipherChat (Yuan et al.)ICLR 2024GPT-3.5-turbo, GPT-4
AutoDAN (Liu et al.)ICLR 2024Vicuna, Llama-2, GPT-3.5/4
ArtPrompt (Jiang et al.)ACL 2024GPT-3.5/4, Gemini, Claude, Llama-2
PAIR (Chao et al.)SaTML 2025GPT-3.5/4, Vicuna, Llama-2, Claude, Gemini
FlipAttack (Liu et al.)ICML 2025GPT-3.5/4/4o, Claude-3.5, Gemini, Llama-3, Mixtral
Simple Adaptive Attacks (Andriushchenko et al.)ICLR 2025GPT-3.5/4o, Claude-3/3.5, Llama-2/3, Gemma-7B, Vicuna-13B, Mistral-7B
Adversarial Reasoning (Sabbaghi et al.)ICML 2025Llama-2-7B, Llama-3-8B, Llama-3-8B-RR, R2D2
Best-of-N Jailbreaking (Hughes et al.)NeurIPS 2025GPT-4o/4o-mini, Claude-3.5-Sonnet, Claude-3-Opus, Gemini-1.5-Flash/Pro, Llama-3-8B
One Step Is Enough / REDA (Jiang et al.)ACL 2025GPT-3.5, GLM-4, Llama-3.1-8B, Qwen2-7B, Vicuna-13B
RACE (Ying et al.)EMNLP 2025GPT-4/4o, Gemini-1.5-Pro/2.0, o1, DeepSeek-R1, Qwen2-7B, Gemma-2-9B, GLM-4-9B
SeqAR (Yang et al.)NAACL 2025GPT-3.5/4/4o, Gemini-1.5-Pro, Llama-2-7B, Llama-3-8B
SEMAICLR 2026GPT-4o, GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-3B, gpt-oss-20B
Align to Misalign (Koo et al.)ICLR 2026GPT-4o, GPT-4o-mini, Claude-3.5-Haiku, Claude-Sonnet-4, Llama-3.1-8B
SlotGCGICLR 2026Llama-2-7B/13B, Llama-3.1-8B, Mistral-7B, Vicuna-7B, Qwen-2.5, GPT-3.5/4o, Gemini-2.0/2.5
MetisICML 2026GPT-3.5/4o, o1, GPT-5-chat, Gemini-2.5-Pro, Claude-3.7, Grok-3, Llama-3-8B/70B, Qwen2.5-7B
AE-CoTICML 2026o1-mini, o3-mini, GPT-5, DeepSeek-R1, Qwen3-235B, Gemini-2.5-Flash, Gemini-3-Pro
MJ Multi-turn (Park et al.)NeurIPS 2026GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B, Mistral-7B, gpt-oss-20B

These studies show that the model families used in our evaluation remain common choices in jailbreak research, supporting comparisons with both existing and future work.


[W4, Q3.2] Significance of gains and standalone versus overlay

Thank you for the comments. Our contribution goes beyond ASR improvements by identifying generic token fragmentation as the mechanism behind ancient-language jailbreaks.

We report standalone and overlay results separately in Table 3, where the overlays achieve the highest ASR on four of five models. We further evaluate their improvements under the same query budget.

[A for W4, Q3.2] We clarify the distinct roles of standalone and overlay configurations and show that ĪCÁRŪS improves existing attacks under matched query budgets.


Details

We compare ĪCÁRŪS with the unmodified prompt and existing attacks under the same query budget on Llama-3.

Table G. Binary ASR (%) across query budgets on Llama-3

MethodK=1K=1K=5K=5K=10K=10K=20K=20
Vanilla0000
ĪCÁRŪS002948
GCG-T2327.83132
GCG-T + ĪCÁRŪS2643.95660
Poems4345.446.248
Poems + ĪCÁRŪS4673.78184
RefusalSupp3642.54547
RefusalSupp + ĪCÁRŪS3863.98085

Vanilla represents the unmodified prompt evaluated without search.

At K=20K=20, standalone ĪCÁRŪS achieves 48% ASR, while its overlays improve existing attacks by 28 to 38 percentage points.

I appreciate the authors' detailed rebuttal and additional experiments. In particular, the controlled experiments on token fragmentation and comprehension analysis provide stronger empirical evidence for the observed jailbreak behavior. The query-budget-matched comparisons also address my concerns regarding evaluation fairness and demonstrate meaningful improvements when ĪCÁRŪS is combined with existing jailbreak methods.

However, my main concerns remain unresolved:

  1. Insufficient mechanistic understanding. While the new experiments provide stronger evidence that token fragmentation contributes to jailbreak effectiveness, they do not fully explain why such perturbations weaken safety alignment while preserving semantic understanding. The analysis still lacks direct evidence regarding internal safety-related mechanisms.
  2. Limited methodological novelty. The authors acknowledge that the search algorithm itself is not novel. More importantly, the new experiments show that generic character perturbations can match or even outperform ancient-language transformations, further raising questions about the distinctive contribution of ancient grammar. A fair comparison using the same search algorithm and query budget remains necessary.
  3. Limited evaluation on recent models. The authors justify their model selection by referencing prior studies but do not provide additional experiments on more recent commercial LLMs. Therefore, the generalizability of the findings to modern safety-aligned models remains unclear.

Overall, the rebuttal strengthens the empirical evaluation but does not sufficiently resolve my fundamental concerns regarding mechanistic insight and technical novelty. I therefore decide to maintain my original rating.

Reviewer_apbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discovers that ancient languages ​​can bypass secure alignment mechanisms primarily due to their grammatical features rather than their vocabulary, and proposes Iterative Combinatorial Ancient Rule Search (ĪCÁRŪS) based on this finding. This method searches for the most efficient rule combination for each request by performing a beam stack search within an ancient grammatical rule base.

Critical strengths
  1. This paper analyzes and verifies that the reason ancient languages ​​can bypass security mechanisms lies mainly in their grammar rather than their vocabulary, and points out that token fragmentation is a significant factor leading to the failure of secure alignment.

  2. This paper proposes a black-box attack algorithm based on ancient language grammar. By performing a beam stack search within an ancient grammatical rule base, it finds efficient rule combinations for different requests, achieving a high ASR (Automatic Sequence Response).

Critical weaknesses
  1. This paper attributes the attack effect primarily to token fragmentation, but this phenomenon is not unique to ancient language grammar. Other character-level perturbation methods, such as Unicode character perturbation, can also cause token fragmentation. Therefore, the unique advantages of ancient language grammar rules compared to general character perturbation methods remain to be further explained. Furthermore, the definitions of vocabulary and grammar in this paper are not clear enough. From a linguistic perspective, grammar typically includes syntax and morphological syntax, while some cases in this paper only involve character-level modifications such as adding diacritics; whether these constitute grammatical transformations remains questionable.

  2. This paper lacks ablation experiments for each component of ĪCÁRŪS. In addition, some closed-source model versions used in the experiments are relatively old, and the query budgets of different methods in the main experiments show significant differences, affecting the fairness of the comparison. It is worth noting that ĪCÁRŪS's ASR on some models is even lower than some baselines requiring only a single query.

Questions for authors
  1. How exactly is the Query Budget calculated in this paper? It is recommended that the authors further clarify the statistical method for the number of queries.

  2. Why does this paper use both StrongREJECT Score and ASR as evaluation metrics? What is the relationship between the two? Specifically, should samples with a StrongREJECT Score exceeding a certain threshold be considered successful attacks, and should ASR be calculated accordingly?

4 comments

We appreciate the thoughtful feedback on our paper, and we included additional comments to further improve our work.


[W1] Ancient Language Specificity and Grammar Definition

Thank you for the comment. Our study investigates why ancient languages bypass safety alignment and identifies generic token fragmentation as the key mechanism behind this effect, rather than any property unique to ancient languages.

We separate vocabulary and grammar and find that the effect is concentrated in orthographic rules. Our controlled experiments further show that generic character transformations reproduce the effect of ancient orthography, supporting this explanation.

We also clarify the distinction between vocabulary and grammar, including why orthography is classified as part of grammar in classical traditions.

[A for W1] We trace the jailbreak effect of ancient languages to generic token fragmentation and clarify the classification of orthographic transformations within grammar.


Details

I. Identifying the mechanism behind ancient-language jailbreaks

We replace ancient marks with generic marks at matched token counts and vary the density of random combining marks on English prompts.

Table A. Binary ASR (%) under matched token counts

ModelNoneAncientDotCaretHyphenAsteriskAcuteRandomZWSP
Llama-3152436333331262117
Qwen367141113132159

Generic marks match or outperform ancient marks, while the nonfragmenting ZWSP remains close to the baseline.

Table B. Token inflation, embedding shift, and ASR under random mark density

Densityρ\rhoδ\deltaASR
010.0015%
0.570.7746%
1130.8162%
2220.8461%
3310.8556%

Random combining marks raise ASR from 15% to 62% without ancient grammar, further supporting token fragmentation as the mechanism.

II. Defining vocabulary and grammar

We distinguish vocabulary and grammar by separating lexical substitutions from changes to orthography, word forms, and word order.

In classical grammatical traditions, orthography was also treated as part of grammar [1, 2, 3, 4]. We therefore include diacritic transformations as orthographic rules and classify our 15 grammar rules into three linguistic levels.

References

[1] Atherton, Catherine. "What Every Grammarian Knows?" The Classical Quarterly, 46(1), 239–260, 1996.

[2] Colson, Francis H. "The Grammatical Chapters in Quintilian I. 4–8." The Classical Quarterly, 8(1), 33–47, 1914.

[3] Scappaticcio, Maria Chiara. "Verrius Flaccus, His Alexandrian Model, or Just an Anonymous Grammarian? The Most Ancient Direct Witness of a Latin Ars Grammatica." The Classical Quarterly, 70(2), 806–821, 2020.

[4] Stojanov, Tomislav. "Orthographies in Grammar Books: Antiquity and Humanism." 2018.

[W2] Component Ablation, Model Selection, and Performance Comparison

Thank you for the comments. We clarify our component analysis, model selection, and query budget evaluation.

We analyze rule stacking and 30 additional synthetic rules, showing the benefit of combining rules and the robustness of our findings beyond the original rule pool.

We selected established model families for comparison with prior studies. We also provide a query-matched evaluation showing that ĪCÁRŪS improves existing attacks under the same budget.

[A for W2] Our analyses demonstrate the benefits of rule stacking and generalization to additional rules. We clarify our model selection and show that ĪCÁRŪS provides additional gains under matched query budgets.


Details

I. Component analysis and rule expansion

We evaluate the effect of stacking up to three rules and extend the original 15 rules with 30 additional synthetic rules across five language families and three linguistic levels.

Table C. Mean StrongREJECT by number of stacked rules

Rules stackedMean StrongREJECT
0 (English)0.048
10.092
20.140
30.164

StrongREJECT increases as more rules are combined, showing the benefit of rule stacking.

Table D. Correlation between token fragmentation and ASR on Llama-3

Rule setρ\rho vs. ASRδ\delta vs. ASR
Original 15 rules0.690.66
Additional 30 rules0.940.88

The relationship remains strong on the additional rules, showing that our findings are not limited to the original rule pool.

II. Model selection

We selected five target models (Llama-3, Qwen3, Gemini 2.5, GPT-3.5, GPT-4o) for comparability with prior jailbreak studies. Recent studies continue to evaluate the same model families, including GPT, Gemini, Llama, and Qwen, as shown below.

Table E. Representative jailbreak studies and target models

Jailbreak paperVenueTarget models
CipherChat (Yuan et al.)ICLR 2024GPT-3.5-turbo, GPT-4
AutoDAN (Liu et al.)ICLR 2024Vicuna, Llama-2, GPT-3.5/4
ArtPrompt (Jiang et al.)ACL 2024GPT-3.5/4, Gemini, Claude, Llama-2
PAIR (Chao et al.)IEEE SaTML 2025GPT-3.5/4, Vicuna, Llama-2, Claude, Gemini
FlipAttack (Liu et al.)ICML 2025GPT-3.5/4/4o, Claude-3.5, Gemini, Llama-3, Mixtral
Simple Adaptive Attacks (Andriushchenko et al.)ICLR 2025GPT-3.5/4o, Claude-3/3.5, Llama-2/3, Gemma-7B, Vicuna-13B, Mistral-7B
Adversarial Reasoning (Sabbaghi et al.)ICML 2025Llama-2-7B, Llama-3-8B, Llama-3-8B-RR, R2D2
Best-of-N Jailbreaking (Hughes et al.)NeurIPS 2025GPT-4o/4o-mini, Claude-3.5-Sonnet, Claude-3-Opus, Gemini-1.5-Flash/Pro, Llama-3-8B
One Step Is Enough / REDA (Jiang et al.)ACL 2025GPT-3.5, GLM-4, Llama-3.1-8B, Qwen2-7B, Vicuna-13B
RACE (Ying et al.)EMNLP 2025GPT-4/4o, Gemini-1.5-Pro/2.0, o1, DeepSeek-R1, Qwen2-7B, Gemma-2-9B, GLM-4-9B
SeqAR (Yang et al.)NAACL 2025GPT-3.5/4/4o, Gemini-1.5-Pro, Llama-2-7B, Llama-3-8B
SEMA (Park et al.)ICLR 2026GPT-4o, GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-3B, gpt-oss-20B
Align to Misalign (Koo et al.)ICLR 2026GPT-4o, GPT-4o-mini, Claude-3.5-Haiku, Claude-Sonnet-4, Llama-3.1-8B
SlotGCGICLR 2026Llama-2-7B/13B, Llama-3.1-8B, Mistral-7B, Vicuna-7B, Qwen-2.5, GPT-3.5/4o, Gemini-2.0/2.5
MetisICML 2026GPT-3.5/4o, o1, GPT-5-chat, Gemini-2.5-Pro, Claude-3.7, Grok-3, Llama-3-8B/70B, Qwen2.5-7B
AE-CoTICML 2026o1-mini, o3-mini, GPT-5, DeepSeek-R1, Qwen3-235B, Gemini-2.5-Flash, Gemini-3-Pro
MJ Multi-turn (Park et al.)NeurIPS 2026GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B, Mistral-7B, gpt-oss-20B

III. Query budget and fairness

Some single-query baselines achieve high ASR on models already vulnerable to those attacks.

We compare attacks under the same query budget to measure the additional benefit of ĪCÁRŪS.

Table F. Binary ASR (%) across query budgets on Llama-3

MethodK=1K=1K=5K=5K=10K=10K=20K=20
GCG-T2327.83132
GCG-T + ĪCÁRŪS2643.95660
Poems4345.446.248
Poems + ĪCÁRŪS4673.78184
RefusalSupp3642.54547
RefusalSupp + ĪCÁRŪS3863.98085

At K=20K=20, ĪCÁRŪS improves ASR by 28 to 38 percentage points over the corresponding baselines.

[Q1] Definition of the query budget

Thank you for the comment. We count each target model generation as one query and evaluate performance using the first KK responses, up to K=20K=20.

For one-shot baselines, we reuse the original response and sample 19 additional responses with temperature 1.0 and top-p 0.9. For LRL, we evaluate four language translations and sample four additional responses per language.

For ĪCÁRŪS, we run beam stack search under the same query budget, counting each evaluated candidate as one query. We evaluate the best result obtained before reaching the query limit or the early stopping threshold of τ=0.8\tau=0.8 in StrongREJECT.

At each budget, we report best-of-K StrongREJECT and cumulative binary ASR, using a separate binary judge for ASR.

[A for Q1] We evaluate methods under the same query budget using additional sampling for baselines and recorded beam search evaluations for ĪCÁRŪS, with the sampling settings explicitly specified.


[Q2] Relationship between StrongREJECT and ASR

Thank you for the comment. StrongREJECT was introduced to address the limitations of binary ASR by measuring how informative and non-refusing a response is on a continuous scale from 0 to 1. In contrast, ASR measures the proportion of successful jailbreaks using a separate GPT-4o judge that produces binary decisions.

We use a StrongREJECT threshold of τ=0.8\tau=0.8 only for early stopping during ĪCÁRŪS search. For final evaluation, we report both metrics independently without thresholding StrongREJECT to calculate ASR.

[A for Q2] StrongREJECT provides a continuous measure that complements binary ASR. Its threshold is used only for search termination, while final evaluation reports both metrics independently.

Thank you to the authors for their response and the additional experiments.

Although these new experiments address my concerns, the fact that "simple character-level perturbations can achieve similar or even better results" suggests that the rules devised for ancient languages ​​lack sufficient distinctiveness, and the proposed method is somewhat limited in terms of overall innovation. Furthermore, the models used in the experiments are relatively dated; given that some of the related research cited by the authors already employs more recent models, the current experiments are insufficient to fully validate the effectiveness of the proposed method on more modern architectures. In light of these factors, I maintain my original rating.

Reviewer_D3AwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work provides a detailed investigation into how ancient languages can undermine the safety alignment of LLMs. Building on these findings, the authors propose ĪCÁRŪS, a jailbreak attack that searches and combines rules from different ancient languages. The proposed method can also be applied to enhance existing jailbreak attacks and the experimental results demonstrate that ĪCÁRŪS achieves promising performance.

Critical strengths

1.The paper provides a comprehensive investigation of the mechanisms underlying ancient-language jailbreaks, supported by extensive experiments. It presents the counterintuitive finding that the effectiveness of ancient-language jailbreaks is not primarily driven by ancient-language vocabulary.

2.The experimental evaluation is comprehensive, covering not only comparisons against existing jailbreak baselines but also evaluations under various defense mechanisms.

3.Some of the discovered rule combinations substantially improve the ASR of existing jailbreak methods like PAIR, Poems, suggesting the potential value of the proposed search space of jailbreak rules in ancitent-language.

Critical weaknesses

1.Despite the extensive analysis of the mechanisms underlying ancient-language jailbreaks, the search strategy employed by ĪCÁRŪS does not demonstrate clear methodological improvements over prior work, such as h4rm3l[1]. It would be better for the authors to clarify the novelty of their search strategy compared with existing approaches.

[1]. Doumbouya, Moussa Koulako Bala, et al. "h4rm3l: A language for composable jailbreak attack synthesis." International Conference on Learning Representations. Vol. 2025. 2025.

2.The authors have shown that token fragmentation is a generic factor to the jailbreak mechanism of ancient-language. In particular, the authors do not demonstrate whether the ancient-language grammar rule pool provides advantages over character and sentence-level transformation rules (such as base64 encoding and flipping the sentence) under the same search algorithm and query budget, or whether combining these transformations could further improve jailbreak performance.

3.The standalone ĪCÁRŪS method achieves relatively low ASR than other baselines. Furthermore, when applied to enhance existing jailbreak methods, it does not demonstrate substantial improvements over previous sota methods on 100 samples of JBB, except on Qwen3-8B.

Questions for authors

1.Do Table 3 and Figure 10 report ASR results evaluated on the same set of data? The results for GCG-T on Llama-3-8B appear inconsistent between the two. It would be better if the authors could clarify this discrepancy.

2.In Section 3.3, the authors define the expected score under independence as (1-(1-a)(1-b)), where a and b are StrongREJECT scores. It is a little confused. Could the authors explain the rationale for using this formula to measure rule interactions?

2 comments

We appreciate the thoughtful feedback on our paper, and we included additional comments to further improve our work.


[W1] Novelty Beyond Existing Search Strategies

Thank you for the comment on the novelty of our approach. We clarify that our contribution is the analysis of a linguistic vulnerability and an attack method built on these findings, rather than a new search algorithm.

While prior methods compose transformations or iteratively refine prompts, we investigate which linguistic properties enable jailbreaks and why. We separate vocabulary and grammar, identify orthographic token fragmentation as a key mechanism, and analyze interactions between rules.

These findings guide ĪCÁRŪS, which combines grammar rules while keeping the request and response in plain English.

We will clarify this distinction and discuss the relevant prior work in the revised manuscript.

[A for W1] Our contribution is identifying a linguistic vulnerability and using these findings to design a rule combination attack, rather than introducing a new black-box search algorithm.


[W2] Ancient Grammar and Generic Token Fragmentation

Thank you for the comment. We clarify that the jailbreak effect is not unique to ancient grammar. Our experiments show that generic character transformations can produce the same effect through token fragmentation.

We demonstrate this through matched token count and random fragmentation experiments. We will also compare generic transformations under the same beam search and query budget.

[A for W2] Ancient grammar provides a way to induce token fragmentation, but the effect is not specific to ancient languages, as shown by our controlled experiments.


Details

I. Matched token count control

We replace ancient marks with generic marks while keeping token counts matched, including a nonfragmenting ZWSP control.

Table A. Binary ASR (%) under matched token counts

ASR (%)NoneAncientDotCaretHyphenAsteriskAcuteRandomZWSP
Llama-3152436333331262117
Qwen367141113132159

Generic marks match or outperform ancient marks, while ZWSP shows little improvement.

II. Random token fragmentation

We vary the density of random combining marks on plain English prompts.

Table B. Token inflation, embedding shift, and ASR under random mark density

Densityρ\rhoδ\deltaASR
010.0015%
0.570.7746%
1130.8162%
2220.8461%
3310.8556%

ASR increases from 15% to 62% without ancient grammar, then declines at higher densities.


[W3] Standalone and Overlay Performance

Thank you for the comment. We clarify that ĪCÁRŪS can be used both as a standalone attack and as an overlay on existing jailbreak prompts.

The overlay improves the performance of all four baseline attacks across five models, achieving the highest ASR on four of the five models. Standalone ĪCÁRŪS also preserves English content and produces directly usable English responses.

[A for W3] Applying ĪCÁRŪS to existing jailbreak prompts consistently improves ASR across all five models, while the standalone attack preserves English input and output.


Details

Comparison with existing jailbreak prompts

We compare four existing jailbreak attacks before and after applying ĪCÁRŪS.

Table C. Binary ASR (%) across five target models

MethodLlama-3Qwen3Gemini 2.5GPT-3.5GPT-4o
GCG-T23910480
+ ĪCÁRŪS6258718015
PAIR2043488040
+ ĪCÁRŪS4354638249
Poems4337727247
+ ĪCÁRŪS8589899062
RefusalSupp3623428023
+ ĪCÁRŪS8571688649

Applying ĪCÁRŪS improves ASR for every baseline across all five models.

[Q1] Table 3 versus Figure 10

Thank you for pointing out this inconsistency. The original Figure 10 incorrectly computed binary ASR by applying a 0.5 threshold to StrongREJECT scores, whereas our ASR evaluation uses a separate judging prompt that produces binary decisions.

We have corrected Figure 10 using the proper binary ASR evaluation. The corrected figure preserves the overall trend, with ASR increasing as the query budget grows and ĪCÁRŪS overlays outperforming their corresponding baselines.

We also reviewed the other figures and tables for the same evaluation error and found no further instances.

[A for Q1] We corrected Figure 10 to use the proper binary ASR evaluation. The overall trend remains unchanged, and our review found no similar evaluation errors elsewhere in the manuscript.


Details

Table D. Representative binary ASR (%) across query budgets

MethodK=1K=1K=5K=5K=10K=10K=20K=20
GCG-T2327.83132
PAIR2028.03032
Poems4345.446.248
RefusalSupp3642.54547
GCG-T + ĪCÁRŪS2643.95660
Poems + ĪCÁRŪS4673.78184
RefusalSupp + ĪCÁRŪS3863.98085

The corrected results show that ĪCÁRŪS overlays maintain their advantage as the query budget increases.


[Q2] The independence baseline for rule interactions

Thank you for the comment. We treat StrongREJECT scores as proxies for success probabilities and use 1−(1−a)(1−b)1-(1-a)(1-b) as the expected combined effect when two rules act independently.

We compare this reference with the observed combined score to measure synergy or interference. Since StrongREJECT scores are not calibrated probabilities, this is an approximate reference.

[A for Q2] We use StrongREJECT scores as probability proxies to compare the observed combined effect with an independence-based reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.