- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper finds that ancient-language grammar rules, particularly orthographic rules, can increase jailbreak success rates by introducing token fragmentation and shifting input embeddings. Based on this observation, the authors propose ĪCÁRŪS, which employs Beam Stack Search to identify effective combinations of grammar rules. Experiments on five open-source and closed-source LLMs demonstrate relatively high attack success rates.
Critical strengths
-
Interesting research perspective. The paper investigates LLM jailbreaks from the perspective of ancient-language grammar and analyzes the different effects of vocabulary and grammatical transformations.
-
Systematic experimental analysis. The authors conduct ablation studies, token fragmentation analysis, and rule combination experiments, providing substantial empirical evidence.
-
Practical applicability. The proposed method requires neither model parameters nor gradient access, and successful attacks produce directly usable English responses.
Critical weaknesses
-
Insufficient mechanistic understanding. The paper primarily relies on correlation analyses to establish relationships between token fragmentation, embedding shifts, and jailbreak success. However, it does not adequately explain why these transformations weaken safety alignment while preserving the model's ability to understand harmful requests. The analysis remains largely at the input representation level.
-
Limited methodological novelty. The use of special-character perturbations resembles existing Unicode-based obfuscation attacks. Moreover, Beam Stack Search essentially performs black-box optimization over combinations of predefined rules, without demonstrating clear technical distinctions from existing iterative jailbreak optimization methods.
-
Insufficient evaluation on recent models. The commercial models evaluated mainly include GPT-3.5-turbo, GPT-4o, and Gemini 2.5 Flash. The absence of more recent commercial LLMs makes it difficult to assess the effectiveness of the proposed attack against more advanced safety alignment mechanisms.
-
Limited significance of the empirical improvements. Numerous existing jailbreak methods have already achieved high attack success rates, making further ASR improvements alone less compelling. In particular, the standalone ĪCÁRŪS method does not consistently outperform existing baselines, while its strongest results rely on combinations with other jailbreak techniques.
Questions for authors
-
Can the authors provide a deeper mechanistic explanation of why token fragmentation weakens safety judgments without substantially impairing semantic understanding? Could analyses of internal representations or safety-related activations help validate this hypothesis?
-
What are the key technical innovations of ĪCÁRŪS compared with existing special-character-based attacks and iterative black-box jailbreak optimization methods?
-
Can the authors extend the evaluation to more recent commercial LLMs and further distinguish the performance gains achieved by standalone ĪCÁRŪS from those obtained through combinations with existing jailbreak methods?
3 comments
[W3, Q3.1] Evaluation on Recent Models
Thank you for the comment. We selected five target models (Llama-3, Qwen3, Gemini 2.5, GPT-3.5, GPT-4o) to facilitate comparisons with prior jailbreak studies and provide a reference for future evaluations.
Recent studies continue to evaluate the same model families, including GPT, Gemini, Llama, and Qwen, as shown below.
[A for W3, Q3.1] Our model selection enables direct comparison with prior jailbreak studies and provides a consistent reference for future research.
Details
I. Model selection in prior jailbreak studies
Table F. Representative jailbreak studies and target models
| Jailbreak paper | Venue | Target models |
|---|---|---|
| CipherChat (Yuan et al.) | ICLR 2024 | GPT-3.5-turbo, GPT-4 |
| AutoDAN (Liu et al.) | ICLR 2024 | Vicuna, Llama-2, GPT-3.5/4 |
| ArtPrompt (Jiang et al.) | ACL 2024 | GPT-3.5/4, Gemini, Claude, Llama-2 |
| PAIR (Chao et al.) | SaTML 2025 | GPT-3.5/4, Vicuna, Llama-2, Claude, Gemini |
| FlipAttack (Liu et al.) | ICML 2025 | GPT-3.5/4/4o, Claude-3.5, Gemini, Llama-3, Mixtral |
| Simple Adaptive Attacks (Andriushchenko et al.) | ICLR 2025 | GPT-3.5/4o, Claude-3/3.5, Llama-2/3, Gemma-7B, Vicuna-13B, Mistral-7B |
| Adversarial Reasoning (Sabbaghi et al.) | ICML 2025 | Llama-2-7B, Llama-3-8B, Llama-3-8B-RR, R2D2 |
| Best-of-N Jailbreaking (Hughes et al.) | NeurIPS 2025 | GPT-4o/4o-mini, Claude-3.5-Sonnet, Claude-3-Opus, Gemini-1.5-Flash/Pro, Llama-3-8B |
| One Step Is Enough / REDA (Jiang et al.) | ACL 2025 | GPT-3.5, GLM-4, Llama-3.1-8B, Qwen2-7B, Vicuna-13B |
| RACE (Ying et al.) | EMNLP 2025 | GPT-4/4o, Gemini-1.5-Pro/2.0, o1, DeepSeek-R1, Qwen2-7B, Gemma-2-9B, GLM-4-9B |
| SeqAR (Yang et al.) | NAACL 2025 | GPT-3.5/4/4o, Gemini-1.5-Pro, Llama-2-7B, Llama-3-8B |
| SEMA | ICLR 2026 | GPT-4o, GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-3B, gpt-oss-20B |
| Align to Misalign (Koo et al.) | ICLR 2026 | GPT-4o, GPT-4o-mini, Claude-3.5-Haiku, Claude-Sonnet-4, Llama-3.1-8B |
| SlotGCG | ICLR 2026 | Llama-2-7B/13B, Llama-3.1-8B, Mistral-7B, Vicuna-7B, Qwen-2.5, GPT-3.5/4o, Gemini-2.0/2.5 |
| Metis | ICML 2026 | GPT-3.5/4o, o1, GPT-5-chat, Gemini-2.5-Pro, Claude-3.7, Grok-3, Llama-3-8B/70B, Qwen2.5-7B |
| AE-CoT | ICML 2026 | o1-mini, o3-mini, GPT-5, DeepSeek-R1, Qwen3-235B, Gemini-2.5-Flash, Gemini-3-Pro |
| MJ Multi-turn (Park et al.) | NeurIPS 2026 | GPT-4.1-mini, Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B, Mistral-7B, gpt-oss-20B |
These studies show that the model families used in our evaluation remain common choices in jailbreak research, supporting comparisons with both existing and future work.
[W4, Q3.2] Significance of gains and standalone versus overlay
Thank you for the comments. Our contribution goes beyond ASR improvements by identifying generic token fragmentation as the mechanism behind ancient-language jailbreaks.
We report standalone and overlay results separately in Table 3, where the overlays achieve the highest ASR on four of five models. We further evaluate their improvements under the same query budget.
[A for W4, Q3.2] We clarify the distinct roles of standalone and overlay configurations and show that ĪCÁRŪS improves existing attacks under matched query budgets.
Details
We compare ĪCÁRŪS with the unmodified prompt and existing attacks under the same query budget on Llama-3.
Table G. Binary ASR (%) across query budgets on Llama-3
| Method | ||||
|---|---|---|---|---|
| Vanilla | 0 | 0 | 0 | 0 |
| ĪCÁRŪS | 0 | 0 | 29 | 48 |
| GCG-T | 23 | 27.8 | 31 | 32 |
| GCG-T + ĪCÁRŪS | 26 | 43.9 | 56 | 60 |
| Poems | 43 | 45.4 | 46.2 | 48 |
| Poems + ĪCÁRŪS | 46 | 73.7 | 81 | 84 |
| RefusalSupp | 36 | 42.5 | 45 | 47 |
| RefusalSupp + ĪCÁRŪS | 38 | 63.9 | 80 | 85 |
Vanilla represents the unmodified prompt evaluated without search.
At , standalone ĪCÁRŪS achieves 48% ASR, while its overlays improve existing attacks by 28 to 38 percentage points.
I appreciate the authors' detailed rebuttal and additional experiments. In particular, the controlled experiments on token fragmentation and comprehension analysis provide stronger empirical evidence for the observed jailbreak behavior. The query-budget-matched comparisons also address my concerns regarding evaluation fairness and demonstrate meaningful improvements when ĪCÁRŪS is combined with existing jailbreak methods.
However, my main concerns remain unresolved:
- Insufficient mechanistic understanding. While the new experiments provide stronger evidence that token fragmentation contributes to jailbreak effectiveness, they do not fully explain why such perturbations weaken safety alignment while preserving semantic understanding. The analysis still lacks direct evidence regarding internal safety-related mechanisms.
- Limited methodological novelty. The authors acknowledge that the search algorithm itself is not novel. More importantly, the new experiments show that generic character perturbations can match or even outperform ancient-language transformations, further raising questions about the distinctive contribution of ancient grammar. A fair comparison using the same search algorithm and query budget remains necessary.
- Limited evaluation on recent models. The authors justify their model selection by referencing prior studies but do not provide additional experiments on more recent commercial LLMs. Therefore, the generalizability of the findings to modern safety-aligned models remains unclear.
Overall, the rebuttal strengthens the empirical evaluation but does not sufficiently resolve my fundamental concerns regarding mechanistic insight and technical novelty. I therefore decide to maintain my original rating.
We appreciate the thoughtful feedback on our paper, and we included additional comments to further improve our work.
Thank you for the comment. We further investigated why token fragmentation weakens safety alignment through controlled interventions and additional comprehension analysis.
Our interventions identify generic token fragmentation as a key mechanism behind ancient-language jailbreaks. We also find that moderate transformations increase ASR, while excessive transformations disrupt comprehension and reduce attack success.
We will clarify these findings in Section 3 and distinguish our behavioral evidence from direct analysis of internal safety activations.
Details
I. Controlled interventions
We test token fragmentation by replacing ancient marks at matched token counts, varying random mark density, and removing combining marks.
Table A. Binary ASR (%) under matched token counts
Generic marks match or outperform ancient marks, while ZWSP shows little improvement.
Table B. Token inflation, embedding shift, and ASR under random mark density
ASR increases from 15% to 62%, then declines at higher fragmentation levels.
Table C. Effect of removing combining marks on Llama-3
Removing combining marks reduces StrongREJECT from 0.83 to 0.13, further supporting the role of orthographic perturbations.
II. Comprehension analysis
We additionally analyzed vocabulary and grammar interactions and classify responses as COMPLY (C), REFUSE (R), or INCOHERENT (I) using GPT-4o.
Table D. Vocabulary and grammar interactions on Llama-3
Arabic and Sanskrit show strong negative interactions, with StrongREJECT dropping close to zero when both transformations are applied.
Table E. Response classification (%) under vocabulary and grammar transformations
Under combined transformations, 96% of Arabic and 95% of Sanskrit responses become incoherent, with only 2% and 4% classified as refusals. In contrast, Chinese responses remain largely coherent and predominantly refuse.
These results suggest that moderate transformations can increase ASR while preserving comprehension, whereas excessive transformations disrupt understanding and reduce attack success rather than restore safety alignment.
Thank you for the comments on methodological novelty. We clarify that our contribution is the analysis of a linguistic vulnerability and an attack method built on these findings, rather than a new search algorithm.
While prior methods compose transformations or iteratively refine prompts, we investigate which linguistic properties enable jailbreaks and why. We separate vocabulary and grammar, identify orthographic token fragmentation as a key mechanism, and analyze interactions between rules.
These findings guide ĪCÁRŪS, which combines grammar rules while keeping the request and response in plain English.