- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the vulnerability of long-context LLMs to many-shot jailbreak attacks from a positional-encoding perspective. The authors find that increasing the RoPE base reduces attack success but also degrades long-context utility. Motivated by this trade-off, they propose Multi-Frequency Alignment, which samples RoPE bases during GRPO training and assigns greater weight to longer prompts. Experiments on Llama-3.1 and Qwen2.5 models show that MFA substantially reduces attack success rates while largely preserving general and long-context capabilities.
Critical strengths
-
Important problem: The paper addresses the timely and practically important problem of safety degradation in long-context LLMs, particularly under many-shot jailbreak attacks.
-
Strong empirical results: MFA substantially reduces attack success rates across multiple model scales, attack types, and context lengths, while largely preserving general and long-context capabilities.
-
Practical method and useful ablations: MFA only modifies training and retains the model’s native RoPE configuration at inference, introducing no additional deployment overhead. The matched fixed-base GRPO baseline and component ablations help isolate the gains from the proposed design.
Critical weaknesses
-
The interpretation of the inference time RoPE-base experiment is ambiguous. The lower ASR does not necessarily indicate that the model’s safety behavior become more robust. An alternative explanation is that using a non-native RoPE base generally disrupts the model’s ability to utilize long contexts and perform in-context learning. Under this interpretation, the model may become less capable of following the many-shot demonstrations, causing the attack to fail as a side effect of impaired long-context functionality. It is therefore unclear whether this experiment reveals a safety-specific positional phenomenon or a more general degradation of long-context processing.
-
The paper does not adequately rule out over-refusal. Since MFA is trained with binary safety rewards and refusal-oriented targets, the lower ASR may partly reflect a generally stronger tendency to refuse. General and long-context benchmarks do not directly measure this behavior.
-
The core idea of training across varied positional configurations is closely related to Randomized Positional Encodings, PoSE, RFS, and particularly HARPE, which already uses multiple RoPE bases during training. The paper should compare against these approaches and clarify whether MFA offers benefits beyond applying existing positional augmentation to safety alignment.
Questions for authors
The experiments focus on Llama-3.1 and Qwen2.5. Can the authors report results on at least one more recent model family with an updated long-context training and alignment recipe? Since the central claim depends on the model’s RoPE configuration and positional sensitivity, evidence that the observed trade-off and MFA’s gains persist on newer models would materially affect my assessment of the method’s generality.
6 comments
Author Response to Reviewer 2YYo (2/3)
We also test the same target requests after benign long-context prefixes:
| Backbone | Method | Input length (tokens) | Benign FRR (%) ↓ | Benign valid assistance (%) ↑ | Benign invalid output (%) ↓ | Unsafe coherent refusal (%) ↑ |
|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | Original | 22.6k–24.3k | 4.4 | 92.0 | 2.8 | 84.5 |
| Llama-3.1-8B-Instruct | Fixed-base GRPO | 22.6k–24.3k | 7.2 | 88.4 | 3.6 | 91.0 |
| Llama-3.1-8B-Instruct | MFA | 22.6k–24.3k | 6.0 | 90.8 | 2.4 | 95.5 |
| Qwen2.5-7B-Instruct | Original | 22.0k–23.7k | 2.8 | 94.4 | 2.0 | 80.5 |
| Qwen2.5-7B-Instruct | Fixed-base GRPO | 22.0k–23.7k | 4.8 | 92.0 | 2.4 | 89.5 |
| Qwen2.5-7B-Instruct | MFA | 22.0k–23.7k | 4.0 | 93.6 | 1.6 | 94.0 |
We count unnecessary full or partial refusal on benign requests as false refusal; a coherent refusal on unsafe requests must be relevant and withhold harmful assistance. Invalid responses are counted separately, not as successful refusals. MFA's false-refusal rates are above the original models' but below matched fixed-base GRPO on both backbones. At long input lengths, its benign valid-assistance rates are 90.8% and 93.6%, while unsafe coherent-refusal rates rise to 95.5% and 94.0%. The gain in safety therefore has a measurable, but bounded, refusal-calibration cost in these evaluations; we will report the rubric, denominators, and audit details in the appendix.
W3: How does MFA differ from prior positional augmentation?
Thank you for pointing us to these papers. We are sorry for the confusion about how MFA relates to prior positional augmentation. We discuss all four works in Appendix A. For length generalization, Ruoss et al. [3] sample an ordered subset of integer positions, PoSE [4] skips positions between chunks, and Random Float Sampling [5] draws positions from a continuous interval. HARPE [6] assigns different RoPE bases to different attention heads during continued pretraining for context extension.
MFA uses base variation during safety training. All responses in a GRPO group share one sampled base, so their rewards are compared under the same positional configuration. We apply length weighting after group-relative advantage normalization to preserve its effect on the update. Both the frozen reference policy and the deployed model use the native base. The ablation below varies base sampling and length weighting under the same safety-training budget.
| Safety-training variant | 256-shot MSJ ASR (%) ↓ | LOONG ↑ |
|---|---|---|
| Fixed-base GRPO | 28.0 ± 2.6 | 27.6 ± 0.4 |
| Base sampling only | 17.5 ± 2.2 | 28.5 ± 0.4 |
| Length weighting only | 15.0 ± 1.9 | 27.9 ± 0.4 |
| MFA | 8.8 ± 1.7 | 28.9 ± 0.3 |
These results quantify the gain over fixed-base GRPO. A direct safety comparison with HARPE [6] would require training its head-wise base assignment on the same prompts, reward, and update budget, with the inference configuration specified for each method.
Author Response to Reviewer 2YYo (3/3)
Q1: Does MFA extend to a newer model family?
Thank you for asking how MFA transfers to models with newer alignment recipes. Our original evaluation covered Llama and Qwen, with a comparison across model sizes within Qwen. To test transfer across families, we evaluated the suggested Mistral-7B-Instruct-v0.3 and GLM-4-9B-Chat models at 128 shots.
| Model | Setting (128-shot) | MSJ ASR (%) ↓ | Benign many-shot accuracy (%) ↑ | Long-context utility ↑ |
|---|---|---|---|---|
| Mistral-7B-Instruct-v0.3 | Original, native base | 82.5 | 71.4 | 24.6 |
| Mistral-7B-Instruct-v0.3 | Original, scaled base () | 61.0 | 64.8 | 20.3 |
| Mistral-7B-Instruct-v0.3 | Fixed-base GRPO, native base | 36.5 | 70.6 | 24.1 |
| Mistral-7B-Instruct-v0.3 | MFA, native base | 14.0 | 71.2 | 24.8 |
| GLM-4-9B-Chat | Original, native base | 47.5 | 78.2 | 30.4 |
| GLM-4-9B-Chat | Original, scaled base () | 31.0 | 73.5 | 26.1 |
| GLM-4-9B-Chat | Fixed-base GRPO, native base | 22.5 | 77.6 | 29.8 |
| GLM-4-9B-Chat | MFA, native base | 9.0 | 78.0 | 30.7 |
These additions test transfer across more RoPE model families, using a common shot count that fits both configured windows; MFA is evaluated at each model's native inference setting. They do not by themselves answer the newer alignment recipe part of your question. We will distinguish family coverage from recency in the manuscript's generality claim rather than present these models as evidence for an untested training recipe.
The updated PDF will include the over-refusal tests, the Mistral and GLM evaluations, and the expanded Related Work discussion. Thank you again for suggesting the benign-prompt evaluation and for helping us place MFA in the context of earlier work.
References
[1] Röttger et al. (NAACL 2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models.
[2] Cui et al. (ICML 2025). OR-Bench: An Over-Refusal Benchmark for Large Language Models.
[3] Ruoss et al. (ACL 2023). Randomized Positional Encodings Boost Length Generalization of Transformers.
[4] Zhu et al. (ICLR 2024). PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training.
[5] Shimizu et al. (Findings of EACL 2026). Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers.
[6] Lian et al. (COLING 2025). Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models.
Author Response to Reviewer 2YYo (1/3)
Thank you for reading the component ablations closely and for your specific suggestions on how to evaluate MFA.
Thank you for raising this alternative explanation. If a non-native base makes it harder for the model to use the demonstrations, that could account for part of the drop in ASR. Table 8 (Appendix B.9) measures ASR on the same harmful final queries with no demonstrations, 32 benign GSM8K QA demonstrations, or 32 harmful PANDAS demonstrations. We report benign-task accuracy separately below.
Even a benign QA prefix changes ASR on the harmful query, and the PANDAS condition has higher ASR at every base. At 5M, lower PANDAS ASR is accompanied by lower LOONG. The sweep therefore identifies sensitivity to the RoPE-base intervention, not a uniquely safety-specific mechanism. MFA, in contrast, is evaluated at the native inference base. In the separate 128-shot benign many-shot task, MFA retains 71.2% accuracy versus 71.4% for the original Mistral model, and 78.0% versus 78.2% for the original GLM model; the matched fixed-base GRPO controls reach 70.6% and 77.6%. We will add corresponding matched-length benign many-shot results for the main Llama and Qwen models.
We appreciate your suggestion to test over-refusal directly, especially because MFA uses a binary safety reward. During rebuttal, we measured false refusals at each model’s native RoPE base on benign prompts from XSTest [1] and the Hard-1K subset of OR-Bench [2]. We also scored coherent refusals on the unsafe XSTest prompts.