- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how narrative wrapping affects LLM refusal across English, modern Chinese, and Classical Chinese. It first introduces GUISE, a benchmark containing harmful requests, benign controls, and held-out wrapper types, with an evaluator that counts warn-then-answer responses as attack successes. Its representation analysis associates wrapping with a shift away from a reference refusal direction. The proposed defense, AXIS, combines DPO, a representation-alignment objective, and a weighted response-supervision objective. Experiments on three models report improvements in the combined safety–usability score.
Critical strengths
-
The paper addresses a relevant safety problem and attempts to distinguish the effects of narrative wrapping from those of language and register.
-
The paper is generally easy to read.
-
The paper makes two contributions: a benchmark for studying this problem and a defence method.
Critical weaknesses
-
The presentation could be improved. Although the paper is generally understandable, the introduction contains many experimental details and numerical results that would fit better in the methods and results sections. These details can distract readers from the main motivation and ideas.
-
I am not convinced that the results sufficiently support the choice of the reference refusal direction. Section 4 describes the representation analysis as correlational. However, Table 3 shows that random and orthogonal anchors also achieve strong safety performance, particularly on the larger models. For example, on GLM-4-9B, the orthogonal anchor achieves 99.3% OOD safety versus 99.8% for the real anchor, while reducing XSTest over-refusal from 14.7% to 4.1%. It is therefore unclear how much of the improvement actually depends on aligning with the original refusal direction.
-
Another concern is that the setting is quite narrow. The method relies on aligning directions estimated from Classical and modern Chinese requests. Although the paper reports transfer to English, it is unclear how the approach would be applied more broadly to other languages and registers. A clearer discussion of how to use the method outside this particular setting would help establish its broader value.
Questions for authors
-
How do you interpret the strong performance of the random and orthogonal anchors in Table 3? What evidence shows that aligning with the original refusal direction is particularly beneficial?
-
How would AXIS be applied to other languages that do not have a clear classical–modern register distinction? How would you choose the reference direction in those settings?
Thank you for highlighting the anchor controls and the applicability to other languages. We have added an intervention-based comparison of anchor choices, clarified the role of refusal-direction overlap, and made the cross-language application procedure explicit.
W1. Introduction contains too many numerical details
Clarification: The introduction highlights 20.4% attack success for bare Classical Chinese requests versus 95.7% for wrapped requests (Table 1). These aggregate rates use 334 and 1,067 intents, respectively; the caption now explicitly distinguishes their denominators. Table 1 retains the cross-register measurements, while the detailed defense comparisons are in Section 7.
W2 / Q1. Why use the original refusal direction instead of random or orthogonal anchors?
Key finding: Where the 1.7B model has room to improve, overlap with the refusal direction predicts held-out-wrapper generalization and a better safety–usability balance. The random anchor is a partial-overlap control: it preserves an initial overlap c₀=0.785 (Sec. 7.2) and therefore has an overlap of approximately c₀²≈0.62 with the reference refusal direction. The orthogonal anchor has zero overlap.
Existing anchor ablation, Qwen3-1.7B (ood834 refusal; Table 3, mean of 3 seeds):
Source: Table 3, ood834 column of the 1.7B block (rows "AXIS (real anchor)", "AXIS, random anchor", "AXIS, orthogonal anchor"); overlaps from the definitions in Sec. 7.2.
The ordering holds across seeds, and all nine paired contrasts are significant (Sec. 7.2).
New intervention: rotation-only training, with no refusal demonstrations (new; Sec. 7.2, paragraph "Rotation without refusal demonstrations").
Source (new): Rotation-only anchor comparison reported in this response; the real-anchor result is also described in Sec. 7.2. Bold: best per column.
Compared with the partial-overlap anchor, the real anchor improves harm334 refusal by 15.0 points (Holm-adjusted p=3e-6) and ood834 refusal by 3.8 points (p=0.02), with 1.4 and 1.2 points more over-refusal on plain benign requests and XSTest, respectively (the rotation-only results in this response). Although the orthogonal anchor yields higher attack refusal, it also refuses many more benign requests: 46.7% versus 8.7% on plain benign requests and 51.4% versus 24.9% on XSTest.
At larger scales, anchor variants already achieve high ood834 refusal (≥97%; Table 3, 4B and 9B blocks), making safety rates less informative for distinguishing directions. The revised claim therefore focuses on the setting where direction choice produces measurable differences in generalization. We report the complete larger-model anchor results in Table 3.
W3 / Q2. Applying AXIS to languages without a classical–modern distinction
Key finding: AXIS requires matched harmful/benign contrasts across two input conditions—not a classical–modern language pair. The objective in Section 5 can be applied using the following recipe:
Existing and new evidence supports the feasibility of this setup:
Thus, the method generalizes conceptually to any setting with a reliable reference condition and a matched target condition. The procedure above explains how to adapt the objective to other languages; those adaptations still require evaluation.