- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a defense method against jailbreak attacks on large audio language models. It aims to defend against these attacks while avoiding refusals of normal requests as much as possible. Specifically, it proposes TRAIL, which tracks how refusal-related hidden states change across model layers, removes components shared with safe requests, and uses rotation instead of vector addition to modify the hidden states. Experiments further show that it improves jailbreak defense while largely preserving performance on normal tasks.
Critical strengths
-
The motivation is clear, and the presentation is easy to follow and understand.
-
The idea of studying how refusal behavior changes across layers is interesting. Using rotation instead of directly adding a refusal vector is also a reasonable design choice.
-
The experiments cover several jailbreak benchmarks and evaluate both defense performance and normal task performance.
Critical weaknesses
-
In terms of novelty, it may not be very different from using cross-layer trajectories in the intermediate layers of text LLMs to guide interventions (for example, Spherical Steering (ICML 2026) also uses rotation-based interventions). Both cross-layer refusal analysis and rotation-based steering have been studied before. The paper should better explain what is new beyond combining these ideas for LALMs. What is different for LALMs?
-
The method combines several changes, but their individual contributions are not clearly isolated. The paper does not directly compare cross-layer directions with additive steering, or single-layer directions with rotation. In other words, the ablation study is insufficient. It mainly examines hyperparameters but does not isolate the contributions of cross-layer trajectory modeling and rotation steering.
-
The evaluation is limited to 2 LALMs. Can this demonstrate generality? It also lacks comparisons with output safety guards (e.g. Azure Content Filter or Llama Guard), which are a practical alternative for filtering harmful responses.
-
A minor issue, Figure 3 is too far from where it is cited. Figure 5 does not need to be here at all. This part (case study) takes up a lot of space, but does not seem important and could be moved to the appendix.
Questions for authors
-
What is new for LALMs compared with existing cross-layer refusal analysis and rotation-based steering in LLMs?
-
Which component mainly contributes to the improvement? Why are cross-layer modeling and rotation both necessary?
-
How well does TRAIL generalize beyond the two evaluated LALMs? Maybe also compare output filter methods.
3 comments
Response to Reviewer QwuK [2/2]
Q3: Cross-Model Generalization.
A3: We added two evaluation models, Qwen2.5-Omni and Llama-Omni, to verify the cross-model generalization of the defense method, as follows:
Tab. R4: ASR of TRAIL on Additional Models.
| Model | Method | FigStep-Audio | AdvBench-Audio | HarmBench |
|---|---|---|---|---|
| Qwen2.5-Omni | w/o TRAIL | 40.8 | 3.8 | 17.5 |
| w/ TRAIL | 32.4 | 0.9 | 12.6 | |
| Llama-Omni | w/o TRAIL | 73.6 | 32.6 | 35.9 |
| w/ TRAIL | 54.4 | 15.9 | 26.5 |
We did not make any setting adjustments for the newly added models; we directly applied the original settings to Qwen2.5-Omni and Llama-Omni. As can be seen, our method also effectively improves the defense capability of the newly added models. These results are reported in Appendix D.2.
Q4: Explore issues about output safety guards.
A4: Thank you for the suggestion. Llama-Guard is a detection-based method, whereas TRAIL is an activation-intervention-based method. They are not directly comparable, but can be complementary when used in series. We adopt a pipeline in which activation steering is applied first, followed by Llama-Guard detection. As shown in Tab. R5, on FigStep-Audio, with Llama-Guard alone, 22.0% of the outputs it judged safe were still judged unsafe by SorryBench. After adding activation steering, this ratio drops to 2.8%. This indicates that combining the two can improve defense effectiveness. This is an interesting finding, and we thank you for the idea. These results are presented in Appendix D.4.
Tab. R5: ASR (%) of Llama-Guard-3-8B with and without TRAIL on FigStep-Audio (lower is better).
| Model | Method | FigStep-Audio |
|---|---|---|
| Qwen2-Audio | Only Llama-Guard-3-8B | 22.0 |
| TRAIL+Llama-Guard-3-8B | 2.8 | |
| Kimi-Audio | Only Llama-Guard-3-8B | 10.8 |
| TRAIL+Llama-Guard-3-8B | 3.2 |
Q5: Suggestions on manuscript organization.
A5: Thank you for your helpful suggestions regarding the manuscript organization. Following your suggestions, we have added an appropriate reference to Figure 3 at a more relevant location, moved the Case Study section to Appendix E, and introduced a new Inference Efficiency Analysis subsection in the main text to replace the space previously occupied by the case study.
Official Comment By Authors
Dear reviewer QwuK ,
Thank you for your valuable comments and constructive suggestions.
As the discussion deadline is approaching, we would appreciate your participation in the discussion so that we have sufficient time to address any further questions or concerns you may have.
Thank you for your time and consideration, and we look forward to hearing from you.
Best Regards,
Authors of Submission 35924
Response to Reviewer QwuK [1/2]
We highly appreciate your positive feedback and valuable suggestions about our paper. We respond to your questions as follows.
Q1: Comparison with Spherical Steering.
A1: Thank you for raising this important concern. We would like to emphasize that, compared with Spherical Steering in LLMs (ICML 2026) [1], our method establishes its novelty at three levels:
(1) Cross-layer Refusal Trajectory Modeling.
Spherical Steering constructs a contrastive prototype direction μ^((l)) at a given layer and rotates activations toward this prototype. Its steering target is a static, layer-wise contrastive direction. In TRAIL, however, the steering target is not a static prototype but a cross-layer refusal trajectory, which characterizes how the refusal representation evolves with layer depth, thereby yielding stronger defense performance. Further details are provided in Challenge I of Section 3.2 in the original manuscript (Page 4, Lines 199–207).
(2) Two-dimensional Subspace-constrained Rotation.
Spherical Steering performs norm-preserving rotation on the full d-dimensional hypersphere, rotating the entire activation direction toward the target prototype. TRAIL, in contrast, projects the activation onto a local two-dimensional rotation basis. The rotation is performed only within this plane, while components outside the plane remain unchanged, thereby minimizing the damage of activation steering to the original semantics.
(3) How is this different for ALLMs?
TRAIL is specifically designed for this setting. Refusal behavior is entangled with cross-modal audio representations, audio jailbreak attacks bring new attack surfaces, and benign audio queries are particularly prone to over-refusal. TRAIL is designed precisely for this setting: it extracts refusal trajectories from audio-language hidden states, decouples the benign over-refusal component, and performs rotation within the trajectory-derived subspace. Therefore, TRAIL is more than a generic rotation primitive; it also addresses the ALLM-specific cross-modal refusal dynamics and the robustness–over-refusal trade-off. We have provided detailed justification for this in Challenge II of Section 3.2 in the original manuscript (Page 5, Lines 223–230), and we will further clarify these differences to make the distinction clearer. In addition, we have added a comparison with Spherical Steering (ICML 2026) [1] adapted to the ALLM setting, as shown below:
Tab. R1: Comparison of ASR (%) between TRAIL and Spherical Steering
As shown in the Tab. R1, TRAIL achieves lower ASR on harmful prompts for both models.
[1] Spherical Steering: Geometry-Aware Activation Rotation for Language Models. ICML 2026
Q2: Core Components and Their Complementary Roles.
A2: Thank you for your insightful comments. We have added ablation experiments on the two core components of our method: cross-layer refusal trajectory modeling and rotation steering, as shown in Tab. R2.
Tab. R2: Ablation Study of TRAIL's Core Components on the FigStep-Audio Dataset.
We have presented the ASR results of ablating the two core components on the FigStep-Audio Dataset. The experiments show that both components contribute to improving defense performance. We will revise the final version to ensure that the ablation is sufficient.
In addition, we have supplemented a comparison of TRAIL with additive steering (i.e., removing the rotation steering) to evaluate their impact on the models' general audio capabilities, as shown in Tab. R3. The data are from AirBench, using its official evaluation settings:
Tab. R3: Comparison of TRAIL and Additive Steering on AirBench.
As shown in the Tab. R3, additive steering degrades model capabilities, supporting our motivation in Challenge II of Section 3.2 regarding its potential disruption of semantic structure.