- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TriSteer, a training-free adaptive activation steering framework for improving the safety of Large Audio Language Models (LALMs). The authors investigate the representation geometry of audio-conditioned hidden states and construct three measurement subspaces: Semantic Space, Decision-Relevant Acoustic Space, and Decision-Irrelevant Acoustic Space.
Based on these subspaces, TriSteer adaptively adjusts both the magnitude and direction of activation steering. The magnitude controller estimates input-specific risk relative to benign reference boundaries, while the direction controller selectively reduces decision-irrelevant acoustic components from an audio-native safety direction.
Experiments on three LALMs and multiple audio safety benchmarks demonstrate substantial reductions in attack success rates while largely preserving benign-task performance. The paper also provides representation analyses and component ablations to investigate the proposed design.
Critical strengths
Important research problem. The paper addresses the safety challenges of large audio language models, particularly the need to distinguish harmful semantic content from acoustic variations that may affect safety decisions.
Interesting audio-specific design. The proposed three-space measurement framework provides an interesting perspective on separating semantic, decision-relevant acoustic, and decision-irrelevant acoustic variations. Combining these measurements with adaptive magnitude and direction control is intuitive and potentially useful.
Strong empirical results. The paper reports substantial reductions in attack success rates across multiple audio safety benchmarks and three LALM backbones. The authors also conduct representation analyses, component ablations, and evaluation with an alternative safety judge, providing useful evidence of the method's effectiveness.
Critical weaknesses
W1. The proposed three-space measurement framework is an interesting contribution, but its incremental effectiveness is not consistently demonstrated. The paper introduces a three-space representation framework that distinguishes semantic, decision-relevant acoustic, and decision-irrelevant acoustic variations. This is arguably the most distinctive methodological contribution of TriSteer compared with existing adaptive activation steering approaches. However, the ablation studies indicate that the primary safety–utility improvements arise from adaptive magnitude control, while the additional benefits of the proposed tri-space-based direction refinement are relatively limited and inconsistent across different models and attack settings. For example, the component ablations show that adaptive direction refinement sometimes provides additional safety improvements, but can also produce comparable or slightly worse results than fixed-direction adaptive-magnitude steering. The supplementary comparisons with Full-only and Full Irr removal further suggest that simpler variants can achieve similar or better performance in certain settings. Although these results support the potential usefulness of the proposed framework, they do not sufficiently establish that the more complex three-space design provides substantial and consistent advantages over simpler adaptive steering mechanisms. W2. The generalizability of the proposed three-space decomposition remains insufficiently validated. The Semantic, Decision-Relevant Acoustic, and Decision-Irrelevant Acoustic spaces are constructed from predefined contrast families through SVD-based representation analysis. The authors demonstrate that the corresponding subspaces capture distinguishable representation variations on held-out contrast pairs. However, these held-out samples are drawn from the same predefined contrast families used for subspace construction. More importantly, the distinction between decision-relevant and decision-irrelevant acoustic information can be context-dependent. An acoustic variation considered irrelevant for one task may carry important safety-related information in another. Therefore, it remains unclear whether the learned subspaces capture generalizable safety-relevant acoustic factors or primarily reflect the specific contrast construction procedures. Evaluation under substantially different acoustic conditions and more diverse audio tasks would strengthen the proposed representation-level contribution. W3. The evaluation of benign utility and cross-model generalization is relatively limited. Although the paper reports strong safety improvements, benign utility is mainly evaluated using the OpenBookQA subset of VoiceBench. This task does not fully represent the diverse capabilities of audio language models, including speech recognition, speech translation, emotion understanding, speaker-related reasoning, and long-form spoken instruction following. Since TriSteer modifies hidden representations across multiple language layers, it is important to investigate whether the intervention affects these capabilities. Furthermore, the evaluation covers three backbones from the Qwen model family. Additional evaluation on different audio model architectures and broader benign audio tasks would provide stronger evidence of generalization. W4. The stability and robustness of the proposed adaptive steering mechanism require further investigation. The proposed controller relies on relatively small sets of harmful and benign examples to construct steering directions and estimate reference boundaries. In particular, the audio-native behavior anchor is constructed from a small number of safe-refusal and unsafe-response examples. Such a construction may be sensitive to the particular samples selected. Moreover, the parameter sensitivity experiments show that changing the steering scale or calibration quantile can substantially affect attack success rates in some settings. Although the paper provides useful sensitivity analyses, it does not sufficiently establish the stability of the proposed framework across independently sampled construction sets or substantially different attack distributions. A more systematic evaluation of calibration robustness would help assess the reliability of TriSteer for practical deployment.
Questions for authors
Incremental contribution of the three-space design: The ablation results suggest that adaptive magnitude control accounts for most of the safety–utility improvements, while direction refinement provides relatively limited and sometimes inconsistent gains. What are the specific advantages of the proposed three-space framework over simpler adaptive-magnitude steering methods? Can the authors identify the conditions under which the three-space design consistently improves performance?
Necessity of the three subspaces: Are all three measurement spaces necessary for effective safety steering? Can the authors provide more fine-grained ablations of the Semantic, Decision-Relevant Acoustic, and Decision-Irrelevant Acoustic spaces to clarify their individual contributions?
Generalization of acoustic representations: How well do the learned subspaces generalize to acoustic variations outside the predefined contrast families? Can the authors evaluate the method on substantially different speaking styles, acoustic environments, or tasks where the relevance of acoustic cues may change?
Benign utility: Can the authors evaluate TriSteer on a broader range of benign audio tasks, such as speech recognition, speech translation, emotion recognition, and spoken instruction following, to demonstrate that the interventions do not compromise general audio understanding?
Calibration robustness: How sensitive are the learned steering directions and risk boundaries to the particular harmful and benign samples used during construction? Can the authors report results across independently sampled construction and calibration sets?
Comparison with simpler adaptive approaches: Can the authors compare TriSteer with a risk-adaptive version of SARSteer or other existing activation steering methods under matched conditions, to better isolate the advantages of the proposed representation-based direction refinement?
13 comments
Three-space ablations support the need for our method
Response to W1 and Q2: Necessity of the three subspaces.
We report w/o Sem and w/o Rel ablations on two models: Qwen2-Audio-7B and Omni-7B. The variants are defined below:
| Variant | Changes to space construction | Changes to control logic |
|---|---|---|
| w/o Sem | Construct Rel directly from raw paired differences without removing Sem; remove only Rel from Irr, without removing Sem. | Retain Rel / Irr acoustic evidence allocation; compute risk and direction refinement using the reconstructed spaces. |
| w/o Rel | Construct Irr from its own paired differences, removing only Sem and no longer removing Rel components. | Retain the Full view and the risk view with this Irr component removed; refit and recalibrate using the original support set. Set Rel excess evidence to 0 and ; Irr excess evidence still determines whether direction refinement is activated. |
VoiceBench refers to its OpenBookQA subset. JALMbench Others, JALMbench AdvWave, FigStep report ASR.
| Model | Method | JALMbench Others ↓ | JALMbench AdvWave ↓ | FigStep ASR ↓ | VoiceBench Acc ↑ |
|---|---|---|---|---|---|
| Qwen2-Audio-7B | TriSteer | 0.44 | 1.60 | 1.20 | 46.75 |
| w/o Sem | 0.44 | 2.40 | 1.60 | 46.75 | |
| w/o Rel | 0.88 | 2.40 | 1.20 | 46.75 | |
| Omni-7B | TriSteer | 0.44 | 6.40 | 1.20 | 81.45 |
| w/o Sem | 0.00 | 8.00 | 1.20 | 81.45 | |
| w/o Rel | 0.88 | 10.40 | 1.60 | 81.45 |
Removing Sem or Rel increases attack success rates in most tested settings.
TriSteer outperforms adaptive SARSteer
Response to W1 and Q6: Comparison with a simpler adaptive steering baseline.
We add input-dependent, layerwise magnitude control while retaining SARSteer's original direction construction and hyperparameters. We reuse its profiling representations from 100 harmful and 100 safe inputs to compute the centers at each layer, replacing the fixed with . Each layer computes this coefficient once during prefill, before its own intervention, and reuses it unchanged during decoding. The original direction and , , and are retained.
VoiceBench refers to its OpenBookQA subset. JALMbench Others, JALMbench AdvWave, FigStep, and AJailbench report ASR. We also report FigStep BRR to assess the balance between refusing harmful requests and answering paired benign requests.
| Model | Method | JALMbench Others ↓ | JALMbench AdvWave ↓ | FigStep ↓ | AJailBench ↓ | FigStep BRR ↑ | Voicebench Acc ↑ | VoiceBench BAR ↑ |
|---|---|---|---|---|---|---|---|---|
| Qwen2-Audio-7B | SARSteer | 12.72 | 48.40 | 13.20 | 18.65 | 77.40 | 42.41 | 99.04 |
| Adaptive SARSteer | 14.91 | 60.00 | 16.80 | 41.97 | 82.40 | 45.54 | 99.52 | |
| TriSteer | 0.44 | 1.60 | 1.20 | 7.77 | 83.60 | 46.75 | 99.76 | |
| Omni-3B | SARSteer | 28.95 | 76.00 | 31.60 | 20.21 | 78.00 | 77.11 | 98.80 |
| Adaptive SARSteer | 28.51 | 82.00 | 33.20 | 19.17 | 77.60 | 77.35 | 99.28 | |
| TriSteer | 1.75 | 2.00 | 2.80 | 2.59 | 89.20 | 77.11 | 99.04 | |
| Omni-7B | SARSteer | 19.30 | 75.60 | 26.80 | 9.84 | 82.00 | 80.48 | 98.55 |
| Adaptive SARSteer | 21.93 | 81.20 | 26.80 | 10.36 | 81.40 | 81.69 | 98.31 | |
| TriSteer | 0.44 | 6.40 | 1.20 | 2.07 | 92.60 | 81.45 | 98.31 |
TriSteer achieves lower attack success rates in these comparisons; adaptive scaling alone does not consistently improve SARSteer.
We thank the reviewer for the careful review and positive assessment. We appreciate your recognition of the importance of safety in large audio language models and the value of distinguishing semantic content from acoustic variations that affect safety decisions. We also thank you for recognizing the intuition and potential utility of our audio-specific design, which combines three-space measurement with adaptive magnitude and direction control. Your positive assessment of the results across three models and multiple audio safety benchmarks, as well as the representation analyses, component ablations, and alternative-judge evaluation, is encouraging. We address your concerns below.