acceptodds
Under review as a conference paper at ICLR 2027

TriSteer: Tri-Space Adaptive Steering for Large Audio Language Model Safety

Abstract

Large Audio Language Models (LALMs) have extended language-model interaction to the audio modality, while introducing new safety risks. We investigate how safety-relevant information is organized in audio-conditioned hidden representations and whether this structure can support inference-time control. A central challenge is to identify which internal semantic and acoustic evidence should trigger and shape an intervention. Our analysis reveals layer-dependent alignment between audio-native and text-only safety directions, while semantic and acoustic variations form reusable structures in the representation space. Based on these findings, we propose TriSteer, a training-free framework that constructs ordered contrast-defined measurement subspaces for semantic, decision-relevant acoustic, and decision-irrelevant acoustic variation. At each layer, TriSteer adaptively sets intervention strength from the current input's deviation relative to fixed benign reference boundaries, while acoustic attribution selectively refines the steering direction. Experiments across three LALMs and multiple safety benchmarks show that TriSteer improves the safety-utility trade-off without model updates. These results demonstrate that the geometry of internal safety representations provides a practical basis for adaptive inference-time control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Audio Language Model, Activation Steering, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TVbzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TriSteer, a training-free adaptive activation steering framework for improving the safety of Large Audio Language Models (LALMs). The authors investigate the representation geometry of audio-conditioned hidden states and construct three measurement subspaces: Semantic Space, Decision-Relevant Acoustic Space, and Decision-Irrelevant Acoustic Space.

Based on these subspaces, TriSteer adaptively adjusts both the magnitude and direction of activation steering. The magnitude controller estimates input-specific risk relative to benign reference boundaries, while the direction controller selectively reduces decision-irrelevant acoustic components from an audio-native safety direction.

Experiments on three LALMs and multiple audio safety benchmarks demonstrate substantial reductions in attack success rates while largely preserving benign-task performance. The paper also provides representation analyses and component ablations to investigate the proposed design.

Critical strengths

Important research problem. The paper addresses the safety challenges of large audio language models, particularly the need to distinguish harmful semantic content from acoustic variations that may affect safety decisions.

Interesting audio-specific design. The proposed three-space measurement framework provides an interesting perspective on separating semantic, decision-relevant acoustic, and decision-irrelevant acoustic variations. Combining these measurements with adaptive magnitude and direction control is intuitive and potentially useful.

Strong empirical results. The paper reports substantial reductions in attack success rates across multiple audio safety benchmarks and three LALM backbones. The authors also conduct representation analyses, component ablations, and evaluation with an alternative safety judge, providing useful evidence of the method's effectiveness.

Critical weaknesses

W1. The proposed three-space measurement framework is an interesting contribution, but its incremental effectiveness is not consistently demonstrated. The paper introduces a three-space representation framework that distinguishes semantic, decision-relevant acoustic, and decision-irrelevant acoustic variations. This is arguably the most distinctive methodological contribution of TriSteer compared with existing adaptive activation steering approaches. However, the ablation studies indicate that the primary safety–utility improvements arise from adaptive magnitude control, while the additional benefits of the proposed tri-space-based direction refinement are relatively limited and inconsistent across different models and attack settings. For example, the component ablations show that adaptive direction refinement sometimes provides additional safety improvements, but can also produce comparable or slightly worse results than fixed-direction adaptive-magnitude steering. The supplementary comparisons with Full-only and Full Irr removal further suggest that simpler variants can achieve similar or better performance in certain settings. Although these results support the potential usefulness of the proposed framework, they do not sufficiently establish that the more complex three-space design provides substantial and consistent advantages over simpler adaptive steering mechanisms. W2. The generalizability of the proposed three-space decomposition remains insufficiently validated. The Semantic, Decision-Relevant Acoustic, and Decision-Irrelevant Acoustic spaces are constructed from predefined contrast families through SVD-based representation analysis. The authors demonstrate that the corresponding subspaces capture distinguishable representation variations on held-out contrast pairs. However, these held-out samples are drawn from the same predefined contrast families used for subspace construction. More importantly, the distinction between decision-relevant and decision-irrelevant acoustic information can be context-dependent. An acoustic variation considered irrelevant for one task may carry important safety-related information in another. Therefore, it remains unclear whether the learned subspaces capture generalizable safety-relevant acoustic factors or primarily reflect the specific contrast construction procedures. Evaluation under substantially different acoustic conditions and more diverse audio tasks would strengthen the proposed representation-level contribution. W3. The evaluation of benign utility and cross-model generalization is relatively limited. Although the paper reports strong safety improvements, benign utility is mainly evaluated using the OpenBookQA subset of VoiceBench. This task does not fully represent the diverse capabilities of audio language models, including speech recognition, speech translation, emotion understanding, speaker-related reasoning, and long-form spoken instruction following. Since TriSteer modifies hidden representations across multiple language layers, it is important to investigate whether the intervention affects these capabilities. Furthermore, the evaluation covers three backbones from the Qwen model family. Additional evaluation on different audio model architectures and broader benign audio tasks would provide stronger evidence of generalization. W4. The stability and robustness of the proposed adaptive steering mechanism require further investigation. The proposed controller relies on relatively small sets of harmful and benign examples to construct steering directions and estimate reference boundaries. In particular, the audio-native behavior anchor is constructed from a small number of safe-refusal and unsafe-response examples. Such a construction may be sensitive to the particular samples selected. Moreover, the parameter sensitivity experiments show that changing the steering scale or calibration quantile can substantially affect attack success rates in some settings. Although the paper provides useful sensitivity analyses, it does not sufficiently establish the stability of the proposed framework across independently sampled construction sets or substantially different attack distributions. A more systematic evaluation of calibration robustness would help assess the reliability of TriSteer for practical deployment.

Questions for authors

Incremental contribution of the three-space design: The ablation results suggest that adaptive magnitude control accounts for most of the safety–utility improvements, while direction refinement provides relatively limited and sometimes inconsistent gains. What are the specific advantages of the proposed three-space framework over simpler adaptive-magnitude steering methods? Can the authors identify the conditions under which the three-space design consistently improves performance?

Necessity of the three subspaces: Are all three measurement spaces necessary for effective safety steering? Can the authors provide more fine-grained ablations of the Semantic, Decision-Relevant Acoustic, and Decision-Irrelevant Acoustic spaces to clarify their individual contributions?

Generalization of acoustic representations: How well do the learned subspaces generalize to acoustic variations outside the predefined contrast families? Can the authors evaluate the method on substantially different speaking styles, acoustic environments, or tasks where the relevance of acoustic cues may change?

Benign utility: Can the authors evaluate TriSteer on a broader range of benign audio tasks, such as speech recognition, speech translation, emotion recognition, and spoken instruction following, to demonstrate that the interventions do not compromise general audio understanding?

Calibration robustness: How sensitive are the learned steering directions and risk boundaries to the particular harmful and benign samples used during construction? Can the authors report results across independently sampled construction and calibration sets?

Comparison with simpler adaptive approaches: Can the authors compare TriSteer with a risk-adaptive version of SARSteer or other existing activation steering methods under matched conditions, to better isolate the advantages of the proposed representation-based direction refinement?

13 comments

We thank the reviewer for the careful review and positive assessment. We appreciate your recognition of the importance of safety in large audio language models and the value of distinguishing semantic content from acoustic variations that affect safety decisions. We also thank you for recognizing the intuition and potential utility of our audio-specific design, which combines three-space measurement with adaptive magnitude and direction control. Your positive assessment of the results across three models and multiple audio safety benchmarks, as well as the representation analyses, component ablations, and alternative-judge evaluation, is encouraging. We address your concerns below.

AuthorsOpenReview ↗

Three-space ablations support the need for our method

Response to W1 and Q2: Necessity of the three subspaces.

We report w/o Sem and w/o Rel ablations on two models: Qwen2-Audio-7B and Omni-7B. The variants are defined below:

VariantChanges to space constructionChanges to control logic
w/o SemConstruct Rel directly from raw paired differences without removing Sem; remove only Rel from Irr, without removing Sem.Retain Rel / Irr acoustic evidence allocation; compute risk and direction refinement using the reconstructed spaces.
w/o RelConstruct Irr from its own paired differences, removing only Sem and no longer removing Rel components.Retain the Full view and the risk view with this Irr component removed; refit and recalibrate using the original support set. Set Rel excess evidence to 0 and r=1r=1; Irr excess evidence still determines whether direction refinement is activated.

VoiceBench refers to its OpenBookQA subset. JALMbench Others, JALMbench AdvWave, FigStep report ASR.

ModelMethodJALMbench Others ↓JALMbench AdvWave ↓FigStep ASR ↓VoiceBench Acc ↑
Qwen2-Audio-7BTriSteer0.441.601.2046.75
w/o Sem0.442.401.6046.75
w/o Rel0.882.401.2046.75
Omni-7BTriSteer0.446.401.2081.45
w/o Sem0.008.001.2081.45
w/o Rel0.8810.401.6081.45

Removing Sem or Rel increases attack success rates in most tested settings.

AuthorsOpenReview ↗

TriSteer outperforms adaptive SARSteer

Response to W1 and Q6: Comparison with a simpler adaptive steering baseline.

We add input-dependent, layerwise magnitude control while retaining SARSteer's original direction construction and hyperparameters. We reuse its profiling representations from 100 harmful and 100 safe inputs to compute the centers μH,μS\mu_H,\mu_S at each layer, replacing the fixed βℓ=1\beta_\ell=1 with βℓ(x)=clip((hℓ−μS)⊤(μH−μS)/∥μH−μS∥22,0,1)\beta_\ell(x)=\mathrm{clip}((h_\ell-\mu_S)^\top(\mu_H-\mu_S)/\|\mu_H-\mu_S\|_2^2,0,1). Each layer computes this coefficient once during prefill, before its own intervention, and reuses it unchanged during decoding. The original direction and α=0.10\alpha=0.10, λ=1\lambda=1, and k=10k=10 are retained.

VoiceBench refers to its OpenBookQA subset. JALMbench Others, JALMbench AdvWave, FigStep, and AJailbench report ASR. We also report FigStep BRR to assess the balance between refusing harmful requests and answering paired benign requests.

ModelMethodJALMbench Others ↓JALMbench AdvWave ↓FigStep ↓AJailBench ↓FigStep BRR ↑Voicebench Acc ↑VoiceBench BAR ↑
Qwen2-Audio-7BSARSteer12.7248.4013.2018.6577.4042.4199.04
Adaptive SARSteer14.9160.0016.8041.9782.4045.5499.52
TriSteer0.441.601.207.7783.6046.7599.76
Omni-3BSARSteer28.9576.0031.6020.2178.0077.1198.80
Adaptive SARSteer28.5182.0033.2019.1777.6077.3599.28
TriSteer1.752.002.802.5989.2077.1199.04
Omni-7BSARSteer19.3075.6026.809.8482.0080.4898.55
Adaptive SARSteer21.9381.2026.8010.3681.4081.6998.31
TriSteer0.446.401.202.0792.6081.4598.31

TriSteer achieves lower attack success rates in these comparisons; adaptive scaling alone does not consistently improve SARSteer.

Reviewer_6S3ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TriSteer, an training-free activation steering method for improving the safety of large audio language models. It constructs three subspaces for semantic, decision-relevant acoustic, and decision-irrelevant acoustic variation to adapt the strength and direction of steering. Experiments on three models show reduced attack success rates while maintaining accuracy on the OpenBookQA subset of VoiceBench.

Critical strengths
  1. The motivation is clear: acoustic cues can be relevant or irrelevant to safety decisions and should be treated differently.

  2. The method achieves strong defense performance across the evaluated models and attack benchmarks.

Critical weaknesses
  1. Table 10 shows that Full-only already achieves most of the safety gains without constructing the three spaces, suggesting that the additional contribution of the three-space design is limited.

  2. General audio QA performance is evaluated only on the OpenBookQA subset of VoiceBench. This limited evaluation does not establish whether TriSteer preserves performance across a broader range of audio understanding tasks.

Questions for authors

Please see the weaknesses above

10 comments

We thank the reviewer for recognizing our clear motivation and TriSteer’s strong safety performance across models and attack benchmarks. Our additional experiments show that TriSteer improves aggregate safety over Full-only across the evaluated acoustic scenarios. We also broaden benign evaluation beyond knowledge-based QA: emotion and spoken intent recognition remain stable, while environmental sound classification is largely preserved.

AuthorsOpenReview ↗

Full-only cannot replace three-space direction control

Response to Weakness 1: The contribution of three-space control beyond Full-only.

We agree that the strong performance of Full-only demonstrates the effectiveness of the shared audio-native anchor and adaptive magnitude control. The contribution of the three spaces is to extend global risk gating to selective control based on representation attribution: they provide distinct views of risk and identify which components of the intervention direction should be attenuated. The effectiveness of Full-only is compatible with this additional control mechanism.

  • Selective changes to direction components cannot be replicated by magnitude scaling alone. At a fixed layer, decompose the anchor into its Irr projection virr\mathbf{v}_{\mathrm{irr}} and the orthogonal remainder v⊥\mathbf{v}_\perp. Direction refinement can be written intuitively as

    \mathbf{v}_{\mathrm{corr}}=\mathbf{v}_\perp+(1-\omega)\mathbf{v}_{\mathrm{irr}},\quad 0\leq\omega\leq1.$$ Full-only scales both components by the same factor, whereas TriSteer can selectively reduce the relative contribution of Irr. When both components are nonzero and $\omega>0$, the refined direction is no longer collinear with the original anchor. Full-only therefore cannot reproduce this nonzero update by adjusting magnitude alone, and subsequent norm restoration does not eliminate the directional difference.
  • The three spaces jointly define what to attenuate and what to preserve. Sem removes measured semantic variation from acoustic attribution, Rel provides a reference for acoustic information to preserve, and Irr identifies the components targeted for attenuation. Ordered residualization, Sem→Rel→Irr, prioritizes the shared components of the first two spaces and makes the final Irr space orthogonal to them. Direction refinement therefore leaves the Sem and Rel projections unchanged before norm restoration; afterward, the components not selectively attenuated are scaled together. This defines a more specific control target than treating all acoustic variation as interference.

  • The three spaces also provide a risk measurement view that Full-only does not explicitly offer. In standardized coordinates, the relevant view removes the measured Irr component before fitting a risk axis:

    (I-P'_{\mathrm{irr}})(\mathbf{z}+\mathbf{n})=\mathbf{z}_R \quad\text{if }\mathbf{n}\in\operatorname{range}(P'_{\mathrm{irr}}).$$ Perturbations within the measured Irr space leave the relevant-view risk score unchanged, a property the Full branch alone does not guarantee. This provides a complementary risk measurement that cannot be obtained simply by changing the threshold of the Full-view score. Combining both views through a maximum allows the controller to retain full-representation risk evidence while also detecting risk that becomes clearer after removing the measured Irr component.

Compared with Full-only, TriSteer thus extends control over whether and how strongly to intervene to control over which representation evidence to use and which directional components to adjust selectively.

AuthorsOpenReview ↗

TriSteer outperforms Full-only in two acoustic scenarios

Response to Weakness 1: Advantages over Full-only in two acoustic scenarios.

Impaired capacity evaluates dangerous-operation requests under conditions such as intoxication, using the Aware metric. Aware excludes SKIP outcomes from its denominator. Overlapping instruction injection superimposes malicious speech on a benign request and measures the proportion of Safe responses. We retain the finalized construction configuration and parameters, and use the official rubrics and metric definitions with DeepSeek-Flash as the judge. Results are listed by model and method below (%, higher is better for both metrics).

ModelMethodImpaired capacity:Aware ↑Overlap injection:Safe ↑
Qwen2-Audio-7BFull-only9.3388.50
TriSteer14.2089.96
Qwen2.5-Omni-3BFull-only9.4096.35
TriSteer15.3897.99
Qwen2.5-Omni-7BFull-only20.9097.08
TriSteer23.8697.63

TriSteer improves on Full-only in the two evaluated acoustic scenarios.

Reference

Wang, Yuxiang, Hongyu Liu, Yijiang Xu, Qinke Ni, Li Wang, Wan Lin, Kunyu Feng, Dekun Chen, Xu Tan, Lei Wang, Jie Shi, and Zhizheng Wu. (2026). VoxSafeBench: Not Just What Is Said, but Who, How, and Where. arXiv preprint arXiv:2604.14548. https://arxiv.org/abs/2604.14548

Reviewer_BYdBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TriSteer, a training-free inference-time steering method for improving the safety of large audio language models (LALMs). It constructs three contrast-based subspaces—semantic, decision-relevant acoustic, and decision-irrelevant acoustic—and uses them to adapt the magnitude and direction of activation steering. Experiments on three Qwen-family models show lower attack success rates across several audio jailbreak benchmarks while largely preserving performance on the evaluated benign task.

Critical strengths

The paper addresses an important audio safety problem with a clearly structured inference-time steering framework, and evaluates it across multiple attack benchmarks with ablation studies and an alternative evaluator.

Critical weaknesses

(1)The “decision-relevant” and “decision-irrelevant” acoustic spaces are defined through selected contrast sets. In particular, Rel contrasts are extracted with task-specific prompts that instruct the model to attend to speech content or paralinguistic cues. Although each paired contrast uses the same prompt at both endpoints, prompt–audio interactions may still shape the resulting representation directions. The paper therefore does not establish that these spaces capture the model’s naturally used acoustic evidence under its default deployment configuration. Validation under default prompts, and sensitivity analyses for prompt wording and acoustic attributes, are needed. (2)Risk fitting uses only 20 harmful inputs, and the unsafe-response anchor for Qwen2.5-Omni-7B is based on four examples. The benign calibration boundary is estimated from 41 inputs. Such small supports may make the fitted directions and thresholds sensitive to individual samples. The paper should report resampling or bootstrap analyses that quantify how changes in the construction data affect the controller and final safety results.

Questions for authors

Please refer to the weakness.

11 comments

We thank the reviewer for recognizing the importance of audio safety, the clear structure of TriSteer, and the evaluation across multiple benchmarks, ablations, and an alternative judge. We have conducted additional experiments on construction-prompt sensitivity and transfer to the default inference configuration, as well as support-set expansion at two scales. The results support that TriSteer remains effective across construction prompts and with larger support sets.

AuthorsOpenReview ↗

Defense remains effective across construction prompts

Response to Weakness 1: Sensitivity to construction prompts.

Reconstruction with default prompts yields safety and utility results close to the original configuration, and none of the tested wording variants causes an overall breakdown. We compare four construction-prompt conditions on all three models: default, short acoustic, original, and paraphrased. The table below illustrates the four conditions using the English happy-attribute prompt. The original and paraphrased prompts are longer, so only excerpts are shown. The default condition does not override the model's default system configuration.

Construction conditionExample prompt
DefaultNo additional prompt; retain the model's default configuration.
Short acousticWhen answering the speaker's questions, pay attention not only to what the speaker says but also to the speaker's paralinguistic information. Respond with appropriate content.
Original (excerpt)If the tone reveals a strong emotion, ignore the literal plainness or neutrality of the text transcript; the emotion conveyed by the tone must dictate your response. Your reply must begin by mirroring and acknowledging the user's emotion before addressing the actual content.
Paraphrased from original (excerpt)Respond to both the spoken content and its emotional tone. When the voice conveys strong emotion despite neutral wording, let that emotion guide your response. Acknowledge it first: share excitement for joy, express solidarity with anger, and offer gentle comfort for sadness.
AuthorsOpenReview ↗

Defense remains effective across construction prompts (continued)

The full comparison, including the default and original conditions, is shown below. VoiceBench refers to its OpenBookQA subset. JALMbench Others, JALMbench AdvWave, FigStep, and AJailbench report ASR. We also report FigStep BRR to assess the balance between refusing harmful requests and answering paired benign requests.

ModelConstruction promptJALMbench Others ↓JALMbench AdvWave ↓FigStep ↓AJailBench ↓FigStep BRR ↑VoiceBench Accuracy ↑VoiceBench BAR ↑
Qwen2-Audio-7BOriginal0.441.601.207.7783.6046.7599.76
Default0.441.601.607.2583.6046.7599.76
Short acoustic0.441.201.206.2283.4046.7599.76
Paraphrased from original0.441.601.207.7783.6046.7599.76
Omni-3BOriginal1.752.002.802.5989.2077.1199.04
Default1.322.003.202.5988.8077.1199.04
Short acoustic1.322.003.202.5988.8077.1199.04
Paraphrased from original1.322.002.802.5988.4077.1199.04
Omni-7BOriginal0.446.401.202.0792.6081.4598.31
Default0.886.801.602.0792.8081.4598.31
Short acoustic0.888.401.202.0792.8081.4598.31
Paraphrased from original0.447.201.202.0793.0081.4598.31

Defense performance remains broadly comparable across the tested construction prompts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Evidence and Manuscript Revisions

We thank the reviewers for their constructive feedback. Our rebuttal provides additional evidence for TriSteer’s central design: combining audio-native steering with complementary risk measurements and selective direction control. In response to the reviewers’ concerns, we have added the following experiments and explanations to the revised manuscript.

  • Dependence on construction prompts and acoustic attributes. Defense performance remains broadly comparable across default, short acoustic, original, and paraphrased construction prompts. Default-prompt Rel spaces capture held-out acoustic changes better than non-matching controls. Cross-attribute experiments further identify shared acoustic structure, while showing that including the target attribute improves coverage. We document default-prompt transfer in Appendix A.4, cross-attribute transfer in Appendix A.5, and construction-prompt sensitivity in Appendix C.6.

  • Contribution beyond simpler steering mechanisms. Removing Sem or Rel increases attack success rates in most reported settings. TriSteer also achieves lower attack success rates than the evaluated adaptive SARSteer variant. Compared with Full-only, TriSteer improves both evaluated official VoxSafeBench metrics and achieves lower aggregate unsafe-response rates across all three models under Qwen3Guard. These results complement the mechanistic distinction: magnitude scaling alone cannot reproduce selective attenuation of direction components. We add the mathematical explanation to Appendix B.1, space ablations to Appendix C.4, adaptive SARSteer comparisons to Appendix C.5, and acoustic-scenario evaluations to Appendix C.8.

  • Preservation of broader audio capabilities and transfer to another model. Additional evaluations on CREMA-D, SLURP-Intent, and ESC-50 show that emotion and spoken intent recognition remain comparable to the unsteered baseline, while environmental sound classification is largely preserved. On Audio Flamingo 3, TriSteer reduces attack success rates relative to SARSteer on the evaluated benchmarks while retaining the base model’s benign QA accuracy. We report the broader benign-task evaluations in Appendix C.9 and Audio Flamingo 3 results in Appendix C.10.

  • Sensitivity to support-set size. We expanded harmful fitting, benign fitting, and independent benign calibration from 20/20/41 to 30/30/60 and 50/50/100, evaluating three candidate sets at each scale. Despite variation in individual metrics, both scales retain low attack success rates in most settings and broadly stable benign QA accuracy. Appendix C.7 reports both expansion scales alongside the original configuration.

Together, these results strengthen the empirical support for TriSteer’s three-space design, its robustness to the tested construction choices, and its balance between safety and utility. The revised appendices include the supplementary results, concise experimental settings, and additional explanation of the control mechanism. We have also made minor corresponding updates to the main text. We appreciate the reviewers’ and Area Chair’s consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.