acceptodds
Under review as a conference paper at ICLR 2027

TADA! Tuning Audio Diffusion Models through Activation Steering

Abstract

Audio diffusion models can synthesize high-fidelity music from text, yet achieving fine-grained control over specific musical attributes remains challenging as their internal mechanisms for representing high-level concepts are poorly understood. In this work, we use activation patching to demonstrate that recent audio diffusion architectures exhibit a semantic bottleneck, where a small, shared subset of consecutive attention layers controls distinct musical concepts, such as the presence of specific instruments, vocals, or genres. Building on this, we systematically evaluate a broad spectrum of steering paradigms, comparing activation steering against prompt-level, score-space, and weight-space interventions, analyzing the interaction between the steering mechanism and the intervention site. Our new benchmark, supported by the extensive user study demonstrates that localized activation steering establishes a new state-of-the-art in audio concept modulation. Audio examples are provided at https://audio-steering.github.io.

Primary area:
interpretability and explainable AI
Keywords:
audio, diffusion models, activation steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NCSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses a form of sensitivity scoring (Eq. 2) to discover that there are some critical layers (named "semantic bottleneck" in the paper) in the music generative model that steer the musical concepts of its outputs. This phenomenon is found to appear in several diffusion-based backbones, i.e., Stable Audio Open, AudioLDM-2 and ACE-Step.

The discussed musical concepts include instrumentation, male/female vocal presence, musical genre, mood, and tempo.

Experimental validation is done on the ACE-Step model and benchmarks localized (i.e., only on "semantic bottleneck") manipulation vs. all-layer manipulation using 8 existing steering methods (Table 1). The benefits of localized manipulation is dependent on steering method, with activation-based steering performing the best.

Critical strengths

The paper presents interesting and informative discoveries that

  • the semantic bottleneck is restricted to 10% or fewer layers of a generative backbone,
  • the relative positions of those layer can vary a lot from model to model (e.g., late in AudioLDM-2, early in ACE-Step).
Critical weaknesses
  • (W1) Limited scope: While the semantic bottleneck phenomenon is interesting and the paper also describes a straightforward method to locate those layers (through Eq. 2), the experimental validation has rather limited scope, i.e., only on diffusion-based, music generation models. This issue is especially substantial since the framework/method, if I understand correctly, is generally applicable to any prompt-conditioned generative models, across domains (e.g., audio, text, vision) and modeling paradigms (e.g., diffusion and autoregression). The paper would be a lot more convincing if the benefits can be shown to apply on multiple domains and types of generative models.

  • (W2) Novelty statement is unclear: If I understand correctly, both the activation patching sensitivity analysis (from Meng et al., 2022) and all benchmarked steering methods are from existing work. In its current form, it is somewhat difficult to identify what the core methodological contribution is, and what impact the work might have on the research field.

Questions for authors
  • (Q1) Could the authors clarify the core algorithmic/methodological contribution of the work? For example, what was not feasible before, and made possible by this work; or, what critical insights the work brings that would change how steering is done going forward.

  • (Q2) From Figure 5 it seems experiments were done on all 3 generative models discussed earlier in the semantic bottleneck discovery, is there a particular reason that all other result tables and figures in main text only include ACE-Step?

  • (Q3) Have the authors conducted similar analyses on LM-based music generation models (e.g., MusicGen)? I'd be curious if the same trends generally transfer from diffusion to autoregressive models.

  • (Q4) Is there a systematic reason for the different steering methods included in each figure/table? For example, I see only localized activation-steering in Table 1 (which gives the impression that localized steering is only applicable to activation-based methods), but both all-layer and localized results for all steering methods in Figure 4. This kind of seemingly random method inclusion also happens in Fig. 5, where only the activation-based CAA results are reported, and then Figure 6, where some prompt-based and score-based methods come back. It would be must easier for readers to grasp the most substantial insights/conclusion if the methods involved are consistent across experiments.

5 comments

We highly appreciate the reviewer's comments and questions. Below, please find our responses to each raised concern.

W1+Q3. Limited scope. Analysis on LM-based generators.

Thank you for this interesting research question!

Prior work [1,2,3] in the image diffusion domain localizes specialized subsets of model layers, but these subsets are disjoint across concept categories. In Stable Diffusion XL (70 cross-attention layers), layers 15–19 control objects while 45–47 control style (see Fig. 5 in [1]), visual text content is controlled by layers 55–57 (see Fig. 2 in [2]). Moreover, recent work on Diffusion Transformers (see Fig. 1 in [3]) reports an even more diffuse distribution of semantic knowledge. These analyses already suggest that there is no bottleneck governing the semanticity of models as a whole in image generators, making our observations specific to audio models.

To better understand whether existence of semantic bottleneck is a universal property of audio generators, we replicate our localization experiment (with same set of counterfactual prompts, concepts, and alignment metric) on the MusicGen-small and MusicGen-medium, which contain, respectively 2424 and 4848 cross-attention layers. We exclude vocal concepts from localization because neither model produces any audible vocals when explicitly conditioned with a prompt.

In Figure 29 (Appendix Q), we present results of this experiment. As is visible, the general trend from diffusion-based generators does not fully transfer to autoregressive models:

  1. Layer specialization is more concept-dependent. The layer impact profile is much more spread out within concepts than for diffusion models. Control over instruments (drums, violin) and genres (reggae, jazz) is distributed between layers. For these concepts, no layer has I(l,c)>0.23I(l,c)>0.23.
  2. The bottleneck is not shared by all the concepts. We do not observe a compact subset of layers consistently dominating across the evaluated concepts. While L5 in MusicGen-small and L13 in MusicGen-medium influence tempo and mood, they show little to no impact on instruments and genres. The concepts peak across 5 layers in MusicGen-small and 6 in MusicGen-medium, whereas in Audio DMs, every concept peaks within 2–3 layers. Consequently, the most impactful layer reaches only I(L5)=0.23I(L5)=0.23 in MusicGen-small and I(L13)=0.13I(L13)=0.13 in MusicGen-medium, against I(L8)=0.49I(L8)=0.49 in ACE-Step, I(L13)=0.51I(L13)=0.51 in Stable Audio and I(L51)=0.36I(L51)=0.36 in AudioLDM2.

These results show that the compact, shared semantic bottleneck observed in the three evaluated audio diffusion models does not extend to MusicGen. Instead, semantic control in this autoregressive model is more distributed and varies across concepts. This suggests that the organization of semantic control depends on the generative architecture, rather than being a universal property of music generators.

[1] Basu, Samyadeep, et al. "On mechanistic knowledge localization in text-to-image generative models." ICML. 2024.

[2] Staniszewski, Łukasz, et al. "Precise parameter localization for textual generation in diffusion models." ICLR. 2025.

[3] Zarei, Arman, et al. "Localizing knowledge in diffusion transformers." NeurIPS. 2025.

W2+Q1. Unclear novelty. What does the paper bring to the research field?

We do not claim algorithmic novelty in activation patching or the steering methods we evaluate. Our contribution is to establish what these tools reveal about semantic control in text-to-music diffusion models and how those findings improve steering. As best stated by Reviewer dzFb, our paper "suggests directions for effective control for controllable music generation models", and these directions are:

1. Global intervention site for concepts. In image diffusion models, the layers that control concepts differ between their categories, so each new concept needs its own layer search. In the audio diffusion models we study, the functional layers are shared across concepts, so localization is a one-time cost per model that transfers to new concepts. As an example, we evaluate steering on 4 concepts (piano, vocal style, guitar type, classical music) that were not used in the patching experiment, and their localized steering with CAA outperforms the all-layer approach.

2. Layers should be selected causally, not discriminatively. A growing number of methods in activation steering choose where to apply steering by how well the concepts can be read out from the given layer: AUSteer [1] measures sign consistency of activation differences between positive-direction and negative-direction runs, while RFMs [2] analyze how probes trained on each layer activation classify the concept and chooses the best-performing one. Such scoring methods indicate where a concept is encoded, not where intervention changes the output. Further details on this experiment are now provided in Appendix R.

To test this distinction, we now add RFM, which was absent from the original submission, and adapt it to ACE-Step (Appendix I.2). Its probes achieve ROC AUCs of 0.920.92–0.970.97 across 21 of 24 layers, indicating that concepts are decodable throughout much of the model. Similarly, AUSteer’s global selection allocates only 2%–16% of its steering budget to the identified bottleneck for piano, mood, and violin. Yet, as shown below, restricting either method to this bottleneck improves steering over its original selection rule:

Intervention selectionRFM, AUC MuQ ↑RFM, AUC CLAP ↑AUSteer, AUC MuQ ↑AUSteer, AUC CLAP ↑
All layers0.042 ± .0030.032 ± .002--
Original selection rule0.065 ± .0040.045 ± .0020.081 ± .0040.058 ± .002
Our semantic bottleneck {7,8}0.118 ± .0050.070 ± .0030.096 ± .0040.063 ± .003

3. The benefit of localization depends on the steering paradigm. Restricting interventions to the bottleneck helps every activation-space method, but degrades weight-space steering and gives mixed results for prompt-level methods. The intervention site should therefore be chosen together with the steering mechanism.

[1] Feng, Zijian, et al. "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.

[2] Zhao, Daniel, et al. "Steering autoregressive music generation with recursive feature machines." ICLR 2026.

Q2. Why do benchmark results include only ACE-Step?

We thank the reviewer for this question. We used ACE-Step for the main benchmark because it is the newer state-of-the-art model, generally producing higher-quality audio than AudioLDM2 and Stable Audio Open, and surpassing them in prompt-following capabilities. During rebuttal, we extended the steering methods comparison on our benchmark to the other two architectures: AudioLDM2 (U-Net) and Stable Audio Open (DiT).

We compare the methods across models on 44 concepts: piano, tempo, mood, and vocal gender, steering in both directions. For each group of baseline methods, we choose the best-performing representatives: PCI (prompt space), FreeSliders (score space), and Concept Sliders (weight space). For activation steering, we compare CAA and AUSteer.

methodAudioLDM2, AUC MuQAudioLDM2, AUC CLAPStable Audio, AUC MuQStable Audio, AUC CLAP
PCI (all)0.00780.00350.0630.030
PCI (loc.)0.00550.00240.0370.018
FreeSliders (all)0.00880.00470.0470.032
FreeSliders (loc.)0.00860.00490.0360.025
Concept Sliders (all)0.00480.00580.0210.024
Concept Sliders (loc.)0.00380.00360.0150.027
CAA (all)0.00620.00620.0470.031
AUSteer (all)0.00820.00920.0480.025
---------------
CAA (loc.)0.00930.00630.0580.037
AUSteer (loc.)0.01090.00790.0630.036

We observe that our original benchmark results on ACE-Step mostly transfer to AudioLDM2 (ALDM2) and Stable Audio Open (SAO): localized activation steering methods (CAA, AUSteer) occupy first (ALDM2 MuQ, SAO MuQ, SAO CLAP) or second (ALDM2 CLAP, SAO CLAP) spots in rankings. Notably, on AudioLDM2 (CLAP), the best-performing method is global AUSteer, and on Stable Audio Open (MuQ), PCI performs as well as localized AUSteer.

Q4. Reason for using different steering methods in figures/tables.

Thank you for pointing this out. The differences reflect presentation constraints and the scope of the original experiments, but we agree that we did not explain them clearly enough.

Table 1 is a compact summary of the main benchmark. To avoid a full-page table, we retained the best-performing localized activation-steering methods, along with the baselines from other steering paradigms. This was a presentation choice, not a restriction on which methods support localization. Table 2 and Figures 5 and 7 (before update, 4 and 6) provide the full comparisons, including both all-layer and localized variants across steering paradigms. These comparisons are important to our conclusion: localization consistently benefits activation-based steering, whereas its effects on other paradigms are mixed.

Figure 6 (initially 5) addresses a different question: whether the benefit of localized activation steering extends across generative models. In the original submission, we evaluated CAA as a representative activation-steering method for this comparison, rather than repeating the entire benchmark on each model. Following the reviewers’ feedback, we have now added benchmarks on AudioLDM2 and Stable Audio Open using the same set of methods—CAA, AUSteer, PCI, FreeSliders, and Concept Sliders—with both all-layer and localized variants.

In the updated version of the paper, we add the new cross-model benchmarks to make the method coverage more consistent across architectures.

Reviewer_dzFbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks: how and where to steer text-to-music models so that the steering is the most effective while preserving audio quality. An empirical study shows that SOTA text-to-music model contains a "semantic bottleneck" (i.e., a few layers in the cross-attention block). Applying conditioning on these layers is proven to be the most effective against baselines which apply conditioning at the prompt, embedding, the entire cross-attention block or during the diffusion process. This paper suggests directions for effective control for controllable music generation models.

Critical strengths
  1. Identifies the semantic bottlenecks in text-to-music generation systems, which provides empirical guidance for future controllable systems.
  2. Applying conditioning on these semantic bottlenecks is well motivated and has proven effective. It also outperforms global conditioning most of the time both subjectively and objectively.
Critical weaknesses

Weakness:

  1. The "shared" bottleneck is partly from averaging and thresholding and the only evidence is in Figure 3. More ablation studies can be done by changing the selection threshold.
  2. Table 2 and Appendix L report localized variants for every method: PCI (loc.), Text Emb (loc.), FreeSliders (loc.), Concept Sliders (loc.), but how can localized approach be applied to the prompt is not explained in the paper. This is confusing since the prompt and embedding do not contain "layers".
  3. Local steering is not a new concept and have been proven effective both in adjacent domains (e.g., image editing), or for audio. This weakens the novelty of the paper.

Weakness (Presentation):

  1. Some concepts (e.g., alignment–preservation) are mentioned in the introduction, but they are only explained later in the paper. Moreover, many baseline models are included, but their names are not self-explanatory. These hinder the natural flow of the paper. I suggest including a new plot and using arrows to point at where the conditioning can be applied and include the baseline's name there. Moreover, some namings can be improved, for instance, the 'ablated' setting in Section 5.4 refers to steering all layers except functional layers.
Questions for authors
  1. LPAPS is used as the audio preservation metric, are there any alternatives? Why is this specific metric used?
  2. What information do the rest of the layers besides the semantic bottlenecks contain?
4 comments

We thank the Reviewer for their time and careful review. Below, we address each point.

W1. The shared bottleneck is from averaging and thresholding; more ablation studies are required.

While the aggregated impact I(l)I(l) in Figure 3 indeed hides how impact varies between concepts, we would like to point the Reviewer to Appendix F.1, where Figures 8–10 report the per-concept scores I(l,c)I(l,c) for all three architectures. They show that the most impactful layers are shared between concepts.

To further check whether our observations do not hinge on our localization methodology, we further test on the stability of selecting the top-2 layers with ACE-Step ({7,8}) under variations:

  • Similarity probes: computing aggregated impact I(l)I(l) only with MuQ-T or CLAP selects the same top-2 set of layers.
  • Reseeding: when generations start from 8 independent initial noise seeds, the same top-2 set is chosen 8/8 times.
  • Choice of prompts: resampling each concept's counterfactual prompts with replacement gives the same top-2 set in 10,000/10,000 draws.
  • Choice of concepts: computing the aggregated I(l)I(l) on every subset of concepts gives the same top-2 layers in: 110/120 3-concept subsets, 207/210 4-concept subsets, 251/252 5-concept subsets, and always the same top-2 layer set 6 or more concept subsets.

Following the Reviewer's suggestion, we also evaluate how sensitive steering performance is to the threshold, using CAA method on 4 concepts (piano, tempo, mood, vocal gender). Below, we vary τ\tau and report the resulting MuQ AUC and CLAP AUC. For reference, we also report steering performance on three random pairs of adjacent layers ({4,5}, {15,16}, {20,21}).

τ\tau rangeselected layersMuQ AUCCLAP AUC
(0.29, 0.49]{8}0.0600.054
(0.094, 0.29]{7,8}0.0700.062
(0.035, 0.094]{6,7,8}0.0690.060
(0.027, 0.035]{6,7,8,14,15}0.0660.060
0all 240.0490.053
—random pairs0.020 / 0.002 / 0.0020.020 / 0.004 / −0.001

The experiment shows that choosing a single, most impactful layer ({8}) for steering is not enough, as it loses 13–14% AUC compared to {7,8}. Notably, once both bottleneck layers are covered by the threshold, its exact value barely matters, as every layer set selected by τ∈(0.027,0.29]\tau \in (0.027, 0.29] lowers the AUC by at most 5%. We observe that adding more layers above the bottleneck does not benefit steering, and the all-layer variant lowers the AUC by 15–30%. Finally, three randomly selected pairs of adjacent layers achieve less than a third of the bottleneck's AUC, suggesting that steering quality depends on where we steer rather than on the exact threshold value.

In the updated version of our submission, we describe these experiments in Appendix F.2 (layer stability) and Appendix L.2 (sensitivity to the threshold)

W2. Localized variants for non-activation methods.

Thank you for pointing this out; we have added precise descriptions in Appendix I. Diffusion backbones read the text conditioning separately in every cross-attention layer. Notably, every such layer can receive a different conditioning during generation. The global variant of a prompt-level method therefore feeds the modified conditioning to all layers (a different prompt in PCI and FreeSliders, an edited embedding in Text Embeddings and Token Embeddings). The localized variant feeds it only to the layers of the semantic bottleneck, keeping the unmodified, neutral conditioning in remaining layers. Precisely:

  • PCI - bottleneck layers receive the concept-altering prompt (PcP_c or P_\tilde{c}), and the other layers keep the neutral prompt PnP_n.
  • Text Embeddings and Token Embeddings - the bottleneck layers receive the edited embedding e′\mathbf{e}', while the others receive the neutral prompt embedding en\mathbf{e}_n.
  • FreeSliders - in the directional predictions of Eq. 5 (ϵθ(xt,Pc)\epsilon_\theta(x_t,P_c) and \epsilon_\theta(x_t,P_\tilde{c})), the bottleneck layers receive the directional prompt (PcP_c and P_\tilde{c}, respectively), while the others keep the neutral one PnP_n.
  • Concept Sliders - the LoRA adapters are attached only to the attention projections of the bottleneck layers and are trained there with the same objective.

In the paper, we added a short general description to Sec. 5.1 and included more detailed per-method descriptions in App. I.1.

W3. Local steering is not novel.

While we agree that restricting steering to a subset of layers has proven beneficial in the past, we believe our analyses provide new insights into how to make such controllable audio generation more effective. In particular:

1. Global intervention site for concepts. In image diffusion models, the layers that control concepts differ between their categories, so each new concept needs its own layer search. Our experiments with audio suggested that the functional layers are shared across concepts, so localization is a one-time cost per model that carries over to new concepts. As an example, we evaluate steering on 4 concepts (piano, vocal style, guitar type, classical music) that were not used in the patching experiment, and their localized steering with CAA outperforms the all-layer approach.

2. Layers should be selected causally, not discriminatively. A growing number of methods in activation steering choose where to apply steering, by how well the concepts can be read out from the given layer: AUSteer [1] measures sign consistency of activation differences between positive-direction and negative-direction runs, while RFM [2] analyzes how probes trained on each layer activation classify the concept and choose the best-performing one. Such scoring methods indicate where a concept is encoded, not where intervention changes the output.

To test this distinction, we now add RFM, which was absent from the original submission, and adapt it to ACE-Step (Appendix R.1). Its probes achieve ROC AUCs of 0.920.92–0.970.97 across 21 of 24 layers, indicating that concepts are decodable throughout much of the model. Similarly, AUSteer’s global selection allocates only 2%–16% of its steering budget to the identified bottleneck for piano, mood, and violin. Yet, as shown below, restricting either method to this bottleneck improves steering over its original selection rule:

Intervention selectionRFM, AUC MuQ ↑RFM, AUC CLAP ↑AUSteer, AUC MuQ ↑AUSteer, AUC CLAP ↑
All layers0.042 ± .0030.032 ± .002--
Original selection rule0.065 ± .0040.045 ± .0020.081 ± .0040.058 ± .002
Our semantic bottleneck {7,8}0.118 ± .0050.070 ± .0030.096 ± .0040.063 ± .003

Full results for this experiment are provided in Appendix R.

3. The benefit of localization depends on the steering paradigm. Restricting interventions to the bottleneck helps every activation-space method, but degrades weight-space steering and gives mixed results for prompt-level methods. The intervention site should therefore be chosen together with the steering mechanism.

We believe these contributions offer valuable insights for the research community and could influence how audio steering methodologies are developed in the future.

[1] Feng, Zijian, et al. "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.

[2] Zhao, Daniel, et al. "Steering autoregressive music generation with recursive feature machines." ICLR 2026.

W4. Readability.

Thank you for these suggestions. We added Figure 4 in Sec. 5.1 that presents a single denoising step and uses arrows to point to where each baseline intervenes. We revised the Introduction, where we now define alignment, preservation, and the steering paradigms, spell out acronyms, and use consistent names for the method families and variants throughout the paper. We also renamed the “ablated” setting to “outside bottleneck”.

Q1. Why LPAPS? Alternative preservation metrics.

We used LPAPS in our main experiments because it is the standard perceptual metric for assessing the similarity between two audio signals in image editing [1,2].

However, in Appendix H.3 we included other decomposable metrics from the Musical Information Retrieval domain. In this experiment, we, next to LPAPS, replicated our evaluation protocol with preservation metrics targeting different measures for audio:

  • Harmonic similarity - per-frame cosine similarity between tracks' chromagrams,
  • Rhythm/beat similarity - F-measure between audio beat sequences,
  • Melody - the raw pitch accuracy between fundamental frequencies,
  • Structural similarity - Pearson correlation of chroma self-similarity matrices.

Table 11 in the revised paper confirms that our ranking does not hinge on LPAPS, as localized activation steering scores best or second-best across all ten columns (55 preservation metrics ×\times 22 alignment metrics).

[1] Manor, Hila, and Tomer Michaeli. "Zero-shot unsupervised and text-based audio editing using ddpm inversion." ICML 2024.

[2] Yang, Yi, et al. "Melodia: Training-free music editing guided by attention probing in diffusion models." AAAI 2026.

Q2. What information is encoded by other layers?

Thank you for this suggestion, which is a really interesting research question on its own. To better understand the role of other layers, we ran a new experiment on all three models with a hypothesis that layers outside the semantic bottleneck develop perceptual, non-semantic audio attributes. To this end, we ran a sweep in which, one at a time, we set the outputs of individual layers to 00 and analyzed how this ablation affected the generated audio.

First, we analyze the impact of cross-attention layers outside the semantic bottleneck. We use 128128 prompts, with 44 different starting noises for each. In Figure 27 (Appendix P), we measure the MuQ and CLAP similarity of a generated clip to the input prompt minus its mean similarity to the other 127 prompts. We find that across all 3 models, zeroing a single cross-attention outside the bottleneck barely changes alignment. Precisely, layers outside the bottleneck drop in their peak MuQ: 0.0120.012, CLAP: 0.0070.007 vs. MuQ: 0.0860.086, CLAP: 0.0590.059 for the bottleneck layers.

Now, to investigate the impact of diffusion model layers on perceptual properties of the generated audio, we perform a second experiment where we set to 00 outputs of transformer blocks, while measuring:

  • audio quality using the mean of the Audiobox Aesthetics [1] scores, evaluating Content Enjoyment, Content Usefulness, Production Complexity, and Production Quality,
  • perceptual distance to original generation (without block ablation) using LPAPS,
  • distributional distance of generations to FMA-Pop [2] music tracks with Fréchet Audio Distance (FAD).

Results presented in Figure 28 (Appendix P) show that Diffusion Transformers (Stable Audio Open, ACE-Step) contain blocks outside the identified bottleneck that are essential for audio quality, which, when disabled:

  1. Lower significantly aesthetics (ACE-Step: L1 -2.1 and L24 -2.9; Stable Audio Open: L4 -2.7 and L24 -1.4).
  2. Increase distributional distance to real, good-sounding tracks (ACE-Step: L1 +6.4 and L24 +10.8; Stable Audio Open: L4 +9.8 and L24 +4.4).
  3. Lead to perceptually (LPAPS) most distinct tracks from the original generations (ACE-Step L1 5.9 and L24 5.2; Stable Audio Open L4 5.8 and L24 4.5), whereas, for reference, LPAPS between two generations with these models from different seeds is 5.95.9 and 4.64.6. Interestingly, for these two models, ablating layers within the bottleneck does not significantly degrade audio quality, indicating a clear separation of perceptual and semantic roles across layers of audio diffusion transformers.

In AudioLDM2, a U-Net architecture, we also observe the impact of the outermost layers on perceptual audio attributes, especially in the later decoder stages (layers 53–64): up to a 0.31 loss in aesthetics and a 0.8 increase in FAD. However, in this model, the bottleneck layers also significantly reduce aesthetics, indicating that the bottleneck carries both semantic and quality signals.

[1] Tjandra, Andros, et al. "Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound." arXiv:2502.05139 (2025).

[2] Gui, Azalea, et al. "Adapting frechet audio distance for generative music evaluation." ICASSP 2024.

Dear Reviewer,

As the discussion period ends in the next 40 hours (November 18 AoE), we would like to know whether our new experiments have addressed your concerns. We remain available for any further questions or clarifications.

Reviewer_e5pTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies fine-grained continuous control of high-level musical concepts such as vocals, tempo, mood, instruments, and genres in text-to-music diffusion models. It uses activation patching across counterfactual prompt pairs to localize a small semantic-bottleneck set of cross-attention layers shared across three architectures, then benchmarks activation-steering methods restricted to those layers against prompt-, score-, and weight-space baselines under an alignment-preservation protocol plus a listening study. The authors report that localized activation steering tops the ranking on Ace-Step, beats global steering and other paradigms, and is corroborated by a 1,279-rating human study and multi-concept steering.

Critical strengths
  1. The finding that a small shared set of cross-attention layers governs high-level musical concepts across is insteresting.
  2. The evaluation is broad for the niche.
Critical weaknesses
  1. The headline gains are thin relative to reported variance, since several best-versus-second distinctions sit within overlapping standard errors and the full benchmark table is shown for only one model. 2.The semantic-bottleneck claim rests on a hand-set threshold applied to a metric whose robustness is not stress-tested, with no sensitivity analysis or statistical test to rule out a threshold artifact.
  2. Generality is overstated, because the main benchmark, multi-concept experiments, and human study run almost entirely on Ace-Step while the other two architectures appear mainly in the localization diagnostic.
Questions for authors
  1. How sensitive are the identified layers and all downstream results to the threshold and the choice of similarity probe, and do they remain stable under reseeding and alternative counterfactual sets?
  2. Given the overlapping standard errors, can you provide paired significance tests showing localized steering reliably beats the strongest baselines rather than sitting within noise?
5 comments

We thank the Reviewer for all the questions and concerns raised, which enabled us to improve our submission. Below, please find our responses.

W1(1)+Q2. Overlapping standard errors, paired significance tests.

Thank you for raising this valid concern. We first clarify how steering effectiveness and its standard error are computed in our benchmark. After calibrating steering strengths for methods, we steer 99 concepts in 22 directions with 1515 strengths per direction, using the same 100100 prompts. For each method, this produces a single AUC value for each (concept, direction, prompt) combination. The final scores (AUCs in Table 1) are averages over all these combinations.

In the initial submission, we first averaged over prompts and directions, and then computed standard errors across the 99 per-concept means. The reported ± therefore mostly reflects how much steering difficulty varies between concepts (e.g., for SAE, MuQ AUC is 0.222 on jazz but only 0.031 on vocal gender). To better measure methods' uncertainty, we now treat the prompts as the sample. For each prompt, we average its AUC over the 18 concept–direction configurations, and compute the standard error across the 100 prompts. As shown in the Table below, the standard error is 0.002–0.005, 2–5× smaller than in the original submission. The results suggest that on MuQ, all three localized activation steering methods rank above all baselines, while on CLAP, localized activation steering performs on par with Concept Sliders.

MethodAUC MuQ ↑AUC CLAP ↑
PCI0.084 ± .0040.049 ± .002
Text Embeddings0.014 ± .0030.011 ± .002
Token Embeddings0.035 ± .0030.024 ± .002
FreeSliders0.079 ± .0040.049 ± .003
Concept Sliders0.086 ± .0040.067 ± .003
AUSteer0.081 ± .0040.058 ± .002
CAA0.075 ± .0040.046 ± .002
AUSteer (loc.)0.096 ± .0040.063 ± .003
CAA (loc.)0.104 ± .0050.065 ± .003
SAE (loc.)0.118 ± .0050.070 ± .003

Following the Reviewer's suggestion, below we also compare the methods pairwise - each localized method with each baseline on the same 100 prompts. For every prompt, we take the difference between the two methods' AUCs and perform a paired t-test. This comparison cancels out the prompt difficulty shared by both methods, allowing it to detect a consistent difference between them even when the standard errors overlap. We call a difference significant if p < 0.05, and not significant (n.s.) otherwise. Because each test has a 5% chance of a false positive, running 21 paired tests per metric (3 localized methods × 7 global baselines) makes it likely that some comparisons come out significant by chance. We therefore apply the Holm correction [1], which adjusts the p-values so that the probability of even a single false positive across all 21 tests remains below 5%.

We report comparisons with the strongest baselines (Concept Sliders, PCI and AUSteer) with AUC difference as % of the baseline's AUC and Holm-corrected p-values:

Relative AUC gainMuQCLAP
vs. Concept Sliders
SAE (loc.)+37% (p < 0.001)+4% (p = 0.49, n.s.)
CAA (loc.)+21% (p < 0.001)−4% (p = 0.49, n.s.)
AUSteer (loc.)+11% (p = 0.002)−7% (p = 0.08, n.s.)
vs. PCI
SAE (loc.)+41% (p < 0.001)+44% (p < 0.001)
CAA (loc.)+24% (p < 0.001)+34% (p < 0.001)
AUSteer (loc.)+14% (p = 0.002)+29% (p < 0.001)
vs. AUSteer
SAE (loc.)+46% (p < 0.001)+20% (p < 0.001)
CAA (loc.)+29% (p < 0.001)+12% (p < 0.001)
AUSteer (loc.)+19% (p < 0.001)+8% (p = 0.002)

Compared with the remaining baselines (FreeSliders, CAA, Token and Text Embeddings), all three localized methods are significantly better on both metrics (all p < 0.001).

Combined, the two provided experiments above show that:

  1. MuQ: all three localized methods (SAE, CAA, AUSteer) significantly outperform every baseline.
  2. CLAP: localized steering is on par with the strongest baseline, Concept Sliders, and significantly outperforms all other baselines.

In the updated manuscript, we highlighted these results in the Introduction, Section 5.3, and Conclusions. For completeness, in the paper (Tables 3 and 39), we also added standard errors to the multi-concept steering results.

[1] Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70.

W1(2)+Q1. Semantic bottleneck and downstream results rely on thresholding.

Yes, we hand-set the threshold τ=0.10\tau=0.10 for consistency across experiments, after inspecting the models' impact profiles (Appendix F.1). Our claim, however, does not rest on τ\tau, as the localization ranks the layers by I(l)I(l), and τ\tau only decides how many top-ranked layers to keep.

To address this concern, we first test the stability of the top-2 layers of ACE-Step ({7,8}):

  • Similarity probes: computing aggregated impact I(l)I(l) only with MuQ-T or CLAP selects the same top-2 set.
  • Reseeding: when generations are performed from 8 independent initial noise seeds, the same top-2 set is chosen 8/8 times.
  • Choice of prompts: resampling each concept's counterfactual prompts with replacement gives the same top-2 set in 10,000/10,000 draws.
  • Choice of concepts: computing the aggregated I(l)I(l) on every subset of concepts gives the same top-2 layers in: 110/120 3-concept subsets, 207/210 4-concept subsets, 251/252 5-concept subsets, and always the same top-2 layer set 6 or more concept subsets.

Additionally, we evaluate the sensitivity of steering performance to the threshold using CAA across 4 concepts (piano, tempo, mood, vocal gender). Below, we vary τ\tau and report the resulting MuQ AUC and CLAP AUC. For reference, we also report steering performance on three random pairs of adjacent layers ({4,5}, {15,16}, {20,21}).

τ\tau rangeselected layersMuQ AUCCLAP AUC
(0.29, 0.49]{8}0.0600.054
(0.094, 0.29]{7,8}0.0700.062
(0.035, 0.094]{6,7,8}0.0690.060
(0.027, 0.035]{6,7,8,14,15}0.0660.060
0all 240.0490.053
random pairs{4,5} / {15,16} / {20,21}0.020 / 0.002 / 0.0020.020 / 0.004 / −0.001

The experiment shows that choosing a single, most impactful layer ({8}) is suboptimal, resulting in a 13–14% loss in AUC compared to {7,8}. Notably, once both bottleneck layers are covered by the threshold, its exact value barely matters, as every layer set selected by τ∈(0.027,0.29]\tau \in (0.027, 0.29] drops the AUC by at most 5%. We observe that adding layers above the bottleneck does not benefit steering, and the all-layer variant lowers the AUC by 15–30%. Finally, three randomly chosen pairs of adjacent layers achieve less than a third of the bottleneck's AUC, suggesting that steering quality depends on where we steer rather than on the exact threshold value.

In the updated version of our submission, we describe these experiments in Appendix F.2 (layer stability) and Appendix L.2 (sensitivity to the threshold)

W2. A full benchmark is shown for only one model.

We are thankful for this suggestion. As explained in the submission, for our benchmark, we have selected Ace-Step as the most capable model, providing the highest-quality sound and the best alignment with the prompt. However, following the reviewer's suggestion, we ran additional experiments with a benchmark on the other two architectures: AudioLDM2 (U-Net) and Stable Audio Open (DiT). Below, we compare the methods across models on 44 concepts: piano, tempo, mood, and vocal gender, with steering in both directions. For each group of baseline methods, we choose the best-performing representatives: PCI (prompt space), FreeSliders (score space), and Concept Sliders (weight space). For activation steering, we evaluate CAA and AUSteer.

Table. Steering Effectiveness (AUC ↑) with MuQ and CLAP.

methodAudioLDM2, AUC MuQAudioLDM2, AUC CLAPStable Audio, AUC MuQStable Audio, AUC CLAP
PCI (all)0.0078±0.00070.0035±0.00050.063±0.0060.030±0.002
PCI (loc.)0.0055±0.00060.0024±0.00040.037±0.0040.018±0.002
FreeSliders (all)0.0088±0.00080.0047±0.00060.047±0.0050.032±0.003
FreeSliders (loc.)0.0086±0.00080.0049±0.00060.036±0.0050.025±0.003
Concept Sliders (all)0.0048±0.00080.0058±0.00070.021±0.0030.024±0.003
Concept Sliders (loc.)0.0038±0.00080.0036±0.00070.015±0.0040.027±0.003
CAA (all)0.0062±0.00060.0062±0.00060.047±0.0050.031±0.003
AUSteer (all)0.0082±0.00080.0092±0.00060.048±0.0040.025±0.003
-
CAA (loc.)0.0093±0.00070.0063±0.00050.058±0.0050.037±0.003
AUSteer (loc.)0.0109±0.00080.0079±0.00060.063±0.0050.036±0.003

Table. Smoothness (↓) with MuQ and CLAP.

methodAudioLDM2, Smoothness MuQAudioLDM2, Smoothness CLAPStable Audio, Smoothness MuQStable Audio, Smoothness CLAP
PCI (all)0.776±0.7170.998±0.77711.872±11.6480.376±0.187
PCI (loc.)0.658±0.5923.405±3.2124.686±4.5000.375±0.162
FreeSliders (all)0.047±0.0160.085±0.0160.115±0.0340.088±0.011
FreeSliders (loc.)0.052±0.0140.075±0.0110.110±0.0230.119±0.016
Concept Sliders (all)0.083±0.0340.039±0.0080.159±0.0390.139±0.036
Concept Sliders (loc.)0.124±0.0680.062±0.0070.224±0.0600.103±0.014
CAA (all)0.048±0.0180.043±0.0160.118±0.0370.138±0.063
AUSteer (all)0.036±0.0090.037±0.0160.092±0.0230.098±0.013
-
CAA (loc.)0.038±0.0110.043±0.0160.077±0.0230.147±0.092
AUSteer (loc.)0.031±0.0060.029±0.0050.068±0.0110.216±0.160

We observe that our original benchmark results on ACE-Step mostly transfer to AudioLDM2 (ALDM2) and Stable Audio Open (SAO): localized activation steering methods (CAA, AUSteer) occupy first (ALDM2 MuQ, SAO MuQ, SAO CLAP) or second (ALDM2 CLAP, SAO CLAP) spots in rankings. Notably, on AudioLDM2 (CLAP), the best-performing method is global AUSteer, and on Stable Audio Open (MuQ), PCI performs best, on par with localized AUSteer.

Our setup, together with these results, is provided in Appendix M.4 of the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.