acceptodds
Under review as a conference paper at ICLR 2027

Styles Are Islands: The Discrete Geography of Music Generation and Why Steering Is One-Way

Abstract

Text-to-music models now write usable music from a sentence. Steering functions give such models a second control. A direction is added to a hidden state, or an affine map is applied to it, and an attribute of the music is dialed up or down without touching the prompt. This is the control a user needs when the request is “more of this” rather than a sentence, or when the model takes no prompt at all. The family has worked well for removing attributes, and it carries an unstated assumption: the control that removes a style also adds it. We test this assumption on a text-to-music model with an automatic listener as the judge, and it fails. The same steering vector removes a style almost completely and adds it hardly at all. Removal recovers most of the distance between the two styles, addition about a tenth, and the two dose curves enclose a hysteresis loop. The natural repair is an affine map that matches the shape of the target class. It is more faithful to the class statistics and fails in the same way, as do re-timed pushes and pushes optimized directly against the listener. We resolve the conflict by asking what a style is in state space. A style is not a region but a thin set of trajectories that occupies no volume. Leaving such a set is certain under any push, and landing on it is impossible for every steering function, affine maps included. Transplanting a real state of the target style reaches the style, whereas the same state restricted to its most discriminating directions does not. The diagnosis yields the method. Transport the state to the target class with the optimal affine map, then project it onto the nearest real trajectory of that class and follow it. Neither step suffices alone. Together they close the loop on electronic texture at both model sizes and part of the way on jazz, with the music label preserved, where the steering vector reaches a tenth of the way.

Primary area:
generative models
Keywords:
generative model, music generation, mechanical interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pTBiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper challenges the assumption that steering functions are reversible and tests this assumption on MusicGen. The findings are that steering can remove a style but is unable to add it, quantifying this gap as a hysteresis loop. The authors argue this happens because a style is not a full-rank volume of the latent space, but instead a subset of it, composed of many thin real trajectories, so thin that it is impossible to land on them. From this result, the authors propose a “transport then project” technique, where at a fixed layer the hidden state is transported to a target class, and then replaced by the nearest trajectory in that class, which then continues with generation. This method reaches the target style on electronic music at both small and medium MusicGen sizes, and partly on jazz.

Critical strengths
  1. The transplant experiment shows that somewhere exists a clear threshold from which a target style can be reached from raw mixing, and furthermore it shows that many of the dimensions are needed for that to succeed.
Critical weaknesses
  1. I think what the authors propose can't be called steering. The hidden state at the selected layer is overwritten with the closest state in the transported class, for each step onward. In this way one track is being effectively replaced by another. The paper does acknowledge that none of the methods proposed preserve the source and that the similarity between outputs and the bank of clips was not measured. Listening to the demo leads me to this conclusion: the outputs are indeed nicer than the affine ones by a long margin, but they have little to do with the original tracks.
  2. The experiments are too narrow for the given claims. The title and abstract make general claims about steering functions and styles, but everything is tested on a single model family, over a single layer of a model, and on three attributes, with only automatic listeners. Tests on other models, for example a diffusion based one (such as Stable Audio Open), would be necessary to support any claim of generalizability.
  3. Lemma 2 doesn’t work. Taken literally it means that any change leaves a class and none can reach one. Furthermore, the ϵ\epsilon-tube version of the argument undoes its first half, as in that manner less than ϵ\epsilon pushes will stay inside the tube, so leaving is no longer a certainty. Finally, the empirical proof that classes/styles live in a thin set does survive (as hard NN beats soft 8-NN), but the proof that they are measure zero is little, as the alpha=0.5 mixing reaches ~70% for target classes.
Questions for authors
  1. Can the authors provide the original clips used for the bank? Or at the very least provide similarity metrics (perceptual ones) such as CLAP or MERT, between the bank and the outputs?
  2. Do the findings hold on other music generation models?
  3. Can the authors provide results from other layers and other attributes?
Reviewer_6jXCOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies inference-time steering of MusicGen (300M and 1.5B) at one middle layer, using a two-arm protocol in which the same steering function is applied to remove a style and to add it, scored by an external automatic listener. The central observation is a strong asymmetry: removal recovers most of the class gap, addition about a tenth, summarized as a hysteresis index. The authors argue that neither class geometry nor the full affine family can explain this, propose that a style is a measure-zero set of real trajectories in state space, support this with transplant experiments, and derive a training-free method that transports the state with the Bures–Wasserstein map and then projects it onto the nearest real target-class trajectory. The method reaches the target on electronic texture at both model sizes and partially on jazz, with music probability preserved.

Critical strengths

A valuable and under-explored question with a fresh perspective. The addition/removal asymmetry in music steering has been noticed but not isolated; the two-arm protocol with fixed direction, operator, and schedule makes it measurable, and the “style as a thin set of trajectories” view is a genuinely different lens from linear steering.

Thorough ablations and a strong result on electronic texture. The ladder of baselines (translation, affine transport, listener-optimized push) and the component ablations (transport only, projection only, nearest-state vs. trajectory) are well designed, and on electronic the proposed method clearly outperforms every affine approach while keeping music probability near that of real clips.

Critical weaknesses

The core theoretical claim is not rigorously established. Modeling a style as a measure-zero set is an idealization with limited empirical support; Lemma 2 applies equally to the contrast class, and the paper’s own mixing experiment (an affine mix at α ≥ 0.5 reaches the style) suggests a neighborhood or basin rather than a null set. The title and the claim that affine steering cannot add a style are broader than the evidence supports.

The method may be retrieval rather than editing, and content preservation is unshown. After the committing steps, the algorithm replays another clip’s hidden states at every step. Without a measurement of similarity to the chosen bank clip, the results are consistent with reproducing that clip. The paper does not show preservation of the source’s melody, rhythm, or structure, and the requirement to start at step one with a pre-stored bank limits interactive use.

Evaluation and generalizability are limited. Verdicts rest on automatic listeners (PANNs, Demucs, MS-CLAP) with no human listening study; experiments cover one model family, one layer per model, and three attributes. Performance is inconsistent across styles (jazz addition at 15% on the small model; drums at 9% on medium) without a mechanistic account of why, and comparisons to nonlinear steering, retrieval-augmented generation, and modern music-editing methods are missing.

Questions for authors

How similar is the projected output to the chosen bank clip (token overlap, embedding distance)? If high, how does the method differ from retrieval?

How do you reconcile Lemma 2 with the α-mixing result, and what does the measure-zero claim add over an off-manifold explanation?

Why does the method succeed on electronic but not on jazz or drums? Is this a property of the evaluators or of the states?

What is the across-seed variance of Table 2, and the sensitivity to k₀ and w?

Reviewer_etpcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether steering functions on a text-to-music model are reversible: does the direction that removes a style also add it? Using MusicGen (300M and 1.5B) at one layer, a two-arm protocol with direction, operator and step held fixed, and automatic listeners (PANNs, MS-CLAP, Demucs), the answer is no. Across six (model, attribute) cells, removal recovers 59 to 101% of the class gap while addition reaches 6 to 16% on genres and below zero on drums. The gap is quantified as a hysteresis index H, the area between the two dose curves, ranging from 0.24 to 0.72.

The paper argues the cause is not class geometry. Observation 1 shows a translation is symmetric in expected log-odds, and the thresholded reader predicts the wrong sign on the measured covariances. Lemma 2 instead posits that a style is a Lebesgue-null set of trajectories inside its cloud, so leaving it is certain under any push and landing on it has probability zero for the whole affine family. Supporting evidence: transplanting a whole real target state reaches 83% of the class level, while the same state restricted to a line, a plane or its 32 most discriminating directions gives 3 to 5%, with a threshold around mixing fraction 0.25 to 0.5.

The proposed method follows: transport the state with the Bures-Wasserstein optimal affine map, then project onto the nearest real trajectory from a bank of target-class states and follow it. On the addition arm over 40 held-out prompts it reaches 116% and 100% on the two electronic cells and 15% and 71% on jazz, against 2 to 12% for the steering vector, with the music label preserved. Table 1 also contributes a taxonomy of existing steering methods by Rayleigh pair and operator.

Critical strengths
  1. The question is sharp and the protocol is right. Reversibility is an unstated assumption across the whole steering literature, and testing it with direction, operator and step fixed across both arms is a contribution the field has not made. The hysteresis index is a clean way to report it. Table 1's classification of existing methods by what their operator does to the class covariance is independently useful.

  2. The transplant experiment (Figure 3b) is the best piece of evidence in the paper. Whole real state at 83%, the same state on a line, a plane or 32 directions at 3 to 5%, with a sharp threshold in mixing fraction, cleanly separates "style is a direction" from "style is a configuration of the whole vector". This is a well-designed causal probe, not a correlational observation, and it does real work.

  3. Reproducibility and candour are both unusually high: pinned checkpoint commit hashes, byte-identical generation verified across nodes, per-clip scores released, CLAP excluded because its jazz AUC is below chance, drums flagged as provisional, and a limitations section that states plainly that little of the source survives and that the method must start at step one.

Critical weaknesses
  1. The headline method may be retrieval, and the paper has not run the control that would settle it. Algorithm 1 overwrites the layer-L state with a bank state verbatim at every step after commitment, so from step k0 the edited layer replays a real target-class clip. The paper states that tempo agreement with the unedited source is 32% for the projection against 18% between unrelated prompts, which is barely above floor, and then says "we have not yet measured how close the output is to the chosen bank clip". That is the one measurement the contribution depends on. If the output is close to the bank clip, the 116% score means the system played back a real electronic clip, and this is audio retrieval rather than steering. The control is cheap: embedding cosine or chroma/MFCC DTW between the output and (a) the chosen bank clip, (b) the source clip, against the unrelated-prompt floor. Scores above 100% (116, 127) sharpen the concern, since 100% is defined as what prompting achieves.

  2. The paper's own Section 3.2 undercuts the asymmetry framing of its abstract and title. Under the symmetric score, the addition arm reaches 44 to 48% and the removal arm 36 to 86%, and the text concedes that "neither arm completes a transfer" and that removal "succeeds" only because leaving is what is scored on that arm. So the honest finding is that a push destroys class membership on both arms and transfers on neither. The abstract still says "removal recovers most of the distance between the two styles, addition about a tenth", which is the scoring convention speaking. Please move the symmetric-score result into the abstract.

  3. Lemma 2 is true but close to vacuous, and the empirics argue against its strict form. Any null set has this property, and real networks are continuous, so a readout cannot be discontinuous at a measure-zero set. The operative claim is the parenthetical one, that an epsilon-neighborhood reader sees addition at O(eps^(p-d)), and that is never quantified: neither d nor eps is estimated. Figure 3b's gradual threshold between alpha 0.25 and 0.5 is evidence for a neighborhood reader with non-trivial tolerance, not for exact membership. I would demote the measure-zero argument and estimate the intrinsic dimension of the trajectory set instead, which turns a tautology into a prediction.

Secondary. Half the attributes tried (piano, major mode, fast tempo) move on neither arm under any method and are excluded in the setup, with no diagnosis of whether this is the evaluator or the states; that materially limits the generality of "steering is one-way". There is no positive control, no attribute where a push does add, so listener insensitivity to additions is not excluded. The motivating application in the introduction, steering a running performance, is exactly what the method cannot do (18% and 14% when started at step 100 or 250). And prompting beats the method on jazz (67 to 119%), so the case rests entirely on prompt-free settings, which are not themselves evaluated.

Questions for authors
  1. How similar is the output audio to the chosen bank clip, relative to the source clip and to an unrelated-prompt floor? Any embedding or DTW similarity would do. This is the question that most affects my assessment.

  2. Why do the projection rows exceed 100% of the class gap (116, 127)? If 100% is what prompting achieves, what makes the edited clips read as more electronic than real electronic prompts?

  3. Can you estimate the intrinsic dimension d of the trajectory set and the tolerance eps, so that the O(eps^(p-d)) prediction becomes testable? Figure 3b's alpha threshold already bounds eps from one side.

  4. Do piano, major mode and fast tempo fail because of their evaluators or their states? Even a quick probe AUC per attribute would tell us whether the exclusion is a listener artifact.

  5. Is there any attribute or model where the addition arm works under a push? A positive control would separate "steering is one-way" from "this listener is insensitive to addition".

  6. "Sigma_1 is estimated at about one sample per dimension": is a sample one clip or one step? With 50 clips at roughly 500 steps the ratio is very different, and the transport map depends on it.

  7. Table 2 says one generation run per row family. How much does the addition arm move across seeds for the projection rows?

Suggestions, not affecting the score: the abstract is dense and the metaphorical register ("the nearest house in the wrong neighborhood") occasionally substitutes for a measurement; Figures 1c, 3c and 3d overlap considerably and could be merged.

Comments on the paper

1 comment

Thank you for the additional experiments which address several of my concerns. However, I remain concerned about whether the proposed method achieves genuine style editing rather than retrieval-based trajectory replacement, particularly given the limited evidence of source-content preservation. The broader theoretical claims may also require further validation. So I keep my score in rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.