acceptodds
Under review as a conference paper at ICLR 2027

MusicSTAR: Structure-Aware Temporal Attention Routing for Song Generation from a Single Long Caption

Abstract

Controllable song generation requires specifying not only a song's overall characteristics, but also how they evolve over time. A single long caption offers a natural way to express this evolution by organizing global characteristics and ordered section-level descriptions into one structured temporal specification. Yet such a description does not by itself ensure reliable temporal control: during autoregressive generation, the model can access conditions from different sections, leading to temporal condition misrouting. Motivated by this observation, we propose MusicSTAR, a structure-aware framework for temporal attention routing from a single long caption. MusicSTAR factorizes section-level conditions and dynamically routes attention to the relevant section while preserving access to acoustic history, with a temporal scheduler adaptively controlling transitions between successive conditions without explicit timestamps or durations. We further introduce local-to-global preference optimization, progressing from section-level controllability to whole-song musicality. Experiments show that MusicSTAR substantially improves fine-grained temporal instruction following while maintaining strong overall song quality, demonstrating that effective temporal control depends not only on what is specified, but also on how that information is accessed during generation. Our demo are available at https://anonymous.4open.science/w/ICLR_30705_demopage/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Song Generation, Controllable Generation, Temporal Attention Routing, Long-Context Conditioning, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nzv3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper propose a problem: Using a long caption to controllable song generation. Current method doesn't follow the tempora; control. They proposed MusicSTAR, using attention routing and ocal-to-global preference optimization to ensure section-level controllability and whole-song musicality.

Critical strengths
  1. The framing is clean, and the solution is convincing.
  2. The demo results sound good.
  3. The paper provides ablation studies for the proposed method in Tables 4 and 5.
Critical weaknesses

1. Many symbols in the method section are undefined or underspecified, which makes the section hard to follow. For example:

  • Undefined symbols. sis_i and eie_i in Eq. 1 are never defined, and neither is qq in Eq. 6.
  • Unclear indexing and notation. tt is called a "step," a "frame," and an "audio query position" at different points, and the paper never says whether these refer to the same index. The relation between LL and $ l_i $ is not stated. The ordering and positional encoding used by Reassemble(⋅)\mathrm{Reassemble}(\cdot) in Eq. 5 are also unspecified, as is where MpreM^{\mathrm{pre}} is applied.
  • Training details. The boundary label, the boundary loss, and the full training objective are never written out. The paper also does not explain how iti_t is set during training.
  • Hyperparameters and settings. The values of τ\tau, Lmin⁡L_{\min}, λt\lambda_t, pexactp_{\mathrm{exact}}, and β\beta are not reported, and it is unclear how λt\lambda_t changes with tt. The reference model in Eq. 14 is not specified, nor are the layers and heads modified by Eqs. 9-10.

A notation table and a hyperparameter table would make the paper much easier to read and reproduce.

2. The related work is thin (about 20 references) and omits several closely related lines of work. In particular:

  • Attention steering (closest to Temporal Attention Redistribution, §3.3.2): PASTA (Zhang et al., ICLR 2024) re-weights attention toward user-specified spans after the fact and then renormalizes, which is very close to Eqs. 9-10. Prompt-to-Prompt (Hertz et al., ICLR 2023) controls generation by editing cross-attention maps.
  • Sequential or time-varying prompts in video generation (the same problem of routing ordered text conditions over time): Phenaki (Villegas et al., ICLR 2023), DiTCtrl (Cai et al., CVPR 2025), and Mind the Time (Wu et al., CVPR 2025).
  • Monotonic alignment (closest to the section pointer, §3.3.1): monotonic attention (Raffel et al., ICML 2017) and MoChA (Chiu & Raffel, ICLR 2018).
  • Fine-grained preference optimization (closest to Section-DPO, §3.5): Fine-Grained RLHF (Wu et al., NeurIPS 2023), Token-level DPO (Zeng et al., ICML 2024), and Step-DPO (Lai et al., 2024). The length-normalized objective in Eq. 13 also resembles SimPO (Meng et al., NeurIPS 2024).
  • Temporally controlled music generation beyond Music ControlNet and Mustango: MuseControlLite (Tsai et al., ICML 2025) and long-form structured generation with Stable Audio (Evans et al., ISMIR 2024).
  • Foundational music and song generation: Jukebox (Dhariwal et al., 2020), MusicLM (Agostinelli et al., 2023), MusicGen (Copet et al., NeurIPS 2023), and SongGen (Liu et al., ICML 2025).
  • Evaluation resources: MuLan (Huang et al., ISMIR 2022) is used both as a reward and as an evaluation metric but is never cited. SongEval (Yao et al., 2025), a benchmark for song aesthetics, would provide a useful independent check, since SongBench is also used as the Song-DPO reward.

3. The evaluation is incomplete. Nothing in the paper measures boundary accuracy or whether the generated sections match the requested ones in number, order, and duration. For example, the authors should report whether the pointer reaches the final section before the end of the sequence. In addition, users cannot specify section durations, so "temporal control" here covers ordering only. The paper should state this clearly as a limitation.

Questions for authors
  1. Test set and baselines.

    • How large is the test set, and where does it come from?
    • How were the section captions converted into the input format each baseline expects?
    • Does every baseline receive the same information?
  2. Structural metrics. Could you report the following?

    • Boundary F-measure (e.g., at ±0.5 s and ±3 s tolerances), along with how the generated boundaries are obtained (from the pointer or from an external structure analyzer).
    • Section count accuracy and order fidelity, e.g., the edit distance between the requested and generated section sequences.
    • Section completion rate, i.e., the fraction of samples in which every requested section appears before the end of the sequence.

    Also, how are failure cases, such as a missing section or a pointer that stops early, counted in these metrics?

Reviewer_hL88OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MusicSTAR generates songs from lyrics and a structured long caption containing global and ordered section-level descriptions. It encodes section conditions separately, uses a learned boundary predictor to advance through sections, and redistributes attention toward the active section while preserving the attention mass assigned to audio history. Evaluation compares overall song quality, section-level attribute adherence, and human ratings against existing systems.

Critical strengths
  • The routing mechanism is well motivated. Separating section conditions before routing avoids mixing information that the model later needs to distinguish. The normalization preserves attention to audio history within each attention row while reallocating attention among conditions.

  • The system results are good. Overall automatic quality scores remain competitive with strong baselines, and Table 3 reports higher human ratings across six dimensions. The comparison with Muse also uses the same section-level attribute descriptions in each model’s input format. Results suggest that the control mechanism can be useful without an loss of song quality.

Critical weaknesses
  • The main evaluation is closed with the signals used for preference training, and the validation is not described well. Section-DPO selects pairs using extracted-attribute MuLan similarity, while Song-DPO uses SongBench.

  • What is the number of evaluated songs or annotators, ratings per sample? The held-out set’s size and construction are also unspecified. It is unclear how generated sections are matched to the sections when computing Table 2.

  • The ablations do not isolate the routing contribution on temporal control. w/o Redistribution removes both redistribution and the section-wise conditioner. Table 4 reports only overall quality metrics, without the section-level adherence scores. Authors don't know if better song-quality than a timestamp-driven variant can establish more reliable section transitions or attribute placement.

Questions for authors
  • With the conditioner and scheduler held fixed, how does the proposed redistribution compare with model w/o redistribution and with applying the same logit penalty?

  • About reward hacking. Can the gains be confirmed by section-level human judgments or an evaluator that was not used to select preference pairs?

  • How is the boundary head trained, and does training use reference section or predicted pointers?

Reviewer_a2g5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MusicSTAR, a structure-aware framework for temporally controllable song generation from a single long caption. The framework introduces a Section-wise Conditioner to isolate section-level representations, a Temporal Attention Scheduler to predict section transitions, and Temporal Attention Redistribution to guide conditioning while preserving attention to acoustic history. The authors further propose a local-to-global preference optimization strategy to improve section-level controllability and overall musicality. Experiments demonstrate improvements in fine-grained instruction following and competitive song generation quality.

Critical strengths

Meaningful research problem. The paper addresses an important challenge in full-song generation: controlling the temporal evolution of musical attributes while maintaining overall musical coherence. Unlike previous approaches, the proposed formulation does not require timestamps or section durations at inference time, offering an interesting direction for more flexible and adaptive song generation.

Coherent architectural design and promising experimental results. The proposed components have clear and complementary roles in temporal conditioning, forming a coherent and technically reasonable framework. The method is evaluated using both objective and subjective metrics, covering overall musical quality and fine-grained instruction following. The experiments demonstrate promising improvements in controllability while maintaining competitive overall song quality.

Critical weaknesses

Limited justification for the single-long-caption formulation.

Although the paper emphasizes the advantages of a single long caption, the proposed representation relies on predefined attributes, explicit section structures, and ordered section-level descriptions. It therefore resembles a concatenation of structured segment-level captions rather than unrestricted natural-language conditioning. The paper does not sufficiently demonstrate its advantages over existing section-conditioned approaches, particularly in handling free-form descriptions without explicit section labels or relative instructions that require reasoning across multiple sections.

Insufficient justification for section-wise conditioning isolation.

The central motivation of this paper is based on the observation that baseline attention is distributed across multiple sections (Figure 1). Accordingly, the proposed architecture encourages each section to primarily attend to its own description and global conditions while suppressing interactions with other sections. Both the motivation and architectural design therefore rely on a strong assumption that section-wise conditioning isolation is beneficial. However, this assumption is not necessarily desirable for music generation, where cross-section dependencies are important for developing coherent musical ideas. Diffuse attention across sections does not necessarily indicate incorrect conditioning, as attending to other sections may support global coherence, while the diagonal attention pattern of MusicSTAR is largely an expected consequence of its routing mechanism. Furthermore, having the entire long caption accessible during generation offers opportunities to model cross-section dependencies, but this potential is not explored in the architectural design or experimental evaluation. Although Appendix D argues that acoustic history preserves cross-section coherence, it may not fully substitute for interpreting explicit relationships between section descriptions. Overall, the paper lacks convincing evidence that section-wise conditioning isolation is preferable to more flexible cross-section interactions in music generation.

Insufficient experimental validation.

The ablation studies do not fully isolate the contributions of individual architectural components in Table 4. Moreover, they lack the fine-grained temporal controllability metrics used in the main evaluation, making it difficult to assess each component's effectiveness in temporal instruction following. Additionally, the overlap between the extracted-attribute evaluation protocol (MuLan protocols) and the automatic signals used to construct Section-DPO preferences (MuLan similarity) raises concerns about the independence of the controllability evaluation.

Questions for authors
  1. For the Temporal Attention Redistribution, how is the penalty parameter (\lambda_t) in Eq. (9) determined? Is it learned or predefined? This is an important design detail that is not clearly explained in the paper. A sensitivity analysis would help clarify the trade-off between enforcing section-specific conditioning and preserving useful cross-section information.

  2. The Temporal Attention Scheduler requires further clarification and experimental validation. Could the authors specify the training objective for boundary prediction, the values of (\tau) and (L_{\min}), and how boundary-class imbalance is handled? Evaluating boundary detection accuracy, transition timing errors, and robustness as the number of sections increases would further demonstrate its effectiveness.

  3. The current ablations do not fully isolate the contributions of individual components; for example, removing Redistribution also removes the Conditioner. Could the authors provide more independent comparisons and report the eight-dimensional controllability scores for Table 4, as in Table 5? Although SongBench assesses fine-grained musical quality, it does not directly measure whether attributes are realized in their intended sections.

  4. The Extracted evaluation protocol in Table 2 appears to overlap with the automatic attribute extraction and similarity measurements used to construct Section-DPO preference pairs. Could the authors clarify the independence of the evaluation and whether alternative methods support the same conclusions?

  5. What instructions were given to listeners when evaluating Song Structure Clarity? In particular, were they asked to assess the naturalness of transitions between sections? Some transitions, such as the verse-to-pre-chorus transition in the demo, sounded less natural to me. Since transition handling is closely related to the proposed scheduler and routing mechanism, a more targeted evaluation would be helpful.

  6. Could the authors clarify what "prompt audio" refers to in Section 3.3.2 (line 260) and define the metric abbreviations in Table 1, particularly CE, CU, PC, and PQ?

Comments on the paper

1 comment

Dear reviewers,

The authors have submitted their rebuttals. Please take a look and interact with the authors if necessary.

Thanks a lot.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.