- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DID, a two-dimensional autoregressive framework for symbolic music generation that explicitly models musical structure across bars and within-bar events. The proposed Decoder-in-Decoder architecture consists of a time decoder that maintains compressed global history across bars and a note decoder that autoregressively generates events within each bar, leveraging both global history and detailed information from the preceding bar. For conditional generation, the authors further introduce Harmonic Plan events, which provide intermediate harmonic guidance through bass pitch classes and relative pitch-class sets.
The proposed method is evaluated on three tasks: music continuation, chord-conditioned piano generation, and accompaniment generation, using the POP909 and POP1k7 datasets. Experimental results demonstrate competitive performance against existing autoregressive and diffusion-based methods, with improvements in several harmonic and rhythmic consistency metrics. Human evaluations further indicate improvements in perceived musical quality, while ablation studies support the effectiveness of the Harmonic Plan and global conditioning components.
Critical strengths
-
Well-motivated architecture for structured music generation. The proposed Decoder-in-Decoder architecture provides a reasonable approach to modeling musical dependencies at different temporal scales. By combining compressed global history with detailed information from the preceding bar, DID balances long-range context and local continuity while retaining the flexibility of autoregressive generation. This design is meaningful for structured symbolic music generation.
-
Promising experimental results across multiple generation tasks. DID is evaluated on three generation tasks using two public datasets, with comparisons against representative autoregressive and diffusion-based methods. The results show improvements in several harmonic and rhythmic consistency metrics, supported by favorable human evaluations. The ablation studies further demonstrate the contributions of Harmonic Plan and global conditioning, providing additional evidence for the effectiveness of the proposed framework.
Critical weaknesses
-
Limited novelty over existing hierarchical autoregressive approaches. The proposed 2D autoregressive formulation is closely related to existing hierarchical music generation methods, such as Museformer, Bar Transformer, and Nested Music Transformer. Moreover, the nested probability factorization itself can also be expressed using conventional 1D autoregressive modeling. Although DID introduces a different architectural organization, the paper does not sufficiently clarify its fundamental advantages over these approaches, making the methodological novelty somewhat limited.
-
Insufficient ablation studies on the core architecture. While the authors evaluate the contributions of Harmonic Plan and global conditioning, they do not provide sufficient ablations for the central Decoder-in-Decoder architecture. In particular, the individual contributions of the time decoder, compressed global history, and complete local history remain unclear. Without these comparisons, it is difficult to determine whether the reported improvements primarily result from the proposed 2D autoregressive design or other architectural components.
-
Inadequate evaluation of long-term musical structure. Although the paper emphasizes improved cross-bar dependencies and long-term structural coherence, the objective evaluation primarily relies on pitch, harmonic, and rhythmic similarity metrics. These metrics do not directly capture higher-level musical structures, such as long-range repetition, sectional organization, and structural development. Furthermore, the best-matching reference-bar evaluation may reward local pattern repetition without reflecting genuine global coherence. More structure-aware evaluations would provide stronger evidence for the paper's central claims.
Questions for authors
Q1. Novelty of the 2D Autoregressive Formulation. The proposed 2D autoregressive factorization appears mathematically equivalent to a conventional 1D autoregressive factorization with bar-level grouping. Could the authors clarify what fundamentally distinguishes DID from existing hierarchical autoregressive approaches, beyond architectural organization? In particular, what modeling capabilities or advantages arise specifically from the proposed 2D formulation?
Q2. Effectiveness of the Core Decoder-in-Decoder Architecture. The current ablation studies primarily focus on Harmonic Plan and global conditioning, while the contributions of the time decoder and the global/local history design remain unclear. Could the authors provide additional ablations or controlled comparisons, such as removing the time decoder or replacing the dual-history mechanism with a conventional autoregressive context? Such comparisons would help establish whether the improvements are attributable to the proposed architecture.
Q3. Evaluation of Long-Term Musical Structure. The paper emphasizes improved cross-bar dependencies and long-term structural coherence, but the current metrics mainly capture pitch, harmonic, and rhythmic similarity. How do the authors justify these metrics as sufficient evidence for long-term structural modeling? Could structure-aware evaluations, such as self-similarity analysis or measures of long-range repetition and variation, provide more direct support for the central claims?
10 comments
Response to Reviewer 9R59 (part 2 / 3 )
[Weakness 2 / Q2: The contributions of the time decoder, compressed global history, and complete local history remain unclear, making it difficult to attribute improvements to the proposed 2D design.]
Reply: Thank you very much for this valuable and constructive suggestion. We sincerely appreciate your attention to the empirical validation of our core architecture, and agree that more targeted ablations would help clarify the contributions of its individual components.
First, Table 1 already compares DID with representative 1D autoregressive baselines, including CP and NMT. DID achieves higher average continuation scores on both POP909 (89.53, compared with 88.06 for CP and 88.04 for NMT) and POP1k7 (86.79, compared with 84.24 and 85.92, respectively). Fig. 3 further demonstrates more stable rhythmic similarity during long-range continuation.
To more directly examine the contributions of the two historical pathways, we additionally evaluate (1) w/o Global History, which removes the compressed history produced by the Time Decoder while retaining complete Local History ; and (2) w/o Local History, which removes while retaining the compressed Global History (Sec. 3.2, Eqs. (6)–(7)).
Additional Ablation: Global–Local History on POP909 Music Continuation.
| Variant | SimPCE ↑ | SimChr ↑ | SimGrv ↑ | PD ↑ | SimSC ↑ | Avg. ↑ |
|---|---|---|---|---|---|---|
| w/o Global History | 85.53 | 71.29 | 83.49 | 47.91 | 84.21 | 74.48 |
| w/o Local History | 95.73 | 91.63 | 92.85 | 67.98 | 98.14 | 89.27 |
| DID (Full) | 96.37 | 92.10 | 92.49 | 68.42 | 98.29 | 89.53 |
As shown in the ablation results, removing Global History leads to substantial performance degradation across all evaluation metrics, highlighting the importance of the Time Decoder's compressed cross-bar context. Furthermore, DID (Full) achieves better overall performance and outperforms the Global-History-only variant on most metrics, supporting the complementary contribution of complete Local History in preserving fine-grained neighboring-bar information. Together, these results provide empirical support for DID's dual-scale historical modeling design.
We hope that the existing comparisons with 1D baselines and these targeted ablations help clarify the benefits of DID's architectural design. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining concerns.
Response to Reviewer 9R59 (part 3 / 3 )
[Weakness 3 / Q3: Current similarity metrics and best-matching reference-bar evaluation may not adequately capture long-term musical structure, including global coherence, structural repetition, and development.]
Reply: Thank you for your insightful comment regarding long-term structural evaluation. We sincerely appreciate your attention to this important aspect of symbolic music generation.
To further examine long-term musical structure, we provide additional evaluations from two complementary perspectives:
(1) Long-range structural consistency. We have conducted a dedicated long-range continuation experiment in Sec. 4.4 (Fig. 3), comparing DID with CP and NMT on the common subset of POP1k7 test examples extending beyond 80 bars. DID maintains nearly stable rhythmic and chromatic similarity as generation length increases, whereas CP and NMT exhibit more noticeable declines. These results provide evidence of DID's ability to preserve rhythmic and tonal coherence over extended generation.
(2) Musical pattern diversity and development. We additionally introduce N-gram Diversity and Reconstruction Accuracy (ReconAcc) on POP1k7 and POP909. N-gram Diversity measures the proportion of unique pitch-class N-grams, reflecting local pattern diversity and helping identify excessive repetition. ReconAcc measures the similarity between generated and reference pitch-class sequences using normalized edit distance, providing a complementary assessment of sequential consistency.
Additional Evaluation: N-gram Diversity and Reconstruction Accuracy.
| Method | POP1k7 N-gram (%) ↑ | POP1k7 ReconAcc (%) ↑ | POP909 N-gram (%) ↑ | POP909 ReconAcc (%) ↑ |
|---|---|---|---|---|
| DID (Ours) | 76.70 | 35.05 | 73.31 | 35.64 |
| MuseTok | 87.12 | 32.95 | 82.52 | 32.94 |
| CP | 69.01 | 33.64 | 60.64 | 30.28 |
| NB | 70.84 | 34.17 | 54.01 | 29.89 |
| REMI | 73.46 | 34.66 | 65.92 | 29.52 |
DID achieves the highest ReconAcc on both datasets, while maintaining competitive N-gram Diversity. Although MuseTok achieves higher N-gram Diversity, its lower ReconAcc indicates that greater local pattern diversity does not necessarily translate into stronger sequential consistency. DID, in contrast, maintains a favorable balance between local pattern diversity and reference-sequence consistency.
Together, the long-range continuation experiment and these additional metrics provide complementary evidence regarding global musical consistency, local pattern repetition, and the balance between musical variation and structural continuity.
We hope that these additional evaluations help address your concerns. We sincerely appreciate your constructive feedback and would be delighted to provide further explanations or engage in additional discussion.
We sincerely hope that our responses and additional experiments address your concerns. We would be grateful if you could kindly reconsider your score in light of these clarifications. We also sincerely invite you to visit our demo page (https://anonymous.4open.science/w/did-cases-785B/) and listen to the generated music samples. We greatly appreciate your constructive feedback and would be delighted to discuss any remaining questions.
Response to Reviewer 9R59 (part 1 / 3 )
Dear Reviewer 9R59,
Thank you for taking the time to review our paper. We sincerely appreciate your positive feedback on our architecture and experimental results, as well as your insightful questions. We would like to address your concerns as follows:
[Weakness 1 / Q1: The proposed 2D autoregressive formulation resembles existing hierarchical music models, and its nested factorization can be expressed through 1D autoregression. Its advantages over prior approaches remain unclear.]
Reply: Thank you for raising this important concern regarding the novelty of DID. We sincerely appreciate the opportunity to clarify its distinctions from existing hierarchical music models. We acknowledge that Museformer, Bar Transformer, and Nested Music Transformer are valuable contributions to symbolic music generation; however, their core modeling approaches differ fundamentally from DID's 2D autoregressive formulation.
To further clarify this distinction, we consider the attention mask structure. Although these methods make valuable contributions, their main decoding processes operate on 1D token sequences, using 2D token masks of shape [Batch, Sequence]. In contrast, DID explicitly separates the bar and event dimensions, resulting in 3D masks of shape [Batch, Bar Sequence, Event Sequence]. This reflects DID's explicit 2D autoregressive formulation.
Specifically, Museformer introduces bar-level summary tokens to support hierarchical attention but retains a 1D sequence of musical events. Bar Transformer uses note-level and bar-level encoders to learn musical structure, while its arrangement decoder generates music along a single sequence. NMT retains 1D autoregressive generation of compound tokens while introducing causal dependencies among the attributes within each token.
In contrast, the core contribution of DID is an explicit 2D autoregressive decoding framework that aligns each time step with a complete musical bar and generates within-bar events along a separate note dimension.This formulation preserves the 2D bar–event structure of music while combining the structural benefits of non-autoregressive methods with the flexibility of autoregressive generation. Its bar-level alignment enables fine-grained conditional control, while autoregressive decoding supports flexible and continuous music generation. Consequently, DID supports music continuation, chord-conditioned piano generation, and accompaniment generation within a unified framework (Tables 1, 2, and 4), whereas representative 1D autoregressive baselines in our experiments are primarily evaluated on music continuation.
Furthermore, to provide a more comprehensive comparison with existing approaches, we have expanded Appendix D of the revised manuscript.
Regarding the concern that DID's nested factorization can be expressed through 1D autoregression, the actual generation process and modeling architecture are fundamentally different. DID explicitly organizes generation into two levels: the outer autoregressive process advances across complete bars, while the inner process generates the ordered events within each bar, conditioned on its bar-level context. This explicit separation defines the generation order and enables direct bar-level condition alignment, which is not inherently provided by a flattened 1D formulation. Furthermore, as discussed in Sec. 4.4, DID demonstrates better rhythmic consistency and more stable chromatic similarity than conventional 1D autoregressive baselines, particularly in long-range continuation.
We hope these comparisons clarify the distinction of DID from prior work and address your concerns. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining questions.