acceptodds
Under review as a conference paper at ICLR 2027

DID: 2D Autoregressive Modeling with a Decoder-in-Decoder Architecture for Symbolic Music Generation

Abstract

Symbolic music generation aims to create compositions from discrete musical events. Existing autoregressive methods flatten multi-bar events into a single sequence, losing the explicit 2D structure, while non-autoregressive approaches often rely on fixed grids, limiting flexibility for variable-length generation. In this work, we propose DID, a 2D autoregressive framework that models musical events across the time and note dimensions.This design preserves the two-dimensional organization of music while retaining autoregressive flexibility. DID follows a Decoder-in-Decoder architecture with two decoders: the time decoder updates a running musical history along the time dimension, while the note decoder uses this memory to generate events along the note dimension. To improve harmonic consistency, we introduce harmonic Plan events as intermediate tokens in the note decoder's output sequence, turning the given conditions into a soft harmonic constraint on the notes that follow. Experiments on multiple public datasets show that DID improves harmonic and rhythmic coherence over prior methods, while supporting flexible generation scenarios, including variable-length music continuation, chord-conditioned piano generation, and accompaniment generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Symbolic Music Generation, 2D Autoregressive Modeling, Decoder-in-Decoder

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9R59OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DID, a two-dimensional autoregressive framework for symbolic music generation that explicitly models musical structure across bars and within-bar events. The proposed Decoder-in-Decoder architecture consists of a time decoder that maintains compressed global history across bars and a note decoder that autoregressively generates events within each bar, leveraging both global history and detailed information from the preceding bar. For conditional generation, the authors further introduce Harmonic Plan events, which provide intermediate harmonic guidance through bass pitch classes and relative pitch-class sets.

The proposed method is evaluated on three tasks: music continuation, chord-conditioned piano generation, and accompaniment generation, using the POP909 and POP1k7 datasets. Experimental results demonstrate competitive performance against existing autoregressive and diffusion-based methods, with improvements in several harmonic and rhythmic consistency metrics. Human evaluations further indicate improvements in perceived musical quality, while ablation studies support the effectiveness of the Harmonic Plan and global conditioning components.

Critical strengths
  1. Well-motivated architecture for structured music generation. The proposed Decoder-in-Decoder architecture provides a reasonable approach to modeling musical dependencies at different temporal scales. By combining compressed global history with detailed information from the preceding bar, DID balances long-range context and local continuity while retaining the flexibility of autoregressive generation. This design is meaningful for structured symbolic music generation.

  2. Promising experimental results across multiple generation tasks. DID is evaluated on three generation tasks using two public datasets, with comparisons against representative autoregressive and diffusion-based methods. The results show improvements in several harmonic and rhythmic consistency metrics, supported by favorable human evaluations. The ablation studies further demonstrate the contributions of Harmonic Plan and global conditioning, providing additional evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. Limited novelty over existing hierarchical autoregressive approaches. The proposed 2D autoregressive formulation is closely related to existing hierarchical music generation methods, such as Museformer, Bar Transformer, and Nested Music Transformer. Moreover, the nested probability factorization itself can also be expressed using conventional 1D autoregressive modeling. Although DID introduces a different architectural organization, the paper does not sufficiently clarify its fundamental advantages over these approaches, making the methodological novelty somewhat limited.

  2. Insufficient ablation studies on the core architecture. While the authors evaluate the contributions of Harmonic Plan and global conditioning, they do not provide sufficient ablations for the central Decoder-in-Decoder architecture. In particular, the individual contributions of the time decoder, compressed global history, and complete local history remain unclear. Without these comparisons, it is difficult to determine whether the reported improvements primarily result from the proposed 2D autoregressive design or other architectural components.

  3. Inadequate evaluation of long-term musical structure. Although the paper emphasizes improved cross-bar dependencies and long-term structural coherence, the objective evaluation primarily relies on pitch, harmonic, and rhythmic similarity metrics. These metrics do not directly capture higher-level musical structures, such as long-range repetition, sectional organization, and structural development. Furthermore, the best-matching reference-bar evaluation may reward local pattern repetition without reflecting genuine global coherence. More structure-aware evaluations would provide stronger evidence for the paper's central claims.

Questions for authors

Q1. Novelty of the 2D Autoregressive Formulation. The proposed 2D autoregressive factorization appears mathematically equivalent to a conventional 1D autoregressive factorization with bar-level grouping. Could the authors clarify what fundamentally distinguishes DID from existing hierarchical autoregressive approaches, beyond architectural organization? In particular, what modeling capabilities or advantages arise specifically from the proposed 2D formulation?

Q2. Effectiveness of the Core Decoder-in-Decoder Architecture. The current ablation studies primarily focus on Harmonic Plan and global conditioning, while the contributions of the time decoder and the global/local history design remain unclear. Could the authors provide additional ablations or controlled comparisons, such as removing the time decoder or replacing the dual-history mechanism with a conventional autoregressive context? Such comparisons would help establish whether the improvements are attributable to the proposed architecture.

Q3. Evaluation of Long-Term Musical Structure. The paper emphasizes improved cross-bar dependencies and long-term structural coherence, but the current metrics mainly capture pitch, harmonic, and rhythmic similarity. How do the authors justify these metrics as sufficient evidence for long-term structural modeling? Could structure-aware evaluations, such as self-similarity analysis or measures of long-range repetition and variation, provide more direct support for the central claims?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 9R59 (part 1 / 3 )

Dear Reviewer 9R59,

Thank you for taking the time to review our paper. We sincerely appreciate your positive feedback on our architecture and experimental results, as well as your insightful questions. We would like to address your concerns as follows:


[Weakness 1 / Q1: The proposed 2D autoregressive formulation resembles existing hierarchical music models, and its nested factorization can be expressed through 1D autoregression. Its advantages over prior approaches remain unclear.]

Reply: Thank you for raising this important concern regarding the novelty of DID. We sincerely appreciate the opportunity to clarify its distinctions from existing hierarchical music models. We acknowledge that Museformer, Bar Transformer, and Nested Music Transformer are valuable contributions to symbolic music generation; however, their core modeling approaches differ fundamentally from DID's 2D autoregressive formulation.

To further clarify this distinction, we consider the attention mask structure. Although these methods make valuable contributions, their main decoding processes operate on 1D token sequences, using 2D token masks of shape [Batch, Sequence]. In contrast, DID explicitly separates the bar and event dimensions, resulting in 3D masks of shape [Batch, Bar Sequence, Event Sequence]. This reflects DID's explicit 2D autoregressive formulation.

Specifically, Museformer introduces bar-level summary tokens to support hierarchical attention but retains a 1D sequence of musical events. Bar Transformer uses note-level and bar-level encoders to learn musical structure, while its arrangement decoder generates music along a single sequence. NMT retains 1D autoregressive generation of compound tokens while introducing causal dependencies among the attributes within each token.

In contrast, the core contribution of DID is an explicit 2D autoregressive decoding framework that aligns each time step with a complete musical bar and generates within-bar events along a separate note dimension.This formulation preserves the 2D bar–event structure of music while combining the structural benefits of non-autoregressive methods with the flexibility of autoregressive generation. Its bar-level alignment enables fine-grained conditional control, while autoregressive decoding supports flexible and continuous music generation. Consequently, DID supports music continuation, chord-conditioned piano generation, and accompaniment generation within a unified framework (Tables 1, 2, and 4), whereas representative 1D autoregressive baselines in our experiments are primarily evaluated on music continuation.

Furthermore, to provide a more comprehensive comparison with existing approaches, we have expanded Appendix D of the revised manuscript.

Regarding the concern that DID's nested factorization can be expressed through 1D autoregression, the actual generation process and modeling architecture are fundamentally different. DID explicitly organizes generation into two levels: the outer autoregressive process advances across complete bars, while the inner process generates the ordered events within each bar, conditioned on its bar-level context. This explicit separation defines the generation order and enables direct bar-level condition alignment, which is not inherently provided by a flattened 1D formulation. Furthermore, as discussed in Sec. 4.4, DID demonstrates better rhythmic consistency and more stable chromatic similarity than conventional 1D autoregressive baselines, particularly in long-range continuation.

We hope these comparisons clarify the distinction of DID from prior work and address your concerns. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining questions.

AuthorsOpenReview ↗

Response to Reviewer 9R59 (part 2 / 3 )

[Weakness 2 / Q2: The contributions of the time decoder, compressed global history, and complete local history remain unclear, making it difficult to attribute improvements to the proposed 2D design.]

Reply: Thank you very much for this valuable and constructive suggestion. We sincerely appreciate your attention to the empirical validation of our core architecture, and agree that more targeted ablations would help clarify the contributions of its individual components.

First, Table 1 already compares DID with representative 1D autoregressive baselines, including CP and NMT. DID achieves higher average continuation scores on both POP909 (89.53, compared with 88.06 for CP and 88.04 for NMT) and POP1k7 (86.79, compared with 84.24 and 85.92, respectively). Fig. 3 further demonstrates more stable rhythmic similarity during long-range continuation.

To more directly examine the contributions of the two historical pathways, we additionally evaluate (1) w/o Global History, which removes the compressed history Ht−2H_{t-2} produced by the Time Decoder while retaining complete Local History Lt−1L_{t-1}; and (2) w/o Local History, which removes Lt−1L_{t-1} while retaining the compressed Global History Ht−1H_{t-1} (Sec. 3.2, Eqs. (6)–(7)).

Additional Ablation: Global–Local History on POP909 Music Continuation.

VariantSimPCE ↑SimChr ↑SimGrv ↑PD ↑SimSC ↑Avg. ↑
w/o Global History85.5371.2983.4947.9184.2174.48
w/o Local History95.7391.6392.8567.9898.1489.27
DID (Full)96.3792.1092.4968.4298.2989.53

As shown in the ablation results, removing Global History leads to substantial performance degradation across all evaluation metrics, highlighting the importance of the Time Decoder's compressed cross-bar context. Furthermore, DID (Full) achieves better overall performance and outperforms the Global-History-only variant on most metrics, supporting the complementary contribution of complete Local History in preserving fine-grained neighboring-bar information. Together, these results provide empirical support for DID's dual-scale historical modeling design.

We hope that the existing comparisons with 1D baselines and these targeted ablations help clarify the benefits of DID's architectural design. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining concerns.

AuthorsOpenReview ↗

Response to Reviewer 9R59 (part 3 / 3 )

[Weakness 3 / Q3: Current similarity metrics and best-matching reference-bar evaluation may not adequately capture long-term musical structure, including global coherence, structural repetition, and development.]

Reply: Thank you for your insightful comment regarding long-term structural evaluation. We sincerely appreciate your attention to this important aspect of symbolic music generation.

To further examine long-term musical structure, we provide additional evaluations from two complementary perspectives:

(1) Long-range structural consistency. We have conducted a dedicated long-range continuation experiment in Sec. 4.4 (Fig. 3), comparing DID with CP and NMT on the common subset of POP1k7 test examples extending beyond 80 bars. DID maintains nearly stable rhythmic and chromatic similarity as generation length increases, whereas CP and NMT exhibit more noticeable declines. These results provide evidence of DID's ability to preserve rhythmic and tonal coherence over extended generation.

(2) Musical pattern diversity and development. We additionally introduce N-gram Diversity and Reconstruction Accuracy (ReconAcc) on POP1k7 and POP909. N-gram Diversity measures the proportion of unique pitch-class N-grams, reflecting local pattern diversity and helping identify excessive repetition. ReconAcc measures the similarity between generated and reference pitch-class sequences using normalized edit distance, providing a complementary assessment of sequential consistency.

Additional Evaluation: N-gram Diversity and Reconstruction Accuracy.

MethodPOP1k7 N-gram (%) ↑POP1k7 ReconAcc (%) ↑POP909 N-gram (%) ↑POP909 ReconAcc (%) ↑
DID (Ours)76.7035.0573.3135.64
MuseTok87.1232.9582.5232.94
CP69.0133.6460.6430.28
NB70.8434.1754.0129.89
REMI73.4634.6665.9229.52

DID achieves the highest ReconAcc on both datasets, while maintaining competitive N-gram Diversity. Although MuseTok achieves higher N-gram Diversity, its lower ReconAcc indicates that greater local pattern diversity does not necessarily translate into stronger sequential consistency. DID, in contrast, maintains a favorable balance between local pattern diversity and reference-sequence consistency.

Together, the long-range continuation experiment and these additional metrics provide complementary evidence regarding global musical consistency, local pattern repetition, and the balance between musical variation and structural continuity.

We hope that these additional evaluations help address your concerns. We sincerely appreciate your constructive feedback and would be delighted to provide further explanations or engage in additional discussion.


We sincerely hope that our responses and additional experiments address your concerns. We would be grateful if you could kindly reconsider your score in light of these clarifications. We also sincerely invite you to visit our demo page (https://anonymous.4open.science/w/did-cases-785B/) and listen to the generated music samples. We greatly appreciate your constructive feedback and would be delighted to discuss any remaining questions.

Reviewer_s4QmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents DID, a hierarchical Transformer model for symbolic music generation. The model first patches bar-level embeddings and feeds them into a time-wise Transformer with causal self-attention to aggregate historical context. Conditioned on history bars 11 to t−1t-1, bar tt leverages a note-level Transformer decoder to predict note tokens within the bar autoregressively. A 1-bar delayed update strategy further allows the decoder to predict bar tt conditioned on bar t−1t-1's complete token sequence (local context) and bars 11 to t−2t-2's aggregated encoding (global history). The paper also proposes a Harmonic Plan strategy, which computes a coarse pitch profile of the target tokens as an intermediate reasoning step, and has the model predict them before decoding the target tokens. Experiments demonstrate the model's performance across three conditional piano arrangement tasks: continuation, accompaniment arrangement, and chord-conditioned generation. An ablation study validates that the inclusion of Harmonic Plan improves performance.

Critical strengths
  • The proposed "decoder-in-decoder" architecture combines compact global history and fine-grained local context for symbolic music generation without a fixed-length constraint.
  • The Harmonic Plan strategy improves model performance by providing intermediate guidance between the given conditions and the target tokens.
Critical weaknesses
  • The main idea of hierarchical symbolic music modeling across the 2D time-pitch plane is not entirely novel. Similar ideas could be seen in earlier works on polyphonic music representation learning [1], as well as more recent models using bar-stream patching [2].

  • Despite the model being called "decoder-in-decoder" and "2D autoregressive", the time-wise Transformer is not strictly an autoregressive decoder, but is more like a temporal aggregator. According to Eq. (3), the time-wise Transformer produces context latents st\mathbf{\mathit{s}}_{t} given history bars 11 to tt. st\mathbf{\mathit{s}}_{t} serves as condition for the note-level decoder to predict bar t+1t+1, but the time-wise Transformer itself does not predict any latents for bar t+1t+1. As a result, the framing of "2D autoregressive" appears to be overclaiming and potentially misleading.

  • There lacks an ablation study on the delayed update strategy to validate its impact.

  • The objective metrics are not entirely intuitive. It is not sufficiently clear what music aspect each metric evaluates, and whether they are duplicate or complementary to each other. Although the results show evaluation scores in favour of the proposed model, it is not entirely clear in what ways the model outperforms the baselines.

  • The introduction of Harmonic Plan lacks sufficient motivation and context in the paper. It appears to be a plug-and-play strategy proposed separately and independently from the "decoder-in-decoder" architecture. This makes the paper read like an aggregation of empirical engineering tricks without sufficient scientific insight.

[1] Z. Wang, et al. "Pianotree vae: Structured representation learning for polyphonic music," in ISMIR 2020.

[2] Y. Wang, et al. "Notagen: Advancing musicality in symbolic music generation with large language model training paradigms," in IJCAI 2025.

Questions for authors
  • Section 3 elaborates on the model inference logic sufficiently, but the training objectives lack clarification. While Transformers are typically trained via next-token prediction, how does the "decoder-in-decoder" design formulate that objective?

  • In Algorithm 1, what is pt\mathbf{\mathit{p}}_t? At timestep t=1t=1, what would be L0\mathbf{\mathit{L}}_0 and s−1\mathbf{\mathit{s}}_{-1}?

  • What symbolic tokenization does the model apply, and how is chord encoded?

  • What is the chord recognition model used for scoring CA?

  • It would be ideal for the demo page to also show samples from the baseline models for a more straightforward comparison.

9 comments
AuthorsOpenReview ↗

Response to Reviewer S4Qm (part 1 / 5 )

Dear Reviewer S4Qm,

Thank you for taking the time to review our paper and we appreciate the positive feedback and your detailed questions regarding our work. We would like to address your concerns as follows:

[Weakness 1: Novelty concerns regarding hierarchical 2D modeling compared to existing works (PianoTree VAE, NotaGen).]

Reply: Thank you for your thoughtful comments regarding the novelty of DID. We sincerely appreciate the opportunity to clarify its distinctions from NotaGen and PianoTree VAE.We acknowledge that both are valuable contributions to symbolic music generation; however, their core modeling approaches differ fundamentally from DID's 2D autoregressive formulation.

Specifically, NotaGen divides 1D ABC sequences into fixed-length character patches that are not aligned with musical units (e.g., one bar may span multiple patches). In contrast, DID aligns each autoregressive time step with a complete musical bar and explicitly models within-bar events along a separate note dimension, forming a 2D autoregressive architecture (NotaGen, Secs. 3.1–3.2, Fig. 2; DID, Secs. 3.1–3.2). This bar-level alignment enables DID to directly incorporate chord and melody conditions into each bar, supporting fine-grained conditional generation (Tables 2 and 4), including chord-conditioned piano generation and accompaniment generation . PianoTree VAE encodes a given music segment into latent variables and decodes them to reconstruct the original segment. In contrast, DID directly predicts the next bar from preceding musical context through 2D autoregressive decoding. Thus, DID supports flexible bar-by-bar generation within a unified framework, whereas PianoTree VAE uses an additional Transformer Decoder to predict future latent segment representations (PianoTree VAE, Secs. 3.2 and 4.7; DID, Sec. 3.2, Eqs. (8)–(9)).

The core contribution of DID is an explicit 2D autoregressive decoding framework that aligns each time step with a complete musical bar and models within-bar events along a separate note dimension. This formulation preserves the 2D bar–event structure of music while combining the structural benefits of non-autoregressive methods with the flexibility of autoregressive generation. Such bar-level alignment enables fine-grained conditional control, while autoregressive decoding supports flexible and continuous music generation. Consequently, DID supports music continuation, chord-conditioned piano generation, and accompaniment generation within a unified framework (Tables 1, 2, and 4), whereas representative 1D Transformer baselines are primarily evaluated on music continuation.

The above discussion highlights the core differences between DID and these two approaches. To further clarify these distinctions and address your concerns, we provide a more detailed comparison below.

Distinction from NotaGen.

(1) Alignment with musical structure. NotaGen divides each bar into fixed-length character patches, meaning that a complete bar may span multiple autoregressive steps (NotaGen, Secs. 3.1–3.2, Fig. 2). Consequently, its autoregressive steps do not directly correspond to musical time units such as beats or bars, and its modeling remains essentially 1D. In contrast, DID treats each complete bar as one autoregressive time step and generates its events along a separate note dimension, explicitly modeling music in 2D (DID, Secs. 3.1–3.2).

(2) Accompaniment generation. Since NotaGen does not align its autoregressive steps with musical time units, it is difficult to directly align melody and chord conditions with the target music for fine-grained accompaniment generation. In contrast, DID explicitly aligns each bar with its corresponding melody and chord conditions, enabling fine-grained accompaniment generation within its 2D autoregressive framework (DID, Secs. 3.1–3.2).

(3) Music representation. NotaGen represents music using ABC notation and generates character sequences, whereas DID adopts structured compound events with explicit musical attributes. Thus, the two methods differ in how musical information is represented and organized for generation.

Distinction from PianoTree VAE.

(1) Reconstruction vs. prediction. PianoTree VAE uses autoregressive decoding to reconstruct fixed-length music segments from latent variables (PianoTree VAE, Sec. 3.2). In contrast, DID directly predicts the next bar from preceding musical context through 2D autoregressive decoding, generating within-bar events before advancing to the next bar (DID, Sec. 3.2, Eqs. (8)–(9)).

AuthorsOpenReview ↗

Response to Reviewer S4Qm (part 2 / 5 )

(2) Architectural design. To predict subsequent music segments, PianoTree VAE requires an additional Transformer Decoder beyond its VAE architecture (PianoTree VAE, Sec. 4.7). In contrast, DID integrates cross-bar prediction and within-bar event generation into a unified Decoder-in-Decoder architecture with two Transformer decoders, providing a more direct framework for continuous music generation (DID, Sec. 3.2).

(3) Music representation. PianoTree VAE primarily models pitch and duration at quantized onset positions (PianoTree VAE, Sec. 3.1.1). In contrast, DID uses structured compound events with richer musical attributes, including chord, tempo, track, pitch, duration, and velocity, providing a more comprehensive and flexible representation.

We have also expanded Appendix D in the revised manuscript, with the additions highlighted in blue. This appendix further clarifies the distinctions between DID and NotaGen and PianoTree VAE, as well as other relevant hierarchical and nested autoregressive baselines.

We hope these comparisons help address your concerns. We sincerely appreciate your feedback and would be delighted to discuss any remaining questions.


[Weakness 2: The time-wise Transformer is not strictly autoregressive, but rather a temporal aggregator, making the 2D autoregressive formulation potentially misleading.]

Reply: Thank you for raising this important concern regarding the autoregressive formulation of DID. We sincerely appreciate your attention to the architectural design and would like to further clarify the information flow and prediction mechanism of the Time Decoder.

First, each completed bar is summarized into a representation (sts_t) using a learnable summary query and the Bar Encoder:

st=BarEnc⁡([qsum;xt,1;…;xt,Nt])0.s_t = \operatorname{BarEnc}([q_{\mathrm{sum}};x_{t,1};\ldots;x_{t,N_t}])_0.

These bar summaries are sequentially fed into the causal Time Decoder (as formulated in Sec. 3.1, Eqs. (2)–(3)).

Second, the Time Decoder predicts continuous latent history representations rather than discrete tokens. Instead of a conventional vocabulary LM head, we use an MLP projection to map its causal hidden state into four latent history tokens: ht−2=Dtime(s≤t−2),h_{t-2} = D_{\mathrm{time}}(s_{\leq t-2}), Ht−2=reshape⁡(MLP⁡(ht−2)),Ht−2∈R4×dnH_{t-2} = \operatorname{reshape}(\operatorname{MLP}(h_{t-2})), \quad H_{t-2} \in \mathbb{R}^{4 \times d_n}

These four tokens encode compressed global history and are concatenated with the complete NOTE events from the preceding bar to condition the Note Decoder (as formulated in Sec. 3.2, Eqs. (6)–(7)):

Pt=[Ht−2;Lt−1].P_t=[H_{t-2};L_{t-1}].

Finally, the Note Decoder generates the current bar autoregressively:

\prod_{n=1}^{N_t} P(e_{t,n}\mid P_t,e_{t,<n}).$$ **Therefore, the Time Decoder does not merely aggregate historical information; it produces learned latent history representations that explicitly guide subsequent bar generation.** Together, the two decoders establish autoregressive generation across bars and within-bar events, as defined by the joint factorization in Eq. (9). We will further clarify this information flow in the revised manuscript. We hope that our response clarifies the autoregressive mechanism of DID and addresses your concern. We would be delighted to engage in further discussion with you. --- **[Weakness 3: Lack of an ablation study on the delayed update strategy.]** **Reply:** Thank you for this valuable suggestion regarding the delayed update strategy. We sincerely appreciate your attention to the architectural design and its empirical validation. To further examine its contribution, we have added a dedicated ablation experiment, as shown in Table 5. Specifically, we compare the full DID, which uses both compressed Global History ($H_{t-2}$) and full-resolution Local History ($L_{t-1}$), with a variant that removes the delayed update and uses (\$H_{t-1} \$) alone. All other model configurations and training settings remain unchanged. **Table 5: Ablation results on POP909 accompaniment generation.** | **Variant** | **OS ↑** | **CA ↑** | **CS ↑** | **Avg. ↑** | | ------------------ | --------- | --------- | --------- | ---------- | | w/o Delayed Update | **85.02** | 82.71 | 78.86 | 82.19 | | DID (Full) | 85.00 | **83.52** | **79.30** | **82.61** | As shown in Table 5, DID (Full) achieves better overall performance, particularly in harmonic consistency, while maintaining comparable rhythmic alignment. **These results suggest that the delayed update strategy helps balance compressed long-range context and fine-grained local information, contributing to more coherent music generation.** We hope that this additional ablation study addresses your concern, and we would be delighted to engage in further discussion with you.
AuthorsOpenReview ↗

Response to Reviewer S4Qm (part 3 / 5 )

[Weakness 4: Insufficient motivation and justification of Harmonic Plan.]

Reply: Thank you for your insightful comment on Harmonic Plan. We appreciate your attention to its motivation and role in our framework.

Our motivation for introducing Harmonic Plan arises from conditional music generation, where recent baselines are predominantly diffusion-based, including FGG and D3PIA. These methods can incorporate conditional guidance and refine predictions throughout the iterative generation process. In contrast, autoregressive models cannot revise previously generated events, and we do not apply any post-hoc intervention to sampled outputs. To provide harmonic guidance during generation, we therefore introduce Harmonic Plan as an intermediate representation before target note generation. The Plan tokens remain accessible to subsequent NOTE predictions through the Note Decoder's attention mechanism, allowing harmonic conditions to implicitly guide event generation without explicitly fixing the final note realization (as formulated in Sec. 3.3, Eqs. (10)–(12), and illustrated by the CHORD–PLAN–NOTE information flow in Fig. 4).

The effectiveness of Harmonic Plan is supported by Table 5: removing it decreases CA from 83.52 to 81.19, CS from 79.30 to 77.60, and OS from 85.00 to 84.41. Fig. 4 further reveals the CHORD–PLAN–NOTE information flow.

We hope that our response clarifies the motivation and effectiveness of Harmonic Plan, and we would be delighted to engage in further discussion with you.


[Weakness 5: Insufficient clarification of evaluation metrics and comparative performance.]

Reply: Thank you for your valuable comments on the evaluation metrics and comparative results. We sincerely appreciate your attention to the evaluation methodology and experimental analysis.

Regarding the evaluation metrics, we have already provided an introduction to each metric in Appendix B.1. To better address your concerns, we have further expanded Appendix B.1 of the revised manuscript, with the additions highlighted in blue, to clarify their computation procedures, evaluation purposes, and complementary roles.

Specifically, for music continuation, SimChr and SimSC assess tonal characteristics from different perspectives, measuring pitch-class similarity and scale consistency, respectively. SimPCE and PD evaluate pitch-distribution properties, capturing pitch-class diversity and overall pitch usage, while SimGrv independently assesses rhythmic-pattern similarity. For conditional generation, CA and CS provide complementary evaluations of harmonic consistency: CA measures exact beat-level chord agreement, whereas CS captures softer progression-level harmonic similarity. OS additionally evaluates rhythmic alignment, complementing these harmony-focused metrics. Together, these metrics assess distinct yet complementary aspects of musical structure.

Regarding comparative performance, we provide a detailed analysis in the Performance Analysis section. As shown in Tables 1, 2, and 4, DID's advantages in SimGrv and SimChr demonstrate the benefits of explicit time–note modeling and dual-scale historical context for rhythmic and tonal consistency, while its strong CA, CS, and OS results support the effectiveness of Harmonic Plan and structured autoregressive decoding in maintaining harmonic and rhythmic consistency.

We hope these additional explanations and experimental results help address your concerns. We sincerely appreciate your valuable feedback and would be delighted to engage in further discussion with you.

Reviewer_gGE7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dual decoder 2D autoregressive framework for variable-length symbolic music generation along time and note dimensions. This is in contrast to existing autoregressive models that flatten both these dimensions into 1D, and non-autoregressive models (e.g. diffusion-based) that can not natively support variable-length music generation. The proposed decoder-in-decoder architecture constitutes a time and a note decoder; while the time decoder updates the running temporal music history, the note decoder uses this history to generate note events constrained by harmonic plan events for consistency. Thus, it achieves better high harmonic and rhythmic coherence compared to existing methods.

Critical strengths

(1). Modelling the interplay between note and time dimensions in symbolic music generation through a decoder-in-decoder architecture is an interesting and novel approach. This can be an important contribution to music generation literature.

(2). The demonstrated quality of generated music outputs indicate very high quality.

(3). Experimental results and analysis is thorough in terms of comparison with various music generation model families. The work includes

(4). The manuscript is very well organized and written clearly. The descriptions and illustrations are easy to follow and understand, and they explain the core ideas of the paper very clearly.

(5). While the proposed technical method and the architecture are well motivated and they are sound technically, the empirical gains do not justify the additional complexity (see weaknesses section for more details)

Critical weaknesses

(1). What are the advantages of explicit bar representations over flattened representation in symbolic music generation? While the explicit bar representation may seem more natural, it is unclear how the 1D flattened representations fall short in music generation quality. It is recommended to motivate the need for explicit 2D representation in terms of concrete generation quality improvements.

(2). The ablation analysis shows that the gains achieved with Global History and Harmonic Planning events look rather modest. Do these apparent modest gains justify the additional complexity of 2D autoregression? It is unclear how 1D autoregression formulation of the proposed method performs in terms of human perception quality and benchmarking metrics. Overall, the current ablation analysis is weak; it does not clearly demonstrate the necessity of the explicit 2D formulation, which is the core contribution of this work.

Questions for authors

(1). What is the practical motivation (e.g. failure modes of existing 1D autoregressive models or diffusion models) behind formulating symbolic music generation as an explicit 2D representation formulation?

(2). How do the modest gains in quantitative results justify the additional modelling complexity of this 2D autoregressive formulation?

8 comments
AuthorsOpenReview ↗

Response to Reviewer gGE7 (part 1 / 2 )

Dear Reviewer gGE7,

Thank you very much for your thoughtful and encouraging review. We sincerely appreciate your recognition of our architectural novelty, technical soundness, and manuscript clarity. We address your concerns as follows:

[Weakness 1: The practical advantages of explicit 2D bar-level modeling over flattened 1D representations remain unclear.]

Reply: We sincerely thank you for this thoughtful observation. We greatly appreciate your suggestion to clarify the practical advantages of explicit 2D modeling.

We would like to highlight two advantages:

  • Continuation quality: Compared with flattened 1D baselines (REMI, CP, NMT, and MuseTok), DID achieves the highest average continuation scores and SimGrv on both datasets (Table 1), indicating better overall generation quality and rhythmic consistency. Moreover, DID maintains more stable rhythmic and chromatic similarity than CP and NMT beyond 80 bars (Fig. 3), demonstrating better long-range stability.
  • Task flexibility: The representative 1D baselines are primarily evaluated on music continuation. In contrast, DID's explicit bar-level organization enables fine-grained alignment of musical conditions with target bars, supporting both chord-conditioned piano generation and accompaniment generation within the same framework (Tables 2 and 4).

We hope these clarifications address your concerns. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining questions.


[Weakness 2: Insufficient ablation studies to justify the added complexity and advantages of explicit 2D autoregression over 1D modeling.]

Reply: Thank you very much for this valuable and constructive suggestion. We fully appreciate your concern regarding whether the performance improvements justify the additional architectural complexity, and agree that more direct evidence would strengthen our evaluation.

First, Table 1 already includes representative 1D autoregressive baselines, CP and NMT. DID achieves higher average continuation scores on both POP909 (89.53, compared with 88.06 for CP and 88.04 for NMT) and POP1k7 (86.79, compared with 84.24 and 85.92, respectively). Fig. 3 further demonstrates more stable rhythmic similarity over long-range continuation.

To further investigate the contribution of the Time Decoder, we additionally consider an ablation removing Global History (Ht−2H_{t-2}) while retaining the complete Local History (Lt−1L_{t-1}). Since Ht−2H_{t-2} is generated by the Time Decoder, this comparison directly examines the contribution of its compressed cross-bar context (Sec. 3.2, Eqs. (6)–(7)).

Additional Ablation: Global History on POP909 Music Continuation.

VariantSimPCE ↑SimChr ↑SimGrv ↑PD ↑SimSC ↑Avg. ↑
DID (w/o Global History)85.5371.2983.4947.9184.2174.48
DID (Full)96.3792.1092.4968.4298.2989.53

The ablation results show that removing Global History consistently degrades performance across all evaluation metrics, indicating that local note-level information alone is insufficient to maintain long-range musical coherence. This highlights the importance of the Time Decoder in capturing cross-bar dependencies and providing complementary long-range context for subsequent generation.

We hope that the existing comparisons with 1D baselines, together with this additional ablation, provide a clearer assessment of DID's architectural benefits. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining concerns.

AuthorsOpenReview ↗

Response to Reviewer gGE7 (part 2 / 2 )

[Q1: What is the practical motivation (e.g., failure modes of existing 1D autoregressive models or diffusion models) behind formulating symbolic music generation as an explicit 2D representation formulation?]

Reply: Thank you very much for this insightful question. We sincerely appreciate the opportunity to clarify the practical motivation behind our 2D formulation.

(1) From the perspective of task flexibility, existing approaches have different limitations:

  • 1D autoregressive methods (e.g., REMI, CP, and NMT) flatten musical events into a single sequence, making explicit bar-level organization and fine-grained condition alignment less straightforward. Consequently, these methods are primarily evaluated on music continuation.
  • Diffusion-based methods preserve 2D musical structures but typically generate within predefined temporal windows, making flexible variable-length continuation less straightforward. They therefore often focus on fine-grained conditional generation.

DID addresses these limitations by explicitly modeling bar progression and within-bar events as two interacting autoregressive dimensions, supporting both flexible continuation and fine-grained bar-aligned conditional generation within a unified framework.

(2) From the perspective of generation quality, our experiments demonstrate two practical benefits:

  • Music continuation: DID achieves the highest average scores on both POP909 and POP1k7 (Table 1), while maintaining more stable rhythmic similarity over extended generation (Fig. 3), demonstrating better continuation quality and long-range stability than 1D baselines.
  • Accompaniment generation: DID achieves the highest average score and CA among the compared methods (Table 2), demonstrating its effectiveness in maintaining harmonic consistency under fine-grained musical conditions.

We hope these explanations clarify the practical motivation and benefits of our formulation. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining questions.


[Q2: How do the modest gains in quantitative results justify the additional modelling complexity of this 2D autoregressive formulation?]

Reply: Thank you very much for raising this important question. We sincerely appreciate your consideration of the trade-off between architectural complexity and practical benefits.

We believe the additional complexity of DID is justified by two key advantages:

  • More faithful modeling of musical characteristics. DID not only represents music through discrete compound events but also explicitly preserves its two-dimensional bar–event organization. This allows the model to capture both the rich attributes of individual musical events and their structural relationships across and within bars, providing a more natural formulation for symbolic music modeling.
  • Greater flexibility across generation tasks. Conventional 1D autoregressive methods (e.g., REMI, CP, and NMT) are primarily evaluated on music continuation, whereas diffusion-based methods (e.g., FGG and D3PIA) mainly focus on fine-grained conditional generation. DID unifies both capabilities, supporting music continuation, chord-conditioned piano generation, and accompaniment generation within a single architecture.

These advantages are supported by our objective and subjective evaluations, where DID achieves the highest average continuation scores on both datasets (Table 1), as well as the best overall human evaluation scores for continuation and accompaniment generation (Table 3).

We hope these explanations clarify why the additional architectural complexity is worthwhile. We sincerely appreciate your constructive feedback and would be delighted to discuss any remaining questions.


We hope that these explanations and additional comparisons address your concerns. We also sincerely invite you to visit our demo page (https://anonymous.4open.science/w/did-cases-785B/) and listen to the generated music samples. We greatly appreciate your encouraging evaluation and constructive feedback, and would be delighted to engage in further discussion with you.

Thank you for clarifying the practical advantages of explicit 2D bar-level modelling for this problem. I suggest the inclusion of these points clearly in the introduction to motivate the proposed method.

Thank you for the clarification regarding the existing ablation studies and for providing additional ablations. The new ablation study clearly shows the impact of Global History component.

This additional information addresses my concerns satisfactorily.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.