acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Autoregressive Music Modeling with Temporal Realignment

Abstract

Language models for symbolic music typically flatten events into a single sequence. When used for conditional generation, this creates a conflict between autoregressive sequence order and musical time: future conditioning information must appear early in the sequence, which can disrupt the underlying temporal relationships. We address this mismatch with model-level temporal realignment. A hierarchical grouping architecture first organizes tokens into timing-aware groups (notes and concurrent onsets). Rotary time embeddings then reassign temporal coordinates to the original musical times while leaving the input sequence and causal factorization unchanged. The resulting model can still be trained with the standard next-token objective. On both symbolic and audio-conditioned generation tasks, temporal realignment substantially improves timing prediction, yields higher coherence scores, and is preferred by human listeners over an otherwise identical model without realignment under random conditioning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Music generation, Music conditioning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rW8TOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

With in-sequence (anticipatory) conditioning, conditioning events are placed earlier in the token sequence than their musical time, so sequence order and musical time disagree. The paper formalises this as delta conditioning with laminar groups. It then proposes a two-level, H-Net-style hierarchy: tokens are compressed into note groups and then into onset-synchronous time groups. The main Transformer uses the original musical time as rotary (RoTE) coordinates, while the input order, causal mask, and next-token objective stay unchanged. Experiments on deduplicated Lakh MIDI cover melody-conditioned accompaniment, random conditioning, and audio-conditioned drum generation. Evaluation uses per-token-type perplexities, the PHALAR coherence score, and a 12-participant listening test with participant-level permutation tests.

Critical strengths

S1. The same-architecture ablation shows large, consistent effects. Across three parameter allocations, realignment reduces time-token perplexity PPLt from 2.57 to 1.80, 2.82 to 1.88, and 2.58 to 1.85 (−28% to −33%), and improves PHALAR by +2.3 to +2.9 (Table 1). In the random-conditioning listening test against the model without realignment, the result is 67/19/14 (win/tie/loss). A participant-level exact permutation test gives a Holm-corrected p = 0.0117 (Table 8; I recomputed the four Holm values and they match). This directly supports the abstract's claim of improvement over "an otherwise identical model without realignment".

S2. The mechanism is simple and causally correct. Only the attention coordinates change. The mask, slots, and KV-cache order are untouched (App. F, Eq. 32), so the method can be reused in other interleaved conditional generators. It extends naturally to mixed audio–MIDI sequences. With audio conditioning, PHALAR rises from 5.40 to 6.35 against a human reference of 6.51, computed on the 2,454 windows that are valid for both models (H.3). The appendix is also careful about data deduplication (Choi et al.'s list) and data accounting, and about consent and blinding in the listening study.

Critical weaknesses

W1. The attribution to the order/time conflict is not isolated. Table 1's flat baseline ("AMT"; from Appendix G.2/G.3 it appears to be the Vanilla Transformer trained on the same representation with the same 10 s lookahead) uses no realignment, yet:

  • its PPLt is 1.83, essentially equal to the best realigned model (Balanced, 1.80);
  • its total PPL of 1.71 is better than every realigned hierarchical model (1.78–1.84), and its instrument/pitch/duration perplexities are also the best;
  • the hierarchical variants without realignment are much worse than this flat model (PPLt 2.57–2.82, PHALAR 2.63–3.25).

The large ablation gap is therefore more consistent with realignment repairing timing information lost by hierarchical compression than with the order/time conflict itself harming modelling, as Fig. 1 and Prop. 1 suggest. The p.5 argument that realignment cannot be done at the token level seems to follow from the representation choice of placing TIME at the end of a group, not from a general limitation. The missing control is a flat Vanilla Transformer with token-level musical-time RoPE, where each token causally inherits the most recent known time, or alternatively AMT's original time-first triplets.

W2. The evidence for an advantage over the flat baseline is thin.

  • The only advantage is PHALAR 5.57 vs. 4.27 in the symbolic row of Table 1. The text does not state the task, the number of songs, or per-system completion rates.
  • Failed or unfinished generations, invalid MIDI, and silent windows are excluded from PHALAR (G.4, p.23) without reporting per-system retention. If retention differs, the means mix generation quality with how often a system produces something scorable. A common-set analysis is needed. The audio experiment already does this.
  • The comparison has no confidence intervals and uses a single seed. The flat baseline is absent from the listening test, and the accompaniment-task listening result is not significant (41/25/34, Holm p = 0.273).
  • p.7 says realignment "retains the efficiency of grouped modeling", but no runtime or FLOPs are reported, and Appendix F concedes that sequence-length savings do not determine end-to-end runtime.

Secondary concerns (less decisive than W1–W2):

W3. Some claims are broader than the evidence. The abstract says that "on both symbolic and audio-conditioned generation tasks, temporal realignment substantially improves timing prediction". In the audio-conditioned setting, PPLt moves only from 1.91 to 1.87 (−2%). The audio "w/o realignment" model gives audio latents no absolute time embedding at all (H.2). That comparison is therefore "shared timestamps vs. none" rather than RoTE vs. alternative ways of injecting time. Moreover, the audio experiments cover only synthesised accompaniment-to-drums on 100 test songs.

Other statements also go beyond the evidence:

  • p.2 states that temporal realignment "consistently outperforms non-realigned baselines for both settings". The only non-realigned comparisons are the same-architecture ablations, and the accompaniment listening comparison is not significant.
  • p.7 introduces the baseline as "the state-of-the-art MIDI generation model" and states that the best realigned model "outperforms the AMT baseline in generation quality". The baseline is retrained on this paper's representation (see Q1), and the advantage holds only on PHALAR, while the baseline's total PPL is better (1.71 vs. 1.78). "Outperforms a flat baseline on PHALAR" would be accurate.
  • The introduction claims "broader applicability" and the conclusion calls the method "a general framework", but the evaluation covers only Lakh MIDI and FluidSynth-rendered accompaniment.

W4. No listening examples. The submission provides no demo page, audio examples, or supplementary material. For a generation paper whose subjective evidence is a 12-listener test, readers cannot judge the claimed timing improvements or the failure modes by ear. Anonymised paired examples (with/without realignment and the flat baseline, under both conditioning tasks) would help.

Questions for authors
  1. Baseline identity. Is "AMT" in Table 1 the Vanilla model of Appendix G.2 (same representation, 10 s lookahead) rather than the original AMT (arrival-time triplets, δ = 5 s)? If so, please rename it and state exactly which AMT mechanisms are kept.
  2. Symbolic PHALAR row. Which task and how many songs does it cover? Please report per-system completion and exclusion counts, and results on the common set of windows valid for all systems.
  3. Flat + musical-time control. Please add a flat Vanilla Transformer with causal token-level musical-time RoPE. If the hierarchy plus realignment remains clearly better, or offers a measured efficiency advantage, I would raise my score.
  4. Lookahead sweep. Without retraining, can you report PPLt at inference lookaheads of 5/10/20 s for all models? A gain that grows with the order/time displacement would directly support the conflict hypothesis.
  5. Audio baseline and PHALAR calibration. Please add an audio "w/o" baseline with absolute time embeddings, PHALAR scores for mismatched or time-shifted pairs as calibration anchors, and song-level bootstrap CIs.

I would raise my score to 3 if Q3/Q4 show benefits beyond the flat model that grow with displacement, or if a listening test against the flat baseline favours the proposed model.

Minor issues

  • Eq. 6 requires strictly monotone coordinates, but under anchored conditioning several events share the same t̃ (Eq. 24). The tie-breaking rule is stated differently on p.3 and p.14.
  • OMCAT's RoTE is applied to feature embeddings, whereas here it is applied to the main network's Q/K. Please state the difference.
  • Per-model parameter counts are not given, and KV heads differ (4 vs. 6).
  • There is a duplicated sentence in H.1, "conditioning,the" on p.7, and "transcribed" is used loosely in the Fig. 4 discussion.
  • Related work: Moonbeam (Guo & Dixon, 2025, arXiv) also uses onset time as a RoPE coordinate for symbolic music and is worth discussing. It is arXiv-only, so I do not count its absence against the paper.

Overall. Realignment is clearly effective within the proposed hierarchical model, and the mechanism is clean. However, the general claim about the order/time conflict, and the need for the hierarchy, are not established against a flat model that already matches the timing prediction without realignment. The advantage over that baseline rests on a single under-specified metric.

6 comments
AuthorsOpenReview ↗

Response to Reviewer rW8T (1/3)

We thank the reviewer for an exceptionally careful review; the requested controls changed the paper. We retrained all symbolic models under identical settings and added the flat Transformer with token-level musical-time RoPE that the reviewer asked for, the lookahead sweep (as a new analysis of span infilling, Table 2), FMD and statistical tests, a new listening test that includes both flat models, compute measurements, and an updated demo page. We also revised the claims that went beyond the evidence. The revised PDF has been uploaded; a version with all changes marked (additions in blue, deletions in red) is provided as supplementary material. Point-by-point answers follow.

Naming in the revision: H-Net is our hierarchical model and Vanilla a flat decoder-only Transformer, each trained with and without temporal realignment; Vanilla w/o realignment is the baseline previously labeled "AMT". For Vanilla, realignment places every token at the musical time in force at that token.

Flat Transformer with token-level musical-time RoPE. We added this control (Vanilla): each token's rotary position is the musical time in force at that token, i.e., the time of the most recent TIME token read up to and including it, so tokens of a time group share a position and their order is left to the causal mask. All four symbolic models (H-Net and Vanilla, each with and without realignment) were retrained with identical data and settings, and the baseline previously labeled "AMT" is now Vanilla w/o realignment. The reviewer is right that realignment does not require the hierarchy: it also improves the flat model's timing prediction, and we removed the claim that realignment cannot be applied at the token level. We now present realignment as architecture-agnostic and the hierarchy as an efficiency choice. On accompaniment, H-Net matches both flat models in PHALAR and FMD, with no significant difference, at only about a quarter of the forward FLOPs, while the flat models keep a lower overall PPL, which we state. The efficiency advantage is also measured in wall-clock time (Appendix G.2): training took 12.3 instead of 21.4 GPU-hours, and generation runs about 1.5× faster (159 vs. 109 tokens/s when filling a 4,096-token window on one H100 at batch size 1), although both savings are smaller than the FLOPs saving.

ModelForward GFLOPs per 4k tokensPPLPPL_tPHALARFMD
H-Net3751.771.795.3743.7
H-Net w/o realignment3751.912.383.7450.1
Vanilla1,4501.701.735.4942.3
Vanilla w/o realignment1,4501.701.805.3342.2

Lookahead sweep. We report it as a new analysis of span infilling, where the model fills masked spans between given segments, split by position within each masked span, with lookaheads of 5, 10, and 20 s at inference and no retraining (Table 2). At the start of a masked span, the model has just read the given music and the conditioning placed ahead of it, and must decide when the first notes enter. There, the time-token perplexity of models without realignment rises at the longest lookahead, whereas with realignment it stays nearly constant; for the flat model, the gain from realignment at the start of a masked span grows with the displacement. In the middle and at the end of a masked span, where the preceding target notes are available, the differences are small, and averaged over all tokens the effect is diluted (Table 8 in the appendix), which is why the overall PPL hides it. We believe this is the direct evidence for the order/time conflict that the reviewer asked for, and it holds beyond the hierarchical model.

Time-token perplexity by position within the masked span and by lookahead (Table 2):

PPL_tStart, Δ=5 s10 s20 sMiddle, Δ=5 s10 s20 sEnd, Δ=5 s10 s20 s
H-Net2.122.122.141.861.831.801.821.811.83
H-Net w/o realignment6.556.248.282.422.402.402.402.352.42
Vanilla2.112.082.121.861.831.801.831.811.81
Vanilla w/o realignment2.492.522.981.931.901.891.861.871.89
AuthorsOpenReview ↗

Response to Reviewer rW8T (2/3)

Evidence for an advantage over the flat baseline. Symbolic PHALAR is computed on accompaniment, 200 test windows of 30 s, as now stated in Appendix G.4. Every model completed all 200 items of both tasks (1,598 of the 1,600 generations on the first attempt, the rest on the second), and all models are scored on the same 955 five-second windows (of 1,000), so retention does not differ across systems. Song-level paired bootstrap intervals for PHALAR (10,000 resamples) and a paired permutation test for FMD show a significant difference from H-Net only for H-Net w/o realignment; the flat models do not differ significantly from it:

Difference from H-NetPHALAR, 95% bootstrap CIFMD, permutation p
H-Net w/o realignment−1.63 [−1.88, −1.38]+6.4 (p ≤ 0.001)
Vanilla+0.12 [−0.03, 0.27]−1.4 (p = 0.16)
Vanilla w/o realignment−0.04 [−0.22, 0.15]−1.4 (p = 0.17)

The gap between our model and the flat baseline observed in our original submission (5.57 vs. 4.27) therefore does not hold in the retrained comparison. Two things changed since our original submission: training now includes span conditioning with equal task weights, and inference presents every window as a training window, including the music that follows it. The flat baseline improved the most, while our model changed little, and we cannot attribute the change to either factor alone. We removed the claim that our model outperforms the baseline in generation quality.

The new listening test (Table 3) includes both flat models. In span infilling, removing realignment from H-Net lowers both criteria significantly, H-Net ties Vanilla, and it is rated significantly smoother than Vanilla w/o realignment. In accompaniment, Vanilla receives the highest ratings, but no difference from H-Net is significant after correction.

Mean ratings with 95% confidence intervals; in parentheses, the paired difference from H-Net; † Holm-adjusted p < 0.05 (Table 3):

TaskModelSmoothnessSoundness
AccompanimentH-Net3.90 ± 0.263.35 ± 0.37
H-Net w/o realignment3.83 ± 0.30 (−0.07)3.39 ± 0.36 (+0.04)
Vanilla4.08 ± 0.21 (+0.19)3.75 ± 0.33 (+0.40)
Vanilla w/o realignment3.85 ± 0.31 (−0.04)3.65 ± 0.37 (+0.29)
SpanH-Net3.34 ± 0.273.17 ± 0.31
H-Net w/o realignment1.98 ± 0.22 (−1.36†)2.38 ± 0.38 (−0.80†)
Vanilla3.38 ± 0.28 (+0.04)3.38 ± 0.30 (+0.21)
Vanilla w/o realignment2.87 ± 0.33 (−0.47†)3.21 ± 0.31 (+0.04)

Claims broader than the evidence; audio comparison. We revised the abstract (the listening claim is now restricted to span infilling, where it is significant), the introduction ("consistently outperforms" and "broader applicability" removed), Section 4.1 (no "state-of-the-art" baseline or "outperforms AMT"), and the conclusion (no "general framework"). For audio conditioning, we agree that time-token perplexity changes little and that the model without realignment receives no timing information for the audio latents, so the comparison contrasts shared timestamps with none rather than alternative ways of injecting time. We now state this limitation in Section 4.2, treat the audio results as supporting evidence, and focus the analysis of the paper on symbolic conditioning, where all of the controls above apply. We therefore did not add the absolute-time audio baseline or calibration anchors for the audio experiment. For symbolic accompaniment, PHALAR now comes with the song-level intervals above, and our scorer also reports, on the same 955 windows, a mismatched-pair floor (each generation re-paired with another song's conditioning) of 0.85 and the human ceiling of 5.98.

AuthorsOpenReview ↗

Response to Reviewer rW8T (3/3)

Listening examples. The demo page at https://listening-examples-iclr-2027.vercel.app now contains, for every clip of both tasks, the outputs of all four symbolic models and the original.

Significance in light of the new results. We want to state plainly what the retrained comparison changes: our model no longer outperforms the flat baseline in generation quality, and we removed that claim. We believe the significance of the paper rests on three contributions that the new results support. First, the formal framework: delta conditioning and the distinction between ordering time and musical time describe a family of in-sequence conditioning schemes, including prefix, block-wise, and anticipation conditioning (Appendix A), and show that moving conditioning ahead of its musical time makes sequence order and musical time disagree; the span analysis (Table 2) shows the effect where the framework locates it, at the start of each masked span, where models without realignment degrade at the longest lookahead. Second, realignment is effective at different levels and in different architectures: applied to tokens in a flat Transformer and to time groups in the hierarchical model, it improves timing prediction in both, and in the hierarchical model it is also crucial for generation quality (PHALAR, FMD, and the listening test on span infilling). Third, the hierarchy improves efficiency at little cost in quality: with realignment, H-Net stays close to Vanilla, with a slightly higher perplexity and no significant difference in PHALAR, FMD, or the listening ratings, while it needs only about a quarter of the forward FLOPs, trains in 12.3 instead of 21.4 GPU-hours, and generates about 1.5× faster (159 vs. 109 tokens/s when filling a 4,096-token window on one H100 at batch size 1). The revised introduction and conclusion state the contributions in these terms.


Reply to questions

Baseline identity. Yes, it was the Vanilla model trained on our representation with the 10-s lookahead, not the original AMT (arrival-time triplets, δ = 5 s). We renamed it Vanilla w/o realignment and describe it as a flat Transformer with anticipation-style (delta) conditioning on our representation; it keeps AMT's interleaving of controls ahead of their sounding time, but not its tokenization.

The other questions, on the symbolic PHALAR row, the flat control with musical-time RoPE, the lookahead sweep, and the audio baseline and PHALAR calibration, are answered in the responses above.

Minor fixes

Per-model parameter counts are now given in Appendix G.2 (132.9M for H-Net and 134.0M for Vanilla), together with their key/value heads (4 and 6). The tie-breaking rule is now stated once in Section 2.2: groups that share an ordering time are ordered by a rule specified by the conditioning scheme, which must place a conditioning group before every target group that depends on it; the mono-constraint allows conditioning groups to share an ordering time and keeps them in musical-time order, which covers anchored and block-wise conditioning, and Appendix A states the anticipation case accordingly. Appendix F now states the difference from OMCAT, which applies RoTE to feature embeddings, whereas we rotate only the queries and keys of the main Transformer. Moonbeam is discussed in Section 5, the paragraph with "conditioning,the" was rewritten, and "transcribed" in the attention-map discussion (now Appendix H) was replaced. We could not locate the duplicated sentence in Appendix H.1 in our source and would appreciate a pointer if it persists in the revision.

Reviewer_oohpOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

This paper adopts the H-Net architecture for symbolic music generation. A music sequence is tokenized as (instrument, pitch, duration) groupings, followed by a time-position token (10ms resolution) after each concurrent note grouping. This forms a hierarchical structure of token -> individual note -> concurrent onset, which effectively fits a two-stage, recursive H-Net architecture, while the music tokenization informs note-level and onset-level boundaries without the need to train a router. This paper also adds RoTE, a positional embedding that encodes absolute timing, to reflect the actual musical timing order at the concurrent-onset level (i.e., "realignment"). In a decoder-only language model that takes a 1-D flattened sequence input, conditional tokens can be interleaved before the target tokens that occur at an earlier time, while this embedding can retain the correct temporal relationship. Two generative tasks are studied to evaluate the proposed model: 1) symbolic accompaniment generation given melody and 5s prompt; 2) symbolic drum generation given audio accompaniment.

Critical strengths
  • This forms a hierarchical structure of token -> individual note -> concurrent onset effectively fits a two-stage, recursive H-Net architecture, while the music tokenization informs note-level and onset-level boundaries without the need to train a router. This is an interesting hierarchical music encoding solution, which offers potential insights for follow-up research in the domain of symbolic music generation.
Critical weaknesses

While I appreciate the idea of hierarchical symbolic music modelling with H-Net, the paper appears somewhat underdeveloped in its writing, organization, and evaluation.

Regarding the paper organization, there should be more formal discussions of the model architecture, the motivation for adopting H-Net, and the specialized design made here. The use of RoTE seems quite a straightforward trick, but the paper spends two entire sections formalizing it. The current Sections 2 and 3 contribute little to readers' understanding of the technical framework. Meanwhile, the paper seems to assume readers have substantial knowledge of H-Net, making it hard to appreciate what's happening here without referring to the original H-Net paper.

Regarding evaluation, the current experiments appear insufficient in a few aspects.

  • For the accompaniment generation task, the H-Net model does not seem to substantially outperform the AMT baseline across a set of PPL metrics. The evaluation could potentially be improved by including more music-established metrics, such as groove consistency [1], FMD [2], etc. Baseline models such as AMT should also appear in the subjective evaluation.
  • The random conditioning task does not make musical sense. There are more reasonable tasks to consider: span inpainting, continuation, accompaniment without 5s prompt, etc. And for accompaniment generation, there are additional viable baseline models to compare with [3].
  • While it is a good attempt to study audio-conditioned symbolic music generation, the task of drum arrangement does not appear to be the most representative one. To better demonstrate the potential of the proposed model, it would be more convincing to evaluate the task of piano cover generation, with established baseline models [4] to compare with.
  • No listening sample is provided.

Additional evaluation results with appropriate metrics and extended evaluation tasks are provided. The evaluation results make sense. A demo page is also provided.

[1] S.-L. Wu, et al. "The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures," in ISMIR 2020.

[2] J. Retkowski, et al. "Frechet music distance: A metric for generative symbolic music evaluation." arXiv preprint arXiv:2412.07948 (2024).

[3] J. Zhao, et al. "Structured multi-track accompaniment arrangement via style prior modelling," in NeurIPS 2024.

[4] C.-P. Tan, et al. "Picogen2: Piano cover generation with transfer learning approach and weakly aligned data," in ISMIR 2024.

Questions for authors

NA

4 comments
AuthorsOpenReview ↗

Response to Reviewer oohp (1/2)

We thank the reviewer for the detailed assessment. We took the evaluation concerns seriously: we retrained all symbolic models, added a span-infilling task, FMD, statistical tests, the flat baseline in a new listening study, and a demo page. The revised PDF has been uploaded; a version with all changes marked (additions in blue, deletions in red) is provided as supplementary material. Point-by-point answers follow.

Naming in the revision: H-Net is our hierarchical model and Vanilla a flat decoder-only Transformer, each trained with and without temporal realignment; Vanilla w/o realignment is the baseline previously labeled "AMT". For Vanilla, realignment places every token at the musical time in force at that token.

Writing and organization (architecture, H-Net background). We revised Section 3.3 to describe the model without assuming familiarity with H-Net: grouping (a causal encoder whose outputs are kept at the closing positions of note and time groups), ungrouping (broadcast of each group state to the following positions plus a learned residual from the encoder), and the composed token → note → time flow are now given as equations (Eqs. 18–20), and we state which H-Net components are not needed because the representation provides the boundaries (no router, smoothing, or ratio loss). Figure 3 now also shows the token representation (notes and time groups) alongside the pipeline, and Appendix F gives the full architecture. We kept the formalization of delta conditioning in Sections 2–3 because the new Table 2 tests its prediction directly: the effect of the displacement between sequence order and musical time grows with the lookahead.

Accompaniment: H-Net does not substantially outperform AMT in PPL. We agree, and we now say so explicitly. In the revised comparison (Table 1), the flat Transformer reaches a lower overall PPL (1.70 vs. 1.77), while H-Net and the flat models do not differ significantly in generation quality (PHALAR, FMD) and H-Net needs only about a quarter of the forward FLOPs. We also added the flat Transformer with realignment, which shows that the time-token improvement from realignment is not specific to our architecture (PPL_t 1.80 → 1.73 for the flat model, 2.38 → 1.79 for H-Net).

More established metrics (groove consistency, FMD). We added FMD (Retkowski et al., 2024; CLaMP 2 embeddings) to Table 1, with a paired permutation test (Appendix G.4): removing realignment from H-Net increases FMD by 6.4 (p ≤ 0.001), and H-Net does not differ significantly from the flat models (p ≥ 0.16). Groove consistency, however, requires beat and bar positions, which our representation does not contain: onsets are quantized to an absolute 10 ms grid without metrical structure, so we use FMD as the additional established metric.

AMT should appear in the subjective evaluation. The new listening study (Table 3) includes the flat baseline (Vanilla w/o realignment, the model previously labeled AMT) as well as the flat model with realignment. Listeners rate every version on smoothness and soundness on a 5-point scale, for accompaniment and for span infilling. In span infilling, removing realignment from H-Net lowers both criteria significantly, H-Net performs on par with the realigned flat model, and it is rated significantly smoother than the flat baseline. In accompaniment, no difference between H-Net and the flat models is significant after correction.

Mean ratings with 95% confidence intervals; in parentheses, the paired difference from H-Net; † Holm-adjusted p < 0.05 (Table 3):

TaskModelSmoothnessSoundness
AccompanimentH-Net3.90 ± 0.263.35 ± 0.37
H-Net w/o realignment3.83 ± 0.30 (−0.07)3.39 ± 0.36 (+0.04)
Vanilla4.08 ± 0.21 (+0.19)3.75 ± 0.33 (+0.40)
Vanilla w/o realignment3.85 ± 0.31 (−0.04)3.65 ± 0.37 (+0.29)
SpanH-Net3.34 ± 0.273.17 ± 0.31
H-Net w/o realignment1.98 ± 0.22 (−1.36†)2.38 ± 0.38 (−0.80†)
Vanilla3.38 ± 0.28 (+0.04)3.38 ± 0.30 (+0.21)
Vanilla w/o realignment2.87 ± 0.33 (−0.47†)3.21 ± 0.31 (+0.04)
AuthorsOpenReview ↗

Response to Reviewer oohp (2/2)

Random conditioning does not make musical sense; span inpainting, continuation, accompaniment without prompt. We added span infilling: after a 2-s prompt, every part is generated in masked spans of about 2 s between given segments. It is used in the listening study and in a new analysis by position within the span and by lookahead (Table 2), where it exposes the order/time conflict most directly: at the start of each masked span, the time-token perplexity of models without realignment rises at the longest lookahead, whereas realigned models stay nearly constant. We keep random conditioning in Table 1 for continuity with the submission (pooled with accompaniment in PPL).

Additional accompaniment baseline [3] and piano cover generation [4]. We agree that these would broaden the evaluation. Our goal is to isolate the effect of temporal realignment under in-sequence conditioning, so the revision prioritizes controlled comparisons in which only the attention coordinates (and the architecture) change, on the same data, tokenization, and training budget. Systems such as [3] differ in representation and training pipeline, so differences would not be attributable to realignment. For audio conditioning, we chose drum generation because drums depend most strongly on precise timing against the accompaniment, which is the property realignment targets.

No listening samples. Anonymized paired examples are now available at https://listening-examples-iclr-2027.vercel.app.

Significance in light of the new results. We want to state plainly what the retrained comparison changes: our model no longer outperforms the flat baseline in generation quality, and we removed that claim. We believe the significance of the paper rests on three contributions that the new results support. First, the formal framework: delta conditioning and the distinction between ordering time and musical time describe a family of in-sequence conditioning schemes, including prefix, block-wise, and anticipation conditioning (Appendix A), and show that moving conditioning ahead of its musical time makes sequence order and musical time disagree; the span analysis (Table 2) shows the effect where the framework locates it, at the start of each masked span, where models without realignment degrade at the longest lookahead. Second, realignment is effective at different levels and in different architectures: applied to tokens in a flat Transformer and to time groups in the hierarchical model, it improves timing prediction in both, and in the hierarchical model it is also crucial for generation quality (PHALAR, FMD, and the listening test on span infilling). Third, the hierarchy improves efficiency at little cost in quality: with realignment, H-Net stays close to Vanilla, with a slightly higher perplexity and no significant difference in PHALAR, FMD, or the listening ratings, while it needs only about a quarter of the forward FLOPs, trains in 12.3 instead of 21.4 GPU-hours, and generates about 1.5× faster (159 vs. 109 tokens/s when filling a 4,096-token window on one H100 at batch size 1). The revised introduction and conclusion state the contributions in these terms.

Thank you for your response to my questions. I appreciate the authors' effort in including the new experimental results and the demo page. The results reinforce the positive impact of temporal realignment, while the plain architectural difference between H-Net and Flat seems less significant in the music generation tasks explored. These experiments may offer more insights for follow-up research in this domain, and I would like to raise my rating from 1 to 3.

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you very much for your follow-up and for raising your rating. Your suggestions, including span infilling, FMD, a listening test with the baselines, and a demo page, have substantially improved the paper. We agree that, with realignment, the architectural difference between H-Net and the flat Transformer matters little for generation quality in the tasks we explored; the revision states this and presents the hierarchy mainly as a way to reach comparable quality at a lower computational cost. We are glad that the results may offer insights for follow-up research in this domain.

Reviewer_NeQrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new formulation of autoregressive and in-sequence conditional modelling for symbolic music. It models a note sequence in different hierarchies: tokens (e.g., pitch, instrument) are first grouped into notes using a Mamba-2 encoder. An attention encoder then groups notes that share the same onset into time groups. These hidden states are input to a transformer using musical time instead of position indices in RoPE to capture relative musical timing. The transformer outputs are then fed to an attention then a Mamba-2 decoder to decode hidden states back to tokens, modelled using the next-token prediction objective. The results indicate that under the new formulation, the model can still be optimized effectively using the next-token objective while preserving musical timing. A listening test validates the effectiveness of realignment using musical timing.

Critical strengths
  1. A novel formulation of hierarchical music sequence modeling. The paper also uses an H-Net-style transformer along with Mamba encoders and decoders. This is the first of its kind in symbolic music to the best of my knowledge.
  2. Instead of direct tokenization followed by auto-regressive modeling, the paper uses a hierarchical representation (i.e., tokens, group tokens with same musical onsets). This is well motivated and makes sense musically.
  3. The results show that realignment using musical timing for conditional generation is crucial. Although prior papers have shown similar findings, this re-iterates that although symbolic music can be tokenized similarly to text, adding music-related inductive biases is beneficial.
  4. The paper is overall well-written and easy to follow.
Critical weaknesses

Weakness [Addressed during rebuttal see responses below]:

  1. Prior works have already studied the use of autoregressive sequence order and musical timing. For example, Guo et al., AAAI 2023, show the effectiveness of using musical timing over token indices. Guo & Dixon, 2025 use a similar type of compressing (i.e., tokens to group), auto-regressive modeling, then decoding using GRU, similar to the Mamba decoding in this paper. They also place conditioning events before the target and use relative musical timing, along with other relative music attributes (e.g., pitch intervals). That is essentially the concept of "model-level temporal realignment" in this paper. The contribution of the findings is hence weakened and relevant works need to be cited.

[1] Guo et.al, A domain-knowledge-inspired music embedding space and a novel attention mechanism for symbolic music modeling, AAAI 2023. [2] Guo and Dixon, Moonbeam: A MIDI foundation model using both absolute and relative music attributes.

  1. The paper mentions AMT "does not explicitly distinguish autoregressive order from musical-time order". But AMT encodes each event's absolute musical time in the tokens, which is how it lets the model reorder events. Also, the proposed method does not outperform AMT in Table 1 besides the onset tokens. The listening test also only compares models with or without re-alignment without comparing with AMT.

I am happy to raise my rating if the authors can add a. an additional listening test, b. add necessary discussions about Table 1, and c. related works on musical timing vs token indices.

Other Suggestions:

  1. Define what is exactly in a group earlier in Section 3. It's now only introduced later in Section 3.3 and more details are in the Appendix. A first time reader like me have to flip back and forward to understand what does the hierachy represent. Moreover, Figure 5 in Appendix will assit the flow of the paper, I suggest move figure 5 to Section 3.3. To make up for the space, Figure 4 can be moved to the Appendix maybe?
Questions for authors

Questions [Addressed during rebuttal see responses below]:

  1. Real differences between the proposed model and AMT on a higher-level? I understand the authors propose a new formulation and modeling method. But on a higher-level, both reorder the sequence such that the conditioning tokens appear before the tokens to the generated right?
  2. Computation indication. The transformer now models a shorter sequence but more Mamba modules are added. Does this reduce the computation overall?
  3. There are more Mamba modules added, do you think the model is as scalable as vanilla transformers?
5 comments
AuthorsOpenReview ↗

Response to Reviewer NeQr (1/3)

We thank the reviewer for the careful reading and for stating concretely what would strengthen the paper. We have addressed all three points: (a) we ran a new listening test that includes the flat baseline (the model previously labeled AMT), both with and without temporal realignment, on accompaniment and on span infilling (Table 3); (b) we extended the perplexity comparisons: all symbolic models were retrained under identical settings, a flat Transformer with realignment was added, Table 1 is now discussed in Section 4.1, and a new analysis by position within the infilled span and by lookahead (Table 2) shows where realignment matters; (c) Section 5 now discusses the papers the reviewer pointed to, Guo et al. (2023) and Moonbeam (Guo & Dixon, 2025), on musical timing versus token indices. The revised PDF has been uploaded; a version with all changes marked (additions in blue, deletions in red) is provided as supplementary material. Point-by-point answers follow.

Naming in the revision: H-Net is our hierarchical model and Vanilla a flat decoder-only Transformer, each trained with and without temporal realignment; Vanilla w/o realignment is the baseline previously labeled "AMT". For Vanilla, realignment places every token at the musical time in force at that token.

Prior work on musical timing vs. token indices (Guo et al., 2023; Moonbeam). Thank you for pointing us to these works; both are now discussed in Section 5, and we agree that relating events by musical time rather than token indices is shared with them. Our work differs in its starting point and in its design.

These works use musical attributes to build better embeddings and attention for symbolic music in general. We start from a specific problem of in-sequence conditioning: when conditioning is placed ahead of the target in a single autoregressive sequence (delta conditioning), sequence order and musical time disagree, and the disagreement grows with how far ahead the conditioning is placed. Realignment restores musical-time coordinates while keeping the interleaved factorization order, and the new Table 3 tests this directly: at the start of an infilled span, the time-token perplexity of models without realignment rises at the longest lookahead, whereas with realignment it stays nearly constant.

In terms of design, both Guo et al. (2023) and Moonbeam are tied to a note-level compound (CP-like) representation, in which every token is a note carrying its own onset, pitch, and other attributes, and Moonbeam decodes each note's attributes with a GRU. Our architecture instead operates on a flat token sequence and only requires that the tokens can be grouped with boundaries identifiable from the generated prefix (a laminar group representation, Section 2.3): any two groups are nested or disjoint, never partially overlapping. Most symbolic music representations satisfy this, e.g., REMI-style event sequences, compound tokens, and AMT's arrival-time triplets, so the grouping stages and realignment apply to them without changing the representation. Representations with separate note-on and note-off events (MIDI-like encodings) do not: a note's offset event falls at a later time, after events of other notes, so the note's span partially overlaps other groups and the architecture cannot be applied directly.

AuthorsOpenReview ↗

Response to Reviewer NeQr (2/3)

AMT and absolute time; Table 1. We agree that AMT encodes the absolute onset time of every event, and we corrected the related-work sentence: the musical time of an anticipated control is available to AMT, but its attention still relates tokens by sequence position, so the temporal relation between interleaved controls and targets must be inferred from token content. Our realignment changes exactly this attention coordinate. Regarding Table 1, the revised table (symbolic part) reads:

ModelPPLPPL_tPHALARFMD
H-Net1.771.795.3743.7
H-Net w/o realignment1.912.383.7450.1
Vanilla1.701.735.4942.3
Vanilla w/o realignment (AMT-style)1.701.805.3342.2

We now discuss it in Section 4.1. Realignment lowers time-token perplexity for both architectures, and the flat models reach a lower overall PPL, which we state explicitly this time. In generation quality, H-Net does not differ significantly from either flat model (PHALAR: song-level bootstrap CIs include zero; FMD: paired permutation test, p ≥ 0.16), while it needs only about a quarter of the forward FLOPs (see the computation question below). We therefore no longer claim that our model outperforms the flat baseline; the hierarchy contributes to efficiency, and realignment contributes to the timing improvement.

We also analyzed perplexity on span infilling, where the model fills masked spans between given segments, by position within each masked span and by the lookahead used at evaluation (Δ = 5, 10, and 20 s, without retraining). Averaged over all tokens, the models differ little, but the time tokens at the start of a masked span expose the problem: there, the model has just read the given music and the conditioning placed ahead of it, and must decide when the first notes enter. Without realignment, their perplexity rises at the longest lookahead, whereas with realignment it stays nearly constant across lookaheads. This is the conflict between sequence order and musical time that motivates our work, and realignment removes it for both architectures. We added this analysis as Table 2 and discuss it in Section 4.1.

Time-token perplexity by position within the masked span and by lookahead (Table 2):

PPL_tStart, Δ=5 s10 s20 sMiddle, Δ=5 s10 s20 sEnd, Δ=5 s10 s20 s
H-Net2.122.122.141.861.831.801.821.811.83
H-Net w/o realignment6.556.248.282.422.402.402.402.352.42
Vanilla2.112.082.121.861.831.801.831.811.81
Vanilla w/o realignment2.492.522.981.931.901.891.861.871.89

Additional listening test including the flat baseline. We ran a new listening study (Table 3; protocol in Appendix G.5–G.6). Listeners rate every version on smoothness and soundness on a 5-point scale, for accompaniment and for span infilling, comparing H-Net, H-Net w/o realignment, Vanilla, and Vanilla w/o realignment (the flat baseline previously labeled AMT). In span infilling, where each generated segment must connect to the given music on both sides, removing realignment from H-Net lowers both criteria significantly, while H-Net performs on par with Vanilla and is rated significantly smoother than Vanilla w/o realignment. In accompaniment, Vanilla receives the highest ratings, but none of its differences from H-Net is significant after correction. Paired audio examples are now available at https://listening-examples-iclr-2027.vercel.app.

Mean ratings with 95% confidence intervals; in parentheses, the paired difference from H-Net; † Holm-adjusted p < 0.05 (Table 3):

TaskModelSmoothnessSoundness
AccompanimentH-Net3.90 ± 0.263.35 ± 0.37
H-Net w/o realignment3.83 ± 0.30 (−0.07)3.39 ± 0.36 (+0.04)
Vanilla4.08 ± 0.21 (+0.19)3.75 ± 0.33 (+0.40)
Vanilla w/o realignment3.85 ± 0.31 (−0.04)3.65 ± 0.37 (+0.29)
SpanH-Net3.34 ± 0.273.17 ± 0.31
H-Net w/o realignment1.98 ± 0.22 (−1.36†)2.38 ± 0.38 (−0.80†)
Vanilla3.38 ± 0.28 (+0.04)3.38 ± 0.30 (+0.21)
Vanilla w/o realignment2.87 ± 0.33 (−0.47†)3.21 ± 0.31 (+0.04)
AuthorsOpenReview ↗

Response to Reviewer NeQr (3/3)

Significance in light of the new results. We want to state plainly what the retrained comparison changes: our model no longer outperforms the flat baseline in generation quality, and we removed that claim. We believe the significance of the paper rests on three contributions that the new results support. First, the formal framework: delta conditioning and the distinction between ordering time and musical time describe a family of in-sequence conditioning schemes, including prefix, block-wise, and anticipation conditioning (Appendix A), and show that moving conditioning ahead of its musical time makes sequence order and musical time disagree; the span analysis (Table 2) shows the effect where the framework locates it, at the start of each masked span, where models without realignment degrade at the longest lookahead. Second, realignment is effective at different levels and in different architectures: applied to tokens in a flat Transformer and to time groups in the hierarchical model, it improves timing prediction in both, and in the hierarchical model it is also crucial for generation quality (PHALAR, FMD, and the listening test on span infilling). Third, the hierarchy improves efficiency at little cost in quality: with realignment, H-Net stays close to Vanilla, with a slightly higher perplexity and no significant difference in PHALAR, FMD, or the listening ratings, while it needs only about a quarter of the forward FLOPs, trains in 12.3 instead of 21.4 GPU-hours, and generates about 1.5× faster (159 vs. 109 tokens/s when filling a 4,096-token window on one H100 at batch size 1). The revised introduction and conclusion state the contributions in these terms.


Reply to questions

High-level difference from AMT. Both place conditioning tokens before the target tokens that depend on them; in fact, in our experiments the token sequence is the same (anticipation with a fixed lookahead, one member of the delta-conditioning family). The difference is what the model attends with: AMT, and our flat baseline, relate tokens by their sequence positions, so a conditioning event placed Δ seconds ahead appears "earlier" than the target it accompanies; with realignment, attention coordinates are the musical times, so the model sees conditioning and target events at their true temporal distance while the factorization order is unchanged. As shown by the span analysis above (Table 2), this matters most where the model must place the first notes after the conditioning, and the gap is largest at the longest lookahead. Realignment is also not tied to anticipation; it applies to any delta-conditioning scheme whose offsets are known.

Does the hierarchy reduce computation overall? Yes. Counting forward FLOPs per 4,096-token training window (causal attention), H-Net costs 375 GFLOPs versus 1,450 GFLOPs for Vanilla, at a similar parameter count (132.9M vs. 134.0M), i.e., about a quarter. The main Transformer runs on time groups (about 13% of the tokens), and the Mamba-2 stacks are linear in sequence length. The wall-clock saving is smaller than the FLOPs saving: training for 200k updates took 12.3 vs. 21.4 GPU-hours on the same cluster, and generation runs about 1.5× faster (159 vs. 109 tokens/s when filling a 4,096-token window on one H100 at batch size 1). These numbers are now in Appendix G.2.

Is the model as scalable as vanilla Transformers? We have not run scaling experiments, so we cannot claim it empirically. Architecturally, the quadratic attention cost applies only to the compressed time-level sequence, and the token-level Mamba-2 stacks scale linearly, so the cost advantage should grow with context length; H-Net (Hwang et al., 2026) reports favorable scaling on language data, but that does not transfer to music automatically. The practical limitation we observed is kernel efficiency: the hierarchy's FLOPs advantage (≈4×) translates into a smaller wall-clock advantage (≈1.7× in training and ≈1.5× in generation). We have added this caveat rather than a scalability claim.

Minor fixes

Following the reviewer's suggestions on presentation, Section 3 now opens by defining note groups and time groups in our representation, and Figure 3 contains a token-sequence inset that expands two time groups into their note groups and tokens; the attention maps formerly in Figure 4 moved to Appendix H (now Figure 6), with a one-sentence summary in Section 4.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.