- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper scales up the Anticipatory Music Transformer by revising its tokenization (TICK-relative onsets and block anticipation) and adopting a training recipe from recent language models (LLaMA-2-style architecture, Muon optimizer), trained on a larger open corpus (GigaMIDI and Aria-MIDI). It then fits data-constrained scaling laws (Muennighoff et al., 2023; Lovelace et al., 2026) to identify the loss-minimizing training configuration for symbolic music. To the best of my knowledge, this is the first scaling attempt for MIDI, although evaluation results show that no scaling curve suggested by Muennighoff or Lovelace et.al fit symbolic music training. This raises questions of whether symbolic can be scaled similar to text or a different scaling law for MIDI is needed. Several claims lack supporting experiments (e.g., long-form generation), and some contributions, such as block anticipation, are not clearly defined or evaluated in isolation.
Critical strengths
- An open training recipe for symbolic music transformers with model checkpoints, code, and fully open training data.
- First attempt to study the scaling behaviour of symbolic (i.e., multi-track MIDI) music transformer. The use of data-constrained scaling laws is clearly motivated. The authors also honestly report the conflicting scaling suggestions from Muennighoff and Lovelace et.al.
- Subjective and objective evaluation results are convincing, especially with a self-curated test set. The paper also shows that automated evaluation metrics deviate from human perceptions, suggesting new directions for automated evaluation for symbolic music.
Critical weaknesses
-
The unpredictable loss from data-constrained scaling law weakens the contributions, and raises questions of whether symbolic music is as scalable as text, at least from the loss-perpective. A more rigorous analysis following the methodology from Lovelace et.al. could be beneficial: Find residual between observed loss and Chinchilla prediction and observe the patterns. A higher-level concern from me is whether this paper will mislead the community towards scaling symbolic music transformers, which in my opinion is still premature. The title and abstract suggest scaling music transformers is possible, whereas the results show that the loss curve cannot be extrapolated using data-constrained scaling laws. A more detailed analysis will strengthen the paper's claim: For instance, whether the discrepancies stem from the different natures of symbolic music and text, or a different scaling formula can be proposed to predict the scaling behaviour.
-
There are several overclaimings in the paper but lack supporting experiments: (1) The introduction promises "we improve the generative modeling recipe ... from datasets, data mixtures, tokenization, model architecture, and optimizers." but the only ablation study conducted is on tokenisation (Table 1). (2) Section 3.2 last sentence: expressive timing generated from AMT might come from the training data instead of the relativized tokenization. (3) The paper mentions: "intriguing long-context generative capabilities", yet only one midi visualization is shown in Figure 5 to support this claim. (3) Table 2 has too many confounding variables and whether the speed-up is from the tokeniser or the optimiser is unclear. More detailed ablation studies are needed.
-
Although both subjective and objective evaluations show that AMT-2 outperforms AMT-1, the improvement might come from the larger and more diverse training data. The benefits of scaling are unclear besides decreasing likihood metrics. More qualitative examples comparing AMT-1 and AMT-2 will be helpful, e.g., better generalization or musicality.
Questions for authors
- The new tokenization uses a “TICK” token, similar to a “Bar” token, which can theoretically generate music infinitely. Can AMT-2 generate a coherent full-song-length MIDI sequence?
- The appendix explains why the time resolution is reduced from 100 to 96. Have the authors verified whether this is necessary?
8 comments
Additional AMT-2 Samples and Comparison to AMT-1 (1/2)
We thank the reviewer for helpful comments regarding the outputs of AMT-2. The reviewer notes: “(1) only one midi visualization is shown… to support [long context capabilities]”, that “(2) more qualitative examples comparing AMT-1 and AMT-2 will be helpful”, and the question of, “(3) can AMT-2 generate a coherent full-song-length MIDI sequence?” We hope to answer all of these by presenting more samples. Thus the following two comments address Critical Weakness 2 part 3 and Critical Weakness 3.
The supplement contained a curated set of:
- 1 BEAT comparison, 6 Lakh Test comparisons, and 4 Suno comparisons for a total of 22 audio files, each 25 seconds.
We have produced twenty more comparisons using the same procedure of five seconds of conditioning from our Lakh Test split. We compare three models:
- AMT-1 Medium at 800k steps (the most capable)
- AMT-2 1.14B at 372k steps (7 epochs) (our large model at the time of submission)
- AMT-2 3.4B at 411k steps (~17 epochs) (our latest model, currently training)
We to each point:
- (1 and 3), AMT-1 degenerates into pathology earlier than AMT-2. AMT-2 sustains coherence longer than AMT-1, but can demonstrate pathology.
- (2), AMT-2 has more diverse part writing and instrumentation. It more effectively develops musical sections.
- Generally, some Lakh MIDI prompts elicit a musically uninteresting response for all models.
What we write below is subjective. Please find the full uncurated set of MIDI at this url if the reviewer is interested: LINK. Each sample name is hyperlinked to a set of audio containing scrubbable samples. All is anonymous. We voice the MIDI with Garageband.
🎶 f5b43b673dc21317747352a3f6d09bdb. For this prompt, the only conditioning information is a single drum hit. It is essentially unconditional autoregressive sampling.
- AMT-1: Emits a plausible continuation from 0:05 - 0:30. Meanders a bit with very sparse instrumentation from 0:30-0:56. Begins to pathologically repeat itself at 2:16
- AMT-2 1.14B: From 0:00 - 0:26, vamps before developing a new section at 0:40. At 1:13, the model writes a fun turnaround but then seems to end the piece. Material thereafter is pathological.
- AMT-2 3.4B: From 0:00 - 0:06 is the comping section of a bossa nova. The classical guitar solos from 0:07 - 0:28. The timing of its runs is more expressive than AMT-1. At 0:27, the piano begins to solo. At 0:44, the end of the phrase introduces trumpets and trombones. At 1:20, the model seems to wish to end the piece.
🎶 f786750038e5309843ef063c1b6320ed. The prompt has a guitar that awkwardly noodles around the C minor pentatonic. We think it’s interesting to see how each model responds to the awkward conditioning.
- AMT-1: 0:06 - 0:33, somewhat plausible material albeit uneventful. The guitar part never returns. After 0:33 seconds, the model degenerates into repetition.
- AMT-2 1.14B: 0:05 - 0:34, the model continues to play in this guitar part matching the ‘level’ of playing. It develops some weak phrasing and retains the impression of a beginner musician. At 0:37 onward, the backing instruments become the focus.
- AMT-2 3.4B: 0:05 - 0:22, the solo guitar phrasing is stronger than the 1.14B model. The model retains the beginner-like style and develops the material more convincingly at 0:22 to 0:46. Then at 0:47, a new section develops and adds a string ensemble.
🎶 f3adfb192fe9b9b91b2966a3fd16c36c
- AMT-1: Stylistically consistent with the prompt. It writes its own material and repeats it, the same behavior we highlight in Figure 5 of our manuscript. The prompt and its continuation are for solo piano, so the number of token per second is low. We argue that increasing the context to 4096 or 8192 in AMT-2 enables this same behavior at higher token densities.
- AMT-2 1.14B: 0:53, the model attempt a ritard near the end of the phrase, but this becomes the new tempo thereafter.
- AMT-2 3.4B: The continued material is stylistically inconsistent with the prompt. However, it produces a section spanning 0:14 to 2:06, and then repeats it. This is evidence of form over a musically plausible duration.
to be continued in the next comment
Additional AMT-2 Samples and Comparison to AMT-1 (2/2)
We will finally highlight two longer samples that do not have a 'head-to-head' comparison with AMT-1. Both are generated by AMT-2 1.14B, the model at the time of submission.
🎶 fe5cdc20d541a754e9038263d0ab78f0
- We enjoy this sample for its peaceful atmosphere and tasteful harmony. One may imagine it fitting an open-world, exploration-style video game.
🎶 f5b787297f8637e985994adc8a9630a9
- We find this sample endearing and light-hearted. Throughout, the interplay between all three instruments is engaging and idiomatic. We think the material is strongest from 0:05 to 0:52.
Critical Weakness 1: Scaling Law Results
We greatly appreciate the reviewer’s careful review and have found their comments have improved our work considerably. We first address Critical Weakness 1, the Scaling Law result, and will later post responses that address their questions and other stated critical weaknesses.
We agree that the current framing over-credits the scaling laws. The totality of our interventions, including scale, helped produce AMT-2, but the data-constrained laws we had tested did not reliably predict by how much. We will revise the abstract, introduction, and conclusion to separate these claims: we will attribute AMT-2's gains to scale and our recipe changes rather than to the scaling-law fit and state the extrapolation miss plainly rather than as a caveat. We do not intend for our work to “mislead the community” to scale. If anything, our results suggest that data still severely limits present day symbolic music models, even with contemporary techniques.
We followed the reviewer’s helpful suggestion to “Find residual between observed loss and Chinchilla prediction” and it was the most informative analysis in our retrospective. We cut the in-sample Chinchilla residual for the largest model and data budget from 0.094 to 0.007 nats and the re-fit Chinchilla extrapolation lands within 0.025 nats for our held-out 7 epoch run, and a new 3.39B run (currently training, 17/20 epochs). Further details are in comment 2/6 of our joint scaling response.
Whether “symbolic music is as scalable as text” is a fascinating question, but our experiments cannot answer it. The Muennighoff et al. data-constrained scaling laws were defined originally on text, but Appendix section Q (pp. 41) states, “it may be of interest for future work to test the applicability of our findings … to different data modalities or model architectures”. The Lovelace et al. scaling laws are stated generally, though validated only on text. Our study examines whether these scaling law forms transfer to a new domain, but not how text and symbolic music compare. Several factors that could explain the extrapolation miss are not specific to music: the Chinchilla fit (comment 2/6), static hyperparameters (comment 4/6), and the functional form (comment 5/6). We cannot separate these from modality effects without a controlled comparison, which is unfortunately beyond our original research questions.
We agree that proposing a “different scaling formula” is an interesting direction, and we’ve since investigated the Symbolic Music Scaling Law (SMS Law) proposed by Qu et al., finding that it extrapolates within 0.001 nats on two held-out runs (
one of which is still training, so the result is preliminaryEDIT: the model has finished training. New details are in the updated PDF and summarized in the changelog: LINK). In comment 5/6 of our joint scaling response, we explain how we adapted their fitting procedure and why this had not been obvious at the time of submission. To facilitate research towards the reviewer’s suggestion of finding a “different scaling formula”, even beyond the SMS Law, our introduction states we will release our scaling law fit data. This fit dataset represents a total of ~3,863.5 GPU hours (161 GPU-days). To our knowledge, this will be the largest scaling law dataset for symbolic music.