acceptodds
Under review as a conference paper at ICLR 2027

Scaling Anticipatory Music Transformers

Abstract

We seek to build capable generative models for symbolic music at the largest scale supported by the limited data available. To this end, we apply recently proposed data-constrained scaling laws to symbolic music and conduct an empirical scaling study on an improved variant of the anticipatory music transformer. The study informs the training configuration for a 1.14 billion parameter model, which achieves state-of-the-art likelihood metrics on the Lakh MIDI dataset and exhibits strong performance in human evaluation, even against closed source commercial systems like Suno v6. The model demonstrates strong extrapolative performance on a new test set of music composed in 2026, and exhibits intriguing long-context generative capabilities. However, the model's final test loss diverges from the scaling prediction, raising methodological questions about data-constrained scaling laws, at least in the symbolic music domain. We release the model checkpoint, source code, scaling law runs, and the aforementioned 2026 test set, and emphasize that all training data are fully open.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
symbolic music, music generation, music generative models, scaling law, data constrained, data-constrained scaling law, sequence model, anticipation, tokenization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X38EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper scales up the Anticipatory Music Transformer by revising its tokenization (TICK-relative onsets and block anticipation) and adopting a training recipe from recent language models (LLaMA-2-style architecture, Muon optimizer), trained on a larger open corpus (GigaMIDI and Aria-MIDI). It then fits data-constrained scaling laws (Muennighoff et al., 2023; Lovelace et al., 2026) to identify the loss-minimizing training configuration for symbolic music. To the best of my knowledge, this is the first scaling attempt for MIDI, although evaluation results show that no scaling curve suggested by Muennighoff or Lovelace et.al fit symbolic music training. This raises questions of whether symbolic can be scaled similar to text or a different scaling law for MIDI is needed. Several claims lack supporting experiments (e.g., long-form generation), and some contributions, such as block anticipation, are not clearly defined or evaluated in isolation.

Critical strengths
  1. An open training recipe for symbolic music transformers with model checkpoints, code, and fully open training data.
  2. First attempt to study the scaling behaviour of symbolic (i.e., multi-track MIDI) music transformer. The use of data-constrained scaling laws is clearly motivated. The authors also honestly report the conflicting scaling suggestions from Muennighoff and Lovelace et.al.
  3. Subjective and objective evaluation results are convincing, especially with a self-curated test set. The paper also shows that automated evaluation metrics deviate from human perceptions, suggesting new directions for automated evaluation for symbolic music.
Critical weaknesses
  1. The unpredictable loss from data-constrained scaling law weakens the contributions, and raises questions of whether symbolic music is as scalable as text, at least from the loss-perpective. A more rigorous analysis following the methodology from Lovelace et.al. could be beneficial: Find residual between observed loss and Chinchilla prediction and observe the patterns. A higher-level concern from me is whether this paper will mislead the community towards scaling symbolic music transformers, which in my opinion is still premature. The title and abstract suggest scaling music transformers is possible, whereas the results show that the loss curve cannot be extrapolated using data-constrained scaling laws. A more detailed analysis will strengthen the paper's claim: For instance, whether the discrepancies stem from the different natures of symbolic music and text, or a different scaling formula can be proposed to predict the scaling behaviour.

  2. There are several overclaimings in the paper but lack supporting experiments: (1) The introduction promises "we improve the generative modeling recipe ... from datasets, data mixtures, tokenization, model architecture, and optimizers." but the only ablation study conducted is on tokenisation (Table 1). (2) Section 3.2 last sentence: expressive timing generated from AMT might come from the training data instead of the relativized tokenization. (3) The paper mentions: "intriguing long-context generative capabilities", yet only one midi visualization is shown in Figure 5 to support this claim. (3) Table 2 has too many confounding variables and whether the speed-up is from the tokeniser or the optimiser is unclear. More detailed ablation studies are needed.

  3. Although both subjective and objective evaluations show that AMT-2 outperforms AMT-1, the improvement might come from the larger and more diverse training data. The benefits of scaling are unclear besides decreasing likihood metrics. More qualitative examples comparing AMT-1 and AMT-2 will be helpful, e.g., better generalization or musicality.

Questions for authors
  1. The new tokenization uses a “TICK” token, similar to a “Bar” token, which can theoretically generate music
infinitely. Can AMT-2 generate a coherent full-song-length MIDI sequence?
  2. The appendix explains why the time resolution is reduced from 100 to 96. Have the authors verified whether this is necessary?
8 comments
AuthorsOpenReview ↗

Critical Weakness 1: Scaling Law Results

We greatly appreciate the reviewer’s careful review and have found their comments have improved our work considerably. We first address Critical Weakness 1, the Scaling Law result, and will later post responses that address their questions and other stated critical weaknesses.

We agree that the current framing over-credits the scaling laws. The totality of our interventions, including scale, helped produce AMT-2, but the data-constrained laws we had tested did not reliably predict by how much. We will revise the abstract, introduction, and conclusion to separate these claims: we will attribute AMT-2's gains to scale and our recipe changes rather than to the scaling-law fit and state the extrapolation miss plainly rather than as a caveat. We do not intend for our work to “mislead the community” to scale. If anything, our results suggest that data still severely limits present day symbolic music models, even with contemporary techniques.

We followed the reviewer’s helpful suggestion to “Find residual between observed loss and Chinchilla prediction” and it was the most informative analysis in our retrospective. We cut the in-sample Chinchilla residual for the largest model and data budget from 0.094 to 0.007 nats and the re-fit Chinchilla extrapolation lands within 0.025 nats for our held-out 7 epoch run, and a new 3.39B run (currently training, 17/20 epochs). Further details are in comment 2/6 of our joint scaling response.

Whether “symbolic music is as scalable as text” is a fascinating question, but our experiments cannot answer it. The Muennighoff et al. data-constrained scaling laws were defined originally on text, but Appendix section Q (pp. 41) states, “it may be of interest for future work to test the applicability of our findings … to different data modalities or model architectures”. The Lovelace et al. scaling laws are stated generally, though validated only on text. Our study examines whether these scaling law forms transfer to a new domain, but not how text and symbolic music compare. Several factors that could explain the extrapolation miss are not specific to music: the Chinchilla fit (comment 2/6), static hyperparameters (comment 4/6), and the functional form (comment 5/6). We cannot separate these from modality effects without a controlled comparison, which is unfortunately beyond our original research questions.

We agree that proposing a “different scaling formula” is an interesting direction, and we’ve since investigated the Symbolic Music Scaling Law (SMS Law) proposed by Qu et al., finding that it extrapolates within 0.001 nats on two held-out runs (one of which is still training, so the result is preliminary EDIT: the model has finished training. New details are in the updated PDF and summarized in the changelog: LINK). In comment 5/6 of our joint scaling response, we explain how we adapted their fitting procedure and why this had not been obvious at the time of submission. To facilitate research towards the reviewer’s suggestion of finding a “different scaling formula”, even beyond the SMS Law, our introduction states we will release our scaling law fit data. This fit dataset represents a total of ~3,863.5 GPU hours (161 GPU-days). To our knowledge, this will be the largest scaling law dataset for symbolic music.

AuthorsOpenReview ↗

Additional AMT-2 Samples and Comparison to AMT-1 (1/2)

We thank the reviewer for helpful comments regarding the outputs of AMT-2. The reviewer notes: “(1) only one midi visualization is shown… to support [long context capabilities]”, that “(2) more qualitative examples comparing AMT-1 and AMT-2 will be helpful”, and the question of, “(3) can AMT-2 generate a coherent full-song-length MIDI sequence?” We hope to answer all of these by presenting more samples. Thus the following two comments address Critical Weakness 2 part 3 and Critical Weakness 3.

The supplement contained a curated set of:

  • 1 BEAT comparison, 6 Lakh Test comparisons, and 4 Suno comparisons for a total of 22 audio files, each 25 seconds.

We have produced twenty more comparisons using the same procedure of five seconds of conditioning from our Lakh Test split. We compare three models:

  • AMT-1 Medium at 800k steps (the most capable)
  • AMT-2 1.14B at 372k steps (7 epochs) (our large model at the time of submission)
  • AMT-2 3.4B at 411k steps (~17 epochs) (our latest model, currently training)

We to each point:

  • (1 and 3), AMT-1 degenerates into pathology earlier than AMT-2. AMT-2 sustains coherence longer than AMT-1, but can demonstrate pathology.
  • (2), AMT-2 has more diverse part writing and instrumentation. It more effectively develops musical sections.
  • Generally, some Lakh MIDI prompts elicit a musically uninteresting response for all models.

What we write below is subjective. Please find the full uncurated set of MIDI at this url if the reviewer is interested: LINK. Each sample name is hyperlinked to a set of audio containing scrubbable samples. All is anonymous. We voice the MIDI with Garageband.

🎶 f5b43b673dc21317747352a3f6d09bdb. For this prompt, the only conditioning information is a single drum hit. It is essentially unconditional autoregressive sampling.

  • AMT-1: Emits a plausible continuation from 0:05 - 0:30. Meanders a bit with very sparse instrumentation from 0:30-0:56. Begins to pathologically repeat itself at 2:16
  • AMT-2 1.14B: From 0:00 - 0:26, vamps before developing a new section at 0:40. At 1:13, the model writes a fun turnaround but then seems to end the piece. Material thereafter is pathological.
  • AMT-2 3.4B: From 0:00 - 0:06 is the comping section of a bossa nova. The classical guitar solos from 0:07 - 0:28. The timing of its runs is more expressive than AMT-1. At 0:27, the piano begins to solo. At 0:44, the end of the phrase introduces trumpets and trombones. At 1:20, the model seems to wish to end the piece.

🎶 f786750038e5309843ef063c1b6320ed. The prompt has a guitar that awkwardly noodles around the C minor pentatonic. We think it’s interesting to see how each model responds to the awkward conditioning.

  • AMT-1: 0:06 - 0:33, somewhat plausible material albeit uneventful. The guitar part never returns. After 0:33 seconds, the model degenerates into repetition.
  • AMT-2 1.14B: 0:05 - 0:34, the model continues to play in this guitar part matching the ‘level’ of playing. It develops some weak phrasing and retains the impression of a beginner musician. At 0:37 onward, the backing instruments become the focus.
  • AMT-2 3.4B: 0:05 - 0:22, the solo guitar phrasing is stronger than the 1.14B model. The model retains the beginner-like style and develops the material more convincingly at 0:22 to 0:46. Then at 0:47, a new section develops and adds a string ensemble.

🎶 f3adfb192fe9b9b91b2966a3fd16c36c

  • AMT-1: Stylistically consistent with the prompt. It writes its own material and repeats it, the same behavior we highlight in Figure 5 of our manuscript. The prompt and its continuation are for solo piano, so the number of token per second is low. We argue that increasing the context to 4096 or 8192 in AMT-2 enables this same behavior at higher token densities.
  • AMT-2 1.14B: 0:53, the model attempt a ritard near the end of the phrase, but this becomes the new tempo thereafter.
  • AMT-2 3.4B: The continued material is stylistically inconsistent with the prompt. However, it produces a section spanning 0:14 to 2:06, and then repeats it. This is evidence of form over a musically plausible duration.

to be continued in the next comment

AuthorsOpenReview ↗

Additional AMT-2 Samples and Comparison to AMT-1 (2/2)

We will finally highlight two longer samples that do not have a 'head-to-head' comparison with AMT-1. Both are generated by AMT-2 1.14B, the model at the time of submission.

🎶 fe5cdc20d541a754e9038263d0ab78f0

  • We enjoy this sample for its peaceful atmosphere and tasteful harmony. One may imagine it fitting an open-world, exploration-style video game.

🎶 f5b787297f8637e985994adc8a9630a9

  • We find this sample endearing and light-hearted. Throughout, the interplay between all three instruments is engaging and idiomatic. We think the material is strongest from 0:05 to 0:52.
Reviewer_DhudOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

This paper demonstrates an empirical scaling study for symbolic music generation. Using the base model of Anticipatory Music Transformer (AMT), the IsoFLOP curves show the typical parabolic pattern under the low training budget domain, where the LMD data remain fresh (1.7B unique tokens x 4 epochs). The extrapolation to the high training budget domain, however, shows that under the parameter size equivalent to AMT-Large (1B), there is a significant gap between the optimal data scale and what can be offered by LMD. This motivates scaling AMT under the data-constrained scaling laws.

To proceed, the paper gathers additional MIDI data sources and reaches the unique token scale of UD=27.86BU_D=27.86\mathrm{B}. By varying model parameter size NN and number of epochs RDR_D, the data-constrained scaling laws aim to predict the loss LL by fitting a formula L=f(N,UD,RD)L = f(N, U_D, R_D). The fitted formula helps extrapolate to the 1B parameter scale and find the optimal data scale Dopt=UD×RDD_{\mathrm{opt}}=U_D\times R_D that produces the lowest loss. The paper thus trains AMT-2, with 1.14B model parameters under the scaling law. However, the model’s final test loss diverges significantly from the scaling prediction.

The AMT-2 model still achieves state-of-the-art BPS in the symbolic music generation literature. To facilitate further testing without teat/train contamination, this paper secures a testing dataset of symbolic music (via multi-instrument transcription from audio) released after June 2026. Objective evaluations on the test set show mixed results, while subjective human studies favour AMT-2. AMT-2 additionally revises the music tokenization from global absolute position to recurring local position in order to address the long-context constraints.

Critical strengths
  • This paper introduces AMT-2, a new-generation Anticipatory Music Transformer that addresses long-context generation constraints and is trained using the optimal data scale under the 1.14B parameter size, as informed by data-constrained scaling laws. The AMT-2 model shows significant improvement over AMT-1 and is competitive with existing large-scale symbolic music LM baselines.

  • The scaling experiment on symbolic music generation offers a substantial technical contribution.

Critical weaknesses
  • Despite the substantial technical effort, the idea of LM data-constrained scaling is not entirely novel, and the scaling results here appear to offer less sufficient or useful insights than expected. The model’s final test loss diverges significantly from the scaling prediction, which calls into question the reliability/confidence of the scaling laws acquired in this specific case study. One additional experiment might be interesting: training two additional 1.14B AMT-2 models using a magnitude lower or higher token count. It would be interesting to validate in this way if the model trained using the scaling law does lead to the optimal condition. Added experiments and discussions address the concern.

  • The writing of the paper appears in a rush and hard to follow. Many descriptions are rather convoluted and hard to parse. The current writing appears to assume readers have substantial prior knowledge of AMT and Lovelace et al. (2026), leaving minimal context for researchers in more general domains to follow smoothly. It also lacks illustration for several key terms. For example, Line 295 mentions "the value of EE" without first defining EE; the value of UDU_D only appears in the figure captions instead of being formally mentioned in the main body; The concrete values of two important parameters are not mentioned at all: δ\delta and tick\tt{tick}.

  • While the testing dataset collects post-2026 music and is sure to avoid test/train contamination, the MIDI could have inevitably introduced artifacts due to transcription errors. Besides more evident pitch errors, Muscriptors is prone to producing prolonged or overly short note durations, which bias the underlying data distribution. This may raise concerns about the dataset's practical utility and the reliability of evaluations conducted on it. Despite potential artifacts in the transcribed data, the evaluation is done with a fair comparison.

Questions for authors
  • In the implementation of AMT-2, what value does δ\delta take, and how long is a tick\tt{tick}? Question addressed.
  • Line 193: "We remedy this by first deciding a subset of instruments to anticipate, and then uniformly sampling a time > δ\delta in the current chunk and anticipating the subsequent tokens." This reads as rather convoluted. Does it mean δ\delta is a variable to be sampled, instead of a fixed param value, during the course of training? Also, it is still not entirely clear why AMT-1 cannot support infilling and accompaniment at the same time. Question addressed
  • In Figure 2, six stars appear to correspond to 2 small, 3 medium, and 1 large AMT-1 model. Which concrete training runs do these stars correspond to? Question addressed.
  • Line 310: "The Form E scaling law predicts this configuration should achieve a final test cross entropy loss of 0.7162. To contextualize this number, the Form E scaling law predicts that the best possible model for N ≤ 1.14billion is one that trains for 100 epochs, which it predicts would achieve a loss of 0.6812." These two Form E statements contradict each other. Typo? Question addressed.
  • Appendix A.1 interprets why 96 positional resolution might work better than 100. The reasoning is conditioned on 120 BPM, but in general, music comes in varying tempos. Does this 96 vs 100 resolution design really matter? Question addressed.
10 comments

We thank the reviewer for their careful review, and we apologize if the reviewer spent disproportionate effort in reading our work. We greatly appreciate the time invested and are actively working to improve the writing clarity. We will first address Critical weakness 1 and Question 4, which are closely related. In subsequent comments, we will address the other stated weaknesses and questions.

Regarding Critical Weakness 1: We agree that our study challenges the “reliability and confidence of the scaling laws” we use, and this is a central finding that we report. One question our work tries to answer is, do data-constrained scaling laws extrapolate in the seemingly appropriate setting of symbolic music? Unfortunately, our answer is that the laws tested at the time did not. Our abstract states that these scaling laws “inform the training configuration for a 1.14 billion parameter model” and in hindsight this language is underspecified. We indeed use the data-constrained scaling laws to choose the configuration that we ultimately run, but the result we obtain diverges from the scaling law’s prediction. We’ve since investigated a different scaling law form, the Symbolic Music Scaling Law (Qu et al.), which improves extrapolation to residuals within 0.001 nats on two held out runs (one run is still training, so the result is preliminary EDIT: the model has finished training. New details are in the updated PDF and summarized in the changelog: LINK). Details are within our joint response comment 5/6.

On novelty, we do not claim to have invented data-constrained scaling laws. What is novel is examining the Lovelace and Muennighoff scaling laws on symbolic music and reporting their limitations.

To address Question 4, Line 310 is not a typo, but it is a confusing sentence. The 7 epoch run is not the predicted optimum, it is the run we could feasibly train given our compute. Form E predicts this 7 epoch run should achieve a loss of 0.7162. As for the optimum, Form E predicts the best possible run of a 1.14B model should train for 100 epochs and achieve a loss of 0.6812. The point we were trying to make here is that, at the time, the scaling law predicted the optimal run is only 0.035 nats per token better than the run we can feasibly train; we judged 0.035 nats is non-negligible but not worth 93 more epochs. Thus, we note that contrary to the reviewer’s summary, AMT-2 was not trained at the then predicted scaling law optimum.

Relatedly, the suggested “order of magnitude higher token count” run would test exactly this prediction (Form E expects it to be better, ≈0.68\approx0.68). However, it would cost roughly 10x our main run, which is beyond our compute during the discussion period.

As for “less sufficient or useful insights than expected”, we have conducted an additional investigation based on reviewer comments that we believe yields additional insight. Specifically,

  • Discarding extremely low data-budget runs from the Chinchilla fitting grid helps extrapolation, and future practitioners could benefit from investigating the residuals before moving to the data-constrained fit procedure. (Comment 2/6 of our joint scaling response)
  • Consistent with our initial claims, measures such as Huber loss do not necessarily correlate with extrapolation. (Also comment 2/6 of our joint scaling response)

We also highlight that we have added an additional held-out run to further investigate extrapolation (comment 3/6 of our joint scaling response), and added additional context as to why the data-constrained scaling laws might fail (comment 4/6 of our joint scaling response).

AuthorsOpenReview ↗

Question 1 and 3

Hello, we are continuing to refine responses to your thoughtful inquiries. In the meantime, we hope to answer these shorter questions.

Question 1a: “In the implementation of AMT-2, what value does delta take?”

In AMT-2, we expand the values of δ\delta to include 1,5,10,20\\{1, 5, 10, 20\\}, each representing the number of seconds ahead in time that may be anticipated. We sample uniformly from the set of possible values at tokenization time. In AMT-1,δ=5\delta = 5 always. When these values become tokens, they are (δ\delta: Token ID):

{
    1: 35139, 
    5: 35140, 
    10: 35141, 
    20: 35142
}

Question 1b: “how long is a tick”?

In our runs, we set the tick token to occur every second. This can be configured. It has token integer ID 17568.

Question 3: “In Figure 2, six stars appear to correspond to 2 small, 3 medium, and 1 large AMT-1 model. Which concrete training runs do these stars correspond to?”

These stars correspond to rows 3, 5, 6, 7, 8, 9 in Table 1 of the original Thickstun et al. “Anticipatory Music Transformer” paper. The stars are every reported run that uses the arrival time encoding with anticipation. We will add the below table to the appendix:

#ConfigParamsStepFLOPsBPS ↓Test loss ↓
3Small128,103,936100k3.2914×10193.2914 \times 10^{19}80.70.902683
5Small128,103,936800k2.6332×10202.6332 \times 10^{20}75.00.839232
6Medium359,708,672100k1.1168×10201.1168 \times 10^{20}74.40.831068
7Medium359,708,672200k2.2336×10202.2336 \times 10^{20}71.50.802315
8Medium359,708,672800k8.9345×10208.9345 \times 10^{20}69.70.780602
9Large780,136,960100k2.5423×10202.5423 \times 10^{20}73.20.819863

Test loss is the computed as ln⁡(ppl⁡(e))/3\ln(\operatorname{ppl}(\mathbf{e}))/3, where ppl⁡(e)\operatorname{ppl}(\mathbf{e}) is the per-event perplexity.

AuthorsOpenReview ↗

Question 2

Hello, we have written another joint response that addresses concerns raised by more than one reviewer: LINK

In particular, we answer Question 2 in comment 2/4. Indeed δ\delta is a value sampled at dataset tokenization time. Our original writing in the manuscript described the procedure of how we create the simultaneous in-filling and accompaniment samples. We believe comment 2/4 describes this more clearly.

Reviewer_UJQpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents AMT-2, a 1.14B-parameter symbolic music model combining relative-time tokenization, block anticipation, an updated training recipe, and a larger training corpus. It compares five data-constrained scaling-law formulations using 215 training observations and uses the selected fit, subject to compute constraints, to configure the final model. AMT-2 improves likelihood metrics and human preferences over AMT-1. However, all fitted scaling laws underestimate its final test loss, raising questions about their extrapolation reliability in this setting.

Critical strengths
  1. The overall improvement in symbolic music generation is supported by complementary evaluations. With evaluation context restricted to 1,024 tokens, AMT-2 reduces Lakh test BPS from AMT-1’s 69.7 to 59.0 (Table 8). Human evaluators also prefer AMT-2 over AMT-1 and BEAT, with significance retained after multiple-comparison correction (Table 4). These results provide substantive evidence of practical progress beyond likelihood improvements alone.

  2. The scaling study provides an informative test of existing laws in a data-constrained domain. Across 90 single-epoch and 125 multi-epoch observations, formulations with similar fit quality recommend substantially different configurations, and the best-fitting formulation is not the most accurate on the held-out run (Figures 3–4). This is useful empirical evidence that fit quality alone may be insufficient for guiding training allocation.

Critical weaknesses
  1. The scope of the scaling-law conclusions is limited by the extrapolation evaluation. Multi-epoch fitting covers at most 496.5M parameters and 0.8B unique tokens, while the main held-out configuration uses 1.14B parameters and 27.86B unique tokens (Table 7; Figure 4). This substantial extrapolation provides a useful stress test, but a single configuration does not establish whether the observed mismatch is systematic or identify its source. Additional held-out configurations and matched-budget comparisons would strengthen the study’s value for selecting training configurations.

  2. The benefits of the anticipation changes are not directly isolated. The main human evaluation assesses prefix continuation, and the AMT-1 comparison explicitly uses autoregressive generation (Appendix A.10.2). Table 1 also combines time-representation and anticipation changes. Thus, while the overall generation improvements are supported, the incremental benefits for cold-start conditioning and joint accompaniment and infilling remain less clear. Targeted evaluations of these capabilities would strengthen the methodological contribution.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Critical Weakness 1: Scaling Result

We thank the reviewer for their thoughtful comments, in particular, we appreciate the comment that our scaling results are useful. We will first address critical weakness 1 and write follow-up comments addressing the remaining stated critical weakness.

Regarding Critical Weakness 1 of the scaling result, we respond to specific concerns below.

“A single configuration does not establish whether the observed mismatch is systematic or identify its source”

  • We identify extremely low data budgets in the Chinchilla fit grid as a major contributor, and re-fit the Chinchilla equation on samples with UD≥400U_D \geq 400M. Comment (2/6) of our joint response details this.

“Multi-epoch fitting covers at most 496.5M parameters and 0.8B unique tokens, while the main held-out configuration uses 1.14B parameters and 27.86B unique tokens”

  • In hopes of improving extrapolation, we increased the sample grid of multi-epoch runs to 24B unique token budgets for a single model size 496M. We use these additional 5 samples in our analysis again in comment (2/6) of our joint response.

“Additional held-out configurations and matched-budget comparisons would strengthen the study’s value for selecting training configurations.”

  • We add an additional held-out sample of a model at (N=3.39BN=3.39\mathrm{B}, UD=27.86BU_D=27.86\mathrm{B}, at 17 of 20 total epochs). The run is not fully annealed in its learning rate schedule, but the preliminary result suggests the mismatch is systematic (with the new UD≥400MU_D \geq 400\mathrm{M} Chinchilla fit, all five previously reported scaling law forms over-predict both held-out runs). This is in comment (3/6) of our joint response. Regarding matched budget comparisons, we agree this is the correct diagnostic to determine which scaling law is most useful for selecting a configuration. Our results test calibration and not ranking, so this is a good direction. We do not have available GPUs to perform this within the discussion period, but we commit to releasing our full scaling law fit data, which contains ~3,863.5 GPU hours (161 GPU-days) of training runs.

We’ve also added another scaling law fit, the Qu et al. Symbolic Music Scaling Law (SMS Law), which extrapolates within 0.001 nats on both held-out runs. We detail how we needed to adapt its stated fitting procedure to our setting in comment 5/6 of our joint response.

AuthorsOpenReview ↗

Critical Weakness 2: The benefits of anticipation

Hello,

We have written another joint response regarding anticipation and its effects: LINK

We have performed an experiment that quantifies the effect of cold starts in AMT-1. To summarize, cold starts occur at the start of tokenized files and therefore impart a bias towards introductory material. This means AMT-1 tends to produce sparse and delayed sequences when prompted by cold start samples. We show that AMT-2 reduces this sparsity and delay in comment 3/4.

We also investigate the impact of anticipation on the validation cross-entropy in our scaling laws fitting samples. We find that the controls reduce event cross entropy, consistently across scales, and that models learn how to use the controls: comment 4/4

AuthorsOpenReview ↗

Critical Weakness 2: Additional Results

Dear UJQp,

In Critical Weakness 2, the reviewer had asked for targeted evaluations of cold-start conditioning and joint accompaniment and in-filling. Our earlier response (comment 3/4) better explained cold starts and quantified the degree to which AMT-2's new training patterns help (Table 5 in the now updated manuscript).

At reviewer Dhud's request, we have now also evaluated joint accompaniment and in-filling directly. We've since added our results to latest revision as Appendix A.2 (more detail in our reply to Dhud: LINK, there are 3 parts to this comment). In the thread, the control pattern of 'joint accompaniment and in-filling' is referred to as 'accompaniment infilling', as reviewer Dhud found this to be a more intuitive term. In short, AMT-1 can perform joint accompaniment and in-filling despite not training on it, so we withdraw our earlier claim that it cannot; AMT-2 makes more use of the context after the span.

We hope our additional experiment makes the benefit of the new control patterns clearer.

Thank you for the helpful suggestion.

Thanks to the authors for the follow-up experiments and clear responses. This is a very solid work. I have raised my score to support acceptance.

Comments on the paper

16 comments
AuthorsOpenReview ↗

Joint Scaling Response (1/6)

Joint Response to Scaling Study (1/6): Overview

All three reviewers raise valid concerns regarding our scaling law analysis. Below is a single response with our findings. For each reviewer, we add more context and pointers within individual replies (to be posted. EDIT: all have been posted).

Our understanding of the raised concerns is:

  • X38E: inability to predict the loss weakens the contribution
  • Dhud: inability to predict the loss raises concerns that our scaling laws are unreliable, and therefore our study is "less sufficient or useful… than expected".
  • UJQp: our conclusion that the data-constrained scaling laws do not extrapolate in our setting is useful, but a single data-point is insufficient to support our claim

We first wish to defend our assertion that reporting the scaling law result is valuable despite the imperfect extrapolation. We are forthcoming about this issue. This is because our negative result reveals previously unknown limitations of these scaling laws and indeed is part of the contribution because it enriches our collective understanding of them in new modalities.

To ground our discussion, we reiterate that our objective is to build capable generative symbolic music models using the latest techniques with our own improvements, convincingly argue these models are powerful, and document the process to promote open science. We make a thorough attempt to use the data-constrained scaling laws as described in their respective publications, and report honest findings. We had asked, do data-constrained scaling laws extrapolate in appropriate settings outside their proposing work? Unfortunately we had found that they do not for symbolic music, but our new results refine this.

We have since conducted a retrospective investigation based on all the reviewers' helpful comments. We divide this into three parts, posted as separate comments below, followed by a summary:

  • Part 1 (2/6 and 3/6): Could we have known ahead of time that our extrapolation would fail? Do we have more evidence to strengthen our claim of extrapolative shortcomings?
  • Part 2 (4/6): Why might the data-constrained scaling laws fail?
  • Part 3 (5/6): Are there other scaling law forms that better fit our data?
  • Summary (6/6)
AuthorsOpenReview ↗

Joint Scaling Response (2/6)

Could we have known ahead of time that our extrapolation would fail? (Chinchilla Residual and Re-fit)

The Chinchilla residuals show that the single-epoch scaling law already exhibits considerable error, even before fitting the data-constrained scaling laws. Prior work observes that Chinchilla tends to "overestimate the [positive] impact of additional training data as we move towards the long data regime… and [models tend to train] more slowly than what scaling laws predict" (Sardana et al. pp. 6). We find that unique data budgets below 400M have higher residuals, so we exclude these runs, and re-fit Chinchilla. This reduces the residual in the large model, large data regime within the fit by over an order of magnitude: from 0.094 nats per token to 0.007.

🖼️ FIGURE 1 (link) 🖼️

Chinchilla residuals (observed − fitted) for all 90 single-epoch runs. With the full grid, residuals are largest at the smallest unique-token budgets (up to −1.77-1.77 nats at N=1.14N = 1.14B, UD=50U_D = 50M), and the fit is off by +0.094+0.094 at N=1.14N = 1.14B, UD=24U_D = 24B. Re-fitting on the 63 runs with UD≥400U_D \geq 400M reduces that residual to +0.007+0.007.

We could have done this analysis without the held-out extrapolations. However, it is not obvious that we should have. The next question is, does this improved Chinchilla fit also improve the data-constrained scaling laws?

We add five additional multi-epoch runs to our fitting data to improve extrapolation. These samples are a model of N=496N = 496M parameters trained on UD=24U_D = 24B unique tokens for 2, 4, 8, 12, and 16 epochs, which is a data budget much closer to our true held-out run. We next re-fit the data-constrained scaling laws using our improved Chinchilla coefficients and with a total of 130 multi-epoch samples.

🖼️ FIGURE 2 (link) 🖼️

Held-out residuals (predicted − observed, nats/token) on the 1.14B, 7-epoch run (observed test loss 0.8402). "Full" uses the full Chinchilla grid, as in the submission; "≥400M" uses the re-fit Chinchilla. Mean log-Huber (×104\times 10^4) is scored on the 130 multi-epoch runs.

LawResidual (Full)Residual (≥400M)Huber (Full)Huber (≥400M)
Chinchilla−0.1511+0.02491.8641.737
Form A−0.1284+0.02890.9261.018
Form B−0.1436+0.02890.7481.018
Form C−0.1438+0.03151.1261.114
Form D−0.1091+0.06880.8630.837
Form E−0.1220+0.05160.7720.825

Extrapolation improves considerably: compare the "≥400M" columns (our new fits) with the "Full" columns (the full Chinchilla grid, as in the submission). However, we believe our main claims still stand because:

  • The extrapolative capabilities of the data-constrained scaling laws remain dubious in this setting, as Chinchilla puzzlingly is the closest, though we concede this could be noise.
  • If one examines the goodness of fit metric of the mean log Huber loss, it isn't obvious that Chinchilla should extrapolate best.
  • This re-analysis is retrospective. At the time of launching our large runs, it's unclear that we should have discarded some samples from fitting. These small unique data budgets are from the stated grid of Lovelace et al., which we followed closely.

Part 1 continues in the next comment (3/6) with a new held-out run.

AuthorsOpenReview ↗

Joint Scaling Response (3/6)

Part 1, continued: a new held-out run

We have since obtained an additional held-out data point to test extrapolation. We have been training a larger model and now have a new held-out sample to test extrapolation. Our new held-out data point has N=3.39N = 3.39B on UD=27.86U_D = 27.86B for 20 epochs. This model is still training, so we take an intermediate checkpoint at 17 epochs. EDIT: the model has finished training. New details are in the updated PDF and summarized in the changelog: LINK. We caution that the learning rate schedule has not fully decayed, so the result is preliminary. We increased the context size of this model from 4096 to 8192, so we evaluate test loss on the first 4096 tokens to match prior runs. The extrapolation results are below (the full plot is at the Figure 3 link):

🖼️ FIGURE 3 (link) 🖼️

Held-out residuals (predicted − observed, nats/token) on the 3.39B run at 17 of 20 epochs (observed test loss 0.8061, first 4,096 tokens of the 8,192-token context). Same fits as Figure 2.

LawResidual (Full)Residual (≥400M)
Chinchilla−0.1767+0.0231
Form A−0.1361+0.0293
Form B−0.1605+0.0293
Form C−0.1186+0.0753
Form D+0.0244+0.2291
Form E−0.1309+0.1190

The takeaway is the same as the previous re-fitting. The original Chinchilla fit with Form D ("Full" column) appears close to the held-out run (+0.024). However, the same Form D fit missed the 1.14B run by −0.109 and the Form D fit with UD≥400U_D \geq 400M misses this run by +0.229. We therefore do not read this as a reliable extrapolation.

To summarize the impact of Part 1:

  • We expanded the multi-epoch fitting grid to large data budgets to help extrapolate. It does not resolve the 'miss' from the original submission.
  • We improved the Chinchilla fit by removing small data budgets. We inspected the residual and reviewed prior literature that observes limitations of the equation. This helps extrapolation considerably.
    • We advise future practitioners to investigate the residuals on their Chinchilla fit before advancing to the data-constrained scaling fits. The Lovelace paper does not report this step.
  • In our original manuscript, we had stated that the Lovelace additive penalty is the most accurate, but in this re-fit setting, that statement no longer holds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.