acceptodds
Under review as a conference paper at ICLR 2027

Quantizing Rhythm with Music Foundation Models

Abstract

Rhythm quantization, mapping the expressive timings of a musical performance into a score's precise note values, must be learned from fewer than a thousand paired performances and fewer than two hundred scores. We show that a pretrained symbolic-music foundation model can be adapted into a rhythm quantizer with no new parameters: fed the performance as its anticipated control stream, the Anticipatory Music Transformer becomes a fixed-lookahead transcriber of entire songs, fine-tuned on interleaved pairs of events from a performance and its corresponding score. Musical pretraining substantially improves this adaptation. Under an identical training recipe and checkpoint-selection rule, the pretrained model more than doubles whole-song strict note F1 relative to random initialization (8.7% vs. 3.2%). We further find that teacher-forced fine-tuning in this data-limited regime is susceptible to compounding errors during whole-song autoregressive inference. To address this, we introduce a three-stage training curriculum that progressively bridges the distribution shift between teacher-forced training and model-generated inference. On the 59-performance ASAP test set, our model reaches 9.33% strict note F1 and 77.93% inter-onset-interval F1, compared with 8.96% and 67.81%, respectively, for the strongest task-specific baseline. These gains are accompanied by a substantially higher rate of producing outputs at the reference score's time scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
rhythm quantization, symbolic music, exposure bias, low-resource learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Nsh9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles whole-song rhythm quantization (mapping expressive musical performance timing into score note values and metric beats) in an extreme data-limited regime (the standard ASAP benchmark contains only 822 training performances from 176 works). The authors propose AMT-Q, which adapts a pretrained 360M symbolic-music foundation model (the Anticipatory Music Transformer) to rhythm quantization without adding any task-specific parameters. By feeding performance events as the anticipated control stream and interleaving them with score events, the model acts as a fixed-lookahead transducer.

To combat severe exposure bias during whole-song autoregressive rollouts, the authors develop a three-stage curriculum: (1) teacher forcing, (2) score-token dropout selected via whole-song validation rollouts, and (3) on-policy DAgger retraining on model-generated contexts anchored to reference weights.

Recognizing that standard whole-song strict note F1 conflates global scale differences, cumulative drift, and local rhythmic quantization, the authors introduce a suite of disentangled metrics: scaled note F1, inter-onset-interval (IOI) F1, scaled IOI F1, and Theil-Sen local time-scale estimation.

Validation on the 59-performance ASAP test set demonstrates that AMT-Q achieves 9.33% strict note F1 and 77.93% IOI F1, matching or outperforming task-specific state-of-the-art baselines. Crucially, controlled comparisons confirm that musical pretraining more than doubles strict F1 over random initialization (8.7% vs. 3.2%), and AMT-Q outputs align with the reference score's time scale substantially more frequently (43/59 vs. 23/59).

Critical strengths
  1. Parameter-Free Foundation Model Adaptation with Rigorous Pretraining Controls: Adapting the Anticipatory Music Transformer into a streaming rhythm quantizer without any new parameters is conceptually elegant. More importantly, the authors conduct an exemplary controlled study: holding the architecture, training updates, data augmentations, and checkpoint selection rules identical, musical pretraining achieves a statistically significant 2.7x increase in strict F1 over random initialization across works, while text pretraining (SmolLM2) fails to bridge the gap. This offers conclusive evidence on the transferability of generative symbolic priors to low-resource transcription.
  2. Methodological Rectification of Rhythm Quantization Evaluation: The paper makes a consequential contribution to the evaluation methodology of symbolic music. By dissecting whole-song strict F1 into global scale mismatch, cumulative temporal drift, and local rhythmic correctness via IOI F1 and Theil-Sen slope estimation, the paper resolves long-standing ambiguities in why past systems failed and provides an actionable blueprint for future benchmarking.
Critical weaknesses
  1. Low Absolute Strict Accuracy and Persistent Whole-Song Drift: Although AMT-Q surpasses prior systems on IOI F1 and time-scale adherence, the absolute strict note F1 remains below 10% (9.33% with DAgger, median 4.75%). This confirms that while the model successfully captures local rhythmic ratios (IOI F1 ~78%), whole-song autoregressive rollouts on complex classical pieces still frequently suffer from irreversible global drift, limiting its current readiness for commercial automated notation software without manual beat anchors.
  2. Evaluation Restricted to a Single Small Benchmark: The experimental validation is conducted exclusively on the ASAP dataset (14 test works, 59 performances). While ASAP is the standard benchmark in recent literature, demonstrating whether AMT-Q generalizes across other genres (e.g., jazz with complex syncopation, pop, or non-piano MIDI datasets) would provide broader validation for the proposed framework.
Questions for authors
  1. Section 5.3 notes that the Spearman rank correlation between whole-song validation strict F1 and test strict F1 across checkpoints is only 0.02. What accounts for this near-zero correlation? Is it driven by the small number of works in the validation (16) and test (14) sets, or by high sensitivity to drift in specific pieces? How can practitioners perform reliable model selection in this regime?
  2. Table 1 shows that while DAgger increases mean strict F1 over works from 9.6% to 14.7%, the median strict F1 slightly drops from 5.14% to 4.75%, and IOI F1 slightly decreases from 78.81% to 77.93%. Does DAgger introduce higher variance across different musical compositions?
  3. Since the primary bottleneck for strict F1 is long-term drift rather than local interval quantization, have the authors considered incorporating periodic global re-synchronization anchors (e.g., bar-line or downbeat estimators) during sliding-window inference?
2 comments
AuthorsOpenReview ↗

Reply to Nsh9 (1/2)

We thank the reviewer for the positive assessment and the careful questions, and for the strengths it identifies:

"Adapting the Anticipatory Music Transformer into a streaming rhythm quantizer without any new parameters is conceptually elegant."

"the authors conduct an exemplary controlled study"

"This offers conclusive evidence on the transferability of generative symbolic priors to low-resource transcription."

"The paper makes a consequential contribution to the evaluation methodology of symbolic music."

Since the submission we applied the same method to a 3.39B-parameter music foundation model, AMT-Q 3B, and evaluated it on two larger out-of-distribution test sets never used for fine-tuning (ATEPP, Gangqinpu; see our general response). Below we address each concern brought up by the reviewer.

Low Absolute Accuracy and Whole-Song Drift

"Low Absolute Strict Accuracy and Persistent Whole-Song Drift"

"the absolute strict note F1 remains below 10%"

"whole-song autoregressive rollouts on complex classical pieces still frequently suffer from irreversible global drift, limiting its current readiness for commercial automated notation software without manual beat anchors."

We agree that absolute strict F1 is still low, and we state this as a limitation. With AMT-Q 3B it rises to 19.68 on ASAP, 25.05 on ATEPP and 26.15 on Gangqinpu, but whole-song drift remains the main failure: strict F1 is close to all-or-nothing per piece. On ATEPP, 497 of the 2,489 recordings are above 50% strict F1, while the median Gangqinpu song scores 8.47. We make no claim of readiness for notation software without beat anchors.

Limited Test Set

"Evaluation Restricted to a Single Small Benchmark"

"The experimental validation is conducted exclusively on the ASAP dataset (14 test works, 59 performances)."

"demonstrating whether AMT-Q generalizes across other genres (e.g., jazz with complex syncopation, pop, or non-piano MIDI datasets) would provide broader validation for the proposed framework."

We now evaluate on two larger out-of-distribution test sets never used for fine-tuning: ATEPP (2,489 classical recordings, with noisier performances obtained from automated transcription) and Gangqinpu (11,425 songs, mostly pop piano arrangements, the genre the review mentions).

Strict note F1 (%) of AMT-Q 3B and the two released models, all fine-tuned on ASAP only; ATEPP and Gangqinpu never used for fine-tuning.

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test (59 recordings)19.688.967.58+10.72 per recording0.018
ATEPP (2,489 recordings)25.0514.749.99+12.24 per work< 0.0001
Gangqinpu (11,425 songs)26.1510.3210.20+15.83 per song< 0.0001

Results on other metrics can be found in parts 1 and 2 of the general reply.

AuthorsOpenReview ↗

Reply to Nsh9 (2/2)

Checkpoint Selection

"Section 5.3 notes that the Spearman rank correlation between whole-song validation strict F1 and test strict F1 across checkpoints is only 0.02."

"What accounts for this near-zero correlation?"

"Is it driven by the small number of works in the validation (16) and test (14) sets, or by high sensitivity to drift in specific pieces?"

"How can practitioners perform reliable model selection in this regime?"

Strict F1 on 14 to 16 works is dominated by a few pieces that either lock onto the beat or drift early in the piece, so a checkpoint's rank on validation is only weakly correlated to its rank on test. With AMT-Q 3B we trained four runs with different seeds and evaluated all 20 checkpoints of each on test:

RunStep selected on validationStrict F1, mean of its 20 checkpointsValidation-to-test rank correlation
Seed 42 (AMT-Q 3B)1800019.86−0.03
Seed 13800018.96+0.13
Seed 22000019.03+0.57
Seed 31800019.65+0.26
All four runs19.38 (all 80 checkpoints)

Validation ranks checkpoints poorly, but the choice matters little: the checkpoints selected on validation average 19.89 strict F1 on test, against 19.38 over all 80 checkpoints. For practitioners we recommend selecting on validation but comparing systems by the mean over several checkpoints or seeds, and reporting scaled and IOI F1 beside strict F1.

DAgger and Variance Across Pieces

"while DAgger increases mean strict F1 over works from 9.6% to 14.7%, the median strict F1 slightly drops from 5.14% to 4.75%, and IOI F1 slightly decreases from 78.81% to 77.93%."

"Does DAgger introduce higher variance across different musical compositions?"

With the 3B model we present DAgger as an optional extension rather than part of the recommended system: it raises IOI F1 but lowers strict and scaled F1. The pattern the review noticed, a higher mean with a lower median, comes from strict F1 being close to all-or-nothing per piece. With AMT-Q 3B, four of the 14 ASAP test works dominate the mean, and 7 of the 59 recordings are above 50%. On ATEPP the mean lead over Beyer & Dai is +12.24 points per work and the median lead +3.00. This is why we report medians and IOI-based measures beside the mean.

Re-Synchronization Anchors

"have the authors considered incorporating periodic global re-synchronization anchors (e.g., bar-line or downbeat estimators) during sliding-window inference?"

Unfortunately, it appears that existing beat-tracking systems aren't robust enough on classical music for this strategy to be effective. We tested re-anchoring to a beat tracker during decoding.

Re-anchoring to a beat tracker: every 69 notes, the output is shifted by whole beats so its beat count matches the tracker (PM2S on ATEPP, librosa on Gangqinpu). Strict F1 (%), mean over works (ATEPP) or songs (Gangqinpu), without and with re-anchoring; Holm p of the change.

Test set, systemOriginalRe-anchoredHolm p
ATEPP (232 works), AMT-Q 3B29.3518.21< 0.0001
ATEPP, Beyer & Dai17.1113.62< 0.0001
ATEPP, Zeng et al.12.5710.92< 0.0001
Gangqinpu test split, AMT-Q 3B27.4430.300.0081

It helps only where the beat tracker is reliable: on the metronomic Gangqinpu songs it raises AMT-Q 3B's strict F1, while on classical ATEPP, where the tracker errs, it lowers strict F1 for every system, significantly after Holm correction. Better beat or downbeat estimators would make this route more useful.

Reviewer_gY2MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for performance-to-score rhythm quantization. The core idea is to adapt the pretrained Anticipatory Music Transformer (AMT) By interleaving performance and score event. To mitigate exposure bias during whole-song generation, the authors introduce a three-stage training curriculum consisting of teacher forcing, score-token dropout, and DAgger. They also propose complementary evaluation metrics to distinguish global time-scale errors from local rhythmic inaccuracies.

Critical strengths
  1. The paper demonstrates that a pretrained symbolic music foundation model can be adapted to rhythm quantization without introducing task-specific parameters. The comparison with random initialization and text-model pretraining provides convincing evidence that music-specific pretraining offers substantial benefits in this data-limited setting.

  2. The authors provide extensive comparisons, ablations, and statistical analyses. In particular, the proposed complementary metrics reveal that much of the performance difference between methods originates from global time-scale mismatches rather than local rhythmic inaccuracies. This provides useful insights into the limitations of existing evaluation protocols and the challenges of rhythm quantization.

Critical weaknesses
  1. Despite using a 360M-parameter pretrained music foundation model and a multi-stage training pipeline, AMT-Q does not significantly outperform Beyer & Dai (2024) in strict note F1 (9.33% vs. 8.96%). Moreover, its apparent advantage in IOI F1 largely disappears after correcting for global time-scale mismatches, suggesting limited improvement in local rhythmic accuracy. This raises questions about whether the additional model complexity and training effort are justified by the performance gains.

  2. The contribution of DAgger is particularly unclear. Adding DAgger only marginally improves strict F1 (8.72% to 9.33%) while reducing IOI F1 and scaled IOI F1. Furthermore, the three DAgger runs do not consistently outperform Stage 2, raising concerns about the robustness of the claimed improvement. The multiple changes in optimization, regularization, and checkpoint selection across stages also make it difficult to isolate the contributions of individual components.

Questions for authors
  1. The three DAgger runs show inconsistent improvements over Stage 2, and their average strict F1 does not exceed that of Stage 2. Can the authors clarify whether DAgger provides a statistically significant and reproducible improvement across multiple random seeds? Additionally, how much of its benefit comes from exposure-bias mitigation versus training on unfiltered performances containing unmatched notes?

  2. While the comparison against random initialization demonstrates the value of music-specific pretraining, it remains unclear whether AMT's anticipation mechanism contributes beyond a general musical prior. Could the authors compare against a similarly sized music model pretrained using standard autoregressive next-token prediction, followed by the same fine-tuning procedure? This would help determine whether anticipation-specific pretraining is essential to the proposed approach.

5 comments
AuthorsOpenReview ↗

Reply to gY2M (1/3)

We thank the reviewer for the careful and rigorous review, and for the strengths it identifies:

"The paper demonstrates that a pretrained symbolic music foundation model can be adapted to rhythm quantization without introducing task-specific parameters."

"The comparison with random initialization and text-model pretraining provides convincing evidence that music-specific pretraining offers substantial benefits in this data-limited setting."

"The authors provide extensive comparisons, ablations, and statistical analyses."

Since the submission we applied the same method to a 3.39B-parameter music foundation model, AMT-Q 3B, and evaluated it on two larger out-of-distribution test sets never used for fine-tuning (ATEPP, Gangqinpu; see our general response). Below we address each concern brought up by the reviewer.

Gains over Beyer and Dai

"AMT-Q does not significantly outperform Beyer & Dai (2024) in strict note F1"

"This raises questions about whether the additional model complexity and training effort are justified by the performance gains."

With AMT-Q 3B the strict F1 lead over Beyer & Dai is significant on all three test sets.

Strict note F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test (59 recordings)19.688.967.58+10.72 per recording0.018
ATEPP (2,489 recordings)25.0514.749.99+12.24 per work< 0.0001
Gangqinpu (11,425 songs)26.1510.3210.20+15.83 per song< 0.0001

On the added complexity and training effort, our general response (comment 4/4) demonstrates that when fine-tuning compute is matched to the released systems, AMT-Q still outperforms task-specific systems in all four metrics on Gangqinpu.

Beyond Time-Scale Calibration

"its apparent advantage in IOI F1 largely disappears after correcting for global time-scale mismatches, suggesting limited improvement in local rhythmic accuracy."

After scaling, AMT-Q 3B's lead is more than time-scale calibration, and post-processing Beyer and Dai's output does not close the gap. Even after providing the best possible global time-scaling to Beyer and Dai's output per recording, which requires access to the ground-truth score and is not available in practice at inference time, unscaled AMT-Q 3B outperforms scaled Beyer and Dai by 8.14 points of strict F1.

Beyer & Dai's output rescaled by one tempo ratio per recording, ASAP test split. AMT-Q is unscaled.

SystemStrict F1AMT-Q 3B leadp (per recording)
Beyer & Dai, as released8.96+10.720.0018
Beyer & Dai, rescaled by a tempo rule chosen before testing8.46+11.220.0008
Beyer & Dai, only halved or doubled (chosen after seeing the test result)9.74+9.940.0047
Beyer & Dai, best ratio per recording (oracle: uses the reference score)11.54+8.140.017

With the optimal time-scale provided for all systems, AMT-Q 3B demonstrates a lead, significant except in scaled IOI F1 on ASAP.

Scaled F1 (%)

Scaled F1 (%) of AMT-Q 3B and the two released models, all fine-tuned on ASAP only; ATEPP and Gangqinpu never used for fine-tuning.

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test22.5911.5410.79+11.05 per recording0.0031
ATEPP32.8017.7612.97+17.55 per work0.0003
Gangqinpu43.1121.4616.51+21.65 per song0.0002

Scaled IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test84.3383.0280.29+1.30 per recording0.31
ATEPP86.3282.8480.55+2.32 per work0.0003
Gangqinpu90.9687.4086.74+3.55 per song0.0002
AuthorsOpenReview ↗

Reply to gY2M (2/3)

DAgger: Significance Across Seeds, Exposure Bias, Unmatched Notes

"The contribution of DAgger is particularly unclear."

"the three DAgger runs do not consistently outperform Stage 2, raising concerns about the robustness of the claimed improvement."

"The three DAgger runs show inconsistent improvements over Stage 2, and their average strict F1 does not exceed that of Stage 2."

"Can the authors clarify whether DAgger provides a statistically significant and reproducible improvement across multiple random seeds?"

"how much of its benefit comes from exposure-bias mitigation versus training on unfiltered performances containing unmatched notes?"

Reproducibility across seeds. We present DAgger as an optional extension of our recommended system, AMT-Q. We find that DAgger on AMT-Q 3B produces statistically significant gains on IOI F1 specifically while having mixed effects on other metrics. (see Reply to N7FD (2/4) for further details). We find that the IOI F1 gain replicates with a second seed.

Test setFine-tuned onIOI F1 gainHolm pSeed
ASAP test, per recording (59)ASAP+6.250.00211
ASAP test, per recording (59)ASAP+4.310.0172
Gangqinpu test split, per songGangqinpu and ATEPP+2.720.00842
Gangqinpu test split, per songGangqinpu and ATEPP+3.040.00021

Exposure bias or unmatched notes?

Experiments reveal almost all of the gain is exposure bias. DAgger's training data differs from supervised fine-tuning in two ways: the score so far is its own output and the performances are unfiltered, keeping every played note, including performance notes with no corresponding score note (unmatched note). Unmatched notes stay in the input, but in every run they are masked from the loss, so the model is never trained to reject them. AMT-Q's fine-tuning stages use filtered performances, with unmatched notes removed. We compare DAgger with SFT on filtered/unfiltered data.

DAgger and two ordinary training runs that differ from it in one factor each: the performances (filtered or unfiltered) and the score written so far that the model reads (its own output or the correct score).

RunPerformancesScore written so farDAgger's IOI F1 gain over this runHolm p
DAggerunfiltered (includes notes with no score note)the model's own output——
Ordinary trainingfiltered (unmatched notes removed)the correct score+6.300.0061
Ordinary training on DAgger's passagesunfiltered (includes notes with no score note): the same passages DAgger usesthe correct score+6.250.0021

Because the gain is indistinguishable over both, using unfiltered performances adds almost nothing, with all of the gain coming from training on contexts from the model's own output, which produces a statistically significant gain in IOI F1.

AuthorsOpenReview ↗

Reply to gY2M (3/3)

Isolating Each Component

"The multiple changes in optimization, regularization, and checkpoint selection across stages also make it difficult to isolate the contributions of individual components."

We ablate each component one at a time on AMT-Q 3B (ASAP test split):

Variant (everything else as in AMT-Q 3B)Strict F1 changep (per recording)
Random initialisation (no music pretraining)−13.80< 0.0001
Initialised from a 3B text model (Llama 3.2) instead of the music model−14.62< 0.0001
No score-token dropout in the second stage−5.130.032
First stage only (no second stage)−8.200.011

On ASAP, continuing from stage 1 through stage 2 improves strict note F1 by 8.20 points over stopping after stage 1 (p = 0.011). The full curriculum also outperforms the same curriculum with score-token dropout turned off by 5.13 points (p = 0.032), so score-token dropout helps on its own. Music-specific pretraining provides a further advantage under the same curriculum, improving strict note F1 by 13.80 points over random initialization and 14.62 points over Llama 3.2 3B initialization.

The optimizer does not matter nearly as much: AMT-Q 3B is trained with Muon; against the same recipe trained with AdamW:

AMT-Q 3B (trained with Muon) minus the same recipe trained with AdamW, ASAP test split, each run averaged over its 20 saved checkpoints; mean paired difference in points (positive = Muon is better).

MeasureMuon minus AdamWHolm p
Strict F1, per recording (59)+0.751.00
Strict F1, per work (14)+1.680.035
IOI F1, per recording−0.061.00
Scaled IOI F1, per recording+0.091.00

On checkpoint selection, four training runs with different seeds score 19.89 strict F1 on average at their selected checkpoints, against 19.38 over all 80 checkpoints, so checkpoint selection is worth about half a point; the mean over each run's 20 checkpoints lies between 18.96 and 19.86, so the four runs agree within one point.

Four training runs of the AMT-Q 3B recipe that differ only in seed, ASAP test split (59 recordings): the step selected on validation, the mean strict F1 (%) of the run's 20 checkpoints on test, and the Spearman rank correlation between validation and test strict F1 across those 20 checkpoints.

RunStep selected on validationStrict F1, mean of its 20 checkpointsValidation-to-test rank correlation
Seed 42 (AMT-Q 3B)1800019.86−0.03
Seed 13800018.96+0.13
Seed 22000019.03+0.57
Seed 31800019.65+0.26
All four runs19.38 (all 80 checkpoints)

Anticipation vs Music Pretraining

"it remains unclear whether AMT's anticipation mechanism contributes beyond a general musical prior."

"Could the authors compare against a similarly sized music model pretrained using standard autoregressive next-token prediction, followed by the same fine-tuning procedure?"

We run this comparison. Two 1B music foundation models with the same architecture and data are pretrained, one with anticipation and the other with ordinary next-token prediction. We then fine-tune these models identically:

MeasureDifferenceHolm p
Strict F1, per work (14)+2.800.55
Strict F1, per recording (59)+3.560.053
IOI F1, per recording (59)+7.390.0009
Scaled IOI F1, per recording (59)+2.770.0009

The anticipatory model scores higher on every measure. The gains in IOI and scaled IOI F1 are significant, but not in strict F1, so anticipation helps rhythm beyond a general musical prior, though we cannot show it raises strict F1.

Reviewer_N7FDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AMT-Q, which adapts a pretrained symbolic music generation model, Anticipatory Music Transformer, for rhythm quantization through interleaved performance and score events. The authors introduce a three-stage training curriculum to address exposure bias during autoregressive inference and propose complementary metrics for evaluating time-scale discrepancies and rhythmic accuracy. Experiments demonstrate the benefits of musical pretraining and achieve performance comparable to task-specific baselines.

Critical strengths

An interesting application of music foundation models to rhythm quantization.

The paper presents a reasonable approach to adapting a pretrained generative music model for performance-to-score conversion by leveraging AMT's existing anticipation mechanism. The comparison between pretrained AMT, randomly initialized AMT, and a pretrained text model provides useful evidence that music-specific pretraining benefits rhythm quantization.

Careful analysis of evaluation metrics and failure modes.

The paper provides a detailed examination of how global time-scale errors and accumulated drift affect whole-song evaluation. The complementary metrics offer useful insights into the limitations of existing rhythm quantization systems.

Critical weaknesses

Limited methodological novelty and insufficient justification for the third training stage.

The proposed three-stage training curriculum primarily combines existing techniques. The first stage adopts standard teacher-forced training, the second stage builds upon score-token dropout adopted from Beyer and Dai (2024), and the third stage applies the well-established DAgger algorithm. The paper would benefit from a clearer discussion of what new technical insights the proposed combination provides beyond existing approaches.

Also, the additional benefit of DAgger training is not convincingly demonstrated. It provides only a marginal improvement in strict F1 while degrading IOI-based metrics compared with Stage 2. Figure 4(a) also shows that DAgger does not consistently improve agreement with the reference time scale. The paper lacks sufficient analysis of which exposure-bias errors DAgger addresses and whether its additional complexity is justified.

Marginal improvements with substantially higher model complexity.

Despite adapting a substantially larger, 360M-parameter foundation model, AMT-Q achieves only marginal improvements in strict note-level F1 over Beyer and Dai (2024), and Section 5.2 explicitly acknowledges that the differences are not statistically significant. Moreover, the reported IOI F1 advantage largely disappears after correcting global time-scale discrepancies, raising questions about whether AMT-Q improves local rhythmic accuracy beyond time-scale calibration.

Given the substantially smaller architecture of Beyer and Dai (2024) and the extensive task-specific fine-tuning required by AMT-Q, the practical benefits of AMT-Q remain unclear without further evidence of improvements in accuracy, efficiency, or generalizability.

Questions for authors
  1. What are the practical advantages of AMT-Q over smaller task-specific models? Could the authors compare parameter counts, training costs, memory requirements, and inference speed to justify adapting a 360M-parameter foundation model when the accuracy improvements are not statistically significant?
  2. What is the specific contribution of DAgger training? Could the authors provide further analysis of the errors corrected by Stage 3 and demonstrate consistent improvements over Stage 2? Given the mixed results in Figure 4(a) and Table 1, which variant is recommended as the final system?
  3. Does AMT-Q offer advantages beyond global time-scale calibration? Given that the IOI F1 improvement over Beyer and Dai (2024) largely disappears after correcting time-scale differences, could a simpler post-processing method applied to Beyer and Dai achieve comparable results?
  4. How generalizable are the benefits of foundation-model adaptation beyond rhythm quantization? Given the substantial task-specific fine-tuning required, could the authors clarify whether pretraining offers advantages in adaptation efficiency, generalization, or transfer to other music understanding tasks beyond what could be achieved with a task-specific system?
6 comments
AuthorsOpenReview ↗

Reply to N7FD (1/4)

We thank the reviewer for the careful review, and for the strengths it identifies:

"An interesting application of music foundation models to rhythm quantization."

"The comparison between pretrained AMT, randomly initialized AMT, and a pretrained text model provides useful evidence that music-specific pretraining benefits rhythm quantization."

"Careful analysis of evaluation metrics and failure modes."

"The complementary metrics offer useful insights into the limitations of existing rhythm quantization systems."

Since the original submission, we have scaled our method to a larger music foundation model, AMT-Q 3B (3.39B parameters), and evaluated it on two larger out-of-distribution test sets never used for fine-tuning. Our general response describes the model. Below we address each concern brought up by the reviewer.

Marginal Gains

"AMT-Q achieves only marginal improvements in strict note-level F1 over Beyer and Dai (2024), and Section 5.2 explicitly acknowledges that the differences are not statistically significant."

"Given the substantially smaller architecture of Beyer and Dai (2024) and the extensive task-specific fine-tuning required by AMT-Q, the practical benefits of AMT-Q remain unclear without further evidence of improvements in accuracy, efficiency, or generalizability."

After scaling to 3.39B parameters, the lead over Beyer and Dai is no longer marginal, even out-of-distribution. AMT-Q 3B leads on strict F1 on ASAP and on two larger datasets never used for fine-tuning, ATEPP (classical; performance MIDI transcribed automatically from audio, so noisier) and Gangqinpu (pop piano) by statistically significant margins.

Strict note F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test (59 recordings)19.688.967.58+10.72 per recording0.018
All of ATEPP (2,489 recordings)25.0514.749.99+12.24 per work< 0.0001
Gangqinpu (11,425 songs)26.1510.3210.20+15.83 per song< 0.0001

Scaled F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test22.5911.5410.79+11.05 per recording0.0031
All of ATEPP32.8017.7612.97+17.55 per work0.0003
Gangqinpu43.1121.4616.51+21.65 per song0.0002

IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test77.7267.8163.04+9.92 per recording0.0010
All of ATEPP74.2074.9370.07−1.15 per work0.71
Gangqinpu73.4063.6869.33+9.72 per song0.0002

Scaled IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test84.3383.0280.29+1.30 per recording0.31
All of ATEPP86.3282.8480.55+2.32 per work0.0003
Gangqinpu90.9687.4086.74+3.55 per song0.0002

In strict and scaled F1, AMT-Q 3B is ahead on all three test sets. In IOI and scaled IOI F1 it is ahead as well, with the exception of IOI F1 on ATEPP, where Beyer and Dai is on par, and scaled IOI F1 on ASAP, where the lead is not significant. A discussion of efficiency and generalization can be found further down in this thread under Computational Cost and Generalization.

AuthorsOpenReview ↗

Reply to N7FD (2/4)

Novelty of the curriculum

"Limited methodological novelty and insufficient justification for the third training stage."

"The proposed three-stage training curriculum primarily combines existing techniques."

"The paper would benefit from a clearer discussion of what new technical insights the proposed combination provides beyond existing approaches."

The stages are known methods. Our contribution is to quantify how much the two-stage fine-tuning curriculum improves rhythm quantization over stopping after the first stage, how much score-token dropout adds, and how much the choice of pretrained model matters under the same curriculum.

Ablations of AMT-Q 3B on the ASAP test split:

Variant (everything else as in AMT-Q 3B)Strict F1 changep (per recording)
Random initialisation (no music pretraining)−13.80< 0.0001
Initialised from a 3B text model (Llama 3.2) instead of the music model−14.62< 0.0001
No score-token dropout in the second stage−5.130.032
First stage only (no second stage)−8.200.011

On ASAP, continuing from stage 1 through stage 2 improves strict note F1 by 8.20 points over stopping after stage 1 (p = 0.011). The full curriculum also outperforms the same curriculum with score-token dropout turned off by 5.13 points (p = 0.032), so score-token dropout helps on its own. Music-specific pretraining provides a further advantage under the same curriculum, improving strict note F1 by 13.80 points over random initialization and 14.62 points over Llama 3.2 3B initialization.

Contribution of DAgger

"the additional benefit of DAgger training is not convincingly demonstrated."

"It provides only a marginal improvement in strict F1 while degrading IOI-based metrics compared with Stage 2."

"What is the specific contribution of DAgger training?"

"Could the authors provide further analysis of the errors corrected by Stage 3 and demonstrate consistent improvements over Stage 2?"

"The paper lacks sufficient analysis of which exposure-bias errors DAgger addresses"

"Given the mixed results in Figure 4(a) and Table 1, which variant is recommended as the final system?"

The recommended system is AMT-Q after the second stage. We present DAgger as a possible extension and analyze it separately. DAgger raises IOI F1 specifically with a mixed effect on other metrics, suiting applications where local rhythmic accuracy matters most.

Test setFine-tuned onIOI F1 gainHolm pSeed
ASAP test, per recording (59)ASAP+6.250.00211
ASAP test, per recording (59)ASAP+4.310.0172
All of ATEPP, per work (232)ASAP+2.340.0211
Gangqinpu test split, per songGangqinpu and ATEPP+2.720.00842
Gangqinpu test split, per songGangqinpu and ATEPP+3.040.00021

Below, we show the difference between DAgger and a control which continues ordinary fine-tuning instead:

MeasureASAP testHolm pAll of ATEPPHolm pGangqinpu testHolm p
Strict note F1+3.471.00−1.960.76−5.090.0060
Scaled F1+1.421.00−4.910.0010−10.420.0018
IOI F1+6.250.0021+2.340.021+2.720.0084
Scaled IOI F1+1.111.00−1.090.0034−0.960.014

Observe that IOI F1 significantly rises on all three test sets. With the time-scaling forgiven (scaled IOI F1), the gain vanishes, so DAgger mainly helps the model keep inter-onset-intervals scaled correctly. Strict and scaled F1 do not rise significantly anywhere, and fall on Gangqinpu (scaled F1 also on ATEPP). We view DAgger as a way to trade-off performance between different metrics, providing improved IOI F1 across the board at the cost of lower strict and scaled F1 on Gangqinpu and lower scaled F1 on ATEPP, best suited for applications where correct local rhythmic structure is most important.

AuthorsOpenReview ↗

Reply to N7FD (3/4)

Beyond Time-Scale Calibration

"the reported IOI F1 advantage largely disappears after correcting global time-scale discrepancies, raising questions about whether AMT-Q improves local rhythmic accuracy beyond time-scale calibration."

"Does AMT-Q offer advantages beyond global time-scale calibration?"

"could a simpler post-processing method applied to Beyer and Dai achieve comparable results?"

After scaling, AMT-Q 3B's lead is more than time-scale calibration, and post-processing Beyer and Dai's output does not close the gap. Even after providing the best possible global time-scaling to Beyer and Dai's output per recording, which requires access to the ground-truth score and is not available in practice at inference time, unscaled AMT-Q 3B outperforms scaled Beyer and Dai by 8.14 points of strict F1.

Beyer & Dai's output rescaled by one tempo ratio per recording, ASAP test split. AMT-Q is unscaled.

SystemStrict F1AMT-Q 3B leadp (per recording)
Beyer & Dai, as released8.96+10.720.0018
Beyer & Dai, rescaled by a tempo rule chosen before testing8.46+11.220.0008
Beyer & Dai, only halved or doubled (chosen after seeing the test result)9.74+9.940.0047
Beyer & Dai, best ratio per recording (oracle: uses the reference score)11.54+8.140.017

With the optimal time-scale provided for all systems, AMT-Q 3B demonstrates a lead, significant except in scaled IOI F1 on ASAP.

Scaled F1 (%)

Scaled F1 (%) of AMT-Q 3B and the two released models, all fine-tuned on ASAP only; ATEPP and Gangqinpu never used for fine-tuning.

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test22.5911.5410.79+11.05 per recording0.0031
ATEPP32.8017.7612.97+17.55 per work0.0003
Gangqinpu43.1121.4616.51+21.65 per song0.0002

Scaled IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & DaiHolm p
ASAP test84.3383.0280.29+1.30 per recording0.31
ATEPP86.3282.8480.55+2.32 per work0.0003
Gangqinpu90.9687.4086.74+3.55 per song0.0002

Generalization

"How generalizable are the benefits of foundation-model adaptation beyond rhythm quantization?"

"could the authors clarify whether pretraining offers advantages in adaptation efficiency, generalization, or transfer to other music understanding tasks beyond what could be achieved with a task-specific system?"

We only tested rhythm quantization and make no claims about other music tasks. Within this task, our results demonstrate robust generalization and out-of-distribution transfer. AMT-Q 3B, fine-tuned on ASAP alone, leads on ATEPP and Gangqinpu, which were never used for fine-tuning (tables under Marginal Gains above), and strict F1 grows monotonically with model size on all three test sets, with the scaling law shown in Figure 1.

🖼️ FIGURE 1 (link) 🖼️

Figure 1. Strict note F1 (%) against non-embedding parameters, both on log scales, on ASAP test, ATEPP and Gangqinpu.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response (1/4): AMT-Q scaled to 3B parameters

We thank the reviewers for their rigorous and constructive reviews. All three praise two parts of the paper: the evidence that music-specific pretraining matters (Nsh9 notes a 2.7x increase in strict F1 over random initialization; N7FD, gY2M, Nsh9), and the evaluation that separates global time-scale errors from local rhythmic accuracy (N7FD, gY2M, Nsh9). gY2M and Nsh9 also note that AMT-Q adapts the foundation model without any task-specific parameters.

The shared concern was the strength of the results: on ASAP, AMT-Q's strict-F1 lead over Beyer & Dai was not statistically significant (N7FD, gY2M), and the evaluation relied on one small benchmark (Nsh9). The paper argues that a music foundation model lets rhythm quantization benefit from scale, so we tested this claim by applying the same method to a 3.39B-parameter AMT (AMT-Q 3B below) and evaluating it on two larger out-of-distribution benchmarks never used for fine-tuning: ATEPP (classical; performance MIDI transcribed automatically from audio, so noisier) and Gangqinpu (pop piano, which Nsh9 asked about).

Scaled-up Foundation Model AMT-Q now builds on AMT v2, a scaled-up Anticipatory Music Transformer from a concurrent ICLR 2027 submission, Scaling Anticipatory Music Transformers (https://openreview.net/forum?id=oXlMzBVPuB), which documents its architecture, tokenization and fully open pretraining corpus; we use its 3.39B-parameter model in place of the 360M AMT v1. The method is unchanged: the same anticipatory layout with a 32-note performance prefix, the same window-by-window decoding, and the same training curriculum. The reported AMT-Q is the model after its second stage; DAgger, the third stage, is analyzed separately in our replies to N7FD and gY2M. What changes is the token grammar, because AMT-Q uses AMT v2's vocabulary:

Submission (AMT v1)Revision (AMT v2)
TimeAbsolute: each onset counted from the start of the windowRelative: each onset is counted from the most recent clock token of its own stream
Score clockNoneTICK, AMT v2's own clock token: one per two notated beats (the score is normalized to 0.5 s per beat)
Performance clockNoneATICK: one per second of performance time
Resolution10 ms (performance); 1/50 beat (score)1/96 s (performance); 1/48 beat (score), so triplets are exact
One noteA triplet: onset, duration, pitchThe same triplet (onset offset, duration, pitch), preceded by one clock token for each clock step passed since that stream's previous note: TICK … TIME DUR NOTE for a score note, ATICK … ATIME ADUR ANOTE for a performance note. Clock tokens never appear inside a triplet
Window1,024 tokens (170 performance notes)4,096 tokens (about 645 notes); both clocks restart in every window

The score and performance clocks are separate since they run at different rates: the score is notated in beats (120 beats per minute), while the performance occurs in seconds with rubato. AMT v2 has a single clock, so ATICK is added in a previously unused row of vocabulary. No additional parameters are used.

Results. Strict note F1 (%), best system in bold. All systems are trained on the ASAP train split only, and AMT-Q is chosen on ASAP validation.

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & Dai (Holm p)
ASAP test (59 recordings)19.688.967.58+10.72 per recording (0.018)
All of ATEPP (2,489 recordings, 232 works)25.0514.749.99+12.24 per work (< 0.0001)
Gangqinpu (14,554 recordings, 11,425 songs)26.1510.3210.20+15.83 per song (< 0.0001)

AMT-Q 3B now leads Beyer and Dai significantly on all three benchmarks. On ATEPP and Gangqinpu, the Holm correction covers the six tests we fixed prior to decoding; on ASAP it covers 12 tests of the ASAP comparison (four evaluation metrics, three tests each).

The next three comments cover the other three measures (2/4), scaling with model size (3/4), and cost and where each reviewer's other points are answered (4/4).

AuthorsOpenReview ↗

General response (2/4): other evaluation metrics

We present results for the other evaluation metrics, with the best system bolded. Scaled F1 is strict F1 maximized over 13 global time-scale ratios. IOI F1 matches each note's inter-onset interval relative to the previous note rather than absolute position, forgiving drift and measuring local rhythmic agreement. Scaled IOI F1 maximizes IOI F1 over the same ratios and forgives both drifting and global scaling.

Scaled F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & Dai (Holm p)
ASAP test22.5911.5410.79+11.05 per recording (0.003)
All of ATEPP32.8017.7612.97+17.55 per work (0.0003)
Gangqinpu43.1121.4616.51+21.65 per song (< 0.001)

IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & Dai (Holm p)
ASAP test77.7267.8163.04+9.92 per recording (0.001)
All of ATEPP74.2074.9370.07−1.15 per work (0.71)
Gangqinpu73.4063.6869.33+9.72 per song (< 0.001)

Scaled IOI F1 (%)

Evaluation setAMT-Q 3BBeyer & DaiZeng et al.Lead over Beyer & Dai (Holm p)
ASAP test84.3383.0280.29+1.30 per recording (0.31)
All of ATEPP86.3282.8480.55+2.32 per work (0.0003)
Gangqinpu90.9687.4086.74+3.55 per song (< 0.001)

AMT-Q 3B leads Beyer & Dai significantly in scaled F1 on all three benchmarks, in IOI F1 on ASAP and Gangqinpu, and in scaled IOI F1 on ATEPP and Gangqinpu.

AuthorsOpenReview ↗

General response (3/4): strict F1 grows with model size on all three benchmarks

To determine whether the scale of the pretrained music foundation model drives the gains, we fine-tune AMT v2 models from 15M to 3B parameters using the AMT-Q recipe and evaluate them on all three benchmarks.

Strict note F1 in percentage points by model size, each size fine-tuned on ASAP with the full AMT-Q recipe: ASAP test split (mean over 59 recordings), all of ATEPP (mean over 2,489 recordings) and Gangqinpu (mean over 11,425 songs).

ModelPretraining tokensASAP testAll of ATEPPGangqinpu
15M24B5.565.409.90
50M24B5.7711.1214.29
125M24B5.9711.8015.58
500M24B8.2515.3117.35
1B24B11.2918.7922.33
1.14B195B12.0520.3022.72
3B431B19.6825.0526.15

Below, we provide the exponent bb of the fitted scaling law strict F1=cNb\text{strict F1} = cN^{b} over non-embedding parameters NN, with 95% confidence intervals over works (ASAP, ATEPP) or songs (Gangqinpu). Every interval excludes 0.

Evaluation setbb, 15M to 1B (same pretraining tokens)bb, 15M to 1B and 3B
ASAP test (14 works)0.119 [0.005, 0.237]0.176 [0.079, 0.276]
All of ATEPP (232 works)0.203 [0.170, 0.236]0.205 [0.176, 0.235]
Gangqinpu (11,425 songs)0.128 [0.124, 0.132]0.134 [0.130, 0.137]

Strict F1 rises with model size on every benchmark. The models pass Beyer and Dai between 500M and 1B on ASAP, between 125M and 500M on ATEPP, and between 15M and 50M on Gangqinpu. Because the 15M to 1B models share the same pretraining corpus, their fit isolates the effect of size.

🖼️ FIGURE 1 (link) 🖼️

Figure 1. Strict note F1 (%) against non-embedding parameters, both on log scales, on ASAP test, all of ATEPP and Gangqinpu.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.