- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MuseDuo, a unified multimodal framework for jointly modeling audio and MIDI, aiming to support a wide range of music tasks, including music generation, source separation, MIDI transcription, audio rendering, timbre transfer, and note-level editing, within a single model.
Specifically, the authors extend a pretrained music generation DiT with a MIDI-VAE that maps discrete piano-roll representations into continuous latent sequences temporally aligned with audio latents. A shared Transformer jointly models the two modalities, while a hybrid training strategy combining independent and coupled noise timesteps allows each modality to serve as either a generation target or a conditioning input at inference time. The authors also employ on-the-fly MIDI pseudo-label generation using automatic transcription models to alleviate the scarcity of large-scale paired audio–MIDI data.
The model is evaluated on ten music tasks across multiple datasets and compared with various task-specific baselines. MuseDuo achieves competitive performance in multi-track generation, source separation, multi-track transcription, and MIDI-to-audio rendering. Ablation studies further demonstrate the importance of the MIDI-VAE, decoupled noise timesteps, and multimodal fusion strategies.
Critical strengths
- A practically valuable research objective.
The paper aims to unify audio and symbolic music modeling tasks that have traditionally been addressed separately, enabling generation, transcription, rendering, and editing within a single set of model weights. This unified framework has clear practical relevance, particularly for music production workflows requiring frequent interactions between audio and MIDI representations.
- Direct empirical support for the core training mechanisms.
The proposed hybrid noise-timestep strategy enables bidirectional conditional generation between audio and MIDI. Table 10 shows that using exclusively coupled noise timesteps substantially degrades performance on several cross-modal tasks, while removing the MIDI-VAE also causes severe degradation in MIDI-related generation tasks. These results provide strong empirical support for the effectiveness of the core design choices.
- Broad experimental coverage with competitive results across multiple tasks.
The paper evaluates music generation, separation, transcription, rendering, and editing against specialized models using datasets such as Slakh, URMP, MoisesDB, and MUSDB18. MuseDuo achieves notable improvements in multi-track generation, source separation, and several cross-modal tasks, while also providing ablation studies and some analysis of inference efficiency.
Critical weaknesses
- Limited Novelty of the Core Multimodal Generative Mechanism
The main technical components of MuseDuo have substantial precedents in prior work. Existing multimodal diffusion models have already employed modality-specific noise levels to support joint and conditional generation, while the DiT backbone, continuous latent representations, and shared-noise multi-track generation also build upon established techniques. Consequently, MuseDuo's primary contribution appears to lie in integrating existing multimodal generative mechanisms into a unified audio–MIDI framework, rather than introducing a fundamentally new modeling principle. Its independent methodological novelty therefore requires further clarification.
Although Table 10 demonstrates the importance of decoupled noise timesteps for cross-modal tasks, the experiments only compare the proposed hybrid strategy with fully coupled timesteps, without investigating fully independent sampling or alternative mixing ratios. Furthermore, the MIDI-VAE largely adapts an existing audio-VAE architecture. The paper would benefit from a clearer explanation of which contributions constitute novel modeling mechanisms and which represent domain-specific adaptations or system-level integration of existing approaches.
- Insufficient Validation of Audio–MIDI and Cross-stem Consistency
Accurate temporal and note-level correspondence is central to unified audio–MIDI modeling, yet the evaluation of consistency in jointly generated outputs remains insufficient. Section 4.10 primarily evaluates audio–MIDI consistency by automatically transcribing the generated audio and comparing the transcription with the generated MIDI. However, the MIDI pseudo-labels used during training are also produced by transcription models such as Basic Pitch and ADTOF. If the evaluation relies on the same or closely related transcription models, the results may be affected by shared systematic biases, making it difficult to fully establish the accuracy of the generated audio–MIDI correspondence at the level of actual musical notes.
Moreover, MuseDuo inherits the shared-noise multi-track generation mechanism from Stemphonic, but different stems are processed without explicit cross-stem communication or synchronization constraints. Table 3 primarily evaluates individual stem quality using KAD, FAD, and CLAP, without directly measuring rhythmic, onset-level, or structural synchronization across generated stems. Therefore, while the results demonstrate multi-track generation capability, they do not sufficiently establish that the generated stems and their corresponding MIDI representations remain reliably consistent in musical structure.
- Insufficient Isolation of Multitask Learning Benefits and Limited Statistical Reliability
Although MuseDuo achieves strong results on several tasks, the current experiments do not sufficiently disentangle the benefits of joint audio–MIDI modeling from those of multitask training itself. For example, compared with Stemphonic*, MuseDuo introduces not only the MIDI modality but also additional cross-modal tasks and associated training data construction strategies. Consequently, improvements in source separation and multi-track generation cannot be attributed solely to joint MIDI modeling. Similarly, the MIDI-VAE ablation simultaneously changes the representation distribution, continuity, and encoding structure, making it difficult to isolate the specific mechanism responsible for the observed performance degradation.
Furthermore, the paper does not report variability across independent training runs, confidence intervals, or statistical significance tests for the main results. Several evaluation sets contain multiple segments from the same musical recordings, while datasets such as URMP have relatively few independent musical pieces. Evaluating uncertainty at the song or performance level would therefore be important for assessing result stability. Additionally, MuseDuo does not consistently outperform specialized models on all tasks, particularly on some single-track transcription and text-to-MIDI metrics, suggesting that claims regarding the overall benefits of unified training should be interpreted more cautiously.
- Incomplete Implementation Details and Limited Reproducibility
The method relies on approximately 24,000 hours of proprietary licensed music data and a large pretrained DiT, but several important details of the final joint audio–MIDI fine-tuning procedure remain insufficiently specified. These include the number of training updates, learning rate, task sampling proportions, complete MIDI-VAE configurations, and pseudo-label generation and filtering procedures. Additionally, Equation (2) combines the audio and MIDI velocity-prediction losses without sufficiently clarifying whether modality-specific normalization or loss balancing is applied to account for differences in latent dimensionality and numerical scale. These details are important for reproducing the training procedure and understanding performance trade-offs across tasks.
More importantly, the authors explicitly state that they do not plan to release their training code, data pipeline, or model checkpoints. Although music licensing restrictions are understandable and the paper evaluates on public datasets, this substantially limits independent verification of the training setup, pseudo-label quality, and certain reimplemented baselines. Given that the main contributions depend on large-scale multitask training and cross-modal modeling, the missing implementation details further weaken the reproducibility and verifiability of the experimental findings.
Questions for authors
- Methodological Novelty and Noise Scheduling
Compared with existing multimodal diffusion models employing modality-specific noise levels, what are the main independent methodological contributions of MuseDuo? Could the authors further explain the rationale for using 50% independent and 50% coupled noise timesteps, and provide comparisons with alternative mixing ratios or fully independent sampling?
- Audio–MIDI and Cross-stem Consistency
Could the authors further validate note-level consistency between jointly generated audio and MIDI using transcription models independent of the pseudo-label teachers, or datasets with ground-truth MIDI annotations? Additionally, could they compare shared versus independent noise and provide direct metrics for onset alignment, rhythmic synchronization, or other forms of musical consistency across generated stems?
- Multitask Contribution and Statistical Reliability
Could the authors provide an audio-only baseline matched in task composition and training budget to disentangle the benefits of joint MIDI modeling from those of additional multitask training? Could they also report performance variability across independent training runs, together with confidence intervals and statistical significance tests aggregated at the song or performance level?
- Training Details and Reproducibility
Could the authors provide the complete joint fine-tuning configuration, including the number of training steps, task sampling proportions, MIDI-VAE hyperparameters, and pseudo-label quality-control procedures? Are the audio and MIDI losses in Equation (2) independently normalized or weighted? Furthermore, given that the model and training code will not be released, could more detailed baseline implementation and data deduplication protocols be provided to facilitate independent verification?
- Flow Matching Conventions
Section 3.1 defines the forward interpolation as , while the velocity prediction target is , which has the opposite sign to the derivative of the interpolation with respect to . Could the authors explicitly specify the reverse-time ODE and numerical integration update used during inference to clarify the time-direction and velocity-sign conventions?
7 comments
[W2, Q2] Cross-stem consistency. Regarding cross-stem consistency, we additionally design and conduct a blind pairwise listening study on multi-track generation. For each prompt + stem combination, raters hear MuseDuo's output and a baseline's output. We ask raters “which clip do the instruments play together better, in time and in key?” (59 raters, 144 pairwise comparisons over 48 distinct prompt + stem combinations) to specifically focus on cross-stem coherence. We will add the exact question and the rating interface to the appendix.
The results (Table R3-3) indicate that MuseDuo outperforms MGE-LDM significantly (61.1% preferred, ), and also outperforms MuseDuo inferenced without shared noise across stems (59.3% preferred, ), indicating the importance and efficacy of the noise sharing technique. MuseDuo and Stemphonic are rated almost the same (48.7% preferred, not significant) on the cross-stem coherence aspect.
Regarding objective metrics evaluating the coherence of multi-track generation, the Stemphonic paper claimed that the COCOLA metric is ineffective due to potential domain mismatch (the training data of COCOLA are mostly synthetic while theirs are mostly real recordings, similar to ours). There was a controlled experiment conducted in Stemphonic, showing that noise sharing is effective in improving mix quality and text control, which matched our impression from our early experiments. Additionally, we compute the Spearman correlation between human preference and the COCOLA metric in every pair of model comparison, and find that the correlation is weak/insignificant (Table R3-3b). We also find that beat alignment is not suitable due to often inaccurate beat detection on stem-wise audio. Thus, we rely on human evaluation to demonstrate the coherence between stems, as our conducted listening study was designed to answer.
Table R3-3. Multi-track generation listening test results (59 raters, 696 votes)
| MuseDuo vs. | Wins / votes | MuseDuo win rate | 95% CI | p-value |
|---|---|---|---|---|
| MGE-LDM | 140 / 229 | 61.1% | 53.9–68.7% | < 0.01 |
| MuseDuo, w/o noise sharing | 137 / 231 | 59.3% | 50.9–67.5% | < 0.05 |
| Stemphonic | 115 / 236 | 48.7% | 41.4–56.0% | > 0.1 |
Table R3-3b. Correlation between COCOLA and human preference (multi-track generation, 95% CI in brackets)
| MuseDuo vs. | COCOLA–human Spearman correlation |
|---|---|
| MGE-LDM | 0.10 [−0.18, 0.37] |
| MuseDuo, w/o noise sharing | 0.18 [−0.13, 0.46] |
| Stemphonic | 0.26 [−0.04, 0.52] |
[W3, Q3] Regarding multitask contribution to performance and statistical reliability.
- Regarding matched audio control, we additionally train Stemphonic* with uniform timestep sampling for this rebuttal, and provide the following ablation to compare how the MIDI modality and timestep sampling strategy affect the multi-track generation and separation tasks. Although the result for Audio-only with uniform timestep setting wins slightly for the separation metrics, joint audio-MIDI setting supports a much wider range of tasks, additionally enabling single-track transcription, multi-track transcription, timbre transfer, MIDI-to-audio rendering, multi-track text-to-MIDI generation, and multiple note-level note editing tasks.
Table R3-4. MuseDuo ablation: {joint, audio-only} × {Uniform(0,1), logit-normal (LN)} timestep sampling. (95% CI of per-clip metrics in [, ].)
| Modalities (named model in paper) | Timestep | Sep. SDRi ↑ | Sep. Mel-L2 ↓ | Sep. KAD<sub>M</sub> ↓ | Sep. FAD<sub>V</sub> ↓ | Gen. CLAP ↑ | Gen. KAD<sub>M</sub> ↓ | Gen. FAD<sub>V</sub> ↓ |
|---|---|---|---|---|---|---|---|---|
| Audio & MIDI (MuseDuo) | Uniform | 5.846 [5.771, 5.996] | 2.613 [2.549, 2.669] | 0.873 | 0.321 | 0.316 [0.312, 0.319] | 4.259 | 0.723 |
| Audio & MIDI | Logit-normal | 2.463 [2.300, 2.699] | 3.963 [3.872, 4.051] | 1.019 | 0.330 | 0.306 [0.303, 0.309] | 4.720 | 0.760 |
| Audio-only | Uniform | 6.300 [6.214, 6.441] | 2.454 [2.397, 2.513] | 0.858 | 0.315 | 0.312 [0.309, 0.316] | 4.262 | 0.678 |
| Audio-only (Stemphonic*) | Logit-normal | 3.705 [3.545, 3.919] | 3.685 [3.603, 3.769] | 0.999 | 0.335 | 0.305 [0.301, 0.308] | 3.727 | 0.671 |
- For statistical reliability, we will add Appendix D to our draft with 95% CIs (bootstrapped) for every per-clip metric, and paired tests for all 77 baseline comparisons and 42 ablation comparisons. 45 of the 77 favor MuseDuo significantly, 20 are not significant, and 12 favor the baseline. (Note that this is after we switch the transcription model used for evaluation from "Basic Pitch + ADTOF" to "MuScriptor-M" following Rev. duvQ's suggestion, so as not to favor our method.) We will soften the wording wherever a claim is based on a difference that isn’t significant.
[W4, Q4] Regarding training details and reproducibility
The training configuration for DiT generative backbone is as follows: we finetune both MuseDuo and Stemphonic* from the same text-to-music base model, with all hyperparameters the same. The only difference is MuseDuo uses an additional MIDI-VAE, and a MIDI head at the final layer of DiT for the MIDI modality.
Table R3-5. Training and inference configurations.
| Setting | MuseDuo | Stemphonic* |
|---|---|---|
| Initialization | Base model (App. A.2) | Base model (App. A.2) |
| Fine-tuning steps | 110k | 110k |
| Model size (Trainable parameters) | 1.6B | 1.6B |
| Batch size | 64 | 64 |
| Learning rate | 1e-4 | 1e-4 |
| Multitrack data (hours) | 2k | 2k |
| Training tasks | MT generation, accompaniment, separation, removal, mix generation, all with corresponding MIDI inputs/outputs | MT generation, accompaniment, separation, removal, mix generation |
| C_m (MIDI latent channels) | 64 | N/A (audio only) |
| Inference steps | 80 | 80 |
| CFG scale | 5.0 | 5.0 |
We also provide the key hyperparameters of our MIDI-VAE below.
Table R3-6. MIDI-VAE hyperparameters
| Group | Hyperparameter | Value |
|---|---|---|
| Input | Piano-roll channels | 88 pitches × {onset, frame} + 5 drum classes (ADTOF) = 181 |
| Input | Piano-roll frame rate | ≈86 Hz |
| Input | Training crop length | 32 s |
| Architecture | Encoder / decoder type | 1D convolution + attention hybrid, as in the audio autoencoder [1] |
| Architecture | Temporal downsampling factor | 7 |
| Architecture | Parameters (encoder + decoder) | 158M |
| Latent | Latent channels C_m | 64 |
| Latent | Latent frame rate | 12.3 Hz (same as the audio latent) |
| Training | Training data | 2k h multitrack audio, Basic Pitch + ADTOF pseudo-labels |
| Training | Reconstruction loss | Binary cross-entropy on onset, frame, and drum rows |
| Training | KL weight β | 1 |
| Training | Optimizer, learning rate, batch size, steps | AdamW, 1e-4, 64, 45k |
- Regarding how the audio and MIDI losses are weighted against each other: the losses on the audio and MIDI latents in Equation (2) are concatenated, i.e., every latent MIDI/audio dimension is weighted equally during training.
[Q5] Regarding flow-matching conventions
We define the target as the denoising direction (draft line 155 "... velocity that transport noise back to data"), and therefore there is no inconsistency. We provide the full clarification below.
Forward path. Data is at and noise at :
Training target. The network regresses
the velocity pointing from noise to data. Equivalently, with . The opposite sign to is therefore by design.
Sampling ODE. Generation solves
backward in time, from to .
Numerical update at inference. On the grid , we start from and, for , take the Euler step
so each step moves along , toward the data. We use 80 steps with equal spacing.
[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. in ICML, 2023.
We tag each response with the weakness (W) and question (Q) it addresses, as indexed in the review.
[W1, Q1] Regarding methodological novelty and noise scheduling. We thank the reviewer for pointing this key aspect out. The closest prior work in terms of generative modeling mechanism, UniDiffuser [3], trains a transformer from scratch on semantically aligned images and text and introduces per-modality timesteps for switching between joint, conditional and marginal sampling. MuseDuo adopts this mechanism, but is special in four respects (Table R3-1):
We will make these distinctions more clear in our introduction and related work.
Table R3-1. Comparing MuseDuo with UniDiffuser
[W1, Q1] Noise timestep independent/joint mixing ratio. We are additionally training a model with "100% independent" timesteps between audio and MIDI modalities, and will follow up with the results and comparison with our "100% joint" and "50% joint, 50% independent" (i.e., MuseDuo default setup) variants once ready.
[W2, Q2] Audio-MIDI consistency. We follow the reviewer's suggestion and additionally use MuScriptor as the transcriber to check the audio–MIDI adherence.
Table R3-2. Audio–MIDI consistency test with different transcribers (recomputed on all 2.5K MoisesDB+MusDB multi-track sample sets, 95% CI listed after ±.)
The audio-MIDI consistency scores are lower with MuScriptor as expected. MuseDuo uses Basic Pitch and ADTOF to transcribe audio for training, so the generation results are closer to the output distribution of Basic Pitch and ADTOF, showing natural differences among transcription systems. Even when measured with MuScriptor, an independent transcriber, the audio–MIDI consistency remains non-trivial. We also refer the reviewer to the multi-track generation examples on our demo page to examine the correspondence between MuseDuo's MIDI and audio outputs.