acceptodds
Under review as a conference paper at ICLR 2027

MuseDuo: Unified Audio-MIDI Music Generation, Separation, Transcription, Rendering, and Editing

Abstract

Music generation has developed as two largely separate fields: symbolic generation, which produces MIDI or scores, and audio generation, which models waveforms. Musicians work across both, composing and arranging in symbolic terms and working directly with timbre and texture in the audio domain. We present MuseDuo, a diffusion/flow transformer (DiT) that is 1) trained on both music mixes and multi-track instrument stems, and 2) jointly models audio and MIDI, enabling a broad range music tasks. MuseDuo comprises three key mechanisms: (i) a unified audio-MIDI multimodal latent rectified-flow architecture with a novel MIDI-VAE, (ii) a concise training recipe that realizes diverse tasks via systematic inference-time configuration of input, output, and conditioning, and (iii) on-the-fly MIDI extraction that alleviates the need for costly annotation. We fine-tune a pretrained text-to-audio music DiT, and evaluate on 10 music tasks spanning text-to-audio, text-to-MIDI, multi-stem audio generation, open-vocabulary source separation, transcription, MIDI-to-audio rendering, timbre transfer, and note-level editing. With a unified design, MuseDuo matches or outperforms 12 recent task-specific baselines on the majority of evaluated tasks and metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
joint audio-MIDI modeling, rectified flow, music generation, music source separation, music transcription, MIDI-to-audio synthesis, timbre transfer, music editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qyXVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuseDuo, a unified multimodal framework for jointly modeling audio and MIDI, aiming to support a wide range of music tasks, including music generation, source separation, MIDI transcription, audio rendering, timbre transfer, and note-level editing, within a single model.

Specifically, the authors extend a pretrained music generation DiT with a MIDI-VAE that maps discrete piano-roll representations into continuous latent sequences temporally aligned with audio latents. A shared Transformer jointly models the two modalities, while a hybrid training strategy combining independent and coupled noise timesteps allows each modality to serve as either a generation target or a conditioning input at inference time. The authors also employ on-the-fly MIDI pseudo-label generation using automatic transcription models to alleviate the scarcity of large-scale paired audio–MIDI data.

The model is evaluated on ten music tasks across multiple datasets and compared with various task-specific baselines. MuseDuo achieves competitive performance in multi-track generation, source separation, multi-track transcription, and MIDI-to-audio rendering. Ablation studies further demonstrate the importance of the MIDI-VAE, decoupled noise timesteps, and multimodal fusion strategies.

Critical strengths
  1. A practically valuable research objective.

The paper aims to unify audio and symbolic music modeling tasks that have traditionally been addressed separately, enabling generation, transcription, rendering, and editing within a single set of model weights. This unified framework has clear practical relevance, particularly for music production workflows requiring frequent interactions between audio and MIDI representations.

  1. Direct empirical support for the core training mechanisms.

The proposed hybrid noise-timestep strategy enables bidirectional conditional generation between audio and MIDI. Table 10 shows that using exclusively coupled noise timesteps substantially degrades performance on several cross-modal tasks, while removing the MIDI-VAE also causes severe degradation in MIDI-related generation tasks. These results provide strong empirical support for the effectiveness of the core design choices.

  1. Broad experimental coverage with competitive results across multiple tasks.

The paper evaluates music generation, separation, transcription, rendering, and editing against specialized models using datasets such as Slakh, URMP, MoisesDB, and MUSDB18. MuseDuo achieves notable improvements in multi-track generation, source separation, and several cross-modal tasks, while also providing ablation studies and some analysis of inference efficiency.

Critical weaknesses
  1. Limited Novelty of the Core Multimodal Generative Mechanism

The main technical components of MuseDuo have substantial precedents in prior work. Existing multimodal diffusion models have already employed modality-specific noise levels to support joint and conditional generation, while the DiT backbone, continuous latent representations, and shared-noise multi-track generation also build upon established techniques. Consequently, MuseDuo's primary contribution appears to lie in integrating existing multimodal generative mechanisms into a unified audio–MIDI framework, rather than introducing a fundamentally new modeling principle. Its independent methodological novelty therefore requires further clarification.

Although Table 10 demonstrates the importance of decoupled noise timesteps for cross-modal tasks, the experiments only compare the proposed hybrid strategy with fully coupled timesteps, without investigating fully independent sampling or alternative mixing ratios. Furthermore, the MIDI-VAE largely adapts an existing audio-VAE architecture. The paper would benefit from a clearer explanation of which contributions constitute novel modeling mechanisms and which represent domain-specific adaptations or system-level integration of existing approaches.

  1. Insufficient Validation of Audio–MIDI and Cross-stem Consistency

Accurate temporal and note-level correspondence is central to unified audio–MIDI modeling, yet the evaluation of consistency in jointly generated outputs remains insufficient. Section 4.10 primarily evaluates audio–MIDI consistency by automatically transcribing the generated audio and comparing the transcription with the generated MIDI. However, the MIDI pseudo-labels used during training are also produced by transcription models such as Basic Pitch and ADTOF. If the evaluation relies on the same or closely related transcription models, the results may be affected by shared systematic biases, making it difficult to fully establish the accuracy of the generated audio–MIDI correspondence at the level of actual musical notes.

Moreover, MuseDuo inherits the shared-noise multi-track generation mechanism from Stemphonic, but different stems are processed without explicit cross-stem communication or synchronization constraints. Table 3 primarily evaluates individual stem quality using KAD, FAD, and CLAP, without directly measuring rhythmic, onset-level, or structural synchronization across generated stems. Therefore, while the results demonstrate multi-track generation capability, they do not sufficiently establish that the generated stems and their corresponding MIDI representations remain reliably consistent in musical structure.

  1. Insufficient Isolation of Multitask Learning Benefits and Limited Statistical Reliability

Although MuseDuo achieves strong results on several tasks, the current experiments do not sufficiently disentangle the benefits of joint audio–MIDI modeling from those of multitask training itself. For example, compared with Stemphonic*, MuseDuo introduces not only the MIDI modality but also additional cross-modal tasks and associated training data construction strategies. Consequently, improvements in source separation and multi-track generation cannot be attributed solely to joint MIDI modeling. Similarly, the MIDI-VAE ablation simultaneously changes the representation distribution, continuity, and encoding structure, making it difficult to isolate the specific mechanism responsible for the observed performance degradation.

Furthermore, the paper does not report variability across independent training runs, confidence intervals, or statistical significance tests for the main results. Several evaluation sets contain multiple segments from the same musical recordings, while datasets such as URMP have relatively few independent musical pieces. Evaluating uncertainty at the song or performance level would therefore be important for assessing result stability. Additionally, MuseDuo does not consistently outperform specialized models on all tasks, particularly on some single-track transcription and text-to-MIDI metrics, suggesting that claims regarding the overall benefits of unified training should be interpreted more cautiously.

  1. Incomplete Implementation Details and Limited Reproducibility

The method relies on approximately 24,000 hours of proprietary licensed music data and a large pretrained DiT, but several important details of the final joint audio–MIDI fine-tuning procedure remain insufficiently specified. These include the number of training updates, learning rate, task sampling proportions, complete MIDI-VAE configurations, and pseudo-label generation and filtering procedures. Additionally, Equation (2) combines the audio and MIDI velocity-prediction losses without sufficiently clarifying whether modality-specific normalization or loss balancing is applied to account for differences in latent dimensionality and numerical scale. These details are important for reproducing the training procedure and understanding performance trade-offs across tasks.

More importantly, the authors explicitly state that they do not plan to release their training code, data pipeline, or model checkpoints. Although music licensing restrictions are understandable and the paper evaluates on public datasets, this substantially limits independent verification of the training setup, pseudo-label quality, and certain reimplemented baselines. Given that the main contributions depend on large-scale multitask training and cross-modal modeling, the missing implementation details further weaken the reproducibility and verifiability of the experimental findings.

Questions for authors
  1. Methodological Novelty and Noise Scheduling

Compared with existing multimodal diffusion models employing modality-specific noise levels, what are the main independent methodological contributions of MuseDuo? Could the authors further explain the rationale for using 50% independent and 50% coupled noise timesteps, and provide comparisons with alternative mixing ratios or fully independent sampling?

  1. Audio–MIDI and Cross-stem Consistency

Could the authors further validate note-level consistency between jointly generated audio and MIDI using transcription models independent of the pseudo-label teachers, or datasets with ground-truth MIDI annotations? Additionally, could they compare shared versus independent noise and provide direct metrics for onset alignment, rhythmic synchronization, or other forms of musical consistency across generated stems?

  1. Multitask Contribution and Statistical Reliability

Could the authors provide an audio-only baseline matched in task composition and training budget to disentangle the benefits of joint MIDI modeling from those of additional multitask training? Could they also report performance variability across independent training runs, together with confidence intervals and statistical significance tests aggregated at the song or performance level?

  1. Training Details and Reproducibility

Could the authors provide the complete joint fine-tuning configuration, including the number of training steps, task sampling proportions, MIDI-VAE hyperparameters, and pseudo-label quality-control procedures? Are the audio and MIDI losses in Equation (2) independently normalized or weighted? Furthermore, given that the model and training code will not be released, could more detailed baseline implementation and data deduplication protocols be provided to facilitate independent verification?

  1. Flow Matching Conventions

Section 3.1 defines the forward interpolation as z(t)=(1−t)z+tϵz(t)=(1-t)z+t\epsilon, while the velocity prediction target is z−ϵz-\epsilon, which has the opposite sign to the derivative of the interpolation with respect to tt. Could the authors explicitly specify the reverse-time ODE and numerical integration update used during inference to clarify the time-direction and velocity-sign conventions?

7 comments

We tag each response with the weakness (W) and question (Q) it addresses, as indexed in the review.

[W1, Q1] Regarding methodological novelty and noise scheduling. We thank the reviewer for pointing this key aspect out. The closest prior work in terms of generative modeling mechanism, UniDiffuser [3], trains a transformer from scratch on semantically aligned images and text and introduces per-modality timesteps for switching between joint, conditional and marginal sampling. MuseDuo adopts this mechanism, but is special in four respects (Table R3-1):

  1. Its modalities are temporally aligned frame by frame: the MIDI-VAE encodes the piano roll at the audio latent’s frame rate. The generative model therefore is required to learn and process the time-wise tight coupling between the MIDI and audio modalities.
  2. It combines cross-modal conditioning with external text and reference-audio conditions, which enables separation, accompaniment, single-pass multi-track transcription, inpainting and note-level editing, supporting a wide range of creative music tasks previously not possible with any single audio- or symbolic-domain music model, to our knowledge.
  3. It is resource-wise friendly to build and adopt:
    • Fine-tuned from a text-to-music checkpoint. Text-to-music models are widely available in the research community. We show that MuseDuo can be finetuned from one of these, demonstrating that no expensive or lengthy custom pretraining is required to gain multi-task music abilities.
    • No MIDI annotations required. MIDI is extracted on the fly from multitrack audio, so it needs no paired MIDI with the audio, i.e., there is no data collection cost involved.
  4. Its key design choices (shared RoPE, input fusion, MIDI-VAE) are shown to be important to its performance in our ablation study (Table 10 in paper), which adds to the reusable insights of our work.

We will make these distinctions more clear in our introduction and related work.

Table R3-1. Comparing MuseDuo with UniDiffuser

UniDiffuserMuseDuo
AlignmentSemanticTemporal (frame-level)
Init.From scratchText-to-music checkpoint
Symbolic dataPaired captionsExtracted on the fly (no GT)
Symbolic latentCLIP + GPT-2MIDI-VAE
Cross-modal positionNoneShared RoPE
FusionConcatConcat / add (ablated)
External cond.NoneText + ref. audio
Evaluated tasks210

[W1, Q1] Noise timestep independent/joint mixing ratio. We are additionally training a model with "100% independent" timesteps between audio and MIDI modalities, and will follow up with the results and comparison with our "100% joint" and "50% joint, 50% independent" (i.e., MuseDuo default setup) variants once ready.

[W2, Q2] Audio-MIDI consistency. We follow the reviewer's suggestion and additionally use MuScriptor as the transcriber to check the audio–MIDI adherence.

Table R3-2. Audio–MIDI consistency test with different transcribers (recomputed on all 2.5K MoisesDB+MusDB multi-track sample sets, 95% CI listed after ±.)

TranscriberOnset F1Frame F1Drums F1
Basic Pitch + ADTOF0.606 ± 0.0070.815 ± 0.0050.665 ± 0.008
MuScriptor-M0.445 ± 0.0080.636 ± 0.0080.576 ± 0.008

The audio-MIDI consistency scores are lower with MuScriptor as expected. MuseDuo uses Basic Pitch and ADTOF to transcribe audio for training, so the generation results are closer to the output distribution of Basic Pitch and ADTOF, showing natural differences among transcription systems. Even when measured with MuScriptor, an independent transcriber, the audio–MIDI consistency remains non-trivial. We also refer the reviewer to the multi-track generation examples on our demo page to examine the correspondence between MuseDuo's MIDI and audio outputs.

[W2, Q2] Cross-stem consistency. Regarding cross-stem consistency, we additionally design and conduct a blind pairwise listening study on multi-track generation. For each prompt + stem combination, raters hear MuseDuo's output and a baseline's output. We ask raters “which clip do the instruments play together better, in time and in key?” (59 raters, 144 pairwise comparisons over 48 distinct prompt + stem combinations) to specifically focus on cross-stem coherence. We will add the exact question and the rating interface to the appendix.

The results (Table R3-3) indicate that MuseDuo outperforms MGE-LDM significantly (61.1% preferred, p<0.01p < 0.01), and also outperforms MuseDuo inferenced without shared noise across stems (59.3% preferred, p<0.05p < 0.05), indicating the importance and efficacy of the noise sharing technique. MuseDuo and Stemphonic are rated almost the same (48.7% preferred, p>0.1p > 0.1 not significant) on the cross-stem coherence aspect.

Regarding objective metrics evaluating the coherence of multi-track generation, the Stemphonic paper claimed that the COCOLA metric is ineffective due to potential domain mismatch (the training data of COCOLA are mostly synthetic while theirs are mostly real recordings, similar to ours). There was a controlled experiment conducted in Stemphonic, showing that noise sharing is effective in improving mix quality and text control, which matched our impression from our early experiments. Additionally, we compute the Spearman correlation between human preference and the COCOLA metric in every pair of model comparison, and find that the correlation is weak/insignificant (Table R3-3b). We also find that beat alignment is not suitable due to often inaccurate beat detection on stem-wise audio. Thus, we rely on human evaluation to demonstrate the coherence between stems, as our conducted listening study was designed to answer.

Table R3-3. Multi-track generation listening test results (59 raters, 696 votes)

MuseDuo vs.Wins / votesMuseDuo win rate95% CIp-value
MGE-LDM140 / 22961.1%53.9–68.7%< 0.01
MuseDuo, w/o noise sharing137 / 23159.3%50.9–67.5%< 0.05
Stemphonic115 / 23648.7%41.4–56.0%> 0.1

Table R3-3b. Correlation between COCOLA and human preference (multi-track generation, 95% CI in brackets)

MuseDuo vs.COCOLA–human Spearman correlation
MGE-LDM0.10 [−0.18, 0.37]
MuseDuo, w/o noise sharing0.18 [−0.13, 0.46]
Stemphonic0.26 [−0.04, 0.52]

[W3, Q3] Regarding multitask contribution to performance and statistical reliability.

  • Regarding matched audio control, we additionally train Stemphonic* with uniform timestep sampling for this rebuttal, and provide the following ablation to compare how the MIDI modality and timestep sampling strategy affect the multi-track generation and separation tasks. Although the result for Audio-only with uniform timestep setting wins slightly for the separation metrics, joint audio-MIDI setting supports a much wider range of tasks, additionally enabling single-track transcription, multi-track transcription, timbre transfer, MIDI-to-audio rendering, multi-track text-to-MIDI generation, and multiple note-level note editing tasks.

Table R3-4. MuseDuo ablation: {joint, audio-only} × {Uniform(0,1), logit-normal (LN)} timestep sampling. (95% CI of per-clip metrics in [⋅\cdot, ⋅\cdot].)

Modalities (named model in paper)TimestepSep. SDRi ↑Sep. Mel-L2 ↓Sep. KAD<sub>M</sub> ↓Sep. FAD<sub>V</sub> ↓Gen. CLAP ↑Gen. KAD<sub>M</sub> ↓Gen. FAD<sub>V</sub> ↓
Audio & MIDI (MuseDuo)Uniform5.846 [5.771, 5.996]2.613 [2.549, 2.669]0.8730.3210.316 [0.312, 0.319]4.2590.723
Audio & MIDILogit-normal2.463 [2.300, 2.699]3.963 [3.872, 4.051]1.0190.3300.306 [0.303, 0.309]4.7200.760
Audio-onlyUniform6.300 [6.214, 6.441]2.454 [2.397, 2.513]0.8580.3150.312 [0.309, 0.316]4.2620.678
Audio-only (Stemphonic*)Logit-normal3.705 [3.545, 3.919]3.685 [3.603, 3.769]0.9990.3350.305 [0.301, 0.308]3.7270.671
  • For statistical reliability, we will add Appendix D to our draft with 95% CIs (bootstrapped) for every per-clip metric, and paired tests for all 77 baseline comparisons and 42 ablation comparisons. 45 of the 77 favor MuseDuo significantly, 20 are not significant, and 12 favor the baseline. (Note that this is after we switch the transcription model used for evaluation from "Basic Pitch + ADTOF" to "MuScriptor-M" following Rev. duvQ's suggestion, so as not to favor our method.) We will soften the wording wherever a claim is based on a difference that isn’t significant.

[W4, Q4] Regarding training details and reproducibility

The training configuration for DiT generative backbone is as follows: we finetune both MuseDuo and Stemphonic* from the same text-to-music base model, with all hyperparameters the same. The only difference is MuseDuo uses an additional MIDI-VAE, and a MIDI head at the final layer of DiT for the MIDI modality.

Table R3-5. Training and inference configurations.

SettingMuseDuoStemphonic*
InitializationBase model (App. A.2)Base model (App. A.2)
Fine-tuning steps110k110k
Model size (Trainable parameters)1.6B1.6B
Batch size6464
Learning rate1e-41e-4
Multitrack data (hours)2k2k
Training tasksMT generation, accompaniment, separation, removal, mix generation, all with corresponding MIDI inputs/outputsMT generation, accompaniment, separation, removal, mix generation
C_m (MIDI latent channels)64N/A (audio only)
Inference steps8080
CFG scale5.05.0

We also provide the key hyperparameters of our MIDI-VAE below.

Table R3-6. MIDI-VAE hyperparameters

GroupHyperparameterValue
InputPiano-roll channels88 pitches × {onset, frame} + 5 drum classes (ADTOF) = 181
InputPiano-roll frame rate≈86 Hz
InputTraining crop length32 s
ArchitectureEncoder / decoder type1D convolution + attention hybrid, as in the audio autoencoder [1]
ArchitectureTemporal downsampling factor7
ArchitectureParameters (encoder + decoder)158M
LatentLatent channels C_m64
LatentLatent frame rate12.3 Hz (same as the audio latent)
TrainingTraining data2k h multitrack audio, Basic Pitch + ADTOF pseudo-labels
TrainingReconstruction lossBinary cross-entropy on onset, frame, and drum rows
TrainingKL weight β1
TrainingOptimizer, learning rate, batch size, stepsAdamW, 1e-4, 64, 45k
  • Regarding how the audio and MIDI losses are weighted against each other: the losses on the audio and MIDI latents in Equation (2) are concatenated, i.e., every latent MIDI/audio dimension is weighted equally during training.

[Q5] Regarding flow-matching conventions

We define the target vv as the denoising direction (draft line 155 "... velocity vv that transport noise back to data"), and therefore there is no inconsistency. We provide the full clarification below.

Forward path. Data is at t=0t=0 and noise at t=1t=1:

z(t)=(1−t) z+t ϵ,dz(t)dt=ϵ−z.z(t) = (1-t)\,z + t\,\epsilon, \qquad \frac{dz(t)}{dt} = \epsilon - z .

Training target. The network v^θ(z(t),t,C)\hat v_\theta(z(t), t, \mathcal{C}) regresses

v=z−ϵ=−dz(t)dt,v = z - \epsilon = -\frac{dz(t)}{dt},

the velocity pointing from noise to data. Equivalently, v=dz/dτv = dz/d\tau with τ=1−t\tau = 1 - t. The opposite sign to dz/dtdz/dt is therefore by design.

Sampling ODE. Generation solves

dz(t)dt=− v^θ(z(t),t,C),z(1)=ϵ∼N(0,I),\frac{dz(t)}{dt} = -\,\hat v_\theta\big(z(t), t, \mathcal{C}\big), \qquad z(1) = \epsilon \sim \mathcal{N}(0, I),

backward in time, from t=1t=1 to t=0t=0.

Numerical update at inference. On the grid 0=t0<t1<⋯<tN=10 = t_0 < t_1 < \dots < t_N = 1, we start from ztN=ϵz_{t_N} = \epsilon and, for i=N,N−1,…,1i = N, N-1, \dots, 1, take the Euler step

zti−1=zti+(ti−ti−1) v^θ(zti,ti,C),ti−ti−1>0,z_{t_{i-1}} = z_{t_i} + (t_i - t_{i-1})\,\hat v_\theta\big(z_{t_i}, t_i, \mathcal{C}\big), \qquad t_i - t_{i-1} > 0,

so each step moves along v^θ\hat v_\theta, toward the data. We use 80 steps with equal spacing.

[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. in ICML, 2023.

Reviewer_XUJWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MuseDuo, a flow-matching DiT model that generates multi-stem music audio, aligned MIDI, or their combinations, with controllability based on reference audio and text descriptions. The model is fine-tuned from a pre-trained text-to-audio music DiT with stem MIDI data transcribed on the fly. Both audio and MIDI (represented as piano roll with separate channels for note onset, sustain, and drum) first go to a VAE for continuous latent extraction, with the frame rate unified and concatenated along the time-step dimension. A single DiT then processes noised fused latents in one pass and predicts the rectified-flow velocity to denoise the data. The audio and MIDI have independent noise levels, so one can serve as condition for the other to generate at inference time, thus uncovering cross-modal transcription/synthesis tasks.

The proposed model is trained using in-house licensed audio data, with stem MIDI transcribed on the fly. Experiments evaluate on 10 tasks covering mix and stem generation in audio and symbolic domains, as well as audio-to-MIDI transcription and MIDI-to-audio synthesis. Results demonstrate enhanced audio fidelity (measured by FAD and KAD) compared to existing text-to-music generation models, superior text control in multi-stem generation, and enhanced quality in audio source separation. While surpassing Muscriptor in multi-track music transcription, its monophonic instrument transcription capability is bounded by the on-the-fly labelling models used during fine-tuning. Additional experiments demonstrate superior text-to-MIDI generation quality and note-level editing capability.

Critical strengths
  • Unified model across a bunch of music generation and understanding tasks. As demonstrated in Table 1, a unified formulation of q(za,zm∣C)q(\textbf{z}_a, \textbf{z}_m | \mathcal{C}) gives rise to a bunch of music generative tasks, including mix and multi-stem generation in audio and symbolic domains, as well as source separation. An independent noise level trick further uncovers cross-modal transcription and synthesis tasks under a unified model. Experiments are conducted in comparison to existing models for individual tasks, and the proposed model leads in most cases.

  • Dual-directional audio and MIDI modelling demonstrates enhanced cross-modal understanding compared to one-directional modelling. The experiments show significant improvements in multi-instrument transcription compared to Muscriptor, indicating the benifit of cross-modal multi-task modelling, which may provide good insights to follow up research in this domain.

Critical weaknesses
  • While the improvement in multi-instrument transcription is impressive, the results on monophonic transcription are largely bounded by the labelling models used in fine-tuning. In addition, the paper did not evaluate the case of polyphonic transcription, such as piano transcription. Since there is quite abundant annotated piano audio-MIDI data, it would be interesting to make use of this and see whether the inclusion of human-labelled (though confined to piano) data could lead to cross-domain enhancement that may benefit multi-instrument or monophonic transcription as well.

  • In the case of multi-stem generation, the reviewer understands that individual stems are processed in a batch that only shares the initial input noise, while the later DiT processes go per-stem and are independent. It is not clear how the stem-wise coherence can be maintained in this way.

Questions for authors
  • Following the last point in Weaknesses, is there any more explicit account for inter-track / inter-stem coherence in the multi-track/stem generation? Are there any metrics to evaluate this aspect?

  • Line 175: ".. the audio is an isolated instrument recording...": Is the model input audio always stem-split audio, or is there a ratio for mix vs stem input?

  • In Line 434, Section 4.2, instead of MusicGen Large, the baseline uses the Medium variant. Is this for comparable model size? What is the number of trainable parameters of the proposed model, and how does it compare to each baseline model?

3 comments

We refer to the reviewer's weaknesses as W1–W2 and questions as Q1–Q3, in the order they appear in the review.

[W2, Q1] Regarding multi-track coherence. We additionally design and conduct a blind pairwise listening study on multi-track generation that focuses on evaluating cross-stem coherence (we also conduct one on timbre transfer; see our response to Reviewer duvQ). For each prompt + stem combination, raters hear MuseDuo's output and a baseline's output and answer “which clip do the instruments play together better, in time and in key?” (59 raters, 144 pairwise comparisons over 48 distinct prompt + stem combinations). We will add the exact question and the rating interface to the appendix.

  • The results (Table R2-1) indicate that MuseDuo outperforms MGE-LDM significantly (61.1% preferred, p<0.01p < 0.01), and also outperforms MuseDuo inferenced without shared noise across stems (59.3% preferred, p<0.05p < 0.05), indicating the importance and efficacy of the noise sharing technique. MuseDuo and Stemphonic are rated almost the same (48.7% preferred, p>0.1p > 0.1 not significant) on the cross-stem coherence aspect.
  • Regarding objective metrics evaluating the coherence of multi-track generation, the Stemphonic paper claimed that the COCOLA [2] metric is ineffective due to potential domain mismatch (the training data of COCOLA are mostly synthetic while theirs are mostly real recordings, similar to ours). There was a controlled experiment conducted in Stemphonic, showing that noise sharing is effective in improving mix quality and text control, which matched our impression in our early experiments. Additionally, we compute the correlation between human preference and the COCOLA metric, and find that the correlation is weak/insignificant (Table R2-1b). We also find that beat alignment is often not suitable due to inaccurate beat detection on stem-wise audio. Thus, we rely on human evaluation to demonstrate the coherence between stems, as our conducted listening study was designed to answer.

Table R2-1. Multi-track generation listening test results (59 raters, 696 votes)

MuseDuo vs.Wins / votesMuseDuo win rate95% CIp-value
MGE-LDM140 / 22961.1%53.9–68.7%< 0.01
MuseDuo, w/o noise sharing137 / 23159.3%50.9–67.5%< 0.05
Stemphonic115 / 23648.7%41.4–56.0%> 0.1

Table R2-1b. Correlation between COCOLA and human preference (multi-track generation, 95% CI in brackets)

MuseDuo vs.COCOLA–human Spearman correlation
MGE-LDM0.10 [−0.18, 0.37]
MuseDuo, w/o noise sharing0.18 [−0.13, 0.46]
Stemphonic0.26 [−0.04, 0.52]

[Q2] Regarding training tasks ratio. Both MuseDuo and Stemphonic* are trained on a variety of tasks; the task ratios and their inputs/outputs are shown in the table below.

Table R2-2. Training tasks ratio

TaskInputOutputTraining ratio
Single-track mix generationText promptMix audio0.2
Multi-track stem generationText prompt + instrument label per stemStems (one per instrument)0.2
Separation and multi-track transcriptionMix audio (reference condition) + target instrument labelTarget stem(s)0.2
RemovalMix audio (reference condition) + label of the instrument to removeMix without the removed instrument0.2
AccompanimentExisting stem(s) (reference condition) + text prompt / instrument labelNew stem(s) that fit the input0.2

[2] Ruben Ciranni, Giorgio Mariani, Michele Mancusi, Emilian Postolache, Giorgio Fabbro, Emanuele Rodolà, and Luca Cosmo, “COCOLA: Coherence-oriented contrastive learning of musical audio representations,” in ICASSP, 2025.

[Q3] Regarding baseline model size. MusicGen medium (1.84B) is chosen since it has similar model size as MuseDuo (1.6B). We report the model size of all included baselines for completeness:

Table R2-3. Baseline model sizes

ModelParameters (inference)
MuseDuo (ours)1.6B
MusicGen-Medium†1.84B
Stable Audio 3 medium1.70B
Stemphonic1.6B
MGE-LDM1.02B
MuseControlLite1.14B
MIDI-DDSP5.93M
Spectrogram Diffusion (base w/ context)412M
MuScriptor-M300M
MuScriptor-L1.3B
Basic Pitch16.8K
ADTOF450K
AudioSep239M
SAM-Audio (Base)1B
YourMT3+ (YPTF.MoE+Multi variant)45.8M
Text2midi0.3B
MIDI-LLM1.5B

† MusicGen-Medium is officially labeled 1.5B, but its release weights contain 1.84B parameters.

[W1] Regarding transcription. We agree that MuseDuo's MIDI output quality and accuracy, in all transcription tasks and in MIDI generation, are bounded by the labeling models (Basic Pitch and ADTOF). However, we note that this may improve with better transcription models and/or leveraging real MIDI data to train MuseDuo, which we leave as future work. As for polyphonic music, which the reviewer asked about: MuseDuo's multi-instrument transcription operates on polyphonic mixtures, and the single-instrument inputs from the Slakh dataset used for evaluation also include polyphonic content (e.g., piano).

AuthorsOpenReview ↗

Follow-up on our response before the discussion period ends

Thank you again for your positive and constructive review. As the discussion period closes soon, we would like to check whether our responses address your questions:

  • Cross-stem coherence (W2, Q1): we conducted a blind listening study focusing specifically on cross-stem coherence, listeners prefer MuseDuo over MGE-LDM (61.1%) and over the same model without shared noise (59.3%). This is now Sec. 4.12 of the revised paper.
  • Mix vs. stem inputs (Q2): the training task ratios are in Table R2-2 and Appendix A.2.
  • Model sizes (Q3): parameter counts for MuseDuo and all baselines are in Table R2-3 and Appendix F.
  • Transcription (W1): we acknowledge the limitation from using Basic Pitch + ADTOF for MIDI pseudo labeling, and note that the Slakh evaluation includes polyphonic instruments such as piano.

Please let us know if anything is still unclear. We would be glad to follow up.

Reviewer_duvQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MuseDuo fine-tunes a pretrained 1.6B rectified-flow text-to-music DiT so that it jointly models audio latents and MIDI. MIDI is encoded as a 181-channel piano roll by a new MIDI-VAE at the audio-latent frame rate. The two modalities are concatenated along the sequence with shared RoPE positions, and each modality has its own noise timestep, decoupled for 50% of training samples. This single checkpoint can therefore sample the joint distribution (generation), MIDI given audio (transcription), and audio given MIDI (rendering). Adding mix/submix channel conditioning and Stemphonic-style stem packing extends the same model to multitrack generation, source separation, multitrack transcription, accompaniment, timbre transfer, and note-level editing (Table 1). Training MIDI is pseudo-labelled from licensed stems with Basic Pitch / ADTOF. The model is compared with 12 task-specific baselines, and Table 10 ablates the main design choices.

Critical strengths

S1. One checkpoint covers a broad set of audio–MIDI tasks, with large margins on several. MIDI-to-audio Onset F1 is 0.794/0.844 vs. 0.544/0.573 for MuseControlLite (Table 6). Multitrack transcription Frame F1 is 0.449/0.613 vs. 0.316/0.537 for the best MuScriptor (Table 5). Held-out separation SDRi is 5.92 vs. 1.42 for MGE-LDM (Table 4). The unified formulation is practical and the engineering effort is substantial; this is what keeps my score near the borderline.

S2. The ablations give reusable design knowledge (Table 10).

  • Tying t_a = t_m collapses the cross-modal tasks: single-track transcription, MIDI-to-audio and timbre-transfer Onset F1 drop to 0.006, 0.008 and 0.004.
  • Replacing the MIDI-VAE with a raw piano roll collapses MIDI generation (CLAP 0.106), while MIDI as a condition (M2A 0.742) is barely affected. A poorly matched representation therefore mainly hurts the modality being generated.
  • Switching to the base model's logit-normal schedule reduces SDRi from 5.85 to 2.46, which suggests that strongly conditioned, near-deterministic tasks need training coverage of extreme noise levels.
Critical weaknesses

W1. The central thesis, that joint MIDI modelling makes the model better at audio, is never isolated, and the paper's own ablation points to another cause.

The paper frames joint modelling as the point of the work: "music generation methods should be aligned to operate in both domains jointly rather than in either one alone" (p.1 l.036–037). It presents Stemphonic* as "isolating the effect of joint MIDI modeling" (p.6 l.317–318), and calls multitrack transcription, separation and generation "the stronger evidence of the benefits of joint audio-MIDI modeling" (p.10 l.514–516). The evidence does not support this attribution:

  • Stemphonic* is not a matched control. Its objective "remains that of Eq. (4)" (p.18 l.927), i.e. logit-normal timesteps (p.17 l.871–874), whereas MuseDuo samples t ~ U(0,1) (p.5 l.248). Changing only the schedule in MuseDuo drops SDRi from 5.85 to 2.46 (Table 10), below Stemphonic*'s 3.71. The schedule alone could therefore account for the whole separation gap in Table 4 (3.71 vs. 5.85).
  • The "raw piano roll" row of Table 10 is not a no-MIDI ablation: it still models MIDI, only with a worse representation. Its MIDI generation collapses (CLAP 0.106), yet SDRi rises to 6.18. In the paper's own data, better symbolic modelling does not translate into better separation.
  • MuseDuo's own initialisation (base model or Stemphonic*), number of updates and task mixture are not reported, so the two models' training budgets cannot be compared either.
  • Multitrack transcription has no audio-only counterpart at all, and the stated reason for the gain, "performing separation and transcription jointly in a single pass" (p.7 l.378–p.8 l.400), is not tested against a separate-then-transcribe cascade.
  • For multitrack generation, the claim that "adding the MIDI modality and additional tasks does not come at the cost of multi-track audio generation" (p.7 l.352–354) rests on the "Stemphonic" row of Table 3, and it is unclear which version and schedule that row uses.

As it stands, the paper shows that one model can do all these tasks, not that doing them jointly helps. To support the thesis, a matched audio-only control (same U(0,1) schedule, initialisation, data, updates and task mixture; ideally the 2×2 of joint vs. audio-only × U vs. LN) should show a clear benefit. Otherwise, the claims on p.1, p.6 and p.10 should be reduced to capability claims.

W2. The perceptual and musical claims are not validated, and the comparison with specialist models is incomplete.

  • No listening study. Every quality claim rests on FAD/KAD, CLAP and transcription-based proxies, and no task has a blind listening test. These proxies do not show whether the reported gains translate into better-sounding music or more coherent arrangements. Even a small blind listening test for multitrack generation (overall quality, musicality, cross-track coherence), MIDI-to-audio / timbre transfer and editing would help. Automatic song-quality predictors such as SongEval (Yao et al., 2025) or SongBench (Wu et al., 2026) can add evidence where they apply to the output, but they cannot replace human evaluation.
  • Multitrack coherence is assumed, not shown. Appendix A.2 treats each stem as an independent batch element, with a per-stem objective and no cross-stem attention; the only coupling between stems is a shared initial noise. Section 3.1 says this "enforces synchronization" (p.4 l.168–169), while A.2 says synchronisation is only "encouraged" (p.18 l.925). Because the loss is per stem, sharing ε during training does not change any single stem's training distribution, so it is unclear how the model learns to coordinate the parts. Nothing in the model targets arrangement-level musicality (complementary voicing, rhythmic locking, parts responding to each other), and Table 3 does not measure it. A direct inter-stem measure is needed, e.g., onset/beat alignment between generated stems or a stem-coherence model such as COCOLA.
  • The demo raises the same concern. In the multitrack examples on the anonymous demo page, the generated vocal stems still contain audible guitar, and the vocals are unintelligible mumbling (the page notes that the model is not trained for lyric-conditioned singing). Since these are presumably selected examples, this casts doubt on both stem purity and the musical value of the multitrack output; a measure of leakage between stems would help.
  • Missing specialist baselines. The headline claim is that MuseDuo "matches or outperforms 12 recent task-specific baselines on the majority of evaluated tasks and metrics" (p.1 l.027–028), which the conclusion upgrades to "comparable or better on almost all tasks and metrics" (p.9 l.485). The baseline set is too weak to support this. Separation is compared only with MGE-LDM and Stemphonic*, the paper's improved re-implementation of Stemphonic, without text-queried separators that have public weights, such as AudioSep (Liu et al., 2023) or SAM Audio (Shi et al., 2025); the MGE-LDM paper itself already compares with AudioSep. Multitrack transcription omits YourMT3+ (Chang et al., 2024), which the paper cites but marks as "supported, not benchmarked" in Table 12. These should be evaluated under the same protocol, or the paper should explain why they cannot be.
  • "Almost all" is overstated even against the reported baselines. All five single-track transcription numbers are below the best baseline (e.g., Slakh Onset 0.508 vs. 0.697, Table 5); URMP timbre-transfer KAD is 28.1 vs. 16.9 (Table 6); major↔minor editability is 0.423 vs. 0.504 (Table 8); text-to-MIDI Chroma/Groove JS are 0.168/0.417 vs. 0.093/0.236 (Table 9); and KAD_M and CLAP trail Stable Audio–M in Table 2. The p.9 conclusions should reflect these trade-offs.

Secondary concerns (less decisive than W1–W2):

W3. Several evaluation details are missing or may favour the proposed model.

  • The paper does not say which transcriber scores the generated audio in Tables 6–8 and in the joint audio–MIDI consistency result (0.61/0.82/0.63, p.9). If it is Basic Pitch / ADTOF, the same models that produced the training MIDI, the metric rewards the model for imitating its own teachers. The scoring transcriber should be named, and the rankings checked with an independent one.
  • For note editing, it is unclear whether editability is measured on the notes that should change or on the whole clip. An identity ("leave the audio unchanged") baseline should be added to Table 8 to show how much of the Fidelity and Editability numbers can be obtained without editing anything.
  • p.8 l.412–414 says MIDI-DDSP was evaluated "on the URMP test split and on a held-out Slakh subset", but Table 7 reports only "curated monophonic Slakh samples" (n = 31 / 77 for M2A / TT), and I could not find URMP numbers anywhere. URMP is MIDI-DDSP's training domain, so this is the comparison that matters most. These results should be reported, or the text corrected.
  • Input information is not matched. MuseControlLite is melody-conditioned (p.8 l.409), while MuseDuo receives full polyphonic MIDI. In multitrack transcription, MuseDuo gets the ground-truth instrument list via <inst> (Table 1); it is not stated whether MuScriptor, which supports instrument conditioning, received the same. Both differences can favour MuseDuo and should be stated with the results.
  • The absolute Slakh multitrack Onset F1 (≈ 0.31) is far below values commonly reported for the MT3 family, which suggests a different protocol; stem matching, crop length and split should be spelled out.
  • No seeds or confidence intervals are reported, while several margins are very small: +0.011 Slakh multitrack Onset (Table 5), +0.007 URMP timbre-transfer Onset (Table 6), +0.005 text-to-MIDI CLAP (Table 9). Bootstrap CIs or multiple seeds are needed to show that these margins are real.

W4. Reproducibility details for MuseDuo itself are missing. Appendix A.2 mainly describes the audio-only stem fine-tuning. For MuseDuo the paper does not report the number of fine-tuning steps, learning rate, multitrack data hours, task sampling ratios, C_m, sampling steps, or CFG settings, nor whether training starts from the base model or from Stemphonic*. The audio autoencoder is cited inconsistently (p.4 l.177 vs. p.4 l.208 and Appendix A.1), so it is unclear which pretrained autoencoder is actually used. Not releasing code, data, or checkpoints is not in itself a reason to reject, but it makes these details all the more important.

Minor issues

  • Writing errors: p.4 l.196 "Section ??"; a duplicated sentence on p.17; an incomplete sentence in §4.1 (l.308–309).
  • §4.4 claims the best realism, but KAD_M is 2.31 vs. 2.08 for Stable Audio–M (Table 2), and fine-tuning makes two MusicFM metrics worse than the base model.
  • "The default ranks first or second on every metric" (p.9 l.466–467) does not hold for MIDI generation CLAP (default 0.321 vs. 0.324 for both addition and tied timesteps).
  • Concatenation is described as "slightly slower" (p.9 l.470–472), but for four tracks it takes 7.3 s vs. 4.5 s, about 62% longer.
  • Accompaniment generation, instrument removal and inpainting are listed as capabilities (p.2 l.066–067, p.6 l.294), but Table 12 marks accompaniment as "supported, not benchmarked", and removal and inpainting have no separate quantitative results. The abstract and Figure 1 should make this clear. The abstract also says "10 music tasks", while Table 12 says "eleven tasks".
  • The URMP dataset is cited as Montesinos et al. (2020), which introduces Solos. Please cite the original URMP paper (Li et al., IEEE TMM 2019, doi:10.1109/TMM.2018.2856090).
  • Using per-modality timesteps so that one model covers joint, marginal and conditional distributions is the central idea of UniDiffuser (Bao et al., ICML 2023). The paper cites it only in related work; §3.5 should credit it where the mechanism is introduced.
Questions for authors

Overall, I value the breadth of a single checkpoint and the informative ablations on decoupled timesteps and the MIDI-VAE. However, the central claim that joint modelling helps is not isolated, output quality is not perceptually validated, and the strongest specialists are missing from the comparison. My current rating is therefore 2 (weak reject). I would raise it to 3 (weak accept) if the rebuttal convincingly addresses questions 1–3 below; question 1 matters most. The other points in W3–W4 and the minor issues would strengthen the paper but are not decisive for my score.

  1. Matched audio-only control (W1). What timestep distribution, initialisation and training budget do Stemphonic* and the "Stemphonic" row in Table 3 use, and where is MuseDuo initialised from? Can you report Tables 3–4 for the matched audio-only control described in W1? If joint training clearly beats it, W1 is resolved. If the two tie, revising the claims on p.1, p.6 and p.10 to capability claims would also resolve W1 for me, since the unified capability is valuable on its own.
  2. Perceptual and multitrack evidence (W2). Can you add a blind listening test and the inter-stem coherence and leakage measures described in W2? How do you explain the guitar in the generated vocal stems on the demo page?
  3. Stronger specialists (W2). Can AudioSep / SAM Audio (separation) and YourMT3+ (multitrack transcription) be evaluated under your protocol? If not, why not?
  4. Evaluation clarifications (W3). Which transcriber scores Tables 6–8 and the p.9 consistency experiment? Is editability computed only on the notes that should change? Do the multitrack-transcription baselines receive the ground-truth instrument list? How are stems matched, and what crop length and split are used?
7 comments

Thank you for raising this. It shows us that our main narrative was not clear enough, and we would like to clarify it.

We did NOT claim that joint MIDI modeling makes the model better at audio in our original draft. As we wrote in our original abstract (line 21 "... a concise training recipe that realizes diverse tasks ...") and 2nd item in contribution list (lines 80 to 83), in this work, we aim to show that by joint MIDI and audio modeling, combining with reference audio condition, MuseDuo is capable of a wide variety of music tasks. Our custom-implemented Stemphonic* baseline serves as the counterpart of MuseDuo without MIDI modality, with matching training data, DiT model architecture/size (less the MIDI input & output projections) and follow the original Stemphonic and common diffusion/flow models for music (e.g., Stable Audio 3, ACE-Step 1.5) to use logit-normal timestep sampling. We follow the reviewer's suggestion to train an additional audio-only checkpoint with our proposed uniform timestep sampling (i.e., the MIDI modality being the only difference, everything else equal to MuseDuo), and report in Table R1-1 the 2x2 ablation results along with their 95% confidence intervals when applicable.

Table R1-1. MuseDuo ablation: {joint, audio-only} × {Uniform(0,1), logit-normal (LN)} timestep sampling. (95% CI of per-clip metrics in [⋅\cdot, ⋅\cdot].)

Modalities (named model in paper)TimestepSep. SDRi ↑Sep. Mel-L2 ↓Sep. KAD$_M$↓Sep. FAD$_V$ ↓Gen. CLAP ↑Gen. KAD_M ↓Gen. FAD_V ↓
Audio & MIDI (MuseDuo)Uniform5.846 [5.771, 5.996]2.613 [2.549, 2.669]0.8730.3210.316 [0.312, 0.319]4.2590.723
Audio & MIDILogit-normal2.463 [2.300, 2.699]3.963 [3.872, 4.051]1.0190.3300.306 [0.303, 0.309]4.7200.760
Audio-onlyUniform6.300 [6.214, 6.441]2.454 [2.397, 2.513]0.8580.3150.312 [0.309, 0.316]4.2620.678
Audio-only (Stemphonic*)Logit-normal3.705 [3.545, 3.919]3.685 [3.603, 3.769]0.9990.3350.305 [0.301, 0.308]3.7270.671

While the result for audio-only with uniform timestep setting wins slightly for the separation metrics, joint audio-MIDI modeling (MuseDuo) supports a much wider range of tasks, including single-track transcription, multi-track transcription, timbre transfer, MIDI-to-audio rendering, multi-track text-to-MIDI generation, and multiple note-level note editing tasks, with competitive performance in each of them as demonstrated in our evaluation. We also note that the advantage of uniform timestep sampling, over the commonly used logit-normal, is our experimental discovery as included in our original ablation study (Table 10 in paper).

We understand that some phrasing in our introduction and experimental findings (Sec. 4) may lead to the confusion that we are claiming "joint audio-MIDI improves audio generation", and will update our draft to prevent misunderstanding.

[W2, Q2, Q3] Regarding perceptual quality and comparisons with specialist models

We thank the reviewer for raising this valid concern and providing actionable suggestions.

[W2.1, W2.2, Q2] We additionally design and conduct listening studies on two representative tasks, multi-track generation and timbre transfer, which have non-overlapping opponent models and focus on generation and editing respectively. Both studies use blind pairwise comparisons: for each input, raters listen to MuseDuo's output and a baseline's output and choose the better one according to the task-specific question. We will add the listening study results to the main text of our paper draft, and the exact questions and the rating interface to the appendix.

  • [W2.1, W2.2] Multi-track generation. We ask raters “which clip do the instruments play together better, in time and in key?” (59 raters, 144 pairwise comparisons over 48 distinct prompt + stem combinations) to specifically focus on cross-stem coherence. The results (Table R1-2) indicate that MuseDuo outperforms MGE-LDM significantly (61.1% preferred, p<0.01p < 0.01), and also outperforms MuseDuo inferenced without shared noise across stems (59.3% preferred, p<0.05p < 0.05), indicating the importance and efficacy of the noise sharing technique. MuseDuo and Stemphonic are rated almost the same (48.7% preferred, p>0.1p > 0.1 not significant) on the cross-stem coherence aspect.
  • [W2.2] Regarding objective metrics evaluating the coherence of multi-track generation, the Stemphonic paper claimed that the COCOLA metric is ineffective due to potential domain mismatch (the training data of COCOLA are mostly synthetic while theirs are mostly real recordings, similar to ours). There was a controlled experiment conducted in Stemphonic, showing that noise sharing is effective in improving mix quality and text control, which matched our impression from our early experiments. Additionally, we compute the Spearman correlation between human preference and the COCOLA metric in every pair of model comparison, and find that the correlation is weak/insignificant (Table R1-2b). We also find that beat alignment is not suitable due to often inaccurate beat detection on stem-wise audio. Thus, we rely on human evaluation to demonstrate the coherence between stems, as our conducted listening study was designed to answer.

Table R1-2. Multi-track generation listening test results (59 raters, 696 votes)

MuseDuo vs.Wins / votesMuseDuo win rate95% CIp-value
MGE-LDM140 / 22961.1%53.9–68.7%< 0.01
MuseDuo, w/o noise sharing137 / 23159.3%50.9–67.5%< 0.05
Stemphonic115 / 23648.7%41.4–56.0%> 0.1

Table R1-2b. Correlation between COCOLA and human preference (multi-track generation, 95% CI in brackets)

MuseDuo vs.COCOLA–human Spearman correlation
MGE-LDM0.10 [−0.18, 0.37]
MuseDuo, w/o noise sharing0.18 [−0.13, 0.46]
Stemphonic0.26 [−0.04, 0.52]
  • [W2.1] Timbre transfer. We ask raters “which clip better plays the ORIGINAL recording’s music on the TARGET instrument?” (63 raters, 664 votes). We compare against MuseControlLite on both pop and orchestral instruments, and MIDI-DDSP on orchestral instruments to match their respective supported target instrument set. MuseDuo outperforms both baselines significantly as we report in Table R1-3 below.

Table R1-3. Timbre transfer listening test results (63 raters, 664 votes)

MuseDuo vs.Wins / votesMuseDuo win rate95% CIp-value
MuseControlLite311 / 44669.7%64.7–74.6%< 0.001
MIDI-DDSP129 / 21859.2%50.2–67.9%< 0.05

[W2.3, Q2] Regarding vocal stem quality. The stem training data of MuseDuo contains mostly instrumental stems (vocal stems make up only 3.2% of all stems). The small number of vocal stems does not come with lyrics annotation (hence the "wordless vocal effect" we noted on our demo page) and admittedly sometimes contain backing instruments, which may have caused the vocal/guitar mix issue the reviewer noticed in that particular demo example. We reiterate that MuseDuo focuses on instrumental music, and note that our evaluation and demo examples across broad instrumental classes reflect our focus.

[W2.4, Q3] Regarding specialist baselines. We follow the reviewer's suggestion and add AudioSep and SAM-Audio (Base, 1B) for separation, and YourMT3+ for transcription. To sum up, MuseDuo outperforms AudioSep on all metrics. SAM-Audio wins on SDRi, but falls short on audio realisticness/quality metrics KAD<sub>M</sub> and FAD<sub>V</sub>. YourMT3+ outperforms both MuScriptor and MuseDuo significantly, likely because Slakh is its in-domain data, i.e., YourMT3+ has seen the same soundfont renders in the Slakh training set. Therefore, in the revised Table 5 in our paper draft, we show YourMT3+ as a reference row and exclude it from the ranking. YourMT3+ is also trained on URMP, and therefore we do not evaluate it on URMP.

Table R1-4. Source separation, full MoisesDB + MusDB test set

ModelSDRi ↑Mel-L2 ↓KAD<sub>M</sub> ↓FAD<sub>V</sub> ↓
Stemphonic*3.713.681.000.33
AudioSep0.324.789.813.12
SAM-Audio Base6.952.613.171.75
MuseDuo (ours)5.852.610.870.32

Table R1-5. Single-track transcription on Slakh

MethodOnset F1 ↑Frame F1 ↑Drums F1 ↑
MuseDuo (ours)0.5080.6540.742
Basic Pitch0.5270.653–
ADTOF––0.757
MuScriptor Small (0.1B)0.6040.6220.760
MuScriptor Medium (0.3B)0.6790.6630.789
MuScriptor Large (1.3B)0.6970.6850.779
YourMT3+ †0.8810.8250.865

† trained on the Slakh training split (in-domain).

Table R1-6. Multi-track transcription on Slakh

MethodOnset F1 ↑Frame F1 ↑Drums F1 ↑
MuseDuo (ours)0.3160.4490.707
MuScriptor Small0.2320.2560.649
MuScriptor Medium0.2660.2810.688
MuScriptor Large0.3050.3160.686
YourMT3+ †0.7220.6660.852

† trained on the Slakh training split (in-domain).

[W2.5] Regarding transcription claims. We agree that MuseDuo's MIDI output quality/accuracy are bounded by its teacher transcribers, Basic Pitch and ADTOF (line 415), as best reflected on single-track transcription performance. We will soften the overall claim in the Conclusion from "comparable or better on almost all tasks and metrics" to "matches or outperforms task-specific baselines on the majority of tasks and metrics". However, we do note that this limitation does not affect our core contribution that joint audio-MIDI modeling enables many tasks for musical creativity, and would also like to add that transcription/MIDI outputs can be improved using stronger transcribers as the teacher.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal summary

We thank all three reviewers for their careful and constructive reviews, and are glad that each sees value in a unified model for a wide range of music tasks. Below, we first clarify our core goal and value of our work along with our evaluation scope and protocol. We then summarize major and minor concerns raised by the reviewers and how address each one to the best of our ability.

Core goal and value. We fine-tune a pretrained text-to-music model, with minimal changes, into a single model that works across audio and MIDI. One set of weights generates mixes, stems, and MIDI from text, separates sources, transcribes audio to MIDI, renders MIDI to audio, transfers timbre, and edits individual notes. Many of these has so far needed its own specialist system. With a finetuning-based approach, we avoid training from scratch which can be time- and resource-wise highly intensive.

How broadly we evaluated it. We compare against recent baselines including many per-task specialists on 10 tasks, with 6 public test sets and 15 baseline systems (MusicGen, Stable Audio 3, Stemphonic, MGE-LDM, AudioSep, SAM-Audio, Basic Pitch, ADTOF, MuScriptor, YourMT3+, Spectrogram Diffusion, MuseControlLite, MIDI-DDSP, Text2midi, and MIDI-LLM). Each task has its own evaluation pipeline and uses the standard metrics of its literature. MuseDuo, with only one checkpoint, matches or outperforms these baselines on the majority of tasks and metrics.

Summary of concerns and responses. From the reviews, we summarize three major areas of concerns raised by the reviewers and several further minor points. We address each below, and plan to include all changes in our revised paper draft (using blue-colored text for clarity).

Major concerns

  1. Whether joint MIDI modeling makes the model better at audio (duvQ Weakness 1 [W1], qyXV W3). We thank the reviewers for raising this, because it shows that our narrative could be more clear. This is not a claim we intend to make. Our message is that fine-tuning a pretrained text-to-music model, with minimal changes, yields one model for a wide range of audio and MIDI tasks. Some of our original wording suggested benefits on audio generation, and we will revise those parts in the paper draft. We also additionally trained the matched audio-only control that the reviewers requested: MuseDuo is on par on generation and slightly worse on separation, but in return it performs many tasks involving MIDI input/output, which the audio-only model cannot. See our response to duvQ [Question 1 (Q1), W1].
  2. Perceptual quality and cross-stem coherence were based on automatic metrics only (duvQ W2, XUJW W2, qyXV W2). We design and run two listening studies, one on multi-track generation focusing on cross-stem coherence, and the other on timbre transfer. For multi-track generation coherence, listeners prefer MuseDuo over MGE-LDM (61.1%) and over MuseDuo without noise sharing (59.3%), and rate it on par with Stemphonic (48.7%). For timbre transfer, they prefer MuseDuo over MuseControlLite (69.7%) and MIDI-DDSP (59.2%). See our responses to duvQ [W2, Q2], XUJW [W2, Q1], and qyXV [W2, Q2].
  3. Novelty relative to existing multimodal diffusion models (qyXV W1). MuseDuo adopts per-modality timesteps from UniDiffuser, which we will credit explicitly in Sec. 3.5. What it adds is frame-level alignment of audio and MIDI, external reference-audio conditions, and a straightforward recipe that starts from a text-to-music checkpoint and needs no MIDI annotations. These aspects collectively make 10+ music tasks possible with just one finetuned model checkpoint. See our response to qyXV [W1, Q1].
AuthorsOpenReview ↗

Rebuttal summary (continued)

Further points

  • Scoring with the teacher transcribers may favor MuseDuo (duvQ W3, qyXV W2). We re-score with an independent transcriber, MuScriptor-M on timbre transfer, MIDI-to-audio, and note-level editing tasks, as well as the consistency between MuseDuo's audio and MIDI outputs. The original conclusions generally do not change. See duvQ [Q4, W3] and qyXV [W2, Q2].
  • Statistical reliability (duvQ W3, qyXV W3). We add bootstrapped 95% confidence intervals and paired significance tests for metrics computed at the clip level, and will remove claims based on non-significant differences in our draft. See duvQ [Q4, W3] and qyXV [W3, Q3].
  • MIDI accuracy is bounded by the teacher transcribers (XUJW W1, duvQ W2). We agree and admitted so in the original paper draft. Stronger teachers or real MIDI data may improve it, but we think this does not affect the core value of our work. See XUJW [W1].
  • Training details and reproducibility (duvQ W4, qyXV W4). We provide the full configuration, multi-task task ratios, and important MIDI-VAE hyperparameters. See qyXV [W4, Q4] and duvQ [W4].
  • Some separation/transcription specialist baselines were missing (duvQ W2). We add AudioSep and SAM-Audio for separation, and YourMT3+ for transcription. MuseDuo outperforms AudioSep. SAM-Audio is better on SDRi, and MuseDuo is better on audio quality. YourMT3+, which is trained on Slakh, is better at transcription there likely due to the evaluation data being in-domain. See duvQ [Q3].
AuthorsOpenReview ↗

The revised paper draft is uploaded

Following up on our summary above: the revised paper is now uploaded, with all changes in blue. Below we state how the main concerns are now addressed in the paper.

Reviewer duvQ tied their assessment to three questions, the first being the most important.

  • Matched audio-only control (Q1): the 2×2 experiment is now in Appendix A.3. We have rewritten the introduction, experiments, and conclusion so that the paper claims exactly what this control supports: joint modeling gives one checkpoint many more tasks, not better audio.
  • Perceptual evidence (Q2): the two blind listening studies, which we conducted for the rebuttal, are now a main-text section (Sec. 4.12). The full protocol and rating interface are in Appendix E.
  • Stronger specialists (Q3): AudioSep, SAM-Audio, and YourMT3+ are now in Tables 4 and 5.

The multi-track listening study also answers a question that XUJW and qyXV both raised: how stems stay coherent when they share only the initial noise. Listeners clearly prefer MuseDuo over the same model without noise sharing.

For qyXV, UniDiffuser is now credited where we introduce per-modality timesteps (Sec. 3.5), together with what MuseDuo adds. We also completed the fully independent timestep run that qyXV asked for. It is now part of the ablation (Table 10) and shows that our 50% hybrid keeps the advantage: it is significantly better on multi-track generation and separation, and trails closely only on one timbre-transfer metric.

Across the paper, note-control results are re-scored with an independent transcriber (MuScriptor-M), and the conclusions are generally unchanged. A new Appendix D gives confidence intervals and paired significance tests for every comparison; where a difference is not significant, we have softened the wording. The full training details, task ratios, MIDI-VAE hyperparameters, and model sizes are in Appendices A.2 and F.

We hope the revised paper makes our claims, and the evidence behind them, easy to check. We sincerely thank the reviewers again for helping us improve our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.