acceptodds
Under review as a conference paper at ICLR 2027

NeuroBand: A Neural Dubber of Any Musical Instrument

Abstract

Humans are born with the ability to associate synchronized music with musical instrument performance videos. However, existing video-to-audio (V2A) models struggle to generate high-quality and temporally synchronized multi-category music. In this work, we address this issue by introducing an audio generation task based on instrument performance videos. To this end, we present InstruVideo, a novel video-text paired benchmark consisting of over 812 hours of clips across 106 different musical instruments and providing a rich and diverse dataset of musical performances. We propose NeuroBand, a diffusion model specifically designed to generate high-fidelity audio from instrument performance videos. NeuroBand leverages Variational Autoencoder (VAE) and vocoders pre-trained on large audio datasets to enhance the quality of the generated audio. Experimental results on three instrumental test sets, e.g., YouTube-Music, Douyin-Music, and MUSIC-solo, demonstrate that NeuroBand pre-trained on large audio datasets can highly enhance the quality of the generated audio. Furthermore, employing flow matching as an optimization technique improves the temporal synchronization between the generated audio and the corresponding video. The dataset, code and models will be made publicly available for further research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-visual generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6r26OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to tackle the "instrument-performance-video to music" task, which is a sub-task in video-to-audio generation.

The audio backbone of this model is built upon the AudioLDM framework: mel VAE, vocoder. Addtionally, Flow-Matching was used to train the generative model.

However, there are severe problems regarding the citation and baselines. For example, MMAudio is one of the most famous V2A model, but is not cited in this paper. This paper's baseline models are mainly from early 2024, competitive models like Frieren or MMAudio are never mentioned. In 2025 and 2026, we see other progresses in V2A built upon MMAudio, and none of them is cited or compared in this paper.

Moreover, the paper considered flow-matching as a novel V2A generative model in line 76 - 77, which is not true in the year of 2026.

My impression is that, the paper is either a manuscript completed in early 2024, or the paper is trying to pick up weak baselines to unfairly improve the performance of their proposed method.

I have to give a "clear reject" to this manuscript

Critical strengths

Clear presentation.

This helped me to realize the context of this paper stopped in the year of 2024.

Critical weaknesses

As mentioned in the Summary session, all strong V2A baselines since the late 2024 are not compared or mentioned or cited.

Specificly, the missing of MMAudio is inacceptable, because this is the first paper that solved the temporal alignment challenge in V2A tasks.

Questions for authors

Did I miss the supplementary file for audio samples? If they were not provided, could you consider to share some audio samples during the rebuttal?

Reviewer_3VR7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

he paper proposes "dub any musical instrument", a video-to-audio task in which a model generates instrument audio from a silent performance video, matching both the instrument's timbre and the performer's timing. To support the task, the authors collect InstruVideo, ~292k 10-second clips (~812 h) across 106 instruments. The clips come from YouTube-8M and VGGSound, plus newly collected Bilibili and Douyin videos, and vocals are removed with Demucs. They also propose NeuroBand, a latent diffusion model. It comes in three variants: LDM, ALDM (AudioLDM VAE/vocoder), and FM (rectified-flow training). The models are evaluated on two in-domain test sets (YouTube-Music, Douyin-Music) and one out-of-distribution set (MUSIC-solo). The metrics are FAD, KL, FD, FAVD and AV-Align, complemented by a pairwise human study reported as Elo scores and ablations over sampling steps and CFG scale.

Critical strengths

A well-motivated task. The paper correctly identifies that general video-to-audio models are developed and evaluated at a coarse category level, whereas performance videos require instrument-faithful generation (timbre, pitch, and timing). Framing music-performance dubbing as a fine-grained task is valuable, even though the current evaluation does not yet measure instrument fidelity directly.

Critical weaknesses
  1. The reference audio is noisy, which calls the whole evaluation suite into question. All reported metrics (FAD, KL, FD, FAVD) compare generated audio against the dataset's own audio, but that audio is not clean instrument sound. The only cleaning step is vocal removal with Demucs (Sec. 3.3), which (i) adds separation artifacts, (ii) does not remove crowd noise, applause or other background sound (the authors themselves describe Douyin-Music as "contaminated by background and crowd noise", Sec. 5.2), and (iii) may partially remove melodic instruments in the vocal range (e.g., erhu, violin, flute, suona), which are exactly the cases the fine-grained claim is about. The paper does not check separation quality or report how much of the test set still contains non-instrument sound. As a result, a model that generates clean, faithful instrument audio may be penalised for not reproducing crowd noise or separation artifacts. In KL, for example, a reference containing applause pushes the classifier's predictions toward non-music classes. This matters to our assessment because every quantitative claim in the paper relies on these metrics. Without a clean reference set (e.g., a manually verified subset), the reported gains cannot be attributed to better instrument dubbing as opposed to better reproduction of the noise.

  2. The methodological contribution is marginal, and it is evaluated against outdated baselines that were not trained on the same data.

    • Composition of existing components. NeuroBand combines off-the-shelf parts without task-specific modification: the CAVP video encoder from Diff-Foley, the SD-v1.4 U-Net, the AudioLDM VAE and vocoder, the rectified-flow recipe of Esser et al. (2024), and standard classifier-free guidance. No component targets instrument timbre, which the paper names as the core challenge of the task. The paper's own tables suggest that most of the improvement comes from replacing the Griffin-Lim vocoder with AudioLDM's pretrained one. NeuroBand (LDM), which follows Diff-Foley's recipe but is trained on InstruVideo, improves on Diff-Foley by only ~1–1.5 FAD. NeuroBand (ALDM), which swaps in the AudioLDM VAE and vocoder, improves by a further ~6 FAD on YouTube-Music (15.94 → 9.80).
    • Unequal training data. The baselines are run with their official public weights (Reproducibility Statement), i.e., trained on other data, while NeuroBand is trained on InstruVideo's own training split. Because the distributional metrics reward matching the reference distribution, including its noise (see Weakness 1), the comparison confounds the method with in-domain training. The reported advantage of NeuroBand could therefore stem simply from the different training data rather than from the model itself. This should have been verified, e.g., by retraining at least some baselines on InstruVideo.
    • Outdated baselines and related work. None of the generative works discussed or compared against is later than 2024, even though video-to-audio and video-to-music generation have advanced considerably since then. Relevant recent works include:

    This matters to our assessment because, without baselines retrained on InstruVideo or comparisons to current methods, the state-of-the-art claims do not show that NeuroBand improves on existing methods. The remaining contribution then rests largely on the dataset.

  3. No audio samples are provided, so the generated audio cannot be assessed. For a paper on audio generation, the only qualitative evidence is Fig. 6: a set of small mel-spectrograms without axes or frequency labels, compared with ground truth. The accompanying text claims that the model captures "the nuances of key strikes and pedal usage" and "the subtleties of bowing techniques and string vibrations". These claims cannot be verified from spectrograms at this resolution, and most readers cannot judge timbre or synchronization from a spectrogram anyway. No demo page or supplementary audio/video examples are provided. This matters to our assessment because, given the concerns about the metrics (Weakness 1) and the limited human study, listening examples (including side-by-side comparisons with the baselines on the same videos) would be the most direct way to judge whether NeuroBand produces faithful, synchronized instrument audio.

Questions for authors

Questions

  1. Data release and mitigation. For the newly collected Bilibili and Douyin videos, what mitigation do the authors plan regarding (i) identifiable human faces, (ii) minors appearing in the videos, and (iii) distribution rights for the recorded performances? Under what license do the authors intend to release the dataset, and will the release include the media files themselves or only links/identifiers?

  2. Baselines trained on different data. Why were the baselines not retrained on InstruVideo, the same data used to train NeuroBand? Without this, how do the authors explain what causes the performance gap: the model design, the in-domain training data, or the pretrained AudioLDM VAE and vocoder?

Suggestions

  1. Rephrase the "dub any musical instrument" framing. The task name (Introduction, paragraph 2) and the title suggest the model works for instruments it has never seen. However, the model is trained on a fixed set of 106 instrument classes, and no experiment tests it on instruments outside that set. MUSIC-solo is described as out-of-distribution, but its instruments appear to overlap with the training classes, so it tests a change in data source rather than new instruments. Unless the authors add a held-out-instrument (zero-shot) evaluation, we suggest a more accurate name such as "multi-instrument performance dubbing", and toning down the "any instrument" claim throughout the paper.

  2. Drop the claim that flow matching is a novel training strategy for video-to-audio. The Introduction (lines 75–77) presents flow matching as "a novel training strategy for video-to-audio models". However, flow matching is already well established for video-to-audio generation, e.g., Frieren (Wang et al., NeurIPS 2024) and MMAudio (Cheng et al., CVPR 2025), neither of which is cited. Moreover, the formulation used here (linear interpolation path, logit-normal timestep sampling) follows Esser et al. (2024) without modification. We suggest presenting flow matching as an adopted design choice, citing the prior V2A works that use it, and removing the unsupported statement that it addresses "gradient vanishing commonly encountered in traditional diffusion models" (lines 320–322).

  3. Report the human study as pairwise preference percentages (Fig. 4). Elo scores are designed for incomplete, unbalanced pairings (as in Chatbot Arena). In this study, every model pair is compared on every video, so pairwise win rates can be reported directly and are far easier to interpret: a gap of ~100 Elo points corresponds to only a ~64% win rate. We suggest reporting win rates with confidence intervals, together with the number of participants. Fig. 4 could also show each per-baseline comparison separately, e.g., NeuroBand's win rate against each baseline, color-coded by baseline, for both preference and alignment. This would be much more informative than a single Elo point per model.

  4. Replace the current ablation study with analyses that bear on the main claims. The ablation on sampling steps and CFG scale (Sec. 5.4, Fig. 5) is a search over inference hyperparameters. It does not support any of the paper's claims and takes space away from the core results. We suggest dropping it entirely and using the space for one of the following:

    • (i) Ablating the actual components of the method, to show where the gains come from. For example: training data (InstruVideo vs. the baselines' data), the VAE and vocoder (Griffin-Lim vs. AudioLDM), and the training objective (diffusion vs. flow matching, using the same sampler and number of steps).
    • (ii) Preferably, a deeper analysis of the noisy cases the data may contain. For example: how much of the training and test data still contains non-instrument sound (crowd noise, applause, separation artifacts); how vocal removal affects instruments in the vocal range; and how the metrics and model ranking change when evaluating on a manually verified clean subset. This would directly address the concerns in Weakness 1.
Reviewer_jiVyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies audio generation from silent videos of musical instrument performances, aiming to balance audio quality, instrument semantics, and temporal synchronization with the performance. The authors construct InstruVideo, which contains approximately 292k 10-second audiovisual clips, over 812 hours, and 106 instrument categories. They clean the reference audio through vocal separation, establish the YouTube-Music and Douyin-Music test sets, and additionally evaluate on MUSIC-solo as an out-of-distribution test set. NeuroBand combines a CAVP video encoder, an audio-pretrained VAE and vocoder, and a conditional U-Net. The paper compares LDM, ALDM, and flow matching (FM) variants. FM uses linear interpolations between ground-truth audio latents and Gaussian noise as training states and regresses the velocity given by their difference. At inference, the conditional velocity field is integrated from noise and decoded into a waveform, without requiring the original audio track. ALDM achieves lower FAD and FAVD across the three test sets, while FM obtains higher AV-Align on the first two. Validation includes objective metrics, human pairwise comparisons covering a limited set of models, and ablations on sampling steps and CFG.

Critical strengths
  1. The task and dataset have clear practical value. The work focuses on timbre and temporal relationships in musical performances, with coverage extending to 106 instrument categories. It provides a concrete setting for evaluating general V2A models on instrument performances. This supports systematic research on instrument-performance dubbing across a broad range of categories.
  2. The gains from audio-specific components are visible across multiple test sets. ALDM obtains FAD values of 9.804/2.315/17.140, compared with 11.906/8.315/21.124 for the best prior methods listed in the respective tables. FAVD also decreases on all three sets. MUSIC-solo adds some evidence beyond the training distribution alongside the other test sets.
Critical weaknesses
  1. The main arguments for FM are not sufficiently aligned with the comparison conditions and observed results. FM uses 50 Euler steps, whereas ALDM uses 25 DPM-solver steps (p. 7). There is no comparison of measured latency, matched computational budgets, or matched target quality, leaving the claim of "faster sampling" without empirical support. FM's AV-Align exceeds ALDM's on YouTube-Music and Douyin-Music, but on MUSIC-solo it is 0.303, below ALDM's 0.320. FD on that set also increases from 24.832 to 43.295. Figure 5 is insufficient to establish consistent synchronization improvements across settings, and linear interpolation and Euler updates alone do not demonstrate that vanishing gradients have been resolved. The paper needs to clarify which conclusions are experimentally supported and incorporate the synchronization-quality trade-off into its main conclusions. This directly affects the defensible scope of the method's contribution.
  2. The benchmark protocol and controls do not adequately separate the benefits of task-specific training from those of the method. NeuroBand is trained on InstruVideo, whereas the reproducibility statement says that prior baselines use official weights. The paper does not yet provide enough information to determine whether training data and opportunities for domain adaptation are comparable. The authors have stated that all Bilibili data are used for training and all Douyin data for testing, with sliding windows mainly applied to these supplementary platforms. YouTube data generally use one 10-second clip per original video. These settings reduce concerns about directly splitting sliding-window clips across training and testing. Several details still need clarification to better interpret the out-of-distribution and long-tail results. These include deduplication across the source YouTube datasets and cross-platform reposts, separation rules based on original video IDs, and the distribution of the 106 categories. There is no evidence here to assert that leakage occurred. The subjective comparison in Figure 4 includes only ALDM, Diff-Foley, and Im2Wav, and therefore cannot support a perceptual advantage over all baselines. The respective benefits of task-specific training and the proposed method require clearer separation.
Questions for authors

1.Please compare ALDM and FM in terms of latency, quality, and synchronization using the same hardware, batch size, duration, and clearly specified CFG settings. Alternatively, please provide results from existing experiments with a matched sampling budget. What supports the claim that FM is "faster"? If such data are unavailable, would you narrow the claims about efficiency and resolving vanishing gradients? 2. Were the baselines trained or fine-tuned on InstruVideo? Please specify their training data, pretrained components, and inference conditions. Please clarify which claims concern direct transfer of general-purpose models and which concern algorithmic improvements under comparable domain adaptation. 3. The paper states that all Bilibili data are used for training and all Douyin data for testing. Please further explain how original video IDs are separated across the source YouTube datasets, whether cross-platform reposts and near duplicates are removed, and the exact test-set sizes and category distributions. How does the 280k figure in Table 1 relate to the more than 283k training samples stated in the text? These details affect the interpretation of the benchmark's generalization evidence. 4. Why does FM exhibit both lower AV-Align and substantially higher FD on MUSIC-solo? Can existing results from repeated sampling or different configurations establish whether these findings are stable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.