- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper tackles the problems of real-world corruptions and fast-improving synthetic song generators by proposing a new dataset BASS, which contains 23 audio corruptions. They also propose CARLA a 2 stage framework that learns a real-music density from human songs and adapts to new generators using a LoRA adaptor trained with fewer samples (at most 100). The main motivation is the low performance of existing SOTA models on unseen generators and corruptions. They combine 4 frozen audio encoders and try to model the density of real audio in stage 1 using projectors and a normalising flow, utilising both real and corrupted songs. To capture unseen generators, they add LoRA modules per generator to learn generator-specific features using few-shot samples.
Critical strengths
-
The Problem is well motivated, interesting and useful, providing people an opportunity to build upon this dataset especially in deployment where new models arise commonly.
-
The 2-stage workflow is interesting and useful in real-world scenarios.
-
Modelling the real audio as a density and then adding adaptors for new generators is a different and interesting approach compared to what others in this domain have used/explored.
-
The synthetic dataset is mostly well-built (see weaknesses), and the controlled perturbations are an added benefit.
Critical weaknesses
-
The model is extremely inefficient and only slightly better than SpecTTTra (SONICS paper), CARLA requires passing the audio through 4 audio encoders, which are complex foundational models, causing much more compute, while only being slightly better than the other compared models.
-
Ablations are not there for selecting foundational models and contribution of individual backbones; why weren't other models like Whisper (and variants of whisper) and other newer models tried?
-
The human songs come from MTG-Jamendo, while synthetic songs come from Suno and Mureka, and the real and fake songs are also not matched by genre, so it could be that the trained model is learning that. And the real audio are all 30-second clips and the model operates at 60 seconds; this doesn't make sense
-
No evaluation on MoM (Batra et al., 2025) as well SONICS (Rahman et al., 2025)
-
Different versions of Suno and Mureka dont really count as unseen as they would have the signatures of their respective platforms, so the authors cant claim that they generalise to unseen generators.
-
No real life corruptions or recordings through microphones; all robustness testing is controlled and does not make a good claim regarding the method.
-
Sensitivity to few-shot samples is too random i think as performance decreases as we go from 100 to 200, not sufficiently explained using embedding based selection or multiple seeds
Questions for authors
-
Why were four audio encoders required? There are no ablations showing the contribution of each encoder individually.
-
Can the authors report inference time, FLOPs, memory consumption, and throughput compared to SpecTTTra?
-
Can the authors evaluate alternative pretrained audio encoders, such as different Whisper versions or more recent encoders?
-
How do the authors ensure that CARLA is not learning dataset-specific differences, such as 30-second clip lengths or unmatched genres?
-
How are different audio durations handled, given that the model takes 1-minute audio inputs?
-
Can the authors evaluate the effects of changing corruption parameters or applying multiple corruptions simultaneously?
-
Can the authors evaluate CARLA on a completely unseen generator, such as Udio from MoM?
-
Can the authors report results across multiple random seeds for support-sample selection and investigate why performance decreases when the number of support samples increases from 100 to 200? This is surprising and may be related to seed variability.
-
Can the authors compare CARLA against simpler baselines, such as frozen pretrained models with few-shot adapters? Some straightforward baselines appear to be missing.