- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on detecting whether individual components of a song were used in training of a music generation model. The authors experiment on multiple DoRA fine-tuning runs of a Stable Audio 3 Small diffusion model. Importantly, the paper distinguishes exact-mixture membership from constituent exposure using candidate-matched denoising scores. They find loss differences consistent with constituent exposure in mixtures that were not themselves used for fine-tuning. Additionally, the paper explores hyperbolic representations of musical components. Hyperbolic representations achieve better containment prediction than Euclidean representations under certain evaluation settings.
Critical strengths
-
The paper investigates an interesting problem of detecting the training exposure of music components. The membership inference/data contamination literature is quite saturated, so focusing on innovative and novel aspects in this branch of research is important. The problem statement is the strongest aspect of the paper in my view.
-
The authors evaluate on multiple datasets and include comparisons between several types of training. (However, see weaknesses). Additionally, the paper includes several additional experiments, ablations, and analyses of its limitations
Critical weaknesses
-
The membership-inference evaluation is limited to fine-tuned models. raising some concerns about the generality and soundness of the results. I am aware of the challenging nature of running membership inference problems (open checkpoints with public data splits are hard to find, and training from scratch is costly). Still, the paper would be strengthened by experimentally exploring a full model training pipeline. (see for example: Dubiński et al., Towards More Realistic Membership Inference Attacks on Large Diffusion Models (WACV 2024))
-
The benefit of hyperbolic representations is limited.
Questions for authors
The comparison between hyperbolic and Euclidean representations may not be fully sound. In Section 4.3, the authors state that 1) some hyperparameters were selected based on the test set, and 2) the tuning was performed only for the hyperbolic model; the Euclidean baseline used the same settings as the hyperbolic model. The difference in results between hyperbolic and Euclidean is small. Would the hyperbolic approach still outperform the Euclidean approach if both were tuned independently (without using test set results)?
I maintain my positive score after the rebuttal.
Thank you for your review and suggestions.