- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces PMus to evaluate how a music continuation relates to its past. It changes the context while scoring the same continuation with an external model. It evaluates five measures through controlled generation experiments and a listening study with 85 musically trained participants.
Critical strengths
- The paper tackles a real gap in music-generation evaluation. Whether the continuation fits well to what came before.
- Its fixed-continuation comparisons are well designed, and the controlled experiments and listening study are very detailed.
Critical weaknesses
- My main concern is that the Self and Partner scores have not been shown to perceptually distinguish continuing one’s own part from responding to another player. PMus-Duo has higher point-estimate agreement on both questions, with no clear advantage for the designated player-specific scores.
- The practical case is also still limited. As the agreement depends strongly on the evaluator, and the study does not establish an advantage over generator-level preferences or usefulness for choosing between outputs from the same generator.
Questions for authors
Appendix C.1 shows that dropping only the first 0.2 s of the target lowers listener agreement from 0.640 to 0.609 for PMus-Self and from 0.634 to 0.602 for PMus-Partner. How much of the reported predictive gain comes from acoustic continuity at the codec splice rather than from musical structure, and can the authors quantify this?
Response to Reviewer YrmV
Thank you for the review and the two precise questions, they were bothering us as well. We ran an experiment to test the distinction between self-continuation and partner response, and an analysis to better understand the relationship between the prediction gain over the generator-preference baseline. Whether PMus can predict which output listeners prefer among different seeds of the same generator remains open. We will report the pilot and baseline analysis in the revision and explain how a follow-up listening study would address that question.
No perceptual distinction between self-continuation and partner response. We built the targeted test (General Response, Parts 1 and 2). When the same listener answers both questions, the answers agree with PMus-Self on the continue-A question in 28 of 45, and with PMus-Partner on the listen-to-B question in 36 of 45.
Question 1: Does the adjusted association in Table 21 give a stable prediction gain on held-out data? No, and the revision says so. All six paired held-out accuracy changes in Table 21 include zero. With instrument reports, adding PMus-Partner moves held-out accuracy from 0.675 to 0.700 (change 0.025, 95% interval −0.022 to 0.075). Without them, it moves from 0.700 to 0.672. The coefficient (0.350, 95% interval 0.094 to 0.607, Holm-corrected p = 0.0224) is an adjusted association, not a prediction gain. A broader result does hold: within generator pairs, all 45 fixed-effect coefficients are positive (General Response, Part 3).
Gain over the generator-preference baseline. The within-pair models show that the context contrasts carry information beyond which generator produced a clip. A resolved accuracy gain over the generator-preference baseline is not shown, and the revision no longer claims one (General Response, Part 3).
Selection among one generator's outputs. This remains untested. A same-model, different-seed listening study is our next experiment (General Response, Part 3).
Question 2: What evidence supports the player-specific scores beyond the shared-context score? Three pieces, two of them new:
On the submitted study's judgments, the player-specific scores do not beat PMus-Duo (Table 17). The planned analysis of the new test will evaluate it.