- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the OmniVPC task and OmniVPC-Bench, a benchmark designed to evaluate whether omni-modal large language models can provide accurate, specific, and targeted vocal performance commentary like professional vocal coaches. The authors evaluate 31 omni-modal models and conduct modality ablations, comparisons with teachers’ original commentary, and genre-label perturbation experiments to investigate models’ audio-visual perception and commentary behaviors. The results show that current models achieve limited overall performance. In the modality ablation experiments, silent video improves performance on some visually relevant questions, whereas audio-only input does not yield consistent gains. Providing the teacher’s original commentary substantially improves technical and artistic evaluation scores. The authors also identify tendencies toward excessive praise, inaccurate selection of evaluation dimensions, and sensitivity to genre labels. The validation relies primarily on benchmark comparisons, modality ablations, and behavioral analyses. The reliability of the conclusions depends on the validity of the annotations and reference answers, the effectiveness of the automatic evaluation methods, and the generalizability of the modality-ablation findings across different model families.
Critical strengths
-
A Valuable Benchmark for Professional Performance Evaluation. This paper introduces OmniVPC-Bench, a benchmark designed to systematically evaluate omni-modal LLMs on professional vocal performance commentary across three dimensions: holistic commentary, technical evaluation, and artistic evaluation. It provides a useful evaluation framework for investigating model capabilities and limitations in specialized performance assessment.
-
Exploration of Model Capabilities in a Specialized Assessment Domain. Professional vocal performance commentary requires not only visual understanding of a performance but also technical and artistic analysis of singing. By extending omni-modal model evaluation to this specialized task, the paper offers a meaningful exploration of the capability boundaries of multimodal models in professional skill assessment.
Critical weaknesses
-
The paper primarily uses teachers’ original comments as references and evaluates models through LLM-as-a-Judge and closed-ended questions. However, teachers’ comments may not cover all issues in a performance, and models may be penalized for making valid observations that are not mentioned by the teachers. Therefore, it remains unclear whether the current metrics genuinely reflect models’ abilities in performance understanding, issue identification, and instructional feedback.
-
The paper finds that audio-only input does not lead to substantial performance improvements, but the corresponding ablation study covers only five Gemini models. This finding may be influenced by the model family, audio input processing, task design, or scoring protocol. Therefore, the evidence is insufficient to support the broader conclusion that current multimodal models rely primarily on visual information and fail to effectively utilize audio information.
-
The paper combines HCC, TES, and AES into OVS using weights of 0.4, 0.3, and 0.3, respectively, and assigns fixed weights to different evaluation levels. However, it remains unclear whether these weights are supported by expert validation or sensitivity analyses. It is important to determine whether alternative weighting schemes would change the model rankings or the paper’s main conclusions.
Questions for authors
-
The dataset contains 127 performance videos, with only 83 in the test set. Have the authors assessed distributional differences across singers, teaching contexts, and singing genres? Is the current dataset sufficiently representative to support the paper’s broader conclusions about multimodal models’ vocal assessment capabilities?
-
How were the weights for HCC, TES, and AES in OVS, as well as the weights assigned to the internal evaluation levels of TES and AES, determined? Have the authors conducted sensitivity analyses to verify that the model rankings and main conclusions remain stable under different weighting schemes?
-
Could the authors provide more representative end-to-end examples showing the original audiovisual inputs, teachers’ comments, and feedback generated by different models, with vocal experts assessing the accuracy and usefulness of the feedback? Such evidence would help establish whether the benchmark scores reflect actual vocal coaching quality.
-
Could the authors provide clearer qualitative comparisons that illustrate the visual cues models recognize, the auditory issues they miss, and the differences between teachers’ feedback and model outputs?
7 comments
Response to Reviewer 7xtx (Part 2/3)
W2(a): "the corresponding ablation study covers only five Gemini models ... may be influenced by the model family"
Thank you very much. The new §4.7 (RQ6) and App. C.6 add four open-weight omni backbones and the audio-only LALM Qwen2-Audio on the matched conditions, and no backbone gains significantly from audio alone.
- Setup: each clip comes in four versions that differ only in modality (
av,audio, mutedvideo, andnone), soaudio−noneis what the soundtrack alone adds andav−videowhat it adds on top of the picture. The reruns use identical clips and windows and the released code, and the re-run omni backbones match their Table 1avscores within the 95% interval. Paired lifts on the 946 non-visual TES/AES items (Table R2 = Figure 11(a, b); McNemar p in parentheses):
| Backbone | audio − none | av − video |
|---|---|---|
| Qwen2-Audio (audio-only) | −3.0 (0.05) | – |
| Qwen2.5-Omni-7B | +1.1 (0.49) | +1.6 (0.19) |
| Qwen3-Omni-Instruct | +1.3 (0.43) | +0.6 (0.69) |
| TLive-Omni-9B | −1.4 (0.35) | +0.7 (0.55) |
| MiniCPM-o-4.5 | +1.8 (0.26) | +2.8 (0.0495) |
| 5 Gemini backbones | −4.9 to −0.1 | −3.0 to +3.0 |
- ① Audio alone gives no significant gain, a dedicated audio LLM included. Pooled over all 10 backbones, the lift is −1.65 points (95% CI [−3.09, −0.27]), and on the 937 items whose evidence is auditory it is −1.07 [−2.58, 0.40] (App. C.6). The audio-only Qwen2-Audio scores 41.9 OVS with its input and 41.3 without it.
- ② Adding audio to video helps little: only MiniCPM-o (+2.8) and Gemini-3-Flash (+3.0, p = 0.03) gain significantly, 2 of the 9 backbones in Figure 11(b). The matched conditions still detect these small gains, so the benchmark can tell which models use audio.
- ③ The vision reference behaves as expected: the vision-only Qwen2.5-VL gains 8.3 points from muted video on the 264 sight items (p = 0.005), within the +4.2 to +17.1 of the Gemini backbones (App. C.6).
Thank you for asking for other model families. Therefore, "not listening" now rests on ten backbones across families and is not specific to the five Gemini models.
W2(b): "audio input processing"
Thank you very much. Audio input processing does not change the result (Figure 11(c)): all 20 open-weight backbones resample audio to 16 kHz in their own front ends, and 44.1 kHz stereo audio on 12 clips changes OVS by only −0.7 to +0.6. We appreciate this check, now part of RQ6. Hence, the audio format does not explain "not listening".
W2(c): "task design, or scoring protocol"
Thank you very much. The same items and scoring register a gain of about 40 points once useful input is supplied: accuracy rises by +39.8 and +40.0 when the coach's transcript replaces the media (p < 0.001; §4.4, App. C.3), whereas the soundtrack yields no such gain. Thank you for separating this factor. As a result, the task design and the scoring protocol do detect useful input, so they do not account for the missing audio gain.
W3: "whether these weights are supported by expert validation or sensitivity analyses"
Thank you very much. The weights were fixed by design and never fitted to any result, and, following your comment, the revised paper adds a sensitivity analysis in the new Table 2 (p. 9, red).
- Design: the level weights decrease along the cascade (L1 > L2 > L3: verdict, observation, advice), and the track weights were also fixed by design.
- Sensitivity analysis: it rescores the stored answers of all 31 backbones under each change, without re-running any model, and compares each new ranking with that of Eq. (2) by Kendall τ (Table R3):
| Change to the scoring | Kendall τ vs. Eq. (2) | Top-3 |
|---|---|---|
| Equal level weights | 0.96 | same |
| Equal track weights | 0.98 | same |
| Item-proportional tracks | 0.82 | same |
| HCC dropped | 0.79 | same |
| L1 dropped | 0.83 | same |
| ∅ class removed | 0.96 | same |
| Point-anchored HCC judge | 0.78 | same |
- Coverage: other level weights, other track weights (equal, or proportional to item counts) and dropping a whole track or level; the last two rows also remove the ∅ class and replace the HCC judge.
Thank you for requesting this analysis. Therefore, the weights are now supported by a sensitivity analysis over the alternative scorings above.
Q2: "whether alternative weighting schemes would change the model rankings"
Thank you very much. The top three do not change under any alternative to Eq. (2) that we tested. Under every change in Table R3 above, they remain Gemini-3.7-Flash, Gemini-3.8-Flash and Gemini-3.6-Flash (§4.7), and across all 16 combinations of level and track weights the best open-weight backbone (TLive-Omni-9B) is unchanged. We are grateful for this question. Hence, the model rankings and our conclusions do not depend on the chosen weights.
Response to Reviewer 7xtx (Part 3/3)
Q1: "distributional differences across singers, teaching contexts, and singing genres ... sufficiently representative"
Thank you very much. The splits are disjoint and show no significant difference on the axes you name, and the conclusions hold in both teaching contexts.
- Singers: no session, singer or piece appears in two splits (§3.3, Step 3), and genres with fewer than three sessions go to test whole (App. A.2).
- Teaching contexts: the corpus spans Chinese vocal-pedagogy classrooms (69 clips) and master classes (58) (§3.2). OVS on English-coached clips exceeds that on Chinese-coached clips by only 0.2 points on average, and English is higher for 18 of the 31 backbones, roughly half of them (App. C.1, Figure C1(c)).
- Genres: the test split matches train+dev in genre mix (χ², p = 0.94), in the L1 gold-class distribution (p = 0.37) and in sung duration (Mann–Whitney p = 0.43), and Table A2 gives per-genre and per-split counts. Figure C1(c) plots closed-item accuracy per genre, and Table R4 of our response to Reviewer swcE gives per-genre accuracies with 95% clip-bootstrap intervals; the smallest genres have very few test clips, so their values are only indicative.
Thank you for raising representativeness. As a result, the test split is representative of the corpus across singers, teaching contexts and singing genres.
Q3: "end-to-end examples ... teachers' comments, and feedback generated by different models"
Thank you very much. The revised PDF contains a full end-to-end case in Table 3 (§4.8, p. 10; protocol in App. C.8).
- Content: test clip v2_057 with its frames, the coach's words and six backbones' commentaries, verdicts and advice, each marked as agreeing with the coach or not; every quoted span is checked by script to occur verbatim in the backbone's output or in the coach's transcript.
- Reading: only Gemini-3.7-Flash rates as a strength the stage choice the coach saw and matches the coach on breath, and it also leads the six on HCC for this clip.
- Scoring anchor: every gold answer comes from the coach's own feedback; the HCC judge is self-consistent (ICC(2,1) = 0.86) and scores a commentary 30.7 points lower when the coach's judgements are inverted (Table 2).
Thank you for this suggestion; the case shows what the scores mean. Therefore, the paper sets the teacher's comments beside the feedback of different models in one end-to-end example.
Q4: "the visual cues models recognize, the auditory issues they miss, and the differences between teachers' feedback and model outputs"
Thank you very much. Three closed-item cases show the pattern you ask about: visual cues are recognized, while auditory issues are missed. Table C5 and Figure 10 contrast the dimensions coaches and backbones address, and the Q3 case sets the coach's words beside six backbones' outputs. The cases below are for Gemini-2.5-FL and 3.8-F under all four conditions.
- v2_022 (Lyrical, high register; visual cue recognized): asked whether "the high note was too tense and needed to be relaxed" is accurate, both backbones answer "no" under
noneandaudio, and "yes" (correct) undervideoandav. The coach said "这个高音……沾着嘴了" ("this high note … is stuck at the mouth"). - v2_091 (Jazz, stage presence; visual cue recognized): both choose "not a notable feature" without the picture and "a clear strength" (correct) with it. The coach said "You seem to be more comfortable in your space."
- v2_003 (Lyrical, high register; auditory issue missed): the coach observed "you struggle to reach the high notes effectively", and both backbones pick a distractor in all four conditions,
audioandavincluded.
Thank you for asking for these cases. Hence, the answer turns correct when the picture is added in the first two cases and the soundtrack never helps in the third, so the models recognize visual cues and miss auditory issues.
Summary of revisions (red in the revised PDF)
| Where | Change | Answers |
|---|---|---|
| §4.7 (new RQ6), Figure 11, Table 2 (p. 9) | Open-weight and audio-only backbones on the matched conditions; pooled audio lift; 44.1 kHz and full-span runs; ranking under alternative scorings; HCC judge checks | W1, W2(a), W2(b), W3, Q2 |
| App. C.6 (pp. 26–28) | Protocol and full results of RQ6, incl. closed-item floors and the own-transcript ceiling | W2(a), W3, Q2 |
| App. E (p. 31) | Future Work, incl. multiple coach references | W1 |
At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!
Response to Reviewer 7xtx (Part 1/3)
Thank you very much for your time and thorough review, and for recognizing OmniVPC-Bench as "a valuable benchmark for professional performance evaluation" and "a meaningful exploration of the capability boundaries of multimodal models." Your concerns are valid observations that the teachers did not mention (W1), whether the audio finding is influenced by the model family (W2(a)), by audio input processing (W2(b)) or by task design and scoring protocol (W2(c)), whether the OVS weights are supported by sensitivity analyses (W3), whether alternative weighting schemes change the rankings (Q2), distributional differences across singers, teaching contexts and genres (Q1), end-to-end examples (Q3), and the visual cues recognized versus the auditory issues missed (Q4). We answer each below.
We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.
In the revised PDF, every passage we revised in response to the reviews is marked in red, including the changes that address concerns raised by several reviewers, so the red passages are our revisions in response to you. They include the new §4.7 (RQ6) with Figure 11 and Table 2 (p. 9), its full protocol and results in the new App. C.6 (pp. 26–28), and the new App. E Future Work (p. 31). The red phrases and cross marks in Table 3 ("contradicts or is absent") and the red elements of our figures, such as markers, lines and bars, are data colours rather than revision marks.
W1: "teachers' comments may not cover all issues in a performance, and models may be penalized for making valid observations that are not mentioned by the teachers"
Thank you very much. Valid observations the coach did not mention are not penalized on the closed tracks, and the revised paper now reports the ranking without the one item class that relies on omission, under which the main findings hold.
Thank you for this point; it led us to report the ranking without the ∅ class. Therefore, a low score on the closed tracks means that a model missed or misjudged what the coach observed, and valid observations the teachers did not mention are not penalized there.