acceptodds
Under review as a conference paper at ICLR 2027

Watching, Not Listening: Benchmarking Omnimodal Vocal Performance Commentary

Abstract

Vocal coaching turns a sung performance into selective, actionable feedback, now within reach of omnimodal large language models (omni models) that watch and listen at once. Omni models have grown into unified audio-visual backbones, while AI for music has moved from understanding to singing assessment. However, vocal commentary needs audio-visual evidence, verifiable expert judgement and selectivity at once, and no benchmark measures all three. We formalize Omnimodal Vocal Performance Commentary (OmniVPC), where a model comments on a sung clip from its video and audio as an expert coach would. We release OmniVPC-Bench, 1,970 items over 127 coached clips in nine genres, with three tracks under four matched media conditions and each coach's own words as the reference. Across 31 omni models, none reaches an OVS of 55: models gain from the picture rather than the sound, and the coach's transcript in place of the media adds about 40 points.

Primary area:
interpretability and explainable AI
Keywords:
omnimodal large language models, vocal pedagogy, hypergraph retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7xtxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the OmniVPC task and OmniVPC-Bench, a benchmark designed to evaluate whether omni-modal large language models can provide accurate, specific, and targeted vocal performance commentary like professional vocal coaches. The authors evaluate 31 omni-modal models and conduct modality ablations, comparisons with teachers’ original commentary, and genre-label perturbation experiments to investigate models’ audio-visual perception and commentary behaviors. The results show that current models achieve limited overall performance. In the modality ablation experiments, silent video improves performance on some visually relevant questions, whereas audio-only input does not yield consistent gains. Providing the teacher’s original commentary substantially improves technical and artistic evaluation scores. The authors also identify tendencies toward excessive praise, inaccurate selection of evaluation dimensions, and sensitivity to genre labels. The validation relies primarily on benchmark comparisons, modality ablations, and behavioral analyses. The reliability of the conclusions depends on the validity of the annotations and reference answers, the effectiveness of the automatic evaluation methods, and the generalizability of the modality-ablation findings across different model families.

Critical strengths
  1. A Valuable Benchmark for Professional Performance Evaluation. This paper introduces OmniVPC-Bench, a benchmark designed to systematically evaluate omni-modal LLMs on professional vocal performance commentary across three dimensions: holistic commentary, technical evaluation, and artistic evaluation. It provides a useful evaluation framework for investigating model capabilities and limitations in specialized performance assessment.

  2. Exploration of Model Capabilities in a Specialized Assessment Domain. Professional vocal performance commentary requires not only visual understanding of a performance but also technical and artistic analysis of singing. By extending omni-modal model evaluation to this specialized task, the paper offers a meaningful exploration of the capability boundaries of multimodal models in professional skill assessment.

Critical weaknesses
  1. The paper primarily uses teachers’ original comments as references and evaluates models through LLM-as-a-Judge and closed-ended questions. However, teachers’ comments may not cover all issues in a performance, and models may be penalized for making valid observations that are not mentioned by the teachers. Therefore, it remains unclear whether the current metrics genuinely reflect models’ abilities in performance understanding, issue identification, and instructional feedback.

  2. The paper finds that audio-only input does not lead to substantial performance improvements, but the corresponding ablation study covers only five Gemini models. This finding may be influenced by the model family, audio input processing, task design, or scoring protocol. Therefore, the evidence is insufficient to support the broader conclusion that current multimodal models rely primarily on visual information and fail to effectively utilize audio information.

  3. The paper combines HCC, TES, and AES into OVS using weights of 0.4, 0.3, and 0.3, respectively, and assigns fixed weights to different evaluation levels. However, it remains unclear whether these weights are supported by expert validation or sensitivity analyses. It is important to determine whether alternative weighting schemes would change the model rankings or the paper’s main conclusions.

Questions for authors
  1. The dataset contains 127 performance videos, with only 83 in the test set. Have the authors assessed distributional differences across singers, teaching contexts, and singing genres? Is the current dataset sufficiently representative to support the paper’s broader conclusions about multimodal models’ vocal assessment capabilities?

  2. How were the weights for HCC, TES, and AES in OVS, as well as the weights assigned to the internal evaluation levels of TES and AES, determined? Have the authors conducted sensitivity analyses to verify that the model rankings and main conclusions remain stable under different weighting schemes?

  3. Could the authors provide more representative end-to-end examples showing the original audiovisual inputs, teachers’ comments, and feedback generated by different models, with vocal experts assessing the accuracy and usefulness of the feedback? Such evidence would help establish whether the benchmark scores reflect actual vocal coaching quality.

  4. Could the authors provide clearer qualitative comparisons that illustrate the visual cues models recognize, the auditory issues they miss, and the differences between teachers’ feedback and model outputs?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 7xtx (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing OmniVPC-Bench as "a valuable benchmark for professional performance evaluation" and "a meaningful exploration of the capability boundaries of multimodal models." Your concerns are valid observations that the teachers did not mention (W1), whether the audio finding is influenced by the model family (W2(a)), by audio input processing (W2(b)) or by task design and scoring protocol (W2(c)), whether the OVS weights are supported by sensitivity analyses (W3), whether alternative weighting schemes change the rankings (Q2), distributional differences across singers, teaching contexts and genres (Q1), end-to-end examples (Q3), and the visual cues recognized versus the auditory issues missed (Q4). We answer each below.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, every passage we revised in response to the reviews is marked in red, including the changes that address concerns raised by several reviewers, so the red passages are our revisions in response to you. They include the new §4.7 (RQ6) with Figure 11 and Table 2 (p. 9), its full protocol and results in the new App. C.6 (pp. 26–28), and the new App. E Future Work (p. 31). The red phrases and cross marks in Table 3 ("contradicts or is absent") and the red elements of our figures, such as markers, lines and bars, are data colours rather than revision marks.


W1: "teachers' comments may not cover all issues in a performance, and models may be penalized for making valid observations that are not mentioned by the teachers"

Thank you very much. Valid observations the coach did not mention are not penalized on the closed tracks, and the revised paper now reports the ranking without the one item class that relies on omission, under which the main findings hold.

  • Only one class relies on omission. Every TES/AES item asks about a point the coach made, except the L1 "not addressed" class (39.8% of the 410 L1 test items, §4.5). By §3.1 a model is correct "insofar as its remarks agree with the coach's", and §3.1 makes selectivity, i.e., which aspects a commentary leaves out, part of the answer, so this class asks whether a model can tell which dimensions an expert would single out.
  • The main findings hold without that class (Table R1):
Evidence (test split)Result
L2 yes/no on the coach's own observations (Table C4, Fig. 9(d))d′d' = −0.40 to 0.42, all 31 backbones
Coach's own transcript instead of media (Table C8), TES/AES27.8/36.5 → 71.7/74.3 (2.5-FL); 37.4/43.7 → 81.9/80.3 (3.8-F)
∅ class removed (new Table 2)Kendall τ = 0.96, same top-3
  • Reading: ① the backbones do not reliably verify observations the coach did make; ② with the coach's own transcript in place of the media, TES/AES accuracy rises sharply, so the items are answerable from what the coach said; ③ with the ∅ class removed, the ranking barely moves.
  • HCC rewards performance-specific correctness (new Table 2). The judge scores the coach's own transcript rewritten as a commentary and then the same text with every judgement inverted, and the score falls by 30.7 points (82/83 clips, Wilcoxon p < 1e-14); using another same-genre clip's content lowers it by 12.3 (72/83, p < 1e-11).
  • The three abilities you name are each scored against what the coach said. Performance understanding is the holistic commentary (HCC). Issue identification is the coach's verdict on each dimension (L1) and the specific observations the coach made (L2), and instructional feedback is the coach's advice (TES L3).
  • Several coaches per performance is direction (i) of the new App. E: the responses of further coaches to the same performances would let a commentary be graded against several experts at once and show where coaches agree.

Thank you for this point; it led us to report the ranking without the ∅ class. Therefore, a low score on the closed tracks means that a model missed or misjudged what the coach observed, and valid observations the teachers did not mention are not penalized there.


AuthorsOpenReview ↗

Response to Reviewer 7xtx (Part 2/3)


W2(a): "the corresponding ablation study covers only five Gemini models ... may be influenced by the model family"

Thank you very much. The new §4.7 (RQ6) and App. C.6 add four open-weight omni backbones and the audio-only LALM Qwen2-Audio on the matched conditions, and no backbone gains significantly from audio alone.

  • Setup: each clip comes in four versions that differ only in modality (av, audio, muted video, and none), so audio − none is what the soundtrack alone adds and av − video what it adds on top of the picture. The reruns use identical clips and windows and the released code, and the re-run omni backbones match their Table 1 av scores within the 95% interval. Paired lifts on the 946 non-visual TES/AES items (Table R2 = Figure 11(a, b); McNemar p in parentheses):
Backboneaudio − noneav − video
Qwen2-Audio (audio-only)−3.0 (0.05)–
Qwen2.5-Omni-7B+1.1 (0.49)+1.6 (0.19)
Qwen3-Omni-Instruct+1.3 (0.43)+0.6 (0.69)
TLive-Omni-9B−1.4 (0.35)+0.7 (0.55)
MiniCPM-o-4.5+1.8 (0.26)+2.8 (0.0495)
5 Gemini backbones−4.9 to −0.1−3.0 to +3.0
  • ① Audio alone gives no significant gain, a dedicated audio LLM included. Pooled over all 10 backbones, the lift is −1.65 points (95% CI [−3.09, −0.27]), and on the 937 items whose evidence is auditory it is −1.07 [−2.58, 0.40] (App. C.6). The audio-only Qwen2-Audio scores 41.9 OVS with its input and 41.3 without it.
  • ② Adding audio to video helps little: only MiniCPM-o (+2.8) and Gemini-3-Flash (+3.0, p = 0.03) gain significantly, 2 of the 9 backbones in Figure 11(b). The matched conditions still detect these small gains, so the benchmark can tell which models use audio.
  • ③ The vision reference behaves as expected: the vision-only Qwen2.5-VL gains 8.3 points from muted video on the 264 sight items (p = 0.005), within the +4.2 to +17.1 of the Gemini backbones (App. C.6).

Thank you for asking for other model families. Therefore, "not listening" now rests on ten backbones across families and is not specific to the five Gemini models.


W2(b): "audio input processing"

Thank you very much. Audio input processing does not change the result (Figure 11(c)): all 20 open-weight backbones resample audio to 16 kHz in their own front ends, and 44.1 kHz stereo audio on 12 clips changes OVS by only −0.7 to +0.6. We appreciate this check, now part of RQ6. Hence, the audio format does not explain "not listening".


W2(c): "task design, or scoring protocol"

Thank you very much. The same items and scoring register a gain of about 40 points once useful input is supplied: accuracy rises by +39.8 and +40.0 when the coach's transcript replaces the media (p < 0.001; §4.4, App. C.3), whereas the soundtrack yields no such gain. Thank you for separating this factor. As a result, the task design and the scoring protocol do detect useful input, so they do not account for the missing audio gain.


W3: "whether these weights are supported by expert validation or sensitivity analyses"

Thank you very much. The weights were fixed by design and never fitted to any result, and, following your comment, the revised paper adds a sensitivity analysis in the new Table 2 (p. 9, red).

  • Design: the level weights decrease along the cascade (L1 > L2 > L3: verdict, observation, advice), and the track weights were also fixed by design.
  • Sensitivity analysis: it rescores the stored answers of all 31 backbones under each change, without re-running any model, and compares each new ranking with that of Eq. (2) by Kendall τ (Table R3):
Change to the scoringKendall τ vs. Eq. (2)Top-3
Equal level weights0.96same
Equal track weights0.98same
Item-proportional tracks0.82same
HCC dropped0.79same
L1 dropped0.83same
∅ class removed0.96same
Point-anchored HCC judge0.78same
  • Coverage: other level weights, other track weights (equal, or proportional to item counts) and dropping a whole track or level; the last two rows also remove the ∅ class and replace the HCC judge.

Thank you for requesting this analysis. Therefore, the weights are now supported by a sensitivity analysis over the alternative scorings above.


Q2: "whether alternative weighting schemes would change the model rankings"

Thank you very much. The top three do not change under any alternative to Eq. (2) that we tested. Under every change in Table R3 above, they remain Gemini-3.7-Flash, Gemini-3.8-Flash and Gemini-3.6-Flash (§4.7), and across all 16 combinations of level and track weights the best open-weight backbone (TLive-Omni-9B) is unchanged. We are grateful for this question. Hence, the model rankings and our conclusions do not depend on the chosen weights.


AuthorsOpenReview ↗

Response to Reviewer 7xtx (Part 3/3)


Q1: "distributional differences across singers, teaching contexts, and singing genres ... sufficiently representative"

Thank you very much. The splits are disjoint and show no significant difference on the axes you name, and the conclusions hold in both teaching contexts.

  • Singers: no session, singer or piece appears in two splits (§3.3, Step 3), and genres with fewer than three sessions go to test whole (App. A.2).
  • Teaching contexts: the corpus spans Chinese vocal-pedagogy classrooms (69 clips) and master classes (58) (§3.2). OVS on English-coached clips exceeds that on Chinese-coached clips by only 0.2 points on average, and English is higher for 18 of the 31 backbones, roughly half of them (App. C.1, Figure C1(c)).
  • Genres: the test split matches train+dev in genre mix (χ², p = 0.94), in the L1 gold-class distribution (p = 0.37) and in sung duration (Mann–Whitney p = 0.43), and Table A2 gives per-genre and per-split counts. Figure C1(c) plots closed-item accuracy per genre, and Table R4 of our response to Reviewer swcE gives per-genre accuracies with 95% clip-bootstrap intervals; the smallest genres have very few test clips, so their values are only indicative.

Thank you for raising representativeness. As a result, the test split is representative of the corpus across singers, teaching contexts and singing genres.


Q3: "end-to-end examples ... teachers' comments, and feedback generated by different models"

Thank you very much. The revised PDF contains a full end-to-end case in Table 3 (§4.8, p. 10; protocol in App. C.8).

  • Content: test clip v2_057 with its frames, the coach's words and six backbones' commentaries, verdicts and advice, each marked as agreeing with the coach or not; every quoted span is checked by script to occur verbatim in the backbone's output or in the coach's transcript.
  • Reading: only Gemini-3.7-Flash rates as a strength the stage choice the coach saw and matches the coach on breath, and it also leads the six on HCC for this clip.
  • Scoring anchor: every gold answer comes from the coach's own feedback; the HCC judge is self-consistent (ICC(2,1) = 0.86) and scores a commentary 30.7 points lower when the coach's judgements are inverted (Table 2).

Thank you for this suggestion; the case shows what the scores mean. Therefore, the paper sets the teacher's comments beside the feedback of different models in one end-to-end example.


Q4: "the visual cues models recognize, the auditory issues they miss, and the differences between teachers' feedback and model outputs"

Thank you very much. Three closed-item cases show the pattern you ask about: visual cues are recognized, while auditory issues are missed. Table C5 and Figure 10 contrast the dimensions coaches and backbones address, and the Q3 case sets the coach's words beside six backbones' outputs. The cases below are for Gemini-2.5-FL and 3.8-F under all four conditions.

  • v2_022 (Lyrical, high register; visual cue recognized): asked whether "the high note was too tense and needed to be relaxed" is accurate, both backbones answer "no" under none and audio, and "yes" (correct) under video and av. The coach said "这个高音……沾着嘴了" ("this high note … is stuck at the mouth").
  • v2_091 (Jazz, stage presence; visual cue recognized): both choose "not a notable feature" without the picture and "a clear strength" (correct) with it. The coach said "You seem to be more comfortable in your space."
  • v2_003 (Lyrical, high register; auditory issue missed): the coach observed "you struggle to reach the high notes effectively", and both backbones pick a distractor in all four conditions, audio and av included.

Thank you for asking for these cases. Hence, the answer turns correct when the picture is added in the first two cases and the soundtrack never helps in the third, so the models recognize visual cues and miss auditory issues.


Summary of revisions (red in the revised PDF)

WhereChangeAnswers
§4.7 (new RQ6), Figure 11, Table 2 (p. 9)Open-weight and audio-only backbones on the matched conditions; pooled audio lift; 44.1 kHz and full-span runs; ranking under alternative scorings; HCC judge checksW1, W2(a), W2(b), W3, Q2
App. C.6 (pp. 26–28)Protocol and full results of RQ6, incl. closed-item floors and the own-transcript ceilingW2(a), W3, Q2
App. E (p. 31)Future Work, incl. multiple coach referencesW1

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Reviewer_qxysOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OmniVPC-Bench, a benchmark for evaluating omnimodal model's ability to provide expert feedback on vocal performances. The benchmark contains 1,970 items derived from 127 performances across nine genres using the original coaches' spoken feedback as references. Evaluations of 31 models and input ablations reveal limited reference agreement and greater benefits from visual than auditory input.

Critical strengths

The benchmark uses coaches' responses from real vocal lessons and master classes which are valuable real-world, human-generated resources.

Critical weaknesses
  1. Sections 3.1-3.3 define correctness through agreement with the original coach and an unmentioned dimension becomes a gold "not addressed" label, and predicting this omission is treated as part of expert selectivity. However, a coach's response is a selective intervention and is not an exhaustive annotation of a performance. An omitted dimension may be acceptable, problematic but lower priority, already discussed, or outside that lesson's focus. This therefore creates a distinction between two valid but different evaluation targets: (1) predicting what this particular coach happened to discuss, and (2) producing accurate and useful coaching from the available performance. The benchmark explicitly chooses the former but several conclusions concern the latter. For example, near chance prediction of "not addressed" does not by itself show that a model cannot identify musically important issues. Likewise, a valid observation absent from the reference is not necessarily a hallucination.

  2. Reference feedback may depend on evidence removed by the media sampling procedure. Appendix A.2 reports original performance spans of 10-545 seconds with a median of 188 seconds, but models receive three concatenated 20 second windows or the full span when sufficiently short. HCC nevertheless uses the full coach transcript as its reference, and other questions are derived from that response. Without establishing that the relevant evidence remains observable in the sampled excerpts, low scores may partly reflect missing information, complicating their interpretation as evidence of perceptual limitations.

Questions for authors
  1. How does the evaluation distinguish valid feedback absent from the reference from unsupported or incorrect feedback? Is there evidence that independent experts can reliably infer the original coach’s selected dimensions from the supplied performance?

  2. Were items checked for answerability from the exact sampled excerpts provided to models? How were observations referring to omitted passages handled, particularly for HCC and temporally anchored questions?

  3. In the judge-sensitivity control (Section 3.4), a commentary written from the coach's own annotated points scores 69.0 on HCC which is 1.7 points above one written from the genre alone (67.3). Has the HCC judge been validated against expert ratings? How sensitive is it to performance-specific correctness relative to factors such as fluency or response length?

8 comments
AuthorsOpenReview ↗

Response to Reviewer qxys (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing that the benchmark is built on "coaches' responses from real vocal lessons and master classes," which are "valuable real-world, human-generated resources." We answer W1–W2 and Q1–Q3 below, starting in this part with W1.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, every passage we revised in response to the reviews is marked in red, including the changes that address concerns raised by several reviewers, so the red passages are our revisions in response to you. They include a new clarifying sentence in §3.2, the new §4.7 (RQ6) with Figure 11 and Table 2 (p. 9), its full protocol and results in the new App. C.6 (pp. 26–28), and the new App. E Limitations and Future Work (p. 31). The red phrases and cross marks in Table 3 ("contradicts or is absent") and the red elements of our figures, such as markers, lines and bars, are data colours rather than revision marks.


W1(a): "a coach's response is a selective intervention ... (1) predicting what this particular coach happened to discuss, and (2) producing accurate and useful coaching"

Thank you very much. Yes, "not addressed" is a core and intended property, but it measures one thing only: salience (which dimensions this coach chooses to address). OmniVPC-Bench measures (1). §3.1 defines a model as correct "insofar as its remarks agree with the coach's."

  • Where it is used. "Not addressed" is a gold answer only for L1 verdict items: 163 of the 410 L1 test items (39.8%), i.e., 163 of all 1,293 test items. The 69.0% figure counts annotation cells before class capping, not test items. We have added one clarifying sentence in §3.2:
What is countedFigure
Annotation cells the coach leaves unmentioned, before class capping (§3.2, Table A3)69.0% of cells
L1 items whose gold answer is "not addressed" (new sentence in §3.2; §4.5)163 of the 410 L1 test items (39.8%); 163 of all 1,293
  • How it is reported. §4.5 reports it separately as salience. It is not used as evidence that models miss issues.
  • What everything else measures. The other 1,130 items, including all of L2, L3 and HCC, are graded against points the coach stated. A yes/no "no" is not an omission: it is an opposite-polarity observation on the same dimension from another clip, so both commentaries back it (App. A.2).
  • The conclusions do not rest on the ∅ class (Table R1):
Evidence (test split)ResultUses omission?
L2 yes/no on the coach's own observations (Table C4)d′d' = −0.40 to 0.42, all 31 backbonesno
Coach's own transcript instead of media (Table C8), TES/AES27.8/36.5 → 71.7/74.3 (2.5-FL); 37.4/43.7 → 81.9/80.3 (3.8-F)no
OVS with the ∅ class removed (new Table 2)Kendall τ = 0.96, same top-3removed
Closed tracks only (TES/AES), ∅ removedτ = 0.80, same top-3removed
  • The first two rows never use omission. Backbones fail to verify observations the coach did make, yet the same items become answerable once the coach's own transcript replaces the media.
  • The last two rows remove the ∅ class, and the ranking keeps its top three. The last row also leaves out HCC.
  • With another same-genre coach's transcript instead, backbones score like no media (§4.4, App. C.3). Genre knowledge does not answer these items.

Thank you for this distinction. Therefore, the benchmark measures agreement with this coach, and its low scores reflect a gap in perceiving the performance on points the coach did make.


W1(b): "near chance prediction of 'not addressed' does not by itself show..."

Thank you very much. The paper reads this result the same way. As noted in W1(a), §4.5 reports near-chance prediction of "not addressed" only as salience. We appreciate this careful reading. Hence, this result is not used as evidence that issues go unnoticed.


W1(c): "a valid observation absent from the reference is not necessarily a hallucination"

Thank you very much. Closed items never penalize an unreferenced remark. The paper does not use the term "hallucination" for such a remark. Where a reference exists, we annotated all 31 backbones' commentaries with the same Step-4 annotator used for the coach transcripts. On dimensions the coach raised, their polarity agrees with the coach 54.3% of the time, vs. 54.4% expected from each backbone's own per-dimension prior. A backbone therefore agrees with the coach no more often than its own habits predict. Thank you for this example. As a result, a valid observation absent from the reference is never counted as an error, and the measured gap concerns points the coach did raise.


AuthorsOpenReview ↗

Response to Reviewer qxys (Part 2/3)


Q1(a): "How does the evaluation distinguish valid feedback absent from the reference from unsupported or incorrect feedback?"

Thank you very much. A remark outside the reference is never graded on the closed tracks. The judges keep incorrect feedback apart from feedback that is absent.

  • Closed tracks (TES/AES). Every item asks about a point the coach made, or the ∅ class at L1. A remark outside the reference is neither rewarded nor penalized there. A yes/no negative comes from a same-genre clip of opposite polarity on the same dimension (§3.3, Step 5).
  • HCC. It compares the commentary with the reference. Q3(b) below shows that it penalizes wrong content.
  • Point-anchored judge (Table 2). It labels each coach point as matched, contradicted, or absent. Incorrect feedback (contradicted) is kept apart from feedback that simply does not cover a point (absent). This judge keeps the OVS ranking (Kendall τ = 0.78, same top-3).

Thank you for this question. Hence, valid feedback absent from the reference is left ungraded, while unsupported or incorrect feedback is what loses points.


Q1(b): "Is there evidence that independent experts can reliably infer the original coach's selected dimensions?"

Thank you very much. We treat the coach's selection as one measured property (salience), reported separately. Experts also checked the reference itself.

  • Reference. Our gold answers come from renowned vocal masters and established musicians, in live, on-site commentary. Three experts spot-checked a stratified sample for transcript fidelity, genre label, and item–evidence agreement (§3.3, Step 7). Closed items are scored exactly against the coach's stated points. The HCC judge is self-consistent (ICC(2,1) = 0.86) and sensitive to whether a commentary matches this coach (Table 2).
  • Salience. L1 "not addressed" items measure salience relative to this coach. §4.5 reports it separately from the other findings (48.5–53.2 for all 31 backbones, Figure 9(b)).
  • Several coaches. Grading against several coaches per performance is direction (i) of the new App. E (p. 31).

Thank you for raising this. Therefore, the coach's selected dimensions enter as a separately reported salience measure, and agreement among experts on them is direction (i) of our Future Work.


W2: "Reference feedback may depend on evidence removed by the media sampling"

Thank you very much. We tested directly whether the sampled excerpts limit the scores, and they do not (new §4.7 and App. C.6, pp. 26–28).

  • ① Coverage does not predict accuracy. Coverage is the share of the sung span inside the three windows. Its median is 0.337, and 11/83 clips are complete. The mean per-backbone Spearman ρ with per-clip closed-item accuracy is −0.085 over 31 backbones. The media-minus-none gain does not grow with coverage either: all 15 ρ (5 Gemini × av/audio/video) are negative, and none is significant.
  • ② The full sung span does not improve OVS (Figure 11(c)). Of the 72 test clips whose sung span exceeds 60 s, we drew 12 (3 per coverage quartile, seed 0 before any call; 162 items). We ran them over the whole span (95–454 s) and over the three windows, in the same session (Table R2):
BackboneMediaHCCTESAESOVS
Gemini-2.5-FL3 × 20 s70.035.739.750.6
full span70.025.639.247.5
Gemini-3.8-F3 × 20 s78.931.443.254.0
full span74.833.940.852.3
  • ③ The audio format does not matter. On the same 12 clips, 44.1 kHz stereo audio cut from the original recordings at the released windows changes OVS by only −0.7 to +0.6 (Figure 11(c)).

As App. C.6 concludes, neither the three windows nor the audio format limits the scores. Thank you for pressing on this point. Therefore, the low scores do not come from evidence removed by the media sampling.


Q2: "Were items checked for answerability from the exact sampled excerpts?"

Thank you very much. We checked this at the level of observations. 86% of test observations (422/491) name no time region (App. C.6). Region-naming items are matched to the windows by design.

  • Temporally anchored items. The three windows are centred in the thirds of the sung span (§3.3, Step 6; App. A.2). The opening, middle, and closing thirds each contribute a window.
  • HCC. HCC uses the full transcript as its reference. When the whole sung span replaces the three windows of av, HCC goes from 70.0 to 70.0 for Gemini-2.5-FL and from 78.9 to 74.8 for Gemini-3.8-F (Table R2; protocol in App. C.6). It does not rise with the full span either.

We are grateful for this check. Hence, most items do not depend on which passage was sampled, and the rest remain answerable from the sampled excerpts.


AuthorsOpenReview ↗

Response to Reviewer qxys (Part 3/3)


Q3(a): "Has the HCC judge been validated against expert ratings?"

Thank you very much. Every HCC score is anchored to an expert's judgement of that performance, because each commentary is scored against the coach's own feedback. The judge is also self-consistent, and the ranking does not hinge on it.

  • Self-consistency. Re-judging the 249 commentaries of Gemini-3.8-Flash, TLive-Omni-9B and Qwen2.5-Omni-7B twice gives ICC(2,1) = 0.86. Model means change by at most 0.7, so the scores are reproducible.
  • Independent judge. Replacing HCC with a point-anchored judge, which labels each coach point as matched, contradicted or absent with a verbatim quote, keeps the OVS ranking (Kendall τ = 0.78, same top-3). Dropping HCC gives τ = 0.79, same top-3 (Table 2; protocol in App. C.6, p. 28). This judge checks the coach's points one by one and still orders the backbones similarly.

Thank you for asking about validation; it led to the judge checks in the revised paper. Therefore, HCC is tied to the expert reference of each performance, is reproducible, and the ranking does not depend on this judge.


Q3(b): "How sensitive is it to performance-specific correctness relative to factors such as fluency or response length?"

Thank you very much. The judge rewards correct, performance-specific content. Neither fluency nor length earns credit. We now test this on all 83 test clips, judging each text against its own clip's c∗c^* (Table R3; rows 1, 2 and 4 are in the new Table 2):

Contrast (first vs. second)What changesMean diffFirst higherWilcoxon p
c∗c^* as a commentary vs. same, every judgement invertedcorrectness+30.782/83<1e-14
this clip's c∗c^* vs. a same-genre clip's c∗c^*which performance+12.372/83<1e-11
c∗c^* vs. fluent, genre-only, content-free commentarycontent vs. fluency+3.758/83<1e-4
Gemini-3.8-Flash's commentary vs. it plus generic filleradded length+8.477/83<1e-13
  • Design. Each pair differs in one property. In the first pair, the coach's own transcript is rewritten as a commentary and then every judgement in it is inverted. In the second, the commentary comes from another clip of the same genre. In the fourth, generic filler is appended to an otherwise unchanged commentary.
  • ① Correctness. Inverting the coach's judgements, or substituting another clip's content of the same genre, costs 12.3–30.7 points, although both texts in each pair read like an expert coach.
  • ② Fluency and length. Coach content scores above fluent content-free text written from the genre alone. Appending fluent filler lowers a commentary's score, even though the padded text is longer and just as fluent.
  • ③ Performance-specific content. HCC also falls when only the genre line of the prompt is swapped and the commentary follows the wrong label (76.3 to 73.2 and 69.1 to 67.5; §4.4).
  • §3.4 control. It compares only commentaries that gpt-4o-mini writes. The gap you point to there, between the commentary written from the coach's points and the one written from the genre alone, is small. Its clearest signal is across references: the same commentary, written from the coach's points, scores 69.0 against its own coach's c∗c^* but 61.9 against another same-genre coach's.

Thank you for pointing to this gap; it prompted the direct tests above. Hence, the judge is sensitive to performance-specific correctness, while fluency and response length earn no credit.


Summary of revisions (red in the revised PDF)

ItemLocationRevisionStatusColor
Units of the two "not addressed" figures§3.2 (Dimensions)One sentence: 69.0% of annotation cells; after class capping, gold answer only for L1 (163 of the 410 L1 test items, 39.8%; 163 of all 1,293); L2, L3 and HCC take theirs from points the coach statedAddedred
Full-span and 44.1 kHz runs on 12 truncated test clips§4.7 (new RQ6), Figure 11(c) (p. 9)Direct tests that the sampled windows and the audio format do not limit the scoresIn the revised PDFred
Ranking without the ∅ class, without HCC, and with a point-anchored judge; HCC judge checksTable 2 (p. 9)OVS ranking unchanged (Kendall τ = 0.96 without ∅; τ = 0.78 with the point-anchored judge), same top-3In the revised PDFred
RQ6 protocol and results; Limitations and Future Work (multiple coach references; time-aligned commentary)App. C.6 (pp. 26–28); App. E (p. 31)New appendix material answering W2, Q2, Q3(a) and Q1(b)In the revised PDFred

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved the paper accordingly. Thank you very much for your support of our work.

Reviewer_swcEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Omnimodal Vocal Performance Commentary , a task that requires models to generate selective, expert-level vocal coaching feedback from singing videos, audio, and genre labels. It presents OmniVPC-Bench, a benchmark comprising 1,970 annotated items from 127 real vocal classroom and master-class clips across nine genres, with session-, singer-, and piece-disjoint splits. The benchmark supports three evaluation tracks: holistic commentary generation (HCC), technique evaluation (TES), and artistry evaluation (AES), which are combined into an overall score (OVS). Experiments across 31 omni-modal models show substantial limitations in current systems, with the best OVS reaching only 54.9. Further analyses reveal that audio input alone provides little benefit, whereas visual information substantially improves judgments on visually grounded items. Models also exhibit strong dependence on expert-provided commentary, sensitivity to genre prompts, systematic over-praising, generic and repetitive feedback, and a mismatch between their advice and coaches' actual priorities. These findings highlight the substantial gap between current omni-modal models and expert-level, evidence-grounded, and selective vocal performance assessment.

Critical strengths
  1. he paper formalizes OmniVPC, a task that requires models to provide expert-like commentary on singing performances based on video and audio inputs. It introduces OmniVPC-Bench, which supports three evaluation tracks, ranging from holistic commentary to a structured verdict–observation–advice cascade, with “not addressed” explicitly treated as a valid gold answer.
  2. The benchmark adopts four matched media conditions (audio-visual, audio-only, muted video, and no media), using identical clips and frames to isolate the contributions of visual and audio information. It also employs session-, singer-, and piecedisjoint data splits and a reproducible, scripted construction pipeline that avoids using outputs from evaluated models as annotations. Additional quality controls include removing items solvable without media, class capping, and expert spotchecks.
  3. Extensive evaluations across 31 models reveal substantial limitations, with the best OVS reaching only 54.9. Models benefit considerably from visual input (+4.2 to +17.1 points), but not from audio alone. Replacing media with the original coach's transcript yields an approximately 40-point improvement, whereas using another coach's transcript provides little benefit. Models are also highly sensitive to genre prompts, with 22–30% of verdicts changing when only the genre label is altered, and exhibit systematic over-praising and a strong bias toward breath-related advice.
Critical weaknesses
  1. The paper's headline claim that no model achieves an OVS of 55 is difficult to interpret because neither a chance-level baseline nor a human-expert ceiling is reported. Although the chance level for L1 is specified as 33.3%, the expected chance performance for TES and AES remains unclear, given their mixtures of binary, four-option, and judge-based evaluations.
  2. A central finding of the paper is that models benefit from visual input but not from audio input. However, the released audio is encoded as 16 kHz mono AAC, while many of the benchmark's audio-related dimensions, such as resonance, vibrato, timbre, and register transitions, depend on fine-grained spectral information. The limited audio bandwidth and mono representation may therefore impair models' ability to identify these characteristics.
  3. Each benchmark clip contains only three 20-second windows, covering at most 60 seconds, whereas the median sung duration is 188 seconds. Consequently, models may be evaluated on only a fraction of the performance, while the coach's reference commentary may describe events occurring outside the provided windows. This mismatch could partly explain why models fail to reproduce the coach's observations.
Questions for authors
  1. Please report (i) an empirical chance/random baseline for L2, L3, TES, AES and OVS (constant answers, shuffled options, permuted gold labels), and (ii) a human ceiling: vocal teachers or coaches who did not see "c" answering the same items from the same media. Without these, "no backbone reaches OVS 55" is not interpretable.
  2. Each clip has one coach's response and correctness is agreement with that coach. Please report inter-coach agreement: for a subset of clips, have a second coach annotate or answer the items, and report agreement on polarity, on which dimensions are addressed, and on L1.
  3. Three of the nine genres have ≤3 test clips (R&B 2, Jazz 3, Coloratura 1) and fewer than three sessions put genres entirely in test, so "nine genres" is thin evidence; please report per-genre results with intervals or qualify the claim. Also, 11 of 31 backbones are Gemini and no audio-only LALM is included — please add at least audio-only (e.g., Qwen-Audio / AudioFlamingo) and video-only references on the matched conditions, which would sharpen the "not listening" conclusion.
  4. Given the 69.0 (oracle content) vs. 67.3 (genre-only) control, please report the correlation between HCC scores and ratings from ≥2 human experts on a sample of commentaries, the judge's self-consistency, and a swap test (does the judge prefer c* to a fluent but content-free commentary?). Please also report OVS under alternative weightings (equal tracks, item-count-proportional, HCC dropped) to show the ranking is not an artifact of Eq. (2).
3 comments
AuthorsOpenReview ↗

Response to Reviewer swcE (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing the paper's formalization with "'not addressed' explicitly treated as a valid gold answer," its "four matched media conditions," its "session-, singer-, and piece-disjoint data splits," and its "extensive evaluations across 31 models." Your concerns are the chance-level baseline (W1(a) & Q1(i)), the human-expert ceiling (W1(b)), the 16 kHz mono audio (W2), the three 20-second windows (W3), a human ceiling from vocal teachers (Q1(ii)), inter-coach agreement (Q2), per-genre results for the smallest genres (Q3(a)), audio-only and video-only references (Q3(b)), the correlation of HCC with human experts (Q4(a)), self-consistency and a swap test (Q4(b)), and OVS under alternative weightings (Q4(c)). We answer each below.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, every passage we revised in response to the reviews is marked in red, including the changes that address concerns raised by several reviewers, so the red passages are our revisions in response to you. They include the new §4.7 (RQ6) with Figure 11 and Table 2 (p. 9), its full protocol and results in the new App. C.6 (pp. 26–28), and the new App. E Future Work (p. 31). The red phrases and cross marks in Table 3 ("contradicts or is absent") and the red elements of our figures, such as markers, lines and bars, are data colours rather than revision marks.


W1(a) & Q1(i): "no model achieves an OVS of 55 is difficult to interpret ... neither a chance-level baseline"

Thank you very much. The revised PDF now reports exact chance floors for the closed items (App. C.6, p. 28), so "no backbone reaches 55" has a lower reference point.

  • Floors (Table R1): "constant answer" always picks the same option, "shuffled options" equals uniform random answering, and "permuted gold" shuffles the gold answers within each item type (B = 200):
Closed itemsConstant answerShuffled optionsPermuted gold
L1 polarity (balanced accuracy)33.325.0–
L2 yes/no (balanced by construction)50.050.0–
L2/L3 four-option descriptions18.3–29.025.0–
TES / AES closed-item floor–19.5 / 26.820.1 / 28.1
  • Reading: the exact floors are TES 19.5 and AES 26.8, and on L1 any constant answer scores 33.3 by construction (balanced accuracy, §3.4 and Eq. (3)).

Thank you for asking for this baseline. Therefore, every score can now be read against an exact chance-level floor.


W1(b): "nor a human-expert ceiling"

Thank you very much. The top of the scale is now anchored by an own-transcript ceiling on the same backbones and items (App. C.6, p. 28).

  • Why this control: HCC and the judged items are free text, for which a random answer is not well defined, so the matched controls give the upper reference:
Gemini-2.5-FL / 3.8-FTESAES
av (Table 1)27.8 / 37.436.5 / 43.7
coach's own transcript, no media (Table C8)71.7 / 81.974.3 / 80.3
  • Reading: with the information the coach had, backbones reach 71.7–81.9 on TES/AES, versus 27.8–43.7 from the media, so the headroom is large and the scale is not capped near 55.

Thank you for this request. Hence, the floors and this ceiling now bound the scale on which every backbone is placed.


W2: "the released audio is encoded as 16 kHz mono AAC ... resonance, vibrato, timbre, and register transitions depend on fine-grained spectral information"

Thank you very much. The 16 kHz mono release does not hide this spectral detail from the models: giving two Gemini backbones 44.1 kHz stereo audio leaves OVS essentially unchanged (new §4.7, Figure 11(c)).

  • Open-weight backbones already work at 16 kHz: all 20 of them resample audio to 16 kHz in their own front ends (read from each checkpoint's processor code or configuration), so higher-rate input would not reach them.
  • 44.1 kHz stereo test: for 12 test clips we cut 44.1 kHz stereo audio from the original recordings at the released windows, located by cross-correlation, and ran it in the same session as the released audio (Table R2, 162 items):
OVSaudio 16 kHzaudio 44.1 kHzav 16 kHzav 44.1 kHz
Gemini-2.5-FL45.646.250.650.1
Gemini-3.8-F51.151.354.053.3
  • Reading: higher fidelity changes OVS by −0.7 to +0.6. App. B.1 gives the audio specification, and the open-weight reruns of RQ6 receive the soundtrack of av.mp4 decoded losslessly to 16 kHz mono (App. C.6).

Thank you for raising the audio format. Therefore, the released encoding does not explain the finding on these spectral dimensions.


AuthorsOpenReview ↗

Response to Reviewer swcE (Part 2/3)


W3: "three 20-second windows ... the coach's reference commentary may describe events occurring outside the provided windows"

Thank you very much. Events outside the windows do not explain the gap: giving two backbones the whole performance does not improve OVS (Figure 11(c)).

  • ① Most observations are not tied to a passage: 422 of 491 test observations (86%, App. C.6) name no time region. The three windows are also centred in the thirds of the sung span (§3.3, Step 6), so the opening, middle and closing thirds each contribute a window.
  • ② Coverage does not predict accuracy. Coverage, the share of the sung span inside the three windows, has a median of 0.337, and 11/83 clips are complete. Its mean per-backbone Spearman ρ with closed-item accuracy is −0.085 over 31 backbones, and the media-minus-none gain does not grow with coverage (all 15 ρ negative, none significant).
  • ③ Whole performance: we drew the 12 truncated clips before any call, 3 per coverage quartile with seed 0, and they last 95–454 s (Table R3):
BackboneMediaHCCTESAESOVS
Gemini-2.5-FL3 × 20 s / full span70.0 / 70.035.7 / 25.639.7 / 39.250.6 / 47.5
Gemini-3.8-F3 × 20 s / full span78.9 / 74.831.4 / 33.943.2 / 40.854.0 / 52.3
  • Reading: OVS does not improve for either backbone, and HCC, whose reference is the coach's full transcript, does not rise with the full span either (70.0 vs. 70.0 and 78.9 vs. 74.8).

Thank you for this concern, which led to the full-span runs. Hence, events occurring outside the provided windows are not what holds the scores down.


Q1(ii): "a human ceiling: vocal teachers or coaches ... answering the same items from the same media"

Thank you very much. The ceiling in the revised paper is built from the human coach's own information: given the coach's own words, backbones reach 71.7–81.9 on TES/AES (Table C8), so the items are answerable from the human reference.

  • This ceiling uses the same backbones and items, with the coach's information in place of the media, so the gap between it and the av scores isolates what the backbones fail to perceive in the clip.
  • Answers from further coaches on the same items belong to the multi-coach direction described under Q2 below.

Thank you for suggesting this reference point. As a result, the scale has an upper reference derived from the human coach, and a ceiling from further coaches is part of the direction under Q2.


Q2: "Please report inter-coach agreement"

Thank you very much. Grading against several coaches per performance is now direction (i) of the new App. E (red), where further coaches' responses to the same performances would show where coaches agree. The current reference is secured as follows.

  • Reference: our gold answers are the coaches' own spoken feedback on the performance they had just heard, and three experts spot-checked a stratified sample for transcript fidelity, genre label and item–evidence agreement (§3.3, Step 7).
  • Judge: the HCC judge is self-consistent (ICC(2,1) = 0.86), and the ranking is stable under an independent judge (τ = 0.78) and without HCC (τ = 0.79; Table 2), so it does not rest on any single judge.

We appreciate this request, which shaped our Future Work. Therefore, inter-coach agreement is set out as direction (i) of our Future Work, and the present ranking rests on an expert-checked reference.


Q3(a): "R&B 2, Jazz 3, Coloratura 1 ... per-genre results with intervals or qualify the claim"

Thank you very much. Following your suggestion, per-genre results with intervals are given below, and the three smallest genres are qualified as indicative. Per-genre TES/AES item accuracy (31-backbone mean) with 95% clip-bootstrap intervals (Table R4; Figure C1(c) plots closed-item accuracy per genre):

Genre (test clips)Accuracy [95% CI]Genre (test clips)Accuracy [95% CI]
Lyrical (28)38.4 [33.7, 43.2]Folk (6)33.3 [21.5, 49.4]
Academic (15)35.2 [30.8, 39.7]Folk-Pop (6)31.3 [19.9, 41.8]
Opera (14)34.1 [28.8, 39.0]Jazz (3)24.6 [9.3, 39.3]
Pop Ballad (8)27.6 [15.2, 39.7]R&B (2) / Coloratura (1)32.5 / 26.6
  • The bootstrap resamples test clips, so each interval shows how much a genre's mean depends on which clips it contains; the intervals overlap and widen as the number of clips falls.
  • Jazz, R&B and Coloratura have 1–3 test clips each, so their values are indicative. The test genre mix matches train+dev (χ², p = 0.94; Table A2).

Thank you for this suggestion; it made the per-genre claim more precise. Hence, the per-genre results now come with intervals and an explicit qualification for the smallest genres.

AuthorsOpenReview ↗

Response to Reviewer swcE (Part 3/3)


Q3(b): "11 of 31 backbones are Gemini and no audio-only LALM is included ... audio-only ... and video-only references on the matched conditions"

Thank you very much. The new §4.7 (RQ6) adds the references you suggest, and no backbone gains significantly from the soundtrack alone.

  • Added backbones (App. C.6): four open-weight omni backbones (Qwen2.5-Omni-7B, Qwen3-Omni-Instruct, TLive-Omni-9B and MiniCPM-o-4.5), the audio-only LALM Qwen2-Audio-7B-Instruct and the vision-only Qwen2.5-VL-7B-Instruct on the matched conditions; the re-run omni backbones match their Table 1 av scores within the 95% interval.
  • Lifts on the 946 non-visual items (Table R5 = Figure 11(a, b); McNemar p in parentheses):
Backboneaudio − noneav − video
Qwen2-Audio (audio-only)−3.0 (0.05)–
Qwen2.5-Omni-7B+1.1 (0.49)+1.6 (0.19)
Qwen3-Omni-Instruct+1.3 (0.43)+0.6 (0.69)
TLive-Omni-9B−1.4 (0.35)+0.7 (0.55)
MiniCPM-o-4.5+1.8 (0.26)+2.8 (0.0495)
5 Gemini backbones−4.9 to −0.1−3.0 to +3.0
  • ① Audio alone (audio − none): pooled over all 10, the lift is −1.65 (95% CI [−3.09, −0.27]), and −1.07 [−2.58, 0.40] on the 937 items with auditory evidence. Qwen2-Audio, built for listening, scores 41.9 OVS with its input and 41.3 without.
  • ② Audio on video (av − video): it helps significantly only MiniCPM-o (+2.8) and Gemini-3-Flash (+3.0), 2 of 9, and the benchmark detects such gains.
  • ③ Vision-only reference: it gains +8.3 points from muted video on the 264 sight items (p = 0.005), so the conditions register a modality gain where a model can use it.

Thank you for suggesting these references. Therefore, "not listening" now rests on ten backbones across families, including a model built only for listening.


Q4(a): "correlation between HCC scores and ratings from ≥2 human experts"

Thank you very much. Every HCC score is anchored to a human expert: HCC scores each commentary against the coach's own feedback on that performance. In addition, an independent point-anchored judge, which labels each coach point as matched, contradicted or absent, keeps the ranking (τ = 0.78, same top-3) while checking the coach's points one by one. We appreciate this proposal. Hence, HCC is tied to the expert's own judgement of each performance, and the ranking does not depend on this one judge.


Q4(b): "self-consistency, and a swap test"

Thank you very much. The new Table 2 (p. 9) reports the swap test, and the judge is self-consistent, with ICC(2,1) = 0.86 over two re-judgings of 249 commentaries.

  • Swap test (83 test clips, the paper's HCC judge, complementing the §3.4 control): the judge compares pairs of texts built to differ in one property (their judgements, the performance they describe, their content, or their length). In the first two pairs, the first text is the coach's own transcript rewritten as a commentary.
  • ① Correctness: inverting the coach's judgements costs 30.7 points (82/83 clips) and another same-genre clip's content 12.3 (72/83).
  • ② Fluency and length: the judge prefers c∗c^* to a fluent content-free commentary (+3.7, 58/83, Wilcoxon p < 1e-4) and lowers a commentary padded with generic filler by 8.4 (77/83).

Thank you for both checks. Therefore, the judge is self-consistent, and the swap test shows that it rewards correct, clip-specific content over fluency or length.


Q4(c): "OVS under alternative weightings"

Thank you very much. For the three alternative weightings you name, the top-3 is unchanged in each: Kendall τ vs. Eq. (2) is 0.98 for equal tracks, 0.82 for item-count-proportional tracks and 0.79 with HCC dropped. Across all 16 combinations of level and track weights, the best open-weight backbone is unchanged, so the ranking is not an artifact of Eq. (2). Thank you for specifying these weightings. As a result, OVS keeps its ranking under the alternative weightings you name.


Summary of revisions (red in the revised PDF)

WhereChangeAnswers
§4.7 (new RQ6), Figure 11, Table 2 (p. 9)Open-weight, audio-only and vision-only backbones; pooled audio lift; 44.1 kHz and full-span runs; alternative scorings; HCC judge checksW2, W3, Q1(ii), Q2, Q3(b), Q4(a)–(c)
App. C.6 (pp. 26–28)Protocol and results of RQ6, incl. closed-item floors and the own-transcript ceilingW1(a) & Q1(i), W1(b), W2, W3, Q3(b)
App. E (p. 31)Future Work, incl. multiple coach referencesQ1(ii), Q2

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score further. Thank you very much!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you very much for your time and thorough reviews. We are glad that you found OmniVPC-Bench "a valuable benchmark for professional performance evaluation" and "a meaningful exploration of the capability boundaries of multimodal models" (7xtx), saw that it is built on "coaches' responses from real vocal lessons and master classes," which are "valuable real-world, human-generated resources" (qxys), and valued its "four matched media conditions," "session-, singer-, and piece-disjoint splits" and "reproducible, scripted construction pipeline" (swcE). Your shared questions concern the single-coach reference, whether the audio finding depends on the Gemini family or the 16 kHz audio, the three 20-second windows, the HCC judge and the OVS weights.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite all reviewers to open the revised PDF to view them.

In the revised PDF, every passage we revised in response to the reviews is marked in red, including the changes that address concerns raised by several reviewers. The red phrases and cross marks in Table 3 ("contradicts or is absent") and the red elements of our figures, such as markers, lines and bars, are data colours rather than revision marks.

We added a new §4.7 (RQ6) with Figure 11 and Table 2 (p. 9), its full protocol and results in a new App. C.6 (pp. 26–28), and a new App. E Future Work (p. 31). The table lists each concern, what we added with its main result, and where it appears:

ConcernWhat we added, and the main resultWhere in the revised PDFReviewer(s)
Single-coach referenceRanking without the L1 "not addressed" class, the only class that relies on what the coach left out (163 of the 1,293 test items): Kendall τ = 0.96 with the same top-3. Every other closed item asks about a point the coach made, so a valid remark outside the reference is neither rewarded nor penalized there. The backbones fail to verify even the coach's own observations (d′d' = −0.40 to 0.42, all 31 backbones), yet the same items become answerable once the coach's own transcript replaces the media, so we read the low scores as a gap in perceiving the performance. Grading against several coaches per performance is direction (i) of Future WorkTable 2, p. 9; App. E (p. 31) (red)all
Is "not listening" specific to Gemini?Four open-weight omni backbones, the audio-only Qwen2-Audio and the vision-only Qwen2.5-VL on the matched conditions: no backbone gains significantly from audio alone; pooled over all 10, the lift is −1.65 (95% CI [−3.09, −0.27]), while the vision-only Qwen2.5-VL gains +8.3 from muted video on the 264 sight items; the matched conditions still detect the small audio-on-video gains of 2 of 9 backbones§4.7, Figure 11(a, b); App. C.6 (red)7xtx, swcE
16 kHz audio44.1 kHz stereo runs on 12 clips: OVS changes by only −0.7 to +0.6, and all 20 open-weight backbones resample audio to 16 kHz§4.7, Figure 11(c) (red)7xtx, swcE
Three 20-second windowsFull-span runs on the same 12 clips: the whole sung span does not raise OVS for either backbone tested (50.6 to 47.5 and 54.0 to 52.3); 86% of test observations name no time region§4.7, Figure 11(c); App. C.6 (red)qxys, swcE
HCC judgeJudge checks on all 83 test clips: inverting the coach's judgements costs 30.7 points (82/83 clips) and another same-genre clip's content 12.3 (72/83); generic filler lowers the score by 8.4 (77/83); ICC(2,1) = 0.86Table 2, p. 9 (red)qxys, swcE
OVS weightsRescoring of all 31 backbones under alternative scorings: Kendall τ is 0.96 with equal level weights, 0.98 with equal tracks, 0.82 item-proportional, 0.79 without HCC and 0.78 with a point-anchored judge; the top three never changeTable 2, p. 9 (red)7xtx, swcE
Floor and ceilingExact closed-item floors and an own-transcript ceiling: floors are TES 19.5 and AES 26.8; with the coach's own transcript, two Gemini backbones reach 71.7–81.9 on TES/AES versus 27.8–43.7 from the mediaApp. C.6, p. 28 (red); Table C8swcE
GenresPer-genre results with 95% clip-bootstrap intervals; the test genre mix matches train+dev (χ², p = 0.94)swcE thread; Table A27xtx, swcE

The results are also reproducible. The open-weight reruns use identical clips and windows and the released code, and the re-run omni backbones match their Table 1 av scores within the 95% interval. The alternative scorings rescore stored answers without re-running any model.

Each reviewer's thread below gives the full evidence and ends with a table of where every change appears. If any point deserves further discussion, we would be glad to elaborate. Thank you again for your valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.