acceptodds
Under review as a conference paper at ICLR 2027

MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

Abstract

Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce MuseBench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, and expert validation. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.28% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models' creative domain expertise. Further analysis points to a consistent failure pattern in which models lag sharply on game arts, recover only the single most salient option on multi-select pairs, and gain little from adaptive key frame selection, suggesting the bottleneck lies in stylistic vocabulary and cultural priors rather than temporal localization.

Primary area:
datasets and benchmarks
Keywords:
Audiovisual Art, Video Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6cAmOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MuseBench, a benchmark for intent-level audiovisual arts understanding in MLLMs. It contains 4,016 questions covering cinematic, static visual, stage performing, and game arts. The benchmark uses expert commentary from video essays, an iterative human-in-the-loop construction pipeline, and mixed single-select/multi-select questions. Evaluation of 28 MLLMs highlights challenges in interpreting artistic choices beyond factual recognition.

Critical strengths

-The benchmark addresses a meaningful and useful problem. Understanding how creative choices convey emotion and meaning is relevant to arts education, media analysis, and creative assistance.

-The coverage of four artistic domains is valuable, particularly the inclusion of stage performance and game arts. This broadens evaluation beyond conventional image aesthetics and cinematography tasks.

-Using video essays as a source of expert knowledge is well motivated. The construction pipeline explicitly addresses evidence alignment, narrator-dependent questions, and distractor quality.

Critical weaknesses

I generally don't have much concern on the paper, and only have several minor ones.

-The novelty mainly lies in the benchmark’s scope and design rather than new algorithms. A clearer comparison with ShotBench [1] would better establish its distinctive contribution.

-Answer-set reliability needs further validation. For example, options D and H in Fig. 19(c) appear semantically equivalent, although only H is marked correct. Independent expert agreement on answers would strengthen confidence.

-The CAA definition for human expert is not clear. There is no a_i, i.e. the correctness regarding a single question for human. So how is the number of human expert in Table 1 come?

[1] Hongbo Liu et al. ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models. NeurIPS, 2025.

Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 6cAm (1/2)

Dear Reviewer 6cAm:

We thank you for recognizing the value of artistic-intent understanding, the coverage of stage performance and game arts, and the motivation for using video essays. We address your questions on ShotBench, answer-set reliability, and the human CAA calculation below.

Response to W1

“The novelty mainly lies in the benchmark’s scope and design rather than new algorithms. A clearer comparison with ShotBench [1] would better establish its distinctive contribution.”

Thank you for suggesting this comparison. MuseBench's contribution lies in defining and evaluating intent-level audiovisual arts understanding across four artistic domains. Its question construction, answer formats, and analyses are organized around connecting observable creative choices to the functions they serve. ShotBench [1] is a closely related contribution, and a direct comparison makes this task distinction clearer.

ShotBench evaluates cinematic language across eight dimensions, including shot size, framing, camera angle, lens size, lighting, composition, and camera movement. MuseBench extends the evaluation to Cinematic Arts, Static Visual Arts, Stage Performing Arts, and Game Arts, covering 11 subdomains. Within these domains, it emphasizes how an artistic technique produces an effect and supports an expressive or functional purpose.

Table R1. Comparison with ShotBench.

AspectShotBenchMuseBench
Domain coverageCinematography across eight dimensionsFour artistic domains and 11 subdomains
Question emphasisCinematic attributes and visual grammar of shotsRelations between artistic choices, their effects, and their functions
EvidenceStill images and video clips from filmsNarrator-removed clips selected from expert video essays
Answer formatFour-option single-choice questions; an option may contain several attributesFour to eight options, combining single-select and multi-select questions
ConstructionTrained annotation and professional cinematography resources with expert verificationExpert-commentary-informed generation, iterative review, and manual verification
Associated contributionShotQA training data and the ShotVL modelCross-domain evaluation, set-valued interpretation, and analysis of 28 MLLMs

For example, identifying a camera angle and explaining how that angle establishes a character's authority are closely connected tasks. MuseBench emphasizes the technique-to-function relation, and extends the same principle to lighting in stage performance, color relationships in visual art, and the communicative function of effects in games.

The mixed answer format is also useful for this target. A creative choice can support several valid analytical dimensions, which a multi-select question can represent explicitly. The submitted results show that this distinction changes model comparisons: InternVL3-78B improves over InternVL3-8B in Overall ACC (37.80% versus 33.07%) but has lower Multi EM (13.53% versus 20.30%). Thus, selecting one plausible interpretation and recovering the full supported set expose different aspects of model capability.

We have added the comparison in Section 2 (pp. 3–4) and Appendix C.1, Table 5 (lines 956–985). The appendix includes illustrative task pairs connecting attribute recognition to technique-to-function reasoning.

AuthorsOpenReview ↗

Author Response to Reviewer 6cAm (2/2)

Response to W2

“Answer-set reliability needs further validation. For example, options D and H in Fig. 19(c) appear semantically equivalent, although only H is marked correct. Independent expert agreement on answers would strengthen confidence.”

Thank you for this suggestion. The intended distinction between D and H concerns the strength of the claim about visual evidence. In the submitted figure, D explicitly says “from the visual cue alone,” adding a claim that the visual cue is sufficient by itself, whereas H describes decisions “based on what they see.” The question targets how the correspondence between an explosion's visible size and its actual impact area helps players assess danger. We have retained H and rewritten D to assert that visual size alone reveals exact damage at every distance, making the distinction explicit in revised Figure 19(c) (p. 41).

Equivalent-option cases are uncommon in the audited sample. Our stratified random audit identifies 3 such questions out of 400 (0.75%; 95% CI: 0.3–2.2%). A further 11 questions (2.75%) receive wording refinements, while 386 questions (96.50%) require no changes.

We also evaluate answer agreement on a 320-question sample, covering all four art categories and both formats. Two domain-matched experts independently answer each item using the evaluation clips, with gold labels, source commentary, and one another's answers hidden.

Table R2. Independent answer agreement, measured before adjudication.

Question formatQuestionsExperts per itemAnswer agreement (%)Additional agreement metric
Single-select160286.88Cohen's κ: 0.79
Multi-select160268.75 (exact set)Mean pairwise Jaccard: 0.82

After adjudication, 303 of 320 original answer sets are confirmed (94.69%), and 17 are updated (5.31%). The option audit quantifies the prevalence of equivalent alternatives, while independent answering directly measures the consistency of the supported interpretations.

The study complements the existing quality-rating AC2 of 0.855. We have added the independent agreement and adjudication outcomes in Appendix D.9.1, Tables 12–13 (lines 1771–1806), and the option audit in Appendix D.11, Table 16 (lines 1833–1876). Section 4.4 (lines 524–529) now presents the agreement and score-sensitivity results in the main experiments. The updated example appears in Figure 19(c). Label-only updates use the stored predictions; rewritten options receive new evaluations on the audited samples. The main leaderboard retains the submitted evaluation and its original scores.

Response to W3

“The CAA definition for human expert is not clear. There is no a_i, i.e. the correctness regarding a single question for human. So how is the number of human expert in Table 1 come?”

Thank you for asking us to clarify this calculation. Each human answer has an item-level correctness indicator, defined in exactly the same way as a model answer. Two domain experts covered all 4,016 questions through disjoint category assignments. Each question therefore has one expert response that can be compared with its reference answer.

For a single-select question ii, let hih_i be the assigned expert's answer, yiy_i the reference answer, and KiK_i the number of options. The correctness indicator is aihuman=1[hi=yi]a_i^{\mathrm{human}}=\mathbb{1}[h_i=y_i], and human CAA is

CAAhuman=100Nsingle∑i∈Saihuman−1/Ki1−1/Ki,\displaystyle \mathrm{CAA}_{\mathrm{human}}=\frac{100}{N_{\mathrm{single}}}\sum_{i\in\mathcal{S}}\frac{a_i^{\mathrm{human}}-1/K_i}{1-1/K_i},

where S\mathcal{S} is the single-select subset and Nsingle=∣S∣N_{\mathrm{single}}=|\mathcal{S}|. A correct response contributes 1 before percentage scaling, and an incorrect response contributes −1/(Ki−1)-1/(K_i-1). The chance adjustment is applied to each question before averaging, accounting for its actual number of options.

The human row in Table 1 reports 90.98% Single CAA, alongside 87.18% Overall ACC and 78.00% Multi EM. Overall ACC averages fully correct responses across both formats, while Multi EM requires the selected option set to match the reference set. The category-specific CAA values use the single-select questions in the corresponding category. Disjoint expert assignments provide the required per-item answers, while the overlapping study described above separately evaluates inter-expert agreement.

We have added the explicit human correctness indicator in Section 3.5 (lines 352–354) and the aggregation formula in Appendix E, Equation (2) (lines 1880–1891). Appendix D.9 retains the expert assignment and performance-baseline details.

[1] Liu H, He J, Jin Y, et al. ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models. NeurIPS, 2025. Paper.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer 6cAm,

Thank you again for your careful review and positive assessment of MuseBench; your comments helped us improve the paper. When your time permits, could you please read our response and check whether your concerns are addressed?

During the rebuttal period, we made every effort to address your concerns as follows.

  • Comparison with ShotBench: We added a direct comparison with ShotBench, clarifying MuseBench's focus on technique-to-function reasoning across four artistic domains and its inclusion of both single-select and multi-select questions.
  • Expert answer study and option audit: We added an independent expert answer study on 320 questions, with 86.88% single-answer agreement and 68.75% multi-answer exact-set agreement. A separate 400-question option audit identified three equivalent-option cases (0.75%), and we clarified the distinction between options D and H in Figure 19(c).
  • Human CAA formulation: We made the human CAA calculation explicit: each expert response supplies an item-level correctness indicator, with chance adjustment applied using each question's number of options before averaging.

We appreciate your constructive feedback, which helped clarify the benchmark's positioning and strengthen its validation. We strongly believe that our responses and the additional results have addressed your concerns. If any concern remains, we would be happy to provide further clarification.

Warm regards, Authors

Reviewer_tVhQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MUSEBENCH, a benchmark for evaluating multimodal large language models on their ability to understand creative intent in the audiovisual arts. The authors curate 4,016 expert-reviewed questions from more than 10,000 candidate video essays, covering four major art categories and 11 subdomains, with variable-option single- and multi-select formats. Questions are constructed using materials with narration, while model evaluation uses audiovisual clips with the narrator audio removed. Zero-shot evaluation of 28 MLLMs shows that the best model achieves 48.28% overall accuracy, below the reported expert baseline of 87.18%. The paper also analyzes performance across categories, question formats, and input modalities.

Critical strengths
  1. The research question is meaningful. The paper moves beyond recognizing video content to examining creative choices and their expressive intent, a capability less commonly addressed by existing video benchmarks. Its four art categories and associated subdomains also provide broad evaluation coverage.

  2. The benchmark construction and empirical analysis are extensive. The paper describes a staged data-construction and human-review process, evaluates 28 models, and further analyzes multi-select performance, category differences, the contribution of audio, and option-position bias. The combination of single- and multi-select questions also attempts to represent the multiple valid dimensions of artistic interpretation.

Critical weaknesses
  1. The treatment of text embedded in the video is insufficiently specified. The paper states that narrator audio is removed during evaluation, but does not clearly explain whether hardcoded subtitles, narration captions, or on-screen annotations are retained or filtered. Since many source videos come from Bilibili and TikTok, on-screen text may contain or summarize the original narration and thus provide an additional textual cue. The authors should clarify how such text is handled and assess its effect on model responses.

  2. Some answer options remain highly similar in wording and insufficiently discriminative. Although the paper reports addressing similar option prefixes and overly academic wording during quality review, such cases remain in the examples. For instance, options B, C, and D in Figure 22(c) share a long prefix and differ only in subtle wording near the end; similar patterns also appear in Figures 19(d) and 21(c). These examples make it difficult to determine whether models are identifying artistic intent or distinguishing fine-grained wording differences. The authors should report the prevalence of such residual cases in the final dataset or explain how human review ensures adequate option discriminability.

  3. The evaluation modalities do not fully match the audiovisual requirements of some questions. Some models receive video and text but no audio, while some questions concern audiovisual coordination in stage performance or the intent behind sound design. The paper’s audio ablation shows that adding audio improves performance for some models. Models without audio may therefore be disadvantaged on these items, and overall rankings may conflate artistic reasoning with modality availability. The authors should report results separately for questions that require audio evidence or stratify model comparisons by input modality. A further taxonomy question concerns Figure 22(d), which places a plant-breeding workflow diagram under Static Visual Arts; the question primarily tests conceptual understanding of a process. The authors could clarify how such items fit the category and the definition of visual necessity.

  4. The text-only shortcut controls do not rule out the possibility that stronger models exploit the question and answer options. The appendix reports no-audiovisual-input controls only for VideoLLaMA2 and Qwen2.5-Omni-7B, and tests Options-only rather than Question + Options. These results do not show whether strong proprietary models can achieve high scores by exploiting semantic or logical relations between the question and options. The authors should provide Question + Options only results for leading models to test this possibility more directly.

  5. The human expert baseline lacks item-level agreement information. Appendix D.9 states that the 87.18% expert baseline was produced by two experts with relevant master’s degrees, assigned to disjoint categories, so each question received only one expert response. The reported score is therefore a combination of responses from two experts rather than an item-level consensus. The reported Gwet AC2 is based on a separate 90-item quality-rating exercise and does not establish agreement on the expert answers. The authors should clarify how expert answers were verified and discuss how single-expert responses affect the interpretation of the 87.18% baseline.

  6. Figure 1 does not render correctly in standard viewing environments. I observed this on macOS and Windows using both Chrome and Edge, in the OpenReview version and in the locally downloaded PDF; the figure was visible only after extracting it with Python. Since the figure provides an overview of the paper, this rendering issue affects its presentation. The authors should fix the submitted PDF and verify that the figure displays correctly in common PDF viewers.

Questions for authors
  1. Narrator removal and on-screen text: Are hardcoded subtitles, narration captions, or other on-screen annotations retained in the evaluation videos? If so, how do the authors rule out the possibility that this text contains information from the original narration and provides a textual shortcut?

  2. Text-only shortcuts for stronger models: Could the authors report results for leading models such as Claude-4.6-Opus and Qwen-3.5-Plus under a Question + Options only setting? This would help determine whether their scores rely on textual cues in the question and options.

  3. Option discriminability: Table 8 reports addressing similar option prefixes and near-duplicate options, yet examples such as Figure 22(c) still contain visibly similar options. How were these residual cases handled during human review? Could the authors report their prevalence in the final dataset and their potential effect on model performance?

  4. Expert baseline: The expert baseline in Appendix D.9 uses disjoint category assignments, with only one expert answering each question. How do the authors ensure that these responses provide a reliable reference? Could they report inter-expert agreement or adjudication results on a subset of overlapping items?

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer tVhQ (1/5)

Dear Reviewer tVhQ:

We thank you for recognizing the motivation of MuseBench, its broad artistic coverage, and the construction and evaluation efforts. We address your questions on textual cues, option design, modality-specific evaluation, the expert baseline, and figure rendering below.

Response to Q1/W1

“Are hardcoded subtitles, narration captions, or other on-screen annotations retained in the evaluation videos? If so, how do the authors rule out the possibility that this text contains information from the original narration and provides a textual shortcut?”

Narration subtitles and answer-revealing explanatory overlays are removed or masked before model evaluation. We manually inspected all 4,016 final evaluation clips and checked the processed videos to ensure that these narration-derived cues had been removed. Artwork-intrinsic text, such as dialogue subtitles, game interfaces, and text used in a visual composition, was retained as part of the artwork.

OCR assisted this full manual review. The construction also checks that questions are answerable from the narrator-removed evaluation clips, with generated clip captions reserved for construction. Table R1 counts text found before processing; the main evaluation uses the processed clips. Table R2 compares unmasked and masked video while holding the audio transcript, questions, options, and evaluation settings fixed. A separate text-only condition supplies extracted on-screen text with the question and options.

Table R1. Text identified before processing during the full manual review and its treatment in the final evaluation clips.

Text typeQuestions inspectedAffected questionsPercentageTreatment during evaluation
Narration subtitles4,0162125.28Blurred in the final evaluation clips
Explanatory overlays or answer-revealing annotations4,0161874.66Blurred; paired control in Table R2
Dialogue subtitles, game UI, or artwork-intrinsic text4,0161,14628.54Retained as part of the artwork

Text categories overlap: 1,449 items (36.08%) contained at least one text type before processing.

Table R2. On-screen text controls on 187 questions (94 single-select, 93 multi-select). Scores are ACC / Single CAA / Multi EM (%). The masking comparison holds the audio transcript, questions, options, and evaluation settings fixed.

ModelMatched questionsBefore explanatory-text maskingExplanatory-text-masked clipsExtracted text + Q+O onlyMasked minus unmasked ACC, 95% CI
Claude-4.6-Opus18752.41 / 68.44 / 30.1149.73 / 63.16 / 29.0333.16 / 44.76 / 10.75-2.67 [-5.94, 0.58]
Qwen-3.5-Plus18750.27 / 67.12 / 26.8848.13 / 63.16 / 25.8131.55 / 42.12 / 9.68-2.14 [-5.31, 1.03]

Before processing, explanatory overlays were identified in 4.66% of inspected questions. Masking them decreases ACC by 2.67 and 2.14 points, while input with masked video and the audio transcript outperforms extracted text plus Q+O by 16.58 points for both models. A comparable-region blur control decreases ACC by 0.53 points for each model. These comparisons quantify the contribution of embedded text and the value of the remaining evaluation inputs.

We have added the audit, paired results, and comparable-region blur control in Appendix D.10, Tables 14–15 (lines 1809–1844). Section 4.4 (lines 517–523) now presents the main results, and Section 3.4 (lines 291–293) clarifies the full manual review.

AuthorsOpenReview ↗

Author Response to Reviewer tVhQ (2/5)

Response to Q2/W4

“Could the authors report results for leading models such as Claude-4.6-Opus and Qwen-3.5-Plus under a Question + Options only setting? This would help determine whether their scores rely on textual cues in the question and options.”

Thank you for suggesting this comparison. MuseBench is designed to make the audiovisual evidence useful for distinguishing plausible interpretations. The correct answer is written before distractor generation, distractors are required to be plausible without the clip, and option positions are randomized. The appendix provides options-only and transcript-plus-question-and-options controls for VideoLLaMA2 and Qwen2.5-Omni-7B. Your proposed setting extends this analysis to the leading proprietary models and directly measures the information supplied by the question and options together.

The added Q+O-only condition contains the task instruction, question, and options. Alongside the two models named in the review, we additionally evaluate GPT-6.1-Sol and Claude-Sonnet-5.5 for this rebuttal. For each model, we compare options-only, Q+O, and the full-input condition using the same items, option order, decoding settings, and scoring.

Table R3. Strong-model language-shortcut controls and full-input results (%).

ModelInputQuestionsOverall ACCSingle CAAMulti EM
GPT-6.1-SolOptions only4,01623.5322.745.02
GPT-6.1-SolQuestion + options4,01632.0536.8710.14
GPT-6.1-SolVideo + audio transcript + question + options4,01658.4266.3140.13
Claude-Sonnet-5.5Options only4,01622.8621.614.63
Claude-Sonnet-5.5Question + options4,01631.1335.519.42
Claude-Sonnet-5.5Video + audio transcript + question + options4,01656.7564.4538.24
Claude-4.6-OpusOptions only4,01621.3418.674.12
Claude-4.6-OpusQuestion + options4,01629.1831.898.64
Claude-4.6-OpusVideo + audio transcript + question + options4,01648.2855.1328.91
Qwen-3.5-PlusOptions only4,01622.4120.364.85
Qwen-3.5-PlusQuestion + options4,01630.3533.939.25
Qwen-3.5-PlusVideo + audio transcript + question + options4,01647.2658.5223.21

Adding video and the Whisper-Large-v3 transcript to Q+O improves ACC by 26.37 and 25.62 points for GPT-6.1-Sol and Claude-Sonnet-5.5, and by 19.10 and 16.91 points for Claude-4.6-Opus and Qwen-3.5-Plus. Single CAA and Multi EM improve for all four models. The gains from video and audio-derived transcripts hold across both answer formats and extend to the two newly evaluated systems.

We now report the Q+O and full-input comparisons in the main experiments, Section 4.4, Table 3 (lines 509–517). Appendix F.1.1, Table 19 (lines 2023–2051) provides all three conditions and Single CAA, which accounts for the different guessing probabilities associated with four to eight options. The two newly evaluated systems are identified as supplementary to the original 28-model main evaluation.

AuthorsOpenReview ↗

Author Response to Reviewer tVhQ (3/5)

Response to Q3/W2

“Table 8 reports addressing similar option prefixes and near-duplicate options, yet examples such as Figure 22(c) still contain visibly similar options. How were these residual cases handled during human review? Could the authors report their prevalence in the final dataset and their potential effect on model performance?”

Thank you for pointing to these examples. A shared opening keeps the subject and terminology constant, while the alternatives test different interpretations of the artistic evidence. We assess whether their claims about a technique's effect or function can be distinguished against the clip. The revised examples make these distinctions explicit while retaining the same keyed answers.

Table R4. Option distinctions and revisions in the cited examples.

ExampleDistinction and revised wording
Fig. 22(c)D retains multiple complementary color relations; B now states one exclusive complementary contrast, and C specifies analogous relations.
Fig. 19(d)A retains the artificial, oppressive atmosphere supported by uniform facades and stark shadows. The overlapping B, C, and F now express sheltered, historical, and landmark-oriented readings.
Fig. 21(c)B retains the contrast between public authority and domestic demands beyond the character's control; C expresses domestic authority, and D treats both settings as neutral backdrops.
Fig. 19(c), D/HH retains decisions based on visible extent; D now asserts that visual size alone reveals exact damage at every distance.

The construction pipeline combines distractor strategies, iterative expert feedback, and manual verification of every retained item. Submitted Table 8, now Table 10, records construction-stage issues and fixes, as clarified in its caption and Appendix D.8. We additionally audited 400 stratified randomly sampled questions to measure residual wording and equivalence issues in the final set.

Table R5. Option audit and paired score sensitivity on 400 questions (200 single-select and 200 multi-select).

MeasureResult
Items requiring no modification386 / 400 (96.50%)
Items requiring clearer wording11 / 400 (2.75%, 95% CI 1.5–4.9%)
Items with equivalent options requiring answer-set correction3 / 400 (0.75%, 95% CI 0.3–2.2%)
Claude-4.6-Opus, original / audit-reviewed ACC and EM on the 400 audited itemsACC 48.75 → 50.25; EM 29.50 → 30.50
Qwen-3.5-Plus, original / audit-reviewed ACC and EM on the 400 audited itemsACC 47.25 → 48.25; EM 23.50 → 24.50

A separate full-benchmark screen identifies 617 questions with an option pair sharing a prefix of at least 50 characters. Experts find clearly distinct claims in 543 (88.0%). The remaining 74 (12.0%) show distinguishable interpretations with minor semantic overlap. This screen identifies repeated phrasing; the random audit above estimates the prevalence of wording and equivalence issues.

On the 400 audited items, Claude's correct-answer count increases from 195 to 201 (+1.50 ACC points), and Qwen's from 189 to 193 (+1.00 point). Both gain two exactly answered multi-select questions (+1.00 EM point). Their relative ordering remains unchanged. These paired comparisons measure sensitivity to the audited corrections; the main leaderboard retains the submitted scores.

The audit and score comparison appear in Appendix D.11, Table 16 (lines 1833–1867), with example explanations at lines 1869–1876. Section 4.4 (lines 524–529) summarizes the score sensitivity alongside the independent answer study.

Reviewer_LwW7OpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MUSEBENCH, a benchmark designed to evaluate intent-level audiovisual arts understanding in multimodal large language models (MLLMs). Unlike existing video understanding benchmarks that primarily focus on perceptual recognition and factual comprehension, MUSEBENCH aims to assess whether models can interpret the artistic intentions behind creative choices, such as cinematography, composition, lighting, editing, and sound design.

The benchmark contains 4,016 questions spanning four major artistic domains and 11 sub-domains, constructed from professional video essays. The authors propose a multi-stage data construction pipeline involving video segmentation, visual captioning, question generation, adversarial distractor generation, and iterative human validation. To accommodate multiple valid artistic interpretations, the benchmark incorporates both single-select and multi-select questions with varying numbers of options.

The authors evaluate 28 MLLMs under zero-shot settings using overall accuracy, chance-adjusted accuracy for single-select questions, and exact match for multi-select questions. Experimental results reveal a substantial gap between current models and human experts, with the best-performing model achieving 48.28% accuracy compared to 87.18% for human experts. Further analysis highlights difficulties in game arts understanding and multi-select reasoning, as well as limited benefits from adaptive keyframe selection.

Critical strengths
  1. The paper explores a meaningful and underexplored direction, moving beyond perceptual recognition toward understanding artistic intent.
  2. The benchmark covers diverse artistic domains and includes multi-select questions and human quality validation.
Critical weaknesses
  1. Concerns about the model evaluation protocol. The evaluation mixes models with different input modalities (V+T and V+A+T), making direct comparisons potentially unfair. More importantly, Table 1 classifies GPT-5.4 as V+A+T, although its official API does not support native audio input. This raises concerns about the accuracy and reproducibility of the reported evaluation setup. The authors should clarify the actual input pipelines and ensure consistent evaluation conditions.
  2. Insufficient validation of subjective ground-truth answers. Artistic interpretation is inherently subjective, and multiple interpretations may be equally valid. Although the benchmark introduces multi-select questions, the predefined answer sets are not validated through independent expert agreement. Each question in the human performance evaluation is answered by only one expert, while the reported inter-annotator agreement measures question quality rather than answer agreement. This raises concerns about the reliability of the ground truth, especially when strict exact-match scoring penalizes alternative but potentially valid interpretations.
Questions for authors
  1. How was audio provided to GPT-5.4, given that its official API does not support native audio input? Can the authors clarify the input pipelines and fairness of cross-model comparisons?

  2. Have the authors measured answer-level agreement among independent experts? How are alternative valid artistic interpretations handled when they differ from the predefined ground truth?

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer LwW7 (1/2)

Dear Reviewer LwW7:

We thank you for recognizing the importance of intent-level audiovisual arts understanding and the breadth of MuseBench. We clarify the evaluation inputs, the role of audio in model comparisons, and the validation of expert-grounded answers below.

Response to Q1/W1

“How was audio provided to GPT-5.4, given that its official API does not support native audio input? Can the authors clarify the input pipelines and fairness of cross-model comparisons?”

Thank you for catching this typo. GPT-5.4 does not support native audio input through the API used in our evaluation. Its V+A+T label in the submitted Table 1 was a typo; we have corrected it to V+T in revised Table 1 (p. 8). It received sampled video frames, a Whisper-Large-v3 transcript of the narrator-removed audio, and the question and options.

The evaluation follows the input pipeline specified in the submitted paper. Section 4.1 (lines 352–355) states: “Audio-capable models receive the full narrator-removed audiovisual clip; other models receive a transcript of the same audio.” The same paragraph specifies 1 fps frame sampling subject to each model's maximum frame count. Section 3.5 and Appendix E define the scoring rules applied to every model. GPT-5.4 follows this documented transcript route, as do Claude-4.6-Opus and Qwen-3.5-Plus.

Our primary goal is to identify each model's capabilities and weaknesses in audiovisual arts understanding. Section 4.2 develops this goal through comparisons with expert performance and capability profiles across artistic domains. Every system answers the same benchmark questions with the same options and scoring rules, using the input route documented in Section 4.1. Native audio and transcripts convey different information, so modality support is part of the capability profile we report. Scores describe what each model can achieve through its supported interface; the within-model ablations below identify the contribution of audio under controlled conditions. Table R1 summarizes the input routes.

Table R1. Evaluation input details. Q+O denotes the question, task instruction, and answer options.

ModelAccessVisual samplingAudio-derived inputEvaluation input
GPT-5.4Official API1 fps, model-specific frame limitWhisper-Large-v3 transcriptV + transcript + Q+O
Claude-4.6-OpusOfficial API1 fps, model-specific frame limitWhisper-Large-v3 transcriptV + transcript + Q+O
Qwen-3.5-PlusOfficial API1 fps, model-specific frame limitWhisper-Large-v3 transcriptV + transcript + Q+O
Qwen2.5-Omni-7BLocal weights1 fps, model-specific frame limitNative audio waveformV+A+Q+O

Section 4.3, Finding 5 and Table 2 isolate audio's contribution through matched within-model comparisons. On all 4,016 questions, the same models are evaluated with V+T and V+A+T inputs:

Table R2. Audio ablation reported in the submitted paper (%).

ModelInputQuestionsOverall ACCSingle CAAMulti EM
MiniCPM-oV+T4,01623.8319.2811.72
MiniCPM-oV+A+T4,01631.3527.0518.90
Gemma-4-E4BV+T4,01621.1920.566.27
Gemma-4-E4BV+A+T4,01627.6128.679.06

Audio helps models answer some benchmark questions. Adding it improves Overall ACC by 7.52 and 6.42 percentage points, respectively, with gains in both question formats. Holding the model and question set fixed isolates this contribution. Together with the domain-specific results in Section 4.2, these analyses diagnose the evidence each model can use and the artistic-understanding tasks on which it falls short.

We have grouped revised Table 1 by actual input modality within each model tier and added the per-model input summary in Appendix F.2, Table 20 (lines 2054–2076). Section 4.1 also specifies the transcript route used by GPT-5.4, Claude-4.6-Opus, and Qwen-3.5-Plus.

AuthorsOpenReview ↗

Author Response to Reviewer LwW7 (2/2)

Response to Q2/W2

“Have the authors measured answer-level agreement among independent experts? How are alternative valid artistic interpretations handled when they differ from the predefined ground truth?”

Thank you for asking us to clarify the validation. The reported AC2 = 0.855 concerns quality ratings, while the 87.18% expert baseline comes from domain-matched experts answering the benchmark. Following your suggestion, the additional overlapping study directly measures agreement on independently selected answers.

The construction process combines expert-commentary-grounded answers with manual verification. Every retained question is checked for answerability from the narrator-removed clips, option quality, and evidence alignment. Multi-select questions represent several supported analytical dimensions, such as attention, atmosphere, and narrative function, within the same item. As described in Appendix G, the reference answers capture source-supported interpretations within the question's answer space. Exact match evaluates recovery of this supported set, while precision, recall, and F1 describe partial agreement.

For the additional validation, two domain-matched experts independently answer each sampled question across all four categories and both formats. They use the evaluation clips and options, with gold labels, source commentary, and other experts' answers hidden. Agreement is measured before adjudication, followed by a third expert's evidence-based review of disagreements.

Table R3. Independent expert answer agreement.

CategorySingle / multi itemsExperts per itemSingle answer agreement (%)Single Cohen's κMulti exact-set agreement (%)Multi mean pairwise Jaccard
Cinematic Arts40 / 40292.500.8975.000.86
Static Visual Arts40 / 40290.000.8470.000.84
Stage Performing Arts40 / 40285.000.7767.500.81
Game Arts40 / 40280.000.7062.500.77

Across 320 independently evaluated questions, experts reach 86.88% single-answer agreement (κ = 0.79) and 68.75% multi-answer exact-set agreement (Jaccard = 0.82). Expert review confirms 94.69% of the original answer sets. These results provide direct answer-level support for the consistency of the benchmark's expert-grounded interpretations.

Alternative interpretations are assessed against the audiovisual evidence. The adjudication records the rationale for each accepted answer and incorporates additional supported options where appropriate. The accompanying score comparison quantifies the effect of these decisions on the model evaluation.

Table R4. Answer-set validation and score comparison on the same audited items.

MeasureResult
Independently audited items320
Items whose original answer set is confirmed303 (94.69%)
Answer sets updated after expert adjudication17 (5.31%)
Questions or options revised13
Claude-4.6-Opus, original / validated-key ACC and EMACC 49.06 → 50.31; EM 30.00 → 31.25
Qwen-3.5-Plus, original / validated-key ACC and EMACC 47.19 → 48.13; EM 23.75 → 24.38

On the matched audited set, validation increases Claude-4.6-Opus's ACC by 1.25 points and Qwen-3.5-Plus's by 0.94 points. Claude retains the lead, with the gap changing from 1.88 to 2.19 points. The study combines direct answer-level validation with a measured check of the resulting model comparison.

We have added the answer-level study and adjudication results in Appendix D.9.1, Tables 12–13 (lines 1771–1806). Section 4.4 (lines 524–529) now presents the agreement results and score sensitivity in the main experiments; Section 3.4 (lines 331–333) distinguishes this study from quality ratings. Label-only updates use the stored predictions, and revised questions or options receive new evaluations on the audited subset. The main leaderboard retains the submitted evaluation and its original scores.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer LwW7,

Thank you again for your careful review and for recognizing the importance of intent-level audiovisual arts understanding; your comments helped us improve the paper. When your time permits, could you please read our response and check whether your concerns are addressed?

During the rebuttal period, we made every effort to address your concerns as follows.

  • Modality labels and input protocols: We corrected GPT-5.4's modality label to V+T and documented its actual inputs (sampled video frames, an audio transcript, and the question and options). We also grouped model comparisons by actual input modality and clarified how the existing within-model ablations isolate audio's contribution.
  • Independent expert answer study: We added an independent expert answer study on 320 questions, distinct from the quality-rating exercise. Before adjudication, single-answer agreement reached 86.88% (κ = 0.79) and multi-answer exact-set agreement reached 68.75% (Jaccard = 0.82).
  • Adjudication and subset sensitivity: We clarified how alternative interpretations are assessed against audiovisual evidence. Expert adjudication confirmed 303 original answer sets and updated 17; paired evaluation on the audited subset changed ACC by 1.25 and 0.94 percentage points for Claude-4.6-Opus and Qwen-3.5-Plus, respectively, preserving their relative ordering.

We appreciate your constructive feedback, which helped clarify the input protocols and strengthen our answer-level validation. We strongly believe that our responses and the additional results have addressed your concerns. We would be grateful if you could reconsider your recommendation in light of these clarifications and additional results. If any concern remains, we would be happy to provide further clarification.

Warm regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.