- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MuseBench, a benchmark for intent-level audiovisual arts understanding in MLLMs. It contains 4,016 questions covering cinematic, static visual, stage performing, and game arts. The benchmark uses expert commentary from video essays, an iterative human-in-the-loop construction pipeline, and mixed single-select/multi-select questions. Evaluation of 28 MLLMs highlights challenges in interpreting artistic choices beyond factual recognition.
Critical strengths
-The benchmark addresses a meaningful and useful problem. Understanding how creative choices convey emotion and meaning is relevant to arts education, media analysis, and creative assistance.
-The coverage of four artistic domains is valuable, particularly the inclusion of stage performance and game arts. This broadens evaluation beyond conventional image aesthetics and cinematography tasks.
-Using video essays as a source of expert knowledge is well motivated. The construction pipeline explicitly addresses evidence alignment, narrator-dependent questions, and distractor quality.
Critical weaknesses
I generally don't have much concern on the paper, and only have several minor ones.
-The novelty mainly lies in the benchmark’s scope and design rather than new algorithms. A clearer comparison with ShotBench [1] would better establish its distinctive contribution.
-Answer-set reliability needs further validation. For example, options D and H in Fig. 19(c) appear semantically equivalent, although only H is marked correct. Independent expert agreement on answers would strengthen confidence.
-The CAA definition for human expert is not clear. There is no a_i, i.e. the correctness regarding a single question for human. So how is the number of human expert in Table 1 come?
[1] Hongbo Liu et al. ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models. NeurIPS, 2025.
Questions for authors
See the weaknesses.
5 comments
Author Response to Reviewer 6cAm (2/2)
Response to W2
“Answer-set reliability needs further validation. For example, options D and H in Fig. 19(c) appear semantically equivalent, although only H is marked correct. Independent expert agreement on answers would strengthen confidence.”
Thank you for this suggestion. The intended distinction between D and H concerns the strength of the claim about visual evidence. In the submitted figure, D explicitly says “from the visual cue alone,” adding a claim that the visual cue is sufficient by itself, whereas H describes decisions “based on what they see.” The question targets how the correspondence between an explosion's visible size and its actual impact area helps players assess danger. We have retained H and rewritten D to assert that visual size alone reveals exact damage at every distance, making the distinction explicit in revised Figure 19(c) (p. 41).
Equivalent-option cases are uncommon in the audited sample. Our stratified random audit identifies 3 such questions out of 400 (0.75%; 95% CI: 0.3–2.2%). A further 11 questions (2.75%) receive wording refinements, while 386 questions (96.50%) require no changes.
We also evaluate answer agreement on a 320-question sample, covering all four art categories and both formats. Two domain-matched experts independently answer each item using the evaluation clips, with gold labels, source commentary, and one another's answers hidden.
Table R2. Independent answer agreement, measured before adjudication.
| Question format | Questions | Experts per item | Answer agreement (%) | Additional agreement metric |
|---|---|---|---|---|
| Single-select | 160 | 2 | 86.88 | Cohen's κ: 0.79 |
| Multi-select | 160 | 2 | 68.75 (exact set) | Mean pairwise Jaccard: 0.82 |
After adjudication, 303 of 320 original answer sets are confirmed (94.69%), and 17 are updated (5.31%). The option audit quantifies the prevalence of equivalent alternatives, while independent answering directly measures the consistency of the supported interpretations.
The study complements the existing quality-rating AC2 of 0.855. We have added the independent agreement and adjudication outcomes in Appendix D.9.1, Tables 12–13 (lines 1771–1806), and the option audit in Appendix D.11, Table 16 (lines 1833–1876). Section 4.4 (lines 524–529) now presents the agreement and score-sensitivity results in the main experiments. The updated example appears in Figure 19(c). Label-only updates use the stored predictions; rewritten options receive new evaluations on the audited samples. The main leaderboard retains the submitted evaluation and its original scores.
Response to W3
“The CAA definition for human expert is not clear. There is no a_i, i.e. the correctness regarding a single question for human. So how is the number of human expert in Table 1 come?”
Thank you for asking us to clarify this calculation. Each human answer has an item-level correctness indicator, defined in exactly the same way as a model answer. Two domain experts covered all 4,016 questions through disjoint category assignments. Each question therefore has one expert response that can be compared with its reference answer.
For a single-select question , let be the assigned expert's answer, the reference answer, and the number of options. The correctness indicator is , and human CAA is
where is the single-select subset and . A correct response contributes 1 before percentage scaling, and an incorrect response contributes . The chance adjustment is applied to each question before averaging, accounting for its actual number of options.
The human row in Table 1 reports 90.98% Single CAA, alongside 87.18% Overall ACC and 78.00% Multi EM. Overall ACC averages fully correct responses across both formats, while Multi EM requires the selected option set to match the reference set. The category-specific CAA values use the single-select questions in the corresponding category. Disjoint expert assignments provide the required per-item answers, while the overlapping study described above separately evaluates inter-expert agreement.
We have added the explicit human correctness indicator in Section 3.5 (lines 352–354) and the aggregation formula in Appendix E, Equation (2) (lines 1880–1891). Appendix D.9 retains the expert assignment and performance-baseline details.
[1] Liu H, He J, Jin Y, et al. ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models. NeurIPS, 2025. Paper.
A gentle reminder
Dear Reviewer 6cAm,
Thank you again for your careful review and positive assessment of MuseBench; your comments helped us improve the paper. When your time permits, could you please read our response and check whether your concerns are addressed?
During the rebuttal period, we made every effort to address your concerns as follows.
- Comparison with ShotBench: We added a direct comparison with ShotBench, clarifying MuseBench's focus on technique-to-function reasoning across four artistic domains and its inclusion of both single-select and multi-select questions.
- Expert answer study and option audit: We added an independent expert answer study on 320 questions, with 86.88% single-answer agreement and 68.75% multi-answer exact-set agreement. A separate 400-question option audit identified three equivalent-option cases (0.75%), and we clarified the distinction between options D and H in Figure 19(c).
- Human CAA formulation: We made the human CAA calculation explicit: each expert response supplies an item-level correctness indicator, with chance adjustment applied using each question's number of options before averaging.
We appreciate your constructive feedback, which helped clarify the benchmark's positioning and strengthen its validation. We strongly believe that our responses and the additional results have addressed your concerns. If any concern remains, we would be happy to provide further clarification.
Warm regards, Authors
Author Response to Reviewer 6cAm (1/2)
Dear Reviewer 6cAm:
We thank you for recognizing the value of artistic-intent understanding, the coverage of stage performance and game arts, and the motivation for using video essays. We address your questions on ShotBench, answer-set reliability, and the human CAA calculation below.
Thank you for suggesting this comparison. MuseBench's contribution lies in defining and evaluating intent-level audiovisual arts understanding across four artistic domains. Its question construction, answer formats, and analyses are organized around connecting observable creative choices to the functions they serve. ShotBench [1] is a closely related contribution, and a direct comparison makes this task distinction clearer.
ShotBench evaluates cinematic language across eight dimensions, including shot size, framing, camera angle, lens size, lighting, composition, and camera movement. MuseBench extends the evaluation to Cinematic Arts, Static Visual Arts, Stage Performing Arts, and Game Arts, covering 11 subdomains. Within these domains, it emphasizes how an artistic technique produces an effect and supports an expressive or functional purpose.
Table R1. Comparison with ShotBench.
For example, identifying a camera angle and explaining how that angle establishes a character's authority are closely connected tasks. MuseBench emphasizes the technique-to-function relation, and extends the same principle to lighting in stage performance, color relationships in visual art, and the communicative function of effects in games.
The mixed answer format is also useful for this target. A creative choice can support several valid analytical dimensions, which a multi-select question can represent explicitly. The submitted results show that this distinction changes model comparisons: InternVL3-78B improves over InternVL3-8B in Overall ACC (37.80% versus 33.07%) but has lower Multi EM (13.53% versus 20.30%). Thus, selecting one plausible interpretation and recovering the full supported set expose different aspects of model capability.
We have added the comparison in Section 2 (pp. 3–4) and Appendix C.1, Table 5 (lines 956–985). The appendix includes illustrative task pairs connecting attribute recognition to technique-to-function reasoning.