acceptodds
Under review as a conference paper at ICLR 2027

AdAlign-Bench: A Benchmark for Human-Aligned Advertisement Subjective Video Understanding

Abstract

Subjective video judgments reflect viewers' experiences and perceptions, raising the question of whether multimodal large language models (MLLMs) align with human evaluations. Existing subjective benchmarks commonly use aggregated human ratings, which measure population-level alignment but offer limited insight into viewer differences and conditions of model–human divergence. We introduce AdAlign, a benchmark centered on population-level subjective rating alignment, using individual responses, psychological traits, and information-processing-related ratings to examine variation in model-human correspondence that aggregate scores may obscure. AdAlign comprises annotations by 225 participants on 393 advertisement videos, covering creativity and multimodal perceptual quality alongside open-ended impressions and self-reported viewing experiences. Current MLLMs align more closely with human ratings of creativity than of perceptual quality, capturing broad rating trends while exhibiting discrepancies in rating agreement and absolute scores. Alignment declines in both higher-rated and higher-disagreement video subsets, more markedly in the former. Under generic viewer prompts, model ratings also exhibit trait-related patterns of correspondence with human judgments. An exploratory case study suggests that correspondence with human information-processing reports may support creativity alignment, while illustrating differences between human and model response-level correlation structures. Together, these findings highlight variation in subjective rating alignment that aggregate performance alone may obscure.

Primary area:
datasets and benchmarks
Keywords:
Subjective video understanding, MLLM-human alignment, Creativity evaluaiton, Multimodal perceptual quality evaluaiton

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_QGQHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the alignment of current multimodel large language models (MLLMs) with actual human evaluations on subjective video judgments. A new benchmark is introduced: Participants were asked to watch advertisement videos and fill out detailed questionaires on the video itself. Additionally, psychological traits and human information processing data on the participants has been gathered on a participant level. The gathered information has been compared to a selection of open and closed source VLMs. As a main result, the paper indicates that the current MLLMs align better towards the assessment of creativity than the perceptual quality.

My current recommendation is to reject the paper. My main concerns are centered around the facts that the paper is written in way which is difficult to read and understand, the narrow scope of the provided annotated dataset, and missing information on several critical aspects of the dataset creation and its limited evaluation. In my opintion, the paper is not ready for publication in its current form, and I would recommend a major revision and scope adjustment before resubmission.

Critical strengths
  1. If stated correctly in terms of scope, content and limitations: The mentioned dataset could still be valuable to a limited group of researchers.
Critical weaknesses
  1. Poor writing: The paper is generally difficult to read, to list a few examples:
    a) The introduction fails to introduce the problem in a simple way which is accessible for a broader audience. Many of the used concepts and differentiations are not explained sufficiently or at all (e.g. subjective vs. objective video understanding, implicit information understanding, etc. are just mentioned without any context).
    b) Many sentences are written in a way where the content is only understandable after multiple cycles of reading: e.g. Line 52-55: "Specifically, we examine alignment across video subsets differing in human disagreement and mean ratings, trait-related variation in model–human correspondence without supplying traits to models, and similarities and differences in the relationships among human and model information-processing-related ratings."
    The authors mention in the AI Use statemtent that generative AI tools were used for language editing, improving the clarity and organization of the manuscript etc. However, I would dare to hypothesize that exactly this approach lead to a poorly readable paper in this case.

  2. Participant sample and data: As pointed out in the limitation section, the participant sample is primarily in the age group of 20 - 30 years. A narrow occupational background, cultural context and social-media use is also indicated. The gathered dataset is hence very specific, and generalization to other age groups and demographics is highly uncertain, limiting the usability.

  3. Missing methodological information: E.g.
    a) Missing dataset information: Information on the gathered video dataset are missing entirely. Information such as on the content (what as advertisement about, its product category), length of ads, quantity of information conveyed etc. are missing and are not analyzed. This would be very relevant, also with regards to the human evaluation results. Further, Youtube videos are indicated in the main text, but not found in the provided vid_url.csv in the supplementary.
    b) Salient impressions are mentioned multiple times, an evaluation procedure is introduced, results are presented to check for alignment between humans and LLMs. However, the authors never introduce the concept nor what they understand under it in this paper's context. Further, the methodology for the LLMs and instructions for annotators to mark the impressions are not described either.

  4. Insufficient evaluation: One of the main motivations mentioned for this paper is the fact that current methods restrict their analysis on a population level, not providing insights into viewer differences. Exactly this aspects seems under evaluated. Almost all evaluations are on a population level with the exception of Figure 2 and even this analysis provides only limited insight.

Questions for authors

See weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer QGQH (1/4)

We sincerely thank you for your careful reading of our manuscript and for your candid, constructive feedback. Your comments helped us identify several places where our presentation, dataset descriptions, and analytical structure were not communicated clearly enough. Taking your suggestions seriously, we have thoroughly revised the manuscript to improve readability, added detailed video-dataset statistics, clarified the salient-impression task and instructions in the main text, and better connected our population-level and individual-level analyses. Our point-by-point responses and revisions are provided below.

Q1. Writing clarity and problem introduction

Thank you for pointing out where our writing and problem introduction were difficult to follow. We agree that introducing technical terms without sufficient context—coupled with overly dense sentence structures—made the paper harder to read than intended. In the revised manuscript, we carefully edited the text throughout to use clearer, more direct language. First, to make the problem immediately accessible to a broader audience, we now open the Introduction with an intuitive advertising scenario:

"In the advertising marketing scenario, audience reactions to a video campaign are rarely uniform. Some advertisements win broad praise, whereas others spark sharp controversy; some are remembered for a single catchy hook, while others leave viewers with diverse impressions depending on their personal tastes and decision-making styles. Much like the saying that 'there are a thousand Hamlets in a thousand people's eyes,' subjective responses to video advertisements vary across both videos and viewers. How well do current multimodal large language models (MLLMs) perform in such subjective evaluation settings? To explore this question, we collect real human audience evaluations and build a benchmark for subjective video understanding."

This scenario naturally distinguishes our task from traditional video benchmarks that rely on a few expert annotators. Objective content understanding focuses on verifiable video facts that people generally agree on (such as what objects appear or what actions occur), and implicit information understanding focuses on inferring hidden messages or narrative intent from those facts. In contrast, subjective video understanding means that the viewer is an essential part of the answer. Because different viewers bring their own tastes and perspectives, their subjective evaluations and salient impressions naturally vary.

We also truly appreciate your feedback on our sentence structure. Following your suggestion, we have broken long, dense sentences throughout the paper into shorter, clearly connected statements. For example, we rewrote the dense overview sentence you quoted in the Introduction into plain, accessible terms that explain how we move from overall population-level alignment to the patterns beneath it:

"In this work, we first evaluate population-level alignment between MLLMs and the audience as a whole, and then examine the underlying patterns that overall population-level alignment may obscure. Specifically, beyond comparing overall scores and salient impressions across the full dataset, we investigate three practical questions. First, does a model still align with human viewers when we look only at higher-rated advertisements or more controversial advertisements with higher viewer disagreement? Second, when a model is prompted to act as a generic ordinary viewer without knowing any viewer's background, how does the correspondence between model and human ratings vary across viewers with different psychological traits? Third, when humans rate an advertisement's creativity, multimodal perceptual quality, and information-processing-related experiences, do models reflect the same relationships among these ratings as humans do, and does jointly evaluating these experiences help or hurt model alignment?"

AuthorsOpenReview ↗

Response to Reviewer QGQH (2/4)

Q2. Participant scope and practical value of the benchmark

Thank you very much for raising this important point. We truly appreciate your valuable feedback, fully agree with your concern, and openly acknowledge this limitation. Because our study required strict in-person lab sessions, our recruitment was limited in practice, resulting in 225 participants aged 18-30 from a Chinese cultural background. As you rightly pointed out, people of different ages and cultures may view ads very differently, so our findings cannot be generalized to all age groups or cultures. Following your feedback, we have clearly stated the age and cultural background of our participants upfront in the Introduction and Appendix A (the Limitations section), and we strictly limit our conclusions to this group rather than overgeneralizing.

Within this boundary, we also hope to explain why this dataset still has practical value:

  1. Relevance of young-adult viewers and precedents in prior studies: Our 18-30 sample includes both university students and young working professionals. Young adults aged 18-30 are an active and important audience segment in digital marketing and online consumer research (e.g., Djafarova & Rushworth, 2017). Moreover, influential laboratory studies on commercial video advertisements have similarly focused on young-adult cohorts—for example, Dmochowski et al. (2014) in Nature Communications studied real commercial ad ratings and preferences using young-adult cohorts (aged 20-29 and 18-22), and Teixeira et al. (2012) in Journal of Marketing Research examined internet video ad engagement using a sample with a mean age of 22. While these precedents do not make our sample representative of all consumers, they support that studying how MLLMs align with this young-adult group still provides a useful starting point for subjective video ad evaluation.
  2. Psychological and consumer diversity within the group: Even within the same age and cultural group, viewers do not react in the same way. Our participants show wide variation across four well-established psychological and consumer scales (Big Five personality, Regulatory Focus, Emotional Contagion, and Consumer Decision-Making Styles), which prior consumer and advertising studies have linked to how people respond to ads.

We take your feedback seriously and plan to expand AdAlign to broader age groups, occupations, and cultural backgrounds in our future work.

References:

  • [1] Djafarova, Elmira, and Chloe Rushworth. "Exploring the credibility of online celebrities' Instagram profiles in influencing the purchase decisions of young female users." Computers in human behavior 68 (2017): 1-7.
  • [2] Dmochowski, J. P., Bezdek, M. A., Abelson, B. P., Johnson, J. S., Schumacher, E. H., & Parra, L. C. (2014). Audience preferences are predicted by temporal reliability of neural processing. Nature communications, 5(1), 4567.
  • [3] Teixeira, T., Wedel, M., & Pieters, R. (2012). Emotion-induced engagement in internet video advertisements. Journal of marketing research, 49(2), 144-159.
AuthorsOpenReview ↗

Response to Reviewer QGQH (3/4)

Q3. Dataset statistics and salient-impression evaluation

Q3(a) Video categories, duration, information quantity, and YouTube sources

Thank you for pointing out the missing information on the video dataset. We agree that details on advertisement content, product categories, and video duration are essential context for interpreting the human evaluation results. Following your feedback, we have added the full product-category and duration distributions for all 393 videos (Tables 1 and 2 below) to Appendix D and summarized them directly in Section 3.2. As shown in Table 1, the dataset covers 12 industry categories, led by Food & Beverage (25.45%), 3C Digital Electronics (16.54%), and Apparel & Underwear (14.76%). As shown in Table 2, video duration ranges from 7.00 s to 232.34 s (mean 54.53 s, median 47.77 s), with most ads falling between 30 and 60 seconds (42.75%).

Table 1. Product-category distribution across the 393 videos.

RankIndustry categoryVideo countPercentageCumulative percentage
1Food & Beverage10025.45%25.45%
23C Digital Electronics6516.54%41.98%
3Apparel & Underwear5814.76%56.74%
4Home Appliances4511.45%68.19%
5Beauty & Skincare328.14%76.34%
6Home & Textiles328.14%84.48%
7Daily Necessities174.33%88.80%
8Shoes & Bags164.07%92.88%
9Local Services61.53%94.40%
10Pet Supplies51.27%95.67%
11Toys & Trendy Playthings41.02%96.69%
12Others (<1% each)133.31%100.00%

Table 2. Video-duration distribution and summary statistics (N=393, total 357.19 min).

Duration rangeVideo countPercentageCumulative percentageSummary statisticValue (s)
<15 s174.33%4.33%Minimum / Maximum7.00 / 232.34
15-30 s7017.81%22.14%Mean / SD54.53 / 34.23
30-60 s16842.75%64.89%25th / 50th (Median) / 75th pct30.05 / 47.77 / 65.99
1-1.5 min7719.59%84.48%5th / 95th pct15.03 / 125.90
1.5-2 min399.92%94.40%——
>2 min225.60%100.00%——

Thank you also for your suggestion regarding the "amount of information conveyed." We appreciate this thoughtful point. Unlike duration or product category, an ad video does not have a single objective number for how much information it carries, since different viewers often feel differently about how informative an ad is. Instead of assigning a fixed objective label, we included two rating items—Visual Complexity and Desire to Postpone Closure (Table 1 in the manuscripts and Appendix B.1), so that viewers could directly rate how rich and informative each video felt to them.

YouTube links: Thank you for catching this detail. Because advertising is cross-border, we used YouTube as one of our main platforms to find ad videos. However, many original YouTube uploads did not have the Chinese subtitles needed for our participants (Section 3.2). To make sure participants could clearly understand every ad, we downloaded the corresponding Chinese-subtitled versions from Chinese video platforms for the lab sessions and initially recorded those download links in our metadata file. Following your comment, we have now added the original YouTube links alongside the subtitled video links in our metadata and explained this clearly in Section 3.2 and Appendix D.

Reviewer_bx4HOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AdAlign, a benchmark for comparing MLLM judgments with human subjective evaluations of short advertisement videos. It contains 393 videos annotated by 225 participants, with approximately 13 raters per video on average, and retains individual content ratings, open-ended impressions, self-reported information-processing experiences, and psychological-trait measures. The evaluation combines Kendall's tau, Pearson correlation, linearly weighted Cohen's kappa, and MAE with salient-impression extraction, subset analyses, trait moderation, and an exploratory prompting case study. The main descriptive findings are stronger correspondence on creativity than on perceptual quality, and lower ranking correspondence in higher-rated and higher-disagreement subsets. These trends depend on the metric: Appendix F shows that MAE improves in the higher-rated creativity and visual-quality subsets even as ranking correlation falls.

Critical strengths
  1. A useful subjective-video resource that retains individual-level information. AdAlign combines content judgments with open impressions, viewing-experience reports, and psychological-trait measures, rather than retaining only aggregate labels. This supports the study of variation across viewers as well as population-level evaluation. This combination is a substantive resource contribution within the stated advertising and participant context.
  2. The evaluation separates distinct meanings of alignment and reports informative boundary conditions. Table 2 shows that strong rank or linear correlation can coexist with much weaker rating-category agreement. Figure, the trait analysis, and the three impression-evaluation pipelines provide complementary views. Reporting the pipelines separately is useful: the unweighted @5 leader is consistent across them, while @3 is sensitive to the evaluator. These distinctions make the benchmark more informative than a single overall score.
Critical weaknesses
  1. The high-rating subset analysis needs a more precise interpretation of the ranking decline. Section 4.3 defines subsets using human means or standard deviations and evaluates them against the original video-level MOS. Lower Kendall's tau within these subsets is an informative conditional observation. However, smaller human-score gaps can make the same model or reference error more consequential for ranking, and finite-rater estimates can affect subset membership. The observed decline alone does not distinguish these effects from an additional limitation specific to highly rated videos.
  2. The precision of the MOS-based comparisons and close model-order claims is not quantified. Section 3.2 reports approximately 13 raters per video on average, and Section 4.1 averages three model responses at temperature 1. The participant count alone does not imply an unreliable MOS, and genuine viewer disagreement should not be treated as annotation error. Nevertheless, the manuscript does not report uncertainty or resampling stability for the MOS-based reference, the key metric differences, or the subset curves.
Questions for authors
  1. How stable are the high-rating and high-disagreement subsets and their alignment curves under appropriate rater resampling? Please report subset score ranges and reference precision, and clarify how the decline in tau should be interpreted alongside the improved high-rating MAE. If an additional high-end model limitation is intended, what baseline distinguishes it from narrower score gaps and uncertainty in the MOS reference?
  2. For the key close comparisons in Table 2, what are the paired metric-difference intervals or rank-stability results, and how variable are the three responses per model-video prompt? Please distinguish comparisons conditional on the observed human reference from sensitivity to participant sampling; not every descriptive table maximum requires a significance claim.
  3. I suggest citing the paper "Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding" and analyzing the connection between the two works.
6 comments
AuthorsOpenReview ↗

Response to Reviewer bx4H (1/4)

Thank you very much for your thorough review and constructive methodological feedback. Your suggestions on rater resampling, paired model-comparison uncertainty, and related work have greatly improved the statistical rigor and clarity of our paper. We have incorporated all corresponding analyses and revisions into the updated manuscript PDF, and our point-by-point responses are provided below.

W1 and Q1. Rater resampling, reference precision, and the τ\tau–MAE pattern

We appreciate this insightful methodological question. Following your advice, we added rater-resampling, subset-distribution, and error-decomposition analyses in Tables 1 and 2 to distinguish score-range changes, human-reference behavior, and model alignment across the 100%, 80%, and 60% subsets. Creativity averages its five subdimensions, and Visual quality averages its six constituent subdimensions.

1. Subset distributions, human split-half baseline, and rater-bootstrap stability

Each video's composite MOS averages its item-wise human means. Higher-rated selection ranks composite MOS; higher-disagreement selection ranks mean item-wise rater SD. We run 500 random split-half comparisons on the observed subsets by splitting each video's raters into two disjoint halves, and 1,000 within-video bootstrap replicates by resampling nn complete rater vectors with replacement per video, recomputing MOS and SD, and reselecting subsets while holding the video pool and model scores fixed (Table 1).

Table 1. Subset distributions, human split-half baseline, and rater-bootstrap stability.

DimensionSelectionRetained (nn)Mean MOS / rangeBetween-video SDWithin-video SD / SEMHuman split-half τ\tau [2.5th, 97.5th pct]Human split-half MAEGemini-3.1-Pro Boot τ\tau [95% interval]Gemini-3.1-Pro Boot MAE [95% interval]
CreativityFull set100% (393)4.811 / 4.9631.2201.282 / 0.3480.6332 [0.6159, 0.6506]0.56750.5696 [0.5574, 0.5819]0.8518 [0.8319, 0.8725]
CreativityHigher-rated80% (314)5.314 / 3.0480.7371.253 / 0.3400.4990 [0.4718, 0.5273]0.55570.4638 [0.4442, 0.4840]0.8260 [0.8054, 0.8488]
CreativityHigher-rated60% (235)5.661 / 1.8630.4601.169 / 0.3170.3592 [0.3225, 0.3979]0.51580.4034 [0.3759, 0.4331]0.7536 [0.7262, 0.7808]
CreativityHigher-disag.80% (314)4.590 / 4.6321.1181.377 / 0.3730.5940 [0.5702, 0.6176]0.60670.5535 [0.5341, 0.5735]0.8851 [0.8558, 0.9135]
CreativityHigher-disag.60% (235)4.394 / 4.1571.0451.463 / 0.3960.5681 [0.5402, 0.5948]0.64460.5391 [0.5110, 0.5666]0.9200 [0.8809, 0.9582]
VisualFull set100% (393)4.981 / 4.8461.0261.251 / 0.3390.5748 [0.5542, 0.5941]0.55210.5014 [0.4863, 0.5168]0.9598 [0.9362, 0.9844]
VisualHigher-rated80% (314)5.407 / 2.7310.5911.197 / 0.3250.4174 [0.3869, 0.4465]0.53020.3709 [0.3466, 0.3948]0.8090 [0.7836, 0.8336]
VisualHigher-rated60% (235)5.689 / 1.8140.3601.118 / 0.3040.2408 [0.2004, 0.2827]0.49570.2854 [0.2483, 0.3224]0.6753 [0.6473, 0.7022]
VisualHigher-disag.80% (314)4.754 / 4.3630.9911.339 / 0.3630.5520 [0.5273, 0.5764]0.59050.4949 [0.4723, 0.5163]1.0365 [1.0054, 1.0674]
VisualHigher-disag.60% (235)4.548 / 4.2080.9511.424 / 0.3860.5323 [0.5003, 0.5608]0.63050.4833 [0.4558, 0.5111]1.1120 [1.0698, 1.1553]
  1. Rater-bootstrap stability: At 60%, mean subset overlap with the original selection is ~95% for higher-rated subsets and ~85% for higher-disagreement subsets. Across all 13 models in both dimensions, paired 60%-minus-100% bootstrap intervals are strictly negative for both τ\tau and MAE under higher-rated selection. Under higher-disagreement selection, τ\tau intervals are below zero for 9/13 Creativity and 4/13 Visual models, whereas MAE intervals are above zero for 11/13 and 13/13 models, showing that absolute error increases more consistently than rank correlation drops.
  2. Score range and reference precision: From 100% to the top 60% higher-rated subset, MOS ranges and between-video SDs shrink by over 62% while item-wise SEM decreases slightly. Video scores become much closer relative to the 0.30-0.35 SEM without noisier item means. Higher-disagreement selection instead increases within-video rater SD and SEM while retaining most of the between-video SD.
  3. Human split-half reference: At the top 60% higher-rated subset, human split-half τ\tau drops sharply while human MAE improves. On higher-disagreement subsets, human τ\tau drops only mildly while MAE increases. Although split-half τ\tau compares two half-sized panels rather than a model against full-panel MOS and is not a strict ceiling, it confirms that the human reference follows the same directional τ\tau–MAE trade-off.
AuthorsOpenReview ↗

Response to Reviewer bx4H (2/4)

2. Why τ\tau decreases while MAE improves on higher-rated videos

To examine why model MAE improves from lower-rated to higher-rated videos across all 13 models, we compared the full set with the bottom 50% (197 videos) and top 50% (196 videos) MOS halves, and decomposed each video-item error e=model score−human MOSe = \text{model score} - \text{human MOS} into its overestimation contribution P=mean(max⁡(e,0))P = \text{mean}(\max(e, 0)) and underestimation contribution N=mean(max⁡(−e,0))N = \text{mean}(\max(-e, 0)), where MAE=P+N\text{MAE} = P + N and signed bias=P−N\text{signed bias} = P - N (Table 2).

Table 2. Signed bias, MAE, and model-score SD across the full set and bottom / top 50% MOS halves.

DimensionModelSigned bias (100% | bottom / top 50%)MAE (100% | bottom / top 50%)Model SD across videos (bottom / top 50%)
CreativityGemini-3.1-Pro-0.295 | -0.271 / -0.3200.809 | 0.951 / 0.6661.492 / 1.015
CreativityGemini-2.5-Pro+0.229 | +0.174 / +0.2830.979 | 1.182 / 0.7751.787 / 1.035
CreativityQwen3-VL-32B-0.039 | +0.347 / -0.4280.784 | 0.857 / 0.7111.107 / 0.853
CreativityGPT-5.4+0.568 | +0.941 / +0.1930.806 | 1.061 / 0.5501.023 / 0.669
CreativityQwen3-Omni-Flash+1.069 | +1.638 / +0.4981.171 | 1.673 / 0.6671.033 / 0.588
Creativity13-Model Range-0.295 to +1.069 | -0.271 to +1.638 / -0.428 to +0.4980.726-1.171 | 0.857-1.673 / 0.524-0.7941.023-1.787 / 0.588-1.035
VisualGemini-3.1-Pro+0.806 | +1.135 / +0.4750.933 | 1.256 / 0.6081.098 / 0.584
VisualGPT-5.4+0.978 | +1.465 / +0.4891.048 | 1.478 / 0.6150.654 / 0.495
VisualQwen3-Omni-Flash+1.606 | +2.208 / +1.0011.617 | 2.214 / 1.0180.636 / 0.289
Visual13-Model Range+0.806 to +1.606 | +1.135 to +2.208 / +0.401 to +1.0010.933-1.617 | 1.256-2.214 / 0.608-1.0180.636-1.205 / 0.289-0.607

At 100%, all 13 models on Visual and 11 of 13 on Creativity have positive signed bias. As human MOS rises from the bottom half to the top half, it moves closer to the models' higher score levels, and model score SD across videos also narrows. For all 13 Visual models and 10 of 13 Creativity models, both the overestimation contribution PP and absolute net bias decrease, directly lowering MAE. Even for the three Creativity models whose absolute net bias is slightly larger in the top half (Gemini-3.1-Pro, Gemini-2.5-Pro, and Qwen3-VL-32B), decomposing MAE=P+N\text{MAE} = P + N shows that their overestimation contribution PP drops sharply from the bottom to the top half, which outweighs any change in NN and reduces total MAE. Together with compressed between-video human MOS gaps, closer absolute scores naturally coexist with lower rank correlation.

3. Revised claims and subset-level analysis

Thank you very much for this insightful suggestion. Your question inspired us to further explore: under comparable score gaps and human reference uncertainty, to what extent does rank correspondence remain stable? This led to the following findings and revisions in Section 4.3:

  1. Reversal of the high- vs. low-rated comparison when score spread is constrained to the same SD: First, as shown in Table 1, when between-video MOS SD shrinks by over 62% in the top 60% subset, the human split-half τ\tau drops sharply alongside model-to-human τ\tau, showing that narrower score gaps naturally lower rank correlation even for human panels. Second, to directly verify whether models actually rank higher-rated videos worse than lower-rated videos once score spread is controlled, we further split the dataset into the top-half (high-rated) and bottom-half (low-rated) pools and sampled 40-video subsets from each pool (Section 4.3 and Appendix). If we sample randomly without constraining the score spread of the two pools, high-rated subsets have a much narrower natural MOS range and show lower τ\tau than low-rated subsets across all 13 models (mean Δτ=τhigh−τlow\Delta\tau = \tau_{\text{high}} - \tau_{\text{low}} is -0.141 on Creativity and -0.131 on Visual). However, when we constrain the high-rated and low-rated subsets to have the exact same item-wise human MOS SD (tested across Narrow, Medium, and Wide SD levels), the conclusion reverses: Δτ\Delta\tau turns positive across both dimensions (13-model mean +0.022 / +0.037 / +0.064 on Creativity and +0.056 / +0.074 / +0.087 on Visual), showing higher rank correspondence on higher-rated videos than on lower-rated ones of the same spread.
  2. Therefore, we revised our conclusions in Section 4.3: Because raw τ\tau values are not directly comparable across subsets with different score spreads, the apparent drop on narrowed top subsets reflects the much narrower score range of those subsets rather than a problem with model capability. Following your feedback, we revised Section 4.3 accordingly: while full-set and narrowed-subset τ\tau values cannot be compared directly without accounting for score range, comparing different models within the same subset remains valid and informative.
AuthorsOpenReview ↗

Response to Reviewer bx4H (3/4)

W2 and Q2. Response variability and paired model-comparison uncertainty

Thank you for highlighting the importance of quantifying response variability and model-comparison uncertainty. Following your suggestion, we report three-run repeatability diagnostics in Table 3 and paired bootstrap confidence intervals in Table 4.

1. Repeatability across three responses

Table 3 reports three-run response variability on complete three-run cases (temperature 1, top-p 0.9, thinking enabled). Score SD is the within-video-item sample SD across the three runs, averaged over videos and items. τ\tau spread is each item's max-minus-min single-run Kendall's τb\tau_b against MOS, averaged across items. Here, a single-run τ\tau is computed for each item across videos using scores from one run, rather than scores averaged over the three runs. Across all models, within-item 3-run Score SD is much smaller than human within-video rater SD, and single-run τ\tau spread is small across runs.

Table 3. Three-response variability across models.

DimensionDiagnosticGPT-5.4Gemini-3.1-ProQwen3.5-PlusGemini-3-FlashGemini-2.5-Pro13-model rangeHuman rater SD
CreativityScore SD0.16030.36090.31540.33060.45260.1603-0.55501.2818
Creativityτ\tau spread (max⁡−min⁡\max-\min)0.01700.01560.02440.01730.02850.0156-0.0444—
VisualScore SD0.10150.30010.23860.25170.35650.1015-0.51581.2507
Visualτ\tau spread (max⁡−min⁡\max-\min)0.01660.02370.02550.04660.05000.0166-0.0759—

2. Paired comparisons and rank stability

We sincerely appreciate your advice to distinguish comparisons conditional on the observed human reference from sensitivity to participant sampling, and to avoid over-claiming significance for every close table maximum. Following your guidance, we evaluate model-comparison uncertainty using 1,000 paired bootstrap replicates under two settings: Setting A resamples the 393 videos with replacement while holding observed MOS and model 3-run means fixed; Setting B uses the same video draw and additionally resamples each video's nn human raters with replacement. Because Gemini-3.1-Pro achieves the highest macro-averaged Kendall's τb\tau_b across both the Creativity and Visual dimensions among all 13 evaluated models, we use it as the reference model to examine paired differences against the next closest competitors. Table 4 reports the observed macro Kendall's τb\tau_b difference (Δτ=Gemini-3.1-Pro−comparator\Delta\tau = \text{Gemini-3.1-Pro} - \text{comparator}) and 95% percentile intervals.

Table 4. Paired macro Kendall's τb\tau_b differences against Gemini-3.1-Pro.

DimensionComparatorObserved Δτ\Delta\tauSetting A: 95% CISetting B: 95% CICI excludes 0 (A / B)
CreativityGPT-5.4+0.0240[-0.0029, +0.0492][-0.0074, +0.0486]No / No
CreativityGemini-2.5-Pro+0.0308[+0.0084, +0.0524][-0.0002, +0.0519]Yes / No
CreativityGemini-3-Flash+0.0544[+0.0352, +0.0731][+0.0263, +0.0702]Yes / Yes
CreativityQwen3.5-Plus+0.0653[+0.0394, +0.0893][+0.0307, +0.0841]Yes / Yes
VisualGPT-5.4+0.0254[-0.0029, +0.0540][-0.0085, +0.0541]No / No
VisualGemini-2.5-Pro+0.0526[+0.0250, +0.0783][+0.0181, +0.0737]Yes / Yes
VisualGemini-3-Flash+0.0452[+0.0187, +0.0731][+0.0109, +0.0713]Yes / Yes
VisualQwen3.5-Plus+0.0632[+0.0322, +0.0947][+0.0261, +0.0925]Yes / Yes

Gemini-3.1-Pro's gains over Gemini-3-Flash and Qwen3.5-Plus exclude zero under both settings in both dimensions, while its smaller margins over GPT-5.4 include zero. Across all 1,000 replicates, Gemini-3.1-Pro ranks first among all 13 models in 95.5%/90.0% (A/B) of replicates on Creativity and 96.2%/90.8% on Visual. We have added Tables 3 and 4 to the Appendix and clarified in Section 4.2 that bold and underlined entries denote observed best and second-best scores rather than statistically separated winners on every metric.

Reviewer_av7LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdAlign-Bench introduces a benchmark for evaluating how multimodal large language models align with human subjective judgments of advertisement videos. It collects annotations from 225 participants on 393 videos, covering creativity, multimodal perceptual quality, open-ended impressions, self-reported viewing experiences, and psychological traits. Its main contributions are the annotated dataset and an evaluation framework combining population-level rating alignment with analyses of viewer differences and video-dependent discrepancies. The authors evaluate 13 model configurations using ranking correlation, rating agreement, absolute error, and impression matching through three LLM-based evaluation pipelines. They further examine high-rating and high-disagreement subsets, trait-related interactions, and information-processing prompts. Reported results indicate stronger alignment on creativity than perceptual quality, trait-related variation in model–human correspondence, and model-dependent effects of expanded prompts.

Critical strengths

Strengths

  1. The paper addresses a relevant question: whether MLLMs agree with human subjective judgments, including differences that average ratings may hide.

  2. The dataset combines ratings from 225 participants on 393 advertisements with individual traits, viewing experiences, and open-ended impressions. These annotations support richer analysis than aggregate scores alone.

  3. The evaluation covers 13 model configurations and distinguishes ranking agreement from absolute scoring errors. The authors also provide detailed annotation items, prompts, and supplementary results.

Critical weaknesses

Weaknesses

  1. The main findings need a human baseline. Lower correlation on highly rated or high-disagreement videos may partly reflect a narrower score range or less reliable human averages. Indeed, the appendix shows that MAE often decreases on highly rated subsets (Figs. 16 and 18). Comparing independent groups of human raters would help establish whether models show an additional limitation.

  2. The open-ended evaluation lacks independent human validation. Each LLM-based pipeline both constructs reference impressions and scores model answers against them. Similar results across three pipelines provide useful evidence, but do not establish whether the references and matches faithfully preserve human responses. A small human audit of both stages would strengthen confidence in this benchmark component.

Questions for authors
  1. Human baseline — requested analysis using existing data. How does agreement between independent groups of human raters change on the highly rated and high-disagreement subsets? Could you provide a split-half human baseline and assess whether the model correlation declines exceed what can be explained by human rating uncertainty and the narrower score range?

  2. Open-ended evaluation — clarification and targeted validation. Were the LLM-generated reference impressions and matching decisions independently checked by humans? If so, please report the audit size, agreement, and main errors. If not, could you provide a small human audit of both stages and assess whether the model comparisons remain consistent?

6 comments
AuthorsOpenReview ↗

Response to Reviewer av7L (1/2)

Thank you very much for your encouraging evaluation of our work and your constructive methodological suggestions. Your comments on adding a human split-half baseline and conducting a human spot-check audit have greatly strengthened our diagnostic analyses and evaluation rigor. We have incorporated all corresponding experiments and revisions into the updated manuscript PDF, and our point-by-point responses are provided below.

W1 and Q1(1). Human split-half baseline and why τ\tau and MAE move in opposite directions across subsets

We appreciate this insightful suggestion. Following your advice, we added a 500-iteration human split-half baseline alongside subset distribution diagnostics across both the higher-rated and higher-disagreement subsets (100%, 80%, and 60% of videos). Creativity averages its five subdimensions, and Visual quality averages its six subdimensions. For each video, we randomly split its raters into two disjoint halves, compute item-wise Kendall's τ\tau and MAE between the two human groups across videos, and average across items over 500 splits (Table 1).

Table 1. Subset rating distributions, human split-half baseline, and model alignment (Gemini-3.1-Pro).

DimensionSelectionRetained (n)Mean MOSMOS rangeBetween-video SDWithin-video SDItem SEMHuman split-half tau [2.5th, 97.5th pct]Human split-half MAEGemini-3.1-Pro tau / MAE
CreativityFull set100% (393)4.8114.9631.2201.2820.3480.633 [0.616, 0.651]0.5680.602 / 0.810
CreativityHigher-rated80% (314)5.3143.0480.7371.2530.3400.499 [0.472, 0.527]0.5560.501 / 0.780
CreativityHigher-rated60% (235)5.6611.8630.4601.1690.3170.359 [0.323, 0.398]0.5160.459 / 0.698
CreativityHigher-disag.80% (314)4.5904.6321.1181.3770.3730.594 [0.570, 0.618]0.6070.564 / 0.854
CreativityHigher-disag.60% (235)4.3944.1571.0451.4630.3960.568 [0.540, 0.595]0.6450.565 / 0.880
VisualFull set100% (393)4.9814.8461.0261.2510.3390.575 [0.554, 0.594]0.5520.534 / 0.932
VisualHigher-rated80% (314)5.4072.7310.5911.1970.3250.417 [0.387, 0.447]0.5300.405 / 0.794
VisualHigher-rated60% (235)5.6891.8140.3601.1180.3040.241 [0.200, 0.283]0.4960.312 / 0.672
VisualHigher-disag.80% (314)4.7544.3630.9911.3390.3630.552 [0.527, 0.576]0.5910.540 / 1.016
VisualHigher-disag.60% (235)4.5484.2080.9511.4240.3860.532 [0.500, 0.561]0.6310.515 / 1.095

Table 1 explains why τ\tau and MAE move in opposite directions in the two subset analyses:

  1. Higher-rated subsets (lower τ\tau, better MAE): From 100% to 60%, Between-video SD drops by over 60%. Because score differences across videos shrink toward the 0.30-0.35 SEM of the human reference, pairwise ranking becomes much harder for both humans and models, so human split-half τ\tau and model τ\tau drop sharply together. At the same time, MAE improves across all 13 models. For the human split-half baseline, Within-video SD is slightly smaller on higher-rated videos, reducing split-half MAE. For models, most MLLMs tend to assign relatively high scores (13/13 models have positive net bias on Visual and 11/13 on Creativity on the full set). As human MOS rises in the higher-rated subsets, overestimation errors decrease substantially across all 13 models and model score dispersion narrows, yielding lower MAE despite compressed inter-video rank differences.
  2. Higher-disagreement subsets (mild τ\tau decline, worse MAE): Filtering for high rater disagreement retains most of the score spread across videos, so relative ordering remains clear and both human split-half τ\tau and model τ\tau decline only mildly. Meanwhile, as within-video rater SD and item SEM increase on high-disagreement videos, human split-half MAE increases and it also becomes harder for models to match the human MOS in absolute terms, leading to larger model MAE.

Note that human split-half τ\tau compares two half-sized rater panels whereas model τ\tau is evaluated against the full-panel MOS, so split-half τ\tau serves as a diagnostic reference rather than a strict upper bound. Together, these results show that the subset trends reflect conditional boundaries of population-level alignment rather than an isolated drop in model capability at the high end. We have added Table 1 to Appendix F and revised Section 4.3 accordingly.

AuthorsOpenReview ↗

Response to Reviewer av7L (2/2)

Q1(2). Validation of the salient-impression pipeline via a blinded 3-annotator human spot-check audit

Thank you for this valuable suggestion. Beyond reporting results across three independent judge families (GPT-5.4, Gemini-3.5-Flash, and DeepSeek-V4-Pro) in the main text (Table 3) to mitigate single-judge bias, we conducted a blinded, full-overlap 3-annotator human audit on 36 stratified videos across all 3 anonymized judge pipelines and 3 candidate models, covering 687 Stage-1 reference points and 1,620 Stage-2 matching judgments (Table 2):

  1. Stage-1 Aggregation Quality: Annotators evaluated three binary criteria. Both Content Faithfulness (whether the reference point and its grouped human responses express the same content) and Sentiment Consistency (whether the reference point reflects the human sentiment polarity) reached 100.0% accuracy, because our pipeline clusters combinable raw responses, computes their positive/negative ratio first, and summarizes strictly from them. Information Validity (whether the point contains substantive content rather than brief, vague remarks such as "content") reached 96.70% accuracy. The 3.30% invalid points arise because instructing the model to preserve minority human impressions occasionally leads it to turn a brief, vague remark into a standalone point; since such points rarely match in Stage 2, they do not materially affect model evaluation.
  2. Stage-2 Matching Quality: Positive matches (score = 0.5 or 1.0) achieved 95.01% accuracy, where errors are mainly borderline 0.5 matches that should be 0, while confusion between 0.5 and 1.0 is rare. Zero matches (score = 0) achieved 92.43% accuracy, where errors mainly consist of unmatched items (0) that partially match (0.5) a reference point.

Table 2. Summary of the 3-annotator blinded human audit (36 stratified videos, 687 Stage-1 points, 1,620 Stage-2 judgments).

Audit stage & metricUnitsAnnotator 1 / 2 / 3 acc.Overall acc. (error)3-way agreementFleiss' kappaError by pipeline (dpsk / gemini / gpt)
Stage 1: Content Faithfulness687100% / 100% / 100%100.00% (0.00%)100.0%—0.0% / 0.0% / 0.0%
Stage 1: Sentiment Consistency687100% / 100% / 100%100.00% (0.00%)100.0%—0.0% / 0.0% / 0.0%
Stage 1: Information Validity68796.1% / 96.9% / 97.1%96.70% (3.30%)98.1%0.8023.2% / 5.1% / 1.6%
Stage 2: Positive Match (0.5/1)84995.6% / 95.1% / 94.4%95.01% (4.99%)91.6%0.4123.9% / 4.2% / 6.5%
Stage 2: Zero Match (0)77190.9% / 92.6% / 93.8%92.43% (7.57%)96.4%0.8279.7% / 6.2% / 6.4%

We have added these human audit results and details to Appendix E.7 (Table 19) in the revised manuscript.

Dear Reviewer av7L,

Thank you again for your time and effort in reviewing our paper. Your comments have been very helpful in improving the quality of our work. As the author-reviewer discussion period will end in about three days, we would like to gently check whether you have had a chance to read our response and whether your concerns have been addressed. Please let us know if you have any further questions.

Thank you again for your time and help.

Best regards,

The Authors

Reviewer_3GXAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the alignment between subjective ratings from multimodal large language models and human judgments of advertisement videos. AdAlign collects annotations from 225 participants on 393 videos, covering creativity, multimodal perceptual quality, self-reported viewing experiences, and open-ended impressions, together with psychological-trait measures. The main evaluation uses video-level mean human ratings as the reference. Individual traits are used to analyze model–human correspondence across viewers and are not supplied as model inputs. The results show generally closer correspondence for creativity than for perceptual quality, and that ranking correlation does not guarantee agreement in absolute scores. The paper also analyzes higher-rated and higher-disagreement subsets and explores changes after adding information-processing items.

Critical strengths

1.The benchmark is well designed. Rather than releasing only aggregated ratings, it retains individual viewer ratings, psychological trait measures, and open-ended impressions, which makes it usable for studying viewer-level variation rather than population averages alone.

2.The empirical evaluation is broad. It covers thirteen open and closed source MLLMs under several alignment metrics, including rank correlation, weighted category agreement, and absolute error, and additionally reports salient-impression extraction under three separate LLM evaluator pipelines. The results are informative in showing that strong ranking correspondence can coexist with scores that are systematically higher than human ratings, particularly on perceptual-quality items.

Critical weaknesses

The decline in rank correlation on higher-rated subsets is worth reporting, but the present results cannot separate a limitation in models' fine-grained discrimination at the high end from the effects of the selection itself. Section 4.3 describes this decline as a boundary of current models' subjective rating alignment, namely that their ability to track human judgments across a broad rating range does not consistently extend to fine-grained differentiation among highly rated videos. Selecting by the mean of human ratings changes the rating range of the retained subset and the proportion of closely spaced video pairs, yet the paper reports no corresponding analysis of within-subset rating range or variance, nor of the stability of the human reference. The existing annotations are sufficient for such an addition: reporting the within-subset rating range and the stability of the human reference, or limiting the claim to the observed drop in subset rank correlation rather than a boundary of model capability, would address this concern.

Questions for authors

1.Section 4.1 averages human and model ratings before reporting linear-weighted κ, whereas the description in Table10 refers to the original rating categories. Please explain how non-integer means are mapped to the categories used for κ, and how cases are handled when some of the three model outputs are null.

2.Salient-impression evaluation uses three pipelines that each construct their own reference set and restricts credit for duplicate matches, so this result does not rest on a single judge. Has the sequence from raw responses to reference points to matching and sentiment assignment been independently spot-checked by humans? Please also give the normalization used for support-weighted Precision@K and Recall@K.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3GXA (1/4)

Thank you very much for your constructive review and insightful methodological feedback. Your comments have been invaluable in helping us deepen our diagnostic analyses, refine our claims in Section 4.3, and strengthen the validation of our evaluation pipeline. We have incorporated all corresponding experiments and revisions into the updated manuscript PDF, and our point-by-point responses are provided below.

W1. Interpretation of rank correlation on higher-rated subsets

We completely agree with your observation: a drop in subset rank correlation does not, on its own, isolate a failure in models' fine-grained capability at the high end, because filtering by mean rating compresses the between-video score range. Prompted by your comment, we conducted (1) subset distribution and human split-half analyses on the original subsets, and (2) a new SD-matched high-vs.-low comparison.

1. Why τ\tau drops on the original top subsets: range compression and human reference bounds

For Creativity (5 subdimensions) and Visual quality (6 subdimensions), retaining the top 100%, 80%, and 60% of videos by mean human MOS sharply compresses between-video dispersion by over 60%, while within-video rater uncertainty (SEM) stays around 0.30-0.35. Because score gaps between videos shrink toward the noise level of the human reference, pairwise ranking becomes harder for both humans and models, so human split-half Kendall's τ\tau drops alongside model-human τ\tau (Table 1). A 1,000-iteration within-video rater bootstrap further confirms that the paired drop in τ\tau from 100% to 60% is strictly negative across all 13 models.

Table 1. Distributional compression and human split-half τ\tau on the original subsets.

DimensionRetained (n)Mean MOSMOS rangeBetween-video SDMean item SEMHuman split-half tau [2.5th, 97.5th pct]Gemini-3.1-Pro tau
Creativity100% (393)4.8114.9631.2200.3480.633 [0.616, 0.651]0.602
Creativity80% (314)5.3143.0480.7370.3400.499 [0.472, 0.527]0.501
Creativity60% (235)5.6611.8630.4600.3170.359 [0.323, 0.398]0.459
Visual100% (393)4.9814.8461.0260.3390.575 [0.554, 0.594]0.534
Visual80% (314)5.4072.7310.5910.3250.417 [0.387, 0.447]0.405
Visual60% (235)5.6891.8140.3600.3040.241 [0.200, 0.283]0.312
AuthorsOpenReview ↗

Response to Reviewer 3GXA (2/4)

2. New SD-matched analysis: comparing high- vs. low-rated videos under equal dispersion

Your comment inspired us to compare how models discriminate among human-perceived good (high-rated) versus poor (low-rated) advertisements while controlling for their rating spread. Specifically, we split the 393 videos into a low-rating pool (bottom 196) and a high-rating pool (top 196) and repeatedly sampled 40-video subsets (averaged over 500 draws). As shown in the "Unconstr." columns of Table 2, if we sample from the two pools without restricting human MOS SD, the high-minus-low rank correlation difference (Δτ=τhigh−τlow\Delta\tau = \tau_{\text{high}} - \tau_{\text{low}}) is negative across all 13 models (mean -0.141 on Creativity and -0.131 on Visual). Crucially, as your comment helped us realize, this unconstrained negative sign masks the underlying pattern simply because the high-rating pool has a much narrower natural score spread.

To compare the two pools on an equal footing, when sampling 40 videos from the high-rating pool and 40 from the low-rating pool, we require both 40-video subsets to have the same human MOS standard deviation (SD). To ensure our conclusion is robust rather than tied to a single arbitrary spread, we set three matched SD levels—Narrow, Medium, and Wide (placed at the 20%, 50%, and 80% positions of the shared SD range achievable by both pools)—and draw 500 SD-matched pairs of subsets at each level. Once between-video MOS dispersion is held equal, Δτ\Delta\tau flips from negative to positive across all three levels in both dimensions (Table 2).

Table 2. High-minus-low rank correlation difference (Δτ=τhigh−τlow\Delta\tau = \tau_{\text{high}} - \tau_{\text{low}}, 40 videos per group, averaged over 500 draws) comparing unconstrained sampling (Unconstr.) against item-wise SD matching (Narrow / Medium / Wide).

ModelCreativity: Unconstr. | Narrow / Medium / WideVisual: Unconstr. | Narrow / Medium / Wide
GPT-5.4-0.121 | +0.045 / +0.055 / +0.072-0.125 | +0.081 / +0.090 / +0.095
Gemini-2.5-Pro-0.179 | +0.018 / +0.016 / +0.026-0.190 | +0.033 / +0.034 / +0.038
Gemini-3-Flash-0.136 | +0.036 / +0.066 / +0.099-0.176 | +0.028 / +0.071 / +0.112
Gemini-3.1-Pro-0.100 | +0.061 / +0.097 / +0.132-0.171 | +0.057 / +0.076 / +0.092
Kimi-k2.5-0.123 | +0.088 / +0.098 / +0.115-0.066 | +0.128 / +0.158 / +0.170
Qwen3-Omni-Flash-0.206 | -0.038 / -0.035 / -0.006-0.157 | +0.005 / +0.014 / +0.020
Qwen3-VL-235B-0.126 | +0.036 / +0.042 / +0.060-0.119 | +0.065 / +0.077 / +0.080
Qwen3-VL-30B-0.178 | -0.007 / -0.003 / +0.008-0.122 | +0.029 / +0.036 / +0.032
Qwen3-VL-32B-0.158 | -0.004 / +0.012 / +0.045-0.180 | +0.058 / +0.049 / +0.037
Qwen3.5-27B-0.142 | -0.006 / +0.010 / +0.052-0.063 | +0.079 / +0.099 / +0.113
Qwen3.5-35B-0.074 | +0.033 / +0.052 / +0.088-0.093 | +0.050 / +0.075 / +0.094
Qwen3.5-Omni-Plus-0.133 | -0.001 / +0.044 / +0.086-0.101 | +0.067 / +0.096 / +0.125
Qwen3.5-Plus-0.158 | +0.019 / +0.032 / +0.054-0.137 | +0.045 / +0.080 / +0.123
13-Model Mean-0.141 | +0.022 / +0.037 / +0.064-0.131 | +0.056 / +0.074 / +0.087

Note: Positive Δτ\Delta\tau indicates stronger rank correspondence on the high-rated group. Repeating the analysis with 60 videos per group yields the same overall pattern across all three SD levels.

3. Revised conclusions in Section 4.3

We sincerely thank you again for this insightful suggestion, which prompted these analyses and helped us revise the claims in Section 4.3 around two takeaways:

  1. Unconstrained subset comparisons compress score spread and invert the true comparison: Directly filtering top-rated subsets or comparing high- vs. low-rated pools without controlling for score dispersion is not an equal comparison, because higher-rated videos have a much narrower natural MOS spread relative to human rating uncertainty. Without SD matching, this range compression yields negative Δτ\Delta\tau across all 13 models and masks their actual relative alignment.
  2. Equal-dispersion comparison reverses Δτ\Delta\tau to positive, and we bound our interpretation carefully: Holding between-video MOS dispersion equal provides a direct comparison and reverses Δτ\Delta\tau from negative to positive across both dimensions, indicating that models generally align slightly better on higher-rated videos than on lower-rated ones of comparable spread. Because video content and human rating uncertainty still vary across rating levels, we carefully bound our claims in the revised manuscript, viewing these patterns as reflecting how alignment changes with subset characteristics and providing a descriptive comparison of rank correspondence under matched human MOS SD.
AuthorsOpenReview ↗

Response to Reviewer 3GXA (3/4)

Q1. Mapping non-integer means for linear-weighted κ\kappa and handling null outputs

Thank you for asking for these clarifications. We have made both rules explicit in the revised manuscript:

  1. Rounding for linear-weighted κ\kappa: We first compute each video's human MOS and 3-run model mean, and then round both to the nearest integer on the Likert scale (1-7, or 1-5 for positive affect) before computing linear-weighted κ\kappa. (Kendall's τ\tau and Pearson's rr use the unrounded means.)
  2. Null outputs and audio gate clarification: For non-audio items, nulls are not allowed; any failed parse was rerun until three valid integer scores were obtained. For audio items (BGM and spoken voice), a video passes the presence gate for a model if >50% of human raters and at least 2 of 3 model runs detect the audio component as present; to compare all evaluated models on a common video set, fine-grained audio ratings are evaluated on the intersection of videos passing this gate across all models. On these videos, if one of a model's three runs outputs null (absent), we average its remaining two non-null runs rather than mapping null to zero or to the lowest rating. We also corrected a wording mistake in Appendix E.5, where "true negatives (TN)" should read "true positives passing the presence gate," and stated all rules clearly in Section 4.1 and Appendices E.2 and E.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

We sincerely thank all reviewers for their constructive feedback and engagement throughout the discussion period. Your suggestions have substantially strengthened our methodology, experimental depth, and presentation clarity. Below, we summarize our key revisions and how each reviewer's concerns have been addressed.

1. Summary of Key Revisions and New Findings

  • Disentangling Subset Range Compression from Model Capability (Reviewers 3GXA, av7L, bx4H):
    • Subset diagnostics & human baseline: We added subset score-range diagnostics, a human split-half baseline, rater-bootstrap stability across higher-rated and higher-disagreement subsets, and an error decomposition explaining why rank correlation (τ\tau) drops while MAE improves on higher-rated subsets.
    • New SD-matched high-vs.-low experiment: Inspired by comments on narrowed score ranges, we compared high-rated and low-rated video pools while matching their human MOS standard deviation (SD). Results confirm that an unconstrained narrower score range lowers rank correlation and masks models' actual discriminative ability: once score dispersion is held equal, the high-minus-low rank correlation difference (Δτ\Delta\tau) reverses from negative to positive, showing that models align slightly better on higher-rated videos than on lower-rated ones of comparable spread. We updated our claims in Section 4.3 accordingly.
  • Human Audit of the Salient-Impression Pipeline & Metric Clarifications (Reviewers 3GXA, av7L): We conducted a blinded three-annotator human audit on Stage-1 reference aggregation and Stage-2 matching judgments, confirming high accuracy and inter-annotator agreement while analyzing remaining minor errors. We also clarified the κ\kappa rounding/null rules and updated the Weighted Recall normalization.
  • Repeatability & Model-Comparison Uncertainty (Reviewer bx4H): We added multi-run response variability diagnostics, paired bootstrap confidence intervals for top-model comparisons, and a discussion connecting our work to physiological audience-response modeling (Wu et al., 2024).
  • Writing Clarity, Dataset Statistics, and Scope Contextualization (Reviewer QGQH): We thoroughly revised the manuscript for readability, added product-category and duration distributions, clarified the salient-impression task and the progression from population- to individual-level analyses, and explicitly scoped our empirical claims to the young-adult demographic represented by our sample.

2. Response Status by Reviewer

  • Reviewer bx4H (Rating: 3, Confidence: 4): All concerns on subset stability, the τ\tau–MAE trade-off, response variability, paired comparison uncertainty, and related work were addressed. Reviewer bx4H confirmed: "The authors have fully addressed my comments. I have no further comments."
  • Reviewer av7L (Rating: 3, Confidence: 2): All concerns on the human split-half baseline and human validation of the salient-impression pipeline were addressed. Reviewer av7L confirmed: "The human baseline addresses my first concern, and the audit substantially addresses my second."
  • Reviewer 3GXA (Rating: 2, Confidence: 3): Reviewer 3GXA noted in their weakness that reporting within-subset rating ranges and human reference stability, or bounding the capability claim in Section 4.3, "would address this concern." We fulfilled both conditions—adding subset range/stability analyses, the SD-matched experiment, and revised claims—and completed the human audit and metric clarifications (Q1 & Q2), which directly parallel the analyses confirmed as resolved by Reviewers bx4H and av7L.
  • Reviewer QGQH (Rating: 1, Confidence: 2): In their follow-up, Reviewer QGQH acknowledged "the clear effort that has been undertaken to improve the paper in terms of clarity and writing" and noted that "the additional information on dataset statistics provide the readers more insight on the study itself." Regarding their remaining concern on participant scope and impact, we fully respect this perspective, openly acknowledged this demographic boundary, and focused our claims on the target young-adult population represented by our sample. Meanwhile, by synthesizing studies from top-tier journals (Nature Communications, Journal of Marketing Research, Nature Human Behaviour, and PNAS) alongside recent multimodal benchmarks, we respectfully noted that prior influential studies on video ads have similarly focused on young-adult populations, showing that conclusions within this scope can still yield broad scientific and practical impact; together with our reusable evaluation framework for viewer-dependent subjective alignment, we hope our findings offer meaningful insights and a useful reference for the ICLR community.

We sincerely thank the Area Chairs and Reviewers again for their time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.