- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the alignment of current multimodel large language models (MLLMs) with actual human evaluations on subjective video judgments. A new benchmark is introduced: Participants were asked to watch advertisement videos and fill out detailed questionaires on the video itself. Additionally, psychological traits and human information processing data on the participants has been gathered on a participant level. The gathered information has been compared to a selection of open and closed source VLMs. As a main result, the paper indicates that the current MLLMs align better towards the assessment of creativity than the perceptual quality.
My current recommendation is to reject the paper. My main concerns are centered around the facts that the paper is written in way which is difficult to read and understand, the narrow scope of the provided annotated dataset, and missing information on several critical aspects of the dataset creation and its limited evaluation. In my opintion, the paper is not ready for publication in its current form, and I would recommend a major revision and scope adjustment before resubmission.
Critical strengths
- If stated correctly in terms of scope, content and limitations: The mentioned dataset could still be valuable to a limited group of researchers.
Critical weaknesses
-
Poor writing: The paper is generally difficult to read, to list a few examples:
a) The introduction fails to introduce the problem in a simple way which is accessible for a broader audience. Many of the used concepts and differentiations are not explained sufficiently or at all (e.g. subjective vs. objective video understanding, implicit information understanding, etc. are just mentioned without any context).
b) Many sentences are written in a way where the content is only understandable after multiple cycles of reading: e.g. Line 52-55: "Specifically, we examine alignment across video subsets differing in human disagreement and mean ratings, trait-related variation in model–human correspondence without supplying traits to models, and similarities and differences in the relationships among human and model information-processing-related ratings."
The authors mention in the AI Use statemtent that generative AI tools were used for language editing, improving the clarity and organization of the manuscript etc. However, I would dare to hypothesize that exactly this approach lead to a poorly readable paper in this case. -
Participant sample and data: As pointed out in the limitation section, the participant sample is primarily in the age group of 20 - 30 years. A narrow occupational background, cultural context and social-media use is also indicated. The gathered dataset is hence very specific, and generalization to other age groups and demographics is highly uncertain, limiting the usability.
-
Missing methodological information: E.g.
a) Missing dataset information: Information on the gathered video dataset are missing entirely. Information such as on the content (what as advertisement about, its product category), length of ads, quantity of information conveyed etc. are missing and are not analyzed. This would be very relevant, also with regards to the human evaluation results. Further, Youtube videos are indicated in the main text, but not found in the provided vid_url.csv in the supplementary.
b) Salient impressions are mentioned multiple times, an evaluation procedure is introduced, results are presented to check for alignment between humans and LLMs. However, the authors never introduce the concept nor what they understand under it in this paper's context. Further, the methodology for the LLMs and instructions for annotators to mark the impressions are not described either. -
Insufficient evaluation: One of the main motivations mentioned for this paper is the fact that current methods restrict their analysis on a population level, not providing insights into viewer differences. Exactly this aspects seems under evaluated. Almost all evaluations are on a population level with the exception of Figure 2 and even this analysis provides only limited insight.
Questions for authors
See weaknesses above.
7 comments
Response to Reviewer QGQH (2/4)
Q2. Participant scope and practical value of the benchmark
Thank you very much for raising this important point. We truly appreciate your valuable feedback, fully agree with your concern, and openly acknowledge this limitation. Because our study required strict in-person lab sessions, our recruitment was limited in practice, resulting in 225 participants aged 18-30 from a Chinese cultural background. As you rightly pointed out, people of different ages and cultures may view ads very differently, so our findings cannot be generalized to all age groups or cultures. Following your feedback, we have clearly stated the age and cultural background of our participants upfront in the Introduction and Appendix A (the Limitations section), and we strictly limit our conclusions to this group rather than overgeneralizing.
Within this boundary, we also hope to explain why this dataset still has practical value:
- Relevance of young-adult viewers and precedents in prior studies: Our 18-30 sample includes both university students and young working professionals. Young adults aged 18-30 are an active and important audience segment in digital marketing and online consumer research (e.g., Djafarova & Rushworth, 2017). Moreover, influential laboratory studies on commercial video advertisements have similarly focused on young-adult cohorts—for example, Dmochowski et al. (2014) in Nature Communications studied real commercial ad ratings and preferences using young-adult cohorts (aged 20-29 and 18-22), and Teixeira et al. (2012) in Journal of Marketing Research examined internet video ad engagement using a sample with a mean age of 22. While these precedents do not make our sample representative of all consumers, they support that studying how MLLMs align with this young-adult group still provides a useful starting point for subjective video ad evaluation.
- Psychological and consumer diversity within the group: Even within the same age and cultural group, viewers do not react in the same way. Our participants show wide variation across four well-established psychological and consumer scales (Big Five personality, Regulatory Focus, Emotional Contagion, and Consumer Decision-Making Styles), which prior consumer and advertising studies have linked to how people respond to ads.
We take your feedback seriously and plan to expand AdAlign to broader age groups, occupations, and cultural backgrounds in our future work.
References:
- [1] Djafarova, Elmira, and Chloe Rushworth. "Exploring the credibility of online celebrities' Instagram profiles in influencing the purchase decisions of young female users." Computers in human behavior 68 (2017): 1-7.
- [2] Dmochowski, J. P., Bezdek, M. A., Abelson, B. P., Johnson, J. S., Schumacher, E. H., & Parra, L. C. (2014). Audience preferences are predicted by temporal reliability of neural processing. Nature communications, 5(1), 4567.
- [3] Teixeira, T., Wedel, M., & Pieters, R. (2012). Emotion-induced engagement in internet video advertisements. Journal of marketing research, 49(2), 144-159.
Response to Reviewer QGQH (3/4)
Q3. Dataset statistics and salient-impression evaluation
Q3(a) Video categories, duration, information quantity, and YouTube sources
Thank you for pointing out the missing information on the video dataset. We agree that details on advertisement content, product categories, and video duration are essential context for interpreting the human evaluation results. Following your feedback, we have added the full product-category and duration distributions for all 393 videos (Tables 1 and 2 below) to Appendix D and summarized them directly in Section 3.2. As shown in Table 1, the dataset covers 12 industry categories, led by Food & Beverage (25.45%), 3C Digital Electronics (16.54%), and Apparel & Underwear (14.76%). As shown in Table 2, video duration ranges from 7.00 s to 232.34 s (mean 54.53 s, median 47.77 s), with most ads falling between 30 and 60 seconds (42.75%).
Table 1. Product-category distribution across the 393 videos.
| Rank | Industry category | Video count | Percentage | Cumulative percentage |
|---|---|---|---|---|
| 1 | Food & Beverage | 100 | 25.45% | 25.45% |
| 2 | 3C Digital Electronics | 65 | 16.54% | 41.98% |
| 3 | Apparel & Underwear | 58 | 14.76% | 56.74% |
| 4 | Home Appliances | 45 | 11.45% | 68.19% |
| 5 | Beauty & Skincare | 32 | 8.14% | 76.34% |
| 6 | Home & Textiles | 32 | 8.14% | 84.48% |
| 7 | Daily Necessities | 17 | 4.33% | 88.80% |
| 8 | Shoes & Bags | 16 | 4.07% | 92.88% |
| 9 | Local Services | 6 | 1.53% | 94.40% |
| 10 | Pet Supplies | 5 | 1.27% | 95.67% |
| 11 | Toys & Trendy Playthings | 4 | 1.02% | 96.69% |
| 12 | Others (<1% each) | 13 | 3.31% | 100.00% |
Table 2. Video-duration distribution and summary statistics (N=393, total 357.19 min).
| Duration range | Video count | Percentage | Cumulative percentage | Summary statistic | Value (s) |
|---|---|---|---|---|---|
| <15 s | 17 | 4.33% | 4.33% | Minimum / Maximum | 7.00 / 232.34 |
| 15-30 s | 70 | 17.81% | 22.14% | Mean / SD | 54.53 / 34.23 |
| 30-60 s | 168 | 42.75% | 64.89% | 25th / 50th (Median) / 75th pct | 30.05 / 47.77 / 65.99 |
| 1-1.5 min | 77 | 19.59% | 84.48% | 5th / 95th pct | 15.03 / 125.90 |
| 1.5-2 min | 39 | 9.92% | 94.40% | — | — |
| >2 min | 22 | 5.60% | 100.00% | — | — |
Thank you also for your suggestion regarding the "amount of information conveyed." We appreciate this thoughtful point. Unlike duration or product category, an ad video does not have a single objective number for how much information it carries, since different viewers often feel differently about how informative an ad is. Instead of assigning a fixed objective label, we included two rating items—Visual Complexity and Desire to Postpone Closure (Table 1 in the manuscripts and Appendix B.1), so that viewers could directly rate how rich and informative each video felt to them.
YouTube links: Thank you for catching this detail. Because advertising is cross-border, we used YouTube as one of our main platforms to find ad videos. However, many original YouTube uploads did not have the Chinese subtitles needed for our participants (Section 3.2). To make sure participants could clearly understand every ad, we downloaded the corresponding Chinese-subtitled versions from Chinese video platforms for the lab sessions and initially recorded those download links in our metadata file. Following your comment, we have now added the original YouTube links alongside the subtitled video links in our metadata and explained this clearly in Section 3.2 and Appendix D.
Response to Reviewer QGQH (1/4)
We sincerely thank you for your careful reading of our manuscript and for your candid, constructive feedback. Your comments helped us identify several places where our presentation, dataset descriptions, and analytical structure were not communicated clearly enough. Taking your suggestions seriously, we have thoroughly revised the manuscript to improve readability, added detailed video-dataset statistics, clarified the salient-impression task and instructions in the main text, and better connected our population-level and individual-level analyses. Our point-by-point responses and revisions are provided below.
Q1. Writing clarity and problem introduction
Thank you for pointing out where our writing and problem introduction were difficult to follow. We agree that introducing technical terms without sufficient context—coupled with overly dense sentence structures—made the paper harder to read than intended. In the revised manuscript, we carefully edited the text throughout to use clearer, more direct language. First, to make the problem immediately accessible to a broader audience, we now open the Introduction with an intuitive advertising scenario:
"In the advertising marketing scenario, audience reactions to a video campaign are rarely uniform. Some advertisements win broad praise, whereas others spark sharp controversy; some are remembered for a single catchy hook, while others leave viewers with diverse impressions depending on their personal tastes and decision-making styles. Much like the saying that 'there are a thousand Hamlets in a thousand people's eyes,' subjective responses to video advertisements vary across both videos and viewers. How well do current multimodal large language models (MLLMs) perform in such subjective evaluation settings? To explore this question, we collect real human audience evaluations and build a benchmark for subjective video understanding."
This scenario naturally distinguishes our task from traditional video benchmarks that rely on a few expert annotators. Objective content understanding focuses on verifiable video facts that people generally agree on (such as what objects appear or what actions occur), and implicit information understanding focuses on inferring hidden messages or narrative intent from those facts. In contrast, subjective video understanding means that the viewer is an essential part of the answer. Because different viewers bring their own tastes and perspectives, their subjective evaluations and salient impressions naturally vary.
We also truly appreciate your feedback on our sentence structure. Following your suggestion, we have broken long, dense sentences throughout the paper into shorter, clearly connected statements. For example, we rewrote the dense overview sentence you quoted in the Introduction into plain, accessible terms that explain how we move from overall population-level alignment to the patterns beneath it:
"In this work, we first evaluate population-level alignment between MLLMs and the audience as a whole, and then examine the underlying patterns that overall population-level alignment may obscure. Specifically, beyond comparing overall scores and salient impressions across the full dataset, we investigate three practical questions. First, does a model still align with human viewers when we look only at higher-rated advertisements or more controversial advertisements with higher viewer disagreement? Second, when a model is prompted to act as a generic ordinary viewer without knowing any viewer's background, how does the correspondence between model and human ratings vary across viewers with different psychological traits? Third, when humans rate an advertisement's creativity, multimodal perceptual quality, and information-processing-related experiences, do models reflect the same relationships among these ratings as humans do, and does jointly evaluating these experiences help or hurt model alignment?"