acceptodds
Under review as a conference paper at ICLR 2027

AffectVBench: Benchmarking Affective Faithfulness in Long-Form Text-to-Video Generation

Abstract

Existing benchmarks have accelerated long-form text-to-video (T2V) generation by assessing visual quality, temporal coherence, and cinematic presentation, but overlook affective faithfulness: whether the intended affect is correctly bound to characters and events, develops coherently across shots, and is supported by perceivable evidence. We introduce AffectVBench, a fine-grained benchmark for this capability. Grounded in Story Prompts and structured references capturing open-vocabulary affective states, intensity, mixtures, and temporal evolution, AffectVBench provides nine evidence-grounded metrics across Accuracy, Development, and Support. It further incorporates blinded anchor-based calibration for Viewer Affect and multidimensional uncertainty-aware probabilistic scoring. Experiments on 13 representative T2V models reveal that current models struggle with multi-character emotion binding, cross-shot affect persistence, and event-driven affect transitions. These findings establish affective faithfulness as a distinct bottleneck for credible long-form video generation.

Primary area:
datasets and benchmarks
Keywords:
Affective Faithfulness, Long-Form Text-to-Video Generation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NTGCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits long-form visual storytelling from the perspective of affective faithfulness. To evaluate characters' affective trajectories in videos generated by frontier T2V models, it establishes a comprehensive benchmark that combines multimodal affective narrative construction via human-agent collaboration with nine affective metrics. Extensive experiments reveal that current video generative models still fall short in affective consistency and faithfulness.

Critical strengths

S1. The paper frames affective faithfulness as a novel evaluation problem for long-form T2V generation and presents a comprehensive benchmark built via a human-agent workflow.

S2. The paper defines a suite of nine metrics for affect detection and understanding, organized into Accuracy, Development, and Support. These metrics provide a more detailed analysis of characters’ affective coherence, fidelity, and transitions in generated videos.

Critical weaknesses

W1. The definitions of the nine proposed metrics are insufficiently explained.

  1. The metrics are partially redundant, as several of them assess overlapping properties. For example, the SB requirement (cr,Ar,τr)(c_r, A_r, \tau_r) is a subset of the SF requirement (cr,Ar,λr,ρr,τr)(c_r, A_r, \lambda_r, \rho_r, \tau_r) (Sec. 4.2). The typical IR structure (source → interrupt → recover) can be viewed as two consecutive TA transitions (Sec. 4.3) and CAL assesses the legibility of the same evidence evaluated by ECEF (Sec. 4.4). Consistently, the TA–IR and ECEF–CAL score pairs in Table 1 are highly correlated across the 13 T2V systems.

  2. Several metrics lack detailed definitions, which makes them difficult to interpret. For example, the transition pattern ηr\eta_r in TA is not clearly defined (Sec. 4.3, L309). It is also unclear how IR (Sec. 4.3), ECEF, and CAL (Sec. 4.4) are scored beyond the generic five-level protocol of Sec. 4.1. Finally, PLCC, SRCC, and KRCC in Table 2 are not defined.

W2. The paper lacks in-depth case analysis and ablation studies.

  1. In Table 3 (Appendix A), PER decreases from 39.11 to 33.03 when the AffectVBench Story Prompt replaces the raw prompt. The reason for this drop should be explained and more failure cases should be provided.

  2. Case comparisons are needed for Table 1 to show whether each method’s results are consistent across the general and affective metrics. One example would be a method that achieves the best general score but poor affective faithfulness, such as SANA-Video (highest PQ of 85.3 but lowest Overall of 47.1), and vice versa.

  3. Only a prompt-construction ablation is conducted (Appendix A). More detailed explanations and experiments are needed to justify the design choices, including the models used for benchmark construction (Sec. 3.2), the judges (Sec. 4.1), and the evaluated baselines (Appendix C.1).

Questions for authors

I have a few questions about Fig. 1 and Fig. 2. The layouts are a little crowded and the paired captions are not that informative.

Q1. Fig. 1(b) shows four agents, but beyond the brief role descriptions in Sec. 3.2, no more technical details are disclosed. Are these agents purely prompt-based, or do they involve tool use and how are the components wired? What's the specific agent design?

Q2. The caption of Fig. 2 does not explain the internal workflow of its three stages, including Prompt Reference Construction, Reference-Conditioned Metric Evaluation, and Anchor-Based Viewer-Affect Evaluation. The interconnecting arrows across panels are also confusing. What information flow are these arrows designed to convey?

5 comments
AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 1 / 3)

[W1:Metric Rationale, Relationships, and Scoring Criteria.]

We thank the reviewer for highlighting the need to clarify the design rationale and operational definitions of the nine metrics. The revised manuscript details the metric definitions in Sections 4.1–4.5, examines their relationships in Section 5.2 and Fig. 3, and provides level-specific rubrics for IR, ECEF, and CAL in Appendix C.2 and statistical definitions in Appendix C.5.

Design Motivation and Diagnostic Value**.** We design the nine metrics to diagnose failures in long-form affective faithfulness across three dimensions: Accuracy, whether the intended affect is faithfully realized for the designated characters; Development, whether affective states evolve consistently with the narrative; and Support, whether the video provides contextually appropriate and perceptually accessible evidence for those states. The nine metrics provide finer-grained diagnostic checks within these dimensions. We acknowledge partial overlap. Separate reporting provides targeted diagnosis of specific failure types, as summarized below.

Metric PairShared BasisDiagnostic Value
SF vs. SBCharacter-conditioned affect requirements.SF assesses affective realization; SB highlights character–affect binding failures across shots.
TA vs. IRIR’s interruption-and-recovery pattern can be represented as two consecutive transitions.TA assesses narrative alignment of transitions; IR specifically highlights interruption-and-recovery failures.
ECEF vs. CAL.The same expressive evidence.ECEF assesses whether the evidence supports the intended affect in context; CAL assesses whether viewers can clearly perceive that evidence.

Metric Correlation Analysis. We provide a correlation matrix covering all nine metrics. The average correlation is 0.59 within dimensions and 0.38 across dimensions, consistent with the grouping of related diagnostic checks. The high TA–IR and ECEF–CAL correlations are also consistent with their shared structure or evidence. These descriptive results characterize the relationships among the metrics; we do not claim statistical independence..

Metric Definitions and Statistical Measures. We clarify TA’s ordered affective transition patterns and the metric-specific criteria underlying the five-level ratings for IR, ECEF, and CAL. We also explicitly define PLCC (Pearson linear correlation coefficient), SRCC (Spearman rank correlation coefficient), and KRCC (Kendall rank correlation coefficient), which quantify linear association, monotonic rank association, and rank concordance, respectively.

AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 2 / 3)

[W2. In-Depth Case Analysis and Comprehensive Ablation Studies]

W2.1: Explaining the PER Decrease and Analyzing Additional Cases. We thank the reviewer for highlighting the decrease in PER. In the Raw Prompt versus AffectVBench Story Prompt comparison, mean PER decreases from 39.11 to 33.03 (−6.08). The constructed Story Prompts specify finer affective trajectories and can require a state to remain perceptible across later narrative phases. These additional persistence requirements offer a possible explanation for the lower score: a video may depict the initial affect or a subsequent change while failing to sustain the required residual state. The aggregate comparison alone does not establish this as the cause for every sample.

We have therefore added a matched-prompt case study in Appendix E.1. For sample 0001, the Prompt requires Ada’s pensiveness to soften into warmth during her interaction with Adeline, with some warmth remaining at the end. The four generated videos show different levels of persistence (PER: 61.3–82.9) and affective faithfulness despite differences in perceptual quality. This case illustrates the type of temporally extended requirement that PER examines; it is not presented as a Raw Prompt versus Story Prompt comparison. Appendix E.2-5 provides additional generation examples across models.

W2.2: General Quality vs. Affective Faithfulness.

SANA-Video achieves the highest PQ (85.3) but the lowest Overall score (47.1), with particularly weak Development performance. Conversely, Vidu Q2 Pro achieves the highest Overall score (63.4) despite a lower PQ (69.1), with more balanced performance across all three affective dimensions. These results demonstrate that high visual quality does not necessarily translate into strong affective faithfulness, highlighting the need for dedicated evaluation of emotional expression in long-form videos.

MethodPQ ↑AccuracyDevelopmentSupportOverall ↑
SANA-Video85.364.624.752.247.1
Vidu Q2 Pro69.168.159.36363.4

W2.3: Rationale for Selecting Construction Models, Judges, and Baselines.

We thank the reviewer for this suggestion. We clarify our selection rationale and provide additional supporting experiments below.

  • Construction Models and Pipeline Ablations. We assign construction models complementary roles in audiovisual interpretation, visual grounding, conflict checking, and prompt compilation, as detailed in Sec. 3.2. The Full Pipeline achieves PCF/ANC scores of 86.2/72.0, outperforming Raw Prompt (76.4/39.8) and Single-Model Construction (63.8/55.3). Removing Agent B or Agent C also reduces performance, supporting their contributions to construction quality. Detailed pipeline ablations are provided in our response to Reviewer zCdw (W4).
  • Judge Selection. We compare Qwen3-VL-32B, Qwen3-VL-8B, and InternVL3.5-8B while holding the videos, references, and Evidence Ledgers fixed. Qwen3-VL-32B achieves the highest human-alignment SRCC on all eight reference-conditioned metrics. Its mean SRCC across these metrics is 0.69, compared with 0.54 for each alternative; for IR, the respective scores are 0.54, 0.32, and 0.35. These results support selecting Qwen3-VL-32B as the final Judge while showing that evaluation remains sensitive to Judge choice (Appendix D.5, Table 13).
JudgeSFSBEABPERTAIRECEFCAL
Qwen3-VL-32B0.760.820.70.60.660.540.690.78
Qwen3-VL-8B0.660.70.560.430.470.320.570.63
InternVL3.5-8B0.640.680.540.410.450.350.590.65
  • Baseline Selection. We prioritize representative T2V systems covering diverse long-video generation and temporal modeling approaches. For commercial models, we select Seedance 1.0 Pro and Vidu Q2 Pro to balance model coverage with the large-scale generation costs of Core-300. We additionally apply output-quality screening, excluding systems whose outputs consistently exhibit severe blurring, prominent artifacts, or unrecognizable character identities. Our comparison is therefore restricted to systems meeting these output-quality criteria.
AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 3 / 3)

[Q1: How are the four agents in Fig. 1(b) implemented, including their prompting, tool use, and interactions?]

We thank the reviewer for identifying the need to clarify the agent design. We have expanded Sec. 3.2 and the appendix F.1 to specify each agent’s model, input permissions, task, and structured output.

Agent Roles. For each Emotion Reference Video, an Orchestrator initializes a Structured Draft with shared character, event, affect, and temporal fields.

Agent A (Seed2.1 Pro**)** analyzes the full audiovisual stream to recover narrative context, character relations, events, and affective triggers. In parallel, Agent B (Qwen3.8 Max) examines only the muted visual stream and auxiliary visual evidence to record observable characters, actions, expressions, and events; it does not read Agent A’s report. Agent C (MiniMax M3) aligns the two reports by character, event, and time, and flags conflicts or claims lacking visual support for human review. Agent D (GPT-5.6-sol) then performs Analysis of Integration and Boundaries: it combines the structured reports according to human review decisions while keeping contextual interpretations, visually verifiable evidence, and unresolved content separate. Agent D produces a metadata draft, not the Story Prompt.

Tool Use and Connections. The Orchestrator supplies timestamped evidence through shot-boundary detection, keyframe sampling, Grounding-DINO, SAM2, SFace, and pose/action analysis. Agents A and B write to separate partitions of the same Structured Draft; subsequent components exchange structured records containing timestamps, confidence, and evidence references rather than free-form conversation histories. Human reviewers verify consequential character, event, affect, and visual-support claims, after which schema validation and human confirmation freeze the Metadata.

Story Prompt Compilation. A freshly initialized GPT-5.6-sol instance serves separately as the Visual Story Compiler. It receives only the frozen Metadata and converts supported narrative and affective information into a Story Prompt for silent T2V generation. Seed2.1 Turbo checks the candidate Prompt against the Metadata, and human reviewers make the final decision. This separation makes the four agents’ respective contributions and the handoff from metadata construction to prompt compilation explicit.


[Q2:Enhance Figure 2 and introduce information flow.]

We thank the reviewer for pointing out that the original caption did not adequately explain the workflow. The revised Fig. 2 and accompanying text organize evaluation into four stages, with the final aggregation and audit shown explicitly. Further workflow details are provided in Appendix C.2.

  • Stage 1: Prompt-Conditioned Reference Construction. The Story Prompt is parsed into traceable Metric Requirements for the first eight metrics and a separate prompt-specific Viewer-Affect Brief. For Core-300, an Anchor Construction Assistant retrieves Low-, Mid-, and High-level candidates from a generation-video library; five human validators assess the candidates, and three anchors per level are retained. Source videos are not used as anchors. For unseen Prompts, the Assistant searches the library and supplements insufficient levels through dynamic construction.
  • Stage 2: Prompt-Blind Evidence Grounding. Temporal segmentation and visual grounding tools process the muted candidate video. Independent Blind Structural Observer and Blind Affect Specialist roles extract observable structure and affective cues without access to the Prompt or target affect. Their aligned, verified outputs form a frozen Blind Evidence Ledger.
  • Stage 3: Requirement Judging and Anchor-Relative VAR. Reference-Conditioned Judges match the Metric Requirements to the Ledger and estimate five-level fulfillment distributions for the first eight metrics. Uncertain local evidence can trigger a bounded Active Probe. In an isolated branch, Viewer Personas compare the candidate video with its prompt-specific anchors under randomized left–right presentation; a frozen Davidson model converts these judgments into VAR. The VAR branch does not receive the Ledger or other metric scores.
  • Stage 4: Aggregation and Auditing. Requirement-level judgments and VAR are combined into the nine metrics, the Accuracy, Development, and Support dimensions, and Overall. The system also reports coverage and uncertainty and audits the evaluation records.

Information Flow. The arrows show artifact transfers: Metric Requirements feed reference-conditioned judging; the Viewer-Affect Brief and Anchor Bank feed VAR comparison; and the generated video enters both blind extraction and VAR comparison. The branches meet only at aggregation. The dashed arrow requests bounded reinspection of uncertain evidence without changing Prompt-derived requirements or feeding scores back to extraction. The appendix details these connections.

Reviewer_zCdwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AffectVBench, a fine-grained benchmark designed to evaluate Affective Faithfulness in long-form T2V generation. The authors argue that existing benchmarks predominantly emphasize visual quality, temporal coherence, and generic text alignment, while overlooking affective dynamics such as multi-character emotion binding, cross-shot emotional evolution, and visual evidence grounding. To address this gap, they design a human–agent pipeline built on real-world videos to construct structured metadata and story prompts, and define nine fine-grained evaluation metrics. Experiments on 13 representative T2V models reveal current bottlenecks in emotional persistence and event-driven affective transitions.

Critical strengths

S1. Clear and insightful motivation.

The paper identifies a critical yet underexplored problem in long video generation and distinguishes between character-expressed affect and viewer-perceived affect, offering a novel framing.

S2. Comprehensive evaluation metric system.

The authors propose three dimensions and nine fine-grained sub-metrics, together forming a systematic and reasonably complete evaluation framework.

S3. Solid experimental scale.

Thirteen mainstream and state-of-the-art T2V models are evaluated, accompanied by human agreement analysis and ablation studies on prompt construction.

Critical weaknesses

W1. The evaluation pipeline relies excessively on a complex and unvalidated cascade of multiple LLM/MLLM components, potentially introducing severe error accumulation and data leakage risks.

The automated evaluation pipeline is highly complex, involving more than ten models serving as "judges" and "detectives" (Section 4.1). However, the paper lacks a quantitative ablation of the robustness of this automated evaluation pipeline itself. If the front-end Blind Evidence Extraction misses evidence or hallucinates, the back-end Reference Conditioned Judge will assign scores based on erroneous evidence.

W2. The dataset scale is small (only 300 videos), insufficient to support the diversity and generalizability of a long-video benchmark.

Section 3.1 shows that the benchmark ultimately contains only 300 candidate videos. For a benchmark evaluating open-vocabulary, multi-character, complex long-form narratives, a scale of 300 prompts/test samples appears rather small and may invite concerns regarding sample bias and overfitting to particular audiovisual styles.

W3. The Development metrics exhibit low human alignment.

Table 2 reports the correlation between automated scores and human judgments. While visually grounded metrics (SB, CAL) perform well, the Development dimension which constitutes the paper's core innovation (PER TA IR VAR) show low correlations, and human agreement itself is also low (IR only 0.66). This indicates that the proposed automated metrics still fail to accurately capture human perception, raising doubts about the authority of the automated scores.

W4. Potential Evaluator Bias

Section 3.2 employs multiple powerful commercial LLM/MLLMs for prompt compilation and metadata distillation. This heavy reliance on top-tier MLLMs to generate detailed story prompts effectively forces the complex visual semantics of the original video into textual descriptions preferred by a specific MLLM. When videos generated by T2V models are evaluated using the same or similar MLLMs, LLM-to-LLM preference bias (in-context/evaluator bias) may arise.

W5. Figure display issues.

Some fonts in Figures 1 are too small and the lines too dense. And personally I don't think putting emoji on "Affect Bench" (Fig.1.c.(II)) is a good choice. This reduces the readability of the figure.

Questions for authors

We kindly ask the authors to address the concerns raised in the Weaknesses section above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 1 / 4)

[W1:How is the multi-model evaluation pipeline validated against error propagation and information leakage?]

We thank the reviewer for highlighting these concerns. We provide quantitative validation and clarify the information controls below. Additional protocol details and ablation results have been added to Appendix C.2, D.4 and Tables 11, 12 of the revised manuscript.

Necessity of the Multi-Model Design. The ablations use identical Core-300 videos, references, and aggregation rules, with human–automatic SRCC measured on the fixed Human-60 subset (Table 12). The full pipeline achieves 0.76, 0.60, and 0.74 for Accuracy, Development, and ECEF/CAL, respectively, compared with 0.37, 0.30, and 0.28 for direct video-to-score judgment. Direct video-to-score evaluation yields higher automatic Accuracy and ECEF/CAL scores but substantially lower human alignment. Removing blinding or specialized visual tools, or using a single blind observer, also reduces alignment. Notably, removing Verification and Active Probe raises automatic scores across all three groups while reducing human alignment, demonstrating that higher scores do not necessarily indicate more reliable evaluation. These results show that the tested simplifications sacrifice human alignment, supporting the need for specialized components in our current pipeline.

Evidence Quality and Human Validation. We replace automatically extracted evidence with a human-verified Evidence Ledger as a diagnostic variant. Human alignment further improves to 0.86, 0.78, and 0.84 across the three dimensions, identifying front-end evidence quality as an important remaining bottleneck in downstream scoring. Meanwhile, VAR follows an independent anchor-based evaluation protocol, and its ablations further demonstrate the contributions of multi-level anchor comparisons and Davidson tie modeling to human alignment.

Role Separation and Information-Leakage Controls. Benchmark construction establishes human-verified affective references, which are frozen before generated-video evaluation. During evaluation, reference-blind observers extract evidence without access to these references; the final judge subsequently uses them for reference-conditioned scoring. These controls keep reference information out of the blind-observation stage and fix scoring targets before evaluation, preventing post hoc reference revision based on evaluation outcomes.

VariantAuto. Acc.Auto. Dev.Auto. ECEF/CALHuman Acc.Human Dev.Human ECEF/CAL
Full Pipeline65.536.767.60.760.60.74
Direct Video-to-Score Judge7624.873.60.370.30.28
w/o Blinding67.236.168.80.70.530.68
w/o Specialized Visual Tools62.429.659.10.550.350.43
Single Blind Observer63.227.861.30.570.360.5
w/o Verification and Active Probe73.842.5740.420.250.36
Human-Verified Evidence Ledger^\dagger------0.860.780.84
VariantAutomatic VAR ScoreHuman Alignment (SRCC)
Full Pipeline44.10.58
Mid Anchor Only33.50.26
Direct Absolute VAR Rating50.50.3
w/o Davidson Tie Modeling40.50.44
AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 2 / 4)

[W2. Limited Benchmark Scale and Concerns about Diversity and Generalizability]

We thank the reviewer for raising concerns about the scale, diversity, and generalizability of our benchmark. We provide the following additional analyses. The corresponding analyses have been added to the Prompt Coverage and Diversity Analysis section of the revised manuscript and are presented in Tables 4–6.

Fine-Grained Affective Coverage. Our 300 story-level prompts cover 47 fine-grained affective states across six broad affective families, including Affection, Relief, Apprehension, Disdain, and Embarrassment. Specifically, 99.3% of prompts contain at least one affective family, 87.7% involve two or more families, and 76.0% contain ordered cross-family transitions, spanning 29 distinct transition patterns. These results demonstrate coverage of affective states, compositions, and temporal transitions that video count alone does not capture.

Annotation Complexity and Comparable Benchmark Scale. Core-300’s 300 story-level prompts are comparable in scale to the 234 high-complexity long-video prompts in LoCoT2V-Bench. Our fine-grained affective annotation requires detailed validation of subject–affect binding, affective persistence, state transitions, and interruption and recovery. Constructing and validating Core-300 involved approximately 1,379.8 person-hours and 59,820 human judgments, reflecting the practical demands of this temporally structured annotation process.

Narrative and Presentation Diversity. We further analyze the prompts across character interactions, scene environments, cinematic language, and visual atmosphere. Specifically, 70.0% involve dyadic interactions, 76.0% include indoor scenes, and 42.3% include outdoor scenes. Additionally, 50.7% contain camera or shot descriptions, 18.7% involve close-ups, and 22.0% include scene transitions. The prompts also describe varied lighting and weather conditions. These statistics document coverage across multiple narrative and visual dimensions, while leaving open the possibility of source-dependent stylistic biases.

AuthorsOpenReview ↗

Response to Reviewer zCdw(Part 3 / 4)

[W3. Human Alignment of Development Metrics]

We thank the reviewer for highlighting the relatively lower human alignment of the Development metrics. We agree that human evaluation is essential for validating automated evaluation. Fine-grained affective evaluation also involves subjective judgments, particularly when assessing temporal development. Human alignment should therefore be interpreted alongside inter-annotator variability and uncertainty**. We view the proposed scores as diagnostic indicators that complement human evaluation, while acknowledging their remaining alignment gaps.** Please refer to Section 5.2, and Table 2 in the revised manuscript.

Human Agreement and Subjectivity. We further report the inter-annotator agreement with 95% confidence intervals across all nine metrics. For PER, TA, and IR, the agreement scores are 0.71 [0.57, 0.82], 0.74 [0.61, 0.84], and 0.66 [0.50, 0.78], respectively, with lower point estimates than those of the Accuracy metrics. IR has the lowest estimated agreement, reflecting greater variability among annotators when judging affective interruption and recovery. These results provide context for interpreting automated–human alignment, while human judgments remain an essential perceptual reference.

Why Development Is More Challenging. Unlike Accuracy, which primarily evaluates the consistency of affective states, subjects, and events, Development requires understanding affective persistence, transitions, and interruption–recovery patterns over time. In particular, IR requires jointly considering the emotional states before and after an interruption and their surrounding context, making it more temporally dependent and subjective. Accordingly, the human–automatic SRCC scores for PER, TA, and IR are 0.60, 0.66, and 0.54, respectively, following a similar trend to inter-annotator agreement.

Comparison with Existing Benchmarks. Prior benchmarks also report challenges in evaluating temporal and affective properties. LoCoT2V-Bench[1] reports SRCCs of 0.50–0.55 for several temporal and visual dimensions and 0.41–0.49 for its narrative and affective dimensions. EmoTrace-Bench[2] reports Fleiss’ κ\kappa of 0.50–0.65 on its Development set and Cohen’s κ\kappa of 0.50–0.58 on its Test set for emotion-related annotations. Our human agreement estimates for PER, TA, and IR are 0.71, 0.74, and 0.66, measured using ordinal Krippendorff’s α\alpha. Because these studies differ in annotation tasks, samples, and agreement measures, their values are not directly comparable. We cite them to contextualize the difficulty of fine-grained affective evaluation, not to claim superior agreement.

DimensionMetricPLCCSRCCKRCCHuman Agreement (95% CI)
AccuracySF0.780.760.590.82 [0.73, 0.89]
SB0.840.820.660.86 [0.78, 0.92]
EAB0.720.70.530.78 [0.66, 0.87]
DevelopmentPER0.630.60.450.71 [0.57, 0.82]
TA0.690.660.50.74 [0.61, 0.84]
IR0.570.540.40.66 [0.50, 0.78]
SupportECEF0.720.690.530.77 [0.64, 0.86]
CAL0.80.780.620.83 [0.73, 0.90]
VAR0.610.580.430.70 [0.55, 0.81]

[1] Zheng X, Wu C, Chen K, et al. LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation[J]. arXiv preprint arXiv:2510.26412, 2025.

[2] Wang Y, Shi L, Zheng Y, et al. EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity[J]. arXiv preprint arXiv:2610.15452, 2026.

Reviewer_MY72OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AffectVBench, a benchmark for affective faithfulness in long-form text-to-video generation. It constructs human-verified Story Prompts and structured affective references from source videos, then evaluates generated videos using nine metrics covering affective accuracy, temporal development, and visual support. The evaluation spans 13 video-generation systems and includes a comparison with human judgments. The results suggest that current systems struggle particularly with sustained affect, event-driven transitions, and correct attribution of emotions to characters.

Critical strengths
  1. The paper proposes a benchmark for sentiment analysis in videos, which can drive progress in the field of sentiment-aware visual generation.
  2. The paper decomposes affective faithfulness into nine metrics across accuracy, development, and support, each grounded in structured, requirement-level references rather than a single coarse score.
Critical weaknesses

1,Although the paper makes considerable efforts in evidence collection to help the benchmark understand affect, the ceiling of affective judgment is still locked by the capability of the off-the-shelf models, making fine-grained affect analysis unreliable. Moreover, these models may be overly sensitive to video quality issues (e.g., synthetic artifacts in generated videos), an influencing factor the paper does not consider.

2,The benchmark is built entirely from the 300 candidate videos curated in Sec. 3, but it is unclear whether it covers the full spectrum of emotions and which fine-grained affective states are included. Potential data bias in the source videos is a fundamental concern that the paper does not analyze.

Questions for authors

1,Since each Story Prompt is derived from a real source video, why not use the source video as a reference? For example, one could run the same evidence extraction on the source video and compare it with the test video.

2,The construction of the VAR Anchor Bank is underspecified: it is unclear whether anchor videos are drawn from the 300 curated source videos? how many anchors exist per target viewer affect, and what human verification protocol are used?

3,Some models natively generate ~30s videos, while Seedance 1.0 Pro and Vidu Q2 Pro concatenate three ~10s clips. Does this difference in generation protocol affect the scores, especially the temporal metrics?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MY72 (Part 1 / 4)

We thank the reviewer for the detailed and constructive feedback. We appreciate your recognition of the value of requirement-level affective evaluation. Your questions about evaluator reliability, benchmark coverage, source-video references, VAR anchors, and generation protocols have helped us strengthen both the experiments and the presentation of AffectVBench.


[W1: Robustness of Fine-Grained Affective Evaluation]

We appreciate the reviewer's concerns regarding the limitations of off-the-shelf MLLMs and the sensitivity of affective evaluation to video quality. We address these concerns from two aspects:

  • Reliability of Fine-Grained Affective Evaluation. AffectVBench improves evaluation reliability through evidence-grounded decomposition and verification, supported by ablation studies.
    • Evidence-grounded decomposition and verification. AffectVBench decomposes affective faithfulness into nine evidence-grounded metrics across Accuracy, Development, and Support, transforming holistic affective judgment into fine-grained, verifiable subproblems. Rather than relying on a single judge for direct evaluation, it decouples evidence extraction from final judgment through reference-blind observation, specialized visual tools, independent verification, and reference-conditioned scoring.
    • Ablation-based validation. Table 12 demonstrates the contribution of these designs. The full pipeline achieves human alignment of 0.76, 0.60, and 0.74 for Accuracy, Development, and ECEF/CAL, respectively, compared with 0.37, 0.30, and 0.28 for direct video-to-score judgment. Removing blinding, specialized visual tools, or verification together with active probing reduces alignment across all three groups; using a single blind observer also lowers alignment. Human-verified evidence further improves alignment to 0.86, 0.78, and 0.84, respectively, identifying evidence quality as an important remaining bottleneck. These results support the effectiveness of the structured evidence pipeline in improving agreement with human judgments. Further details are provided in Section 4.1 and Appendix D3, D.4.
Evaluation VariantAccuracyDevelopmentECEF/CAL
Full Pipeline0.760.60.74
Direct Video-to-Score Judge0.370.30.28
w/o Blinding0.70.530.68
w/o Specialized Visual Tools0.550.350.43
w/o Verification and Active Probe0.420.250.36
Single Blind Observer0.570.360.5
Human-Verified Evidence Ledger$^\dagger$0.860.780.84
  • Sensitivity to Video Quality. AffectVBench addresses video-quality effects through quality-aware benchmark design and controlled validation.
    • Quality-aware benchmark design. We excluded T2V models prone to severe blur, synthetic artifacts, or unidentifiable characters. We also manually screened 691 candidate Story Prompts, retaining 300 for which most T2V models could generate sufficiently identifiable characters, narrative events, and intended emotional expressions. Moreover, CAL explicitly assesses the visibility of affective cues, distinguishing affective support from general visual quality.
    • Controlled quality-sensitivity analysis. We evaluated Context-Forcing under three quality conditions (Table 14). As video quality deteriorates, both human ratings and the automatic scores for all three affective dimensions decline, with Support showing the largest decrease. Human–automatic SRCC is 0.78, 0.72, and 0.64 under high-, moderate-, and low-quality conditions, respectively. These results indicate that automatic evaluation tracks human-assessed changes across quality levels, although alignment weakens as quality deteriorates. Further details are provided in Appendix D.6.
QualityVQA ↑Accuracy (H/A)Development (H/A)Support (H/A)Overall (H/A)H-A SRCC ↑
High84.670.3/69.442.6/41.764.1/63.259.0/58.10.78
Moderate69.368.1/67.141.2/40.358.3/56.755.9/54.70.72
Low5264.3/62.939.1/37.849.0/46.850.8/49.20.64
AuthorsOpenReview ↗

Response to Reviewer MY72 (Part 2 / 4)

[W2: Affective Coverage and Potential Data Bias]

We have added analyses of emotional coverage, temporal dynamics, and contextual content to characterize the benchmark’s composition and clarify its limitations. These analyses and the corresponding discussion are provided in Sections 3.2, 5.3, and 6, Appendix B.2–B.5, and Tables 4–6 of the revised manuscript.

Emotional coverage and temporal dynamics. We group open-vocabulary emotion descriptions into six broad families, encompassing basic emotions and fine-grained states such as affection, relief, apprehension, disdain, and embarrassment. 99.3% of prompts contain at least one emotion family, and 87.7% contain two or more. Moreover, 77.7% include explicit emotional or facial-state transition cues, while 76.0% contain ordered transitions across emotion families, spanning 29 distinct transition patterns. These results document multi-emotion content and temporal structure, supporting evaluation of both affective states and their development in long-form videos.

Potential data Bias. We have added a quantitative content analysis to make the benchmark’s composition transparent and provide a clearer basis for assessing potential data bias. Specifically, 96.7% of prompts contain narrative or event-related cues, 82.0% include character interactions, 94.3% contain scene or environmental descriptions, and 63.3% include cinematic camera cues. Additionally, 59.7% contain at least two types of environmental cues. We acknowledge that source-video selection may introduce sampling bias. Expanding the benchmark requires substantial human annotation and verification, making a meaningful increase in scale infeasible within the current revision timeline. We will explicitly discuss this limitation and its implications for generalizability in the Conclusion section. AffectVBench is designed to evaluate long-form affective expression across diverse narrative conditions, and this task-specific goal guides our curation. Its model comparisons should therefore be interpreted within the benchmark’s documented emotional and narrative coverage, with broader source coverage remaining an important direction for future work.


[Q1: Why not use evidence extracted from the source video as the reference for evaluating generated videos?]

Real source videos provide naturalistic, coherent examples of emotional states, event–affect relationships, and temporal development. We use these examples to construct Story Prompts and structured affective requirements.

AffectVBench evaluates Text-conditioned Affective Faithfulness: whether generated videos express the affective intent specified by the Story Prompt. The same narrative and emotional intent can be conveyed through diverse facial expressions, gestures, emotional intensities, and temporal patterns. Direct comparison with source-video evidence could impose source-specific details that the text does not require, penalizing otherwise valid realizations. We therefore evaluate generated videos against structured, prompt-grounded affective requirements, using source videos to ground benchmark construction.

Our goal is to add a fine-grained affective dimension to video-generation evaluation, complementing existing quality assessment and providing diagnostic feedback to improve generation quality.

Further details are provided in Section 4.1, the Real Video Reference discussion in Section 5.1, and Appendix C.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer MY72 (Part 3 / 4)

Q2: What are the anchor sources, scale, and human validation procedures?

We clarify these points below. The revised manuscript provides further details in Sec. 4.1 (Stage 1), Appendix C.4, and Table 7.

Anchor Source and Organization. The 300 curated source videos are used to construct Story Prompts, from which prompt-specific Viewer-Affect Briefs are drafted and human-verified. The source videos are not used as anchors. For each frozen Brief, an Anchor Construction Assistant retrieves and ranks candidates from a Generation Video Library. Anchors are Brief-specific because the intended viewer affect depends on the narrative context and temporal arc.

Anchor Scale. For each Brief, the Assistant proposes four library videos at each realization level—Low, Mid, and High. Human validation retains three official anchors and one designated backup per level. Across 300 Briefs, the selected bank contains 2,700 official anchors and 900 backups. Rejected candidates and their replacements are recorded separately from these selected-bank counts.

Human Validation Protocol. Five validators independently assess each candidate using only the muted video and its frozen Brief; the proposed level, generation source, and model identity are hidden. They record affective observability, agreement with the intended viewer-affect direction, realization level, and timestamped supporting or contradictory evidence. Acceptance requires at least four of five validators to agree on observability, direction, and level, with an interquartile range (IQR) of at most one level. For each Brief, at least 80% of valid cross-level comparisons must also support Low < Mid < High. Candidates failing these checks are replaced by further library candidates and reviewed anew, rather than relabeled after review. Briefs, official anchors, backups, validation records, and video identities are frozen before candidate-video evaluation.

ItemPer Viewer-Affect BriefTotal (300 Briefs)
Curated source videos—300 (not anchors)
Prompt-specific Viewer-Affect Briefs1300
Anchor realization levels3 (Low/Mid/High)900 level instances
Selected candidates per level41,200 per level
Official anchors per level3900 per level
Total official anchors92,700
Designated backups3900
Independent validators per candidate55 per candidate

Comments on the paper

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

As the discussion period is nearing its end, we would like to take this opportunity to ensure that we have adequately addressed all your concerns. If you have any remaining questions, concerns, or suggestions, please feel free to let us know. We sincerely appreciate your valuable feedback and would be happy to clarify any outstanding issues to further improve our work.

Thank you again for your time and effort in reviewing our paper and engaging in the discussion.

Best regards, The Authors

AuthorsOpenReview ↗

General Response

We thank the reviewers and Area Chair for their careful feedback. During rebuttal, we revised the manuscript, added experiments, and expanded the appendix to address questions about benchmark construction, metric design, and evaluation reliability. Blue section headings mark revised content.


Main revisions and additional evidence

  • Benchmark coverage and construction. We added analyses of Core-300’s fine-grained affective states, temporal transitions, narrative content, characters, scenes, and cinematic cues, together with a discussion of source-dependent coverage. We also clarified the construction agents’ roles, tool use, information flow, and human review, and added a 60-prompt construction ablation (MY72 W2; zCdw W2, W4; NTGC W2, Q1).
  • Evaluation workflow and references. We revised Fig. 2 and its accompanying text to explain the four stages and the artifacts passed between them. We clarified why source videos inform Prompt construction without serving as direct scoring references or VAR anchors, and specified the Anchor Bank’s retrieval, scale, and human validation protocol (MY72 Q1–Q2; NTGC Q2).
  • Metric definitions and result interpretation. We expanded the definitions and scoring criteria of the nine metrics, added their correlation analysis, and defined the statistical measures used for human alignment. Matched-prompt cases now illustrate why general video quality and affective faithfulness can diverge. We also discuss the PER decrease in the Raw Prompt comparison without treating an illustrative case as proof of its aggregate cause (NTGC W1–W2).
  • Evaluation reliability and protocol sensitivity. We added human-agreement estimates, evidence-extraction results, evaluation-pipeline ablations, and fixed-input Judge comparisons. Further analyses examine video-quality sensitivity and native versus segmented generation protocols (MY72 W1, Q3; zCdw W1, W3–W4; NTGC W2).

Clarifications in response to follow-up comments. The subsequent discussion focused particularly on the number of metrics and the reliability of MLLM-assisted evaluation. We have addressed these points directly:

  • Why report nine metrics? The metrics are organized into Accuracy (what affect is expressed, by whom, and in relation to which event), Development (whether it persists, changes as intended, and recovers when required), and Support (whether appropriate cues exist, are legible, and evoke the intended viewer response). Related metrics may share evidence, but their separate scores identify different generation failures. The revised definitions, examples, and correlation analysis make these relationships explicit.
  • How reliable is MLLM-assisted scoring? AffectVBench separates prompt-blind evidence extraction from requirement-level judgment and evaluates VAR through an independent anchor-relative branch. Table 2 compares all nine scores with independent human ratings; the appendix ablations test the contributions of the pipeline components, evidence quality, and Judge choice. These results support the use of automatic scores for fine-grained diagnosis alongside human evaluation, while making their metric-dependent alignment visible.

We appreciate the reviewers’ continued engagement. Their feedback has helped us make the evaluation procedure, the purpose of each metric, and the interpretation of the results clearer.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

With only a few hours remaining in the discussion period, we would be very grateful if you could take a final look at our responses and revised manuscript. We have addressed your concerns point by point and added substantial clarifications, analyses, and experiments accordingly. If any concern remains unresolved, please let us know as soon as possible so that we can provide a final clarification before the discussion closes. Your response would be extremely valuable to us at this stage.

Thank you again for your time, careful review, and engagement throughout the discussion.

Best regards,
The Authors

Dear Reviewers,

The rebuttal period is coming to an end. Please read the authors' responses and update your reviews if appropriate. It would be helpful to share your thoughts as soon as possible to allow enough time for a productive discussion. Thank you!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.