- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits long-form visual storytelling from the perspective of affective faithfulness. To evaluate characters' affective trajectories in videos generated by frontier T2V models, it establishes a comprehensive benchmark that combines multimodal affective narrative construction via human-agent collaboration with nine affective metrics. Extensive experiments reveal that current video generative models still fall short in affective consistency and faithfulness.
Critical strengths
S1. The paper frames affective faithfulness as a novel evaluation problem for long-form T2V generation and presents a comprehensive benchmark built via a human-agent workflow.
S2. The paper defines a suite of nine metrics for affect detection and understanding, organized into Accuracy, Development, and Support. These metrics provide a more detailed analysis of characters’ affective coherence, fidelity, and transitions in generated videos.
Critical weaknesses
W1. The definitions of the nine proposed metrics are insufficiently explained.
-
The metrics are partially redundant, as several of them assess overlapping properties. For example, the SB requirement is a subset of the SF requirement (Sec. 4.2). The typical IR structure (source → interrupt → recover) can be viewed as two consecutive TA transitions (Sec. 4.3) and CAL assesses the legibility of the same evidence evaluated by ECEF (Sec. 4.4). Consistently, the TA–IR and ECEF–CAL score pairs in Table 1 are highly correlated across the 13 T2V systems.
-
Several metrics lack detailed definitions, which makes them difficult to interpret. For example, the transition pattern in TA is not clearly defined (Sec. 4.3, L309). It is also unclear how IR (Sec. 4.3), ECEF, and CAL (Sec. 4.4) are scored beyond the generic five-level protocol of Sec. 4.1. Finally, PLCC, SRCC, and KRCC in Table 2 are not defined.
W2. The paper lacks in-depth case analysis and ablation studies.
-
In Table 3 (Appendix A), PER decreases from 39.11 to 33.03 when the AffectVBench Story Prompt replaces the raw prompt. The reason for this drop should be explained and more failure cases should be provided.
-
Case comparisons are needed for Table 1 to show whether each method’s results are consistent across the general and affective metrics. One example would be a method that achieves the best general score but poor affective faithfulness, such as SANA-Video (highest PQ of 85.3 but lowest Overall of 47.1), and vice versa.
-
Only a prompt-construction ablation is conducted (Appendix A). More detailed explanations and experiments are needed to justify the design choices, including the models used for benchmark construction (Sec. 3.2), the judges (Sec. 4.1), and the evaluated baselines (Appendix C.1).
Questions for authors
I have a few questions about Fig. 1 and Fig. 2. The layouts are a little crowded and the paired captions are not that informative.
Q1. Fig. 1(b) shows four agents, but beyond the brief role descriptions in Sec. 3.2, no more technical details are disclosed. Are these agents purely prompt-based, or do they involve tool use and how are the components wired? What's the specific agent design?
Q2. The caption of Fig. 2 does not explain the internal workflow of its three stages, including Prompt Reference Construction, Reference-Conditioned Metric Evaluation, and Anchor-Based Viewer-Affect Evaluation. The interconnecting arrows across panels are also confusing. What information flow are these arrows designed to convey?
5 comments
Response to Reviewer zCdw(Part 2 / 3)
[W2. In-Depth Case Analysis and Comprehensive Ablation Studies]
W2.1: Explaining the PER Decrease and Analyzing Additional Cases. We thank the reviewer for highlighting the decrease in PER. In the Raw Prompt versus AffectVBench Story Prompt comparison, mean PER decreases from 39.11 to 33.03 (−6.08). The constructed Story Prompts specify finer affective trajectories and can require a state to remain perceptible across later narrative phases. These additional persistence requirements offer a possible explanation for the lower score: a video may depict the initial affect or a subsequent change while failing to sustain the required residual state. The aggregate comparison alone does not establish this as the cause for every sample.
We have therefore added a matched-prompt case study in Appendix E.1. For sample 0001, the Prompt requires Ada’s pensiveness to soften into warmth during her interaction with Adeline, with some warmth remaining at the end. The four generated videos show different levels of persistence (PER: 61.3–82.9) and affective faithfulness despite differences in perceptual quality. This case illustrates the type of temporally extended requirement that PER examines; it is not presented as a Raw Prompt versus Story Prompt comparison. Appendix E.2-5 provides additional generation examples across models.
W2.2: General Quality vs. Affective Faithfulness.
SANA-Video achieves the highest PQ (85.3) but the lowest Overall score (47.1), with particularly weak Development performance. Conversely, Vidu Q2 Pro achieves the highest Overall score (63.4) despite a lower PQ (69.1), with more balanced performance across all three affective dimensions. These results demonstrate that high visual quality does not necessarily translate into strong affective faithfulness, highlighting the need for dedicated evaluation of emotional expression in long-form videos.
| Method | PQ ↑ | Accuracy | Development | Support | Overall ↑ |
|---|---|---|---|---|---|
| SANA-Video | 85.3 | 64.6 | 24.7 | 52.2 | 47.1 |
| Vidu Q2 Pro | 69.1 | 68.1 | 59.3 | 63 | 63.4 |
W2.3: Rationale for Selecting Construction Models, Judges, and Baselines.
We thank the reviewer for this suggestion. We clarify our selection rationale and provide additional supporting experiments below.
- Construction Models and Pipeline Ablations. We assign construction models complementary roles in audiovisual interpretation, visual grounding, conflict checking, and prompt compilation, as detailed in Sec. 3.2. The Full Pipeline achieves PCF/ANC scores of 86.2/72.0, outperforming Raw Prompt (76.4/39.8) and Single-Model Construction (63.8/55.3). Removing Agent B or Agent C also reduces performance, supporting their contributions to construction quality. Detailed pipeline ablations are provided in our response to Reviewer zCdw (W4).
- Judge Selection. We compare Qwen3-VL-32B, Qwen3-VL-8B, and InternVL3.5-8B while holding the videos, references, and Evidence Ledgers fixed. Qwen3-VL-32B achieves the highest human-alignment SRCC on all eight reference-conditioned metrics. Its mean SRCC across these metrics is 0.69, compared with 0.54 for each alternative; for IR, the respective scores are 0.54, 0.32, and 0.35. These results support selecting Qwen3-VL-32B as the final Judge while showing that evaluation remains sensitive to Judge choice (Appendix D.5, Table 13).
| Judge | SF | SB | EAB | PER | TA | IR | ECEF | CAL |
|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-32B | 0.76 | 0.82 | 0.7 | 0.6 | 0.66 | 0.54 | 0.69 | 0.78 |
| Qwen3-VL-8B | 0.66 | 0.7 | 0.56 | 0.43 | 0.47 | 0.32 | 0.57 | 0.63 |
| InternVL3.5-8B | 0.64 | 0.68 | 0.54 | 0.41 | 0.45 | 0.35 | 0.59 | 0.65 |
- Baseline Selection. We prioritize representative T2V systems covering diverse long-video generation and temporal modeling approaches. For commercial models, we select Seedance 1.0 Pro and Vidu Q2 Pro to balance model coverage with the large-scale generation costs of Core-300. We additionally apply output-quality screening, excluding systems whose outputs consistently exhibit severe blurring, prominent artifacts, or unrecognizable character identities. Our comparison is therefore restricted to systems meeting these output-quality criteria.
Response to Reviewer zCdw(Part 3 / 3)
[Q1: How are the four agents in Fig. 1(b) implemented, including their prompting, tool use, and interactions?]
We thank the reviewer for identifying the need to clarify the agent design. We have expanded Sec. 3.2 and the appendix F.1 to specify each agent’s model, input permissions, task, and structured output.
Agent Roles. For each Emotion Reference Video, an Orchestrator initializes a Structured Draft with shared character, event, affect, and temporal fields.
Agent A (Seed2.1 Pro**)** analyzes the full audiovisual stream to recover narrative context, character relations, events, and affective triggers. In parallel, Agent B (Qwen3.8 Max) examines only the muted visual stream and auxiliary visual evidence to record observable characters, actions, expressions, and events; it does not read Agent A’s report. Agent C (MiniMax M3) aligns the two reports by character, event, and time, and flags conflicts or claims lacking visual support for human review. Agent D (GPT-5.6-sol) then performs Analysis of Integration and Boundaries: it combines the structured reports according to human review decisions while keeping contextual interpretations, visually verifiable evidence, and unresolved content separate. Agent D produces a metadata draft, not the Story Prompt.
Tool Use and Connections. The Orchestrator supplies timestamped evidence through shot-boundary detection, keyframe sampling, Grounding-DINO, SAM2, SFace, and pose/action analysis. Agents A and B write to separate partitions of the same Structured Draft; subsequent components exchange structured records containing timestamps, confidence, and evidence references rather than free-form conversation histories. Human reviewers verify consequential character, event, affect, and visual-support claims, after which schema validation and human confirmation freeze the Metadata.
Story Prompt Compilation. A freshly initialized GPT-5.6-sol instance serves separately as the Visual Story Compiler. It receives only the frozen Metadata and converts supported narrative and affective information into a Story Prompt for silent T2V generation. Seed2.1 Turbo checks the candidate Prompt against the Metadata, and human reviewers make the final decision. This separation makes the four agents’ respective contributions and the handoff from metadata construction to prompt compilation explicit.
[Q2:Enhance Figure 2 and introduce information flow.]
We thank the reviewer for pointing out that the original caption did not adequately explain the workflow. The revised Fig. 2 and accompanying text organize evaluation into four stages, with the final aggregation and audit shown explicitly. Further workflow details are provided in Appendix C.2.
- Stage 1: Prompt-Conditioned Reference Construction. The Story Prompt is parsed into traceable Metric Requirements for the first eight metrics and a separate prompt-specific Viewer-Affect Brief. For Core-300, an Anchor Construction Assistant retrieves Low-, Mid-, and High-level candidates from a generation-video library; five human validators assess the candidates, and three anchors per level are retained. Source videos are not used as anchors. For unseen Prompts, the Assistant searches the library and supplements insufficient levels through dynamic construction.
- Stage 2: Prompt-Blind Evidence Grounding. Temporal segmentation and visual grounding tools process the muted candidate video. Independent Blind Structural Observer and Blind Affect Specialist roles extract observable structure and affective cues without access to the Prompt or target affect. Their aligned, verified outputs form a frozen Blind Evidence Ledger.
- Stage 3: Requirement Judging and Anchor-Relative VAR. Reference-Conditioned Judges match the Metric Requirements to the Ledger and estimate five-level fulfillment distributions for the first eight metrics. Uncertain local evidence can trigger a bounded Active Probe. In an isolated branch, Viewer Personas compare the candidate video with its prompt-specific anchors under randomized left–right presentation; a frozen Davidson model converts these judgments into VAR. The VAR branch does not receive the Ledger or other metric scores.
- Stage 4: Aggregation and Auditing. Requirement-level judgments and VAR are combined into the nine metrics, the Accuracy, Development, and Support dimensions, and Overall. The system also reports coverage and uncertainty and audits the evaluation records.
Information Flow. The arrows show artifact transfers: Metric Requirements feed reference-conditioned judging; the Viewer-Affect Brief and Anchor Bank feed VAR comparison; and the generated video enters both blind extraction and VAR comparison. The branches meet only at aggregation. The dashed arrow requests bounded reinspection of uncertain evidence without changing Prompt-derived requirements or feeding scores back to extraction. The appendix details these connections.
Response to Reviewer zCdw(Part 1 / 3)
[W1:Metric Rationale, Relationships, and Scoring Criteria.]
We thank the reviewer for highlighting the need to clarify the design rationale and operational definitions of the nine metrics. The revised manuscript details the metric definitions in Sections 4.1–4.5, examines their relationships in Section 5.2 and Fig. 3, and provides level-specific rubrics for IR, ECEF, and CAL in Appendix C.2 and statistical definitions in Appendix C.5.
Design Motivation and Diagnostic Value**.** We design the nine metrics to diagnose failures in long-form affective faithfulness across three dimensions: Accuracy, whether the intended affect is faithfully realized for the designated characters; Development, whether affective states evolve consistently with the narrative; and Support, whether the video provides contextually appropriate and perceptually accessible evidence for those states. The nine metrics provide finer-grained diagnostic checks within these dimensions. We acknowledge partial overlap. Separate reporting provides targeted diagnosis of specific failure types, as summarized below.
Metric Correlation Analysis. We provide a correlation matrix covering all nine metrics. The average correlation is 0.59 within dimensions and 0.38 across dimensions, consistent with the grouping of related diagnostic checks. The high TA–IR and ECEF–CAL correlations are also consistent with their shared structure or evidence. These descriptive results characterize the relationships among the metrics; we do not claim statistical independence..
Metric Definitions and Statistical Measures. We clarify TA’s ordered affective transition patterns and the metric-specific criteria underlying the five-level ratings for IR, ECEF, and CAL. We also explicitly define PLCC (Pearson linear correlation coefficient), SRCC (Spearman rank correlation coefficient), and KRCC (Kendall rank correlation coefficient), which quantify linear association, monotonic rank association, and rank concordance, respectively.