- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that once generated videos get longer, subject consistency cannot be judged only on the subjects visible at the start, because people and vehicles keep entering and leaving the scene. The authors build DynSC-Eval on top of Grounding DINO, SAM2 and DINOv3. Every eligible subject is tracked across its visible lifespan, and six metrics are computed per track. Three are local: they count sudden appearances, sudden disappearances and abrupt mutations, and measure frame-to-frame drift. The other three are global and measure how far each track moves away from its initial appearance. The evaluation is validated in two ways. A synthetic experiment with moving shapes shows the metrics react to injected errors while VBench SC barely moves. A benchmark of models at 5s, 15s and 60s separates models that VBench scores as similar. In the second half, the metrics become a DiffusionNFT reward on driving videos. The authors report lower inconsistency for Wan-2.1-1.3B, SANA-2B and ReSim, and for Wan models extended to 10s and 30s through curriculum fine-tuning.
Critical strengths
- The core observation is correct and timely: first-frame or reference-subject protocols simply do not see objects that show up later, and this matters more as generation length grows.
- Splitting the metrics into event counts versus continuous drift, and local versus global, is genuinely informative; Table 2 shows LongLive and Helios failing in quite different ways.
- The synthetic experiment is built with care, using paired clean and corrupted videos, nested perturbation sets and three seeds, and the separate local-only and global-only injections show the metrics are not redundant.
- Using the evaluation as a training reward is a reasonable way to show the metric is useful in practice, and it is tried on T2V, I2V and longer durations.
Critical weaknesses
- For a metric paper, the main gap is the lack of human-alignment validation: there is no human rating correlation and no event-level precision or recall on real generated videos.
- Sudden appearance and disappearance events are inferred from track onsets and terminations, so detector misses, occlusions and ID switches may be counted as generation errors.
- Key hyperparameters such as the mutation threshold, reference window, boundary margin and minimum track length lack sensitivity analysis, and the threshold changes from 0.35 to 0.16 for driving without justification.
- The optimization evaluation is circular because the reward is built from the same metrics used for evaluation, so gains on DynSC-Eval are partly expected.
- Technically the work mostly assembles existing components, and the reward weights of 0.3, 0.3, 0.3 and 0.1 are given without justification.
- The metrics are never run on real videos, which should score near zero, and no error analysis of the perception pipeline.
- The global metrics may penalize legitimate appearance changes caused by viewpoint, pose, scale or lighting.
Questions for authors
- How well do the six metrics correlate with human judgments of subject consistency, and what are the event-level precision and recall on annotated generated videos?
- What scores does DynSC-Eval produce on real driving and general videos, and how often do tracker failures trigger false events?
- How stable are the model rankings in Table 2 when the threshold, reference window, margin, detector or prompt vocabulary change?
- How do you separate true identity drift from natural viewpoint and pose variation in the global metrics?
- Can you report VBench Subject Consistency and a human preference study for the post-training results?
- How does DynSC-Eval compare quantitatively with first-frame-based or reference-based subject consistency protocols on real generated videos?
- How were the reward weights chosen, and how sensitive are the results to them?
If the rebuttal can address the core issues, I would consider raising my score.
6 comments
Rebuttal (2/5)
Q2: Evaluation on real videos.
A2: Thank you for this constructive question. We evaluated 300 real videos across four scenarios: animals, pedestrians, soccer, and driving, covering three durations (5s, 15s, and 30s), with 100 videos per duration and 25 per scenario. We used the default threshold and report all six metrics below.
| Duration | Videos | SR ↓ | ER ↓ | MAD ↓ | ED ↓ | MRD ↓ | WFD ↓ |
|---|---|---|---|---|---|---|---|
| 5s | 100 | 0.0897 | 3.6070 | 0.0972 | 0.2408 | 0.1768 | 0.3306 |
| 15s | 100 | 0.1221 | 3.9968 | 0.1014 | 0.2960 | 0.2150 | 0.4056 |
| 30s | 100 | 0.1549 | 3.6385 | 0.1066 | 0.3406 | 0.2498 | 0.4680 |
The results show that only a small fraction of eligible subject trajectories are flagged with potential inconsistency events. Furthermore, ER remains relatively stable at 3.61–4.00 events per 100 track-seconds across durations, while MAD also changes only slightly. We further analyzed the false positives in real videos and found that the vast majority arise from two particularly challenging cases: (1) occlusion and visibility changes of small subjects, and (2) unusual initial poses followed by substantial legitimate motion. These cases highlight the inherent difficulty of distinguishing legitimate appearance variations from genuine subject inconsistencies under challenging visual conditions.
Furthermore, as reported in A1, our independent audit of 80 generated videos confirms that 148 of 190 manually adjudicated candidate events correspond to genuine subject inconsistencies, yielding 77.89% event-level precision.
Q3: Robustness to threshold selection.
A3: Thank you for raising this question. We conducted two additional sensitivity analyses.
1. Model-Ranking Robustness Across 1,400 Videos
We evaluated all 1,400 generated videos, including 300 commercial 5s videos, 500 open-source 5s videos, 300 commercial 15s videos, and 300 long-form 60s videos, covering animals, pedestrians, vehicles, and soccer. We varied the general-purpose threshold from its default to . Each model was evaluated on 100 videos, and model rankings were compared within the same video category and duration.
| Video Set | Models | τ = 0.30 | τ = 0.35 | τ = 0.40 |
|---|---|---|---|---|
| Commercial 5s | 3 | 1.00 / 1.00 | 1.00 / 1.00 | 0.50 / 0.33 |
| Open-source 5s | 5 | 0.90 / 0.80 | 1.00 / 1.00 | 1.00 / 1.00 |
| Commercial 15s | 3 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
| Long-form 60s | 3 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
The results demonstrate strong ranking stability, particularly for longer videos. Most importantly, both 15s and 60s model rankings remain completely unchanged across all three thresholds, demonstrating that our main comparative conclusions for longer videos do not depend on the precise choice of . The 5s open-source rankings also remain highly consistent. While the commercial 5s ranking shows some sensitivity at , this involves only three models and does not affect the complete ranking stability observed in the longer-duration evaluations.
2. Robustness of RL Improvements to Threshold Selection
We further investigated whether the reported improvements from DynSC-based RL depend on the driving threshold Using the original Wan-2.1-1.3B Base and DynSC-RL models, we re-evaluated the same paired driving videos at across 5s, 10s, and 30s durations. The following table reports paired mean SR differences, defined as ΔSR = SR(RL) − SR(Base). Negative values indicate improved consistency.
| Duration | τ | Base → DynSC-RL | ΔSR [95% CI] ↓ |
|---|---|---|---|
| 5s | 0.14 | 0.2629 → 0.2329 | −0.0300 [−0.0725, 0.0117] |
| 5s | 0.16 | 0.2334 → 0.1992 | −0.0342 [−0.0769, 0.0074] |
| 5s | 0.18 | 0.2073 → 0.1811 | −0.0263 [−0.0676, 0.0144] |
| 10s | 0.14 | 0.4165 → 0.3635 | −0.0530 [−0.0899, −0.0149] |
| 10s | 0.16 | 0.3767 → 0.3309 | −0.0458 [−0.0828, −0.0082] |
| 10s | 0.18 | 0.3414 → 0.2934 | −0.0480 [−0.0835, −0.0115] |
| 30s | 0.14 | 0.5061 → 0.4698 | −0.0363 [−0.0930, 0.0171] |
| 30s | 0.16 | 0.4640 → 0.4476 | −0.0164 [−0.0725, 0.0382] |
| 30s | 0.18 | 0.4220 → 0.4070 | −0.0150 [−0.0700, 0.0385] |
Crucially, DynSC-RL consistently achieves lower SR than the Base model across all nine duration–threshold combinations. The direction of improvement never reverses when the threshold changes.
Rebuttal (3/5)
Q4: Distinguishing identity drift from legitimate viewpoint and pose changes.
A4: Thank you for raising this insightful question. We would like to clarify that DynSC-Eval distinguishes abrupt local inconsistencies from accumulated global appearance changes through complementary metrics. Local metrics (SR, ER, MAD) focus on sudden changes between nearby observations, while global metrics (ED, MRD, WFD) quantify appearance deviations accumulated along continuous subject trajectories. We do not assume that every global appearance deviation necessarily represents an identity inconsistency, as legitimate viewpoint and pose changes can also contribute to feature variation.
To systematically investigate this concern, we conducted two complementary sets of controlled 3D experiments, covering both legitimate appearance variations and deliberately injected inconsistencies.
1. Robustness to Legitimate Appearance Changes
We constructed controlled 3D rendering experiments involving vehicles and humans, keeping subject identity, geometry, and material properties fixed while varying only legitimate motion and viewing conditions. All videos were rendered at 8 seconds, 8 FPS, and resolution, using the predefined threshold .
- Vehicle experiments: Three 3D vehicle models with fixed material configurations, testing rotations (0°, 45°, 90°, 180°), occlusion levels (0%, 25%, 50%, 75%), and illumination attenuation (0%, 25%, 50%, 75%), totaling 36 conditions.
- Human experiments: Two 3D human identities undergoing viewpoint changes, translation, partial occlusion, and natural articulated motions, including walking, waving, and arm raising, totaling 32 conditions.
| Subject | Normal Variations | Conditions | SR ↓ | ER ↓ | Max. Adjacent Feature Drift ↓ |
|---|---|---|---|---|---|
| Vehicle | Rotation, Occlusion, Illumination | 36 | 0 | 0 | 0.05258 |
| Human | Rotation, Translation, Occlusion, Articulation | 32 | 0 | 0 | 0.14047 |
| Total | 68 | 0 | 0 | 0.14047 |
None of the 68 legitimate variation conditions triggered false local inconsistency events (SR = ER = 0), even under rotations up to 180°, occlusion up to 75%, and natural human articulation. All experiments contained valid subject trajectories, and the human experiments maintained successful detection and tracking across all frames.
Furthermore, the maximum adjacent DINOv3 feature distances were only 0.05258 for vehicles and 0.14047 for humans, substantially below the predefined threshold of 0.35. These findings demonstrate that legitimate continuous appearance changes do not necessarily trigger abrupt inconsistency detection.
2. Sensitivity to Genuine Inconsistencies
To verify that this robustness does not arise from an insensitive evaluation pipeline, we additionally introduced two types of controlled inconsistencies:
- Severe Appearance Mutation: Abruptly replacing a four-wheeled passenger vehicle with a six-wheeled cargo vehicle and subsequently restoring it.
- Sudden Disappearance/Reappearance: Abruptly removing a vehicle and its shadow and subsequently restoring them.
Across 12 videos containing 24 ground-truth inconsistency event boundaries, DynSC-Eval correctly detected all 24 without additional false positives, achieving 100% precision and recall under these controlled conditions.
Rebuttal (1/5)
A1: Thank you for this valuable suggestion! To further validate the perceptual reliability of DynSC-Eval, we conducted three complementary human evaluations: (1) manual verification of detected inconsistency events, (2) correlation analysis with human consistency ratings, and (3) human evaluation of improvements after RL fine-tuning.
1. Event Detection Precision
We randomly sampled 80 five-second generated videos covering four scenarios: animals, pedestrians, soccer, and driving. Human annotators manually inspected the inconsistency events detected by DynSC-Eval. Among 190 candidate events with definitive human judgments, 148 were confirmed as genuine subject inconsistency events, while 42 were identified as false positives, yielding an event detection precision of 77.89%.
2. Correlation with Human Judgments
We further recruited 20 human participants to evaluate 20 fixed five-second videos generated by five models (Seedance 2.0, Kling V3, MiniMax H3, SANA-Video-2B, and Wan2.1-1.3B), using four shared prompts. Each video received four numerical consistency ratings on a 1–4 scale.
We calculated video-level Spearman correlations between human ratings and automatic consistency metrics. For DynSC-Eval, we reversed the signs of its error metrics so that higher values consistently indicate better subject consistency.
DynSC-avg reports the mean of the component-wise Spearman correlations
DynSC-Eval shows stronger numerical agreement with human judgments than VBench Subject Consistency(Vbench-SC) and VMBench Temporal Coherence Score(VMBench-TCS) in this evaluation. These findings further support the perceptual relevance of our proposed metrics, beyond their sensitivity demonstrated in the synthetic experiments.
3. Human Verification of RL Improvements
We additionally evaluated whether improvements in DynSC-Eval translate into visually observable reductions in subject inconsistency after RL fine-tuning. For randomly sampled paired driving videos before and after RL, we manually counted subject inconsistency events across 20 pairs of 5s videos, 20 pairs of 10s videos, and 15 pairs of 30s videos.
We also conducted an independent human preference evaluation involving 20 participants. Participants evaluated subject consistency, motion naturalness, and visual quality. We report preference scores where our consistency optimization win, tie, and loss are assigned 1, 0.5, and 0, respectively, averaged equally across video pairs. Thus, 0.5 indicates no preference. All results support improvements in consistency and visual quality.
Overall, these complementary human evaluations support three important conclusions: