acceptodds
Withdrawn from ICLR 2027

Towards Subject Consistency over Dynamic Subject Sets in Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We argue that as video generation extends to longer durations, subject consistency should be evaluated over *dynamic subject sets*. We therefore introduce **DynSC-Eval**, an evaluation framework that dynamically tracks eligible subjects throughout their visible lifespans and measures local continuity and global identity preservation using six complementary object-level metrics, with explicit detection of inconsistency events. To validate its effectiveness, we design synthetic experiments that actively inject inconsistency events, demonstrating both the sensitivity of DynSC-Eval and the limitations of existing metrics. Evaluations of diverse models on 5s, 15s, and 60s video generation further reveal substantial subject consistency differences that are obscured by conventional metrics. Beyond evaluation, we construct rewards from DynSC-Eval and apply DiffusionNFT post-training in an autonomous-driving testbed. On 5s generation, our approach reduces the six inconsistency metrics by an average of 13.82% for Wan-2.1-1.3B and 5.66% for SANA-2B, with improvements also observed on the I2V model ReSim. Qualitative comparisons further demonstrate the effectiveness of our method. We then extend generation to 10s and 30s through curriculum learning and show that consistency optimization remains effective while largely preserving other capabilities.

Primary area:
generative models
Keywords:
video diffusion models, temporal consistency, video generation evaluation, reinforcement learning, long video generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GovPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that once generated videos get longer, subject consistency cannot be judged only on the subjects visible at the start, because people and vehicles keep entering and leaving the scene. The authors build DynSC-Eval on top of Grounding DINO, SAM2 and DINOv3. Every eligible subject is tracked across its visible lifespan, and six metrics are computed per track. Three are local: they count sudden appearances, sudden disappearances and abrupt mutations, and measure frame-to-frame drift. The other three are global and measure how far each track moves away from its initial appearance. The evaluation is validated in two ways. A synthetic experiment with moving shapes shows the metrics react to injected errors while VBench SC barely moves. A benchmark of models at 5s, 15s and 60s separates models that VBench scores as similar. In the second half, the metrics become a DiffusionNFT reward on driving videos. The authors report lower inconsistency for Wan-2.1-1.3B, SANA-2B and ReSim, and for Wan models extended to 10s and 30s through curriculum fine-tuning.

Critical strengths
  1. The core observation is correct and timely: first-frame or reference-subject protocols simply do not see objects that show up later, and this matters more as generation length grows.
  2. Splitting the metrics into event counts versus continuous drift, and local versus global, is genuinely informative; Table 2 shows LongLive and Helios failing in quite different ways.
  3. The synthetic experiment is built with care, using paired clean and corrupted videos, nested perturbation sets and three seeds, and the separate local-only and global-only injections show the metrics are not redundant.
  4. Using the evaluation as a training reward is a reasonable way to show the metric is useful in practice, and it is tried on T2V, I2V and longer durations.
Critical weaknesses
  1. For a metric paper, the main gap is the lack of human-alignment validation: there is no human rating correlation and no event-level precision or recall on real generated videos.
  2. Sudden appearance and disappearance events are inferred from track onsets and terminations, so detector misses, occlusions and ID switches may be counted as generation errors.
  3. Key hyperparameters such as the mutation threshold, reference window, boundary margin and minimum track length lack sensitivity analysis, and the threshold changes from 0.35 to 0.16 for driving without justification.
  4. The optimization evaluation is circular because the reward is built from the same metrics used for evaluation, so gains on DynSC-Eval are partly expected.
  5. Technically the work mostly assembles existing components, and the reward weights of 0.3, 0.3, 0.3 and 0.1 are given without justification.
  6. The metrics are never run on real videos, which should score near zero, and no error analysis of the perception pipeline.
  7. The global metrics may penalize legitimate appearance changes caused by viewpoint, pose, scale or lighting.
Questions for authors
  1. How well do the six metrics correlate with human judgments of subject consistency, and what are the event-level precision and recall on annotated generated videos?
  2. What scores does DynSC-Eval produce on real driving and general videos, and how often do tracker failures trigger false events?
  3. How stable are the model rankings in Table 2 when the threshold, reference window, margin, detector or prompt vocabulary change?
  4. How do you separate true identity drift from natural viewpoint and pose variation in the global metrics?
  5. Can you report VBench Subject Consistency and a human preference study for the post-training results?
  6. How does DynSC-Eval compare quantitatively with first-frame-based or reference-based subject consistency protocols on real generated videos?
  7. How were the reward weights chosen, and how sensitive are the results to them?

If the rebuttal can address the core issues, I would consider raising my score.

6 comments
AuthorsOpenReview ↗

Rebuttal (1/5)

Q1: Human judgement.

A1: Thank you for this valuable suggestion! To further validate the perceptual reliability of DynSC-Eval, we conducted three complementary human evaluations: (1) manual verification of detected inconsistency events, (2) correlation analysis with human consistency ratings, and (3) human evaluation of improvements after RL fine-tuning.

1. Event Detection Precision

We randomly sampled 80 five-second generated videos covering four scenarios: animals, pedestrians, soccer, and driving. Human annotators manually inspected the inconsistency events detected by DynSC-Eval. Among 190 candidate events with definitive human judgments, 148 were confirmed as genuine subject inconsistency events, while 42 were identified as false positives, yielding an event detection precision of 77.89%.

VideosEvaluated EventsTrue PositivesFalse PositivesPrecision ↑
801901484277.89%

2. Correlation with Human Judgments

We further recruited 20 human participants to evaluate 20 fixed five-second videos generated by five models (Seedance 2.0, Kling V3, MiniMax H3, SANA-Video-2B, and Wan2.1-1.3B), using four shared prompts. Each video received four numerical consistency ratings on a 1–4 scale.

We calculated video-level Spearman correlations between human ratings and automatic consistency metrics. For DynSC-Eval, we reversed the signs of its error metrics so that higher values consistently indicate better subject consistency.

MetricVideo-level Spearman ρ ↑
DynSC-avg0.5605
VBench-SC0.3642
VMBench-TCS0.4786

DynSC-avg reports the mean of the component-wise Spearman correlations

DynSC-Eval shows stronger numerical agreement with human judgments than VBench Subject Consistency(Vbench-SC) and VMBench Temporal Coherence Score(VMBench-TCS) in this evaluation. These findings further support the perceptual relevance of our proposed metrics, beyond their sensitivity demonstrated in the synthetic experiments.

3. Human Verification of RL Improvements

We additionally evaluated whether improvements in DynSC-Eval translate into visually observable reductions in subject inconsistency after RL fine-tuning. For randomly sampled paired driving videos before and after RL, we manually counted subject inconsistency events across 20 pairs of 5s videos, 20 pairs of 10s videos, and 15 pairs of 30s videos.

DurationVideo PairsBefore RL ↓After RL ↓Reduction ↑
5s20291451.7%
10s20726213.9%
30s15977126.8%

We also conducted an independent human preference evaluation involving 20 participants. Participants evaluated subject consistency, motion naturalness, and visual quality. We report preference scores where our consistency optimization win, tie, and loss are assigned 1, 0.5, and 0, respectively, averaged equally across video pairs. Thus, 0.5 indicates no preference. All results support improvements in consistency and visual quality.

DurationConsistency ↑Motion Naturalness ↑Visual Quality ↑
5s0.713 [0.563, 0.850]0.675 [0.488, 0.838]0.638 [0.525, 0.738]
10s0.688 [0.550, 0.838]0.525 [0.363, 0.688]0.650 [0.550, 0.763]
30s0.821 [0.713, 0.917]0.704 [0.638, 0.767]0.650 [0.575, 0.713]

Overall, these complementary human evaluations support three important conclusions:

  1. Detection reliability: 77.89% of manually adjudicated detected events represent genuine subject inconsistency.
  2. Perceptual alignment: DynSC-Eval exhibits stronger numerical correlation with human consistency judgments than VBench Subject Consistency and TCS on the evaluated videos.
  3. Practical effectiveness: RL fine-tuning reduces manually identified inconsistency events across 5s, 10s, and 30s videos, with additional human preference evidence supporting perceptible improvements.
AuthorsOpenReview ↗

Rebuttal (2/5)

Q2: Evaluation on real videos.

A2: Thank you for this constructive question. We evaluated 300 real videos across four scenarios: animals, pedestrians, soccer, and driving, covering three durations (5s, 15s, and 30s), with 100 videos per duration and 25 per scenario. We used the default threshold τ=0.35\tau = 0.35 and report all six metrics below.

DurationVideosSR ↓ER ↓MAD ↓ED ↓MRD ↓WFD ↓
5s1000.08973.60700.09720.24080.17680.3306
15s1000.12213.99680.10140.29600.21500.4056
30s1000.15493.63850.10660.34060.24980.4680

The results show that only a small fraction of eligible subject trajectories are flagged with potential inconsistency events. Furthermore, ER remains relatively stable at 3.61–4.00 events per 100 track-seconds across durations, while MAD also changes only slightly. We further analyzed the false positives in real videos and found that the vast majority arise from two particularly challenging cases: (1) occlusion and visibility changes of small subjects, and (2) unusual initial poses followed by substantial legitimate motion. These cases highlight the inherent difficulty of distinguishing legitimate appearance variations from genuine subject inconsistencies under challenging visual conditions.

Furthermore, as reported in A1, our independent audit of 80 generated videos confirms that 148 of 190 manually adjudicated candidate events correspond to genuine subject inconsistencies, yielding 77.89% event-level precision.

Q3: Robustness to threshold selection.

A3: Thank you for raising this question. We conducted two additional sensitivity analyses.

1. Model-Ranking Robustness Across 1,400 Videos

We evaluated all 1,400 generated videos, including 300 commercial 5s videos, 500 open-source 5s videos, 300 commercial 15s videos, and 300 long-form 60s videos, covering animals, pedestrians, vehicles, and soccer. We varied the general-purpose threshold from its default τ=0.35\tau = 0.35 to τ∈{0.30,0.35,0.40}\tau\in \{0.30, 0.35, 0.40\}. Each model was evaluated on 100 videos, and model rankings were compared within the same video category and duration.

Video SetModelsτ = 0.30τ = 0.35τ = 0.40
Commercial 5s31.00 / 1.001.00 / 1.000.50 / 0.33
Open-source 5s50.90 / 0.801.00 / 1.001.00 / 1.00
Commercial 15s31.00 / 1.001.00 / 1.001.00 / 1.00
Long-form 60s31.00 / 1.001.00 / 1.001.00 / 1.00

The results demonstrate strong ranking stability, particularly for longer videos. Most importantly, both 15s and 60s model rankings remain completely unchanged across all three thresholds, demonstrating that our main comparative conclusions for longer videos do not depend on the precise choice of τ=0.35\tau = 0.35. The 5s open-source rankings also remain highly consistent. While the commercial 5s ranking shows some sensitivity at τ=0.40\tau = 0.40, this involves only three models and does not affect the complete ranking stability observed in the longer-duration evaluations.

2. Robustness of RL Improvements to Threshold Selection

We further investigated whether the reported improvements from DynSC-based RL depend on the driving threshold τ=0.16.\tau = 0.16. Using the original Wan-2.1-1.3B Base and DynSC-RL models, we re-evaluated the same paired driving videos at τ∈{0.14,0.16,0.18}\tau\in \{0.14, 0.16, 0.18\} across 5s, 10s, and 30s durations. The following table reports paired mean SR differences, defined as ΔSR = SR(RL) − SR(Base). Negative values indicate improved consistency.

DurationτBase → DynSC-RLΔSR [95% CI] ↓
5s0.140.2629 → 0.2329−0.0300 [−0.0725, 0.0117]
5s0.160.2334 → 0.1992−0.0342 [−0.0769, 0.0074]
5s0.180.2073 → 0.1811−0.0263 [−0.0676, 0.0144]
10s0.140.4165 → 0.3635−0.0530 [−0.0899, −0.0149]
10s0.160.3767 → 0.3309−0.0458 [−0.0828, −0.0082]
10s0.180.3414 → 0.2934−0.0480 [−0.0835, −0.0115]
30s0.140.5061 → 0.4698−0.0363 [−0.0930, 0.0171]
30s0.160.4640 → 0.4476−0.0164 [−0.0725, 0.0382]
30s0.180.4220 → 0.4070−0.0150 [−0.0700, 0.0385]

Crucially, DynSC-RL consistently achieves lower SR than the Base model across all nine duration–threshold combinations. The direction of improvement never reverses when the threshold changes.

AuthorsOpenReview ↗

Rebuttal (3/5)

Q4: Distinguishing identity drift from legitimate viewpoint and pose changes.

A4: Thank you for raising this insightful question. We would like to clarify that DynSC-Eval distinguishes abrupt local inconsistencies from accumulated global appearance changes through complementary metrics. Local metrics (SR, ER, MAD) focus on sudden changes between nearby observations, while global metrics (ED, MRD, WFD) quantify appearance deviations accumulated along continuous subject trajectories. We do not assume that every global appearance deviation necessarily represents an identity inconsistency, as legitimate viewpoint and pose changes can also contribute to feature variation.

To systematically investigate this concern, we conducted two complementary sets of controlled 3D experiments, covering both legitimate appearance variations and deliberately injected inconsistencies.

1. Robustness to Legitimate Appearance Changes

We constructed controlled 3D rendering experiments involving vehicles and humans, keeping subject identity, geometry, and material properties fixed while varying only legitimate motion and viewing conditions. All videos were rendered at 8 seconds, 8 FPS, and 640×360640\times360 resolution, using the predefined threshold τ=0.35\tau=0.35.

  • Vehicle experiments: Three 3D vehicle models with fixed material configurations, testing rotations (0°, 45°, 90°, 180°), occlusion levels (0%, 25%, 50%, 75%), and illumination attenuation (0%, 25%, 50%, 75%), totaling 36 conditions.
  • Human experiments: Two 3D human identities undergoing viewpoint changes, translation, partial occlusion, and natural articulated motions, including walking, waving, and arm raising, totaling 32 conditions.
SubjectNormal VariationsConditionsSR ↓ER ↓Max. Adjacent Feature Drift ↓
VehicleRotation, Occlusion, Illumination36000.05258
HumanRotation, Translation, Occlusion, Articulation32000.14047
Total68000.14047

None of the 68 legitimate variation conditions triggered false local inconsistency events (SR = ER = 0), even under rotations up to 180°, occlusion up to 75%, and natural human articulation. All experiments contained valid subject trajectories, and the human experiments maintained successful detection and tracking across all frames.

Furthermore, the maximum adjacent DINOv3 feature distances were only 0.05258 for vehicles and 0.14047 for humans, substantially below the predefined threshold of 0.35. These findings demonstrate that legitimate continuous appearance changes do not necessarily trigger abrupt inconsistency detection.

2. Sensitivity to Genuine Inconsistencies

To verify that this robustness does not arise from an insensitive evaluation pipeline, we additionally introduced two types of controlled inconsistencies:

  • Severe Appearance Mutation: Abruptly replacing a four-wheeled passenger vehicle with a six-wheeled cargo vehicle and subsequently restoring it.
  • Sudden Disappearance/Reappearance: Abruptly removing a vehicle and its shadow and subsequently restoring them.

Across 12 videos containing 24 ground-truth inconsistency event boundaries, DynSC-Eval correctly detected all 24 without additional false positives, achieving 100% precision and recall under these controlled conditions.

Reviewer_t8pqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies subject consistency evaluation in long-video generation when the visible subject set changes through entry, exit, and occlusion. DynSC-Eval dynamically constructs eligible subject trajectories and uses DINOv3 object features to define three local metrics, SR, ER, and MAD, and three global metrics, ED, MRD, and WFD. These characterize anomalous events, short-term appearance variation, and long-term identity drift. The authors inject local and global errors into controlled videos of geometric objects to demonstrate sensitivity relative to VBench Subject Consistency, and evaluate multiple models at 5, 15, and 60 seconds. They then combine selected metrics into rewards and apply the existing DiffusionNFT algorithm to post-train Wan, SANA, and ReSim in an autonomous-driving testbed, extending the Wan experiments to 10 and 30 seconds. The paper reports improvements in all six DynSC-Eval metrics for the post-trained models, while the measured VBench quality and dynamics scores remain broadly similar to the baselines, with some small increases and decreases rather than uniform preservation.

Critical strengths

S1. The problem formulation addresses a practical gap in multi-subject long-video evaluation, and the metric decomposition has diagnostic value.

The paper shifts the evaluation unit from whole frames or a fixed initial set of subjects to the union of eligible subject sets over time, and distinguishes local disruptions from long-term drift (§3.1, Equations 1-2, and Appendix B.3). SR and ER respectively measure the fraction of tracks affected by errors and event frequency per unit of track time, rather than relying on a single average similarity. ED, MRD, and WFD capture different forms of long-term variation. The synthetic experiments use paired scenes, nested corruption sets, and three seeds, with separate local/global injections (§3.2 and Appendices B.4-B.5). This provides more interpretable evidence of complementarity than a model leaderboard alone.

S2. The evaluation-to-post-training experiments have reasonably broad coverage and explicitly examine some degenerate solutions.

The authors compare post-training against the same domain-adapted baselines, covering two T2V models and one I2V model, and demonstrate improvements in the same direction for 10-second and 30-second Wan models (Tables 3-4). The local-only, global-only, and combined-reward ablations are particularly useful: local-only optimization reduces Dyn from 92 to 85, whereas the combined reward maintains 92; global-only optimization worsens SR (Table 5). Appendix D.2 also includes a reference-regularization ablation. These results acknowledge and investigate the risk that consistency rewards suppress motion, making the application study more informative than a demonstration of improved self-evaluation scores alone.

Critical weaknesses

W1. The acquired PDF leaves an important gap between controlled sensitivity and validity on realistic generated videos, including after reward optimization.

The paper provides substantial relevant evidence: paired synthetic videos with clean conditions and three seeds (§3.2 and Appendices B.4-B.5), temporal pooling and boundary truncation (Appendix B.2), qualitative before/after examples (Figure 5), and reward/regularization ablations (Table 5 and Appendix D.2). My concern is the remaining transfer from those controls to real-video failure identification. The synthetic scenes do not establish how reliably normal occlusion, legitimate pose/illumination changes, and tracker errors are separated from generation inconsistencies. Within the acquired PDF, I did not locate a quantitative real-video event audit or independent human subject-consistency validation. The paper does explain that the lower driving threshold is intended to increase sensitivity under stricter consistency requirements (§4.3); what remains unclear is its empirical calibration and the robustness of the conclusions to threshold choice, not the stated motivation for using two thresholds.

For post-training, Equation 3 uses beta-smoothed SR, MAD, WFD, and MRD, and all six DynSC-Eval metrics share the track/feature pipeline. Improvements in ER and ED therefore provide additional statistics but do not independently validate that pipeline. Figure 5 and the separate VBench quality/motion measurements offer supporting evidence and should not be discounted. However, the reported measurements do not quantify whether the subject-consistency gains persist with comparable eligible-subject coverage and object motion. A whole-video dynamics score can also reflect camera/ego-motion. I have not established that subjects disappear, tracks fragment, or object motion is suppressed after optimization; these remain alternative explanations to check. A limited blind/event audit of existing paired outputs, accompanied by basic eligible-track coverage statistics, or equivalent independent validation, would substantially strengthen the claim. This is not a request for every possible diagnostic or new large-scale training.

W2. The incremental contribution over the closest object-level event evaluation needs a more precise comparison.

The quantitative comparison is principally against VBench (Figure 3, Table 2, and Appendix B.5). VMBench, published at ICCV 2025, already evaluates abnormal object appearance/disappearance using GroundedSAM2, CoTracker, and rules for legitimate transitions (§3.1 and supplementary Appendix B.5). Its published Equation 10 and the appearance/disappearance part of SR are related object-level anomaly aggregates. They are not established to be identical: DynSC-Eval adds mutation and global-drift measurements, uses its own eligible tracks and decision rules, and VMBench's inspected code averages separate appearance/disappearance scores rather than SR's union of event types. No VMBench citation was found in the submitted PDF. A pinned revision of the currently public VMBench implementation, with a commit dated March 12, 2025, also contains periodic re-detection and mask updates (lines 171-193 and 254-285). The publication establishes prior methodological precedent; the commit timestamp alone does not establish when that exact code first became public.

This supports a concern about the unquantified increment, not a conclusion that the paper is unoriginal or that the two frameworks are equivalent. DrivingGen, already cited by the authors, provides additional first-frame object-feature context; its missing numerical comparison is not a separate rejection reason. The paper would be stronger with a focused comparison against VMBench TCS on existing outputs, or a sufficiently precise analysis of what it misses. A same-detector/encoder fixed-versus-dynamic control could separately isolate the benefit of later-subject coverage. These are ways to resolve the same central question, not a mandatory list of all baselines or ablations.

Questions for authors
  1. [Metric validity] How are eligible tracks and compatible objects defined operationally, including a track that disappears through occlusion and later returns? Appendix B.2 already specifies temporal pooling and four-frame feature truncation; please clarify the additional event/association rules rather than repeat those settings. Is there a limited real-video event audit or other independent calibration supporting the selected thresholds and the perceptual discrimination claimed in §3.3? A representative subset of existing outputs would be sufficient to inform my assessment.
  2. [Incremental contribution] How does DynSC-Eval differ from VMBench's appearance/disappearance evaluation in subject discovery, event decisions, and aggregation, and what additional failures do its mutation/global-drift measurements reveal? Could a focused comparison on existing videos, or a precise mechanistic analysis, establish the increment? A same-backbone first-frame-only/dynamic-set control would be another useful way to isolate coverage benefits, rather than an additional required experiment.
  3. [Post-training validation and specification] Do the gains persist with comparable eligible-track coverage and object motion, and is there a limited independent paired assessment beyond the shared DynSC pipeline? The no-object penalty, Figure 5, Table 5, and Appendix D.2 are relevant existing controls. Please also specify the component-cost normalization and beta smoothing before Equation 3 and the reference-regularization formula/weight; the within-prompt standardization of rollout rewards is already described.
5 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

Q1: Validity on realistic generated videos and human perceptual evaluation

A1: We thank the reviewer for raising this concern. We emphasize that controlled synthetic experiments are essential to our work, as they directly validate our motivation and the effectiveness of DynSC-Eval under known consistency failures. In fact, our supplementary materials already include an anonymous webpage showcasing qualitative comparisons before and after RL fine-tuning, allowing direct inspection of the improvements in subject consistency. We encourage the reviewer to examine these examples.

We also agree that validating these findings on realistic generated videos is equally important. To further validate the perceptual reliability of DynSC-Eval, we conducted three complementary human evaluations: (1) manual verification of detected inconsistency events, (2) correlation analysis with human consistency ratings, and (3) human evaluation of improvements after RL fine-tuning.

1. Event Detection Precision

We randomly sampled 80 five-second generated videos covering four scenarios: animals, pedestrians, soccer, and driving. Human annotators manually inspected the inconsistency events detected by DynSC-Eval. Among 190 candidate events with definitive human judgments, 148 were confirmed as genuine subject inconsistency events, while 42 were identified as false positives, yielding an event detection precision of 77.89%.

VideosEvaluated EventsTrue PositivesFalse PositivesPrecision ↑
801901484277.89%

2. Correlation with Human Judgments

We further recruited 20 human participants to evaluate 20 fixed five-second videos generated by five models (Seedance 2.0, Kling V3, MiniMax H3, SANA-Video-2B, and Wan2.1-1.3B), using four shared prompts. Each video received four numerical consistency ratings on a 1–4 scale.

We calculated video-level Spearman correlations between human ratings and automatic consistency metrics. For DynSC-Eval, we reversed the signs of its error metrics so that higher values consistently indicate better subject consistency.

MetricVideo-level Spearman ρ ↑
DynSC-avg0.5605
VBench-SC0.3642
VMBench-TCS0.4786

DynSC-avg reports the mean of the component-wise Spearman correlations

DynSC-Eval shows stronger numerical agreement with human judgments than VBench Subject Consistency(Vbench-SC) and VMBench Temporal Coherence Score(VMBench-TCS) in this evaluation. These findings further support the perceptual relevance of our proposed metrics, beyond their sensitivity demonstrated in the synthetic experiments.

3. Human Verification of RL Improvements

We additionally evaluated whether improvements in DynSC-Eval translate into visually observable reductions in subject inconsistency after RL fine-tuning. For randomly sampled paired driving videos before and after RL, we manually counted subject inconsistency events across 20 pairs of 5s videos, 20 pairs of 10s videos, and 15 pairs of 30s videos.

DurationVideo PairsBefore RL ↓After RL ↓Reduction ↑
5s20291451.7%
10s20726213.9%
30s15977126.8%

We also conducted an independent human preference evaluation involving 20 participants. Participants evaluated subject consistency, motion naturalness, and visual quality. We report preference scores where our consistency optimization win, tie, and loss are assigned 1, 0.5, and 0, respectively, averaged equally across video pairs. Thus, 0.5 indicates no preference. All results support improvements in consistency and visual quality.

DurationConsistency ↑Motion Naturalness ↑Visual Quality ↑
5s0.713 [0.563, 0.850]0.675 [0.488, 0.838]0.638 [0.525, 0.738]
10s0.688 [0.550, 0.838]0.525 [0.363, 0.688]0.650 [0.550, 0.763]
30s0.821 [0.713, 0.917]0.704 [0.638, 0.767]0.650 [0.575, 0.713]

Overall, these complementary human evaluations support three important conclusions:

  1. Detection reliability: 77.89% of manually adjudicated detected events represent genuine subject inconsistency.
  2. Perceptual alignment: DynSC-Eval exhibits stronger numerical correlation with human consistency judgments than VBench Subject Consistency and TCS on the evaluated videos.
  3. Practical effectiveness: RL fine-tuning reduces manually identified inconsistency events across 5s, 10s, and 30s videos, with additional human preference evidence supporting perceptible improvements.
AuthorsOpenReview ↗

Rebuttal (2/4)

Q2: Robustness to threshold selection.

A2: Thank you for raising this concern. We conducted two additional sensitivity analyses: (1) model-ranking stability across all 1,400 generated videos, and (2) robustness of RL improvements across different thresholds.

1. Model-Ranking Robustness Across 1,400 Videos

We evaluated all 1,400 generated videos, including 300 commercial 5s videos, 500 open-source 5s videos, 300 commercial 15s videos, and 300 long-form 60s videos, covering animals, pedestrians, vehicles, and soccer. We varied the general-purpose threshold from its default τ=0.35\tau = 0.35 to τ∈{0.30,0.35,0.40}\tau\in \{0.30, 0.35, 0.40\}. Each model was evaluated on 100 videos, and model rankings were compared within the same video category and duration.

The following table reports Spearman ρ / Kendall τ correlations between model rankings at each threshold and those at the default threshold (0.35).

Video SetModelsτ = 0.30τ = 0.35τ = 0.40
Commercial 5s31.00 / 1.001.00 / 1.000.50 / 0.33
Open-source 5s50.90 / 0.801.00 / 1.001.00 / 1.00
Commercial 15s31.00 / 1.001.00 / 1.001.00 / 1.00
Long-form 60s31.00 / 1.001.00 / 1.001.00 / 1.00

The results demonstrate strong ranking stability, particularly for longer videos. Most importantly, both 15s and 60s model rankings remain completely unchanged across all three thresholds, demonstrating that our main comparative conclusions for longer videos do not depend on the precise choice of τ=0.35\tau = 0.35. The 5s open-source rankings also remain highly consistent. While the commercial 5s ranking shows some sensitivity at τ=0.40\tau = 0.40, this involves only three models and does not affect the complete ranking stability observed in the longer-duration evaluations.

2. Robustness of RL Improvements to Threshold Selection

We further investigated whether the reported improvements from DynSC-based RL depend on the driving threshold τ=0.16.\tau = 0.16. Using the original Wan-2.1-1.3B Base and DynSC-RL models, we re-evaluated the same paired driving videos at τ∈{0.14,0.16,0.18}\tau\in \{0.14, 0.16, 0.18\} across 5s, 10s, and 30s durations. The following table reports paired mean SR differences, defined as ΔSR = SR(RL) − SR(Base). Negative values indicate improved consistency.

DurationτBase → DynSC-RLΔSR [95% CI] ↓
5s0.140.2629 → 0.2329−0.0300 [−0.0725, 0.0117]
5s0.160.2334 → 0.1992−0.0342 [−0.0769, 0.0074]
5s0.180.2073 → 0.1811−0.0263 [−0.0676, 0.0144]
10s0.140.4165 → 0.3635−0.0530 [−0.0899, −0.0149]
10s0.160.3767 → 0.3309−0.0458 [−0.0828, −0.0082]
10s0.180.3414 → 0.2934−0.0480 [−0.0835, −0.0115]
30s0.140.5061 → 0.4698−0.0363 [−0.0930, 0.0171]
30s0.160.4640 → 0.4476−0.0164 [−0.0725, 0.0382]
30s0.180.4220 → 0.4070−0.0150 [−0.0700, 0.0385]

Crucially, DynSC-RL consistently achieves lower SR than the Base model across all nine duration–threshold combinations. The direction of improvement never reverses when the threshold changes.

AuthorsOpenReview ↗

Rebuttal (3/4)

Q3: Concerns regarding post-training effectiveness

A3: We thank the reviewer for this constructive suggestion. Our additional human evaluations (A1), optical-flow-based metrics, and subject coverage analysis provide complementary evidence that DynSC-RL improves perceptual consistency rather than merely optimizing the reward metrics.

1. Independent Validation of Consistency Improvements

As detailed in A1, our human evaluations show that manually identified subject inconsistency events decrease by 51.7%, 13.9%, and 26.8% for 5s, 10s, and 30s videos, respectively. Independent human preference evaluations further favor DynSC-RL, particularly for 30s videos, where the consistency preference score reaches 0.821 [0.713, 0.917].

We additionally evaluated the complete paired Wan-2.1-1.3B evaluation sets, comprising 100, 100, and 50 video pairs at 5s, 10s, and 30s, respectively. We used RAFT optical flow, independent of our Consistency reward, to measure bidirectional valid-region RGB warping errors.

DurationRegionBase → RL Warp L1 ↓Δ [95% CI]
5sFull0.01927 → 0.01754−0.00174 [−0.00221, −0.00127]
5sSubject0.03516 → 0.02873−0.00642 [−0.00872, −0.00431]
10sFull0.02018 → 0.01879−0.00140 [−0.00158, −0.00122]
10sSubject0.04129 → 0.03915−0.00214 [−0.00388, −0.00042]
30sFull0.03334 → 0.02607−0.00727 [−0.00970, −0.00489]
30sSubject0.05792 → 0.05520−0.00273 [−0.00642, 0.00112]

Full-region warping errors decrease significantly across all three durations, while subject-region errors also improve significantly at 5s and 10s. Independent VBench Subject Consistency further improves from 0.93964 to 0.94875 at 5s and from 0.91668 to 0.92826 at 10s, with both paired confidence intervals above zero.

Together with the human evaluations, these results provide independent evidence that the improvements are not confined to our reward formulation.

2. Subject Coverage and Motion Preservation

We further examined whether the improvements could be explained simply by reducing the number of evaluated subjects.

DurationEligible Tracks (Base → RL)Total Visible Track Duration (s)Global Flow (px/s)
5s8.28 → 8.3826.66 → 29.5333.27 → 31.61
10s13.35 → 12.7353.57 → 52.9738.27 → 31.82
30s23.44 → 18.14104.93 → 83.9766.76 → 56.51

For 5s and 10s videos, subject coverage remains broadly comparable before and after optimization. In particular, at 5s, the average eligible-track count slightly increases, while total subject visibility increases significantly by 2.87 seconds per video (95% CI [1.32, 4.51]). At 10s, total visible track duration changes only modestly (53.57 → 52.97 s). For 30s videos, although the number and total duration of eligible tracks decrease, this should not be directly interpreted as suppression of meaningful subject motion. Our objective is to maintain the consistency of plausible subjects, not to maximize the number of detected trajectories. In long-form generation, transient, visually implausible, or unstable subject appearances can themselves constitute generation artifacts. Qualitative inspection suggests that prominent scene participants remain visible after optimization, while some brief, implausible subject appearances are reduced.

Q4: Clarification of evaluation scope and definition.

A4: Thank you for raising this question. An "eligible track" is a confirmed subject trajectory with valid segmentation and sufficient feature observations for the corresponding metric: local metrics require consecutive observations supporting adjacent-window comparison, while global metrics require a longer sequence with valid, non-overlapping endpoint windows. A "compatible subject" is defined operationally through continuous tracking: as long as a subject remains visible within the frame and can be continuously detected and associated with the same track, its visual consistency is evaluated. Partial occlusions or appearance changes do not automatically terminate a track, provided that continuous tracking can be maintained.

When a subject leaves the frame or becomes fully occluded and tracking continuity is lost, its subsequent reappearance is treated as a new trajectory. We do not attempt to recover its previous identity across such discontinuities. This explicitly limits our evaluation to continuously observable subjects and avoids requiring long-term re-identification, which is a separate and more challenging problem.

Reviewer_L5BXOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a set of evaluation metrics DynSC-Eval. It first tracks all subjects in existing grounding DINO + SAM2 tracking videos, and then calculates different metrics for each trajectory using DINOv3 features. According to the author's claim, it has a higher discriminability than VBench. It also achieves improvements in specific domains when applied as rewards in the RL process of downstream tasks.

Critical strengths
  1. The proposed solution by the author is indeed helpful in alleviating the current field's evaluation pain points.
  2. The author further verifies the effectiveness of the proposed indicators through subsequent RL experiments, which is more convincing.
  3. The indicators designed by the author are reasonable, with clear motivation and appropriate methods.
Critical weaknesses
  1. Although the author used the proposed metrics as a reward and performed the RL process, I have a concern that the current results may be at risk of reward hacking. The reward uses the four metrics of DynSC, and Tab. 3–5 also evaluates using the six metrics of DynSC. Although the evaluation includes two additional categories not involved in training, they are essentially related to the reward signal in a similar way and are not completely decoupled. For the evaluation of the RL-trained model, I suggest adding more objective metrics (such as optical flow), subjective metrics, the Subject Consistency metric from VBench, or performance metrics on more downstream tasks to further enhance persuasiveness.

  2. While I think this work is meaningful, similar works [1,2] have also been published before. I hope the authors can include more comparisons with previous works. If there is time, I think comparing results using metrics from other datasets as rewards for RL would further enhance persuasiveness.

  3. The author claims that the metrics have better discriminability compared to VBench, but I have a concern that better discriminability does not necessarily mean accurate estimation. I think a experiments aligned with human preferences would better demonstrate the effectiveness of the metrics.

[1] VMBench: A Benchmark for Perception-Aligned Video Motion Generation [2] VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness

Questions for authors

Please refer to the critical weaknesses above.

4 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Q1: Alignment with human judgments and real-world validation

A1: We thank the reviewer for the constructive suggestion. In fact, our supplementary materials already include an anonymous webpage showcasing qualitative comparisons before and after RL fine-tuning, allowing direct inspection of the improvements in subject consistency.

To further validate the perceptual reliability of DynSC-Eval, we conducted three complementary human evaluations: (1) manual verification of detected inconsistency events, (2) correlation analysis with human consistency ratings, and (3) human evaluation of improvements after RL fine-tuning.

1. Event Detection Precision

We randomly sampled 80 five-second generated videos covering four scenarios: animals, pedestrians, soccer, and driving. Human annotators manually inspected the inconsistency events detected by DynSC-Eval. Among 190 candidate events with definitive human judgments, 148 were confirmed as genuine subject inconsistency events, while 42 were identified as false positives, yielding an event detection precision of 77.89%.

VideosEvaluated EventsTrue PositivesFalse PositivesPrecision ↑
801901484277.89%

2. Correlation with Human Judgments

We further recruited 20 human participants to evaluate 20 fixed five-second videos generated by five models (Seedance 2.0, Kling V3, MiniMax H3, SANA-Video-2B, and Wan2.1-1.3B), using four shared prompts. Each video received four numerical consistency ratings on a 1–4 scale.

We calculated video-level Spearman correlations between human ratings and automatic consistency metrics. For DynSC-Eval, we reversed the signs of its error metrics so that higher values consistently indicate better subject consistency.

MetricVideo-level Spearman ρ ↑
DynSC-avg0.5605
VBench-SC0.3642
VMBench-TCS0.4786

DynSC-avg reports the mean of the component-wise Spearman correlations

DynSC-Eval shows stronger numerical agreement with human judgments than VBench Subject Consistency(Vbench-SC) and VMBench Temporal Coherence Score(VMBench-TCS) in this evaluation. These findings further support the perceptual relevance of our proposed metrics, beyond their sensitivity demonstrated in the synthetic experiments.

3. Human Verification of RL Improvements

We additionally evaluated whether improvements in DynSC-Eval translate into visually observable reductions in subject inconsistency after RL fine-tuning. For randomly sampled paired driving videos before and after RL, we manually counted subject inconsistency events across 20 pairs of 5s videos, 20 pairs of 10s videos, and 15 pairs of 30s videos.

DurationVideo PairsBefore RL ↓After RL ↓Reduction ↑
5s20291451.7%
10s20726213.9%
30s15977126.8%

We also conducted an independent human preference evaluation involving 20 participants. Participants evaluated subject consistency, motion naturalness, and visual quality. We report preference scores where our consistency optimization win, tie, and loss are assigned 1, 0.5, and 0, respectively, averaged equally across video pairs. Thus, 0.5 indicates no preference. All results support improvements in consistency and visual quality.

DurationConsistency ↑Motion Naturalness ↑Visual Quality ↑
5s0.713 [0.563, 0.850]0.675 [0.488, 0.838]0.638 [0.525, 0.738]
10s0.688 [0.550, 0.838]0.525 [0.363, 0.688]0.650 [0.550, 0.763]
30s0.821 [0.713, 0.917]0.704 [0.638, 0.767]0.650 [0.575, 0.713]

Overall, these complementary human evaluations support three important conclusions:

  1. Detection reliability: 77.89% of manually adjudicated detected events represent genuine subject inconsistency.
  2. Perceptual alignment: DynSC-Eval exhibits stronger numerical correlation with human consistency judgments than VBench Subject Consistency and TCS on the evaluated videos.
  3. Practical effectiveness: RL fine-tuning reduces manually identified inconsistency events across 5s, 10s, and 30s videos, with additional human preference evidence supporting perceptible improvements.
AuthorsOpenReview ↗

Rebuttal (2/3)

Q2: Concerns regarding reward hacking and additional downstream evaluation metrics.

A2: We sincerely thank the reviewer for this insightful and constructive suggestion! We have also been seeking more convincing independent metrics to verify that our consistency improvements are not merely the result of reward hacking.

We additionally evaluated the complete paired Wan-2.1-1.3B evaluation sets, comprising 100, 100, and 50 video pairs at 5s, 10s, and 30s, respectively. We used RAFT optical flow, independent of our Consistency reward, to measure bidirectional valid-region RGB warping errors.

DurationRegionBase → RL Warp L1 ↓Δ [95% CI]
5sFull0.01927 → 0.01754−0.00174 [−0.00221, −0.00127]
5sSubject0.03516 → 0.02873−0.00642 [−0.00872, −0.00431]
10sFull0.02018 → 0.01879−0.00140 [−0.00158, −0.00122]
10sSubject0.04129 → 0.03915−0.00214 [−0.00388, −0.00042]
30sFull0.03334 → 0.02607−0.00727 [−0.00970, −0.00489]
30sSubject0.05792 → 0.05520−0.00273 [−0.00642, 0.00112]

Full-region warping errors decrease significantly across all three durations, while subject-region errors also improve significantly at 5s and 10s. Independent VBench Subject Consistency further improves from 0.93964 to 0.94875 at 5s and from 0.91668 to 0.92826 at 10s, with both paired confidence intervals above zero.

Together with the human evaluations, these results provide independent evidence that the improvements are not confined to our reward formulation.

Q3: Comparison with related works.

A3: Thank you for this constructive comment. DynSC-Eval differs from VBench-2.0 and VMBench in two fundamental aspects: full-video dynamic subject coverage and unified local–global consistency evaluation.

Unlike the first-frame-anchored human identity evaluation in VBench-2.0, DynSC-Eval continuously discovers and evaluates subjects throughout the entire video, including those entering the scene later. Moreover, our framework is not restricted to human identities but supports general subjects such as vehicles, animals, and pedestrians. Besides, while VMBench evaluates temporal coherence primarily through appearance/disappearance events (TCS) and motion-related structural deformation, DynSC-Eval explicitly combines local inconsistency detection with global appearance-drift measurements. This enables us to identify abrupt subject mutations as well as gradual identity-related appearance changes along continuous trajectories, even when no disappearance or reappearance occurs.

Therefore, our contribution is not simply another appearance similarity metric, but a systematic evaluation framework that jointly addresses dynamic subject coverage, abrupt local inconsistencies, and accumulated global appearance drift.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q4: Using metrics from other benchmarks as rewards for RL optimization.

A4: Thank you for your insightful comments. We conducted additional post-training experiments on Wan-2.1-1.3B, using VBench-SC and adapted WBench-SC as alternative training rewards.

For each alternative reward, we tested two β/γ configurations: (0.50, 0.05) and (0.75, 0.10), resulting in four additional training runs. All runs used the same training settings apart from the reward and specified hyperparameter configurations, with 75 optimization updates. All checkpoints were evaluated at step 75 using the same 100 held-out prompts, generation seeds, and inference settings.

Wan 5s Policyβ / γVBench-SC ↑Dynamic Videos / 100 ↑Global Flow (px/s)Imaging Quality ↑
Base—0.939649233.27450.71399
DynSC Reward0.50 / 0.050.948759231.60690.71818
VBench-SC Reward0.50 / 0.050.9974300.38020.68177
VBench-SC Reward0.75 / 0.100.9986500.33820.71132
Adapted WBench-SC Reward0.50 / 0.050.949678625.15570.71633
Adapted WBench-SC Reward0.75 / 0.100.973995010.84240.71968

First, VBench-SC reward optimization shows clear signs of reward hacking. Although both configurations achieve nearly perfect consistency scores (0.99743 and 0.99865), none of the 100 generated videos remain dynamic. In other words, the model achieves high consistency scores largely by suppressing motion rather than genuinely improving subject consistency.

Besieds, we observe a similar but less severe issue with adapted WBench-SC. As its consistency score increases from 0.94967 to 0.97399, the number of dynamic videos drops from 86 to 50 out of 100. In contrast, DynSC-RL maintains the same number of dynamic videos as the Base model (92/100), while still improving VBench-SC from 0.93964 to 0.94875.

Finally, DynSC-RL also preserves imaging quality. Its imaging quality score slightly improves over Base and is notably higher than the original VBench-SC reward configuration, with a difference of 0.03641. After tuning, the gap with VBench-SC becomes smaller, while tuned WBench-SC achieves slightly better imaging quality.

Overall, these results suggest that DynSC provides a better balance between subject consistency and motion preservation.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer L5BX,

We sincerely appreciate your constructive comments and positive assessment of our work. We hope that our detailed rebuttal and additional experimental results have adequately addressed your concerns. As the discussion period approaches its end, we would warmly welcome any further feedback or remaining questions.

Thank you for the time and appreciation that you have dedicated to our work.

Best regards,
Authors of Submission 14773

Reviewer_FCmTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DynSC-Eval for evaluating subject consistency over dynamically changing subject sets. It tracks subjects throughout their visible lifespans and defines six metrics for local inconsistency events and long-term identity drift. Validation includes controlled synthetic corruptions and evaluations across video generators and durations. The metrics are further used as rewards for DiffusionNFT post-training in autonomous-driving scenarios.

Critical strengths
  • Well-motivated evaluation setting: Tracking subjects that enter after the first frame addresses a concrete limitation of fixed-subject evaluation. Separating abrupt failures from long-term identity drift also provides more informative diagnostics than a single aggregate consistency score.
  • Useful controlled experiments: Injecting known inconsistencies at different rates provides a clear test of metric sensitivity. The local-only and global-only experiments further demonstrate the complementary responses of the proposed metrics. 14773_Towards_Subject_Consiste
  • Promising application to post-training: The reward improves the reported consistency scores across multiple T2V and I2V models, with further results at 10s and 30s. Reward-component and regularization ablations provide additional evidence for its practical utility within the driving domain.
Critical weaknesses
  • Unverified robustness to normal appearance changes: DINOv3 feature distances may change when a subject turns or is viewed from a different angle, without any identity error. Also, reasonable occlusion should not count as abnormal disappearance. The synthetic tests do not adequately examine these cases, and false-positive rates under legitimate changes are not reported.
  • Narrow validation of reward optimization: Although metric evaluation covers several domains, post-training is tested only on driving videos. Since driving scenarios involve less viewpoint/pose change, these experiments do not verify whether the reward remains reliable under large subject rotations or substantial viewpoint changes, where legitimate appearance variation could be penalized.
  • Underspecified tracking and event detection: The paper does not explain how 'compatible subjects' are identified, how object IDs are associated after occlusion, or how appearance changes affect tracking. These details are essential to distinguish genuine generation errors from tracker failures, which could introduce false events or reset identity references.
  • Missing comparisons with subject-level metrics: Comparisons focus on VBench SC, without directly evaluating subject-level metrics such as DrivingGen and WBench, both discussed in Appendix A. Such comparisons are needed.
  • No validation against human judgments: Greater sensitivity than VBench does not gurantee greater accuracy. The paper does not validate event detection against manual annotations or report agreement with human consistency ratings.
Questions for authors
  • How often does DynSC-Eval incorrectly penalize normal subject pose changes, viewpoint changes, or temporary occlusion?

  • Are post-training results available beyond driving, particularly for subjects undergoing substantial viewpoint changes?

  • How is a 'compatible subject' defined, and how are track ids maintained through occlusion and appearance changes? How to distinguish tracking failures from generation errors?

  • How does DynSC-Eval compare with existing subject-level metrics?

  • Can the authors provide event-detection precision/recall against manual annotations or correlation with human ratings?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Q1: Concern about robustness to normal appearance changes.

A1: Thank you for raising this valuable question. We conducted two additional sets of controlled 3D experiments: (1) Normal Variation Controls, which evaluate robustness to natural rotation, translation, occlusion, illumination changes, and human articulation; (2) Inconsistency Injection Controls, which assess sensitivity to genuine inconsistencies by deliberately introducing structural mutations and abnormal disappearance/reappearance events.

Firstly, we constructed controlled 3D rendering experiments involving both vehicles and humans, keeping subject identity, geometry, and material properties fixed while varying only legitimate motion and viewing conditions. All videos were rendered at 8 seconds, 8 FPS, and 640×360640\times 360 resolution, with the predefined threshold τ=0.35\tau = 0.35.

  • Vehicle experiments: We used three 3D vehicle models with fixed material configurations, testing rotations (0°, 45°, 90°, 180°), occlusion levels (0%, 25%, 50%, 75%), and illumination attenuation (0%, 25%, 50%, 75%), resulting in 36 experimental conditions.
  • Human experiments: We used two 3D human identities (one male and one female) to evaluate viewpoint changes, spatial translation, partial occlusion, and natural articulated motions, including walking, waving, and arm raising, across 32 experimental conditions.
SubjectNormal VariationsConditionsSR ↓ER ↓Max. Adjacent Feature Drift ↓
VehicleRotation, Occlusion, Illumination36000.05258
HumanRotation, Translation, Occlusion, Articulation32000.14047
Total-68000.14047

None of the 68 normal variation conditions triggered false inconsistency events (SR = ER = 0), even under rotations up to 180°, occlusion up to 75%, and natural human articulation. Moreover, DINOv3 features remained locally stable, with maximum adjacent feature distances of only 0.05258 for vehicles and 0.14047 for humans.

To further verify that the absence of false positives does not simply result from an insensitive detector, we introduced two types of explicit subject inconsistencies using the same 3D vehicle models and rendering environment:

  • Severe Appearance Mutation: Between 5 and 6 seconds, a four-wheeled passenger vehicle was abruptly replaced with a six-wheeled cargo vehicle and subsequently restored, simulating a sudden structural inconsistency.
  • Sudden Disappearance/Reappearance: The vehicle and its shadow were abruptly removed at 4.75 seconds and restored at 6.25 seconds, simulating an abnormal disappearance and reappearance event.

Each intervention was evaluated using three material configurations and two rotation conditions (0° and 180°), resulting in 12 videos containing 24 ground-truth inconsistency event boundaries.

Inconsistency TypeCorrectly DetectedFalse PositivesPrecision / Recall
Appearance Mutation120100% / 100%
Disappearance / Reappearance120100% / 100%
Total240100% / 100%

All 24 injected inconsistency event boundaries were correctly detected without additional false positives, achieving 100% precision and recall in these controlled experiments.

Q2: Limited generalization of reward optimization to scenarios with substantial viewpoint and pose changes.

A2: Thank you for raising this question. We would like to clarify that our consistency evaluation and reward formulation are general-purpose and not specifically designed for autonomous driving. We adopt driving as a representative testbed for post-training because it contains multiple moving subjects, frequent interactions, and continuous viewpoint changes. Our evaluation focuses on subject consistency within continuous visible trajectories in a single shot, rather than long-term identity re-identification.

Furthermore, our additional controlled experiments in A1 provide direct evidence of robustness to legitimate appearance variations. Our post-training results also provide empirical evidence that the reward does not simply suppress normal driving behaviors. Qualitative comparisons show that vehicles continue to perform natural turning, overtaking, and viewpoint changes after RL, while exhibiting improved appearance stability. We invite the reviewer to inspect these qualitative comparisons on the anonymous webpage provided in our supplementary materials. These results support the practical effectiveness of our reward.

AuthorsOpenReview ↗

Rebuttal (2/3)

Q3: Clarification of evaluation scope and definition.

A3: Thank you for raising this question. A "compatible subject" is defined operationally through continuous tracking: as long as a subject remains visible within the frame and can be continuously detected and associated with the same track, its visual consistency is evaluated. Partial occlusions or appearance changes do not automatically terminate a track, provided that continuous tracking can be maintained.

When a subject leaves the frame or becomes fully occluded and tracking continuity is lost, its subsequent reappearance is treated as a new trajectory. We do not attempt to recover its previous identity across such discontinuities. This explicitly limits our evaluation to continuously observable subjects and avoids requiring long-term re-identification, which is a separate and more challenging problem.

Q4: Missing Comparisons with Existing Subject-Level Consistency Metrics.

A4: Thank you for raising this question. In fact, we have already discussed and compared existing subject consistency evaluation methods in Lines 075–079 of the main paper and in the Related Works section. We would like to further clarify a fundamental difference between DynSC-Eval and existing subject-level metrics such as DrivingGen and WBench: the definition and coverage of the evaluated subject set. Specifically, DrivingGen initializes subject trajectories using first-frame detection boxes, while WBench initializes tracking using masks of designated subjects in the initial image or first frame. Under these evaluation protocols, subjects entering the scene later are not automatically incorporated into the evaluated set. In contrast, DynSC-Eval continuously discovers and tracks subjects throughout the video, allowing newly appearing subjects to be evaluated from the beginning of their own valid trajectories.

To quantify the importance of this distinction, we conducted a coverage analysis on 1,400 generated videos. We define first-frame coverage as the number of valid trajectories initialized in the first frame divided by the total number of valid trajectories identified throughout the video, aggregated over each evaluation set. The results reveal a substantial evaluation coverage gap for first-frame-based protocols, which becomes particularly pronounced in longer videos.

Video SetVideosFull-Video TracksFirst-Frame TracksFirst-Frame Coverage
Commercial 5s3004,1682,73865.69%
Open-source 5s5007,2403,09342.72%
Commercial 15s3007,6872,79036.30%
Long-form 60s30019,6692,39712.19%

This is precisely the motivation behind DynSC-Eval: subject consistency evaluation should evolve with the dynamic subject set throughout the video, rather than remain restricted to subjects visible at the beginning. By continuously incorporating newly appearing subjects, DynSC-Eval enables evaluation of their identity preservation, abrupt local inconsistencies, and subsequent appearance changes, providing substantially broader coverage, especially for long-form video generation.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q5: Human judgement.

A5: Thank you for this valuable suggestion. To further validate the perceptual reliability of DynSC-Eval, we conducted three complementary human evaluations: (1) manual verification of detected inconsistency events, (2) correlation analysis with human consistency ratings, and (3) human evaluation of improvements after RL fine-tuning.

1. Event Detection Precision

We randomly sampled 80 five-second generated videos covering four scenarios: animals, pedestrians, soccer, and driving. Human annotators manually inspected the inconsistency events detected by DynSC-Eval. Among 190 candidate events with definitive human judgments, 148 were confirmed as genuine subject inconsistency events, while 42 were identified as false positives, yielding an event detection precision of 77.89%.

VideosEvaluated EventsTrue PositivesFalse PositivesPrecision ↑
801901484277.89%

2. Correlation with Human Judgments

We further recruited 20 human participants to evaluate 20 fixed five-second videos generated by five models (Seedance 2.0, Kling V3, MiniMax H3, SANA-Video-2B, and Wan2.1-1.3B), using four shared prompts. Each video received four numerical consistency ratings on a 1–4 scale.

We calculated video-level Spearman correlations between human ratings and automatic consistency metrics. For DynSC-Eval, we reversed the signs of its error metrics so that higher values consistently indicate better subject consistency.

MetricVideo-level Spearman ρ ↑
DynSC-avg0.5605
VBench-SC0.3642
VMBench-TCS0.4786

DynSC-avg reports the mean of the component-wise Spearman correlations

DynSC-Eval shows stronger numerical agreement with human judgments than VBench Subject Consistency(Vbench-SC) and VMBench Temporal Coherence Score(VMBench-TCS) in this evaluation. These findings further support the perceptual relevance of our proposed metrics, beyond their sensitivity demonstrated in the synthetic experiments.

3. Human Verification of RL Improvements

We additionally evaluated whether improvements in DynSC-Eval translate into visually observable reductions in subject inconsistency after RL fine-tuning. For randomly sampled paired driving videos before and after RL, we manually counted subject inconsistency events across 20 pairs of 5s videos, 20 pairs of 10s videos, and 15 pairs of 30s videos.

DurationVideo PairsBefore RL ↓After RL ↓Reduction ↑
5s20291451.7%
10s20726213.9%
30s15977126.8%

We also conducted an independent human preference evaluation involving 20 participants. Participants evaluated subject consistency, motion naturalness, and visual quality. We report preference scores where our consistency optimization win, tie, and loss are assigned 1, 0.5, and 0, respectively, averaged equally across video pairs. Thus, 0.5 indicates no preference. All results support improvements in consistency and visual quality.

DurationConsistency ↑Motion Naturalness ↑Visual Quality ↑
5s0.713 [0.563, 0.850]0.675 [0.488, 0.838]0.638 [0.525, 0.738]
10s0.688 [0.550, 0.838]0.525 [0.363, 0.688]0.650 [0.550, 0.763]
30s0.821 [0.713, 0.917]0.704 [0.638, 0.767]0.650 [0.575, 0.713]

Overall, these complementary human evaluations support three important conclusions:

  1. Detection reliability: 77.89% of manually adjudicated detected events represent genuine subject inconsistency.
  2. Perceptual alignment: DynSC-Eval exhibits stronger numerical correlation with human consistency judgments than VBench Subject Consistency and TCS on the evaluated videos.
  3. Practical effectiveness: RL fine-tuning reduces manually identified inconsistency events across 5s, 10s, and 30s videos, with additional human preference evidence supporting perceptible improvements.
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer FCmT,

We hope that our detailed rebuttal and additional experimental results have adequately addressed your concerns. As the discussion period approaches its end, we would warmly welcome any further feedback or remaining questions. We sincerely hope that you could reconsider your rating in light of the additional evidence and clarifications.

Thank you again for your time and thoughtful consideration. We look forward to hearing from you.

Best regards,
Authors of Submission 14773

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.