acceptodds
Under review as a conference paper at ICLR 2027

DUOSTATE: EVALUATING SELECTIVE STATE CONTROL IN VIDEO GENERATION

Abstract

Following a video script requires more than keeping characters recognisable: the intended person must change, the other must remain unchanged, and a later instruction may require restoring an earlier state. We introduce CharacterLedgerBench, a framework for evaluating this selective state control. A paired protocol tests four branches of the same two-character scene: no edit, an edit to either character, and an edit followed by restoration after occlusion. A region-grounded evidence-to-ledger judge tracks the characters, records visible attributes from timestamped regions, and checks their state changes against the script. It distinguishes recognition of the requested states from completion of the full process in the prescribed order and time windows. On an independent diagnostic set, regional evidence and ledger-based rules improve success-detection F1 from 0.653 to 0.869 over direct full-frame judging. Strict verification accepts only 5.0% of the benchmark's original reversal clips. Providing enough time for every stage and matching edits across roles improves some results, but state control remains difficult. CharacterLedgerBench connects these failures to specific characters and events, establishing an evaluation target beyond visual consistency.

Primary area:
datasets and benchmarks
Keywords:
video generation, image-to-video generation, selective state control, character consistency, role-bound evaluation, temporal evaluation, datasets and benchmarks, vision-language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gie2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DuoState, a benchmark of 30 scenes with two characters and four instruction branches for evaluating selective state control in video generation. The authors also propose a judge that combines regional visual evidence, VLM observations, a state ledger, and temporal rules. Experiments evaluate four video generation models and compare the judge with direct assessment of full frames.

Critical strengths

The paper aims to study a worthwhile problem: changing the states of selected people while preserving the states of others. This is an good aspect of video controllability that identity consistency alone cannot fully capture.

Critical weaknesses
  1. Limited benchmark contribution. Selective state control is a general problem, yet DuoState reduces it to only 30 scenes involving exactly two people and a rigid four branch protocol. The paper does not establish why this narrow design is necessary or representative of state control in video generation. The reversal branch also requires occlusion, although it is unclear why occlusion should be necessary for evaluating state restoration. The paper does not clearly specify whether the provided images are used as reference images or first frames. However, the model input configurations and qualitative results suggest that they serve as first frames. These images are constructed by concatenating two separately generated portraits, which is very likely to introduce unnatural spatial relationships and inconsistent backgrounds, potentially interfering with the evaluation.
  2. Unclear novelty, unfair comparisons, and room for improvement in presentation. Related benchmarks such as OSCBench (Han et al., ACL 2026) and TC-Bench (Feng et al., ACL 2025 Findings) already evaluate state changes and temporal transitions, but are not discussed. The additional value of DuoState is unclear. Model comparisons are also confounded by different generation settings. The readability is relatively poor. For example, Excessive tags and data details like index number in Figure 2 and 4 make them difficult to read.
Questions for authors
  1. Why is selective state control restricted to two characters and four fixed branches, and why is occlusion required for reversal? These constraints seem unnecessarily restrictive for evaluating a general state control capability.
  2. Are the concatenated character images used as reference images or first frames? The authors should rule out potential interference from unnatural spatial relationships and inconsistent backgrounds, which may confound the evaluation of state control.
  3. What additional evaluation capabilities does DuoState provide beyond OSCBench (ACL 2026) and TC-Bench (ACL 2025 Findings)? The overlap with existing benchmarks and the inconsistent generation settings make it difficult to assess the significance of the reported results.
Reviewer_cTtcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper conducted an evaluation regarding selective state control in video generation tasks. It evaluates video generative models' ability to properly register attributes to different subjects according to the instruction, and the ability to restore previously modified attributes after occlusion. The authors performed the evaluation on several existing methods and give some analysis based on the results.

Critical strengths
  1. The 4-branch design captures the intended cases (changing attribute of a certain subject / person and keep the others fixed, restoring attributes after occlusion, etc.)
  2. The dimensions of the evluation are reasonable and comprehensive.
Critical weaknesses
  1. The manuscript is hard to follow, and contains unclear or confusing information. For instance, at line 184, what does mean "Each item’s stored prompt"? In Figure 6, some descriptions (e.g., f and h) does not align with what's displayed in the figure.
  2. The prompts given to the generators are unclear. The paper does not explicitly state what prompts are used, and there is no example and no detailed explanation. In addition, the scripts shown in the examples (e.g., Figure 2) do not seem to be proper prompts that align with human input. This raises the question of whether the prompts used in the evaluation align with typical inputs of the generators.
  3. The dataset used for evaluation seems to lack variation and realism. It is limited to 2 person videos, and most examples given in the paper depict two standing figures in front of a grey background. These data lack variation and do not reflect typical use of video generation models.
  4. The attribute classification (color, position, and status) is coarse and unbalanced, and its motivation is unclear. The status seems to be an umbrella term covering a wide variety of attributes, whereas color and position are clear and concentrated concepts. Furthermore, some attributes has ambiguous classification. For instance, "putting glasses on/off", which is an example of status change (line 161) can also be expressed as a position change. The paper does not explain why it is necessary to classify attribute changes in this study, nor what insights can be obtained by analyzing them. It is also unclear how this coarse classification is defined and used in the evaluation.
  5. The scale of the dataset is too small (30 scenes, 120 cases) for reliably evaluating performance of modern video generation models. This scale is too small compared to, e.g. VBench (introduced at line 103 of the main paper).
Questions for authors
  1. (W2) The prompts used in the evaluation need clarification. Please address the following questions: What prompts are given to the generators? Are there examples of the prompts in the paper or appendix? How are the prompts written or collected, and do they match typical human inputs to video generation models? Are the scripts in Figure 2 used as prompts, or are they converted into prompts? If they are converted, how is the conversion done? Are the same prompts used for all generators? How are prompts about the occlusion / occluder formulated?
  2. (W3) The variation and realism of the evaluation dataset are unclear. Please address the following questions: How was the evaluation dataset collected or constructed? Why do most examples show two standing figures in front of a grey background? How were these examples selected? Do the conclusions still hold on more diverse and realistic videos that reflect typical use of video generation models?
  3. (W4) The motivation and design of the attribute classification are unclear. Please address the following questions: Why is it necessary to classify attribute changes into color, position, and status in this study? What insights does this analysis provide that cannot be obtained from other evaluation metrics? Does the imbalance between attribute classes affect the results and conclusion? How is the ambiguity of classification handled?
Reviewer_Z3A1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DuoState, a benchmark for evaluating selective state control in two-character image-to-video generation. It uses four matched branches per scene to evaluate whether models can preserve both characters' states, change only the target character's state, and restore that state after occlusion when required. The authors also propose a region-grounded evaluator that tracks character states and verifies the requested temporal sequence. Experiments on 480 videos from four generators show that identity preservation does not guarantee successful state control. The evaluator improves success-detection F1 from 0.653 to 0.869 on a separate diagnostic set, while strict verification finds that only 6 of 120 original reversal clips complete the required sequence.

Critical strengths
  • The matched four-branch design provides a useful way to evaluate selective state control. It separates identity preservation, update execution, role attribution, and restoration under shared scene conditions. For example, CogVideoX preserves identity in 58/60 intervention clips but executes only 4/60 requested updates, showing why these distinctions matter. The duration and symmetric-edit controls further strengthen the comparisons by accounting for differences in temporal coverage and edit difficulty.

  • The evaluator is supported by human validation and controlled comparisons. On the same 200 diagnostic videos, regional crops improve success-detection F1 from 0.653 to 0.772, while adding the ledger and temporal rules raises it to 0.869. The authors also evaluate agreement with human annotations and test whether the structured assessment improves results across different VLM backends.

Critical weaknesses
  • The temporal metadata and generation settings need clarification. Table 15 reports that the CogVideoX q029 clip has 49 frames at 16 fps, corresponding to roughly 3 seconds, while Figure 8 shows timestamps extending to 5 seconds. This inconsistency raises concerns about the timing information used for evaluation. The paper also does not clearly explain how the full-duration controls were generated or which settings were held fixed, making it difficult to attribute the increase from 6 to 11 strict reversal successes to additional temporal coverage alone.

  • The validation of strict script-level judgments remains incomplete. Table 21 reports agreement with human annotations for the attribute-level conjunction, but Table 22 does not separately compare automatic and human verdicts under the stricter temporal requirements. The construction and selection of the 200-video diagnostic set are also not sufficiently described, making it difficult to assess how well the reported F1 improvement generalizes to the main benchmark.

  • The presentation is somewhat repetitive and overly cautious. The paper repeatedly emphasizes the limits of its evaluation claims, particularly around attribute-level and strict temporal verification (e.g., lines 428-429). Some of these explanations could be shortened to make the main findings clearer.

Questions for authors
  • Table 15 reports 49 frames at 16 fps for the CogVideoX q029 clip, while Figure 8 shows timestamps extending to 5 seconds. How do the authors explain this discrepancy? Were the full-duration control clips regenerated, retimed, or extended, and which generation settings were kept fixed? Please clarify whether the timestamp inconsistency affects the timing-based results.

  • Does Table 22 report independent agreement between automatic and human strict verdicts, or the results of final human adjudication? Could the authors provide the corresponding confusion counts and clarify whether any of the six accepted clips involve disputed non-target preservation judgments?

  • How were the 200 diagnostic videos selected, and how were their human reference labels established? Please also clarify whether these videos were independent of evaluator development and whether the three judge configurations used comparable evaluation budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.