- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies concurrent action omission in multi-action, multi-subject video generation, where multiple actions are requested for the same subject at overlapping time intervals. The authors analyze two sources of failure, inter-event interference in the text representation and intra-event action competition in video-text attention, and propose CoRAct, a training-free method that addresses them through event-anchored subject representations and subject-anchored concurrent-action guidance. They also introduce an 84-prompt benchmark with action-level temporal annotations and an evaluation protocol that measures whether each designated subject performs each action for a sufficient portion of its requested interval. Experiments are conducted on three video-generation backbones in both image-to-video and text-to-video settings.
Critical strengths
-
The paper tackles an important problem of multi-action, multi-subject video generation.
-
The paper raises an important definition of "action omission" and tries to tackle it from two dimensions, while providing an end-to-end evaluation protocol specifically targeting this problem.
Critical weaknesses
-
In the definition of Omission Rate in Appendix C, the authors state that "A performer reported as unclear is not counted as another subject, so o_a = 0 in that case", which is quite confusing and possibly erroneous. By standard, such cases should at least be excluded from the final results, and if such cases are frequent, the reliability of the rating model (Qwen) should be questioned. This also contradicts the definition of omission given by the authors in Line 134, i.e., "An action is omitted when its designated subject does not perform it over most of its interval".
-
Typically, concurrent actions should be performed in different parts of the frame. For instance, in Figure 1, "removing glasses" happens in the face region of the subject, while "returning a high five" happens in the hand region of the subject, and I see no reason why the attention for these concurrent actions should compete with each other if an ideal model is presented. In addition, some actions are inherently more complicated than others and thus require more attention support; manually raising the support of all concurrent actions equally may therefore harm the generation of such events.
-
The evaluation is unconvincing. First, the provided qualitative samples are far from abundant. Nevertheless, from Figure 4 first row (CoRAct), neither sample is satisfying. In the left sample, the subject does not turn back to the audience, while in the right sample, there are severe artifacts where the head of the subject turns back while its flower-holding arm does not, both of which violate the provided prompt. Therefore, I do not think these actions should be assessed as successful in the evaluation. Considering that these samples have been manually selected, this makes the overall evaluation even less convincing. This also raises questions about the qualification of the MLLM automatic evaluator used in the paper. I believe human-based evaluation is necessary for the problem setup in this paper, which is also lacking.
-
The writing is poor, and some notations and sections need to be clarified. Overall, I find the paper difficult to follow, and several parts are quite confusing. For example, the distinction among actions, events, and co-required actions is not always clear, and the mathematical formulation in Sections 3.1 and 3.2 introduces many notations in rapid succession, making the proposed method difficult to follow. In particular, the definition of the "action direction" through a Jacobian-vector product and the subsequent construction of the subject representation are not sufficiently intuitive. Some claims are also stated too strongly, such as "The support also decides the outcome" in Section 2.4. The definitions and relationships among OM, OM_P, and OM_S also require more clarification.
-
In Table 4, why do the statistics for backbone Wan2.2 A14B not match the statistics in Table 2 under image-to-video and mixed split?
Questions for authors
Please refer to Weaknesses
3 comments
I thank the authors for their detailed response. Although multi-action generation is an interesting and important problem, I do not think the paper is ready for publication yet.
-
Poor qualitative results. Neither CoRAct nor the baselines appear to successfully execute the given text prompts. While the authors argue that the baselines perform worse, this does not establish that CoRAct effectively solves the problem. Perhaps the authors could consider a simpler setting. Additionally, I could not find the additional video samples mentioned in the response, despite checking both the main paper and the supplementary materials.
-
The human evaluation involves only nine annotators and ten unique prompts, which is insufficient to establish robust conclusions. Moreover, the 83 evaluated actions are not independent, as multiple actions belong to the same prompt. This raises concerns about the validity of the reported statistical significance.
-
I remain unconvinced on Weakness 2 by the authors' response. In many cases, such as the example in Figure 1, concurrent actions should occur in different spatial regions of the frame. Furthermore, prioritizing the least-supported action may inevitably reduce the model's attention to other, potentially more complex actions.
Overall, I will maintain my original rating.
We thank the reviewer for the follow-up and for the time spent on our response.
1. Qualitative results. We agree that the prompts are not executed in full by any method. The qualitative results show where the methods differ. The baselines can place an event at the right time and still drop one of its concurrent actions or let another subject perform it, as with the sheet music and the glasses in Fig. 1. In the same example, CoRAct performs both on the designated subjects, which is the failure the paper targets, and the larger human study below agrees with this. The method is new in two respects. It changes the representation of a subject from one event to the next, using the response of the text encoder to the actions of its current event. It also ties the attention support of several actions to one subject and one interval, whereas temporal binding methods decide when each part of the prompt is read and attention guidance for images raises a neglected concept anywhere in a frame. We apologize that the additional video samples could not be found.
2. Human evaluation. We additionally ran a user study after the response. Twenty-five annotators made 1,348 judgments of 260 actions in the videos of four methods on ten prompts, and for every action they marked whether it appears, who performs it, and over which frames. The videos of CoRAct come from the current version of our objective. Under human judgment, 13.3% of the requested actions never appear with CoRAct, against 26.6% with SwitchCraft, 34.8% with TS-Attn, and 56.6% with TIE. The differences are 13.3, 21.6, and 43.3 points. These intervals come from a bootstrap that resamples prompts, so the actions of one prompt stay together and their dependence is accounted for. CoRAct performs 49.9% of the actions as requested, against 38.6%, 32.7%, and 22.1%, and its video is preferred in 28 of 50 choices.
3. Spatial regions of concurrent actions. We agree that concurrent actions often occupy different parts of one subject, such as the mouth and each hand in Fig. 1. Our support takes, for each action separately, the maximum over the locations of its subject, so two actions can be supported at different parts of the same subject. What it does not do is assign an action to a specific part. Doing so requires knowing the body parts of the subject, which restricts the method to human subjects and to image-to-video generation, where the first frame provides them. We therefore leave part-level support to future work, for example by tracking parts across frames through the self-attention of the video model. We also agree that emphasizing the least supported action can take attention from the others. The emphasis is soft, so every co-required action keeps a positive weight, and a part-level formulation would reduce this trade-off further.
We thank the reviewer again for the careful reading. These comments will shape the next version of the paper.
Response to Reviewer 8kvN
We thank the reviewer for the detailed reading. The review questions the soundness of the evaluation. The human evaluation it asks for is already part of the paper, and a negative control now tests the judge directly. The remaining points leave every reported number unchanged, so the findings stand, including the significant reduction of absent actions over existing training-free methods.
W1. Unclear performers. When the judge cannot tell who performs an action, it reports the performer as unclear. Appendix C does not count such an action as performed by another subject, since that requires evidence of a different performer, and the action must still appear and cover its interval. It occurs only on the sequential split, for 0.6% of the judged actions, mostly when only the performing hands are visible. Counting these actions as omitted instead would raise the sequential omission rate by at most 3.2 points, with similar increases for every method, so the ranking is unchanged.
W2. Why concurrent actions compete. Actions of different body parts often share a region, as when a subject smiles while taking off glasses, and attention is not confined to the performing body part, so a broad, dominant action can take most of the subject's attention. As Section 2.4 measures, when a second action is requested together with an action, its support falls to 0.78 of its value alone, below one in 35 of 47 cases. If attention were split by body part, the support of the first action would not change. The guidance does not raise all actions equally either. Its objective is a soft minimum over the co-required actions in each frame (Eq. 4), so it lifts the action with the least support and gives little weight to actions that are already well supported.
W3. Qualitative results and human evaluation. In the left example of Figure 4, TS-Attn, SwitchCraft and TIE all leave out covering the mouth, and TS-Attn claps instead of waving, while CoRAct covers the mouth and waves. Turning back is missed by every method, CoRAct included, and in the right example CoRAct turns the head but leaves the arm with the flower still. The revised caption describes these cases, and we add randomly sampled examples.
The human evaluation (Section 4.1, Appendix C) judges every action with the frames it covers, under the same 70% rule as the judge. Annotators find more actions performed as requested with CoRAct than with any other method (Table A), significantly more than with TS-Attn and TIE. Its videos come from Wan2.2 A14B I2V, as the revised Section 4.1 specifies, except TIE, which exists only for Wan2.2 TI2V-5B. Against the annotated frames, our judge also matches the performed intervals far better than StoryEval (IoU .53 against .30), a significant difference over prompts.
Table A. Human evaluation, share of actions performed as requested (%).
A negative control tests the judge itself (Table B). It reports unrequested actions far less often than requested ones, stating them in the prompt does not raise this rate, and it is no more lenient to CoRAct than to the backbone. Re-judging the same videos reproduces 99.2% of labels.
Table B. Negative control on Wan2.2 TI2V-5B I2V, 912 queries (% reported present).
W4. Writing and notation. Table C restates the terms of the revised Section 3.
Table C. Terms and metrics.
The action direction is the frozen encoder's response on the subject row to adding the verb and object embeddings, one Jacobian-vector product per action (Eq. 1), and the text-side edit rotates the subject row toward their bounded average, keeping its norm (Eqs. 2 and 3).
W5. Table 4 and Table 2. Table 4 reports all 84 prompts, as the revised caption states. Weighting the two splits of Table 2 by size reproduces Table 4 (Table D), so the tables agree.
Table D. Wan2.2 A14B I2V, Table 2 against Table 4.
We hope the clarified evaluation and the added results address the concerns of the review, and we are glad to clarify anything further.