- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MTS-Bench tests whether video generators produce the spatial changes implied by an action, rather than just showing the action. It contains 337 paired I2V/T2V cases and evaluates nine I2V and eight T2V generators on a 150-case subset. Videos are scored along five dimensions using a VLM evaluator, with human ratings used for validation and calibration.
Critical strengths
- The distinction between an action and its spatial outcome is worth studying. Examples such as opening a drawer and relative vehicle motion make the problem clear. The case-specific criteria also allow reasonable variations instead of requiring one reference trajectory.
- I found the evaluator analysis useful. In particular, a scorer can rank generators similarly to humans while struggling to distinguish individual videos. Reporting this limitation helps readers judge which comparisons are trustworthy.
Critical weaknesses
My main concern is the reliability of the evaluation. The human reference is based on a single annotator, and calibration cannot resolve potential errors in the annotator’s judgments. In addition, the average IF–SCP gap does not show how often a video correctly executes the requested action but fails to realize its spatial consequences. These limitations make it difficult to fully assess the reliability of the reported scores and the conclusions drawn from them.
Questions for authors
-
Among videos where the requested action is successfully executed, how often does the expected spatial consequence still fail to occur? An analysis of IF and SCP on the same videos would help distinguish failures in action execution from failures in realizing the expected spatial change.
-
Have the SCP ratings been checked by a second independent annotator? It would be useful to know whether different annotators agree on the expected spatial consequences and how well the generated videos realize them.
-
Does the IF–SCP gap remain when both dimensions are evaluated on the same videos using raw scores, and how does it change after calibration? Since IF uses raw scores while SCP is calibrated, this comparison would help clarify whether the reported gap is affected by the calibration procedure.
3 comments
Thanks for this careful response, which addresses a large part of my concern. I maintain my score.
Thank you for your Positive feedback
Dear Reviewer snkh,
Thank you very much for your encouraging comments and thoughtful consideration of our rebuttal. We truly appreciate your support and favorable recommendation.
Best regards,
The Authors
Rebuttal by Authors
We sincerely thank Reviewer snkh for the thoughtful and constructive feedback. We appreciate your recognition of the importance of distinguishing actions from their spatial consequences, the flexibility of our case-specific rubrics, and our careful analysis of evaluator reliability, particularly the distinction between generator-level ranking and video-level discrimination. In the following response, we provide experiments to address your major concern.
A1. We thank the reviewer for this helpful suggestion.We directly examine IF and SCP within the same video, conditioning on fully successful instruction execution (IF = 3). We report two thresholds: SCP ≤ 1, indicating critical failure or limited success, and SCP ≤ 2, capturing all cases where spatial consequences are not fully realized.
Our results show that successful instruction execution does not necessarily imply correct spatial consequence realization, providing direct video-level evidence of the action–consequence gap.
A2. Thank you for pointing this out. Yes, two additional annotators independently scored the same uniformly sampled subset of 150 videos (75 per modality), covering all applicable dimensions. We compute ordinal Krippendorff’s α jointly from A/B/C ratings and average QWK over the three annotator pairs, exact agreement and within-one-point agreement. These results quantify the reproducibility of the final scores across independent annotators, as reported in Sec. 5.4 and Appendix B.3 (Table 8).
A3. We thank reviewer for your helpful suggestion. To examine the effect of calibration, we report the fully raw IF–SCP gap with the deployed calibrated gap on the same matched videos.
The raw IF–SCP gaps remain positive but are relatively small, while calibration increases their magnitude. As described in Sec. 4.3, our proposed calibration is fitted to human ratings for each dimension independently, rather than to the IF–SCP gap itself. Its purpose is to improve numerical agreement with human judgments. And our human-rated analysis in A1 provides complementary evidence independent of calibration. Among videos with fully successful instruction following (IF = 3), 25.4% of I2V and 22.2% of T2V videos receive SCP ≤ 1. These findings support that successful action execution does not necessarily imply correct spatial consequence realization.