acceptodds
Under review as a conference paper at ICLR 2027

Mind the Space: Benchmarking Spatial State Evolution in Video World Models

Abstract

A generated video can depict the requested action while failing to realize its spatial consequences. We introduce MTS-Bench (Mind the Space Benchmark), which evaluates this distinction through interventions whose task-supported spatial consequences are not fully narrated in the prompt. Its 337 curated cases pair image-to-video (I2V) and text-to-video (T2V) formulations across scene, view, and coupled motion. Case-specific rubrics assess initial-state fidelity, instruction execution, consequence prediction, temporal consistency, and endpoint completion. We evaluate nine I2V and eight T2V generators on a fixed 150-case subset, using a VLM scorer audited against blinded human ratings and selectively calibrated by dimension. The highest-scoring generator obtains 80.2 on instruction following and 56.2 on consequence prediction in I2V; the corresponding T2V scores are 77.2 and 62.4 (out of 100). Matched qualitative examples expose action without the required spatial change, while dimension profiles distinguish generators with similar aggregate scores. Evaluator analysis further shows that strong agreement with aggregate rankings can coexist with weak discrimination between videos. MTS-Bench makes intervention-induced spatial evolution an explicit, diagnosable evaluation target for video world models.

Primary area:
datasets and benchmarks
Keywords:
Video World Models, Video Generation, Spatial Reasoning, Dynamic Spatial Evolution, Benchmarking, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_snkhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MTS-Bench tests whether video generators produce the spatial changes implied by an action, rather than just showing the action. It contains 337 paired I2V/T2V cases and evaluates nine I2V and eight T2V generators on a 150-case subset. Videos are scored along five dimensions using a VLM evaluator, with human ratings used for validation and calibration.

Critical strengths
  1. The distinction between an action and its spatial outcome is worth studying. Examples such as opening a drawer and relative vehicle motion make the problem clear. The case-specific criteria also allow reasonable variations instead of requiring one reference trajectory.
  2. I found the evaluator analysis useful. In particular, a scorer can rank generators similarly to humans while struggling to distinguish individual videos. Reporting this limitation helps readers judge which comparisons are trustworthy.
Critical weaknesses

My main concern is the reliability of the evaluation. The human reference is based on a single annotator, and calibration cannot resolve potential errors in the annotator’s judgments. In addition, the average IF–SCP gap does not show how often a video correctly executes the requested action but fails to realize its spatial consequences. These limitations make it difficult to fully assess the reliability of the reported scores and the conclusions drawn from them.

Questions for authors
  1. Among videos where the requested action is successfully executed, how often does the expected spatial consequence still fail to occur? An analysis of IF and SCP on the same videos would help distinguish failures in action execution from failures in realizing the expected spatial change.

  2. Have the SCP ratings been checked by a second independent annotator? It would be useful to know whether different annotators agree on the expected spatial consequences and how well the generated videos realize them.

  3. Does the IF–SCP gap remain when both dimensions are evaluated on the same videos using raw scores, and how does it change after calibration? Since IF uses raw scores while SCP is calibrated, this comparison would help clarify whether the reported gap is affected by the calibration procedure.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank Reviewer snkh for the thoughtful and constructive feedback. We appreciate your recognition of the importance of distinguishing actions from their spatial consequences, the flexibility of our case-specific rubrics, and our careful analysis of evaluator reliability, particularly the distinction between generator-level ranking and video-level discrimination. In the following response, we provide experiments to address your major concern.

Q1. How often does a video successfully execute the instructed action but fail to realize its spatial consequences?

A1. We thank the reviewer for this helpful suggestion.We directly examine IF and SCP within the same video, conditioning on fully successful instruction execution (IF = 3). We report two thresholds: SCP ≤ 1, indicating critical failure or limited success, and SCP ≤ 2, capturing all cases where spatial consequences are not fully realized.

Scoring sourceModalitySCP ≤ 1 given IF = 3SCP ≤ 2 given IF = 3
SolI2V29/607 (4.8%)120/607 (19.8%)
SolT2V56/464 (12.1%)94/464 (20.3%)
HumanI2V29/114 (25.4%)70/114 (61.4%)
HumanT2V16/72 (22.2%)36/72 (50.0%)

Our results show that successful instruction execution does not necessarily imply correct spatial consequence realization, providing direct video-level evidence of the action–consequence gap.

Q2. Have the SCP ratings been checked by a second independent annotator?

A2. Thank you for pointing this out. Yes, two additional annotators independently scored the same uniformly sampled subset of 150 videos (75 per modality), covering all applicable dimensions. We compute ordinal Krippendorff’s α jointly from A/B/C ratings and average QWK over the three annotator pairs, exact agreement and within-one-point agreement. These results quantify the reproducibility of the final scores across independent annotators, as reported in Sec. 5.4 and Appendix B.3 (Table 8).

ModalityValid video–dimension unitsOrdinal αMean pairwise QWKPairwise exact agreementPairwise agreement within 1 point
I2V3260.6260.66154.78%89.09%
T2V3210.5550.56952.13%87.95%

Q3. Does the IF–SCP gap remain without calibration on the same videos, and how much does calibration affect it?

A3. We thank reviewer for your helpful suggestion. To examine the effect of calibration, we report the fully raw IF–SCP gap with the deployed calibrated gap on the same matched videos.

ModalityMatched videosRaw IFRaw SCPRaw IF−SCPCalibrated IF−SCP
I2V1,24266.6765.43+1.23≈ +14.00
T2V1,13661.1259.24+1.88≈ +5.10

The raw IF–SCP gaps remain positive but are relatively small, while calibration increases their magnitude. As described in Sec. 4.3, our proposed calibration is fitted to human ratings for each dimension independently, rather than to the IF–SCP gap itself. Its purpose is to improve numerical agreement with human judgments. And our human-rated analysis in A1 provides complementary evidence independent of calibration. Among videos with fully successful instruction following (IF = 3), 25.4% of I2V and 22.2% of T2V videos receive SCP ≤ 1. These findings support that successful action execution does not necessarily imply correct spatial consequence realization.

Thanks for this careful response, which addresses a large part of my concern. I maintain my score.

AuthorsOpenReview ↗

Thank you for your Positive feedback

Dear Reviewer snkh,

Thank you very much for your encouraging comments and thoughtful consideration of our rebuttal. We truly appreciate your support and favorable recommendation.

Best regards,

The Authors

Reviewer_WwgLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MTS-Bench, a benchmark designed to investigate whether video generation models can accurately depict spatial consequences that arise from actions, even when those consequences are not explicitly mentioned in the prompt. The benchmark comprises both t2v and i2v cases, drawn from 12 existing video datasets, with each case accompanied by specific scoring criteria. The study utilizes GPT-5.6 Sol to evaluate each video across five dimensions, with the scoring criteria for some of these dimensions calibrated against human evaluations.

Critical strengths
  1. The task definition is clearly accurate. Beyond instruction following and consistency, the inclusion of unspecified spatial consequences in the evaluation criteria addresses a genuine gap, from a video might adhere to the provided prompt while still depicting an incorrect scene.
  2. The analysis of the judge model in Table 4 offers insights that extend well beyond this specific benchmark. In t2v tasks, demonstrates that rank correlation alone is insufficient to validate the effectiveness of a judge model.
  3. Development use cases and audit use cases are kept independent of one another.
Critical weaknesses
  1. The validation of the judges is insufficient. All human annotations were performed by a single annotator, and each modality is represented by only 40 cases. Moreover, these same cases used for auditing were also involved in the design of the prompts. The calibration maps were likewise refitted using these same 40 cases, while Table 5 excludes certain generative models from the evaluation, it fails to exclude the cases themselves. Therefore, it is essential to involve additional annotators and conduct validation using an entirely new set of cases.
  2. The difference between IF and SCP is somewhat blurred in the provided examples, for instance, while IF assesses the metric of acceleration, the same concept is represented in SCP primarily through the shrinking gap between values
  3. minor issue: in your table 10, why it shows MiniMax H3 is under verification, it looks weird.
Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank Reviewer WwgL for recognizing the significance of our task definition in evaluating implicit spatial consequences, the clear design of MTS-Bench, and the broader insights from our evaluator analysis. We particularly appreciate your acknowledgment that generator-level rank correlation alone is insufficient to validate evaluator reliability, as demonstrated in Table 4. We address your questions below.

Q1. Judge validation relies on a single annotator and 40 cases per modality that also informed development. Additional annotators and validation on entirely new cases are needed.

A1. Thank you for pointing this out. As discussed in the paper (Sec. 4.3), prompt development and calibration serve different roles: calibration is a post-hoc numerical mapping applied to the fixed evaluator outputs and does not modify the scoring prompt. The calibration audit evaluates development-fitted mappings before the final 40-case refit, while the revised Table 5 reports transfer to both held-out generators (LOGO) and unseen cases using frozen calibration mappings. We have now complemented these analyses with new-case validation and additional independent annotators.

Validation on new cases. We expanded the human reference to 1,420 videos covering 150 cases. The new evaluation includes 110 previously unused cases per modality, comprising 437 valid I2V videos and 439 valid T2V videos from 96 source groups per modality. We applied the original deployed calibration mappings unchanged, without further selection or refitting. As the table shows, MAE decreases in all four settings and this provides evidence for the frozen calibration’s effectiveness beyond its development and audit cases.

ModalityDimensionRaw MAE ↓Frozen calibrated MAE ↓ΔMAE [95% CI]Severe-error rate: Raw → Cal.
I2VSCP0.9520.835−0.117 [−0.206, −0.022]28.0% → 0.0%
I2VDSC1.0200.836−0.185 [−0.272, −0.100]25.4% → 0.0%
T2VSCP0.8050.718−0.087 [−0.163, −0.009]20.7% → 1.4%
T2VDSC0.8040.739−0.065 [−0.129, +0.002]19.7% → 2.6%

Additional annotators and agreement. Two additional annotators independently re-rated the same uniformly sampled subset of 150 videos, with 75 per modality, using the fixed rubrics. We compute ordinal Krippendorff’s α jointly from A/B/C ratings, average QWK over the three annotator pairs, exact agreement and within-one-point agreement.

ModalityValid video–dimension unitsOrdinal αMean pairwise QWKPairwise exact agreementPairwise agreement within 1 point
I2V3260.6260.66154.78%89.09%
T2V3210.5550.56952.13%87.95%

The results show moderate inter-annotator agreement, with 87.95–89.09% of pairwise ratings differing by no more than one point. These additional experiments provide evidence of both calibration generalization to unseen cases and the reproducibility of human judgments. The calibration-transfer results are reported in Sec. 5.4 (Table 5), and the final-rating agreement protocol and results in Appendix B.3 (Table 8).

Q2. The distinction between IF and SCP is unclear, particularly why acceleration is evaluated under IF while a shrinking gap is evaluated under SCP.

A2. Thank you for your comment. As defined in Tab. 2, IF evaluates the execution of explicitly instructed actions, while SCP assesses the spatial consequences implied by the initial state and intervention. These dimensions are causally related and may share visual evidence, but assess distinct aspects of spatial evolution. In the driving example (Fig. 3), IF focuses on executing the instructed driving behavior, whereas SCP examines whether the relative distances and positions between vehicles evolve correctly throughout the maneuver. The distinction is even clearer in the pepper-cutting example (Fig. 10): IF assesses whether the cutting action is performed, while SCP evaluates the resulting shapes and spatial configuration of the separated pepper pieces. We have clarified the IF/SCP distinction in Sec. 3.3.

Q3. MiniMax H3 marked as “under verification” in Table 10.

A3. Thank you for pointing this out. The label referred to pending verification of the checkpoint and generation settings, rather than the evaluation results. We have now confirmed the configuration used in our experiments, as detailed below, and have replaced the placeholder in Appendix E (Table 11).

GeneratorModeModel / checkpointOutputKey settings
MiniMax H3I2V/T2VMiniMax-H3/FL2VA768p, 124 frames, 24 FPS50 steps; guidance 1.0; embedded guidance 6.0; seed 0

References

[1] MiniMax-H3: https://huggingface.co/MiniMaxAI/MiniMax-H3

Thanks, the author's response solved my major concerns, I'd like to keep my rating.

AuthorsOpenReview ↗

Thank you for your Positive feedback

Dear Reviewer WwgL,

We sincerely thank you for your positive feedback and for taking the time to consider our rebuttal. We greatly appreciate your recommendation for acceptance.

Best regards,
The Authors

Reviewer_3zcfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the problem in current video-world-model evaluation: a generator can depict an instructed action without realizing its implied spatial consequences. Separating instruction following from spatial consequence prediction is a useful and well-motivated design choice. MTS-Bench introduces a useful and timely evaluation target, and its discussion of measurement limitations is stronger than in many benchmark papers.

Critical strengths

The benchmark provides a reasonably rich task structure, covering image-to-video and text-to-video generation, multiple motion sources, viewpoints, and diagnostic dimensions. The case-specific rubrics make the evaluation more targeted than generic visual-quality or prompt-following metrics.

The paper is commendably transparent about evaluator limitations. It compares several VLM judges against human ratings, reports video-level as well as generator-level agreement, and correctly notes that aggregate ranking agreement does not imply reliable discrimination between individual videos.

Critical weaknesses
  1. The key Spatial Consequence Prediction score still relies heavily on a proprietary VLM evaluator whose video-level direction accuracy is only 61.0% for I2V and 70.2% for T2V. Calibration improves numerical agreement, but a monotone calibration cannot repair incorrectly ordered video pairs or semantic misjudgments.

  2. The human reference set is small and uses a single annotator. Since the benchmark depends on whether an unstated spatial consequence is genuinely implied by the initial state and intervention, the paper should measure inter-annotator agreement on both the case rubrics and the final ratings.

  3. The calibration audit is not a fully independent evaluation: the audit cases informed earlier prompt diagnostics, audit acceptance is a selection step, and the final mapping is refitted on all 40 annotated cases. The reported calibration gains should therefore be interpreted cautiously.

  4. Common-support aggregation excludes unjudgeable examples, which may make scores conditional on evaluability. The paper should report unjudgeable rates by model and dimension, together with coverage-adjusted results. It would also be useful to test sensitivity to temporal frame sampling, evaluator resolution, and repeated evaluator calls.

Questions for authors

All my questions and concerns are listed in weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

We sincerely thank Reviewer 3zcf for recognizing the importance of evaluating spatial consequences beyond instruction following, our comprehensive design and case-specific rubrics of MTS-Bench, and our careful analysis of VLM evaluator reliability. We appreciate your positive feedback, particularly on our distinction between video-level discrimination and generator-level ranking agreement. Here we address your main concerns below.

Q1.1. The reliability of the SCP evaluator.

A1. We thank the reviewer for highlighting this concern. To characterize Spatial Consequence Prediction (SCP) evaluator reliability, we extend our video-level human-alignment analysis to a larger evaluation set, covering 271 I2V and 257 T2V videos, and first report the absolute differences between calibrated SCP scores and human ratings on the 0–3 scale.

ComparisonVideosAbsolute difference < 11 ≤ absolute difference < 2Absolute difference ≥ 2
GPT-5.6-Sol vs Human · I2V271204 (75.28%)67 (24.72%)0 (0.00%)
GPT-5.6-Sol vs Human · T2V257189 (73.54%)68 (26.46%)0 (0.00%)

These results support reasonable video-level agreement with human scores: most SCP scores differ from human ratings by less than one point, with no discrepancies of two points or more observed on this subset. Human–human final rating agreement is reported in Sec. 5.4 and Appendix B.3 (Table 8).

Q1.2. Can calibration address pairwise misorderings and semantic errors in SCP evaluation?

A1.2. Thank you for your comment. We first clarify that our monotone calibration aims to align the evaluator's numerical scores with human ratings, rather than correct semantic misjudgments or pairwise misorderings. Nevertheless, calibration can substantially improve score-level agreement, even when the uncalibrated GPT-5.6-Sol score is less accurate than that of another VLM. We provide three illustrative SCP cases below. In all three cases, calibration reduces GPT-5.6-Sol's absolute error, outperforming the comparison VLM in numerical agreement with the human rating. These examples illustrate the benefit of score-scale alignment.

Case / GeneratorHumanOther VLMSol RawSol CalibratedAbsolute Error: Raw / Other / Cal.
I2V · 0245-01 / Wan 2.61Gemini 3.5 Flash: 231.7972 / 1 / 0.797
T2V · 0277-01 / MiniMax H32Gemini 3.5 Flash: 301.2312 / 1 / 0.769
I2V · 0047-01 / HunyuanVideo 1.51InternVL3.5-8B: 231.7632 / 1 / 0.763

To further assess SCP discrimination, we conduct a pairwise analysis on the 110 held-out cases per modality. Within each case, we construct pairs with non-tied human SCP ratings and examine whether Sol correctly preserves their ordering. Results are averaged equally across valid cases and GPT-5.6-Sol ties are counted separately. Importantly, direction accuracy increases from 62.41%/64.79% for one-point human-rating differences to 78.94%/81.64% for differences of at least two points in I2V/T2V. This indicates more reliable discrimination when the spatial consequences differ substantially, although incorrect orderings remain since our SCP calibration uses a strictly monotone ordinal-q mapping.

ModalityPairsDirection Acc. ↑Misordering ↓Tie ↓Acc. (gap=1) ↑Acc. (gap≥2) ↑
I2V · SCP45170.21%28.30%1.49%62.41%78.94%
T2V · SCP41969.61%25.94%4.45%64.79%81.64%
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q2. The human reference set is small and uses a single annotator. Inter-annotator agreement should be measured for both the case rubrics and the final ratings.

A2. We thank the reviewer for this valuable suggestion. We have expanded our human reference set to 1,420 videos across 150 cases and evaluated the original frozen calibration on the newly annotated data (see A3).

To access inter-annotator agreement, two additional annotators independently evaluated a uniformly sampled subset of 150 videos (75 I2V and 75 T2V), assessing both case-rubric assessments on a three-level scale (1–3) and final video ratings on a four-level scale (0–3). We report ordinal Krippendorff's α across all three annotators, computed separately for rubric assessments and final ratings. For final ratings, we additionally report Quadratic Weighted Kappa (QWK) averaged over the three annotator pairs, along with exact and within-one-point agreement rates.

ModalityValid video–dimension unitsCase rubrics: ordinal αFinal ratings: ordinal αMean pairwise QWKExact agreementPairwise agreement within 1 point
I2V3260.8410.6260.66154.78%89.09%
T2V3210.8220.5550.56952.13%87.95%

Our results show strong agreement on case-specific rubrics (α = 0.841/0.822) and moderate agreement on final video ratings (α = 0.626/0.555), suggesting that evaluating spatial evolution is more challenging than assessing the rubrics themselves. Moreover, 87.95–89.09% of pairwise ratings differ by no more than one point. The final rating agreement and annotation protocol are reported in Sec. 5.4 and Appendix B.3 (Table 8).

Q3. The calibration audit is not fully independent because the audit cases informed prompt diagnostics and calibration acceptance, and the final mappings were refitted on all 40 annotated cases.

A3. Thank you for your comment. The scoring prompt is fixed before calibration, which only fits a post-hoc numerical mapping to the resulting scores and does not modify or feed back into prompt development. Additionally, we conducted an additional evaluation on 110 new cases per modality that were not used for prompt development, calibration selection, or parameter fitting. We applied the original final mappings fitted on the 40-case to them for demonstrating the generalization.

ModalityDimensionVideos / Cases / Source groupsRaw MAE ↓Frozen calibrated MAE ↓ΔMAE [95% CI]Severe-error rate: Raw → Cal.
I2VSCP437 / 110 / 960.9520.835−0.117 [−0.206, −0.022]28.0% → 0.0%
I2VDSC437 / 110 / 961.0200.836−0.185 [−0.272, −0.100]25.4% → 0.0%
T2VSCP439 / 110 / 960.8050.718−0.087 [−0.163, −0.009]20.7% → 1.4%
T2VDSC439 / 110 / 960.8040.739−0.065 [−0.129, +0.002]19.7% → 2.6%

The frozen calibration reduces MAE in all four settings. These results provide additional evidence beyond the original audit that the deployed calibration improves agreement with human ratings on new cases.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3)

Q4. Potential evaluability bias from common-support aggregation: report unjudgeable rates by model and dimension, coverage-adjusted results, and sensitivity to temporal frame sampling, evaluator resolution, and repeated evaluator calls.

A4. We thank the reviewer for highlighting this important concern.

Coverage analysis. We have checked the finalized scoring records and added a coverage analysis by model and dimension. The unjudgeable rate is 0% for every model–dimension combination. Missing scores consist of API content-policy refusals and absent scoring records: 10 and 7, respectively, for I2V, and 7 and 2 for T2V. Overall video-level scoring coverage is 98.74% (1,333/1,350) for I2V and 99.25% (1,191/1,200) for T2V.

Coverage-adjusted evaluation. The tables below combine dimension-wise coverage with the original scores, available-case aggregation, and full-support missing-score bounds. Available-case aggregation preserves the original rankings in both modalities. The table shows that all T2V ranks remain unchanged, and only Wan 2.6 and Kling v3 Standard at I2V ranks 4–5 may swap. We will include these results in the revision.

I2V

ModelISP/IF/SCP/DSC coverageTSC coverageUnjudgeable rateCommon OverallAvailable OverallMissing-score bounds
Gemini Omni Flash148/150 (98.67%)35/35 (100.00%)0%71.4671.64[70.87, 71.93]
Happy Horse 1.1148/150 (98.67%)35/35 (100.00%)0%70.3970.27[69.51, 70.58]
MiniMax H3149/150 (99.33%)35/35 (100.00%)0%68.3968.75[68.38, 68.91]
Wan 2.6144/150 (96.00%)34/35 (97.14%)0%67.4667.01[64.47, 68.24]
Kling v3 Standard150/150 (100.00%)35/35 (100.00%)0%66.7566.04[66.04, 66.04]
LTX-2.5150/150 (100.00%)35/35 (100.00%)0%61.8762.50[62.50, 62.50]
Cosmos 3 Super149/150 (99.33%)35/35 (100.00%)0%59.7159.89[59.54, 60.07]
HunyuanVideo 1.5150/150 (100.00%)35/35 (100.00%)0%58.7058.18[58.18, 58.18]
CogVideoX-5B145/150 (96.67%)34/35 (97.14%)0%49.5649.16[47.53, 50.77]

T2V

ModelISR/IF/SCP/DSC coverageTSC coverageUnjudgeable rateCommon OverallAvailable OverallMissing-score bounds
Gemini Omni Flash150/150 (100.00%)29/29 (100.00%)0%72.2972.60[72.60, 72.60]
Happy Horse 1.1149/150 (99.33%)29/29 (100.00%)0%68.7669.10[68.71, 69.25]
MiniMax H3149/150 (99.33%)28/29 (96.55%)0%63.5163.53[62.87, 64.09]
Wan 2.6149/150 (99.33%)29/29 (100.00%)0%62.0261.56[61.19, 61.73]
LTX-2.5 Pro149/150 (99.33%)29/29 (100.00%)0%58.9658.44[58.09, 58.63]
HunyuanVideo 1.5149/150 (99.33%)29/29 (100.00%)0%55.0754.76[54.44, 54.97]
Kling v3 Standard149/150 (99.33%)29/29 (100.00%)0%53.0153.64[53.35, 53.89]
CogVideoX-5B147/150 (98.00%)29/29 (100.00%)0%45.0545.08[44.20, 45.80]

Evaluation sensitivity. We have also added sensitivity analyses for temporal frame sampling, evaluator resolution, and repeated evaluator calls. Frame-count and resolution changes are compared with the newly collected default run (24 frames, 224 × 224); the repeat comparison uses the archived and newly collected default calls under the same recorded evaluator configuration. These checks show limited raw-grade variation and broadly stable generator rankings.

Sensitivity analysisSettings comparedI2V: MAD (all / SCP); rank ρT2V: MAD (all / SCP); rank ρ
Temporal frame sampling24 vs. 12 frames, 224 × 2240.257 / 0.242, ρ = 1.0000.216 / 0.227, ρ = 1.000
Evaluator resolution224 × 224 vs. 448 × 448, 24 frames0.272 / 0.265, ρ = 0.9330.240 / 0.267, ρ = 1.000
Repeated evaluator callsArchived vs. new call, 24 frames, 224 × 2240.226 / 0.205, ρ = 0.9330.195 / 0.210, ρ = 1.000
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer 3zcf,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #43816

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Authors' Responses

We appreciate the reviewers' thoughtful feedback and constructive engagement. To facilitate reviewer–AC discussion, we summarize the raised strengths and concerns below. Check marks indicate points explicitly raised by each reviewer.

StrengthsR-3zcfR-WwgLR-snkh
Well-motivated evaluation of implicit spatial consequences✓✓✓
Informative analysis of VLM evaluator reliability✓✓✓
Targeted, case-specific evaluation criteria✓✓
Broad coverage across modalities, motion sources, and viewpoints✓
Clear presentation✓✓
Concerns raised
Limited human reference & single-annotator reliance✓✓✓
Independent calibration validation on unseen cases✓✓
SCP discrimination & the scope of calibration✓
Clarification of the IF/SCP distinction✓
Same-video IF–SCP analysis & calibration effect✓
Scoring coverage & evaluator sensitivity✓
Incomplete MiniMax H3 configuration details✓
Rating333

We are encouraged that the reviewers recognize our focus on spatial consequences beyond action execution, our case-specific criteria, and the key finding that model-level ranking agreement does not guarantee reliable per-video discrimination. Below, we summarize the additional experiments and clarifications addressing each concern:

  • Expanded human annotations (R-3zcf, R-WwgL, R-snkh): We expanded the human reference to 1,420 videos (150 cases), with two additional annotators independently re-rating a 150-video subset (75/modality). Final ratings achieve 0.661/0.569 mean pairwise QWK and 89.09%/87.95% within-1-point agreement (I2V/T2V); case-rubric agreement is detailed for R-3zcf (full protocol and results are in Sec. 4.3, 5.4; App. B.3, Table 8).

  • Calibration generalization (R-3zcf, R-WwgL): Without refitting on 110 unseen cases/modality, frozen calibration lowers SCP MAE (0.952 to 0.835 in I2V; 0.805 to 0.718 in T2V) and DSC MAE. Leave-one-generator-out validation likewise improves MAE across all 9 I2V and 8 T2V generators, confirming cross-case and cross-generator transfer (Sec. 5.4, Table 5).

  • SCP discrimination and the role of calibration (R-3zcf): On held-out cases, SCP pairwise direction accuracy is 70.21%/69.61% (I2V/T2V), rising to 78.94%/81.64% when human ratings differ by ≥2 points. Monotone calibration improves score alignment while preserving pairwise orderings, so both are evaluated separately.

  • IF vs. SCP distinction (R-WwgL, R-snkh): We clarified action execution vs. spatial consequences via the driving and pepper-cutting examples (Sec. 3.3). Same-video analysis shows 25.4% (I2V) and 22.2% (T2V) of human-rated IF = 3 videos receive SCP ≤ 1, showing execution without spatial realization. Raw-score IF–SCP gaps and calibration effects are also reported.

  • Coverage and evaluator sensitivity (R-3zcf): Video-level coverage is 98.74%/99.25% (I2V/T2V) across models and dimensions, with 0% unjudgeable records. Rankings hold under available-case aggregation and missing-score bounds (T2V fixed; only I2V 4–5 may swap). Varying frame count, resolution, and repeats yields 0.933–1.000 raw-score ranking correlations, confirming aggregate stability.

  • Verified generation settings (R-WwgL): We confirmed the MiniMax H3 checkpoint and generation configuration.

We once again thank the reviewers and AC for their thoughtful feedback, which strengthened our empirical validation and clarified the evaluation protocol.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.