acceptodds
Under review as a conference paper at ICLR 2027

SpatialSherlock: Diagnosing Spatial Inconsistencies in AI-Generated Videos

Abstract

Diagnosing spatial inconsistencies in AI-generated videos requires distinguishing changes explained by camera motion, object motion, and occlusion from conflicts with the depicted scene evolution. We introduce a data pipeline that connects spatial skills reconstructed from public datasets with structured human diagnoses of generated videos. The annotations specify affected regions, required spatial corrections, and explanations, making the spatial basis of each judgment separately assessable. These references support both diagnostic reasoning supervision and component-level reinforcement-learning feedback. The pipeline curates 92,000 examples across four diagnostic tasks from more than 20,000 generated videos. Using this supervision, we train spatialsherlock, based on Qwen3.5-9B, through supervised fine-tuning and joint reinforcement learning. The model forms local diagnoses and checks them against available intervening frames before aggregating video-level judgments and temporal evidence. spatialsherlock achieves 81.3% accuracy on frame-pair diagnosis and 78.7% on source-video diagnosis, exceeding the strongest tested commercial baselines by 40.3 and 16.7 percentage points, respectively. It also improves on its backbone across all nine external spatial benchmarks evaluated zero-shot, with gains of 1.6–16.6 percentage points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Consistency, Spatial Reasoning, Video Understanding, Vision-Language Models, AI-Generated Videos, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DXp5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialSherlock studies the diagnosis of spatial inconsistencies in generated videos, including whether apparent discrepancies can be explained by intervening camera motion, object motion, or occlusion. Its main contribution is a data pipeline combining approximately 200,000 spatial-skill examples from public datasets with structured human diagnoses of more than 20,000 generated videos. The latter are curated into 92,000 examples across Judge, Explain, Diagnose, and Video tasks, with references for regions, spatial corrections, explanations, and temporal evidence. A Qwen3.5-9B backbone is trained through SFT and joint GRPO with component-level rewards. At inference, local diagnoses are conditionally verified against intervening frames before aggregation. The model achieves 81.3% on frame-pair Diagnose and 78.7% on source-video diagnosis, and improves over its backbone on all nine evaluated external spatial benchmarks.

Critical strengths
  1. A useful form of structured supervision for a consequential diagnostic problem. The data records connect a consistency judgment to affected regions, required spatial corrections, and an explanation, rather than providing only a scalar quality label. Verifying endpoint discrepancies against intervening events also addresses a real ambiguity in video diagnosis. Keeping all examples derived from a source video in the same partition is an appropriate safeguard against leakage between these closely related examples (Section 3 and Appendix C). This is a substantive data and task-design contribution despite using an existing backbone and standard training algorithms.

  2. Strong empirical results with informative controls. The reported Diagnose and Video scores exceed the strongest tested commercial baselines by 40.3 and 16.7 percentage points, respectively; gains of 1.6–16.6 points over the backbone also appear across nine external spatial benchmarks. Importantly, Appendix H.2 compares RL with continued SFT from the same checkpoint using the same four-task pool, task mixture, and references, with comparable compute including rollout and reward evaluation. RL improves Video from 60.3% to 78.7% and Diagnose from 70.7% to 81.3%. The data-source ablations further support combining spatial-skill and diagnostic supervision. These controls make the training contribution more convincing than an SFT-versus-RL comparison that also introduces additional task data.

Critical weaknesses
  1. The reliability of the new diagnostic benchmark is insufficiently documented. Appendix F appropriately uses human evaluators to check explanations against references, and Diagnose requires both a correct judgment and explanation. However, the paper does not report annotator/evaluator counts, inter-rater agreement, adjudication, or whether evaluation is blinded and independent of reference construction. These details matter because plausible camera/scene interpretations and alternative valid spatial corrections may make reference matching subjective. The 300-example Diagnose set is also selected from cases flagged as challenging, making a clear sampling protocol particularly important. Without an annotation and scoring reliability audit, the large gains on this newly constructed benchmark are harder to assess precisely.

  2. The video comparison leaves inference-procedure effects unresolved. SpatialSherlock enumerates intervals and boundary pairs, performs local diagnosis, and uses a second model call to verify diagnoses when intervening frames are available. Appendix F scores the aggregated source-video output, but does not make clear whether commercial baselines receive the same interval proposals, verification opportunity, and comparable frame/token/call budgets. Nor is temporal verification isolated by an inference ablation. Thus, the reported 16.7-point Video advantage cannot yet be attributed clearly between the trained model and the inference procedure. This limits the comparative interpretation and the evidence for the proposed verification stage, while leaving the frame-pair and external-benchmark improvements encouraging.

Questions for authors
  1. Annotation and evaluation reliability. How many independent people annotate and score each example, how are disagreements and alternative valid diagnoses handled, and are model identities hidden from evaluators? Please clarify the challenging-case sampling rule and report agreement or an audit on the existing test set. This information could substantially change confidence in the diagnostic gains.

  2. Comparable video inference. Were all Table 1 baselines evaluated with the same interval/boundary proposals and the same opportunity to verify an initial diagnosis against intervening frames? Please state the per-video frame, token, and model-call budgets, and compare the full procedure with verification removed on the existing Video test set. A comparison under matched procedures would clarify both baseline fairness and the contribution of temporal verification.

4 comments
AuthorsOpenReview ↗

Response 1/2: Annotation, evaluation reliability, and scoring metrics

Thank you for your positive assessment of SpatialSherlock, including its frame-pair and external-benchmark improvements. We agree with your constructive suggestions on documenting benchmark reliability and clarifying video inference comparisons. This comment explains annotation, evaluation, and scoring metrics. Part 2 reports the expanded results and video comparisons; a separate supplement provides the video evaluation prompt.

1. Annotation and reference construction

Approximately 30 annotators from two teams participated. Each example passed through three different people in sequence: annotation, validation, and final review. Failure at any review stage triggered a restart of the complete process with new personnel. Examples for which consensus could not be reached were excluded, accounting for approximately 10%.

The algorithm team additionally audited retained annotations at a 1% sampling rate, checking more than 1,000 examples across batches. The audit covered consistency judgments, temporal evidence, region accuracy and completeness, corresponding spatial corrections, and whether explanations were visually grounded. Any detected error triggered another final review of the entire batch. A batch was accepted only after an audit found no errors.

2. Test-set construction and challenging-case selection

The Judge, Explain, and Video test sets were obtained by randomly splitting the reviewed annotation dataset and retaining the corresponding ground-truth references (GT).

The 300 Diagnose examples satisfied two criteria: they were flagged as challenging during final review, and their final diagnoses were jointly endorsed by the annotation and algorithm teams. Final reviewers actively questioned and checked the preceding annotation and validation, flagging cases they could not judge quickly. To ensure an unambiguous scoring reference, we selected 300 flagged examples with a unique, undisputed diagnostic answer covering both the consistency judgment and explanation. Cases with disputed diagnoses or without a unique diagnostic answer were excluded.

3. Scoring personnel, blinding, and verification

Each model output was handled sequentially by two evaluators: one assigned the score, and the other verified it. Neither evaluator had participated in constructing the GT for that example. Outputs from different models were mixed, with model identities hidden from evaluators. Scores were based on agreement between the prediction and GT in the judgment and explanatory facts. Predictions that failed to meet the task-specific reference requirements were scored as incorrect. No disagreements arose during scoring and verification.

The original accuracies follow Appendix F. Explain requires a correct explanation given the judgment; Diagnose requires both a correct judgment and explanation; Video additionally requires correct temporal evidence. Explanations are assessed through the human scoring and verification procedure described above.

4. Complementary evaluation metrics

To complement human evaluation, we added BERTScore-F1 and a DeepSeek-based score for the explanation component of Diagnose. These assess semantic similarity and consistency with reference facts, respectively. We also added Judge macro-F1.

For the DeepSeek score, DeepSeek V4.1 Flash outputs probabilities over the same four categories used by the language reward:

  1. Complete coverage of reference facts without errors or unsupported claims.
  2. At least one complete matching fact, with omissions only.
  3. At least one complete matching fact, alongside errors or unsupported claims.
  4. No complete matching fact.

The score is the probability-weighted sum with weights (1,0.5,0.25,0)(1, 0.5, 0.25, 0). The complete results are provided in Part 2.

AuthorsOpenReview ↗

Response 2/2: Expanded results and video inference comparisons

5. Expanded comparison results

We expanded the comparison to include the three open-source spatial models from Table 2, the original Qwen3.5-9B backbone, and additional commercial models.

Accuracies and macro-F1 are percentages; BERTScore-F1 and DeepSeek scores use a 0–1 scale and evaluate Diagnose explanations. Judge negatives are inconsistent cases and positives are consistent cases; its mean equally weights the two classes. In this table, “—” indicates that the model provider does not support video evaluation. Bold marks the highest value in each column.

ModelVideoDiagnoseJudgeExplain
Acc.Acc.BERTScore-F1DeepSeekNeg.Pos.MeanMacro-F1Acc.
Gemini 3.1 Pro56.334.30.860.3786.533.560.057.056.7
Gemini 3.5 Flash50.741.00.860.4392.516.054.346.461.0
Qwen3.8 Flash46.037.00.850.3739.087.063.060.768.0
Qwen3.8-Max62.030.70.860.3146.080.563.362.173.7
gpt6-sol—52.30.880.6043.592.568.066.082.3
hy4—35.70.840.3638.577.057.856.172.0
MiMo-V2.6-Pro52.340.70.850.4854.070.062.061.868.7
doubao-seed-2.1-pro48.344.00.860.4583.034.058.555.873.0
DeepSeek V4.1 Flash—32.00.840.3148.569.058.858.358.3
intern-s2—34.70.850.3859.061.060.060.066.3
Qwen3.5-9B (backbone)35.030.30.820.3032.067.049.547.966.3
VST-7B-RL32.028.70.810.2633.560.046.845.864.7
SN-SI-1.146.334.00.830.2734.068.551.349.870.3
Cambrian-S-7B36.330.00.810.2131.072.551.849.657.3
SpatialSherlock78.781.30.880.6881.585.083.383.284.0

SpatialSherlock achieves 81.3% Diagnose accuracy, exceeding the strongest comparison baseline by 29.0 percentage points. Its BERTScore-F1 of 0.88 is tied for the highest, and its DeepSeek score of 0.68 exceeds the strongest comparison model's 0.60. These complementary automatic metrics support the explanation-quality gains observed in human evaluation.

6. Video inference procedures and baseline comparisons

The four commercial models in the current manuscript were evaluated using direct video input and a video analysis prompt. SpatialSherlock uses local diagnosis, conditional temporal verification, and aggregation at the source-video level. These configurations therefore use different inference procedures.

The direct-video prompt asks whether observed changes follow natural temporal progression and legitimate camera movement within a single continuous 3D world. It distinguishes 3D relationships from image projection changes and asks the model to consider camera movement, occlusion, object motion, and lighting before declaring an inconsistency. It requests an overall verdict and comprehensive reasoning. The prompt is provided in the supplementary comment.

We additionally evaluated these four commercial models using the SpatialSherlock pipeline. Video accuracies under the two configurations are:

ModelSpatialSherlock pipelineDirect video input + prompt
Gemini 3.1 Pro37.3%56.3%
Gemini 3.5 Flash28.0%50.7%
Qwen3.8 Flash40.0%46.0%
Qwen3.8-Max61.3%62.0%

All four models achieved higher accuracy with direct video input; the two Gemini models improved by 19.0 and 22.7 percentage points, respectively. In our experiments, Gemini frequently mistook viewpoint changes, such as camera rotation, for spatial inconsistencies during two-frame diagnosis. These false positives affected subsequent processing and the final video judgment. Its higher accuracy on inconsistent cases and lower accuracy on consistent cases in Judge show a corresponding tendency.

The current manuscript therefore reports the better-performing direct-video configuration for these four commercial models. SpatialSherlock's 78.7% Video accuracy exceeds the strongest direct-video baseline (62.0%) by 16.7 percentage points. Under the SpatialSherlock pipeline, the best result among these four commercial models is 61.3%, corresponding to a gap of 17.4 points. These comparisons support the reported baseline configuration and provide additional comparisons using the SpatialSherlock pipeline. The 16.7-point gap describes the overall performance difference between the reported model-and-procedure configurations.

AuthorsOpenReview ↗

Supplement: Direct-video evaluation prompt

You will watch a video and determine whether it is spatially and temporally consistent. Treat the video as an observation of a single continuous 3D physical world in which time naturally progresses forward and the camera legitimately moves (panning, rotating, tilting, zooming). Entities in the video can move, act, and interact; the scene can naturally evolve over time (doors opening, objects being carried, lighting changing). Your task is to determine whether all observable changes throughout the video can be explained by "the natural progression of time and legitimate camera movement within the same continuous 3D world." A dimension is considered inconsistent only when some entity, spatial relationship, scene structure, or action state exhibits a discontinuity that cannot be explained by a continuous physical process. [Judgment Rules]

  1. When judging position, orientation, left/right, front/back, distance, and height, use the 3D scene coordinate system, not the image coordinate system. The camera's rotation angle determines how the image changes—a 180° orbit swaps left and right in the image, but a 120° rotation may not. You cannot infer whether 3D positions have changed simply from whether left and right are swapped in the image.
  2. Use the entity's own frontal facing direction as the positive direction—"front" is the 3D direction its face points toward, "back" is the opposite. Do not use image foreground/background to judge 3D front/back. Left/right of body parts are based on the frontal facing direction and do not change with camera angle.
  3. Camera distance and focal length changes alter projection size. When judging scale, look at the same object's proportional relationship relative to scene anchors.
  4. Changes in entity position, posture, and scene state throughout the video must be gradual and explainable by physical processes. Discontinuities (teleportation, instant costume changes, objects appearing/vanishing out of thin air) are inconsistent. Permitted changes include: entities moving naturally, objects being carried, doors opening/closing, lighting gradually changing, and field-of-view changes due to camera movement.
  5. Before judging that something has "disappeared" or "appeared," first rule out each of—(1) Camera cropping: cut off at the image edge; (2) Foreground occlusion: blocked by a foreground object; (3) Out of field of view: camera turning causes it to leave the visible range; (4) Front-back coverage: camera moves to the other side of the subject, making something invisible; (5) Subject displacement occlusion: the subject moves and occludes or reveals other objects; (6) Temporal progression: the entity naturally leaves or enters the frame area over time. If any holds, it is consistent.
  6. Color shifts, texture differences, and reflection changes of the same material under different lighting/white balance, as well as changes in visible surfaces due to viewpoint, are all legitimate. Gradual lighting changes over time in a video are likewise legitimate.
  7. Focus on "whether entities/structures/relationships have undergone holistic replacement, disappearance, addition, or fundamental change." For entities or structures that can already be matched, their surface details (accessories, clothing texture, color brightness, material reflections, background texture, etc.) are not within the judgment scope. The granularity is "whether something has been replaced by a different thing," not "whether a small element on the same thing is completely identical."
  8. Do not make forced inferences about occluded, depth-of-field blurred, or insufficient-resolution regions. When evidence is insufficient, output "undetermined." [Three Verdict Levels]
  • Consistent: All changes in the video can be explained by "the natural progression of time and legitimate camera movement within the same continuous 3D world."
  • Inconsistent: At least one real 3D scene discontinuity exists that cannot be explained by a continuous physical process.
  • Undetermined: Video information is insufficient to make a determination. If the video is unclear but a reasonable judgment has been made within the visible content, lean toward consistent. Complete the analysis of the video first, then output the conclusion in the following fixed format with immutable order for programmatic parsing: === OVERALL VERDICT === <Consistent / Inconsistent / Undetermined> === COMPREHENSIVE REASON === <Comprehensive reasoning. State all discovered issues completely without omission; for each issue, explain what aspect it concerns (entity, spatial relationship, scene structure, action state, etc.) and why it cannot be explained by camera movement or temporal progression. If no issues exist, explain how the observed changes in the video are reasonably explained.> === END ===
AuthorsOpenReview ↗

Follow-up on our response to your comments

Thank you again for your encouraging review and detailed questions. Across our responses, we have clarified the benchmark construction and evaluation reliability protocols—specifically detailing the three-stage annotation workflow, 1% batch audits, selection criteria for the 300 unambiguous Diagnose test cases, blinded two-evaluator scoring independent of ground-truth construction, and complementary automatic metrics (BERTScore-F1 and DeepSeek score). We also evaluated the commercial video baselines under both direct video input and the SpatialSherlock multi-stage pipeline (alongside sharing the full evaluation prompt), showing consistent gains of 16.7 and 17.4 percentage points, respectively.

As the discussion period draws to a close, we wanted to kindly check whether our responses and additional comparisons address your concerns, or if there are any remaining points we could further clarify.

Reviewer_WY8HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a diagnostic pipeline for detecting spatial inconsistencies in AI-generated videos, distinguishing genuine inconsistencies from plausible changes caused by camera motion, object movement, and occlusion. The authors employ approximately 200,000 spatial skill samples and 92,000 diagnostic task samples derived from over 20,000 generated videos to train Qwen3.5-9B through supervised fine-tuning and joint reinforcement learning. The framework addresses four key tasks: inconsistency detection, region localization, correction suggestions, and explanation generation.

Critical strengths
  1. Spatial inconsistency remains a common challenge in AI-generated videos, while manual inspection is costly. Therefore, developing an automated, AI-driven diagnostic pipeline is both meaningful and practically valuable.

  2. The paper proposes a comprehensive diagnostic pipeline that integrates spatial inconsistency detection, localization, explanation, and correction suggestions, demonstrating a coherent design and practical applicability.

  3. The paper constructs 200,000 spatial skill samples and 92,000 diagnostic task samples, with extensive evaluations across nine benchmarks. The diagnostic accuracy increases from 41.0% to 81.3%, demonstrating the effectiveness of the proposed framework.

Critical weaknesses
  1. The reward design for the explanation task lacks clarity. Further justification of its formulation and effectiveness is needed.

  2. The Region and Region+Correction components appear redundant. Their distinct roles and contributions should be clarified.

  3. The paper should justify the choice of GRPO over alternative RL algorithms, such as PPO and GSPO, through theoretical discussion or empirical comparisons.

  4. Although Table 1 shows improvements in diagnosis and explanation, comparisons limited to Gemini and Qwen are insufficient. Additional baselines, such as LLaVA, GPT, Claude, and InternVL, would strengthen the evaluation of SpatialSherlock.

  5. Further ablation studies are needed to assess the individual contributions of different reward components and their combined effects on overall performance.

  6. The evaluation relies solely on accuracy, whose definition remains unclear. Additional metrics, such as F1 and BERTScore, along with robustness evaluations under noise perturbations, would provide a more comprehensive assessment.

Questions for authors

please see the weakness section

4 comments
AuthorsOpenReview ↗

Response 1/3: Language Reward and Distinct Roles of the Spatial Rewards

Thank you for recognizing the value of SpatialSherlock. We agree with your constructive suggestions on reward design, optimization, baseline coverage, evaluation metrics, and robustness. We have completed additional experiments and will incorporate the results and clarifications into the revised manuscript.

1. Language reward

Qwen3.5-9B compares the predicted and human-reference explanations in their summary fields, allowing semantically equivalent wording. It evaluates fact coverage and correctness using four categories:

  • A: All key reference facts covered, without errors or unsupported claims.
  • B: At least one complete matching fact, with omissions only.
  • C: At least one complete matching fact, alongside errors or unsupported claims.
  • D: No complete match to a key reference fact.

For category-label log-probabilities ℓk\ell_k, we compute

R_{\mathrm{lang}}=p_A+0.5p_B+0.25p_C.$$ The weights $(1,0.5,0.25,0)$ reward complete, correct explanations, give partial credit for omissions, and assign lower scores to incorrect or unsupported content. The probability-weighted expectation preserves the scorer's relative preferences across categories and provides graded feedback. We will clarify these definitions and calculations in **Section 4.2 and Appendix E**. **2. Distinct roles of Region and Region+Correction** We agree that their distinct roles need clearer explanation. Spatial diagnosis requires complete coverage of anomalous regions and appropriate corrections associated with those regions. Region evaluates overall coverage; Region+Correction evaluates individual region–correction correspondences. IoU serves a different purpose in each component. **Region supervises overall coverage.** The Region reward, $R_{\mathrm{union}}$, computes IoU between the unions of predicted and reference boxes. Missing anomalous area reduces the intersection, whereas excess coverage increases the union. This evaluates how completely and precisely the prediction covers the annotated anomalous area. Predictions with identical unions receive identical scores even when their numbers or arrangements of boxes differ. This accommodates different box decompositions while still penalizing larger boxes that include irrelevant background. **Region+Correction grounds each correction in its target region.** The Region+Correction reward, $R_{\mathrm{match}}$, first establishes one-to-one correspondences through Hungarian matching that maximizes total IoU. It then weights each matched IoU by agreement on correction actions and, where applicable, movement direction. For a nonempty valid matching, $$R_{\mathrm{match}}=\frac{1}{|M^\star|} \sum_{(i,j)\in M^\star}\omega_{ij}\operatorname{IoU}(b_i^\star,\hat b_j).$$ Here, $M^\star$ is the matching, $b_i^\star$ and $\hat b_j$ are reference and predicted boxes, and $\omega_{ij}$ measures correction agreement. Matched IoU ties correction credit to localization, so a proposed correction must apply to the corresponding region. Incorrect actions or directions lower the score even with accurate localization. Zero-weight pairs remain in the denominator and lower the average. **Two examples illustrate their complementary roles.** First, consider two disjoint reference regions of equal area. Suppose the model predicts only one box, exactly matching one reference region, with a fully correct correction. The matching reward is 1 because the average includes only the actual matched pair. The union reward is 0.5 because half the reference area is missing. Thus, union IoU detects a coverage failure that a high average matching score can overlook. Second, suppose all reference boxes are predicted exactly, but every correction is incorrect and receives zero weight. The union reward remains 1, while the matching reward becomes 0. Complete localization therefore still requires supervision of region-specific corrections. These distinctions also explain why both signals are useful during optimization. When the correctness gate is satisfied, Region rewards accurate localization even if the correction is wrong, while Region+Correction reflects the correction error. Likewise, identical overall coverage receives consistent Region feedback, while the matching term evaluates individual region–correction correspondences. Together, they supervise coverage completeness and correction accuracy while accommodating different box decompositions at the coverage level. Our ablation in Response 2/3 evaluates their combined contribution as joint spatial supervision. We will expand **Appendix E** with these distinct objectives and examples, and clarify matching, averaging, and correction weighting in **Appendix E**.
AuthorsOpenReview ↗

Response 2/3: PPO Comparison and Reward Ablations

3. PPO comparison and reward ablations

GRPO computes rewards for multiple responses to the same visual input and estimates advantages through within-group reward comparisons, without an additional value network. This accommodates our structured scores for judgments, regions, corrections, and explanations while avoiding value-function fitting.

We also implemented PPO with generalized advantage estimation (GAE), initializing its value network VV from Qwen3.5-9B. As policy updates change the response distribution and expected returns, the value network must track these changes; its estimates affect policy advantages. To improve stability, we:

  • Pretrained the value network before joint policy and value optimization.
  • Updated it three times as frequently as the policy, at one-tenth of the policy learning rate.
  • Used PPO clipping of 0.150.15. GRPO used asymmetric clipping with ϵlow=0.20\epsilon_{\mathrm{low}}=0.20 and ϵhigh=0.28\epsilon_{\mathrm{high}}=0.28.

Under these configurations, PPO took approximately five times GRPO's total training time.

For joint diagnosis with nonempty reference regions, the full reward is

R=C(α+βRunion+γRmatch+δRlang).R=C(\alpha+\beta R_{\mathrm{union}}+\gamma R_{\mathrm{match}}+\delta R_{\mathrm{lang}}).

The correctness gate CC requires the correct judgment and, for Video, the correct temporal option. Ablations retain correctness supervision and selectively include language supervision, joint spatial supervision, or both. Retained coefficients remain unchanged, without retuning or renormalization.

Results are percentages; Judge reports the equally weighted class mean. All RL rows use GRPO unless marked PPO; “—” denotes an unevaluated task. Bold marks column maxima.

ConfigurationVideoDiagnoseJudgeExplain
SFT—66.772.073.3
SFT + continued SFT60.370.778.076.3
SFT + PPO, full reward75.384.788.082.7
SFT + RL, correctness only40.050.394.062.0
SFT + RL, correctness + language68.773.085.386.7
SFT + RL, correctness + spatial60.364.086.370.7
SpatialSherlock (SFT + GRPO, full reward)78.781.383.384.0

PPO performs better on Diagnose and Judge; GRPO performs better on Video and Explain. GRPO's 3.4-percentage-point Video advantage and lower measured training cost support our choice for complete-video diagnosis.

The ablations show distinct effects of the supervision:

  • Correctness only: Judge reaches 94.0%, but Diagnose and Video reach only 50.3% and 40.0%, showing that classification alone is insufficient for complete diagnosis.
  • Language supervision: Removing it reduces Diagnose from 81.3% to 64.0% and Explain from 84.0% to 70.7%, supporting explicit supervision of explanation correctness.
  • Joint spatial supervision: Adding both spatial terms to correctness and language rewards raises Video from 68.7% to 78.7% and Diagnose from 73.0% to 81.3%. Judge and Explain change from 85.3% and 86.7% to 83.3% and 84.0%, respectively. This supports the combined spatial contribution to video and joint diagnosis, with trade-offs across tasks.

Continued SFT and full RL start from the same SFT checkpoint and use the same four-task data, task mixture, and reference annotations, with comparable compute. Full RL improves Video and Diagnose over continued SFT by 18.4 and 10.6 percentage points.

We will explain the optimizer choice in Section 4.1, document PPO settings and training cost in Appendix D, expand Table 4 in Appendix H, and discuss these comparisons in Section 5.1.

AuthorsOpenReview ↗

Response 3/3: Expanded Baselines, Evaluation Metrics, and Noise Robustness

4. Expanded baselines and complementary evaluation metrics

We added six commercial models: gpt6-sol, hy4, MiMo-V2.6-Pro, doubao-seed-2.1-pro, DeepSeek V4.1 Flash, and intern-s2, alongside the existing Gemini and Qwen baselines. We also evaluated the three open-source spatial models from Table 2 and the original Qwen3.5-9B backbone.

Accuracies follow Appendix F: Judge requires a correct consistency judgment; Explain requires a correct explanation given the correct judgment; Diagnose requires both. Video requires correct judgment, explanation, and supporting temporal evidence. Explanations are assessed by human evaluators. Judge uses 200 cases per class, Explain and Diagnose each use 300 frame pairs, and Video uses 300 complete source videos.

We added Judge macro-F1 (the mean of both class F1 scores) and two metrics for Diagnose explanations. BERTScore-F1 measures semantic similarity to the reference. The DeepSeek score uses DeepSeek V4.1 Flash with the four categories and probability-weighted scoring in Response 1/3.

Accuracies and macro-F1 are percentages; explanation scores use a 0–1 scale. Judge negatives are inconsistent cases and positives are consistent cases; Mean equally weights the classes. “—” denotes an interface without video support in this evaluation. Bold marks column maxima.

ModelVideoDiagnoseJudgeExplain
Acc.Acc.BERTScore-F1DeepSeekNeg.Pos.MeanMacro-F1Acc.
Gemini 3.1 Pro56.334.30.860.3786.533.560.057.056.7
Gemini 3.5 Flash50.741.00.860.4392.516.054.346.461.0
Qwen3.8 Flash46.037.00.850.3739.087.063.060.768.0
Qwen3.8-Max62.030.70.860.3146.080.563.362.173.7
gpt6-sol—52.30.880.6043.592.568.066.082.3
hy4—35.70.840.3638.577.057.856.172.0
MiMo-V2.6-Pro52.340.70.850.4854.070.062.061.868.7
doubao-seed-2.1-pro48.344.00.860.4583.034.058.555.873.0
DeepSeek V4.1 Flash—32.00.840.3148.569.058.858.358.3
intern-s2—34.70.850.3859.061.060.060.066.3
Qwen3.5-9B (backbone)35.030.30.820.3032.067.049.547.966.3
VST-7B-RL32.028.70.810.2633.560.046.845.864.7
SN-SI-1.146.334.00.830.2734.068.551.349.870.3
Cambrian-S-7B36.330.00.810.2131.072.551.849.657.3
SpatialSherlock78.781.30.880.6881.585.083.383.284.0

SpatialSherlock exceeds the strongest comparison baseline by 16.7, 29.0, 15.3, and 1.7 percentage points on Video, Diagnose, Judge Mean, and Explain, respectively. It achieves the highest Judge macro-F1 and DeepSeek score, and ties for the highest BERTScore-F1. We will expand Table 1 and Section 5.1, and clarify the metrics in Appendix F.

5. Robustness to Gaussian input noise

We evaluated the trained SpatialSherlock with fixed weights under Gaussian input noise. RGB pixels are normalized to [0,1][0,1], perturbed, and clipped:

\epsilon_t\sim\mathcal{N}(0,\sigma^2),$$ with $\sigma\in\{0.01,0.05,0.20\}$. Units and metric definitions follow the table above. Macro-F1 is computed from each condition's confusion matrix before rounding. | Input condition | Video | Diagnose | | | Judge | | | | Explain | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | | Acc. | Acc. | BERTScore-F1 | DeepSeek | Neg. | Pos. | Mean | Macro-F1 | Acc. | | Original input | 78.7 | 81.3 | 0.88 | 0.68 | 81.5 | 85.0 | 83.3 | 83.2 | 84.0 | | $\sigma=0.01$ | 78.0 | 80.7 | 0.87 | 0.68 | 81.5 | 82.0 | 81.8 | 81.7 | 85.3 | | $\sigma=0.05$ | 70.7 | 74.3 | 0.87 | 0.60 | 78.0 | 68.5 | 73.3 | 73.2 | 80.0 | | $\sigma=0.20$ | 40.0 | 44.0 | 0.80 | 0.34 | 71.0 | 32.0 | 51.5 | 49.6 | 71.3 | At $\sigma=0.01$, Video and Diagnose decrease by only 0.7 and 0.6 percentage points. Stronger noise reduces task performance and explanation quality. At $\sigma=0.20$, Video and Diagnose fall to 40.0% and 44.0%, Judge macro-F1 falls from 83.2% to 49.6%, and the DeepSeek score falls from 0.68 to 0.34. Judge accuracy drops more sharply for consistent cases, indicating more frequent misclassification of noisy consistent inputs as inconsistent. These results show performance close to the original results under the tested low-noise condition and sensitivity to stronger noise. We will add **a standalone appendix section on noise robustness**, including the protocol, complete results, and analysis, with a brief discussion in **Section 5.2**.
AuthorsOpenReview ↗

Follow-up on our response to your comments

Thank you again for your time, thoughtful review, and actionable suggestions. Across our three responses, we have provided the requested clarifications and experiments—specifically detailing the language reward formulation and how the Region (union IoU coverage) and Region+Correction (matched correspondence) rewards play distinct roles, comparing GRPO against PPO alongside full reward and SFT ablations, expanding Table 1 with additional commercial and spatial baselines plus Judge macro-F1, BERTScore-F1, and DeepSeek scores, and evaluating robustness under Gaussian input noise.

As the discussion period wraps up, we wanted to kindly check whether our responses and planned revisions address your points, or if there is anything else we can clarify.

Reviewer_fqWsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SpatialSherlock, a post-trained VLM based on Qwen3.5-9B, designed to improve spatial reasoning and serve as a diagnostic tool for spatial inconsistencies in AI generated videos. The authors construct spatial-skill data from existing datasets and annotations, and diagnostic data from human-annotated generated videos, using Qwen3.8-Max to generate task examples and additional reasoning annotations. They also introduce a benchmark drawn from the held-out diagnostic data, covering four tasks, i.e., Judge, Explain, Diagnose, and Video. SpatialSherlock outperforms the evaluated Gemini and Qwen models on this benchmark and shows strong performance against open-source VLMs on external spatial reasoning benchmarks as well.

Critical strengths

The paper specifically addresses spatial inconsistencies regarding "object pose and spatial relations" considering camera motions and viewpoint changes, which is interesting and indeed useful for validating AI generated videos and may have potential to enhance their spatial consistency. The authors further demonstrate improved performance across multiple existing spatial reasoning benchmarks beyond their own, reporting strong results on several of them.

Critical weaknesses

I believe SpatialSherlock could be useful in practice, but the current manuscript requires substantial improvement towards acceptance.

First, the data construction procedure, statistics, quality control, ... etc are central to the contribution of papers like this, yet several important details remain unclear. For example, regarding the spatial-skill data construction part, authors didn't give clear ideas on how they use labels like camera parameters and 3D geometry with ScanNet. How are intrinsic/extrinsic parameters processed and provided to VLM or are they converted to text instructions? As of my knowledge, most of VLMs have low understanding on low level geometry + raw info, so it is unclear how much of the supervision is derived from the original geometry. Need explanation here. The generated-video annotation pipeline also lacks details, such as how many annotators participated, how was annotation reliability assessed, and what proportion of examples was filtered and what was the filtering algorithm? For both cases details on recipes need clarification and explanation.

Second, the quantitative evaluation needs further improvement. This paper does not include validating on standard VLM benchmarks beyond spatial reasoning to evaluate whether targeted SFT/RL is still able to preserve the backbone’s general multimodal capabilities. Also, it would strengthen the paper if Tab1 includes other open-source VLM baselines from Tab2, and I believe Tab4 should be discussed in the main paper. One more thing is, the paper starts with the motivation by emphasizing the value of diagnosing AI-generated videos. Although the proposed benchmark contains videos from multiple generative models, it would strengthen the story to use SpatialSherlock itself as an evaluator for comparing the spatial consistency of different video generation models. Finally, the qualitative analysis provided in the paper is limited and needs additional failure cases analysis.

Minor comments are

  • Fig3 also needs improvement to be more self-contained and clear, especially Fig3(b) part.
  • Have a question for "consistent" cases: are explanations intentionally annotated to be generic, such as “Positions, orientations, and spatial relationships remain coherent throughout the video” (Fig4)? or is the model capable of answer with detailed explanation for consistent case as well?
Questions for authors

See weakness for the details.

4 comments
AuthorsOpenReview ↗

Response 1/3: Data Construction, Quality Control, and General Multimodal Capabilities

Thank you for recognizing the practical value of SpatialSherlock. We agree with your constructive suggestions on data construction, evaluation coverage, and presentation. We have completed additional evaluations and ablations, and will incorporate the results and methodological clarifications into the revised manuscript. Below, we address your comments in order.

1. How camera parameters and geometry provide spatial-skill supervision

We first convert geometric annotations into spatial relations programmatically. These relations and reference-frame descriptions are then provided as text, together with images, to Qwen3.8-Max for QA and reasoning generation.

For example, in ScanNet++, we project the instance-annotated scene mesh using camera intrinsics and COLMAP world-to-camera poses, then determine image-relative left/right relations from visible object regions. For object-relative relations, 3D-Orienter (point clouds) or Orient Anything V2 (images) estimates missing semantic front directions. Relative positions are then computed in the reference object's coordinate frame. Qwen3.8-Max integrates the images, computed relations, and orientation cues to produce answers and reasoning.

ScanNet and ARKitScenes require adaptations to their coordinate conventions and annotations; HSSD provides usable semantic-orientation annotations. For this task, MS-COCO supports image-relative judgments because it lacks native 3D geometry and camera parameters. We will expand Appendix C with this pipeline, dataset-specific adaptations, and auxiliary-model references. The spatial-skill pool contains approximately 200,000 examples.

2. Generated-video annotation and quality control

The pipeline has three stages:

  • Initial screening: Qwen3.8-Max performs retain/reject classification to remove empty shots, blur, ghosting, and other low-quality or uninformative examples, filtering approximately 30%.
  • Human annotation and review: Approximately 30 annotators from two teams participate. Each example undergoes annotation, verification, and final review by three different people. A rejection triggers the full workflow again with new personnel; approximately 10% are discarded because consensus cannot be reached. Our algorithm team additionally audits 1% of retained annotations, totaling over 1,000 across batches. The audit checks judgments and temporal evidence, region accuracy and completeness, region-specific corrections, and visually supported explanations. Any detected error triggers another final review of the entire batch; acceptance requires an audit with no detected errors.
  • Reasoning construction and second screening: Qwen3.8-Max generates reasoning and answers from visual inputs and human annotations, flagging suspected annotation errors for human review. When the original annotation is confirmed, reviewers provide more detailed visual evidence for reasoning construction. Qwen3.5-9B checks reasoning–answer compatibility to filter outputs with weak logical coherence. Approximately 15% are removed at this stage.

The resulting diagnostic pool contains 92,000 task examples: 40,000 Judge, 15,000 Explain, 12,000 Diagnose, and 25,000 Video. We will document these procedures and filtering statistics in Appendix C.

3. General multimodal capabilities

We evaluated six additional benchmarks covering general visual QA (MMBench, RealWorldQA, MMStar), multidisciplinary reasoning (MMMU), and text/diagram understanding (OCRBench, AI2D). Models and their respective backbones use the same evaluation protocols. Scores are on a 0–100 scale, including rescaled OCRBench scores. We first report changes from each model's backbone:

BenchmarkSpatialSherlock ΔVST-7B-RL ΔSN-SI-1.1 ΔCambrian-S-7B Δ
MMBench+0.9−0.5−1.1+0.7
RealWorldQA−4.60.0−6.9−1.3
MMMU−7.6−9.2−3.3+1.8
OCRBench+1.4−0.3−4.3+0.8
AI2D−2.5−0.4+2.9−0.1
MMStar−3.4−0.4−2.8−0.6

The corresponding absolute scores are:

ModelMMBenchRealWorldQAMMMUOCRBenchAI2DMMStar
Qwen3.5-9B (backbone)90.180.378.489.290.279.7
SpatialSherlock91.075.770.890.687.776.3
VST-7B-RL83.068.549.486.183.563.5
SN-SI-1.183.563.857.686.384.265.7
Cambrian-S-7B80.464.848.064.876.943.9

SpatialSherlock improves on MMBench and OCRBench but declines on four benchmarks, most notably MMMU (−7.6 points). We regard this as a reasonable trade-off for its gains in spatial diagnosis and across all nine external spatial benchmarks, and will explicitly discuss it. Its absolute scores remain higher than those of the three compared spatial models on all six general benchmarks.

AuthorsOpenReview ↗

Response 2/3: Expanded Baselines and Complementary Evaluation Metrics

4. Expanded baselines and complementary metrics

Following your suggestion, we evaluated the three open-source spatial models from Table 2 on the four diagnostic tasks. We also included the original Qwen3.5-9B backbone and additional commercial models. We added Judge macro-F1 and two metrics for Diagnose explanations: BERTScore-F1 and a DeepSeek-based score.

The original accuracies follow Appendix F. Explain requires a correct explanation given the judgment; Diagnose requires both a correct judgment and explanation; Video additionally requires correct temporal evidence. Explanations are assessed by human evaluators. The two new explanation metrics evaluate the explanation component of Diagnose.

For the DeepSeek score, DeepSeek V4.1 Flash outputs probabilities over the same four categories used by the language reward: (1) complete coverage of reference facts without errors or unsupported claims; (2) at least one complete matching fact, with omissions only; (3) at least one complete matching fact, alongside errors or unsupported claims; and (4) no complete matching fact. The score is the probability-weighted sum with weights (1,0.5,0.25,0)(1, 0.5, 0.25, 0).

All results are consolidated below. Accuracies and macro-F1 are percentages; BERTScore-F1 and DeepSeek scores use a 0–1 scale. Judge negatives are inconsistent cases and positives are consistent cases; its mean equally weights the two classes. “—” indicates that the model provider does not support video evaluation. Bold marks the highest value in each column.

ModelVideoDiagnoseJudgeExplain
Acc.Acc.BERTScore-F1DeepSeekNeg.Pos.MeanMacro-F1Acc.
Gemini 3.1 Pro56.334.30.860.3786.533.560.057.056.7
Gemini 3.5 Flash50.741.00.860.4392.516.054.346.461.0
Qwen3.8 Flash46.037.00.850.3739.087.063.060.768.0
Qwen3.8-Max62.030.70.860.3146.080.563.362.173.7
gpt6-sol—52.30.880.6043.592.568.066.082.3
hy4—35.70.840.3638.577.057.856.172.0
MiMo-V2.6-Pro52.340.70.850.4854.070.062.061.868.7
doubao-seed-2.1-pro48.344.00.860.4583.034.058.555.873.0
DeepSeek V4.1 Flash—32.00.840.3148.569.058.858.358.3
intern-s2—34.70.850.3859.061.060.060.066.3
Qwen3.5-9B (backbone)35.030.30.820.3032.067.049.547.966.3
VST-7B-RL32.028.70.810.2633.560.046.845.864.7
SN-SI-1.146.334.00.830.2734.068.551.349.870.3
Cambrian-S-7B36.330.00.810.2131.072.551.849.657.3
SpatialSherlock78.781.30.880.6881.585.083.383.284.0

SpatialSherlock exceeds the strongest comparison baseline by 16.7, 29.0, 15.3, and 1.7 percentage points on Video, Diagnose, Judge mean, and Explain, respectively. Relative to its backbone, Video improves from 35.0% to 78.7% and Diagnose from 30.3% to 81.3%. It achieves the highest Judge macro-F1 and ties for the highest BERTScore-F1. Its DeepSeek score is also the highest, at 0.68 versus 0.60 for the strongest comparison model. We will incorporate this expanded comparison into Table 1 and update the accompanying discussion.

AuthorsOpenReview ↗

Response 3/3: Training Ablations, Practical Applications, and Qualitative Analysis

5. Training and reward ablations in Table 4

We agree that Table 4 warrants discussion in the main text. We will expand it to include PPO and reward ablations alongside the existing SFT and continued-SFT controls.

For joint diagnosis with nonempty reference regions, the full reward is C(α+βRunion+γRmatch+δRlang)C(\alpha+\beta R_{\mathrm{union}}+\gamma R_{\mathrm{match}}+\delta R_{\mathrm{lang}}). Here, CC is the correctness gate; the three reward terms evaluate region coverage, region–correction matching, and language agreement. The ablated combinations retain the CC gate; all retained coefficients keep their original values, without retuning or renormalization. Results are percentages; Judge reports the class mean.

ConfigurationVideoDiagnoseJudgeExplain
SFT—66.772.073.3
SFT + continued SFT60.370.778.076.3
SFT + RL (PPO)75.384.788.082.7
SFT + RL (CC)40.050.394.062.0
SFT + RL (C(α+δRlang)C(\alpha+\delta R_{\mathrm{lang}}))68.773.085.386.7
SFT + RL (C(α+βRunion+γRmatch)C(\alpha+\beta R_{\mathrm{union}}+\gamma R_{\mathrm{match}}))60.364.086.370.7
SpatialSherlock (SFT + GRPO, full reward)78.781.383.384.0

The comparisons support the following conclusions:

  • RL versus continued SFT: Both start from the same SFT checkpoint and use the same four-task data, task mixture, and reference annotations, with comparable compute. The full method improves Video and Diagnose over continued SFT by 18.4 and 10.6 percentage points.
  • Correctness alone: Using only CC achieves 94.0% on Judge but 50.3% on Diagnose, showing that accurate classification alone is insufficient for complete diagnosis.
  • Language reward: Removing it reduces Diagnose and Explain to 64.0% and 70.7%. We observe that the model can make the correct judgment and localize the anomalous region yet explain it incorrectly.
  • Region and correction rewards: With only correctness and language rewards, Explain reaches 86.7%. Video and Diagnose reach 68.7% and 73.0%, versus 78.7% and 81.3% for the full method. Bounding boxes help the model understand the cause of an inconsistency, supporting complete diagnosis.
  • PPO versus GRPO: PPO performs better on Diagnose and Judge, whereas our GRPO configuration performs better on Video and Explain.

6. Comparing video generators with SpatialSherlock

We used the same prompts to generate 50 videos per model. The prompts come from sources unrelated to our training and evaluation data. SpatialSherlock judges each complete video as spatially consistent or inconsistent; the table reports the percentage judged consistent.

Video generatorSpatial consistency rate
Seedance 2.576%
Wan 3.072%
MiniMax H370%
Seedance 2.066%
HappyHorse 1.162%
Kling V358%
Kling V3 Omni58%
HappyHorse 1.054%
Grok Imagine Video52%
Vidu Q3 Pro50%
Hailuo 2.346%
Vidu Q2 Pro42%
Veo 3.138%

Rates range from 38% to 76% on this prompt set. This experiment demonstrates the use of SpatialSherlock to compare generators under shared prompts and provide diagnoses for videos judged inconsistent. We will add the setup and results to the revised manuscript.

7. Failure cases

Our observed failures concentrate on large changes in shot scale, such as transitions from frontal extreme long shots to oblique frontal close shots or close-ups. Depth differences between people are less pronounced in the extreme long shot but become pronounced in the close view. The model can mistake these differences for changes in their actual distances. We will add a dedicated appendix with multiple illustrated failure cases and detailed analyses in an upcoming manuscript update.

8. Clarity of Figure 3

We agree that Figure 3 is crowded, particularly panel (b). We will improve its layout and explanations to make the figure clearer and more self-contained.

9. Detailed explanations for consistent cases

Consistent examples do not use a prescribed generic explanation template. Qwen3.8-Max generates reasoning and answers from the specific visual inputs and human diagnostic annotations. SpatialSherlock can also provide detailed explanations for consistent cases.

To judge consistency, the model checks all regions and rules out suspicious areas; a single confirmed spatial contradiction suffices for an inconsistent judgment. Accordingly, we observe reasoning traces that are approximately 50%–100% longer for consistent cases.

Figure 4 shows the answer portion of the second inference call, with the corresponding reasoning omitted. After establishing consistency in its reasoning, the answer has no anomalous changes to summarize, so the wording can look similar across examples. We will add detailed explanations and analyses of consistent cases in the upcoming manuscript update.

AuthorsOpenReview ↗

Follow-up on our response to your concerns

Thank you once again for your thorough review and constructive feedback. Following your suggestions, we have added comprehensive clarifications and new experiments across our three responses—specifically detailing our geometry-to-text spatial relation pipeline and three-stage quality control statistics, evaluating general multimodal preservation across six standard benchmarks, expanding Table 1 with open-source/commercial baselines and new explanation metrics, applying SpatialSherlock to benchmark 13 video generators, and providing failure case analyses along with detailed explanations for consistent cases.

As the discussion period is drawing to a close, we wanted to kindly check whether our responses and planned revisions adequately address your concerns, or if there are any remaining points we could further clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Additional Experiments

We thank the Area Chair and all three reviewers for their careful reading and constructive feedback. We have revised the manuscript to incorporate the additional experiments and clarifications discussed in our responses. The main updates are summarized below, with the corresponding reviewers indicated.

  1. Expanded comparisons and evaluation metrics (fqWs, WY8H, DXp5; Sec. 5.1, Appendix F). We added six general-purpose models, the Qwen3.5-9B backbone, and three open-source spatial models to the diagnostic comparison. We also added Judge macro-F1, BERTScore-F1, and a DeepSeek-based explanation score alongside human evaluation. SpatialSherlock leads all four diagnostic tasks, with a 29.0-point Diagnose advantage over the strongest comparison baseline.

  2. Data construction and evaluation reliability (fqWs, DXp5; Appendices C and F). We expanded the descriptions of geometry-based spatial-skill reconstruction and annotation quality control, including three-person annotation and review, batch audits covering over 1,000 examples, and filtering statistics. We also clarified that test selection is independent of the teacher's answer correctness and that all samples derived from held-out source videos are excluded from training.

  3. Training and reward ablations (fqWs, WY8H; Sec. 5.4, Appendix I). We added PPO and reward-combination comparisons and brought the training ablations and their discussion into the main text. The results show the contribution of combined feedback to complete diagnosis and the task trade-offs between PPO and GRPO: PPO performs better on Diagnose and Judge, while GRPO performs better on Video and Explain.

  4. Video inference comparisons (DXp5; Appendices F.4-F.5). We compared direct video input with the SpatialSherlock pipeline for four Gemini/Qwen baselines and provided the full direct-video prompt. Direct input performs better for all four. Based on these results, all comparison models reporting Video scores use direct input; SpatialSherlock retains its local diagnosis, conditional verification, and video aggregation pipeline.

  5. General multimodal capabilities (fqWs; Sec. 5.2, Appendix G.2). We added six general multimodal benchmarks, reporting absolute scores and changes from each model's backbone. The revised discussion explicitly reports improvements on two benchmarks and declines on four, including a 7.6-point decline on MMMU.

  6. Robustness to input noise (WY8H; Sec. 5.2, Appendix H). We added Gaussian input-noise tests at three intensities with fixed model weights. The results document near-original Video and Diagnose performance under low noise and degradation under stronger noise.

  7. Failure analysis and presentation (fqWs; Appendix K, Figs. 3 and 5). We added three illustrated failure cases showing how changes in shot scale and viewpoint can lead to false diagnoses of distance inconsistencies. We also improved the readability of Figure 3.

  8. Application to video-generator comparison (fqWs; Sec. 5.5, Appendix J). We added an evaluation of 13 video generators using shared prompts independent of our training and evaluation data, with 50 videos per generator. We report the proportion judged spatially consistent by SpatialSherlock, ranging from 38% to 76% on this prompt set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.