- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SpatialSherlock studies the diagnosis of spatial inconsistencies in generated videos, including whether apparent discrepancies can be explained by intervening camera motion, object motion, or occlusion. Its main contribution is a data pipeline combining approximately 200,000 spatial-skill examples from public datasets with structured human diagnoses of more than 20,000 generated videos. The latter are curated into 92,000 examples across Judge, Explain, Diagnose, and Video tasks, with references for regions, spatial corrections, explanations, and temporal evidence. A Qwen3.5-9B backbone is trained through SFT and joint GRPO with component-level rewards. At inference, local diagnoses are conditionally verified against intervening frames before aggregation. The model achieves 81.3% on frame-pair Diagnose and 78.7% on source-video diagnosis, and improves over its backbone on all nine evaluated external spatial benchmarks.
Critical strengths
-
A useful form of structured supervision for a consequential diagnostic problem. The data records connect a consistency judgment to affected regions, required spatial corrections, and an explanation, rather than providing only a scalar quality label. Verifying endpoint discrepancies against intervening events also addresses a real ambiguity in video diagnosis. Keeping all examples derived from a source video in the same partition is an appropriate safeguard against leakage between these closely related examples (Section 3 and Appendix C). This is a substantive data and task-design contribution despite using an existing backbone and standard training algorithms.
-
Strong empirical results with informative controls. The reported Diagnose and Video scores exceed the strongest tested commercial baselines by 40.3 and 16.7 percentage points, respectively; gains of 1.6–16.6 points over the backbone also appear across nine external spatial benchmarks. Importantly, Appendix H.2 compares RL with continued SFT from the same checkpoint using the same four-task pool, task mixture, and references, with comparable compute including rollout and reward evaluation. RL improves Video from 60.3% to 78.7% and Diagnose from 70.7% to 81.3%. The data-source ablations further support combining spatial-skill and diagnostic supervision. These controls make the training contribution more convincing than an SFT-versus-RL comparison that also introduces additional task data.
Critical weaknesses
-
The reliability of the new diagnostic benchmark is insufficiently documented. Appendix F appropriately uses human evaluators to check explanations against references, and Diagnose requires both a correct judgment and explanation. However, the paper does not report annotator/evaluator counts, inter-rater agreement, adjudication, or whether evaluation is blinded and independent of reference construction. These details matter because plausible camera/scene interpretations and alternative valid spatial corrections may make reference matching subjective. The 300-example Diagnose set is also selected from cases flagged as challenging, making a clear sampling protocol particularly important. Without an annotation and scoring reliability audit, the large gains on this newly constructed benchmark are harder to assess precisely.
-
The video comparison leaves inference-procedure effects unresolved. SpatialSherlock enumerates intervals and boundary pairs, performs local diagnosis, and uses a second model call to verify diagnoses when intervening frames are available. Appendix F scores the aggregated source-video output, but does not make clear whether commercial baselines receive the same interval proposals, verification opportunity, and comparable frame/token/call budgets. Nor is temporal verification isolated by an inference ablation. Thus, the reported 16.7-point Video advantage cannot yet be attributed clearly between the trained model and the inference procedure. This limits the comparative interpretation and the evidence for the proposed verification stage, while leaving the frame-pair and external-benchmark improvements encouraging.
Questions for authors
-
Annotation and evaluation reliability. How many independent people annotate and score each example, how are disagreements and alternative valid diagnoses handled, and are model identities hidden from evaluators? Please clarify the challenging-case sampling rule and report agreement or an audit on the existing test set. This information could substantially change confidence in the diagnostic gains.
-
Comparable video inference. Were all Table 1 baselines evaluated with the same interval/boundary proposals and the same opportunity to verify an initial diagnosis against intervening frames? Please state the per-video frame, token, and model-call budgets, and compare the full procedure with verification removed on the existing Video test set. A comparison under matched procedures would clarify both baseline fairness and the contribution of temporal verification.
4 comments
Response 2/2: Expanded results and video inference comparisons
5. Expanded comparison results
We expanded the comparison to include the three open-source spatial models from Table 2, the original Qwen3.5-9B backbone, and additional commercial models.
Accuracies and macro-F1 are percentages; BERTScore-F1 and DeepSeek scores use a 0–1 scale and evaluate Diagnose explanations. Judge negatives are inconsistent cases and positives are consistent cases; its mean equally weights the two classes. In this table, “—” indicates that the model provider does not support video evaluation. Bold marks the highest value in each column.
| Model | Video | Diagnose | Judge | Explain | |||||
|---|---|---|---|---|---|---|---|---|---|
| Acc. | Acc. | BERTScore-F1 | DeepSeek | Neg. | Pos. | Mean | Macro-F1 | Acc. | |
| Gemini 3.1 Pro | 56.3 | 34.3 | 0.86 | 0.37 | 86.5 | 33.5 | 60.0 | 57.0 | 56.7 |
| Gemini 3.5 Flash | 50.7 | 41.0 | 0.86 | 0.43 | 92.5 | 16.0 | 54.3 | 46.4 | 61.0 |
| Qwen3.8 Flash | 46.0 | 37.0 | 0.85 | 0.37 | 39.0 | 87.0 | 63.0 | 60.7 | 68.0 |
| Qwen3.8-Max | 62.0 | 30.7 | 0.86 | 0.31 | 46.0 | 80.5 | 63.3 | 62.1 | 73.7 |
| gpt6-sol | — | 52.3 | 0.88 | 0.60 | 43.5 | 92.5 | 68.0 | 66.0 | 82.3 |
| hy4 | — | 35.7 | 0.84 | 0.36 | 38.5 | 77.0 | 57.8 | 56.1 | 72.0 |
| MiMo-V2.6-Pro | 52.3 | 40.7 | 0.85 | 0.48 | 54.0 | 70.0 | 62.0 | 61.8 | 68.7 |
| doubao-seed-2.1-pro | 48.3 | 44.0 | 0.86 | 0.45 | 83.0 | 34.0 | 58.5 | 55.8 | 73.0 |
| DeepSeek V4.1 Flash | — | 32.0 | 0.84 | 0.31 | 48.5 | 69.0 | 58.8 | 58.3 | 58.3 |
| intern-s2 | — | 34.7 | 0.85 | 0.38 | 59.0 | 61.0 | 60.0 | 60.0 | 66.3 |
| Qwen3.5-9B (backbone) | 35.0 | 30.3 | 0.82 | 0.30 | 32.0 | 67.0 | 49.5 | 47.9 | 66.3 |
| VST-7B-RL | 32.0 | 28.7 | 0.81 | 0.26 | 33.5 | 60.0 | 46.8 | 45.8 | 64.7 |
| SN-SI-1.1 | 46.3 | 34.0 | 0.83 | 0.27 | 34.0 | 68.5 | 51.3 | 49.8 | 70.3 |
| Cambrian-S-7B | 36.3 | 30.0 | 0.81 | 0.21 | 31.0 | 72.5 | 51.8 | 49.6 | 57.3 |
| SpatialSherlock | 78.7 | 81.3 | 0.88 | 0.68 | 81.5 | 85.0 | 83.3 | 83.2 | 84.0 |
SpatialSherlock achieves 81.3% Diagnose accuracy, exceeding the strongest comparison baseline by 29.0 percentage points. Its BERTScore-F1 of 0.88 is tied for the highest, and its DeepSeek score of 0.68 exceeds the strongest comparison model's 0.60. These complementary automatic metrics support the explanation-quality gains observed in human evaluation.
6. Video inference procedures and baseline comparisons
The four commercial models in the current manuscript were evaluated using direct video input and a video analysis prompt. SpatialSherlock uses local diagnosis, conditional temporal verification, and aggregation at the source-video level. These configurations therefore use different inference procedures.
The direct-video prompt asks whether observed changes follow natural temporal progression and legitimate camera movement within a single continuous 3D world. It distinguishes 3D relationships from image projection changes and asks the model to consider camera movement, occlusion, object motion, and lighting before declaring an inconsistency. It requests an overall verdict and comprehensive reasoning. The prompt is provided in the supplementary comment.
We additionally evaluated these four commercial models using the SpatialSherlock pipeline. Video accuracies under the two configurations are:
| Model | SpatialSherlock pipeline | Direct video input + prompt |
|---|---|---|
| Gemini 3.1 Pro | 37.3% | 56.3% |
| Gemini 3.5 Flash | 28.0% | 50.7% |
| Qwen3.8 Flash | 40.0% | 46.0% |
| Qwen3.8-Max | 61.3% | 62.0% |
All four models achieved higher accuracy with direct video input; the two Gemini models improved by 19.0 and 22.7 percentage points, respectively. In our experiments, Gemini frequently mistook viewpoint changes, such as camera rotation, for spatial inconsistencies during two-frame diagnosis. These false positives affected subsequent processing and the final video judgment. Its higher accuracy on inconsistent cases and lower accuracy on consistent cases in Judge show a corresponding tendency.
The current manuscript therefore reports the better-performing direct-video configuration for these four commercial models. SpatialSherlock's 78.7% Video accuracy exceeds the strongest direct-video baseline (62.0%) by 16.7 percentage points. Under the SpatialSherlock pipeline, the best result among these four commercial models is 61.3%, corresponding to a gap of 17.4 points. These comparisons support the reported baseline configuration and provide additional comparisons using the SpatialSherlock pipeline. The 16.7-point gap describes the overall performance difference between the reported model-and-procedure configurations.
Supplement: Direct-video evaluation prompt
You will watch a video and determine whether it is spatially and temporally consistent. Treat the video as an observation of a single continuous 3D physical world in which time naturally progresses forward and the camera legitimately moves (panning, rotating, tilting, zooming). Entities in the video can move, act, and interact; the scene can naturally evolve over time (doors opening, objects being carried, lighting changing). Your task is to determine whether all observable changes throughout the video can be explained by "the natural progression of time and legitimate camera movement within the same continuous 3D world." A dimension is considered inconsistent only when some entity, spatial relationship, scene structure, or action state exhibits a discontinuity that cannot be explained by a continuous physical process. [Judgment Rules]
- When judging position, orientation, left/right, front/back, distance, and height, use the 3D scene coordinate system, not the image coordinate system. The camera's rotation angle determines how the image changes—a 180° orbit swaps left and right in the image, but a 120° rotation may not. You cannot infer whether 3D positions have changed simply from whether left and right are swapped in the image.
- Use the entity's own frontal facing direction as the positive direction—"front" is the 3D direction its face points toward, "back" is the opposite. Do not use image foreground/background to judge 3D front/back. Left/right of body parts are based on the frontal facing direction and do not change with camera angle.
- Camera distance and focal length changes alter projection size. When judging scale, look at the same object's proportional relationship relative to scene anchors.
- Changes in entity position, posture, and scene state throughout the video must be gradual and explainable by physical processes. Discontinuities (teleportation, instant costume changes, objects appearing/vanishing out of thin air) are inconsistent. Permitted changes include: entities moving naturally, objects being carried, doors opening/closing, lighting gradually changing, and field-of-view changes due to camera movement.
- Before judging that something has "disappeared" or "appeared," first rule out each of—(1) Camera cropping: cut off at the image edge; (2) Foreground occlusion: blocked by a foreground object; (3) Out of field of view: camera turning causes it to leave the visible range; (4) Front-back coverage: camera moves to the other side of the subject, making something invisible; (5) Subject displacement occlusion: the subject moves and occludes or reveals other objects; (6) Temporal progression: the entity naturally leaves or enters the frame area over time. If any holds, it is consistent.
- Color shifts, texture differences, and reflection changes of the same material under different lighting/white balance, as well as changes in visible surfaces due to viewpoint, are all legitimate. Gradual lighting changes over time in a video are likewise legitimate.
- Focus on "whether entities/structures/relationships have undergone holistic replacement, disappearance, addition, or fundamental change." For entities or structures that can already be matched, their surface details (accessories, clothing texture, color brightness, material reflections, background texture, etc.) are not within the judgment scope. The granularity is "whether something has been replaced by a different thing," not "whether a small element on the same thing is completely identical."
- Do not make forced inferences about occluded, depth-of-field blurred, or insufficient-resolution regions. When evidence is insufficient, output "undetermined." [Three Verdict Levels]
- Consistent: All changes in the video can be explained by "the natural progression of time and legitimate camera movement within the same continuous 3D world."
- Inconsistent: At least one real 3D scene discontinuity exists that cannot be explained by a continuous physical process.
- Undetermined: Video information is insufficient to make a determination. If the video is unclear but a reasonable judgment has been made within the visible content, lean toward consistent. Complete the analysis of the video first, then output the conclusion in the following fixed format with immutable order for programmatic parsing: === OVERALL VERDICT === <Consistent / Inconsistent / Undetermined> === COMPREHENSIVE REASON === <Comprehensive reasoning. State all discovered issues completely without omission; for each issue, explain what aspect it concerns (entity, spatial relationship, scene structure, action state, etc.) and why it cannot be explained by camera movement or temporal progression. If no issues exist, explain how the observed changes in the video are reasonably explained.> === END ===
Follow-up on our response to your comments
Thank you again for your encouraging review and detailed questions. Across our responses, we have clarified the benchmark construction and evaluation reliability protocols—specifically detailing the three-stage annotation workflow, 1% batch audits, selection criteria for the 300 unambiguous Diagnose test cases, blinded two-evaluator scoring independent of ground-truth construction, and complementary automatic metrics (BERTScore-F1 and DeepSeek score). We also evaluated the commercial video baselines under both direct video input and the SpatialSherlock multi-stage pipeline (alongside sharing the full evaluation prompt), showing consistent gains of 16.7 and 17.4 percentage points, respectively.
As the discussion period draws to a close, we wanted to kindly check whether our responses and additional comparisons address your concerns, or if there are any remaining points we could further clarify.
Response 1/2: Annotation, evaluation reliability, and scoring metrics
Thank you for your positive assessment of SpatialSherlock, including its frame-pair and external-benchmark improvements. We agree with your constructive suggestions on documenting benchmark reliability and clarifying video inference comparisons. This comment explains annotation, evaluation, and scoring metrics. Part 2 reports the expanded results and video comparisons; a separate supplement provides the video evaluation prompt.
1. Annotation and reference construction
Approximately 30 annotators from two teams participated. Each example passed through three different people in sequence: annotation, validation, and final review. Failure at any review stage triggered a restart of the complete process with new personnel. Examples for which consensus could not be reached were excluded, accounting for approximately 10%.
The algorithm team additionally audited retained annotations at a 1% sampling rate, checking more than 1,000 examples across batches. The audit covered consistency judgments, temporal evidence, region accuracy and completeness, corresponding spatial corrections, and whether explanations were visually grounded. Any detected error triggered another final review of the entire batch. A batch was accepted only after an audit found no errors.
2. Test-set construction and challenging-case selection
The Judge, Explain, and Video test sets were obtained by randomly splitting the reviewed annotation dataset and retaining the corresponding ground-truth references (GT).
The 300 Diagnose examples satisfied two criteria: they were flagged as challenging during final review, and their final diagnoses were jointly endorsed by the annotation and algorithm teams. Final reviewers actively questioned and checked the preceding annotation and validation, flagging cases they could not judge quickly. To ensure an unambiguous scoring reference, we selected 300 flagged examples with a unique, undisputed diagnostic answer covering both the consistency judgment and explanation. Cases with disputed diagnoses or without a unique diagnostic answer were excluded.
3. Scoring personnel, blinding, and verification
Each model output was handled sequentially by two evaluators: one assigned the score, and the other verified it. Neither evaluator had participated in constructing the GT for that example. Outputs from different models were mixed, with model identities hidden from evaluators. Scores were based on agreement between the prediction and GT in the judgment and explanatory facts. Predictions that failed to meet the task-specific reference requirements were scored as incorrect. No disagreements arose during scoring and verification.
The original accuracies follow Appendix F. Explain requires a correct explanation given the judgment; Diagnose requires both a correct judgment and explanation; Video additionally requires correct temporal evidence. Explanations are assessed through the human scoring and verification procedure described above.
4. Complementary evaluation metrics
To complement human evaluation, we added BERTScore-F1 and a DeepSeek-based score for the explanation component of Diagnose. These assess semantic similarity and consistency with reference facts, respectively. We also added Judge macro-F1.
For the DeepSeek score, DeepSeek V4.1 Flash outputs probabilities over the same four categories used by the language reward:
The score is the probability-weighted sum with weights (1,0.5,0.25,0). The complete results are provided in Part 2.