- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SYNCR introduces a simulation-grounded framework for evaluating and improving cross-video reasoning in multimodal large language models (MLLMs). It defines eight tasks across temporal alignment, spatial tracking, comparative reasoning, and holistic synthesis. Using Habitat, Kubric, and CLEVRER, the framework constructs 4,000 evaluation questions and 15,960 training questions with simulator-derived ground-truth labels. Experiments on 22 MLLMs reveal substantial weaknesses in physical comparison and scene integration. Supervised fine-tuning improves Qwen3-VL-8B's average SYNCR accuracy from 32.6% to 61.6%.
Critical strengths
-
Data construction. Simulator-derived annotations provide precise labels for temporal, spatial, and physical reasoning tasks. The shared generators also support both controlled evaluation and targeted training.
-
Extensive training and generalization experiments. The authors examine supervised fine-tuning, reinforcement learning, task variations, and simulation-to-real transfer.
Critical weaknesses
-
Unclear need for genuine cross-video reasoning. Several tasks, such as collision counting and kinematic comparison, can be solved through independent per-video analysis and simple aggregation. The paper lacks video concatenation and independent-reasoning baselines to establish the need for joint cross-video processing.
-
Limited task realism and temporal complexity. The tasks rely on synthetic, template-based multiple-choice questions without clear validation against real user needs. All video clips are at most 20 seconds long, limiting the evaluation of long-range and complex cross-video reasoning.
-
Insufficient isolation of training contributions. Multi-task SFT uses substantially more training examples than single-task SFT, making it difficult to separate task synergy from increased training exposure.
Questions for authors
-
How does the proposed multi-video setting compare with simply concatenating the videos or analyzing each video independently? Can the authors provide baseline comparisons to demonstrate the limitations of these alternatives and the benefits of joint cross-video reasoning?
-
Can the authors report annotator agreement?
-
How sensitive are the results to frame sampling rates and visual resolution, especially for synchronization and kinematic comparison? Can the authors evaluate these effects under controlled frame budgets?