acceptodds
Under review as a conference paper at ICLR 2027

SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

Abstract

Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabilities and testing how to improve them requires both reliable labels and targeted supervision. We introduce SYNCR, a simulator-grounded framework that connects these two needs through shared task generators. Built on Habitat, Kubric, and CLEVRER, SYNCR derives answers from environment state and provides 4,000 evaluation questions and 15,960 training questions over disjoint videos, spanning eight cross-video reasoning tasks. Visual ablations and human evaluation assess dependence on the supplied evidence and answer recoverability. Evaluation of 22 multimodal large language models reveals persistent difficulties in physical comparison and scene integration that increasing model size does not consistently resolve. Supervised fine-tuning raises Qwen3-VL-8B’s average SYNCR accuracy from 32.6% to 61.6%, with gains extending to task configurations and video sources absent from training for those tasks. Transfer to real footage is most consistent for temporal ordering: accuracy improves by 9.0–20.5 percentage points on constructed Assembly101 and Panoptic ordering sets across three checkpoints spanning two model families and two model sizes, with additional gains on existing temporal reasoning benchmarks. These results establish SYNCR as a controlled setting for diagnosing cross-video reasoning failures, testing their learnability, and identifying where synthetic supervision transfers.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, video understanding, cross-video reasoning, synthetic data, spatio-temporal reasoning, physical reasoning, supervised fine-tuning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nyCpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SYNCR introduces a simulation-grounded framework for evaluating and improving cross-video reasoning in multimodal large language models (MLLMs). It defines eight tasks across temporal alignment, spatial tracking, comparative reasoning, and holistic synthesis. Using Habitat, Kubric, and CLEVRER, the framework constructs 4,000 evaluation questions and 15,960 training questions with simulator-derived ground-truth labels. Experiments on 22 MLLMs reveal substantial weaknesses in physical comparison and scene integration. Supervised fine-tuning improves Qwen3-VL-8B's average SYNCR accuracy from 32.6% to 61.6%.

Critical strengths
  1. Data construction. Simulator-derived annotations provide precise labels for temporal, spatial, and physical reasoning tasks. The shared generators also support both controlled evaluation and targeted training.

  2. Extensive training and generalization experiments. The authors examine supervised fine-tuning, reinforcement learning, task variations, and simulation-to-real transfer.

Critical weaknesses
  1. Unclear need for genuine cross-video reasoning. Several tasks, such as collision counting and kinematic comparison, can be solved through independent per-video analysis and simple aggregation. The paper lacks video concatenation and independent-reasoning baselines to establish the need for joint cross-video processing.

  2. Limited task realism and temporal complexity. The tasks rely on synthetic, template-based multiple-choice questions without clear validation against real user needs. All video clips are at most 20 seconds long, limiting the evaluation of long-range and complex cross-video reasoning.

  3. Insufficient isolation of training contributions. Multi-task SFT uses substantially more training examples than single-task SFT, making it difficult to separate task synergy from increased training exposure.

Questions for authors
  1. How does the proposed multi-video setting compare with simply concatenating the videos or analyzing each video independently? Can the authors provide baseline comparisons to demonstrate the limitations of these alternatives and the benefits of joint cross-video reasoning?

  2. Can the authors report annotator agreement?

  3. How sensitive are the results to frame sampling rates and visual resolution, especially for synchronization and kinematic comparison? Can the authors evaluate these effects under controlled frame budgets?

Reviewer_j8b1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SYNCR, a framework for training and evaluating cross-video reasoning capabilities. Built on Habitat/HM3D, Kubric, and CLEVRER, it constructs eight tasks covering temporal alignment, spatial tracking, comparison, and scene integration, with 4,000 evaluation questions and 15,960 training questions. The paper also evaluates 22 multimodal models and systematically investigates the effects of supervised fine-tuning, reinforcement learning, configuration shifts, and transfer to real-world videos. Experimental results demonstrate that SFT improves model performance on unseen SYNCR test instances. The most consistent and reproducible transfer gains on real-world videos are primarily observed in temporal ordering tasks.

Critical strengths

A useful connection between evaluation and supervision. The shared task generators allow the authors to evaluate task-level weaknesses and directly test their learnability. Simulator annotations provide access to identities, trajectories, geometry, and temporal offsets that would be costly to annotate manually.

Clear empirical validation. The paper includes visual-input ablations, a human reference, training-seed experiments, incorrect-label supervision, alternative answer formats, and statistical tests. The explicit acknowledgment that simulator-derived labels do not guarantee visual recoverability is also appropriate.

Critical weaknesses

The value of the eight-task suite needs clearer justification. SYNCR builds on established simulators and scene assets. Reusing these resources is reasonable, but it places the novelty primarily in the task formulations, video combinations, and supervision construction. Several tasks resemble extensions of established spatial or temporal tasks, for example, comparing independently estimated collision counts or peak speeds across videos. The paper does not sufficiently explain what capability coverage these formulations add beyond existing spatial-intelligence and video-reasoning work, including VSI-Bench and SenseNova-SI.

The evidence for complementarity among the eight tasks is limited. Fig. 18 shows that the largest improvements are concentrated on the trained tasks, while transfer to excluded tasks is generally weak. Although multi-task training substantially improves synchronization relative to synchronization-only training, the training budgets differ. Thus, the additional value of the full suite over smaller, task-aligned training sets remains unresolved.

The model analysis identifies performance differences but provides limited diagnosis of their causes. The evaluation establishes that model scaling and reasoning-specialized post-training have uneven effects across tasks. However, task-level accuracy does not distinguish failures in per-video perception from failures in cross-video correspondence or relational reasoning. For example, poor kinematic comparison may arise from inaccurate speed estimation, whereas counting errors may arise from missed detections or failed cross-view deduplication. These possibilities imply different research directions.

The transferability and generalization are not well established. The paper does provide detailed task-level results in Figure 5 and Table 19. My concern is their interpretation and attribution, rather than the absence of a breakdown. The strongest replicated results often preserve the trained operation; that is, the constructed Assembly101 and Panoptic sets retain the ordering task, and CrossVid procedural sequencing closely relates to SYNCR ordering. Likewise, the simulator-swap experiments hold out task–simulator combinations rather than entire simulators from training.

The experiments do not sufficiently isolate improvements in cross-video relationship processing. The all-video, single-video, and no-video controls are useful. However, better performance with additional videos can result from access to additional objects or events, without improved correspondence or integration across streams. For example, synchronization remains near chance on EPFL, and the trained route-planning model reaches 39.5% even without videos, indicating exploitable nonvisual regularities. More targeted controls are needed to establish which cross-video relations the model has learned to use.

Questions for authors

What is the purpose of the task formulations relative to existing spatial and temporal reasoning resources? Could you provide a task-level comparison identifying the new relations or failure modes introduced by SYNCR, particularly relative to VSI-Bench, SenseNova-SI, and related synthetic supervision datasets? Please also distinguish reused videos and annotations from newly rendered data and newly generated QA.

What does the full eight-task mixture add beyond smaller, task-aligned supervision? Can you compare ordering-only or temporal-task training, a mixture excluding ordering, and the full suite under matched training budgets on the principal real-video transfer benchmarks? This would help distinguish task complementarity from additional training exposure.

How should the reported transfer be divided into familiar-task domain transfer and unseen-task generalization? Could you explicitly map evaluation tasks to their related training tasks and report results according to this distinction? Where task types are excluded from training, which improvements are robust across seeds and model families?

Can the gains be localized to cross-video correspondence or integration? Could you evaluate trained models with targeted video removal or replacement, or compare against a baseline that extracts information independently from each video before combining it? For counting in particular, can improved object visibility be separated from improved cross-camera deduplication?

Reviewer_8X5dOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SyncR, a framework for evaluating and training cross-video reasoning in MLLMs. It is built on three simulators (Habitat, Kubric, and CLEVRER), and answers are derived from simulator state. There are eight tasks grouped into four families: temporal alignment, spatial tracking, comparative reasoning, and holistic synthesis. The benchmark provides 4,000 evaluation questions and 15,960 training questions over disjoint videos.

THe paper evaluates 22 different models comprising proprietary and open source models. On several tasks, all of the evaluated modes fall well short of a human majority-vote reference. LoRA fine-tuning on the training split raises Qwen3-VL-8B from 32.6% to 61.6% on held-out videos. Gains persist under changed task structures and new task–simulator pairings. On real footage, the most consistent transfer is in temporal ordering.

Critical strengths
  • Simulator-derived labels give exact ground truth for quantities that are hard to annotate at scale: temporal offsets, 3D distances, and velocities. Because the same generators produce both evaluation and training data, a failure can be diagnosed and then tested for learnability under one task definition.
  • The evaluation is broad, and the task-level breakdown shows failures that an average would hide. For example, ordering and synchronization behave very differently.
Critical weaknesses
  • Downstream motivation is thin. The introduction motivates cross-video reasoning with two examples: counting objects across overlapping views and reconstructing events from shuffled clips. It does not connect the capability to concrete applications such as multi-camera surveillance, driving, or multi-view robotics. The experiment closest to such a setting is six-camera counting on nuScenes. There, absolute accuracy stays low, and the larger benefit from extra cameras after training does not replicate across seeds or for InternVL3.5-8B (Table 21). This makes it hard to judge whether SyncR teaches the capabilities those applications need.
  • Evaluation subset sizes are unclear and possibly misreported. The paper uses three different item subsets per task:
    • 500 items for open-weight models,
    • the first 100 for proprietary models and humans (L208–210),
    • and the first 200 for all fine-tuning results.
  • The clearest real-footage transfer is Sequential Ordering, which shuffles gapped segments of a single video. Labels for this task can be produced from any unlabeled real video.
  • The analysis explains little. The results sections report many comparisons but offer little analysis of why models fail. For instance:
    • Why do open-weight models handle ordering but sit at chance on synchronization?
    • What do Object Counting errors look like?
    • There are no qualitative failure examples. Similarly, MVU-Eval is first used without a citation in Sec. 3.3 (cited later at L401). The paper never describes what its Temporal Reasoning and Comparison subsets measure or why those two were chosen.
  • Answers may not be recoverable from the video for Kinematic Comparison. Three observations point this way:
    • The human majority vote reaches only 68%.
    • Gemini-3-Flash scores 4%, well below chance, which suggests the simulator-defined peak 3D speed sometimes conflicts with the object that looks fastest.
Questions for authors
  • How many items per task were actually used for the proprietary and human evaluations?
  • What are individual annotator accuracy and inter-annotator agreement on Kinematic Comparison and Spatial Measurement? Why do proprietary models score below chance on Kinematic Comparison?
  • What do the MVU-Eval Temporal Reasoning and Comparison subsets evaluate, and why were these two chosen as the reference for the correlation analysis?
  • Which concrete applications motivate cross-video reasoning, and which SyncR tasks correspond to them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.