acceptodds
Under review as a conference paper at ICLR 2027

CrossTimeBench: Benchmarking Cross-Temporal Video Reasoning in Complex Multi-Event Scenarios

Abstract

Recent advances in video reasoning have increasingly focused on improving the temporal understanding capabilities of Multimodal Large Language Models(MLLMs). However, existing benchmarks mainly focus on short clips and single-segment reasoning. While such benchmarks are useful for assessing basic temporal understanding, they are insufficient for evaluating complex multi-event scenarios where relevant evidence is distributed across different parts of a video. In this paper, we introduce the CrossTimeBench, a cross-temporal video reasoning benchmark designed to systematically evaluate model reasoning over complex multi-event videos. Each question in the benchmark requires the integration of information from more than two temporal segments, constructed through both mining and synthesis from existing datasets. The benchmark covers 8 major categories and 35 subcategories, including 2,607 videos and 8,496 QA pairs (with 6,628 unique samples due to multi-label annotations). Among them, 48.4% of open-ended QA pairs and 94.6% of closed-form QA pairs are newly constructed. In addition, we propose a fine-grained evaluation framework based on the benchmark and conduct extensive baseline experiments on a wide range of mainstream open-source and closed-source MLLMs. Our experimental results show that, although current models achieve strong performance on existing benchmarks, they still exhibit substantial limitations in cross-temporal reasoning tasks. Meanwhile, the evaluation results demonstrate noticeable differences across open-ended and closed-form QA settings, as well as among different question types. These observations provide new insights into the limitations of current video understanding models.

Primary area:
datasets and benchmarks
Keywords:
Video Reasoning, Temporal Understanding, Cross-Temporal Reasoning, Multimodal Benchmark, Large Multimodal Models, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_8ZP6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CrossTimeBench for reasoning across temporally separated video events. It combines mined and synthetic questions with a detailed task taxonomy. Models are evaluated using multiple-choice accuracy and open-ended answer metrics.

Critical strengths
  • The benchmark addresses an important capability: connecting evidence across separated video events.

  • The authors release data and evaluation code. This allows concrete checks of the construction and scoring procedures.

Critical weaknesses
  • The inference code displays choices with indices starting at 0. The prompt instead asks for indices starting at 1. The scorer expects indices starting at 0. This creates ambiguity and may affect models differently. The authors should confirm whether this implementation produced the reported results.

  • Missing responses and integer-parsing failures can be excluded from the accuracy denominator. A broad exception handler silently skips errors. Different models may therefore be scored on different subsets. The paper does not report how many predictions were excluded.

  • The validation prompt in Figure 9 examines the question text without the video or supporting evidence. This cannot establish that multiple temporal segments are necessary. Text-only, single-segment, and evidence-removal controls are needed.

  • The released evidence annotations are incomplete. Of 6,628 unique questions, 2,698 lack the range field provided for the synthetic examples. Nine examples also have reference answers absent from their choice lists. The authors should explain the evidence schema and correct these invalid items.

  • Models are evaluated with different frame budgets. Claude receives fewer frames than the stated maximum for other models. Without matched visual coverage or oracle evidence, the comparison mixes evidence access with reasoning ability.

Questions for authors
  • Which exact prompt and scoring versions produced the reported tables?

  • Can you rerun evaluation with consistent option indices and a fixed denominator?

  • How many predictions were missing, unparseable, or otherwise excluded for each model?

  • What evidence shows that the questions require multiple temporal segments?

Reviewer_FzVCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CrossTimeBench, a benchmark designed to evaluate the ability of video models to integrate evidence from temporally non-adjacent video segments. The authors extracted questions from multiple existing datasets and synthesized additional questions based on ActivityNet event annotations to cover question types that are underrepresented in existing data. Ultimately, a multi-label benchmark was constructed, covering 8 major categories and 35 subcategories (though it appears there are actually 34), comprising 2,607 video clips and 8,496 labeled question-answer pairs. The evaluation covered both open-ended and multiple-choice question types and compared seven multimodal models.

Critical strengths

S1. The objective of this study is of significant value. Treating “relevant evidence distributed across multiple clips” as an explicit structural constraint better captures the ability the authors aim to measure than simply extending the video duration. Section 3.1 also clearly distinguishes between temporal localization, cross-clip alignment, and combinatorial reasoning.

S2. The data preparation is thorough, and the diagnostic dimensions are comprehensive. The data mining process, the targeted synthesis of missing question types, and the multi-label design all demonstrate a significant level of effort. Additionally, reporting results for both open-ended and multiple-choice questions allows us to observe the performance of the same model under different output constraints.

Critical weaknesses

W1. The overview of related work is incomplete. The related work section discusses LongVideoBench and KFS-Bench but does not mention OmniVideoBench (ICLR 2026). OmniVideoBench investigates multi-step evidence-based reasoning in videos, provides step-by-step reasoning annotations, and includes temporal reasoning problems that require correlating information from different time points in a video. We recommend that the authors broaden the scope of their review and explicitly discuss and distinguish the contributions of this work from those of OmniVideoBench.

W2. Selection of proprietary models and their representativeness. The abstract claims to have conducted a comprehensive evaluation of mainstream open-source and closed-source MLLMs, but Section 4.1 includes only Claude-Sonnet-4-6 and does not explain why the GPT or Gemini series models were not evaluated. The average accuracy of Claude reported in Table 2 is 0.09, which is far below the random level. This suggests that the issue may lie in the answer format or parsing rather than a genuine gap in capability; while the paper attributes this to a reduction in the number of input frames, I believe this cannot account for such sub-random-level performance.

Questions for authors

Q1. Have the authors considered OmniVideoBench (ICLR 2026) and other similar benchmarks involving multi-step video reasoning? Please discuss this in the “Related Work” section and briefly explain how the cross-temporal tasks in this paper relate to and differ from them.

Q2. Why is Claude-Sonnet-4-6 the only proprietary model included, while the GPT or Gemini series models were not? How do the authors explain the 0.09 accuracy rate on the “choose one out of four” question? This accuracy is even lower than that of random guessing. Could the authors report the syntactic analysis failure rate for each model, as well as the distribution of raw outputs for multiple-choice questions, and include at least one additional model from the GPT or Gemini series?

Q3. The paper mentions 35 labels, but my count shows only 34. Could you please clarify this?

Reviewer_AJUEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrossTimeBench, a benchmark that evaluates the ability of multimodal large language models to integrate information across different video segments and understand complex relationships among multiple events. It also establishes a fine grained evaluation framework. Experiments show that current mainstream models still exhibit substantial limitations in reasoning across temporal segments, with significant performance differences across question answering formats and question types. These findings reveal the limitations of existing video understanding models.

Critical strengths
  1. This work focuses on video understanding across temporal segments and requires models to integrate information from at least two segments. Its research objective is clear and practically meaningful.
  2. The task taxonomy is detailed, covering 8 major categories and 35 subcategories, which facilitates analysis of model performance across different cognitive tasks.
  3. The evaluation includes both open ended and closed ended question answering. These complementary formats provide multiple perspectives on model capabilities.
Critical weaknesses
  1. The benchmark evaluates a very limited number of black box large models. As Table 2 shows, Claude Sonnet 4.6 is the only proprietary black box model in the evaluation. Please include more proprietary black box models, such as models from the GPT and Gemini families.
  2. Claude Sonnet 4.6 performs substantially worse than even small open source models with 7B parameters. This result is counterintuitive. Please evaluate additional versions of Claude and further explain why Claude Sonnet 4.6 performs substantially worse than 7B models.
  3. Please highlight the differences between CrossTimeBench and existing benchmarks more clearly. Please provide a table that compares CrossTimeBench with the latest related video reasoning benchmarks from the past two years across multiple dimensions.
  4. Data quality requires more detailed reporting. Please report whether the data undergoes human verification, how many samples human verification filters out, and the level of agreement among expert annotators. If data annotation relies too heavily on large models, data quality is difficult to guarantee.
Questions for authors

See above.

Reviewer_6hojOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrossTimeBench, a benchmark designed to evaluate the cross-temporal, multi-event reasoning capabilities of multimodal models through question-answering tasks that require synthesizing evidence from multiple discontinuous video segments. Experiments reveal that existing models still exhibit significant limitations in connecting scattered events and integrating temporal evidence.

Critical strengths
  1. The paper investigates whether a model can combine evidence from multiple, non-contiguous temporal segments within a video to answer questions.
  2. CrossTimeBench establishes integrating evidence from multiple discontinuous time segments as a design requirement for all its problems.
Critical weaknesses
  1. The core distinction of CrossTimeBench lies in its requirement that each question incorporate evidence from multiple, non-contiguous time segments. However, the annotation of multiple evidence intervals does not necessarily imply that every one of those intervals is essential for answering the question.

  2. Open-source models support up to 64 frames of input, while closed-source models are limited to 32 frames. This performance gap may stem, at least in part, from insufficient coverage of evidence rather than a difference in reasoning capability; therefore, controlled experiments with a standardized number of frames are required.

Questions for authors
  1. Can the authors verify that answering each question requires multiple evidence segments, rather than a single segment

  2. Do the performance differences persist when open- and closed-source models receive the same number of frames

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.