- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces CrossTimeBench for reasoning across temporally separated video events. It combines mined and synthetic questions with a detailed task taxonomy. Models are evaluated using multiple-choice accuracy and open-ended answer metrics.
Critical strengths
-
The benchmark addresses an important capability: connecting evidence across separated video events.
-
The authors release data and evaluation code. This allows concrete checks of the construction and scoring procedures.
Critical weaknesses
-
The inference code displays choices with indices starting at 0. The prompt instead asks for indices starting at 1. The scorer expects indices starting at 0. This creates ambiguity and may affect models differently. The authors should confirm whether this implementation produced the reported results.
-
Missing responses and integer-parsing failures can be excluded from the accuracy denominator. A broad exception handler silently skips errors. Different models may therefore be scored on different subsets. The paper does not report how many predictions were excluded.
-
The validation prompt in Figure 9 examines the question text without the video or supporting evidence. This cannot establish that multiple temporal segments are necessary. Text-only, single-segment, and evidence-removal controls are needed.
-
The released evidence annotations are incomplete. Of 6,628 unique questions, 2,698 lack the range field provided for the synthetic examples. Nine examples also have reference answers absent from their choice lists. The authors should explain the evidence schema and correct these invalid items.
-
Models are evaluated with different frame budgets. Claude receives fewer frames than the stated maximum for other models. Without matched visual coverage or oracle evidence, the comparison mixes evidence access with reasoning ability.
Questions for authors
-
Which exact prompt and scoring versions produced the reported tables?
-
Can you rerun evaluation with consistent option indices and a fixed denominator?
-
How many predictions were missing, unparseable, or otherwise excluded for each model?
-
What evidence shows that the questions require multiple temporal segments?