- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The authors propose a new video benchmark,
Kairos, comprising of long-videos (10-30 mins) automatically annotated with fine-grained temporal annotations (actions, entities, interactions, memory of contextual cues). It consists of ~19K videos (5.4K hours) sourced from YouTube/Bilibili. - A subset of the dataset is used to create video QA benchmark,
Kairos-Benchwith 820 videos and ~3K QA pairs. First, an evidence of varying temporal granularity is sampled (e.g., reference frame, or multiple-shots). Then, this is passed to Gemini 2.5 Flash to generate a QA pair with distractors and reasoning. Questions that can be answered solely from text are removed and a manual check is conducted. - Several open/closed models are evaluated on Kairos-Bench with Gemini-3.1-Pro being the best. The authors also train SFT baselines with Qwen2.5-7B on the train split of Kairos and show that is boosts performance.
The annotation process is roughly as described below.
1. Each video is first split into shots. For each shot, frames are sampled at 1 FPS.
2. The first frame of each shot serves as the reference frame which is captioned in detail and entities are detected used to create an Entity Bank per video. The subsequent frames are treated as differential frames - only what change occurs w.r.t. to reference frames is annotated. Speech and non-speech sounds are also separately annotated.
3. Each shot boundary is also annotated to record the kind of cut it entails.
The benchmark questions are dissected into 17 different capabilities (e.g., temporal ordering) across 4 axes (e.g., temporal and causal reasoning).
Critical strengths
- The size of the benchmark is substantial (19K long videos). Most existing benchmarks optimise for clean but small evaluation sets. It is reasonable that a smaller subset is used to construct a benchmark (820 videos).
- The text-only baseline is a much needed experiment and shows that
Kairos-benchdoes not suffer from text-only bias compared to other long video benchmarks. - The authors show that SFT on this benchmark (train set) improves performance not just on Kairos but also on other long video benchmarks. This indicates that the benchmark does have useful samples and could be included in post-training datasets for general video models.
Critical weaknesses
-
Deeper analysis of results needed. While Table 2 shows results for many open/closed models - it feels more like a dump of all numbers, I did not learn anything/new surprising about the models from this benchmark. For example, are certain models better at long term temporal reasoning tasks? Likewise, are there scaling patterns for similar model families which could tell if newer versions of a model are going to solve this benchmark or not. Moreover, qualitative analysis of failure patterns can tell something unique about the models (assuming the benchmark does test for unique aspects) - this is not present in the paper.
-
Presentation. The section describing benchmark constructions is quite difficult to understand. Also, Fig 2 does not provide a clear picture of the step-by-step annotation pipeline. All this makes it hard to parse the strengths/uniqueness of this benchmark in a flurry of video benchmarks that already exist. Similarly, the qualitative samples shown in Fig 5 are unreadable and do not add anything to a deeper understanding of the benchmark.
-
Lack of simpler baselines. While the authors evaluate multiple models on the benchmark, including some simpler baselines can establish confidence in the hardness of the benchmark.
- Instead of providing (video, question) directly to a model, a dense captioning model can be used to generate a detailed text summary of the video and then passed along with the question. How does this "captioning only" baseline do? This will tell whether the benchmark actually tests, e.g., for long term memory, or for multi-shot reasoning?
- Agentic baselines: if let an agentic approach [2], e.g., to select frames/shots before passing onto the MLLM, how well does it do?
- How time-sensitive is the benchmark? For example, similar to [1], if you shuffle the frames, or change order of the shots, how does that affect performance?
[1] Seeing the Arrow of Time in Large Multimodal Models. Zihui Xue, Mi Luo, Kristen Grauman. NeurIPS 2025.
[2] Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory. Chang Nie, Jiaju Wei, Junlan Feng, Chaoyou Fu, Caifeng Shan. ArXiv 2026.
Questions for authors
Much of my questions are stated in the Weaknesses. Some other minor questions are listed:
- How was the taxonomy to curate videos selected? Were some topics pre-defined? How were the search queries formulated?
- It is not clear how the audio processing part of a single-shot annotations contributes? Can you please clarify?
8 comments
Q2. Fig.2 and Fig.5 do not show the step-by-step pipeline or clear qualitative examples, and the uniqueness of the benchmark is unclear.
Thank you for these helpful suggestions. In the revised version, we will reorganize Fig.2 and Fig.5 and refine the descriptions of the benchmark construction process. Specifically, in Fig.2, we will illustrate the step-by-step processing of a single video, highlighting the annotations generated at each stage. For Fig.5, we will replace the original qualitative examples with a step-by-step illustration tracing a representative question-answer-reasoning triple through the entire benchmark construction process, providing a clearer understanding of how the benchmark is constructed. Meanwhile, given the limited space in the main paper, we will move the original qualitative examples to the appendix and expand them into multiple more detailed and readable figures, offering a more comprehensive view of the dataset and benchmark characteristics.
We illustrate the construction procedure step by step.
- Shots: TransNetV2 splits the video into shots (median 91 shots), and frames are taken at 1 fps (Appendix B.1).
- Audio: faster-whisper transcribes the speech sentence by sentence and Qwen2-Audio summarizes the environmental sound of every 30s. Each sentence is attached to the first frame inside its span, and each sound summary to the reference frame of its window.
- Reference frame (the first frame of each shot): Qwen3-VL-8B writes a full description and lists typed entities, given the previous 20 shot descriptions, the entity bank and the unused audio (Appendix B.3).
- Entity bank: entities are linked across shots by their normalized names (Appendix B.4).
- Differential frames: every later frame of the shot is described as its change from the previous frame, from both images. Every 300 frames, the chain restarts from a new reference frame.
- Transitions: one record per cut (technique, narrative purpose, visual contrast).
- Questions: a sampler draws an evidence window, which fixes the tier and the admissible capabilities. Gemini-2.5-Flash writes the question, the answer, three distractors from the same video and the rationale in one call. A question is removed if at least three of five text-only models answer it. Human reviewers check the rest against the video, and a question is kept only if every review verdict is a pass.
Kairos-Bench is the only benchmark here written from automatic annotation, and the least answerable without the video.
- Automatic and scalable: the other benchmarks are written by hand or by an LLM from human annotations, while Kairos-Bench is written by an LLM from our automatic 1 fps annotation, and humans only review the result.
- The corpus behind it: among annotated video corpora, only Kairos describes every second of 10–30-minute videos, with an entity bank, a record per cut and aligned audio, while others caption short clips (LLaVA-Video-178K[1], ShareGPT4Video[2]), a shot or scene (Vript[3], FineVideo[4]), a moment every few seconds (Ego4D[5]) or a few events per video (ActivityNet Captions[6]).
- Least answerable without the video: the lowest text-only lift (+9.6, against +12.4 to +35.6), with the second-longest questions.
- Tied to its evidence: each question records its evidence timestamps, which fix its tier (T1–T5) and the capability it can test.
- Verified and useful: the annotation is audited (Table 10), and training data from the same pipeline on other videos improve three external benchmarks (Table 4).
| Benchmark | Videos | Video length | Questions written by | Evidence time per question | Text-only lift |
|---|---|---|---|---|---|
| EgoSchema | > 5,000 clips | 3 min | LLM from human narrations | no | +33.6 |
| Video-MME | 900 | 11 s – 1 h | humans | no | +29.8 |
| LongVideoBench | 3,763 | 8 s – 1 h | humans | referred moment | +31.0 |
| LVBench | 103 | 68 min (mean) | humans | time reference | +12.4 |
| HourVideo | 500 | 20–120 min | LLM from narrations, refined by humans | no | +19.0 |
| CG-Bench | 1,219 | 10–80 min | humans | clue interval | +35.6 |
| Video-Holmes | 270 | 1–5 min | LLM from human annotations | no | +30.9 |
| Kairos-Bench | 820 | 10–30 min | LLM from our automatic 1 fps annotation | timestamps | +9.6 |
We will add this discussion and the comparison table to the revised version.
[1] LLaVA-Video: Video Instruction Tuning with Synthetic Data. Zhang Y, Wu J, Li W, et al. TMLR 2025.
[2] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. Chen L, Wei X, Li J, et al. NeurIPS 2024.
[3] Vript: A Video Is Worth Thousands of Words. Yang D, Huang S, Lu C, et al. NeurIPS 2024.
[4] FineVideo. Farré M, Marafioti A, Tunstall L, et al. Hugging Face 2024.
[5] Ego4D: Around the World in 3,000 Hours of Egocentric Video. Grauman K, et al. CVPR 2022.
[6] Dense-Captioning Events in Videos. Krishna R, et al. ICCV 2017.
Q3. Simpler baselines and agentic methods.
Captioning-only baseline. To test whether the benchmark measures long-term memory and multi-shot reasoning, Qwen3-VL-8B answers every question from the question alone, or with one of two texts: a detailed summary of the whole video written by a captioning model (Qwen3-VL-8B from 32 frames), or the Kairos annotation (one timestamped description per shot), each without and with the video.
| Input | Overall | A (moment) | B (within shot) | C (across shots) | D (whole video) | T5 (more than 15 min) |
|---|---|---|---|---|---|---|
| question | 36.6 | 34.5 | 36.8 | 39.4 | 35.1 | 40.5 |
| question + summary | 43.9 | 40.3 | 48.5 | 43.6 | 40.4 | 45.2 |
| question + Kairos | 56.8 | 62.9 | 54.2 | 55.8 | 45.3 | 55.2 |
| video + question | 42.7 | 42.3 | 42.6 | 41.4 | 49.8 | 47.9 |
| video + question + summary | 43.6 | 41.8 | 47.4 | 40.1 | 47.1 | 44.4 |
| video + question + Kairos | 56.5 | 64.2 | 52.0 | 56.4 | 44.0 | 52.5 |
- A summary does not answer the questions. It adds 7.3 points to the question alone and only 0.9 to the video.
- What answers them is a timestamped memory of every moment. The Kairos annotation keeps every shot with its time. It adds 20.2 points to the question alone (13.8 to the video) and beats the summary by 12.2 points across shots (C) and by 10.0 on questions whose evidence spans more than 15 minutes (T5). This is the long-term memory and cross-shot reasoning the benchmark tests: the answer depends on details in specific moments far apart in the video, which a summary loses.
Agentic methods. We compare frame selection and agentic systems against two references: uniform frames and the frames at the annotated evidence (an oracle). Three training-free selectors [7–9] choose the 32 frames for a frozen InternVL3.5-8B. Three published agentic systems [10–12] run with their own code and each is compared with its own backbone.
| Method | Gain Accuracy (%) |
|---|---|
| InternVL3.5-8B, 32 frames | |
| frames at the annotated evidence (oracle) | +9.6 |
| InternVL3.5-8B, training-free frame selection | |
| AKS[5] | +2.1 |
| CLIP top-32[6] | +2.2 |
| ITS[7] | +2.9 |
| agentic systems | |
| Light-Omni[8] (backbone Qwen2.5-Omni-7B) | +1.9 |
| OmniAgent[9] (backbone Qwen2.5-Omni-7B) | +3.9 |
| VideoMind[10] (backbone Qwen2-VL-7B) | +0.9 |
The room for choosing frames is large (frames at the evidence: +9.6), but training-free selectors gain only 2.1–2.9 points and the agents 0.9–3.9. Finding the evidence in a long video is a key bottleneck, which is what a long-term-memory benchmark should measure.
[7] Adaptive Keyframe Sampling for Long Video Understanding. Tang X, Qiu J, Xie L, et al. CVPR 2025.
[8] Learning Transferable Visual Models From Natural Language Supervision. Radford A, Kim J W, Hallacy C, et al. ICML 2021.
[9] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. Liu S, Zhao C, Xu T, et al. CVPR 2025.
[10] Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory. Nie C, Wei J, Feng J, et al. IEEE TMM 2026.
[11] Native Active Perception as Reasoning for Omni-Modal Understanding. Xing Z, Xu R, Wang Y, et al. ICML 2026.
[12] VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning. Liu Y, Lin K Q, Chen C W, et al. ICLR 2026.
Thank you for the detailed and constructive review. We appreciate that you value the scale of Kairos, the text-only baseline and the transfer of our fine-tuning to other long-video benchmarks. Your suggestions helped us make the evidence clearer, and we answer your questions below.
Q1. Deeper analysis of the results: are some models better at long-range reasoning, are there scaling patterns within model families, and what do failure patterns reveal?
Thank you for this question. We also find these results informative and interesting. The paper reports accuracy per capability (A-D) (Table 2) and per temporal tier (T1-T5) for OpenQA (Table 3), but space did not allow a deeper analysis, so we will add the following analysis to the revised version.
A–D are the capability levels (A: single frame, B: within a shot, C: across shots, D: whole video), and T5 holds the questions whose evidence spans more than 15 minutes.
These results support the central point of the paper: long videos must be understood at the level of specific seconds, and Kairos makes this measurable. We will add this analysis in the revised version.