acceptodds
Under review as a conference paper at ICLR 2027

Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics

Abstract

Many emerging video language modeling tasks require systems to move beyond clip-level abstraction and model visual content as it unfolds over extended time horizons. However, most existing video datasets rely on coarse or sparsely aligned supervision, which compresses temporal variation and limits the ability of models to learn reusable representations of continuous visual dynamics. We introduce Kairos, a video dataset for video-language modeling with time-resolved annotations. Kairos consists of long-duration videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal alignment. The annotations capture ongoing actions, entity appearances and attributes, interactions, and evolving contextual cues along the video timeline. This time-resolved structure supports fine-grained evaluation, long-range modeling and reasoning, instruction data construction, representation learning, and video generation. Kairos provides a general-purpose foundation for modeling visual experiences over time.

Primary area:
datasets and benchmarks
Keywords:
Video-Language Understanding, Multi-Modal Language Model, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UEcQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The authors propose a new video benchmark, Kairos, comprising of long-videos (10-30 mins) automatically annotated with fine-grained temporal annotations (actions, entities, interactions, memory of contextual cues). It consists of ~19K videos (5.4K hours) sourced from YouTube/Bilibili.
  • A subset of the dataset is used to create video QA benchmark, Kairos-Bench with 820 videos and ~3K QA pairs. First, an evidence of varying temporal granularity is sampled (e.g., reference frame, or multiple-shots). Then, this is passed to Gemini 2.5 Flash to generate a QA pair with distractors and reasoning. Questions that can be answered solely from text are removed and a manual check is conducted.
  • Several open/closed models are evaluated on Kairos-Bench with Gemini-3.1-Pro being the best. The authors also train SFT baselines with Qwen2.5-7B on the train split of Kairos and show that is boosts performance.

The annotation process is roughly as described below.

1. Each video is first split into shots. For each shot, frames are sampled at 1 FPS.
2. The first frame of each shot serves as the reference frame which is captioned in detail and entities are detected used to create an Entity Bank per video. The subsequent frames are treated as differential frames - only what change occurs w.r.t. to reference frames is annotated. Speech and non-speech sounds are also separately annotated.
3. Each shot boundary is also annotated to record the kind of cut it entails.

The benchmark questions are dissected into 17 different capabilities (e.g., temporal ordering) across 4 axes (e.g., temporal and causal reasoning).

Critical strengths
  • The size of the benchmark is substantial (19K long videos). Most existing benchmarks optimise for clean but small evaluation sets. It is reasonable that a smaller subset is used to construct a benchmark (820 videos).
  • The text-only baseline is a much needed experiment and shows that Kairos-bench does not suffer from text-only bias compared to other long video benchmarks.
  • The authors show that SFT on this benchmark (train set) improves performance not just on Kairos but also on other long video benchmarks. This indicates that the benchmark does have useful samples and could be included in post-training datasets for general video models.
Critical weaknesses
  • Deeper analysis of results needed. While Table 2 shows results for many open/closed models - it feels more like a dump of all numbers, I did not learn anything/new surprising about the models from this benchmark. For example, are certain models better at long term temporal reasoning tasks? Likewise, are there scaling patterns for similar model families which could tell if newer versions of a model are going to solve this benchmark or not. Moreover, qualitative analysis of failure patterns can tell something unique about the models (assuming the benchmark does test for unique aspects) - this is not present in the paper.

  • Presentation. The section describing benchmark constructions is quite difficult to understand. Also, Fig 2 does not provide a clear picture of the step-by-step annotation pipeline. All this makes it hard to parse the strengths/uniqueness of this benchmark in a flurry of video benchmarks that already exist. Similarly, the qualitative samples shown in Fig 5 are unreadable and do not add anything to a deeper understanding of the benchmark.

  • Lack of simpler baselines. While the authors evaluate multiple models on the benchmark, including some simpler baselines can establish confidence in the hardness of the benchmark.

    • Instead of providing (video, question) directly to a model, a dense captioning model can be used to generate a detailed text summary of the video and then passed along with the question. How does this "captioning only" baseline do? This will tell whether the benchmark actually tests, e.g., for long term memory, or for multi-shot reasoning?
    • Agentic baselines: if let an agentic approach [2], e.g., to select frames/shots before passing onto the MLLM, how well does it do?
    • How time-sensitive is the benchmark? For example, similar to [1], if you shuffle the frames, or change order of the shots, how does that affect performance?

[1] Seeing the Arrow of Time in Large Multimodal Models. Zihui Xue, Mi Luo, Kristen Grauman. NeurIPS 2025.

[2] Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory. Chang Nie, Jiaju Wei, Junlan Feng, Chaoyou Fu, Caifeng Shan. ArXiv 2026.

Questions for authors

Much of my questions are stated in the Weaknesses. Some other minor questions are listed:

  • How was the taxonomy to curate videos selected? Were some topics pre-defined? How were the search queries formulated?
  • It is not clear how the audio processing part of a single-shot annotations contributes? Can you please clarify?
8 comments

Thank you for the detailed and constructive review. We appreciate that you value the scale of Kairos, the text-only baseline and the transfer of our fine-tuning to other long-video benchmarks. Your suggestions helped us make the evidence clearer, and we answer your questions below.

Q1. Deeper analysis of the results: are some models better at long-range reasoning, are there scaling patterns within model families, and what do failure patterns reveal?

Thank you for this question. We also find these results informative and interesting. The paper reports accuracy per capability (A-D) (Table 2) and per temporal tier (T1-T5) for OpenQA (Table 3), but space did not allow a deeper analysis, so we will add the following analysis to the revised version.

A–D are the capability levels (A: single frame, B: within a shot, C: across shots, D: whole video), and T5 holds the questions whose evidence spans more than 15 minutes.

ModelOverallABCDT5
Gemini-3.1-Pro58.163.751.559.259.158.7
GPT-5.556.756.958.554.756.859.4
Gemini-2.5-Flash53.458.553.348.549.350.2
InternVL3-78B51.748.058.049.149.354.0
InternVL3.5-38B47.045.449.344.851.647.5
GLM-4.5V46.545.153.540.442.742.1
Nova-2-Lite46.446.749.145.436.937.5
Qwen3-VL-30B-A3B45.543.650.741.843.544.4
Qwen3-VL-8B42.342.044.539.444.044.0
InternVL3-8B41.642.242.739.441.839.8
InternVL3.5-8B40.739.942.739.738.739.4
GPT-4o38.241.434.836.744.941.3
GPT-4o-mini31.530.130.633.334.736.7
  • Models that read the whole video lead, because the questions ask about specific moments. The two Gemini models, which receive the whole video, rank first and third. Given 32 frames instead, Gemini-2.5-Flash falls from 53.4 to 40.8. For our fine-grained benchmark, what matters is seeing the right seconds (discussed in Q3 and Q4).
  • Newer and larger models are better, and the benchmark is far from solved. A larger model of the same family gains 3.2–10.1 points (Qwen3-VL 8B → 30B-A3B, InternVL3.5 8B → 38B, InternVL3 8B → 78B), and a newer generation gains 4.7 (Gemini-2.5-Flash → Gemini-3.1-Pro) to 18.5 (GPT-4o → GPT-5.5). The best model reaches 58.1% and the best open model 51.7%, which leaves ample room for future models.
  • Long-range accuracy grows with model strength. The best models at T5 are among the best overall (GPT-5.5 59.4, Gemini-3.1-Pro 58.7, InternVL3-78B 54.0), and a larger model of the same family scores higher at T5 (InternVL3 +14.2, InternVL3.5 +8.1, Qwen3-VL +0.4). The input separates the models: the Gemini models read the video at 1 fps and catch the one second a T1 question asks about, so they lead on single moments (A: 63.7, 58.5) and score lower at T5 than at T1 (−4.9, −8.3).
  • Failure patterns. Of all questions, 2.3% are solved by none (0.24% expected by chance). The hardest 16.8% (solved by at most three models) concentrate on audio, transitions, counting and ordering. On 58% of the unsolved questions, more than half of the models pick the same wrong option, which differs from the answer in one detail that sparse frames cannot resolve.
  • Kairos-Bench measures the cost of missing the evidence. Every question records its evidence timestamps, so accuracy splits by evidence span, and the cost of missing the evidence can be measured directly: a frozen InternVL3.5-8B gains 9.6 points when its 32 frames are taken at the evidence instead of uniformly (see Q3).

These results support the central point of the paper: long videos must be understood at the level of specific seconds, and Kairos makes this measurable. We will add this analysis in the revised version.

Q2. Fig.2 and Fig.5 do not show the step-by-step pipeline or clear qualitative examples, and the uniqueness of the benchmark is unclear.

Thank you for these helpful suggestions. In the revised version, we will reorganize Fig.2 and Fig.5 and refine the descriptions of the benchmark construction process. Specifically, in Fig.2, we will illustrate the step-by-step processing of a single video, highlighting the annotations generated at each stage. For Fig.5, we will replace the original qualitative examples with a step-by-step illustration tracing a representative question-answer-reasoning triple through the entire benchmark construction process, providing a clearer understanding of how the benchmark is constructed. Meanwhile, given the limited space in the main paper, we will move the original qualitative examples to the appendix and expand them into multiple more detailed and readable figures, offering a more comprehensive view of the dataset and benchmark characteristics.

We illustrate the construction procedure step by step.

  1. Shots: TransNetV2 splits the video into shots (median 91 shots), and frames are taken at 1 fps (Appendix B.1).
  2. Audio: faster-whisper transcribes the speech sentence by sentence and Qwen2-Audio summarizes the environmental sound of every 30s. Each sentence is attached to the first frame inside its span, and each sound summary to the reference frame of its window.
  3. Reference frame (the first frame of each shot): Qwen3-VL-8B writes a full description and lists typed entities, given the previous 20 shot descriptions, the entity bank and the unused audio (Appendix B.3).
  4. Entity bank: entities are linked across shots by their normalized names (Appendix B.4).
  5. Differential frames: every later frame of the shot is described as its change from the previous frame, from both images. Every 300 frames, the chain restarts from a new reference frame.
  6. Transitions: one record per cut (technique, narrative purpose, visual contrast).
  7. Questions: a sampler draws an evidence window, which fixes the tier and the admissible capabilities. Gemini-2.5-Flash writes the question, the answer, three distractors from the same video and the rationale in one call. A question is removed if at least three of five text-only models answer it. Human reviewers check the rest against the video, and a question is kept only if every review verdict is a pass.

Kairos-Bench is the only benchmark here written from automatic annotation, and the least answerable without the video.

  • Automatic and scalable: the other benchmarks are written by hand or by an LLM from human annotations, while Kairos-Bench is written by an LLM from our automatic 1 fps annotation, and humans only review the result.
  • The corpus behind it: among annotated video corpora, only Kairos describes every second of 10–30-minute videos, with an entity bank, a record per cut and aligned audio, while others caption short clips (LLaVA-Video-178K[1], ShareGPT4Video[2]), a shot or scene (Vript[3], FineVideo[4]), a moment every few seconds (Ego4D[5]) or a few events per video (ActivityNet Captions[6]).
  • Least answerable without the video: the lowest text-only lift (+9.6, against +12.4 to +35.6), with the second-longest questions.
  • Tied to its evidence: each question records its evidence timestamps, which fix its tier (T1–T5) and the capability it can test.
  • Verified and useful: the annotation is audited (Table 10), and training data from the same pipeline on other videos improve three external benchmarks (Table 4).
BenchmarkVideosVideo lengthQuestions written byEvidence time per questionText-only lift
EgoSchema> 5,000 clips3 minLLM from human narrationsno+33.6
Video-MME90011 s – 1 hhumansno+29.8
LongVideoBench3,7638 s – 1 hhumansreferred moment+31.0
LVBench10368 min (mean)humanstime reference+12.4
HourVideo50020–120 minLLM from narrations, refined by humansno+19.0
CG-Bench1,21910–80 minhumansclue interval+35.6
Video-Holmes2701–5 minLLM from human annotationsno+30.9
Kairos-Bench82010–30 minLLM from our automatic 1 fps annotationtimestamps+9.6

We will add this discussion and the comparison table to the revised version.

[1] LLaVA-Video: Video Instruction Tuning with Synthetic Data. Zhang Y, Wu J, Li W, et al. TMLR 2025.

[2] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. Chen L, Wei X, Li J, et al. NeurIPS 2024.

[3] Vript: A Video Is Worth Thousands of Words. Yang D, Huang S, Lu C, et al. NeurIPS 2024.

[4] FineVideo. Farré M, Marafioti A, Tunstall L, et al. Hugging Face 2024.

[5] Ego4D: Around the World in 3,000 Hours of Egocentric Video. Grauman K, et al. CVPR 2022.

[6] Dense-Captioning Events in Videos. Krishna R, et al. ICCV 2017.

Q3. Simpler baselines and agentic methods.

Captioning-only baseline. To test whether the benchmark measures long-term memory and multi-shot reasoning, Qwen3-VL-8B answers every question from the question alone, or with one of two texts: a detailed summary of the whole video written by a captioning model (Qwen3-VL-8B from 32 frames), or the Kairos annotation (one timestamped description per shot), each without and with the video.

InputOverallA (moment)B (within shot)C (across shots)D (whole video)T5 (more than 15 min)
question36.634.536.839.435.140.5
question + summary43.940.348.543.640.445.2
question + Kairos56.862.954.255.845.355.2
video + question42.742.342.641.449.847.9
video + question + summary43.641.847.440.147.144.4
video + question + Kairos56.564.252.056.444.052.5
  1. A summary does not answer the questions. It adds 7.3 points to the question alone and only 0.9 to the video.
  2. What answers them is a timestamped memory of every moment. The Kairos annotation keeps every shot with its time. It adds 20.2 points to the question alone (13.8 to the video) and beats the summary by 12.2 points across shots (C) and by 10.0 on questions whose evidence spans more than 15 minutes (T5). This is the long-term memory and cross-shot reasoning the benchmark tests: the answer depends on details in specific moments far apart in the video, which a summary loses.

Agentic methods. We compare frame selection and agentic systems against two references: uniform frames and the frames at the annotated evidence (an oracle). Three training-free selectors [7–9] choose the 32 frames for a frozen InternVL3.5-8B. Three published agentic systems [10–12] run with their own code and each is compared with its own backbone.

MethodGain Accuracy (%)
InternVL3.5-8B, 32 frames
frames at the annotated evidence (oracle)+9.6
InternVL3.5-8B, training-free frame selection
AKS[5]+2.1
CLIP top-32[6]+2.2
ITS[7]+2.9
agentic systems
Light-Omni[8] (backbone Qwen2.5-Omni-7B)+1.9
OmniAgent[9] (backbone Qwen2.5-Omni-7B)+3.9
VideoMind[10] (backbone Qwen2-VL-7B)+0.9

The room for choosing frames is large (frames at the evidence: +9.6), but training-free selectors gain only 2.1–2.9 points and the agents 0.9–3.9. Finding the evidence in a long video is a key bottleneck, which is what a long-term-memory benchmark should measure.

[7] Adaptive Keyframe Sampling for Long Video Understanding. Tang X, Qiu J, Xie L, et al. CVPR 2025.

[8] Learning Transferable Visual Models From Natural Language Supervision. Radford A, Kim J W, Hallacy C, et al. ICML 2021.

[9] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. Liu S, Zhao C, Xu T, et al. CVPR 2025.

[10] Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory. Nie C, Wei J, Feng J, et al. IEEE TMM 2026.

[11] Native Active Perception as Reasoning for Omni-Modal Understanding. Xing Z, Xu R, Wang Y, et al. ICML 2026.

[12] VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning. Liu Y, Lin K Q, Chen C W, et al. ICLR 2026.

Reviewer_nSaiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a large-scale dense video caption dataset, together with a benchmark for evaluating the performance of SOTA MLLMs. The dataset is annotated automatically using a pipeline, and the benchmark is filtered and polished by human.

Critical strengths
  1. The motivation of the paper is clear, as the high-quality dense video caption dataset is important for both video understanding and video generation tasks.

  2. The designed pipeline is well motivated, and the benchmark is reviewed by human.

Critical weaknesses
  1. The pipeline is standard and may have some potential limitations:

(1) The long video is split into shots, and the shots are annotated independently. This may introduce the issue of temporal hacking; that is, some shots may be used to provide more evidence, improve video quality, or convey metaphors. The key themes of the whole video may not be well aligned with the shots.

(2) Following this pipeline, the structure of the annotated video captions and the granularity of events may be homogeneous.

(3) There is a lack of quantitative analysis of the annotated dataset.

  1. To my understanding, after SFT on the proposed benchmark, the model may be able to capture richer information. However, the 16-f model performs even better than the 64-f model, which seems strange.
Questions for authors

See weakness.

3 comments

Thank you for the review, and for recognizing that high-quality dense video captions matter for both video understanding and video generation, and that our pipeline is well motivated. Your concerns point to properties of the annotation that the paper did not make explicit enough, and we answer them below.

Q1. The shots are annotated independently, which may cause temporal hacking, i.e. some shots may be used to provide more evidence, improve video quality, or convey metaphors. The key themes of the video may not align with the shots.

Each shot is described in the context of the earlier shots, not independently. We respectfully clarify that shots are not annotated independently. When describing a shot, the annotation model (Qwen3-VL-8B) receives the descriptions of the previous 20 shots, the video's entity bank (whose names it must reuse) and the audio, which runs across cuts (section 2.1, Appendix B.3–B.4). Without the entity bank, the entities per video double (+109%) and the share of entities seen in more than one shot halves (−51%) (Appendix I). Every cut also gets a transition record with the editing technique, the visual contrast and the narrative purpose (section 2.1, Appendix B.3). This records the role a shot plays. The theme of the whole video is carried into each shot by the 20-shot narrative context, and Kairos-Bench tests it with whole-video questions (Capability level D, e.g., narrative summarization).

We understand temporal hacking as defined in [1]: "a phenomenon where models shortcut by fixating on select frames, missing the full video narrative". If you meant a different kind of temporal hacking, please let us know and we will gladly discuss it.

We agree that this is a key risk for long-video benchmarks, and Kairos-Bench was already designed against it (section 3). Its distractors come from other moments of the same video, its questions name the moment in words rather than by a timestamp, and text-only models and human reviewers remove the questions that do not need the video.

To test whether Kairos-Bench can be answered without the right frames or without their order, we evaluate five models on Kairos-Bench using frames sampled uniformly across the video or around the evidence relevant to each question. We test three frame orders: chronological, shuffled, and reversed. We further consider shot-level shuffling, where the order of shots is randomized while the frame order within each shot is preserved. We also evaluate a setting that uses only frames sampled from the first and last 10% of each video.

32 framesQwen3-VL-8BInternVL3-8BInternVL3.5-38BGLM-4.5VGemini-2.5-Flash
spread over the video, chronological42.743.447.747.542.1
spread over the video, shuffled41.541.746.746.039.7
spread over the video, reversed41.041.446.346.340.1
only the first and last 10% of the video41.841.446.746.140.7
around the evidence, chronological62.458.663.066.362.2
around the evidence, shuffled61.057.262.664.159.6
around the evidence, reversed58.354.760.461.455.7
around the evidence, shots shuffled61.756.862.564.959.7
  • Order matters where the question is about order. Around the evidence, reversing costs 10.3–15.3 points on the 520 questions about order (action sequencing, state tracking, temporal ordering, narrative summarization) and 0.9–4.6 on the others. Spread over the video, frames are about 19-56s apart, so a short event appears in at most one of them and reordering costs only 1.0–2.4 points.
  • Shot order matters across cuts. Shuffling only the shots barely changes ordering within a shot (action sequencing, n = 201: −3.5 to +3.0 points) but costs ordering across shots (temporal ordering, n = 230: −6.1 to −8.7) and narrative summarization (n = 40: −12.5 to −30.0).
  • The frames must show the evidence. With the same number of frames, accuracy is 15.2–20.1 points higher around the evidence than spread over the video. Frames from only the first and last 10% of the video score 3.6–5.3 points below spread frames on the 2,050 questions whose evidence lies in between. A model that fixates on a few misleading frames, as in temporal hacking, would score lower on Kairos-Bench.

[1] Unhackable Temporal Rewarding for Scalable Video MLLMs. Yu E, Lin K, Zhao L, et al. ICLR 2025.

Q2. Following this pipeline, the structure of the annotated video captions and the granularity of events may be homogeneous.

The pipeline gives every video the same structure, but each video fills it with its own content. The structure is what videos share in general: shots, the seconds within a shot, cuts, recurring entities, speech and sound. The content differs from video to video, because each record describes its own frame, with its own entities, speech and sounds.

The uniform structure is intentional, it makes every statement of the annotation temporally grounded. We sample one frame per second and give every frame one time-stamped record, and inside a shot a record states only what changes from the previous second.

To test whether the shared structure makes the text repetitive, we compare Kairos with the same pipeline describing every frame on its own, on the 200 audit videos (Appendix I):

MeasureKairos (references + differentials)every frame described on its own
word trigrams shared with the previous second's record (same shot)32.4%52.8%
near-duplicates of the previous second's record (≥ 80% shared)12.8%21.6%
records identical to an earlier record of the same video15.7%13.3%
records opening with one of the stream's three most common three-word openings17.4%19.2%

Describing changes cuts the overlap with the previous second from 52.8 to 32.4% and the near-duplicates from 21.6 to 12.8%. Identical records are slightly more frequent (15.7 vs 13.3%) because a short change record, such as "Nothing has changed.", recurs in static footage, while describing each frame on its own repeats whole descriptions. This is what the differential design is for: each record spends its words on what is new, while every second keeps its own record.

Events are described at several granularities, not one. A change within one second is one differential, and a longer action is the chain of differentials that follows it within the shot. A shot is described by its reference, a cut by its transition record, and each entity is tracked across the whole video. Accordingly, the benchmark questions use evidence spans from a single moment (T1) to more than 15 minutes (T5).

Q3. There is a lack of quantitative analysis of the annotated dataset.

Thank you for this comment. The paper reports quantitative analysis of the dataset at several levels, which we summarize here. Corpus statistics are in section 2.2, Figure 3 and Appendix C (Table 6, Figures 6–10): taxonomy, video durations, shots per video, shot durations, and scenarios. Benchmark statistics are in section 3.3, Figure 4, Table 7 and Figures 13–18, and annotation quality in Appendices G–J (Tables 10–13). The sanity checks of the benchmark are quantified too: the text-only audit removes 19,920 of 26,363 generated questions, and Kairos-Bench has the lowest text-only lift of the ten benchmarks in Table 1. We add the content measures in Q2 above. We would be grateful if you could specify any further analysis you would like to see.

Q4. The 16-f model performs even better than the 64-f model, which seems strange.

Thank you for the question. The 16-, 32- and 64-frame results in Table 4 come from one fine-tuned model. We trained it once, at 32 frames, and changed only the number of test frames (Table 4: "At evaluation stage, we ablate the frame budget over {16, 32, 64}"), to see whether it uses more frames than it was trained with. On Kairos-Bench it scores lower with 64 frames than with 32, likely because the extra frames add redundant tokens, a longer context and a frame spacing unseen in training. On the three external benchmarks, by contrast, 64 frames score above 16 (e.g., LVBench 42.15 vs 39.18). Our results show that the choice of frames matters more: with the same 32 frames, five models gain 15.2–20.1 points when the frames are taken around the evidence (Q1 above). Models thus need to learn to find the few relevant seconds, and per-second annotation such as Kairos can supervise this, which makes this work timely and important. We hope Kairos-Bench draws the community's attention to how poorly long-video models retrieve the frames a question needs.

Dear Reviewer,

We would like to kindly follow up on our previous response. We would greatly appreciate it if you could take a moment to review our clarifications and let us know whether they address your concerns.

As there are less than 24 hours remaining in the discussion period, we would be especially grateful if you could share any feedback before the deadline.

If you have any remaining questions or suggestions, we would be more than happy to discuss them further.

Thank you very much for your time and consideration!

Reviewer_B9tROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces KAIROS, a large-scale video-language dataset containing 19,004 long-form videos (5,420 hours) with fine-grained annotations at 1 FPS. The annotation pipeline captures spatial details, temporal changes, entity tracking, and audio information. The authors also introduce KAIROS-Bench, with 3,011 human-verified questions, and demonstrate its effectiveness through extensive model evaluations and fine-tuning experiments.

Critical strengths
  • Large-scale, fine-grained annotations. KAIROS provides richer temporal supervision than conventional video captioning datasets. Its automated annotation pipeline is scalable and supports detailed spatiotemporal understanding.

  • Comprehensive evaluation and practical utility. The benchmark includes human verification and leakage filtering. Fine-tuning experiments show improvements on multiple external benchmarks, demonstrating the dataset's usefulness.

Critical weaknesses
  • Concerns about annotation quality. A substantial number of automatically generated benchmark questions are rejected due to incorrect answers, raising concerns about annotation reliability. The quality of the much larger automatically generated training dataset is not sufficiently validated.

  • Insufficient ablation studies. The paper does not clearly isolate the benefits of fine-grained temporal annotations from those of additional training data. Comparisons with simpler annotation strategies, such as standard video captions or reference-frame-only descriptions, are missing.

  • Potential error accumulation in the annotation pipeline. The proposed pipeline generates differential frame descriptions conditioned on preceding observations and maintains entity identities across shots through an automatically constructed entity bank. Errors in earlier descriptions or entity identification may therefore propagate through subsequent annotations. The paper lacks sufficient analysis of such error accumulation and its impact on the reliability of fine-grained temporal supervision.

Questions for authors
  • How reliable are the automatically generated training annotations, and how do annotation errors affect downstream performance?

  • Can the authors provide ablations comparing full KAIROS annotations against simpler alternatives, such as conventional video captions or reference-frame-only annotations?

  • Since fine-tuning uses only 32 sampled frames per video, how effectively does it leverage the dense 1 FPS annotations?

  • Have the authors evaluated how annotation errors accumulate over time, particularly in differential frame descriptions and cross-shot entity tracking? How does annotation accuracy vary with shot duration or the number of preceding differential descriptions?

4 comments

Thank you for your time, accurate summary of our contributions, and thoughtful questions. Your questions on annotation quality, ablations and error accumulation helped us make the evidence clearer. We answer them below.

Q1. How reliable are the automatically generated training annotations?

The annotations are of high quality, as our human audit shows. To measure the training annotations directly, human evaluators audited a stratified random sample of the whole corpus (Appendix G: 200 videos, over 50 hours, all 12 domains). Description precision is 96.6%, the hallucination rate is 2.2%, and entity links reach 93.1% precision and 97.6% recall (Table 10). This quality comes from the pipeline's design (section 2.1): each description is written from its own frames and previous annotation, entities keep their names from the entity bank, and every shot starts from a new reference description.

Most of the wrong answers found in the benchmark review arise when the questions are written, not in the annotation. The review rejects 2,860 of the 6,443 audited questions for a wrong answer (Appendix J, Table 13). A reviewer rejects a question if its answer is wrong or any of its three distractors is right, so a question can fail even when every description it was written from is correct. Questions written from a single description are rejected for a wrong answer 16% of the time, far above the 3.4% of descriptions with an error in the audit (precision 96.6%, Appendix G, Table 10). Whole-video questions, which combine 10–12 descriptions, are rejected 70% of the time. The error rate thus grows with how much information the question writer has to combine, which is why every Kairos-Bench question is verified by humans. The training annotation itself is measured by the audit above.

Q2. Ablation of the full annotation against simpler ones (e.g., video caption, references only).

Thank you for this valuable suggestion. To separate the benefit of the fine-grained annotation from that of more training data, we build three training sets of the same size, 5,000 question–answer items each. Gemini-2.5-Flash writes all of them about Kairos videos outside Kairos-Bench, and only the annotation it reads differs:

  • Kairos: per-shot references, per-second differentials and transition records.
  • ref-only: one description per shot, as in Vript[1] and FineVideo[2]. Lacking per-second and transition records, it yields no items for 5 of the 17 capabilities (e.g., change detection).
  • vid-caption: one dense caption per video, written by Qwen3-VL-8B from 32 frames, as in ShareGPT4Video[3] and LLaVA-Video-178K[4]. It has no shot-level or second-level records.

Each set is used to train Qwen2.5-VL-7B with the paper's recipe. We report accuracy per temporal tier (T1: moment-level, T2: within 60 seconds, T3: within 300 seconds, T4: within 15 minutes, T5: beyond 15 minutes).

Training corpusKairos-BenchT1T2T3T4T5
N/A42.940.345.738.642.046.0
vid-caption47.546.348.943.948.547.9
ref-only49.948.352.547.448.350.2
Kairos52.649.457.447.849.953.3

With the same number of training frames, the Kairos annotation achieves the highest accuracy. It outperforms vid-caption and ref-only by 5.1 and 2.7 points, respectively, and consistently leads across all tiers.

The gap is largest at T2 (1–60 s), where the per-second records matter. T2 questions mostly ask how a shot unfolds, and the simpler annotations yield far fewer T2 items (vid-caption 806, ref-only 235, Kairos 1,611). Kairos leads there by 8.5 and 4.9 points. At T5 (>15 min), vid-caption yields only 26 items compared with 695 for Kairos, highlighting the difficulty of capturing long-range temporal structure with coarse annotations.

[1] Vript: A Video Is Worth Thousands of Words. Yang D, Huang S, Lu C, et al. NeurIPS 2024.

[2] FineVideo. Farré M, Marafioti A, Tunstall L, et al. Hugging Face 2024.

[3] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. Chen L, Wei X, Li J, et al. NeurIPS 2024.

[4] LLaVA-Video: Video Instruction Tuning with Synthetic Data. Zhang Y, Wu J, Li W, et al. TMLR 2025.

Q3. Do annotation errors accumulate over time, in the differentials and in cross-shot entity tracking? How do annotation errors affect downstream performance? How does accuracy vary with shot length and with the number of preceding differentials?

An error in a reference description does not accumulate along its differentials. To test whether an error in the reference spreads along the chain, we take one shot with at least 20 differentials from each of 200 sampled videos and regenerate its first 20 differentials three times: from the shot's own reference (original), from the reference of another video (swap), and from its own reference with 1–3 visual facts made false (perturb). A judge (Gemini-2.5-Flash) checks each claim against the differential's two frames. The table gives the percentage of differentials with a contradicted claim, which would grow with position if errors spread. None of the column grows with position. Thus, accuracy remains largely stable as the number of preceding differentials increases, with no evidence of cumulative error propagation even after 20 differentials.

Positionoriginalswapperturb
13.0%4.1%9.6%
2–51.4%6.5%5.6%
6–103.9%3.5%5.8%
11–150.2%4.4%4.9%
16–203.5%2.6%5.5%

Errors do not accumulate in cross-shot entity tracking either. Each mention is matched to the video's entity bank by its normalized name, and a match only adds an appearance: the entry's name and description stay as first registered, so a wrong link remains a single mistake rather than a drift (93.1% precision, 97.6% recall, Table 10).

Annotation errors lower downstream accuracy on every benchmark. To measure how annotation errors affect downstream performance, we changed one visual fact in 965 of the 5,000 answers of the Kairos training set of Q2 (19.3%, far above the 2.2% hallucination rate of the annotation audit) and retrained with everything else unchanged. This perturbation reduces accuracy on both Kairos-Bench and external benchmarks.

Training dataKairos-BenchLongVideoBenchLVBenchVideo-MME
N/A42.961.141.860.0
original52.662.142.262.0
perturbed50.961.640.661.4

Description accuracy remains stable across shot lengths. To test whether long shots are described less accurately, we verify all claims in 600 descriptions drawn from shots of different durations. Precision is 95.2% for shots of 0–5s, 95.6% for 5–20 s, and 96.4% for 20–300 s. This is expected: each differential describes only the change between two consecutive frames, not the whole shot, so its accuracy hardly depends on the shot's length.

Q4. Fine-tuning uses only 32 sampled frames per video: how well does it use the dense 1 fps annotation?

The dense annotation enters through the training targets. Each training item is written from the annotation of its evidence window, the per-shot and per-second records it covers, so its question, answer and reasoning state what happens at specific seconds. The Q2 above isolates this effect: with the same 32 uniform frames, the Kairos annotation can train a model 5.1 points better on Kairos-Bench than vid-caption items, and 8.5 points better at T2 (in-shot questions).

For fine-tuning, we adopt uniform frame sampling to isolate the benefits of our fine-grained annotations from those of specialized frame selection strategies. This allows us to assess whether our annotations can consistently enhance video understanding across different models, independent of evidence-aware frame selection.

With the same number of frames, five models gain 15.2–20.1 points when the frames show the evidence. To measure how much the choice of frames matters, we evaluate five models on Kairos-Bench using 32 frames, sampled either uniformly across the video or around the evidence relevant to each question.

32 framesQwen3-VL-8BInternVL3-8BInternVL3.5-38BGLM-4.5VGemini-2.5-Flash
spread over the video42.743.447.747.542.1
around the evidence62.458.663.066.362.2

Finding those frames remains an open problem. To test whether existing methods find them in long videos, we compare frame selection and agentic systems against two references: uniform frames and the frames at the annotated evidence (an oracle). Three training-free selectors [5–7] choose the 32 frames for a frozen InternVL3.5-8B. Three published agentic systems [8–10] run with their own code and each is compared with its own backbone.

MethodGain Accuracy (%)
InternVL3.5-8B, 32 frames
frames at the annotated evidence (oracle)+9.6
InternVL3.5-8B, training-free frame selection
AKS[5]+2.1
CLIP top-32[6]+2.2
ITS[7]+2.9
agentic systems
Light-Omni[8] (backbone Qwen2.5-Omni-7B)+1.9
OmniAgent[9] (backbone Qwen2.5-Omni-7B)+3.9
VideoMind[10] (backbone Qwen2-VL-7B)+0.9

The oracle leaves a 9.6-point margin, but the selectors gain only 2.1–2.9 points and the agents 0.9–3.9. The selectors score frames by their CLIP similarity to the question, and their frames come within 2s of the evidence for only 49–50% of the questions (uniform frames: 27%). Kairos-Bench questions name a moment by what happens in it rather than by a timestamp (section 3.4), and the appearance of a single frame matches such a description only in part. The agents fail at the same step: VideoMind's predicted span covers the evidence for 45.6% of the questions, and it answers 46.0% of those correctly against 39.6% of the rest.

Locating the evidence is thus a key bottleneck of long-video understanding. Kairos makes it measurable, because every question records its evidence time, and its 1 fps annotation can supervise models that learn to find the evidence. We hope Kairos-Bench draws the community's attention to how poorly long-video models retrieve the frames a question needs.

[5] Adaptive Keyframe Sampling for Long Video Understanding. Tang X, Qiu J, Xie L, et al. CVPR 2025.

[6] Learning Transferable Visual Models From Natural Language Supervision. Radford A, Kim J W, Hallacy C, et al. ICML 2021.

[7] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. Liu S, Zhao C, Xu T, et al. CVPR 2025.

[8] Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory. Nie C, Wei J, Feng J, et al. IEEE TMM 2026.

[9] Native Active Perception as Reasoning for Omni-Modal Understanding. Xing Z, Xu R, Wang Y, et al. ICML 2026.

[10] VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning. Liu Y, Lin K Q, Chen C W, et al. ICLR 2026.

Dear Reviewer,

We would like to kindly follow up on our previous response. We would greatly appreciate it if you could take a moment to review our clarifications and let us know whether they address your concerns.

As there are less than 24 hours remaining in the discussion period, we would be especially grateful if you could share any feedback before the deadline.

If you have any remaining questions or suggestions, we would be more than happy to discuss them further.

Thank you very much for your time and consideration!

Comments on the paper

3 comments

Dear reviewers,

Thank you for your reviews and constructive feedback.

The authors have submitted their rebuttal. Please take a moment to review it and consider whether your concerns have been adequately addressed. It would be helpful if you could briefly indicate:

  • Whether the rebuttal addresses your main concerns;

  • Whether any significant issues remain unresolved;

  • Whether your overall recommendation has changed.

If appropriate, please also update your review and recommendation in the system.

Thank you for your time and valuable contributions.

Best regards,

AC

AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear reviewers,

We have uploaded a revised version of the paper. All changes are marked in blue, and the main text uses the additional page allowed for the revision. The revised manuscript incorporates the changes we committed to in our responses to the reviewers.

  1. Figure 2 now shows the annotation pipeline step by step, with the record that each step produces, and Figure 5 follows one question through the construction of the benchmark.
  2. Section 3.4 adds an analysis of the results. Appendix N (Table 17) gives the accuracy by capability level.
  3. Section 3.4 adds a comparison with existing benchmarks. Appendix O (Table 18) gives the comparison table.
  4. A time-sensitivity paragraph in Section 3.4 state that only 520 questions (17.3%) concern event order. Appendix P (Tables 19–21) reports the frame-order and evidence-position results separately for these questions, with an example.

We would greatly appreciate it if you could take a look at the revised manuscript and let us know whether our revisions have addressed your concerns.

Thank you again for your time and constructive feedback.

Area_Chair_X2KNOpenReview ↗

AC update

Dear authors,

Thank you for your time and contributions to the discussion.

I understand that you may be concerned if some reviewers have not provided follow-up comments or explicitly acknowledged how their concerns have been addressed in the rebuttal and revised manuscript. Please be assured that the AC will carefully consider all available information, including the original reviews, author rebuttal, revised manuscript, and discussion comments, as part of the final assessment.

The absence of further reviewer comments does not mean that the authors’ responses and revisions have been overlooked. I will carefully examine the issues raised, assess the authors’ clarifications and revisions, and take the discussion into account when forming my recommendation. I will also consult with the reviewers as appropriate to ensure that the relevant issues are fully considered.

Thank you again for your constructive engagement and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.