acceptodds
Under review as a conference paper at ICLR 2027

ChronosKV: Dual-Clock Temporal Encoding for Streaming Video LLMs

Abstract

Streaming video LLMs process unbounded inputs by retaining a bounded visual cache and retrieving a sparse subset at question time. We demonstrate that this answer-time path suffers from a twofold information bottleneck: first, internal query-key attention locates the evidence little better than uniform sampling; second, standard position re-indexing preserves local token order while entirely collapsing absolute elapsed time. To resolve both limitations in a training-free framework, we introduce ChronosKV. ChronosKV pairs a lightweight contrastive vision-language retriever, delivering high-coverage evidence at constant query latency, with a dual-clock temporal encoding. By decoupling Rotary Position Embedding (RoPE) frequency bands band-by-band, ChronosKV routes high-frequency components to a compact ordinal track for local order, while steering low-frequency components to an absolute-time track for elapsed intervals. To evaluate what existing benchmarks miss, we introduce Interval-QA, a diagnostic suite isolating event order, recency, and relative metric distance. Across online streaming and long-video suites, ChronosKV attains the highest average accuracy of the compared systems, and restores the long-range elapsed time that previous position assignments discard.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, video large language models, KV cache retrieval, rotary position embedding, temporal encoding, long video question answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_srovOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce ChronosKV, a training-free temporal encoding for streaming video LLMs. For that the embedding is decoupled into order and temporal distance between frames, introducing a dual-clock temporal encoding. The modifications are light-weight because of only query-time position index changes.

The authors introduce a new benchmark including the time distance between events to account for the temporal distance called Interval-QA.

Critical strengths
  • Introducing a light-weight training-free temporal embedding on already existing video LLMs, might have great impact on already existing systems.
Critical weaknesses
  • W1 Insufficient documentation around new Interval-QA Benchmark: While this paper introduces a new temporal encoding it also introduces the benchmark it measures the main uplift in performance as well. The newly introduced one has been annotated with Claude. The protocol for that is insufficiently documented.
  • W2 Only limited improvement on public benchmarks: Table 8 shows only marginal improvements on public benchmarks. This indicates that the dual-clock encoder adds only marginal improvement compared to ordered positional encodings. The main improvement is measured on the newly introduced Interval-QA benchmark which makes a transparent protocol highly important.
Questions for authors

In section 4 a frontier multimodal modal has been mentioned without clarifying which one it was. Which exact Anthropic model has been used? How have the Claude annotations been validated? Can you please provide more transparency around the benchmark creation?

Reviewer_LtdJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ChronosKV, a training-free framework for streaming video LLMs that addresses two limitations of KV-cache-based reasoning: inaccurate historical frame retrieval and the loss of temporal information during position re-indexing. The method uses an external SigLIP retriever to identify relevant frames and introduces a dual-clock RoPE encoding that preserves local token order and global temporal intervals. The authors also introduce Interval-QA to evaluate event ordering, recency, and relative temporal distance. Experiments across several video benchmarks show consistent improvements over existing methods.

Critical strengths
  1. The paper identifies an interesting limitation of existing streaming video LLMs. The dual-clock RoPE design is well motivated and provides a simple way to preserve both local ordering and elapsed time without additional training.

  2. The experiments are fairly comprehensive, covering multiple backbones, benchmarks, and recent baselines. The controlled ablations effectively separate the contributions of retrieval and positional encoding. The efficiency analysis also shows that the method introduces relatively little overhead.

Critical weaknesses
  1. Limited improvement in metric temporal reasoning. Although the dual-clock encoding substantially improves recency prediction, relative-distance accuracy remains close to chance (54.2% on LLaVA-OneVision). The improvement over ordered positions is only 2.2 points. This raises questions about whether the method truly preserves usable metric temporal information or mainly improves coarse recency judgments.

  2. Limited novelty of some components. External vision-language retrieval is a well-established technique, and frequency-dependent RoPE modifications have been explored in previous work. While applying these ideas to streaming KV-cache retrieval is meaningful, the distinction from existing positional encoding approaches could be better established.

Questions for authors
  1. Why does dual-clock encoding provide only marginal improvements on relative-distance questions despite the substantial gains in recency? Can the authors analyze accuracy as a function of the difference between the two candidate temporal intervals?

  2. How does the proposed dual-clock encoding compare with existing frequency-aware RoPE scaling or interpolation methods under the same retrieval setting?

  3. The external retriever searches a growing collection of frame embeddings. How does retrieval latency scale with video duration, especially for substantially longer streams?

Reviewer_QnV7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets streaming video-QA and identifies two issues: internal QK retrieval is barely better than uniform sampling, and existing position assignments either lose order or collapse elapsed time. The authors propose ChronosKV, which pairs an external SigLIP retriever with a training-free dual-clock RoPE assignment. They also introduce Interval-QA to test order, recency, and relative distance. Experiments across streaming/offline benchmarks and two 7B backbones show consistent gains, especially on recency.

Critical strengths

(1) The research problem is clear and practically relevant. The dual-clock idea is simple, plug-and-play, and well motivated by RoPE’s frequency structure. (2) The Interval-QA benchmark fills a genuine gap by separating positional layout from retrieval variance. Ablations are informative, and the efficiency overhead is modest. The paper is generally well written and the training-free claim is well supported.

Critical weaknesses

(1) The performance gains are uneven: recency improves substantially, but relative distance improves only marginally, and standard streaming/offline gains are also small. (2) The evaluation is limited to two 7B backbones with 1D RoPE, so the generality of the method is unclear. Limited evaluation cannot well support the effectiveness of the plug-and-play and training-free claim. (3) Interval-QA is useful but relatively small and annotated by a frontier model. The annotation reliability is not fully established.

Questions for authors

(1) Add statistical significance or confidence intervals for the main comparisons, especially relative distance. (2) Clarify how band thresholds are selected and test sensitivity. (3) Include at least one additional backbone or discuss expected behavior on multi-axis RoPE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.