acceptodds
Under review as a conference paper at ICLR 2027

Event-Adaptive Temporal Patching for Token-Budgeted Video Grounding and Chaptering

Abstract

Video multimodal large language models (MLLMs) typically compress a video into a fixed number of visual tokens before temporal reasoning, using uniform sampling and fixed temporal patches. This can underrepresent brief actions and transitions, whereas increasing the sampling rate usually increases downstream processing cost. We introduce Event-Adaptive Temporal Patching (EATP), a query-independent, training-free front end that decouples observation density from the number of encoded visual units. EATP observes a dense candidate stream and uses exact dynamic programming over coarse visual descriptors to partition it into contiguous, variable-length temporal groups under a fixed visual- token budget. The resulting schedule allocates shorter groups to regions with large descriptor changes and longer groups to relatively stable content. A length- aware temporal patch embedding, derived from the pretrained two-frame kernel, encodes each group without fine-tuning. Experiments on QVHighlights moment retrieval and VidChapters chapter generation show training-free improvements over native uniform sampling, supporting the benefit of combining dense observation with content-adaptive temporal grouping rather than dense sampling alone. On QVHighlights, EATP improves mIoU by 2.16 points, strict R@[email protected] by 8.04 points and R@[email protected] by 3.14 points. On VidChapters, it improves F1, tIoU, SODA, CIDEr, and GRACE by 4.5, 1.4, 1.7, 6.5, and 1.0 points, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Video Chaptering, Moment Retrieval, Training-free

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bDEPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is very rough overall, clearly an auto-research work with no visualization, and poor figures.

Critical strengths

None

Critical weaknesses
  1. Omission of highly relevant prior work and insufficient justification for novelty. FlexiVideo (CVPR 2026) has already addressed the visual confusion caused by fixed two-frame encoding by proposing continuous adaptive segmentation and dynamic temporal patch embedding prior to visual encoding, while also offering a training-free version. The current paper fails to cite or compare against this work. Although EATP differs in its fixed-group dynamic programming and specific projection rules, the authors need to clearly articulate the substantive contributions arising from these differences and conduct a comparison against the closest method using matched backbones and computational budgets.

  2. Ambiguity regarding the evaluation set version, raising issues with the comparability of external results. Section 4.1 describes the evaluation set as the original QVHighlights test split; however, the results cited in Table 2 for models like GPT-4o and Gemini actually originate from the re-annotated QVHighlights-TimeLens dataset, which features different query and temporal annotations compared to the original. The authors do not specify which version of the data was used for their local Qwen3.5-9B and EATP models. If the original dataset was used, the external results are not directly comparable; if the TimeLens version was used, the experimental protocol and table labels require correction.

  3. Significant internal inconsistency in core results. For the EATP configuration described as having a 2 FPS / 768-frame budget, Table 2 reports mIoU, R@[email protected], and R@[email protected] values ​​of 66.35 / 70.00 / 62.00, whereas Table 3 reports 65.26 / 67.00 / 60.00 (with Figure 2 also adopting the latter mIoU figure). Consequently, the reported mIoU gain shifts from +2.16 to +1.07, and the R@[email protected] gain shifts from +3.14 to +0.14. The paper offers no explanation for these discrepancies in configuration or evaluation; thus, the primary performance gains claimed in the abstract and main tables cannot currently be verified. 4. The ablation study on maximum group length fails to meet the stated control conditions. The lower section of Table 4 claims to fix the candidate stream while varying only Kmax⁡K_{\max}, yet it includes a native baseline where Kmax⁡=2K_{\max}=2. Under the main experimental setup, a 4 FPS candidate stream comprises approximately 4G frames; however, G groups with a maximum length of 2 can only accommodate 2G frames, making it impossible to cover the full set of candidates. Consequently, that specific row inevitably alters either the number of candidate frames or the output budget, disqualifying it as a controlled experiment isolating the effect of maximum group length. The authors should explicitly state the actual number of candidate frames and output tokens for each row.

  4. Missing alignment rules between variable-length grouping and real-world time hinder reproducibility and result interpretation. While the paper notes that candidate frames possess real-world timestamps, it fails to define the representative time and duration for each variable-length group, or how these map to the model's temporal inputs and positional encodings. Given that different groups cover varying real-world durations, this information is critical for understanding the temporal grounding and chaptering mechanisms. The current description is insufficient to reproduce the temporal alignment process or to determine whether changes in temporal information processing influenced the gains in localization performance.

Questions for authors

N/A

1 comment

Weakness 1 — prior work and novelty. Thank you for identifying FlexiVideo as a close predecessor. Section 2.2 now cites it, including its training-free variant. FlexiVideo forms segments using a visual-change threshold and transforms a shared temporal kernel, whereas EATP uses dynamic programming to assign all observed candidates to exactly the native number of groups under explicit group-size bounds and averages pretrained two-frame responses within each group. These distinctions define our contribution; they do not establish superior accuracy or wider applicability. A same-backbone, matched-budget comparison with FlexiVideo is still needed before claiming a performance advantage over it. Our reported gains are relative to the native and fixed-group controls.

Weakness 2 — which QVHighlights annotations were evaluated. Table 2 identifies our benchmark as QVHighlights-TimeLens. Its local Qwen3.5-9B baseline and EATP use the same TimeLens annotations and evaluation queries, allowing a direct comparison of the effect of EATP. We present the published TimeLens scores as benchmark references; differences in models and training mean they cannot isolate the effect of EATP. We will specify the exact TimeLens annotation version and the query and video counts for each local result, so readers can verify the evaluation set used.

Weakness 3 — results and the Kmax⁡=2K_{\max}=2 control. Thank you for noting the differing scores. Table 2 and the Kmax⁡=16K_{\max}=16 row of Table 4 use the selected setting and report 66.35/70.00/62.00 on QVHighlights. The EATP row at the 2-FPS downstream budget in Table 3 and the Kmax⁡=8K_{\max}=8 row of Table 4 use a different group-length setting and report 65.31/67.00/60.00. Both EATP settings use nominal 4-FPS candidates and retain the native output-unit budget. Figure 2 now shows 66.35 for the Kmax⁡=16K_{\max}=16 setting; we will label it in the figure. The native baseline reports 64.19/66.86/53.96. The Table 4 native-reference case has G output groups. With Kmax⁡=2K_{\max}=2, at most 2G candidate frames fit, so this row cannot simultaneously use the approximately 4G candidates of the 4-FPS run. The comparable DP sweep begins at Kmax⁡=4K_{\max}=4 and uses nominal 4-FPS candidates at the same G; when C=4G, its four-frame groups match Uniform K=4. We will report the actual candidate and output-group counts for each row and reconcile the VidChapters GRACE values of 61.10 (main) and 60.96 (ablation) before submission.

Weakness 4 — group timestamps and positional encoding. Section 3.4 now defines the observed interval of each group using the first and last candidate-frame timestamps and prepends that interval as text to the corresponding visual block. Each group produces one visual temporal unit, with the grid length fixed at G, so a group covering more seconds does not take more visual positions. We will clarify that the reported results retain the model's original visual positional encoding: a group's real duration is stated in its timestamp text, but does not rescale its visual positions. We tried changing the positional encoding for variable-length groups, but performance decreased in our preliminary experiments, so we kept the original encoding. We will explore other ways to represent real time in future work. The revised discussion also notes that the timestamp text may contribute to the localization gains, which therefore cannot be attributed solely to group-boundary placement.

Reviewer_aiUZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes EATP, a training-free method to improve the token efficiency of video inputs to VLMs. Instead of sampling frames uniformly, EATP decodes a denser candidate stream and employs dynamic programming to adaptively allocate compute based on information density. With Qwen3.5-9B, the proposed approach reports gains on both temporal localization and dense captioning.

Critical strengths
  1. The paper targets a practical problem: how to allocate temporal capacity in video LLMs under a fixed token budget. Its approach of adapting temporal patches to content is intuitive.
  2. The method is training-free and compatible with pretrained weights. The experiments compare it against controls that separate the effect of denser observation from that of adaptive grouping.
Critical weaknesses
  1. The generality and superiority of the proposed approach is questionable. All experiments use a single model, Qwen3.5-9B. A component of the module is based on a two-frame temporal conv3d. Therefore, it may not apply to many video LLMs. Furthermore, no equal-budget comparison is performed with other adaptive token compression methods for video processing.
  2. The presentation is misleading in places and unbalanced in detail. The cited scores in Table 2 are from QVHighlights-TimeLens and are of no comparison value to the paper's own results. Table 2 also extends beyond page margin. In Sec. 3, much space is spent on unnecessary minor implementation details.
Questions for authors

See weaknesses.

1 comment

Weakness 1 — generality and adaptive baselines. Thank you for raising the limited evidence for generality and the absence of an equal-budget comparison with other adaptive video-token compression methods. Our experiments use Qwen3.5-9B, whose visual front end forms two-frame temporal patches. The native and fixed-group controls test EATP within this model, but do not establish generality across video language models or superiority over other adaptive methods. We will make this scope clear in the revision and add a same-backbone, matched-budget comparison if it can be completed.

Weakness 2 — external QVHighlights scores. Table 2 reports results on QVHighlights-TimeLens. The published scores provide a useful benchmark reference, while the two local Qwen3.5-9B rows test the effect of EATP using the same model, annotations, and evaluation queries. Relative to this local baseline, EATP improves mIoU by 2.16 points, R@[email protected] by 3.14 points, and R@[email protected] by 8.04 points. Because the published systems differ in their models and training, their scores are not a controlled test of EATP itself. We will specify the exact TimeLens annotation version and the number of queries and videos used for our local comparison.

Reviewer_Rp4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the task of temporal localization and chapter generation in videos under a constrained visual token budget. We propose "Event-Adaptive Temporal Patching" (EATP), a training-free and query-agnostic approach. Our method decouples the observation density of candidate frames from the number of temporal units used for downstream tasks. Specifically, we first obtain a dense set of candidate frames; then, leveraging coarse-grained visual descriptors, we employ dynamic programming to partition these frames into a fixed number of contiguous, variable-length segments, such that the squared reconstruction error of the descriptors within each segment is minimized.

Critical strengths
  1. Decouple dense observation from a fixed encoding budget, and provide a complete implementation approach compatible with pre-trained models.
  2. The experiment includes matched controls relevant to the core hypothesis and demonstrates potential gains across two tasks.
Critical weaknesses
  1. Under the nominally identical budget of 2 FPS / 768 frames, Table 2 reports QVHighlights results for EATP as 66.35 / 70.00 / 62.00, while Table 3 reports 65.26 / 67.00 / 60.00—corresponding to mIoU, R@[email protected], and R@[email protected], respectively. Figure 2 also uses the value 65.26. Meanwhile, the results in Table 4 (with Kmax⁡=8K_{\max}=8) are 65.31 / 67.00 / 60.00, which still do not perfectly align with those in Table 3.
  2. The paper states that Kmax⁡K_{\max} is selected based on performance across various tasks, but it does not specify which data split was used for this selection. Given that the primary results reported are derived from the test set, the authors should clarify whether an independent validation set was employed.
Questions for authors
  1. Please explain the differences in the QVHighlights results presented in Tables 2, 3, and 4, as well as in Figure 2, and provide the complete configuration for each row. How does the row in Table 4 with Kmax⁡=2K_{\max}=2 simultaneously satisfy the constraints of a fixed set of candidate streams, a fixed number of groups, and complete coverage?
  2. On which data partition is Kmax⁡K_{\max} selected for the two tasks? Are the exact same evaluation samples used for each table? Please provide the sample sizes, the complete prompts, the method for parsing the generated results, and the rules for handling multiple valid segments as well as invalid outputs.
  3. Can the author provide evidence that the DP (Dynamic Programming) segmentation boundaries align more closely with task-relevant event boundaries? Furthermore, how does the method perform in cases involving semantic shifts with minimal visual change, versus segments where there is significant visual change but the underlying semantics remain continuous?
  4. The visualization in Figure 1 is poorly executed; while it displays both the native paths and the EATP paths, it fails to clearly indicate the event boundaries, changes in descriptors, or the rationale behind the grouping boundaries selected by the DP. Consequently, it is difficult for the reader to discern which regions require smaller groups, or where the native dual-frame groupings might inadvertently conflate distinct events.
  5. EATP partitions continuous video into groups of varying durations, while the downstream component continues to receive a fixed number of temporal units. Although the paper highlights that candidate frames possess ground-truth timestamps, it does not clearly explain how this temporal information is incorporated into the model following the grouping process.
  6. The authors correctly point out that an identical number of downstream tokens does not imply the same total cost. However, the paper does not report end-to-end latency, throughput, or peak VRAM usage, nor does it provide a breakdown of overheads—such as those for decoding, descriptor computation, DP (data parallelism/projection), and initial-layer projection.
1 comment

For clarity, Kmax⁡K_{\max} means the maximum number of candidate frames assigned to one group; G is the number of visual time units passed onward. Dynamic programming (DP) finds the lowest-cost grouping under these constraints. The three QVHighlights scores below are mIoU (mean overlap between predicted and reference time windows), R@[email protected] and R@[email protected] (percentages of queries whose predicted time windows have an IoU greater than the stated threshold with the annotated windows). GRACE is the chaptering score used for VidChapters; ASR means the video's speech transcript. Weakness 1 / Question 1 — conflicting QVHighlights numbers. Thank you for finding the inconsistent numbers. The native baseline samples the video at 2 FPS, with at most 768 frames. EATP instead samples candidate frames at 4 FPS and groups them so that Qwen3.5-9B receives the same number of visual time units as in the native baseline. Table 2 and the Kmax⁡=16K_{\max}=16 row of Table 4 report our selected setting: 66.35 mIoU, 70.00 R@[email protected], and 62.00 R@[email protected]. The 2-FPS-budget EATP row in Table 3 and the K_{\max}=8 row of Table 4 report a different group-length setting: 65.31/67.00/60.00. Both EATP settings observe nominal 4-FPS candidates and retain the native output-unit budget. We have made the distinct settings clear in the results text and tables. Table 4's K_{\max}=2 row is a native-reference case: G groups of at most two frames cannot cover the approximately 4G candidates observed at 4 FPS, so its candidate stream is capped near the native 2G frames. We have separated this row from the fixed-candidate-count comparison. Weakness 2 / Question 2 — selection split, samples, prompt, and output handling. Section 4.3 now states that we choose K_{\max}=16 for QVHighlights using mIoU on a held-out subset of the training data, and Kmax⁡=8K_{\max}=8 for VidChapters using validation GRACE; it says both choices precede test evaluation. For QVHighlights the validation videos are stated to be disjoint from the TimeLens evaluation videos. We will add the construction and size of each validation set, the scores for the settings considered, and the annotation release, split, query count, and video count for each reported result in the Appendix. We will provide the complete moment-retrieval prompt, decoding and parsing rules, explain how multiple or malformed time windows are handled, and state whether invalid outputs remain in the metric denominator in the Appendix. Question 3 — whether chosen groups follow semantic events. EATP groups frames according to changes in coarse visual features, not annotated event labels. A meaningful event change can be visually subtle, while a camera or lighting change can be visually large without marking a new event. Figure 2 compares the model's chapter-boundary accuracy when sampled frames are near versus far from annotated chapter boundaries; it does not test whether EATP places its group boundaries at those annotations. We will clarify this distinction in the paper and describe the method as adaptive allocation based on visual changes. Question 4 — Figure 1 visualization. We agree that Figure 1 is an architecture overview, not evidence for why a particular boundary was selected. The caption describes the three stages, but the present figure does not show descriptor changes, group lengths, actual timestamps, or a concrete case where native two-frame patches merge distinct events. We will add a labeled example with the candidate frames, the descriptor-change pattern, the native pairs, the selected groups and their time intervals. Question 5 — real time after variable-length grouping. Each candidate has a real presentation timestamp. The ordered candidates are divided into G consecutive groups. In Section 3.4, we explain that each group receives a textual start–end timestamp formed from the actual times of its first and last frames. This interval is shown to one decimal place immediately before the group's visual block. Question 6 — total cost. EATP keeps the pretrained 9B model and the number of visual time units it receives at the native budget, so the main model processes a comparable number of visual inputs. The additional work is confined to the visual front end: decoding denser candidate frames, computing compact 8-by-8-grid descriptors, grouping them, and projecting each group. We expect this preprocessing to be modest relative to inference with the 9B model, but will quantify its actual cost. We will add a matched comparison of overall latency and peak memory, with separate timings for decoding, descriptors, grouping, and visual projection, using the same videos, hardware, batch size, and checkpoint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.