acceptodds
Under review as a conference paper at ICLR 2027

HTSC-AT: HIERARCHICAL TEMPORAL SEMANTIC COMPRESSION WITH ADAPTIVE TOKENIZATION FOR EFFICIENT LONG VIDEO UNDERSTANDING

Abstract

Long video understanding with multimodal large language models (MLLMs) is bottlenecked by quadratic attention complexity and severe temporal redundancy in visual tokens. We present HTSC-AT, a three-stage framework for efficient long-context reasoning on lightweight open-source MLLMs. Stage 1 introduces adaptive visual tokenization driven by motion-semantic change intensity, reducing token count by 66.3% while improving QA accuracy by +5.3%. Stage 2 performs event-aware temporal compression via dual-branch boundary detection and hierarchical intra-event clustering, achieving up to 720× compression while preserving semantically critical evidence. Stage 3 employs a sparse-dense hybrid decoder that maintains dense intra-event attention and structured sparse interevent attention, reducing peak GPU memory from 28.7 GB to 10.4 GB. Across MLVU, VideoMME, LongVideoBench, and LVBench, HTSC-AT achieves stateof-the-art results among open-source small-scale models—outperforming VideoXL2-8B by 4.1 points on MLVU Test—while supporting over 10,000 frames on a single-node 8×H800 GPU setup.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long Video Understanding, Token Compression, Sparse Attention, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_nQQDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HTSC-AT for efficient long-video understanding with lightweight MLLMs. The framework has three stages: adaptive visual tokenization based on motion-semantic change, event-aware temporal compression, and sparse-dense attention over the compressed event representation. The paper evaluates the proposed method on several long-video benchmarks and shows promising results.

Critical strengths
  1. The paper considers token reduction at several levels of the long video pipeline. The overall design is coherent: adaptive visual tokenization, event-level compression, and structured sparse attention are meant to address different sources of redundancy.

  2. The experiments are extensive. The paper evaluates on multiple long-video benchmarks and includes separate analyses for token allocation, temporal compression, and sparse attention, together with runtime and memory measurements

Critical weaknesses
  1. In Sec. 2.2.2, the paper determines how many tokens NtN_t to keep for each frame, but does not explain how these NtN_t tokens are actually selected or constructed. In Sec. 2.3.1, the ActionFormer branch is described as operating on 30-token windows over T(1)T^{(1)} (T(1)T^{(1)} contains a variable number of spatial tokens per frame). It is therefore unclear whether ActionFormer predicts boundaries at the spatial-token level or on some frame-level representation. The semantic discontinuity branch also computes similarity between adjacent tokens in $T^{(1)}$. Two different regions within the same frame could produce a large mutation score despite there being no temporal boundary. The paper should clarify the representation used by both branches.

  2. Several important design claims are not supported by the ablations. There is no CLIP-only and optical-flow-only comparison (Eq. 2) for the claimed dual-stream design, no isolation of the two TEBD boundary branches, and no ablation of the proposed TPE. The sparse-connectivity comparison is also not degree-matched: the random baseline samples three events per query event, whereas the proposed design uses six anchors. Why not compare against six randomly sampled events?

Questions for authors
  1. Since efficiency is a key claim, I would also like to see TTFT and end-to-end latency compared with existing long-video MLLMs under the same hardware and input budget (e.g., a cap of 1024 frames), rather than only against the uniform-token baseline. The three-stage compression pipeline seems likely to add some preprocessing overhead.
  2. After NtN_t is computed in Eq. (3), how are the actual $N_t$ spatial tokens obtained from the original CLIP token grid?
  3. How are overlapping ActionFormer predictions combined for window size 30 and stride 10?
  4. Can the authors provide the missing component-level ablations to demonstrate the necessarity of each part?
  5. Can the authors add a degree-matched random baseline, e.g. random six anchors with the same adjacent-event connections?
  6. Minor formatting issue: "Results." on line 368 is isolated on a separate line.
Reviewer_J2bEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HTSC-AT is a compression method for long video understanding. The method goal is to compress the information using an adaptive tokenization strategy to compress the data which allows to significantly reduce the memory footprint of video processing. This method rely on CLIP embeddings that help find most relevant semantic changes in a given video in order to discard the less relevant still information while ensuring the number of token per frame does not exceed a threshold. A second stage try to summarize the selected tokens into a toen summary through hierarchical clustering. A third stage implement an attention mechanism to select the most relevant tokens.

On the experiment side, the authors used only a frozen Qwen 2.5-BL backbone. The three stage method is built on top of the frozen methods. The author claim that their additional module used less than 1M additional parameters to train. They evaluate their method on MLVU, VideoMME, LongVideoBench, LVBench, VideoEval-Pro, CharadesSTA, V-Star.

Critical strengths

On the strength side, the authors perform a good ablation over the different phases and strategies. They show that their method is competitive on some benchmarks with open-source large language models.They also analyze the performance efficiency with respect to an uniform baseline. The appendix material contains additional experiments and details about the method and lists the hyper parameters used.

Critical weaknesses

On the weaker side, the paper fails to compare against training-free methods such as AKS based methods. Could have been interesting to see how they compare from a performance and speed point of view. Also would have appreciated to see comparison with other methods such as LongVu or VideoTree. A main weakness of this paper is that it seems to have been tailored for the frozen Qwen 2.5-BL backbone, so it’s difficult to know whether this method has the potential to generalize or not (and how much work might be needed to adapt it to another backbone). Similarly, the use of a private training dataset will not make the experiments reproducible, so it might be hard to evaluate the correctness of the experimental setup. Also, there isn’t any ablation over the training data corpora. There should be at least one analysis over the public corpora, the private one and the joint one to see how data impact performances. It also seems that claiming the state of the art on open source small scale model is a bit of a long strech, this method is indeed better on some benchmarks (but there is no quantification of the noise, so hard to say if it’s statistically significant or not), and it’s worse on other benchmarks. So it’s competitive, might be more efficient, but not state of the art (unless the authors can make an ablation across several seeds and show that their results are consistent).

Lastly the biggest flag is that the authors claim that they did not use AI for this work or the manuscript but a lot of citations seems to have been hallucinated (Like for Qwen2.5-vl technical report, LongVideoBench and a lot more). So, AI was used directly or indirectly in this work, meaning that the authors either decided to not report the use of AI as required or they just did not verify the source of the bibliography data.

Questions for authors
  • Do you have any idea on how the method can generalize on other frozen backbone or other training dataset?
  • Did you try with another model than CLIP for stage 1?
Reviewer_wAYzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HTSC-AT, a three-stage framework for efficient long video understanding with a frozen Qwen2.5-VL-7B backbone. It combines adaptive visual token allocation based on motion and semantic changes, event-aware hierarchical temporal compression, and hybrid sparse and dense attention. Experiments report competitive performance on long video benchmarks, a 66.3% reduction in visual tokens, and lower inference time and GPU memory usage.

Critical strengths
  1. The paper presents a coherent three-stage approach to reducing redundancy and computational costs in long video understanding. It combines adaptive visual token allocation based on motion and semantic changes, hierarchical temporal compression guided by events, and hybrid sparse and dense attention informed by event structure. These components address complementary aspects of representation and computational efficiency.
  2. The framework is modular and operates with a frozen VLM backbone. The reported reductions in token count, inference time, and GPU memory suggest practical potential for reducing the resource requirements of long video understanding with existing VLMs.
Critical weaknesses
  1. The training protocols for the ablations are insufficiently specified, making architectural contributions difficult to isolate. The proposed modules are trained on approximately 420 GB of proprietary video data alongside public data. However, it is unclear whether the configurations in Tables 2–5 are trained separately with matched data, training budgets, and initialization, or evaluated by replacing modules or connectivity patterns in a shared checkpoint. The former requires controlled training conditions, while the latter may introduce a mismatch between training and inference. Please clarify these protocols and provide cumulative ablations under consistent settings to distinguish the contributions of the three modules from the benefits of additional training. Dependence on proprietary training data also limits reproducibility.
  2. Evaluation inputs and efficiency measurement procedures are insufficiently documented. The paper does not specify the input frame count, sampling strategy, resolution, and scope of end-to-end timing for each benchmark. These choices directly affect both accuracy and computational costs, particularly for a method focused on efficient long video understanding. Please report accuracy, actual input token counts, end-to-end latency, and peak GPU memory as video duration and input frame count increase under a consistent evaluation protocol. Timing should include the overhead of optical flow estimation, event detection, and clustering.
  3. The method relies on numerous manually chosen thresholds and scene-dependent settings without sufficient robustness analysis. These include event boundary, semantic saliency, and duplicate detection thresholds, as well as clustering and anchor selection rules. Some parameters are adjusted by video domain. The experiments do not adequately establish sensitivity to these choices or the tuning required for unseen domains. Please provide sensitivity analyses for key parameters and compare a shared parameter configuration against domain-specific configurations.
Questions for authors

Figure 1 and the description following Eq. 17 suggest that hybrid sparse and dense attention is an additional module between the vision encoder and the frozen LLM. Please clarify its placement and whether it modifies attention within the LLM. If it neither reduces the number of tokens entering the LLM nor changes the LLM’s internal attention, its computational savings appear limited to the added module itself. Why is this additional token mixing needed before the LLM?

Reviewer_7nxEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HTSC-AT proposes a trainable video compression frontend for a frozen Qwen2.5-VL-7B backbone. It combines motion-semantic adaptive token allocation, event-boundary detection and hierarchical intra-event clustering, and sparse-dense attention over event summaries. Training uses QA, captioning, and temporal consistency objectives. The paper reports improved long-video understanding results and reductions in token count, attention cost, and memory. The compressed representations, event partition, and sparse connectivity are independent of the downstream question.

Critical strengths
  1. The framework addresses redundancy at complementary frame, event, and cross-event scales. Event-coherent compression is a reasonable way to avoid fragmenting related evidence across fixed windows.

  2. The paper evaluates several long-video benchmarks and reports ablations of token allocation, segmentation, and attention connectivity. The inclusion of captioning supervision and reported generation and grounding metrics reflects an effort to assess broader information preservation, although the evaluation protocols require clarification.

Critical weaknesses
  1. Architectural integration and trainability are insufficiently specified. It is unclear whether the original Qwen vision encoder and visual merger are used, whether CLIP supplies routing features or final representations, and where the adapter enters the frozen model. Discrete allocation, thresholding, clustering, and top-k operations also require an explicit account of gradient handling and trainable parameters.

  2. The main table compares complete video models, while ablations largely use simple alternatives such as uniform allocation, fixed windows, and single-level clustering. Strong token selection/merging methods and trainable compression frontends under a shared backbone and matched budgets are needed to isolate the proposed design from additional training. Evaluation and efficiency accounting remain unclear. The paper should reconcile stage-wise token counts and the denominator of the 720× compression claim, specify subsets and annotations used for QA, LCUS, ROUGE-L and grounding, and explain the different ablation scores. Timing should explicitly account for visual encoding, optical flow, event detection, clustering, and LLM inference, including how the eight GPUs are used.

Questions for authors
  1. Please provide the complete visual feature path, the encoder checkpoint identities, the adapter insertion point, and a module-level list of trainable parameters. How are gradients handled through discrete compression decisions?

  2. How does the frontend compare with strong selection/merging and trainable resampling alternatives under the same frozen LLM, visual inputs, output token budget, and, where applicable, training data and objectives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.