acceptodds
Withdrawn from ICLR 2027

DB-Video: Detail-Preserving Dual-Branch Compression for Long Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Large Language Models (VideoLLMs) face significant challenges in long video understanding due to the expensive computational cost of processing massive visual tokens. Token compression is hence desired to preserve the fundamental video semantics while simultaneously retaining the fine-grained details most relevant to the query. In this paper, we propose DB-Video, which introduces the novel detail-preserving dual-branch compression to achieve this objective. Our approach employs a dual-branch mechanism, i.e., a base semantic branch to maintain global context and a query-guided residual branch to capture query-specific high-resolution details. To preserve such fine-grained details, we further propose the reconstruction loss to enforce visual fidelity in the compressed latent space. Benefited by the proposed dual-branch compressor and reconstruction loss, DB-Video achieves a token compression ratio 17x while preserving 98% of the original performance. Experimental results demonstrate that DB-Video outperforms other training-based compression methods and generalizes well across three model architectures, offering a more efficient strategy for long video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LKheOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes DB-Video, a dual-branch compression method for long video understanding. A low-resolution branch preserves basic video semantics, while a high-resolution branch extracts and compresses query-relevant details. Training combines a QA loss with a feature reconstruction loss, using gradient isolation to limit conflicts between the objectives. Experiments show that DB-Video reduces visual tokens by 17× while retaining approximately 98% of the base model’s average performance across four long-video benchmarks.

Critical strengths
  1. The paper addresses a practical problem. It substantially reduces visual tokens while retaining performance close to the original model, with reductions in computation and latency.

  2. The design is intuitive. The low-resolution branch preserves basic semantics, while the high-resolution branch adds query-relevant details. The reconstruction loss and gradient isolation are also well motivated.

  3. The experiments cover multiple long-video benchmarks, fine-grained tasks, and three models, with ablations of the main components.

Critical weaknesses
  1. Incomplete baseline comparisons. The backbones and most baselines are relatively dated and may not fully reflect recent progress. Comparisons with relevant methods such as XComp and EPIC, both cited in the paper, are missing. Most training-based results are taken from prior work without matching backbones, training data, or compression budgets, making it difficult to isolate the contribution of the proposed method.

  2. The distinction from prior two-stream designs is unclear. SlowFast-LLaVA [1] already uses two pathways to balance spatial detail and temporal coverage, while Slow-Fast Architecture for Video MLLMs [2] combines a compact overview with instruction-guided access to detailed visual features. Although DB-Video differs in its implementation, these closely related approaches are neither discussed nor compared against, leaving the added value of its dual-branch design and training strategy insufficiently established.

  3. Unclear benefit–cost trade-off of the dual-branch design. In Table 5, the base branch alone achieves higher overall accuracy on Video-MME. Although the dual-branch design improves long-video performance, accuracy and computational cost should be compared under the full frame setting to justify the high-resolution branch.

  4. Limited evidence for preserving query-relevant details. Query guidance and reconstruction loss provide relatively small gains, with no analysis of their stability. Corresponding component ablations on fine-grained tasks are also missing, making it difficult to attribute the gains specifically to better preservation of query-relevant details.

  5. Limited task coverage. Evaluation focuses mainly on multiple-choice video QA, without direct evaluation on tasks such as captioning or explicit temporal grounding. Whether the compression preserves broader video understanding capabilities remains unclear.

[1] Mingze Xu et al. SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models. arXiv:2407.15841, 2024.

[2] Min Shi et al. Slow-Fast Architecture for Video Multi-Modal Large Language Models. arXiv:2504.01328, 2025.

Questions for authors

The method is evaluated across different backbones, but only at the 7B scale. Would it maintain a similar accuracy–compression trade-off on smaller or larger models? This would help clarify how its effectiveness depends on model capacity.

Reviewer_M1EgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DB-Video, a training-based video-token compression method for long-video understanding. It combines two inputs: a base semantic branch using sparse, low-resolution frames, and a query-guided residual branch using denser, higher-resolution frames. The high-resolution branch first interacts with the text query through cross-attention and is then compressed with 3D convolutions and self-attention. Its output is added to the low-resolution branch representation before the LLM.

The paper further proposes a latent reconstruction loss that reconstructs the high-resolution visual features from the compressed representation. It uses gradient isolation so that the reconstruction loss does not update the cross-attention layer, while the VQA loss updates that layer. The method is trained on a 25K subset of LLaVA-Video multiple-choice data, evaluated on Video-MME, LongVideoBench, MLVU, and LVBench, and reported to retain about 98% of Qwen2.5-VL performance under a stated 17× token reduction. Additional experiments consider fine-grained tasks, short-video benchmarks, generalization to LLaVA-Video and LLaVA-OneVision, efficiency, branch ablations, the reconstruction loss, gradient isolation, frame counts, and λ.

Critical strengths

The two-branch decomposition addresses a relevant trade-off. Using sparse low-resolution frames for global semantics and denser high-resolution frames for local evidence is a reasonable design for long-video understanding. Unlike a purely learned bottleneck, the base branch explicitly preserves a broad, low-cost semantic view, while the residual branch can supply details that are useful for the particular question. The branch ablation in Table 5 suggests that the dual-branch model is particularly helpful on medium/long videos and LVBench, which is the intended regime.

Critical weaknesses

The token-counting protocol and fairness of the main comparisons are not sufficiently established, which affects both the 17× claim and the main benchmark interpretation. The paper counts 1024 visual tokens per 448×448 frame and 256 tokens per 224×224 frame in Figure 4 and Eq. 12. However, the public Qwen2.5-VL configuration uses a temporal patch size of 2 and spatial merge size of 2. Under the native post-merger video-tokenization path, two 448×448 frames would correspond to 256 LLM visual tokens, i.e. 128 tokens per frame on average, rather than 1024. It is therefore unclear whether Figure 4 counts pre-merger ViT patches, post-merger LLM tokens, or tokens after a modified projection pipeline.

The novelty is primarily an incremental combination, and the evidence for the proposed mechanism is weaker than the claims made for it. Text-guided or question-conditioned visual compression, learned compression/resamplers, modality pre-fusion, and auxiliary feature-consistency objectives have all been explored in prior work, including LongVU, LLaVA-Mini, EPIC, and question-conditioned compression systems. The paper’s distinction must therefore rest on the particular combination of low-resolution base tokens, high-resolution query-conditioned residual tokens, latent reconstruction, and gradient isolation. The current experiments do not fully isolate this contribution.

Questions for authors

Do you use the native Qwen2.5-VL temporal patch size of 2 and spatial merge size of 2? If not, which component is bypassed or replaced? If the native merger is used, why are 1024 tokens per 448×448 frame counted in Figure 4? If the merger is not used, how is this design kept comparable to Qwen2.5-VL and to token-pruning baselines?

For Table 2, how many tokens does each method actually send into the LLM after all merging/pruning/compression steps? Is “6%” computed relative to the same token representation for FastV, VisionZip, FastVID, and DB-Video? Please report actual token counts rather than only retention percentages.

Please provide minimal pseudocode for the gradient isolation. In particular, specify where detach() or separate backward passes are placed and show that L_rec has zero gradient for the cross-attention parameters while L_vqa still has nonzero gradient there. Also specify the MSE reduction, whether F_h is detached, whether the projector is frozen, and how α is updated.

What is the end-to-end inference cost for the main 128/512-frame configuration, including video decoding, both visual encodings, cross-attention, compression, reconstruction-head overhead during training, and LLM decoding? Please report wall-clock latency and peak GPU memory.

Reviewer_cigiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DB-Video, a dual-branch visual token compression framework for long video understanding. The method combines a low-resolution base semantic branch for preserving global video context with a query-guided residual branch for capturing fine-grained details. A reconstruction loss with gradient isolation is further introduced to improve information preservation during compression. Experiments on four long-video benchmarks show that DB-Video achieves a 17× reduction in visual tokens fed to the LLM while retaining approximately 98% of the original model's performance.

Critical strengths

The paper addresses an important problem in efficient long-video understanding, and the dual-branch architecture provides an intuitive approach to balancing global semantics and query-relevant details. The experimental results demonstrate a favorable accuracy–efficiency trade-off, with substantial visual token reduction and relatively small performance degradation.

Critical weaknesses
  1. Most compression baselines in the main experiments are from 2024–2025, while several highly relevant recent methods are missing, including QViC-MF (CVPR 2026), Tempo (ECCV 2026), and FLoC (ICLR 2026). In addition, Video-XL-Pro explores reconstructive token compression, which is closely related to the proposed reconstruction objective, but is not discussed or compared. XComp (NeurIPS 2025) is mentioned in the related work but excluded from the experiments. Moreover, the existing comparisons involve different backbones, training configurations, and token budgets, making it difficult to establish the superiority of the proposed compression method. More comprehensive comparisons under consistent settings are needed.

  2. Although the dual-branch design is presented as a main contribution, the ablation results show relatively modest improvements over the base semantic branch alone. In Table 5, the dual-branch model achieves 59.2 on Video-MME, slightly below the base semantic branch (59.7), while improving LVBench from 36.5 to 38.8. Furthermore, removing cross-attention results in only minor performance degradation (Table 10). These results raise questions about whether the additional architectural complexity is sufficiently justified and whether simpler compression designs could achieve comparable performance.

Questions for authors

I think the authors need to strengthen the comparison with recent video compression methods under matched token budgets and evaluation settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.