acceptodds
Under review as a conference paper at ICLR 2027

AdaDPC: Adaptive Dual-Path Computation for Efficient Long-Video Understanding

Abstract

Long-video understanding requires models to process large visual token sequences, yet useful visual evidence does not necessarily require full updates at every layer. The key challenge is therefore to reduce redundant visual state refinement without losing access to information that may still support the answer. We propose AdaDPC, an adaptive dual-path computation framework that explicitly separates visual state refinement from visual evidence access. The Selective Refinement Path performs full attention and FFN updates on dynamically selected visual blocks, while the Query-Guided Access Path allows text queries to retrieve evidence from the remaining visual states at low cost. Both paths operate on a shared evolving visual state sequence, allowing visual blocks to change computational roles across layers. Cross-Layer Priority Feedback further uses question-relevant evidence discovered during access to guide subsequent refinement within a fixed budget. Across eight benchmarks, AdaDPC outperforms the dense model, improving MLVU Dev from 65.3 to 69.7 and TimeScope from 76.3 to 81.9, gains of 4.4 and 5.6 percentage points, respectively. Meanwhile, AdaDPC reduces FLOPs by 55.5%, demonstrating that broad visual evidence access can be preserved without uniformly applying expensive refinement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Long Video Understanding, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jEhDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AdaDPC, which fully updates selected visual blocks while preserving text access to the remaining states through QGA. Cross-layer priority feedback guides subsequent refinement, improving performance across eight benchmarks while reducing LLM FLOPs.

Critical strengths
  • The figures are well presented
  • Efficient long-video understanding is an important problem with clear practical value.
Critical weaknesses
  • The query-aware design raises concerns about its practicality in multi-turn conversations.
  • The method relies on a complicated combination of SRP, QGA, PF, routing, scheduling, and gating.
  • The evaluation relies on the relatively dated Qwen2.5-VL-3B backbone. The paper also does not verify whether the method remains effective on larger models.
Questions for authors
  • How can the method be integrated into SGLang or vLLM, and does it deliver end-to-end speedups?
9 comments
AuthorsOpenReview ↗

Response to W1, W2 & W3 by Reviewer jEhD

W1: The query-aware design raises concerns about its practicality in multi-turn conversations.

Response to W1:

Thank you for raising this concern. Query-guided access is compatible with reusing cached visual states across conversation turns. The router directly scores visual-token states, while QGA uses text queries to access visual keys and values. For multi-turn conversations over the same video, the per-layer visual assignments and KV states are retained after the initial prefill and reused across turns. Subsequent turns process only the newly appended text, accessing cached visual evidence through SRP and QGA without repeating visual refinement. Crucially, unrefined visual tokens remain accessible, allowing later questions to retrieve evidence beyond the initially refined blocks.

To validate this in practice, we have implemented and evaluated multi-turn video QA in vLLM with both 3B and 7B models. As shown in Tables R1–R2, AdaDPC achieves 2.23–3.33× speedups on 3B and 2.22–3.33× on 7B over Dense across 256–1,024 sampled frames. These measurements provide direct evidence that AdaDPC supports multi-turn video conversations while retaining substantial runtime benefits.

Table R1 reports the 3B multi-turn latency.

Table R1. Multi-turn video QA latency in vLLM for the 3B models. Dense denotes Qwen2.5-VL-3B.

\hline \text{Video (s)} & \text{Frames} & \text{Dense (s)} & \text{AdaDPC (s)} & \text{Speedup} \\\\ \hline 60 & 256 & 11.118 & \mathbf{4.98} & \mathbf{2.23}\times \\\\ 600 & 512 & 28.307 & \mathbf{10.372} & \mathbf{2.73}\times \\\\ 1800 & 512 & 28.254 & \mathbf{10.566} & \mathbf{2.67}\times \\\\ 3600 & 1024 & 86.825 & \mathbf{26.044} & \mathbf{3.33}\times \\\\ \hline \end{array}$$ Table R2 reports the corresponding 7B multi-turn latency. **Table R2. Multi-turn video QA latency in vLLM for the 7B models. Dense denotes Qwen2.5-VL-7B.** $$\begin{array}{rrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Dense (s)} & \text{AdaDPC (s)} & \text{Speedup} \\\\ \hline 60 & 256 & 17.78 & \mathbf{7.998} & \mathbf{2.22}\times \\\\ 600 & 512 & 43.576 & \mathbf{16.022} & \mathbf{2.72}\times \\\\ 1800 & 512 & 43.527 & \mathbf{15.994} & \mathbf{2.72}\times \\\\ 3600 & 1024 & 127.179 & \mathbf{38.139} & \mathbf{3.33}\times \\\\ \hline \end{array}$$ --- > W2: The method relies on a complicated combination of SRP, QGA, PF, routing, scheduling, and gating. **Response to W2:** Thank you for this comment. AdaDPC is a compact dual-path architecture, rather than a combination of six independent methods. Its methodological novelty is to decouple visual refinement from evidence access: visual states remain readable even when costly updates are skipped, and relevance discovered during access guides subsequent computation. Both paths share visual states and Q/K/V projections, while routing, scheduling, gating, and feedback are lightweight controls within this same architecture. --- > W3: The evaluation relies on the relatively dated Qwen2.5-VL-3B backbone. The paper also does not verify whether the method remains effective on larger models. **Response to W3:** Thank you for this comment. We have extended our evaluation to the larger **Qwen2.5-VL-7B backbone**. As shown in Table R3, **AdaDPC consistently outperforms the corresponding Dense baseline on all eight benchmarks, reproducing the same positive trend observed with the 3B backbone.** These results support AdaDPC’s effectiveness beyond the original 3B setting. **Table R3. Results on the Qwen2.5-VL-7B backbone. Dense denotes the original Qwen2.5-VL-7B baseline. Bold indicates the highest score in each column.** $$\begin{array}{lrrrrrrrr} \hline \text{Model} & \text{LVBench} & \text{LongVideoBench} & \text{MLVU Dev} & \text{VideoMME} & \text{MVBench} & \text{VSI-Bench} & \text{TOMATO} & \text{TimeScope} \\\\ \hline \text{Dense} & 45.3 & 60.7 & 68.3 & 65.1 & 69.6 & 35.6 & 25.6 & 81.0 \\\\ \text{Dense + SFT} & 45.2 & 59.8 & 68.1 & 64.7 & 67.0 & 36.7 & 26.2 & 81.9 \\\\ \hline \text{FastV} & 43.3 & 60.8 & 65.7 & 66.0 & 67.0 & 39.0 & 23.5 & 79.3 \\\\ \text{ToMe} & 43.1 & 60.8 & 66.1 & 65.8 & 67.2 & 38.7 & 23.0 & 80.2 \\\\ \text{MoBA} & 42.3 & 58.3 & 64.7 & 63.4 & 67.5 & 34.8 & 23.5 & 78.8 \\\\ \text{FrameFusion} & 42.8 & 61.1 & 66.3 & \mathbf{66.1} & 66.8 & 39.3 & 22.2 & 81.9 \\\\ \hline \text{AdaDPC} & \mathbf{46.4} & \mathbf{62.8} & \mathbf{71.3} & 65.8 & \mathbf{70.4} & \mathbf{40.6} & \mathbf{28.6} & \mathbf{84.2} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Q1 by Reviewer jEhD (1/3): vLLM Integration and Single-Request End-to-End Latency

Q1: How can the method be integrated into SGLang or vLLM, and does it deliver end-to-end speedups?

Response to Q1:

Thank you for this important question. We have implemented AdaDPC in vLLM and verified model-level end-to-end speedups. The integration uses vLLM’s custom model registration interface, with routing metadata and dual-path attention adapted to its native paged KV cache and prefill/decode execution. We reuse vLLM’s cache management and generation mechanisms, optimize routing-index construction, cache access, and operator execution, and use CUDA Graphs on compatible execution paths to reduce dynamic-routing overhead.

All speedups below are relative to the corresponding Dense baseline. “Model E2E” denotes model-level end-to-end latency.

Under matched hardware, video inputs, and output lengths, AdaDPC achieves model-level end-to-end speedups of 1.59–2.06× on 3B (Table R4). These timings cover visual encoding through completion of generation, excluding CPU preprocessing, rather than only LLM execution or isolated kernels.

Table R4. Single-request inference latency in vLLM for the 3B models. Dense denotes Qwen2.5-VL-3B. Model E2E includes visual encoding and generation but excludes CPU preprocessing.

\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM (s)} & \text{Speedup} & \text{Model E2E (s)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 1.874 & 1.00\times & 3.371 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.028} & \mathbf{1.82}\times & \mathbf{2.119} & \mathbf{1.59}\times \\\\ \hline 600 & 512 & \text{Dense} & 4.944 & 1.00\times & 7.957 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{2.269} & \mathbf{2.18}\times & \mathbf{4.457} & \mathbf{1.79}\times \\\\ \hline 1800 & 512 & \text{Dense} & 4.953 & 1.00\times & 7.973 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{2.282} & \mathbf{2.17}\times & \mathbf{4.485} & \mathbf{1.78}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 15.885 & 1.00\times & 21.923 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{6.234} & \mathbf{2.55}\times & \mathbf{10.647} & \mathbf{2.06}\times \\\\ \hline \end{array}$$ **On 7B, the corresponding model-level end-to-end speedups are 1.45–1.79×** (Table R5), using the same timing scope. **Table R5. Single-request inference latency in vLLM for the 7B models. Dense denotes Qwen2.5-VL-7B. Model E2E includes visual encoding and generation but excludes CPU preprocessing.** $$\begin{array}{rrlrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM (s)} & \text{Speedup} & \text{Model E2E (s)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 3.227 & 1.00\times & 4.73 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.968} & \mathbf{1.64}\times & \mathbf{3.263} & \mathbf{1.45}\times \\\\ \hline 600 & 512 & \text{Dense} & 8.081 & 1.00\times & 11.092 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{4.368} & \mathbf{1.85}\times & \mathbf{6.956} & \mathbf{1.59}\times \\\\ \hline 1800 & 512 & \text{Dense} & 8.063 & 1.00\times & 11.077 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{4.354} & \mathbf{1.85}\times & \mathbf{6.941} & \mathbf{1.60}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 24.321 & 1.00\times & 30.369 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{11.773} & \mathbf{2.07}\times & \mathbf{16.937} & \mathbf{1.79}\times \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Q1 by Reviewer jEhD (2/3): Concurrent vLLM Inference and P95 TTFT/TPOT

Response to Q1 (continued): Concurrent inference.

We additionally evaluate 24 requests at concurrency C=4, each generating 32 tokens (Tables R6–R7). P95 denotes the 95th percentile, TTFT denotes time to first token, and TPOT denotes time per output token. All speedups are relative to the corresponding Dense baseline. TPOT includes generation waiting caused by concurrent prefill.

On 3B, P95 TTFT improves by 1.58–2.05× (Table R6). P95 TPOT improves by 1.23–2.03× across all tested settings.

Table R6. Concurrent inference in vLLM for the 3B models: 24 requests, C=4, and 32 generated tokens per request. TPOT includes generation waiting caused by concurrent prefill.

\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{TTFT P95 (s)} & \text{Speedup} & \text{TPOT P95 (ms/token)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 13.407 & 1.00\times & 12.22 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{8.503} & \mathbf{1.58}\times & \mathbf{9.93} & \mathbf{1.23}\times \\\\ \hline 600 & 512 & \text{Dense} & 32.939 & 1.00\times & 550.23 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{19.129} & \mathbf{1.72}\times & \mathbf{322.9} & \mathbf{1.70}\times \\\\ \hline 1800 & 512 & \text{Dense} & 32.98 & 1.00\times & 549.17 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{19.166} & \mathbf{1.72}\times & \mathbf{327.86} & \mathbf{1.68}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 87.903 & 1.00\times & 2150.91 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{42.797} & \mathbf{2.05}\times & \mathbf{1058.4} & \mathbf{2.03}\times \\\\ \hline \end{array}$$ **On 7B, P95 TTFT improves by 1.50–1.80×** (Table R7). P95 TPOT improves by **1.31–1.78×** across all tested settings. **Table R7. Concurrent inference in vLLM for the 7B models: 24 requests, C=4, and 32 generated tokens per request. TPOT includes generation waiting caused by concurrent prefill.** $$\begin{array}{rrlrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{TTFT P95 (s)} & \text{Speedup} & \text{TPOT P95 (ms/token)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 13.338 & 1.00\times & 12.12 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{8.549} & \mathbf{1.56}\times & \mathbf{9.253} & \mathbf{1.31}\times \\\\ \hline 600 & 512 & \text{Dense} & 44.835 & 1.00\times & 746.55 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{29.964} & \mathbf{1.50}\times & \mathbf{504.87} & \mathbf{1.48}\times \\\\ \hline 1800 & 512 & \text{Dense} & 45.295 & 1.00\times & 757.64 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{28.579} & \mathbf{1.58}\times & \mathbf{480.33} & \mathbf{1.58}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 122.95 & 1.00\times & 3005.5 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{68.397} & \mathbf{1.80}\times & \mathbf{1685.79} & \mathbf{1.78}\times \\\\ \hline \end{array}$$
Reviewer_6AbHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents AdaDPC, a method for cutting compute in video-language models on long videos. It rests on the observation that a visual token can still be useful evidence for the question even when it doesn't need its representation updated by every layer. The main idea is to separate refinement (full attention and FFN updates of a visual token) from access (letting the question read that token's information).

At each layer the model: 1) per-layer router scores 8×4 spatial blocks of visual tokens, 2) a cosine-decaying budget picks the top-scoring blocks for full refinement, and 3) unlike standard Mixture-of-Depths, the skipped blocks aren't dropped. The question tokens can still read them through a cheap, gated text-to-visual cross-attention (the Query-Guided Access Path).

A Cross-Layer Priority Feedback mechanism uses the access path's attention weights to spot question-relevant blocks that weren't refined. Those blocks get guaranteed refinement slots in the next layer, so blocks can switch between paths across depth.

The method is built on Qwen2.5-VL-3B and fine-tuned on about 373K video samples. It reduces LLM FLOPs by 55.5% at 32K visual tokens and improves over both the original model and a matched fine-tuned dense baseline on eight benchmarks.

Critical strengths
  1. Clear new idea. The paper separates two things that efficient video-LLM methods usually treat as one. Specofocallu, refining a visual token and accessing it. Pruning, merging and Mixture-of-Depths methods either compute a token fully or lose it. AdaDPC keeps skipped tokens readable by the question at almost no cost, which fills a real gap in conditional-computation methods.

  2. Simple, cheap addition. The Query-Guided Access Path is a small change to a standard Mixture-of-Depths layer consisting in a gated cross-attention from the few question tokens to the skipped visual tokens. It adds only 0.36% FLOPs over refinement alone.

  3. Allocation that adapts across layers. Cross-Layer Priority Feedback reuses attention weights the model already computes to find question-relevant blocks that were skipped and refine them in the next layer. Blocks can therefore switch roles between layers instead of being pruned permanently, and Fig. 4 shows this happening.

  4. Large efficiency gains with no accuracy loss. LLM FLOPs drop by 55.5%, and accuracy improves over the dense model on all eight benchmarks, with the biggest gains on long videos (MLVU +4.4, TimeScope +5.6). Kernel-level speedups on one profiled layer (attention 5.24×, FFN 2.65×) suggest the FLOPs savings are achievable in practice.

  5. Fair comparison against the dense model. Dense + SFT, refinement-only and the full method share the same data, steps, learning rate and batch size. The refinement-only vs. full comparison (Table 2) cleanly isolates what the access path adds: gains on all eight benchmarks, e.g., +4.3 MVBench and +3.8 MLVU.

  6. Thorough ablations. Each design choice is tested correctly, the access gate, block size (token-level vs. 16/32/64-token blocks), budget schedule (cosine vs. fixed ratios vs. top-p) and feedback ratio λ. Each component contributes.

Critical weaknesses
  1. There is no fair accuracy–efficiency comparison.
    • Table 1 compares against FastV, ToMe, MoBA and FrameFusion without reporting their FLOPs or token-retention ratios. These baselines are also training-free, while AdaDPC is fine-tuned.
    • The closest trained baselines (p-MoD, γ-MoD, VideoLLM-MoD) are cited but not compared.
    • The paper doesn't test the simplest way to keep access to all tokens: leave the skipped tokens in the main attention as keys and values. QGA instead adds a separate, question-only cross-attention, and this choice isn't justified against the simpler alternative.
  2. The efficiency claims are incomplete.
    • Only theoretical FLOPs are reported, with no end-to-end latency or throughput.
    • Kernel profiling covers only one decoder layer (layer 20).
    • Overheads from gathering and top-K selection aren't measured.
    • K and V are still computed and stored for every token at every layer, so KV-cache memory isn't reduced. That cache is often the real bottleneck for long videos.
Questions for authors

The baseline numbers are questionable.

  • Possible problem in Table 6. MVBench has 20 subtasks with 200 questions each, so every subtask accuracy must be a multiple of 0.5. AdaDPC's values fit this. But every Dense + SFT value ends in .15 or .65 (e.g., 81.15, 44.15, 74.65), which can't happen with 200 questions.
  • The fine-tuned dense baseline looks under-trained. Dense + SFT often scores below the original Dense model (VideoMME 61.5 → 60.3, MVBench 65.7 → 64.9). The learning rate is very low (2e-6) and the number of training steps isn't reported.
  • Together, these weaken the central claim that AdaDPC improves accuracy beyond what fine-tuning alone provides. The authors should explain both.
6 comments
AuthorsOpenReview ↗

Response to W1 by Reviewer 6AbH:

W1.1: Table 1 compares against FastV, ToMe, MoBA and FrameFusion without reporting their FLOPs or token-retention ratios. These baselines are also training-free, while AdaDPC is fine-tuned.

Response to W1.1:

Thank you for highlighting the need to clarify computational budgets and training settings. We now provide the measured FLOPs reductions and budget configurations for all four baselines and AdaDPC under the same 32K visual-token context (Table R1), making their accuracy–compute trade-offs explicit.

We also clarify that MoBA was fine-tuned on the same training data as AdaDPC, rather than evaluated as a training-free baseline. AdaDPC outperforms this data-matched MoBA baseline on all eight benchmarks while achieving a greater FLOPs reduction (55.5% vs. 41.1%).

On the 3B backbone, AdaDPC exceeds the strongest result among FastV, ToMe, and FrameFusion on every benchmark, with the largest margins on MLVU Dev (+4.3 percentage points), TOMATO (+2.5), and TimeScope (+2.4). Compared with ToMe specifically, AdaDPC also achieves a greater FLOPs reduction (55.5% vs. 31.4%) while improving accuracy across all eight benchmarks. These results establish favorable accuracy–compute trade-offs against both training-free and fine-tuned baselines.

Table R1. LLM FLOPs reductions and budget configurations under a 32K visual-token context. Reductions are relative to Dense. MoBA and AdaDPC are fine-tuned on the same training data.

\hline \text{Method} & \text{Training} & \text{LLM FLOPs reduction} & \text{Budget configuration} \\\\ \hline \text{FastV} & \text{Training-free} & 60.6\\% & \text{Layer}=2;\\;\text{visual keep}=50\\% \\\\ \text{ToMe} & \text{Training-free} & 31.4\\% & \text{Progressive merging; final visual keep}=50\\% \\\\ \text{FrameFusion} & \text{Training-free} & 81.7\\% & \text{COST}=0.3;\\;\text{sim. threshold}=0.6;\\;\text{switch ratio}=0.1 \\\\ \text{MoBA} & \text{Fine-tuned} & 41.1\\% & \text{Block size}=256;\\;\text{Top-K}=8 \\\\ \text{AdaDPC} & \text{Fine-tuned} & 55.5\\% & \text{32-token blocks; cosine refinement schedule} \\\\ \hline \end{array}$$ --- > W1.2: The closest trained baselines (p-MoD, γ-MoD, VideoLLM-MoD) are cited but not compared. > W1.3: The paper doesn't test the simplest way to keep access to all tokens: leave the skipped tokens in the main attention as keys and values. QGA instead adds a separate, question-only cross-attention, and this choice isn't justified against the simpler alternative. **Response to W1.2 & W1.3:** Thank you for highlighting these missing comparisons. We have added **p-MoD, $\gamma$-MoD, and VideoLLM-MoD** under our **8K visual-token ablation setting**, using this setting given the available compute budget (Table R2). **AdaDPC outperforms all three trained baselines on all three benchmarks.** In particular, at nearly identical FLOPs reductions (**45.1% for AdaDPC vs. 45.3% for p-MoD**), AdaDPC improves MLVU Dev, VideoMME, and TOMATO by **3.8, 3.1, and 1.4 percentage points**, respectively. It also achieves greater FLOPs savings than $\gamma$-MoD and VideoLLM-MoD. These direct comparisons substantiate its accuracy–compute advantages over the closest trained baselines. We also evaluated the suggested alternative, **Full KV**, which **retains skipped visual states as keys and values in the main attention.** Under the same 8K ablation setting, **AdaDPC improves MLVU Dev, VideoMME, and TOMATO by 2.1, 3.4, and 0.3 percentage points over Full KV, while increasing the FLOPs reduction from 41.2% to 45.1%** (Table R2). **The distinction is that Full KV also introduces attention from selected visual queries to skipped visual keys. QGA instead provides the additional access through text queries, preserving evidence access without these extra visual-to-visual attention interactions.** The comparison therefore directly supports the separate access path over simply retaining all keys and values in the main attention. **Table R2. Comparisons with trained MoD baselines and the Full-KV alternative under the 8K visual-token ablation setting. LLM FLOPs reductions are relative to Dense. Bold indicates the highest accuracy in each benchmark column.** $$\begin{array}{lrrrr} \hline \text{Method} & \text{LLM FLOPs reduction} & \text{MLVU Dev} & \text{VideoMME} & \text{TOMATO} \\\\ \hline \text{p-MoD} & 45.3\\% & 58.7 & 58.4 & 22.9 \\\\ \gamma\text{-MoD} & 41.1\\% & 53.8 & 56.2 & 20.7 \\\\ \text{VideoLLM-MoD} & 40.8\\% & 56.1 & 53.9 & 21.2 \\\\ \hline \text{Full KV} & 41.2\\% & 60.4 & 58.1 & 24.0 \\\\ \text{AdaDPC} & 45.1\\% & \mathbf{62.5} & \mathbf{61.5} & \mathbf{24.3} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to W2 by Reviewer 6AbH (1/2): 3B/7B LLM Latency, Throughput, and Peak Memory

W2.1: Only theoretical FLOPs are reported, with no end-to-end latency or throughput.

Response to W2.1:

We thank the reviewer for requesting practical efficiency measurements. Tables R3–R4 report measured LLM inference latency, prefill/decode throughput, and whole-model peak memory on a single NVIDIA A800 GPU, covering both 3B and 7B models across 256–1,024 sampled frames. Latency and throughput cover LLM execution, while peak memory covers the entire model.

All speedups and memory reductions below are relative to the corresponding dense baseline. In Tables R3–R4, latency is measured in seconds, throughput in token/s, and whole-model peak memory in GiB. Each “Ratio” column reports the throughput ratio for the preceding phase; “Reduction” denotes the percentage reduction in whole-model peak memory.

On 3B, AdaDPC achieves 2.56–2.70× LLM speedups, with higher prefill and decode throughput in every tested setting (Table R3). At 1,024 frames, whole-model peak memory decreases by 53.27%.

Table R3. Measured inference efficiency of the 3B models on a single NVIDIA A800 GPU. Dense denotes Qwen2.5-VL-3B.

\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM latency (s)} & \text{Speedup} & \text{Prefill (token/s)} & \text{Ratio} & \text{Decode (token/s)} & \text{Ratio} & \text{Peak (GiB)} & \text{Reduction} \\\\ \hline 60 & 256 & \text{Dense} & 2.725 & 1.00\times & 19241.71 & 1.00\times & 29.78 & 1.00\times & 13.34 & - \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.008} & \mathbf{2.70}\times & \mathbf{40700.56} & \mathbf{2.12}\times & \mathbf{144.31} & \mathbf{4.85}\times & \mathbf{9.513} & \mathbf{28.69}\\% \\\\ \hline 600 & 512 & \text{Dense} & 5.743 & 1.00\times & 13715.30 & 1.00\times & 29.89 & 1.00\times & 19.676 & - \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{2.242} & \mathbf{2.56}\times & \mathbf{32318.41} & \mathbf{2.36}\times & \mathbf{126.72} & \mathbf{4.24}\times & \mathbf{11.326} & \mathbf{42.44}\\% \\\\ \hline 1800 & 512 & \text{Dense} & 5.766 & 1.00\times & 13662.98 & 1.00\times & 29.87 & 1.00\times & 19.676 & - \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{2.251} & \mathbf{2.56}\times & \mathbf{32176.79} & \mathbf{2.36}\times & \mathbf{126.66} & \mathbf{4.24}\times & \mathbf{11.326} & \mathbf{42.44}\\% \\\\ \hline 3600 & 1024 & \text{Dense} & 16.515 & 1.00\times & 8359.17 & 1.00\times & 29.05 & 1.00\times & 32.351 & - \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{6.432} & \mathbf{2.57}\times & \mathbf{21064.28} & \mathbf{2.52}\times & \mathbf{101.89} & \mathbf{3.51}\times & \mathbf{15.118} & \mathbf{53.27}\\% \\\\ \hline \end{array}$$ **On 7B, AdaDPC achieves 2.19–2.45× LLM speedups**, with higher prefill and decode throughput in every tested setting (Table R4). At 1,024 frames, whole-model peak memory decreases by **42.37%**. These measurements demonstrate practical runtime and memory benefits beyond theoretical FLOPs savings. **Table R4. Measured inference efficiency of the 7B models on a single NVIDIA A800 GPU. Dense denotes Qwen2.5-VL-7B.** $$\begin{array}{rrlrrrrrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM latency (s)} & \text{Speedup} & \text{Prefill (token/s)} & \text{Ratio} & \text{Decode (token/s)} & \text{Ratio} & \text{Peak (GiB)} & \text{Reduction} \\\\ \hline 60 & 256 & \text{Dense} & 3.755 & 1.00\times & 10993.66 & 1.00\times & 37.88 & 1.00\times & 22.318 & - \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.714} & \mathbf{2.19}\times & \mathbf{23716.46} & \mathbf{2.16}\times & \mathbf{88.01} & \mathbf{2.32}\times & \mathbf{18.269} & \mathbf{18.14}\\% \\\\ \hline 600 & 512 & \text{Dense} & 8.704 & 1.00\times & 8287.36 & 1.00\times & 33.92 & 1.00\times & 29.178 & - \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{3.742} & \mathbf{2.33}\times & \mathbf{19276.02} & \mathbf{2.33}\times & \mathbf{78.87} & \mathbf{2.33}\times & \mathbf{20.208} & \mathbf{30.74}\\% \\\\ \hline 1800 & 512 & \text{Dense} & 8.674 & 1.00\times & 8316.31 & 1.00\times & 33.96 & 1.00\times & 29.178 & - \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{3.773} & \mathbf{2.30}\times & \mathbf{19101.65} & \mathbf{2.30}\times & \mathbf{78.85} & \mathbf{2.32}\times & \mathbf{20.208} & \mathbf{30.74}\\% \\\\ \hline 3600 & 1024 & \text{Dense} & 25.008 & 1.00\times & 5453.20 & 1.00\times & 23.12 & 1.00\times & 42.894 & - \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{10.200} & \mathbf{2.45}\times & \mathbf{13265.55} & \mathbf{2.43}\times & \mathbf{65.83} & \mathbf{2.85}\times & \mathbf{24.721} & \mathbf{42.37}\\% \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to W2 by Reviewer 6AbH (2/2): Kernel Profiling, Routing Overheads, and KV-Cache Quantization

W2.2: Kernel profiling covers only one decoder layer (layer 20).

Response to W2.2:

Thank you for highlighting the limited scope of the original profiling. We supplement the original layer-20 result with expanded kernel-profiling statistics for both model scales (Table R5), reporting mean, median, and maximum values, rather than relying on a single reported speedup. Mean self-attention speedups are 5.43× (3B) and 6.19× (7B), while mean FFN speedups are 2.61× and 2.60×, respectively. The table also reports FFN-local peak-memory reductions, complementing the whole-model memory measurements above.

Table R5. Kernel profiling on a single NVIDIA A800 GPU. Memory reduction refers to the FFN local peak.

\hline \text{Metric} & \text{3B Mean} & \text{Median} & \text{Maximum} & \text{7B Mean} & \text{Median} & \text{Maximum} \\\\ \hline \text{Self-attention speedup} & 5.43\times & 2.91\times & 12.55\times & 6.19\times & 3.00\times & 14.43\times \\\\ \text{FFN speedup} & 2.61\times & 1.90\times & 4.83\times & 2.60\times & 1.87\times & 4.86\times \\\\ \text{FFN memory reduction} & 44.64\\% & 47.75\\% & 79.90\\% & 44.03\\% & 47.15\\% & 79.91\\% \\\\ \hline \end{array}$$ --- > W2.3: Overheads from gathering and top-K selection aren't measured. **Response to W2.3:** We appreciate the reviewer’s attention to these implementation overheads. **Our LLM latency measurements cover the complete LLM execution, including full-sequence Q/K/V projections, routing, Top-K selection, gathering, and QGA.** Their execution costs are therefore included in the reported latency, although not separately broken down. The measured **2.19–2.70× speedups represent net acceleration after accounting for these overheads**, rather than estimates extrapolated from isolated kernel timings. --- > W2.4: K and V are still computed and stored for every token at every layer, so KV-cache memory isn't reduced. That cache is often the real bottleneck for long videos. **Response to W2.4:** Thank you for highlighting the KV-cache bottleneck. **The default BF16 implementation retains full-sequence KV states, so its computation savings do not by themselves reduce the KV-cache footprint.** To directly address this bottleneck, we evaluate **branch-selective KV-cache quantization**, converting only the QGA KV cache from **BF16 to INT4 while leaving the remaining KV states in BF16.** On **Qwen2.5-VL-7B with a 32K visual-token context**, this reduces KV-cache memory by **32.15–32.17%** relative to the corresponding BF16 configuration, with accuracy decreases of only **0.2 percentage points on LongVideoBench and 0.1 on MLVU Dev** (Table R6). **These results demonstrate that AdaDPC’s computational savings can be complemented by substantial KV-cache compression, while preserving access to all visual states and nearly unchanged accuracy.** **Table R6. QGA KV-cache quantization on Qwen2.5-VL-7B with a 32K visual-token context. Only the QGA KV cache is quantized. KV-cache memory reductions are relative to the corresponding BF16 configuration.** $$\begin{array}{llrr} \hline \text{Benchmark} & \text{QGA KV precision} & \text{Score} & \text{KV-cache memory reduction} \\\\ \hline \text{LongVideoBench} & \text{BF16} & 62.8 & - \\\\ \text{LongVideoBench} & \text{INT4} & 62.6 & \mathbf{32.15}\\% \\\\ \hline \text{MLVU Dev} & \text{BF16} & 71.3 & - \\\\ \text{MLVU Dev} & \text{INT4} & 71.2 & \mathbf{32.17}\\% \\\\ \hline \end{array}$$
Reviewer_1adZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaDPC, an adaptive dual-path framework that separates expensive visual-state refinement from lightweight visual-evidence access for efficient long-video understanding. It dynamically routes selected visual blocks through full attention and FFN updates, while allowing text queries to access the remaining visual states and using cross-layer feedback to guide later computation. Experiments across eight benchmarks show consistent gains over dense baselines while reducing LLM inference FLOPs by 55.5%.

Critical strengths
  1. The paper presents a clear and intuitive motivation by distinguishing visual-state refinement from visual-evidence access, which provides a reasonable perspective for reducing redundant computation in long-video understanding.

  2. The proposed dual-path design is technically coherent, combining selective refinement, lightweight query-guided access, and cross-layer priority feedback within a unified visual state sequence.

  3. The empirical results are strong: AdaDPC consistently outperforms the dense baseline across eight video-language benchmarks while reducing LLM inference FLOPs by 55.5%.

Critical weaknesses
  1. The methodological novelty is somewhat limited, as AdaDPC largely builds on existing MoD-style conditional computation, block-wise routing, and query-based visual access. The paper could better clarify what is fundamentally new compared with prior efficient multimodal computation methods.

  2. The efficiency analysis is not fully end-to-end. Although the paper reports a 55.5% reduction in LLM FLOPs, it still retains full-sequence Q/K/V projections and introduces additional QGA attention, while complete inference latency, throughput, and memory measurements are limited.

  3. The generality of the method is not sufficiently validated. Most experiments are conducted on a single Qwen2.5-VL-3B backbone, and the category-level results also show regressions on a non-trivial number of subtasks, leaving questions about robustness across model scales and task types.

Questions for authors
  1. Could the authors clarify the key methodological novelty of AdaDPC compared with prior MoD-style conditional computation and query-based visual access methods?

  2. Could the authors provide end-to-end efficiency results, including inference latency, throughput, and peak memory, rather than mainly reporting FLOPs and kernel-level speedups?

  3. How well does AdaDPC generalize beyond Qwen2.5-VL-3B? Results on another backbone or model scale would help assess the robustness of the proposed routing strategy.

9 comments
AuthorsOpenReview ↗

Response to W1 & Q1 by Reviewer 1adZ:

Response to W1 & Q1:

W1: The methodological novelty is somewhat limited, as AdaDPC largely builds on existing MoD-style conditional computation, block-wise routing, and query-based visual access. The paper could better clarify what is fundamentally new compared with prior efficient multimodal computation methods. Q1: Could the authors clarify the key methodological novelty of AdaDPC compared with prior MoD-style conditional computation and query-based visual access methods?

Thank you for the opportunity to clarify. The key novelty of AdaDPC is to separate visual-state refinement from visual-evidence access, and to use access to guide subsequent refinement.

At each routed layer, SRP selectively performs full updates, while QGA keeps the remaining visual states accessible to text queries. Both paths operate on a shared, evolving visual sequence. Cross-Layer Priority Feedback further uses the relevance discovered during access to prioritize skipped blocks for refinement in the next layer, within the existing refinement budget. Our contribution therefore lies in coordinating refinement, access, and subsequent computation allocation, using established routing and cross-attention operators.

Reviewer 6AbH explicitly recognizes this novelty, describing AdaDPC as a “Clear new idea” and stating:

“AdaDPC keeps skipped tokens readable by the question at almost no cost, which fills a real gap in conditional-computation methods.”

Reviewer 6AbH further highlights:

“Cross-Layer Priority Feedback reuses attention weights the model already computes to find question-relevant blocks that were skipped and refine them in the next layer.”

These observations reinforce the methodological distinctions outlined above. We will make them more explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to W2 & Q2 by Reviewer 1adZ (1/2): 3B LLM Latency, Throughput, and Peak Memory

Response to W2 and Q2:

W2: The efficiency analysis is not fully end-to-end. Although the paper reports a 55.5% reduction in LLM FLOPs, it still retains full-sequence Q/K/V projections and introduces additional QGA attention, while complete inference latency, throughput, and memory measurements are limited.

Q2: Could the authors provide end-to-end efficiency results, including inference latency, throughput, and peak memory, rather than mainly reporting FLOPs and kernel-level speedups?

Thank you for these comments. The reported 55.5% LLM FLOPs reduction already accounts for full-sequence Q/K/V projections and the additional QGA computation (§3.3 and Appendix C.3). To complement this analysis, we provide measured LLM inference latency, prefill/decode throughput, and whole-model peak memory for both 3B and 7B across 256–1,024 sampled frames.

Experimental setup. All measurements in Tables R1–R3 were conducted on a single NVIDIA A800 GPU. Latency and throughput refer to LLM execution, while the peak memory reported in Tables R1–R2 covers the entire model. All speedup and memory-reduction ratios are calculated relative to the corresponding dense baseline.

In Tables R1–R2, video duration and LLM latency are measured in seconds, prefill/decode throughput in token/s, and whole-model peak memory in GiB. “Reduction” denotes the percentage reduction in whole-model peak memory.

With full-sequence Q/K/V projections and QGA retained, AdaDPC achieves 2.56–2.70× LLM speedups for 3B (Table R1), with higher prefill and decode throughput in every tested setting. At 1,024 frames, whole-model peak memory decreases from 32.351 to 15.118 GiB (53.27%).

Table R1. Measured inference efficiency of the 3B models. Dense denotes Qwen2.5-VL-3B.

\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM latency (s)} & \text{Speedup} & \text{Prefill (token/s)} & \text{Ratio} & \text{Decode (token/s)} & \text{Ratio} & \text{Peak (GiB)} & \text{Reduction} \\\\ \hline 60 & 256 & \text{Dense} & 2.725 & 1.00\times & 19241.71 & 1.00\times & 29.78 & 1.00\times & 13.34 & - \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.008} & \mathbf{2.70}\times & \mathbf{40700.56} & \mathbf{2.12}\times & \mathbf{144.31} & \mathbf{4.85}\times & \mathbf{9.513} & \mathbf{28.69}\\% \\\\ \hline 600 & 512 & \text{Dense} & 5.743 & 1.00\times & 13715.30 & 1.00\times & 29.89 & 1.00\times & 19.676 & - \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{2.242} & \mathbf{2.56}\times & \mathbf{32318.41} & \mathbf{2.36}\times & \mathbf{126.72} & \mathbf{4.24}\times & \mathbf{11.326} & \mathbf{42.44}\\% \\\\ \hline 1800 & 512 & \text{Dense} & 5.766 & 1.00\times & 13662.98 & 1.00\times & 29.87 & 1.00\times & 19.676 & - \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{2.251} & \mathbf{2.56}\times & \mathbf{32176.79} & \mathbf{2.36}\times & \mathbf{126.66} & \mathbf{4.24}\times & \mathbf{11.326} & \mathbf{42.44}\\% \\\\ \hline 3600 & 1024 & \text{Dense} & 16.515 & 1.00\times & 8359.17 & 1.00\times & 29.05 & 1.00\times & 32.351 & - \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{6.432} & \mathbf{2.57}\times & \mathbf{21064.28} & \mathbf{2.52}\times & \mathbf{101.89} & \mathbf{3.51}\times & \mathbf{15.118} & \mathbf{53.27}\\% \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to W2 & Q2 by Reviewer 1adZ (2/2): 7B Inference Efficiency and Kernel Profiling

Response to W2 and Q2 (continued):

On 7B, AdaDPC achieves 2.19–2.45× LLM speedups (Table R2), with higher prefill and decode throughput in every tested setting. At 1,024 frames, whole-model peak memory decreases from 42.894 to 24.721 GiB (42.37%). These measurements retain full-sequence Q/K/V projections and QGA.

Table R2. Measured inference efficiency of the 7B models. Dense denotes Qwen2.5-VL-7B.

\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM latency (s)} & \text{Speedup} & \text{Prefill (token/s)} & \text{Ratio} & \text{Decode (token/s)} & \text{Ratio} & \text{Peak (GiB)} & \text{Reduction} \\\\ \hline 60 & 256 & \text{Dense} & 3.755 & 1.00\times & 10993.66 & 1.00\times & 37.88 & 1.00\times & 22.318 & - \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.714} & \mathbf{2.19}\times & \mathbf{23716.46} & \mathbf{2.16}\times & \mathbf{88.01} & \mathbf{2.32}\times & \mathbf{18.269} & \mathbf{18.14}\\% \\\\ \hline 600 & 512 & \text{Dense} & 8.704 & 1.00\times & 8287.36 & 1.00\times & 33.92 & 1.00\times & 29.178 & - \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{3.742} & \mathbf{2.33}\times & \mathbf{19276.02} & \mathbf{2.33}\times & \mathbf{78.87} & \mathbf{2.33}\times & \mathbf{20.208} & \mathbf{30.74}\\% \\\\ \hline 1800 & 512 & \text{Dense} & 8.674 & 1.00\times & 8316.31 & 1.00\times & 33.96 & 1.00\times & 29.178 & - \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{3.773} & \mathbf{2.30}\times & \mathbf{19101.65} & \mathbf{2.30}\times & \mathbf{78.85} & \mathbf{2.32}\times & \mathbf{20.208} & \mathbf{30.74}\\% \\\\ \hline 3600 & 1024 & \text{Dense} & 25.008 & 1.00\times & 5453.20 & 1.00\times & 23.12 & 1.00\times & 42.894 & - \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{10.200} & \mathbf{2.45}\times & \mathbf{13265.55} & \mathbf{2.43}\times & \mathbf{65.83} & \mathbf{2.85}\times & \mathbf{24.721} & \mathbf{42.37}\\% \\\\ \hline \end{array}$$ Table R3 additionally summarizes kernel profiling for both model scales. **Table R3. Kernel profiling on a single NVIDIA A800 GPU. Memory reduction refers to the FFN local peak.** $$\begin{array}{lrrrrrr} \hline \text{Metric} & \text{3B Mean} & \text{Median} & \text{Maximum} & \text{7B Mean} & \text{Median} & \text{Maximum} \\\\ \hline \text{Self-attention speedup} & 5.43\times & 2.91\times & 12.55\times & 6.19\times & 3.00\times & 14.43\times \\\\ \text{FFN speedup} & 2.61\times & 1.90\times & 4.83\times & 2.60\times & 1.87\times & 4.86\times \\\\ \text{FFN memory reduction} & 44.64\\% & 47.75\\% & 79.90\\% & 44.03\\% & 47.15\\% & 79.91\\% \\\\ \hline \end{array}$$ **These measurements demonstrate that the computational savings translate into practical runtime and memory benefits.**

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.