- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes AdaDPC, which fully updates selected visual blocks while preserving text access to the remaining states through QGA. Cross-layer priority feedback guides subsequent refinement, improving performance across eight benchmarks while reducing LLM FLOPs.
Critical strengths
- The figures are well presented
- Efficient long-video understanding is an important problem with clear practical value.
Critical weaknesses
- The query-aware design raises concerns about its practicality in multi-turn conversations.
- The method relies on a complicated combination of SRP, QGA, PF, routing, scheduling, and gating.
- The evaluation relies on the relatively dated Qwen2.5-VL-3B backbone. The paper also does not verify whether the method remains effective on larger models.
Questions for authors
- How can the method be integrated into SGLang or vLLM, and does it deliver end-to-end speedups?
9 comments
Response to Q1 by Reviewer jEhD (1/3): vLLM Integration and Single-Request End-to-End Latency
Q1: How can the method be integrated into SGLang or vLLM, and does it deliver end-to-end speedups?
Response to Q1:
Thank you for this important question. We have implemented AdaDPC in vLLM and verified model-level end-to-end speedups. The integration uses vLLM’s custom model registration interface, with routing metadata and dual-path attention adapted to its native paged KV cache and prefill/decode execution. We reuse vLLM’s cache management and generation mechanisms, optimize routing-index construction, cache access, and operator execution, and use CUDA Graphs on compatible execution paths to reduce dynamic-routing overhead.
All speedups below are relative to the corresponding Dense baseline. “Model E2E” denotes model-level end-to-end latency.
Under matched hardware, video inputs, and output lengths, AdaDPC achieves model-level end-to-end speedups of 1.59–2.06× on 3B (Table R4). These timings cover visual encoding through completion of generation, excluding CPU preprocessing, rather than only LLM execution or isolated kernels.
Table R4. Single-request inference latency in vLLM for the 3B models. Dense denotes Qwen2.5-VL-3B. Model E2E includes visual encoding and generation but excludes CPU preprocessing.
\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM (s)} & \text{Speedup} & \text{Model E2E (s)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 1.874 & 1.00\times & 3.371 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.028} & \mathbf{1.82}\times & \mathbf{2.119} & \mathbf{1.59}\times \\\\ \hline 600 & 512 & \text{Dense} & 4.944 & 1.00\times & 7.957 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{2.269} & \mathbf{2.18}\times & \mathbf{4.457} & \mathbf{1.79}\times \\\\ \hline 1800 & 512 & \text{Dense} & 4.953 & 1.00\times & 7.973 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{2.282} & \mathbf{2.17}\times & \mathbf{4.485} & \mathbf{1.78}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 15.885 & 1.00\times & 21.923 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{6.234} & \mathbf{2.55}\times & \mathbf{10.647} & \mathbf{2.06}\times \\\\ \hline \end{array}$$ **On 7B, the corresponding model-level end-to-end speedups are 1.45–1.79×** (Table R5), using the same timing scope. **Table R5. Single-request inference latency in vLLM for the 7B models. Dense denotes Qwen2.5-VL-7B. Model E2E includes visual encoding and generation but excludes CPU preprocessing.** $$\begin{array}{rrlrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{LLM (s)} & \text{Speedup} & \text{Model E2E (s)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 3.227 & 1.00\times & 4.73 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{1.968} & \mathbf{1.64}\times & \mathbf{3.263} & \mathbf{1.45}\times \\\\ \hline 600 & 512 & \text{Dense} & 8.081 & 1.00\times & 11.092 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{4.368} & \mathbf{1.85}\times & \mathbf{6.956} & \mathbf{1.59}\times \\\\ \hline 1800 & 512 & \text{Dense} & 8.063 & 1.00\times & 11.077 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{4.354} & \mathbf{1.85}\times & \mathbf{6.941} & \mathbf{1.60}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 24.321 & 1.00\times & 30.369 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{11.773} & \mathbf{2.07}\times & \mathbf{16.937} & \mathbf{1.79}\times \\\\ \hline \end{array}$$Response to Q1 by Reviewer jEhD (2/3): Concurrent vLLM Inference and P95 TTFT/TPOT
Response to Q1 (continued): Concurrent inference.
We additionally evaluate 24 requests at concurrency C=4, each generating 32 tokens (Tables R6–R7). P95 denotes the 95th percentile, TTFT denotes time to first token, and TPOT denotes time per output token. All speedups are relative to the corresponding Dense baseline. TPOT includes generation waiting caused by concurrent prefill.
On 3B, P95 TTFT improves by 1.58–2.05× (Table R6). P95 TPOT improves by 1.23–2.03× across all tested settings.
Table R6. Concurrent inference in vLLM for the 3B models: 24 requests, C=4, and 32 generated tokens per request. TPOT includes generation waiting caused by concurrent prefill.
\hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{TTFT P95 (s)} & \text{Speedup} & \text{TPOT P95 (ms/token)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 13.407 & 1.00\times & 12.22 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{8.503} & \mathbf{1.58}\times & \mathbf{9.93} & \mathbf{1.23}\times \\\\ \hline 600 & 512 & \text{Dense} & 32.939 & 1.00\times & 550.23 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{19.129} & \mathbf{1.72}\times & \mathbf{322.9} & \mathbf{1.70}\times \\\\ \hline 1800 & 512 & \text{Dense} & 32.98 & 1.00\times & 549.17 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{19.166} & \mathbf{1.72}\times & \mathbf{327.86} & \mathbf{1.68}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 87.903 & 1.00\times & 2150.91 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{42.797} & \mathbf{2.05}\times & \mathbf{1058.4} & \mathbf{2.03}\times \\\\ \hline \end{array}$$ **On 7B, P95 TTFT improves by 1.50–1.80×** (Table R7). P95 TPOT improves by **1.31–1.78×** across all tested settings. **Table R7. Concurrent inference in vLLM for the 7B models: 24 requests, C=4, and 32 generated tokens per request. TPOT includes generation waiting caused by concurrent prefill.** $$\begin{array}{rrlrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Model} & \text{TTFT P95 (s)} & \text{Speedup} & \text{TPOT P95 (ms/token)} & \text{Speedup} \\\\ \hline 60 & 256 & \text{Dense} & 13.338 & 1.00\times & 12.12 & 1.00\times \\\\ 60 & 256 & \text{AdaDPC} & \mathbf{8.549} & \mathbf{1.56}\times & \mathbf{9.253} & \mathbf{1.31}\times \\\\ \hline 600 & 512 & \text{Dense} & 44.835 & 1.00\times & 746.55 & 1.00\times \\\\ 600 & 512 & \text{AdaDPC} & \mathbf{29.964} & \mathbf{1.50}\times & \mathbf{504.87} & \mathbf{1.48}\times \\\\ \hline 1800 & 512 & \text{Dense} & 45.295 & 1.00\times & 757.64 & 1.00\times \\\\ 1800 & 512 & \text{AdaDPC} & \mathbf{28.579} & \mathbf{1.58}\times & \mathbf{480.33} & \mathbf{1.58}\times \\\\ \hline 3600 & 1024 & \text{Dense} & 122.95 & 1.00\times & 3005.5 & 1.00\times \\\\ 3600 & 1024 & \text{AdaDPC} & \mathbf{68.397} & \mathbf{1.80}\times & \mathbf{1685.79} & \mathbf{1.78}\times \\\\ \hline \end{array}$$
Response to W1, W2 & W3 by Reviewer jEhD
Response to W1:
Thank you for raising this concern. Query-guided access is compatible with reusing cached visual states across conversation turns. The router directly scores visual-token states, while QGA uses text queries to access visual keys and values. For multi-turn conversations over the same video, the per-layer visual assignments and KV states are retained after the initial prefill and reused across turns. Subsequent turns process only the newly appended text, accessing cached visual evidence through SRP and QGA without repeating visual refinement. Crucially, unrefined visual tokens remain accessible, allowing later questions to retrieve evidence beyond the initially refined blocks.
To validate this in practice, we have implemented and evaluated multi-turn video QA in vLLM with both 3B and 7B models. As shown in Tables R1–R2, AdaDPC achieves 2.23–3.33× speedups on 3B and 2.22–3.33× on 7B over Dense across 256–1,024 sampled frames. These measurements provide direct evidence that AdaDPC supports multi-turn video conversations while retaining substantial runtime benefits.
Table R1 reports the 3B multi-turn latency.
Table R1. Multi-turn video QA latency in vLLM for the 3B models. Dense denotes Qwen2.5-VL-3B.
\hline \text{Video (s)} & \text{Frames} & \text{Dense (s)} & \text{AdaDPC (s)} & \text{Speedup} \\\\ \hline 60 & 256 & 11.118 & \mathbf{4.98} & \mathbf{2.23}\times \\\\ 600 & 512 & 28.307 & \mathbf{10.372} & \mathbf{2.73}\times \\\\ 1800 & 512 & 28.254 & \mathbf{10.566} & \mathbf{2.67}\times \\\\ 3600 & 1024 & 86.825 & \mathbf{26.044} & \mathbf{3.33}\times \\\\ \hline \end{array}$$ Table R2 reports the corresponding 7B multi-turn latency. **Table R2. Multi-turn video QA latency in vLLM for the 7B models. Dense denotes Qwen2.5-VL-7B.** $$\begin{array}{rrrrr} \hline \text{Video (s)} & \text{Frames} & \text{Dense (s)} & \text{AdaDPC (s)} & \text{Speedup} \\\\ \hline 60 & 256 & 17.78 & \mathbf{7.998} & \mathbf{2.22}\times \\\\ 600 & 512 & 43.576 & \mathbf{16.022} & \mathbf{2.72}\times \\\\ 1800 & 512 & 43.527 & \mathbf{15.994} & \mathbf{2.72}\times \\\\ 3600 & 1024 & 127.179 & \mathbf{38.139} & \mathbf{3.33}\times \\\\ \hline \end{array}$$ --- > W2: The method relies on a complicated combination of SRP, QGA, PF, routing, scheduling, and gating. **Response to W2:** Thank you for this comment. AdaDPC is a compact dual-path architecture, rather than a combination of six independent methods. Its methodological novelty is to decouple visual refinement from evidence access: visual states remain readable even when costly updates are skipped, and relevance discovered during access guides subsequent computation. Both paths share visual states and Q/K/V projections, while routing, scheduling, gating, and feedback are lightweight controls within this same architecture. --- > W3: The evaluation relies on the relatively dated Qwen2.5-VL-3B backbone. The paper also does not verify whether the method remains effective on larger models. **Response to W3:** Thank you for this comment. We have extended our evaluation to the larger **Qwen2.5-VL-7B backbone**. As shown in Table R3, **AdaDPC consistently outperforms the corresponding Dense baseline on all eight benchmarks, reproducing the same positive trend observed with the 3B backbone.** These results support AdaDPC’s effectiveness beyond the original 3B setting. **Table R3. Results on the Qwen2.5-VL-7B backbone. Dense denotes the original Qwen2.5-VL-7B baseline. Bold indicates the highest score in each column.** $$\begin{array}{lrrrrrrrr} \hline \text{Model} & \text{LVBench} & \text{LongVideoBench} & \text{MLVU Dev} & \text{VideoMME} & \text{MVBench} & \text{VSI-Bench} & \text{TOMATO} & \text{TimeScope} \\\\ \hline \text{Dense} & 45.3 & 60.7 & 68.3 & 65.1 & 69.6 & 35.6 & 25.6 & 81.0 \\\\ \text{Dense + SFT} & 45.2 & 59.8 & 68.1 & 64.7 & 67.0 & 36.7 & 26.2 & 81.9 \\\\ \hline \text{FastV} & 43.3 & 60.8 & 65.7 & 66.0 & 67.0 & 39.0 & 23.5 & 79.3 \\\\ \text{ToMe} & 43.1 & 60.8 & 66.1 & 65.8 & 67.2 & 38.7 & 23.0 & 80.2 \\\\ \text{MoBA} & 42.3 & 58.3 & 64.7 & 63.4 & 67.5 & 34.8 & 23.5 & 78.8 \\\\ \text{FrameFusion} & 42.8 & 61.1 & 66.3 & \mathbf{66.1} & 66.8 & 39.3 & 22.2 & 81.9 \\\\ \hline \text{AdaDPC} & \mathbf{46.4} & \mathbf{62.8} & \mathbf{71.3} & 65.8 & \mathbf{70.4} & \mathbf{40.6} & \mathbf{28.6} & \mathbf{84.2} \\\\ \hline \end{array}$$