- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DAES, a training-free framework for selecting video evidence under a fixed visual-token budget. Instead of selecting complete frames, it separates each frame into ordinary visual and visual-text evidence using reflection-based cues. It estimates option-conditioned relevance in an MLLM’s Value space, calibrates scores using evidence-specific null distributions, and selects evidence across answer options and timestamps. Evaluation on subtitle-rendered Video-MME-v2 and LongVideoBench shows improvements over frame-selection baselines. The paper also includes component ablations, an original-video experiment, and transfer to different answer models.
Critical strengths
- A well-motivated alternative to whole-frame selection. Selecting visual-text and ordinary visual evidence independently allows the model to retain different information at different timestamps without paying for every complete frame. The type-specific calibration addresses the problem that raw relevance scores for these evidence types may not be directly comparable. Together, these choices form a coherent approach to allocating limited visual context, rather than simply increasing the number of input frames.
- Meaningful gains with useful supporting experiments. With the same Qwen3.5-9B answerer and comparable visual-token budgets, DAES improves over the strongest reported frame-selection baseline by 4.63 percentage points on Video-MME-v2 and 2.19 points on LongVideoBench. The component ablations provide evidence that the main design choices help. Reusing the selected evidence with Qwen3.5-27B and InternVL3.5-8B also produces gains, suggesting that the selected evidence is useful beyond a single answerer.
Critical weaknesses
- The benefit of V/T decoupling and its broader applicability are not sufficiently established. The Full32 ablation changes both V/T decomposition and selection granularity, so it does not establish whether explicit V/T separation is better than simpler token selection. The main evaluation also uses subtitle-rendered multiple-choice videos, while the original-video experiment compares only against uniform sampling. The Q-only ablation does not evaluate open-ended answering. Please evaluate a clearly specified question-only adaptation on VideoEval-Pro, an open-ended long-video QA benchmark. Include matched full-frame and type-agnostic token-selection baselines, controlling the selector, relevance computation, candidate pool, answerer, and token budget. This would clarify whether V/T decoupling itself remains useful without constructed subtitles or predefined answer options.
- Evidence selection may depend on how answer options are written and ordered. DAES computes option-specific relevance and cycles through options when selecting evidence. However, evidence matching a distractor may not help distinguish the correct answer, potentially wasting budget or making selection sensitive to option wording and order. Please test meaning-preserving distractor paraphrases and option permutations only in the selector input, keeping the video, question, and answerer inputs fixed. Reporting changes in selected evidence and final QA performance would establish whether the gains are robust to these irrelevant changes in presentation.
- The practical efficiency trade-off is unclear. Matching the answerer’s visual-token budget does not account for DAES’s additional processing of original and reflected frames or its option-conditioned scoring. Covering more timestamps may also increase answerer-side visual encoding before masking. Without end-to-end measurements, it is unclear whether the accuracy gains justify the additional computation. Please report latency and peak memory against the main baselines on the same hardware, separating preprocessing, evidence selection, and answering. Distinguish one-time calibration and reusable per-video work from per-question costs, including the case where multiple questions share a video.
Questions for authors
See weakness.
6 comments
W2. Sensitivity to option order and distractor wording
Tables 4(b) and 22 report the results under randomized option order and meaning-preserving distractor paraphrases. Importantly, these perturbations are applied only to the selector inputs, while the answerer inputs remain unchanged, allowing us to isolate their effects on evidence selection.
Appendix B.3 provides representative paraphrases, including active/passive reformulations (e.g., "The boy is hugged by his sister" to "His sister hugs the boy"), synonym substitutions (e.g., "closes the door" to "shuts the door"), and minor word-order changes (e.g., "A crisper and drier texture" to "A drier and crisper texture"). These transformations preserve the original meaning while changing the surface wording.
| Setting | Accuracy (%) | Change (pp) | Patch Overlap (%) | Timestamp Overlap (%) |
|---|---|---|---|---|
| Original DAES | 40.88 | — | — | — |
| Randomized option order | 40.72 | -0.16 | 95.25 | 95.79 |
| Distractor paraphrases | 40.59 | -0.29 | 89.32 | 88.96 |
The accuracy drops are small (0.16 and 0.29 pp), while the selected patches and timestamps maintain high overlap with the original DAES selections. These results demonstrate that DAES is robust to option-order and distractor-wording perturbations, both in terms of final QA performance and evidence selection stability.
W3. End-to-end efficiency, memory, and reusable computation
We have added a comprehensive efficiency comparison in Table 7 of the revised manuscript. All methods are evaluated on the same RTX PRO 6000 Blackwell (96 GB), using the BF16 Qwen3.5-9B answerer with batch size one. We report the latency of three stages: Selector (including selector-side visual encoding and selection), Vision (answerer-side visual encoding), and LLM (prefill and decoding). For end-to-end latency, Q-Level measures independent per-question inference, while V-Level amortizes shared video features across four questions from the same video.
| Method | Acc. (%) | Selector (s/Q) | Vision (s/Q) | LLM (s/Q) | Q-Level E2E (s/Q) | V-Level E2E (s/Q) | Peak Mem. (GB) |
|---|---|---|---|---|---|---|---|
| Uniform32 | 32.16 | 0.000 | 0.210 | 1.347 | 2.204 | 1.598 | 21.20 |
| AKS | 35.00 | 2.215 | 0.200 | 1.368 | 9.857 | 4.041 | 25.64 |
| Q-Frame | 34.66 | 0.703 | 0.489 | 1.350 | 5.113 | 2.592 | 25.16 |
| WFS-SB | 36.00 | 2.905 | 0.467 | 1.365 | 5.634 | 4.785 | 25.96 |
| VTR-VLM | 36.25 | 4.643 | 0.174 | 1.369 | 20.654 | 6.386 | 43.67 |
| QCA | 35.28 | 2.215 | 0.207 | 1.369 | 9.907 | 4.075 | 25.64 |
| FlashVID | 37.47 | 1.597 | 0.000 | 1.326 | 8.090 | 3.247 | 26.01 |
| V-CAST | 32.31 | 1.405 | 0.000 | 1.314 | 7.311 | 3.019 | 24.43 |
| Unified Patch | 38.56 | 1.515 | 1.139 | 1.391 | 11.344 | 4.244 | 26.81 |
| DAES (2B) | 40.88 | 1.667 | 0.514 | 1.335 | 8.471 | 3.561 | 27.59 |
DAES achieves the highest accuracy among the compared methods, with competitive end-to-end latency and moderate memory consumption. Its broader temporal coverage (59.34 vs. 32 timestamps per question) incurs additional answerer-side visual encoding compared with Uniform32 (0.514 vs. 0.210 s/Q), but contributes to an 8.72 pp accuracy improvement under the same visual-token budget.
Compared with Unified Patch, DAES further improves accuracy by 2.32 pp while reducing answerer-side visual encoding time from 1.139 to 0.514 s/Q. This is consistent with its more selective temporal coverage (59.34 vs. 255.77 timestamps per question). Despite slightly higher selector latency, DAES achieves lower end-to-end latency at both Q-Level (8.471 vs. 11.344 s/Q) and V-Level (3.561 vs. 4.244 s/Q), with comparable peak memory consumption.
Regarding reusable computation, video-level feature sharing reduces DAES's amortized latency from 8.471 to 3.561 s/Q, while query-dependent selection and answer generation remain independent. These results provide a comprehensive evaluation of DAES's accuracy–efficiency trade-off.
Response
Thank you for the detailed rebuttal and additional experiments. The new results address some of my original concerns, and I have three remaining clarification questions:
-
For VideoEval-Pro setup, were the videos evaluated in their original form without additional subtitle rendering? Did all methods use the same visual-token budget?
-
For the original-video and VideoEval-Pro experiments, were the null distributions reused from the main experiments or constructed separately? If reconstructed, what data were used for calibration?
-
In Table 7, the sum of Selector, Vision, and LLM latency is much smaller than the Q-Level end-to-end latency. What accounts for this difference?
Overall, I appreciate the authors' efforts to address my concerns. These clarifications would help me better assess the new results.
W1. V/T decoupling, original-video inputs, and open-ended QA
Controlling for patch-level selection and answer options.
We agree that the original Full32 ablation changes both evidence typing and selection granularity. We therefore introduce Unified Patch, which ranks individual patches by option-conditioned relevance without V/T separation, using the same model configuration and visual-token budget (Tables 19–20).
DAES consistently outperforms Unified Patch on both benchmarks, demonstrating the benefit of V/T decoupling beyond generic patch-level selection.
We further conduct a controlled ablation on VideoEval-Pro to examine whether this benefit depends on predefined answer options. Unlike multiple-choice QA, evidence selection here is guided solely by the question, without candidate answer options. All methods use the same Qwen3.5-2B selector and Qwen3.5-9B answerer.
DAES further improves over Unified Patch by 1.63 pp, showing that the benefit of V/T decoupling persists without predefined answer options. Together, these results support the contribution of V/T decoupling beyond both generic patch-level selection and option-conditioned relevance estimation.
Original RGB with competitive baselines.
We further extend the original-video evaluation to multiple baselines at 16, 32, and 64 FFE (Table 3). Results at 32 FFE are summarized below:
Original RGB inputs contain no additionally rendered subtitles, although naturally occurring scene text and embedded subtitles remain. DAES outperforms the strongest competing baseline by 0.93 pp on Video-MME-v2 and 1.35 pp on LVB. These results demonstrate that its benefits also extend to original video inputs, although the gains are smaller than with subtitle rendering.
Open-ended QA.
DAES mainly focuses on decoupling Visual and Textual Evidence and adaptively selecting relevant patches based on Query–Patch relevance in the MLLM Value space. Since evidence selection can be guided directly by the question, it does not inherently require predefined answer options. We evaluate this capability on VideoEval-Pro. The model receives only the video and question and generates free-form answers, evaluated against reference answers using GPT-6.1 Sol for semantic correctness.
DAES outperforms Q-Frame by 3.10 pp and Uniform32 by 12.95 pp, demonstrating its effectiveness in open-ended video QA.