- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces LEAP which is a framework for long form AV QA that addresses the difficulty of processing hour scale recordings within limited model context windows.
Critical strengths
- LEAP addresses an important challenge in long video understanding, where limited context windows make it difficult to preserve fine grained visual and acoustic information across hour scale recordings.
Critical weaknesses
-
I see limited methodological differentiation and insufficient comparison with closely related retrieval frameworks. The central idea of identifying relevant temporal regions before performing detailed multimodal reasoning is already established in prior long video understanding research. In particular, Magnet (Chowdhury et al., NeurIPS 2025) explores multimodal retrieval, temporal localization, selective visual processing and evidence aggregation for grounded question answering. It remains unclear how substantially the design choices of LEAP advance beyond existing hierarchical and multimodal retrieval approaches. Even though the paper includes several retrieval ablations, it does not provide a direct comparison with Magnet or a sufficiently detailed discussion distinguishing the two approaches. Since learned evidence localization is central to the proposed method, stronger comparisons with related retrieval systems would help establish the novelty and significance of the contribution.
-
A central motivation of LEAP is scalable processing of long audio visual recordings. However, while the method maintains bounded peak context and memory it still performs a localization pass over every temporal block for each question. So the total retrieval computation grows with recording duration, even though the answer context remains fixed. The paper reports substantial localization token reductions and latency improvements for transcript based retrieval, but the overall computational tradeoff relative to alternative retrieval methods is less clear. More so, the impact of repeated media processing, transcript generation and question dependent localization on end to end latency and computational cost requires further examination. Without a comprehensive comparison under matched computational budgets, it is difficult to assess whether LEAP offers a meaningful efficiency advantage over existing retrieval based approaches.
Questions for authors
-
How does LEAP differ fundamentally from existing multimodal evidence retrieval approaches such as Magnet (Chowdhury et al., 2025), which also combines audio visual retrieval, temporal localization, selective processing and evidence based reasoning? In particular, what advantages arise specifically from the supervised window localization objective and fixed block selection strategy? The authors should ideally provide a more detailed comparison with Magnet and an experimental comparison under matched backbone and computational settings.
-
What is the end to end computational cost, including total processed tokens, latency, media encoding and transcript preprocessing, across different recording lengths? How does this compare with existing retrieval based methods under similar computational budgets, particularly when multiple questions are asked about the same recording?
-
LEAP retains 3 blocks and 3 windows per selected block regardless of recording duration or question complexity. How does the method perform when answering requires evidence distributed across more than 3 distinct temporal blocks or across many distant events? Could the authors analyze accuracy and evidence coverage as a function of the number and temporal distribution of required evidence segments? This would help clarify whether the fixed retrieval budget limits performance on more complex long duration reasoning tasks.
8 comments
W2 + Q2
[...] while the method maintains bounded peak context and memory [...] the total retrieval computation grows with recording duration [...]
What is the end to end computational cost, including total processed tokens, latency, media encoding and transcript preprocessing, across different recording lengths? [...] particularly when multiple questions are asked about the same recording?
A2: Thank you for separating per-pass cost from total cost. The paper's bound is on per-pass context and peak memory (§3.5). LEAP's total work grows with duration by one bounded pass per 600 s block. A whole-clip pass (32 frames plus full audio) grows too, at about 47k tokens per hour, and stops at the position limit near 81 minutes, while LEAP's passes continue to any length.
Per question (medians). Cumulative prefill is 2.2–2.4× one whole-clip pass where one exists; the cached row counts each block prefix once per video (Several questions on one recording below). Times are query-to-answer on one H100 with media decoded fresh (Table 2).
| TraceAV | LVOmniBench | VideoOdyssey | MMOU | |
|---|---|---|---|---|
| Cumulative prefill (k tokens) | 67.4 | 62.3 | 148.1 | 20.9 |
| Cumulative prefill, cached (k tokens) | 38.8 | 37.7 | 42.5 | 17.4 |
| Time, LEAP (s) | 12.4 | 15.6 | 30.0 | 5.5 |
Across recording lengths. In a separate timing on one H100 NVL†, one question at a time with media decoded live, LEAP's median time on VideoOdyssey grows with the number of blocks, from 19 s near 1 h to 42 s at 2.6–3.5 h, and its peak allocated memory stays at or below 85.5 GB. In the same timing, TraceAV's official recipe did not complete 19 of the 48 sampled questions on one card, all on recordings of 35 minutes or more, while LEAP answered all 48 on one card. Where both complete, LEAP's median time stays within a few seconds of the official recipes on TraceAV, LVOmniBench and MMOU.
Several questions on one recording. Each block pass places the media before the question, so a block's media prefix is identical across questions and can be cached per video. On VideoOdyssey, at 10.6 questions per video, caching cuts cumulative prefill to 0.29× (table above). Served from a vLLM prefix cache on four GPUs, the localization of a video's later questions runs 1.6× faster than without it; the cache takes about 5.4 GiB of GPU memory per hour of video.
Preprocessing. Transcription runs once per recording, before any question: Whisper large-v3 takes about 3.3 minutes per recording hour on one H100. Media decoding is included in every per-question time above.
W2 (matched budgets)
Without a comprehensive comparison under matched computational budgets, it is difficult to assess whether LEAP offers a meaningful efficiency advantage over existing retrieval based approaches.
A3: We appreciate the push for matched budgets. LEAP significantly leads, on TraceAV, LVOmniBench and MMOU, the whole clip with full audio matched to it in frames (288, LEAP's frame ceiling; App. B.4) or in total tokens† (as many frames as fit LEAP's cumulative prefill on each question, capped at 60k, with the same outline and answer adapter). It also leads significantly on TraceAV and MMOU where both read the same number of tokens:
| Benchmark | LEAP − token-matched whole clip (pp) |
|---|---|
| TraceAV | +4.27 [+2.66, +6.02] |
| LVOmniBench | +3.75 [+1.12, +6.45] |
| MMOU | +1.86 [+1.04, +2.69] |
| TraceAV, same token count | +2.14 [+0.33, +3.99] |
| MMOU, same token count | +1.34 [+0.51, +2.21] |
The context-filled montage packs 32 frames and 10 s audio slices into one pass of up to 60k tokens, more than LEAP's answer pass; with the answer adapter on both, LEAP leads it significantly on VideoOdyssey in every quintile of the montage's audio coverage (Fig. 15). At the same answer budget, our trained score's four-benchmark mean accuracy is significantly above ImageBind's and SigLIP's (Table 11, A1).
Q3
How does the method perform when answering requires evidence distributed across more than 3 distinct temporal blocks or across many distant events? [...]
A4: Thank you for asking about evidence spread across many blocks. Where the annotated evidence touches four or more distinct 600 s blocks, LEAP still leads the 32-frame whole clip given the same outline and answer adapter on TraceAV, by a significant 8.1 pp†.
The three-block budget is a hyperparameter, set by the length of one answer pass rather than by the method: a fourth block fits once that pass is widened by a third (Fig. 7), and each pass stays bounded however long the recording. On questions whose evidence touches four or more blocks, a fourth block raises the share of annotated evidence seconds covered†:
| Benchmark | Coverage gain (pp) |
|---|---|
| TraceAV | +8.8 |
| VideoOdyssey | +3.5 |
The answer pass also reads a transcript outline of the whole recording, and its gain is largest where the windows miss the evidence (MMOU 48.0 → 55.9 on those questions, Table 14).
Thank You for Your Insightful Suggestions
Dear Reviewer,
Thank you so much for your insightful suggestions! In our responses, we have addressed each of your concerns:
- Differences from MAGNET and a direct comparison (W1, Q1). The revised PDF cites MAGNET in §2; we set out three structural differences, and in Table 11 our trained window score significantly outperforms MAGNET's retrieval signal, ImageBind audio-visual similarity, in evidence coverage (71.7% vs. 56.2% on VideoOdyssey) and accuracy, while the supervised objective and the block ranking each significantly raise the evidence retained there.
- End-to-end cost across recording lengths and for several questions per recording (W2, Q2). We reported tokens, time, media decoding and transcription per question and across recording lengths: each 600 s block adds one bounded pass, transcription runs once per recording, and caching each block's media prefix across a recording's questions cuts cumulative prefill to 0.29× on VideoOdyssey.
- Comparison under matched computational budgets (W2). With the same outline and answer adapter, LEAP significantly leads the whole clip with full audio matched to its total tokens per question (TraceAV +4.27 pp), and at the same answer budget our trained score's mean accuracy is significantly above that of the ImageBind and SigLIP rankings (Table 11).
- Evidence spread across more than three blocks (Q3). Where the annotated evidence touches four or more 600 s blocks, LEAP still significantly leads the 32-frame whole clip on TraceAV (+8.1 pp); the three-block budget is a hyperparameter bounded by one answer pass, and a fourth block raises evidence coverage on these questions.
Could you kindly let us know whether our responses have addressed your concerns? If you have any further questions, please let us know, and we will address them.
Thank you again for your time and valuable feedback!
We thank the reviewer for engaging with how LEAP relates to multimodal retrieval frameworks and with its end-to-end cost.
Conventions. pp = percentage points. Baselines are named as in Table 6. A paired difference is read on the same questions as LEAP-side minus comparator, with a two-sided 95% bootstrap interval resampled over source videos (at least 10,000 draws); significant means the interval excludes zero. Denominators are the full official question sets with failures scored wrong. TraceAV accuracy is over all questions unless marked general (the mean of its twelve general sub-tasks used in Table 1). Results marked † are new in this rebuttal.
W1 + Q1
A1: Thank you for the pointer; we will cite MAGNET. The two systems retrieve under different constraints. MAGNET (AVHaystacksQA) searches a collection of up to 500 videos of 5–25 minutes: it retrieves the top-k videos by ImageBind similarity, runs one Qwen2.5-Omni agent per retrieved video, and a GPT-4o meta-agent aggregates their outputs into free-form step-wise answers scored by BLEU and GPT judgment. LEAP searches inside one recording of up to four hours that does not fit the backbone's context, and answers multiple-choice questions.
Three structural differences follow. (i) Where the context bound holds. MAGNET assigns one agent to each retrieved video of 5–25 minutes; every LEAP pass reads one 600 s block or at most nine 75 s windows, at any recording length. (ii) What scores relevance. MAGNET ranks with an external embedding model; LEAP's score is the logit the answering backbone itself, under a localization adapter, assigns to each window's letter when a block's windows are offered as lettered options, read jointly over native-rate audio and frames, so ranking adds no model and no pass. (iii) Where the answer is formed. LEAP answers in one bounded pass of the same backbone over all retained windows; MAGNET aggregates per-video agent outputs in a separate meta-agent, GPT-4o in its stated configuration.
MAGNET retrieves by ImageBind audio-visual similarity, so we added an ImageBind audio-visual ranking variant to Table 11†: the frame–question and audio–question cosines averaged at each 2 s step, each window scored by its best step and each block by its best window, with the same candidate grid, block and window caps, and answerer as the other rows. Final-window coverage is the share of questions with annotated evidence in an answer-pass window.
Our trained score covers significantly more evidence than the ImageBind ranking on VideoOdyssey and MMOU and answers significantly more accurately on both. It also answers MMOU significantly more accurately than SigLIP and bge-m3, and its four-benchmark mean accuracy is significantly above those of ImageBind and SigLIP.
What the supervised objective buys. Over the base selector (the same selection run by the base model, no adapter mounted) on the same grid, it significantly raises final-window coverage, by 10.94 pp pooled and 16.57 pp on VideoOdyssey. With the answer adapter (the LoRA that answers from the retained windows) and the transcript outline held fixed†, it raises accuracy significantly on VideoOdyssey and MMOU.
What the block ranking buys. On VideoOdyssey, where a median question takes 13 passes, ranking retains significantly more annotated evidence than equally spaced blocks, by 23.07 pp (App. C.6), and answers significantly more accurately, by 3.20 pp.