acceptodds
Under review as a conference paper at ICLR 2027

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

Abstract

Hour-scale audio-visual question answering is constrained by a context dilemma: dense whole-recording encoding rapidly exhausts context limits, whereas uniform temporal compression severely dilutes fine-grained acoustic and visual evidence. We introduce LEAP, a framework where the model retrieves its own evidence without placing the whole recording in one context. LEAP divides a recording into fixed-duration blocks, applying a lightweight localization pass to each block to score short candidate windows. The highest-ranked windows are pooled and re-encoded in a single bounded answer pass. Consequently, the answer input and peak context remain independent of the recording duration. By decoupling evidence localization from reasoning, our framework can localize candidate temporal windows over pre-computed transcripts without decoding media frames, while preserving fine-grained visual and non-speech evidence by routing the final answering pass over raw audio-visual streams. LEAP trains both stages: a localization LoRA improves the selected windows, and an answer LoRA improves the answers read from the same windows. The block grid natively supports causal queries, enabling LEAP to support streaming inference without streaming-specific training. Across several AVQA benchmarks, LEAP improves over the Qwen3-Omni-30B-A3B baseline by 4.5–16.8%, and transfers to a second omni-modal backbone, MiniCPM-o 4.5, surpassing its published results by 3.1–13.0%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Form Audio-Visual Question Answering, Temporal Evidence Localization, Memory-Efficient Video Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hzaeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LEAP which is a framework for long form AV QA that addresses the difficulty of processing hour scale recordings within limited model context windows.

Critical strengths
  • LEAP addresses an important challenge in long video understanding, where limited context windows make it difficult to preserve fine grained visual and acoustic information across hour scale recordings.
Critical weaknesses
  • I see limited methodological differentiation and insufficient comparison with closely related retrieval frameworks. The central idea of identifying relevant temporal regions before performing detailed multimodal reasoning is already established in prior long video understanding research. In particular, Magnet (Chowdhury et al., NeurIPS 2025) explores multimodal retrieval, temporal localization, selective visual processing and evidence aggregation for grounded question answering. It remains unclear how substantially the design choices of LEAP advance beyond existing hierarchical and multimodal retrieval approaches. Even though the paper includes several retrieval ablations, it does not provide a direct comparison with Magnet or a sufficiently detailed discussion distinguishing the two approaches. Since learned evidence localization is central to the proposed method, stronger comparisons with related retrieval systems would help establish the novelty and significance of the contribution.

  • A central motivation of LEAP is scalable processing of long audio visual recordings. However, while the method maintains bounded peak context and memory it still performs a localization pass over every temporal block for each question. So the total retrieval computation grows with recording duration, even though the answer context remains fixed. The paper reports substantial localization token reductions and latency improvements for transcript based retrieval, but the overall computational tradeoff relative to alternative retrieval methods is less clear. More so, the impact of repeated media processing, transcript generation and question dependent localization on end to end latency and computational cost requires further examination. Without a comprehensive comparison under matched computational budgets, it is difficult to assess whether LEAP offers a meaningful efficiency advantage over existing retrieval based approaches.

Questions for authors
  • How does LEAP differ fundamentally from existing multimodal evidence retrieval approaches such as Magnet (Chowdhury et al., 2025), which also combines audio visual retrieval, temporal localization, selective processing and evidence based reasoning? In particular, what advantages arise specifically from the supervised window localization objective and fixed block selection strategy? The authors should ideally provide a more detailed comparison with Magnet and an experimental comparison under matched backbone and computational settings.

  • What is the end to end computational cost, including total processed tokens, latency, media encoding and transcript preprocessing, across different recording lengths? How does this compare with existing retrieval based methods under similar computational budgets, particularly when multiple questions are asked about the same recording?

  • LEAP retains 3 blocks and 3 windows per selected block regardless of recording duration or question complexity. How does the method perform when answering requires evidence distributed across more than 3 distinct temporal blocks or across many distant events? Could the authors analyze accuracy and evidence coverage as a function of the number and temporal distribution of required evidence segments? This would help clarify whether the fixed retrieval budget limits performance on more complex long duration reasoning tasks.

8 comments

We thank the reviewer for engaging with how LEAP relates to multimodal retrieval frameworks and with its end-to-end cost.

Conventions. pp = percentage points. Baselines are named as in Table 6. A paired difference is read on the same questions as LEAP-side minus comparator, with a two-sided 95% bootstrap interval resampled over source videos (at least 10,000 draws); significant means the interval excludes zero. Denominators are the full official question sets with failures scored wrong. TraceAV accuracy is over all questions unless marked general (the mean of its twelve general sub-tasks used in Table 1). Results marked † are new in this rebuttal.

W1 + Q1

In particular, Magnet (Chowdhury et al., NeurIPS 2025) explores multimodal retrieval, temporal localization, selective visual processing and evidence aggregation for grounded question answering. [...] it does not provide a direct comparison with Magnet or a sufficiently detailed discussion distinguishing the two approaches.

How does LEAP differ fundamentally from existing multimodal evidence retrieval approaches such as Magnet [...]? In particular, what advantages arise specifically from the supervised window localization objective and fixed block selection strategy? The authors should ideally provide a more detailed comparison with Magnet and an experimental comparison under matched backbone and computational settings.

A1: Thank you for the pointer; we will cite MAGNET. The two systems retrieve under different constraints. MAGNET (AVHaystacksQA) searches a collection of up to 500 videos of 5–25 minutes: it retrieves the top-k videos by ImageBind similarity, runs one Qwen2.5-Omni agent per retrieved video, and a GPT-4o meta-agent aggregates their outputs into free-form step-wise answers scored by BLEU and GPT judgment. LEAP searches inside one recording of up to four hours that does not fit the backbone's context, and answers multiple-choice questions.

Three structural differences follow. (i) Where the context bound holds. MAGNET assigns one agent to each retrieved video of 5–25 minutes; every LEAP pass reads one 600 s block or at most nine 75 s windows, at any recording length. (ii) What scores relevance. MAGNET ranks with an external embedding model; LEAP's score is the logit the answering backbone itself, under a localization adapter, assigns to each window's letter when a block's windows are offered as lettered options, read jointly over native-rate audio and frames, so ranking adds no model and no pass. (iii) Where the answer is formed. LEAP answers in one bounded pass of the same backbone over all retained windows; MAGNET aggregates per-video agent outputs in a separate meta-agent, GPT-4o in its stated configuration.

MAGNET retrieves by ImageBind audio-visual similarity, so we added an ImageBind audio-visual ranking variant to Table 11†: the frame–question and audio–question cosines averaged at each 2 s step, each window scored by its best step and each block by its best window, with the same candidate grid, block and window caps, and answerer as the other rows. Final-window coverage is the share of questions with annotated evidence in an answer-pass window.

Ranking score (Table 11)Coverage, VideoOdyssey (%)Coverage, MMOU (%)Accuracy, four-benchmark mean (%)
Our trained score71.794.852.28
ImageBind, audio-visual†56.288.549.80
SigLIP, frames65.888.950.38
bge-m3, transcript58.988.351.18

Our trained score covers significantly more evidence than the ImageBind ranking on VideoOdyssey and MMOU and answers significantly more accurately on both. It also answers MMOU significantly more accurately than SigLIP and bge-m3, and its four-benchmark mean accuracy is significantly above those of ImageBind and SigLIP.

What the supervised objective buys. Over the base selector (the same selection run by the base model, no adapter mounted) on the same grid, it significantly raises final-window coverage, by 10.94 pp pooled and 16.57 pp on VideoOdyssey. With the answer adapter (the LoRA that answers from the retained windows) and the transcript outline held fixed†, it raises accuracy significantly on VideoOdyssey and MMOU.

What the block ranking buys. On VideoOdyssey, where a median question takes 13 passes, ranking retains significantly more annotated evidence than equally spaced blocks, by 23.07 pp (App. C.6), and answers significantly more accurately, by 3.20 pp.

W2 + Q2

[...] while the method maintains bounded peak context and memory [...] the total retrieval computation grows with recording duration [...]

What is the end to end computational cost, including total processed tokens, latency, media encoding and transcript preprocessing, across different recording lengths? [...] particularly when multiple questions are asked about the same recording?

A2: Thank you for separating per-pass cost from total cost. The paper's bound is on per-pass context and peak memory (§3.5). LEAP's total work grows with duration by one bounded pass per 600 s block. A whole-clip pass (32 frames plus full audio) grows too, at about 47k tokens per hour, and stops at the position limit near 81 minutes, while LEAP's passes continue to any length.

Per question (medians). Cumulative prefill is 2.2–2.4× one whole-clip pass where one exists; the cached row counts each block prefix once per video (Several questions on one recording below). Times are query-to-answer on one H100 with media decoded fresh (Table 2).

TraceAVLVOmniBenchVideoOdysseyMMOU
Cumulative prefill (k tokens)67.462.3148.120.9
Cumulative prefill, cached (k tokens)38.837.742.517.4
Time, LEAP (s)12.415.630.05.5

Across recording lengths. In a separate timing on one H100 NVL†, one question at a time with media decoded live, LEAP's median time on VideoOdyssey grows with the number of blocks, from 19 s near 1 h to 42 s at 2.6–3.5 h, and its peak allocated memory stays at or below 85.5 GB. In the same timing, TraceAV's official recipe did not complete 19 of the 48 sampled questions on one card, all on recordings of 35 minutes or more, while LEAP answered all 48 on one card. Where both complete, LEAP's median time stays within a few seconds of the official recipes on TraceAV, LVOmniBench and MMOU.

Several questions on one recording. Each block pass places the media before the question, so a block's media prefix is identical across questions and can be cached per video. On VideoOdyssey, at 10.6 questions per video, caching cuts cumulative prefill to 0.29× (table above). Served from a vLLM prefix cache on four GPUs, the localization of a video's later questions runs 1.6× faster than without it; the cache takes about 5.4 GiB of GPU memory per hour of video.

Preprocessing. Transcription runs once per recording, before any question: Whisper large-v3 takes about 3.3 minutes per recording hour on one H100. Media decoding is included in every per-question time above.

W2 (matched budgets)

Without a comprehensive comparison under matched computational budgets, it is difficult to assess whether LEAP offers a meaningful efficiency advantage over existing retrieval based approaches.

A3: We appreciate the push for matched budgets. LEAP significantly leads, on TraceAV, LVOmniBench and MMOU, the whole clip with full audio matched to it in frames (288, LEAP's frame ceiling; App. B.4) or in total tokens† (as many frames as fit LEAP's cumulative prefill on each question, capped at 60k, with the same outline and answer adapter). It also leads significantly on TraceAV and MMOU where both read the same number of tokens:

BenchmarkLEAP − token-matched whole clip (pp)
TraceAV+4.27 [+2.66, +6.02]
LVOmniBench+3.75 [+1.12, +6.45]
MMOU+1.86 [+1.04, +2.69]
TraceAV, same token count+2.14 [+0.33, +3.99]
MMOU, same token count+1.34 [+0.51, +2.21]

The context-filled montage packs 32 frames and 10 s audio slices into one pass of up to 60k tokens, more than LEAP's answer pass; with the answer adapter on both, LEAP leads it significantly on VideoOdyssey in every quintile of the montage's audio coverage (Fig. 15). At the same answer budget, our trained score's four-benchmark mean accuracy is significantly above ImageBind's and SigLIP's (Table 11, A1).

Q3

How does the method perform when answering requires evidence distributed across more than 3 distinct temporal blocks or across many distant events? [...]

A4: Thank you for asking about evidence spread across many blocks. Where the annotated evidence touches four or more distinct 600 s blocks, LEAP still leads the 32-frame whole clip given the same outline and answer adapter on TraceAV, by a significant 8.1 pp†.

The three-block budget is a hyperparameter, set by the length of one answer pass rather than by the method: a fourth block fits once that pass is widened by a third (Fig. 7), and each pass stays bounded however long the recording. On questions whose evidence touches four or more blocks, a fourth block raises the share of annotated evidence seconds covered†:

BenchmarkCoverage gain (pp)
TraceAV+8.8
VideoOdyssey+3.5

The answer pass also reads a transcript outline of the whole recording, and its gain is largest where the windows miss the evidence (MMOU 48.0 → 55.9 on those questions, Table 14).

AuthorsOpenReview ↗

Thank You for Your Insightful Suggestions

Dear Reviewer,

Thank you so much for your insightful suggestions! In our responses, we have addressed each of your concerns:

  1. Differences from MAGNET and a direct comparison (W1, Q1). The revised PDF cites MAGNET in §2; we set out three structural differences, and in Table 11 our trained window score significantly outperforms MAGNET's retrieval signal, ImageBind audio-visual similarity, in evidence coverage (71.7% vs. 56.2% on VideoOdyssey) and accuracy, while the supervised objective and the block ranking each significantly raise the evidence retained there.
  2. End-to-end cost across recording lengths and for several questions per recording (W2, Q2). We reported tokens, time, media decoding and transcription per question and across recording lengths: each 600 s block adds one bounded pass, transcription runs once per recording, and caching each block's media prefix across a recording's questions cuts cumulative prefill to 0.29× on VideoOdyssey.
  3. Comparison under matched computational budgets (W2). With the same outline and answer adapter, LEAP significantly leads the whole clip with full audio matched to its total tokens per question (TraceAV +4.27 pp), and at the same answer budget our trained score's mean accuracy is significantly above that of the ImageBind and SigLIP rankings (Table 11).
  4. Evidence spread across more than three blocks (Q3). Where the annotated evidence touches four or more 600 s blocks, LEAP still significantly leads the 32-frame whole clip on TraceAV (+8.1 pp); the three-block budget is a hyperparameter bounded by one answer pass, and a fourth block raises evidence coverage on these questions.

Could you kindly let us know whether our responses have addressed your concerns? If you have any further questions, please let us know, and we will address them.

Thank you again for your time and valuable feedback!

Reviewer_jiZ9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LEAP is a 2-pass retrieve-then-answer pipeline for long AV QA on Qwen3-Omni-30B-A3B (and MiniCPM-o 4.5).

  1. Localization. The recording is cut into fixed 600s blocks, each windowed into eight 75s candidate windows. A localization LoRA reads each block once (sparse frames plus full-rate audio). It scores the windows by option-letter logits (“In which time window does the evidence appear? A…H”) and ranks blocks by their maximum window logit.
  2. Answering. The top 3 blocks × top 3 windows (≤9 windows, about 11 minutes) are re-encoded at 32 frames per window plus the audio. A separate answer LoRA answers them in a single pass, together with a ≤4k-token, question-ranked Whisper transcript outline.

The localization LoRA is trained on LongVALE event-caption grounding. The answer LoRA is trained on 3,404 OmniVideo-100K instances (one gold segment plus two distractor segments from other recordings). A “transcript channel” variant scores windows from ASR text with the base model and the same pipeline is applied causally to StreamArena. The paper reports +4.5 to +16.8 pp over same-stack official-recipe baselines on TraceAV, LVOmniBench, VideoOdyssey and MMOU, with transfer to MiniCPM-o 4.5.

Critical strengths
  1. Evaluations are good. The method is compared with suitable and compatible pipelines.
  2. Bounded per-pass context is genuinely useful at hour scale. On VideoOdyssey (1–4 h) the whole recording cannot be encoded and LEAP's gain there is significant: 53.7 vs. 36.9 for the same-stack official recipe and 41.9 for the whole clip with the same answer LoRA. Per-task results (Table 20) show gains across most categories.
  3. The paper is honest about cost. It reports that cumulative prefill is 2.2–2.4× a whole-clip pass, that prefix caching trades memory linear in duration for prefill and it gives per-stage latency. It also honestly reports negative or null results, e.g., the outline hurts on LVOmniBench and the BCE vs. CE selector makes no accuracy difference.
  4. The diagnostic analyses are informative, in particular the hit/miss conversion analysis against human clue intervals (CG-Bench, Fig. 9) and the certificate-length analysis on VideoOdyssey (Tables 18–19).
Critical weaknesses
  1. The paper's own data undercuts the "learned" localization claim. The title and contributions center on a trained localizer, yet:

    • Table 2: the transcript channel scores windows with the untrained base model over ASR text and matches the media channel with the trained localization LoRA on every main benchmark (64.6 vs. 63.6, 46.5 vs. 45.8, 52.3 vs. 53.7, 66.2 vs. 66.3). It does so with about 5× fewer localization tokens and lower latency.
    • Table 11: off-the-shelf bge-m3 cosine over window transcripts beats the trained adapter on TraceAV (60.8 vs. 59.2) and is within about 1–3 points on LVOmniBench and VideoOdyssey. Only on MMOU does the adapter clearly lead.
    • Table 10: the localization LoRA's contribution is real but modest relative to the full gap over baselines.

    A fair reading is that most of the gain comes from bounded multi-window re-reading, the answer LoRA, and transcript-derived text, not from learning to localize. The paper should reframe accordingly. It should also explain why it deploys the media channel "throughout" when the cheaper transcript channel is equivalent on all four headline benchmarks; the only counterexample offered is CG-Bench, at −1.8 pp.

  2. Headline gains confound three factors. The Table 1 comparison against the official recipe mixes:

    • (a) retrieval;
    • (b) a trained answer LoRA (3.4k instances), which alone lifts the whole clip from 36.9 to 41.9 on VideoOdyssey;
    • (c) a 4k-token question-ranked transcript from an external ASR model (Whisper large-v3), which no baseline receives. Removing it costs 3.1 / 2.5 / 1.1 pp on TraceAV / VideoOdyssey / MMOU (Table 10).

    The clean comparison is Table 8: same answer LoRA, no outline, whole clip vs. selected windows, restricted to clips where both fit. There the retrieval gain is +0.7 pp on TraceAV and +1.0 pp on OmniVideoBench. It is larger only on LVOmniBench and MMOU. This table belongs in the main text, together with a baseline that gives the whole clip or montage the same transcript outline. The abstract's "4.5–16.8%" should also say percentage points and name the baseline.

  3. Limited novelty relative to localize-then-answer work. Training a localizer and an answerer as two parameter-efficient adapters on one shared frozen backbone is the design of SeViLA (Yu et al., NeurIPS 2023), which is not cited. There, a Localizer and an Answerer are both PEFT-tuned from BLIP-2; the Localizer picks language-aware keyframes for the Answerer, and the Answerer supplies pseudo-labels to refine the Localizer. The other ingredients are each well established and partly cited:

    • chunked or divide-and-conquer scanning (Zoom-Zero D&C, RevisionLLM);
    • text/ASR as the retrieval channel (DrVideo, Video-RAG, VSI);
    • multi-interval packing (VideoAgent, VideoTree, and similar).

    The incremental pieces are the extension to omni-modal backbones with full-rate audio and the specific within-block letter-logit scoring. These are useful engineering, but the paper should say plainly that this is its contribution.

  4. The cross-block ranking has no training signal. Blocks are ranked by max_k ℓ_{m,k} compared across independent passes. The localization loss (Eq. 3) is a within-pass softmax cross-entropy, which is invariant to adding a constant to all logits in a pass, so the absolute logit level that cross-block comparison depends on is never supervised. In addition, every training clip contains the evidence ("Every training clip fits in one block"), so the selector never sees a block with no evidence and is never trained to give it a low score. Table 15 shows the ranking works through the within-pass margin, but that is an emergent property, not a learned one. The method would be more principled with negative (evidence-free) blocks and either a cross-block objective or an explicit "no evidence" option. The paper already tries such a prompt in an ablation (C.1); it should be used in training.

  5. The training data is biased toward what the base model already does.

    • The localization corpus is LongVALE caption grounding ("in which window does this caption occur"), not question-conditioned evidence search. It is filtered to keep only items where the base model already answers yes on the target window and no on every other window (40k of 70k kept). That removes exactly the cases where the base model fails, which may explain why the learned selector adds little over the base selector on text-rich channels.
    • The answer adapter trains on one gold segment plus two distractors drawn from other recordings. Such distractors are trivially separable by scene. At inference the distractors come from the same recording, and there are up to 9 windows, not 3. Please report results with same-recording distractors and with larger segment counts.
  6. The "O(1)" framing overstates the efficiency advantage.

    • Peak per-pass context is bounded, but so is the context of any fixed-budget baseline, including the context-filled montage the paper compares against.
    • Query work is M+1 passes, linear in duration. Prefill is 2.2–2.4× a whole-clip pass, and the outline requires transcribing the full recording.
    • VideoOdyssey latency is a median of 30 s per question.

    The relevant question is accuracy at matched compute. Please add (i) whole-clip or montage baselines at matched total tokens or wall-clock, and (ii) latency for the baselines in Table 2. Also note that the streaming setting re-scans all causal blocks per query, so cost grows with stream history unless the cache, which grows linearly, is retained.

  7. The benchmarks are dominated by speech and text priors.

    • Text-only (stem plus options) already reaches 52.0 on TraceAV and 48.8 on MMOU (Fig. 3).
    • On Qwen, TraceAV and LVOmniBench accuracy "barely moves from the first window on" (Fig. 8). One 75 s window plus the outline is about as good as nine, which suggests these scores depend little on how much media is retrieved.
    • The largest per-task gains on VideoOdyssey are Speech Fact Retrieval (39.9→72.8) and OCR, while the vision-only track gains much less (41.2→44.1, Table 18).

    The "audio-visual perception" framing should be qualified. Most of the measured gain is speech-evidence retrieval, which the transcript results above corroborate.

  8. Fixed budget and grid.

    • B=3, W=3, and δ=75 s are fixed globally and capped by one answer pass. Questions whose evidence spans more than three blocks are unreachable by construction.
    • Δ=600 s is set by the longest LongVALE training clip, not by the task.
    • Blocks and windows do not overlap, and the effect of evidence straddling a boundary is not analyzed.
    • Coverage counts any overlap with a 75 s window as a hit, which is lenient.

    An adaptive budget, or at least a boundary-straddling analysis, is needed.

  9. Comparisons with other methods are thin on the same backbone.

    • The only same-stack retrieval/agent baseline actually run, the OmniRAG-Agent loop, scores far below the bare backbone (34.8 vs. 57.8 on MMOU). That suggests a broken integration, not a meaningful comparison.
    • AVP and Audio-Guided numbers come from the TraceAV paper.
    • No trained selector (TimeSearch-R, Zoom-Zero, video-SALMONN-R3-style) is run on the same backbone.
    • The released OmniVideo-100K SFT checkpoint, a whole-clip model with no retrieval, beats LEAP on MMOU (70.1 vs. 66.3, Table 7). The main text should acknowledge this.
    • Gemini-3-Flash remains well ahead on LVOmniBench (59.0 vs. 45.8).
  10. The MiniCPM-o "transfer" is not like-for-like. The abstract's "3.1–13.0% over published results" compares to published numbers produced by other pipelines. The paper criticizes exactly this practice for Qwen and fixes it there with same-stack re-runs. The MiniCPM variant also changes the localization objective (BCE), the frame rates, the LoRA targets (adding MLP), the training data (13k instances including LongVALE), and drops the outline. It is a re-tuning, not a transfer. Please add same-stack MiniCPM baselines for the main benchmarks.

  11. Ablation controls.

  • "LEAP without answer training" answers with the localization LoRA mounted. That adapter was trained only to pick time-window letters and is not a neutral control. The base weights should be used.
  • StreamArena results use a local Qwen3.6-27B judge with one vote, while published rows use the benchmark judge with multi-turn context. Judge agreement with the official judge is not reported, so Table 9's two blocks cannot be compared.
  • The contamination audit (B.6) reports only the identifier-level conclusion. The frame-hash and audio-fingerprint findings are described but never reported.
  1. Presentation. The main text is nine pages of conclusions whose evidence lives in a 38-page appendix. Many claims say only "significantly," with no number in the main text. Aggregations are inconsistent:
  • TraceAV is the general-subtask mean in Tables 1 and 10 (60.9) but all-question accuracy in Tables 2, 8, and 13 (63.6).
  • Several tables say they "compare only with each other" because of different tail-block splits.

The prose style is idiosyncratic and dense ("The block grid is fixed by the clock"). A reader cannot reconstruct the attribution of gains without cross-referencing about 10 tables.

Sources:

Questions for authors
  1. With the same answer LoRA and the same transcript outline given to the whole clip or montage, what is the remaining gain from retrieval on each benchmark (full denominator and both-fit)?
  2. Why is the media channel deployed when the base-selector transcript channel matches it on all four main benchmarks? On which benchmark or question types does the learned media localizer clearly win?
  3. Can you add evidence-free blocks and a cross-block or "no evidence" objective to localizer training, and report the effect on block ranking?
  4. How does the answer LoRA perform when trained with same-recording distractors and up to 9 segments?
  5. What are accuracy and latency at matched total compute for the whole-clip/montage baselines and for the base-selector transcript channel?
  6. Do the "without answer training" rows change if answered with base weights instead of the localization LoRA?
  7. What did the frame-hash and audio-fingerprint contamination levels find?
  8. How does LEAP relate to SeViLA's localizer/answerer design? How does it compare with a trained selector (e.g., TimeSearch-R or Zoom-Zero) on the same backbone?
10 comments

We thank the reviewer for an exceptionally careful reading of the paper and its appendix.

Conventions. pp = percentage points. Baselines are named as in Table 6. A paired difference is read on the same questions as LEAP-side minus comparator, with a two-sided 95% bootstrap interval resampled over source videos (at least 10,000 draws); significant means the interval excludes zero. Denominators are the full official question sets with failures scored wrong. TraceAV accuracy is over all questions unless marked general (the mean of its twelve general sub-tasks used in Table 1). Results marked † are new in this rebuttal.

W2 + Q1 (confounded headline gains)

Headline gains confound three factors. [...] The clean comparison is Table 8 [...] There the retrieval gain is +0.7 pp on TraceAV and +1.0 pp on OmniVideoBench. [...] This table belongs in the main text, together with a baseline that gives the whole clip or montage the same transcript outline. The abstract's "4.5–16.8%" should also say percentage points and name the baseline.

With the same answer LoRA and the same transcript outline given to the whole clip or montage, what is the remaining gain from retrieval on each benchmark (full denominator and both-fit)?

A1: Thank you for specifying the control exactly; we ran it†. With the same answer LoRA and the same outline given to the whole clip (the context-filled montage on VideoOdyssey), LEAP's gain stays significant on all four benchmarks under both denominators:

LEAP − (whole clip + same outline + same answer LoRA)TraceAVLVOmniBenchVideoOdysseyMMOU
full denominator+4.18 [+2.44, +5.97]+4.34 [+1.72, +7.00]+8.57 [+5.57, +11.48]+2.56 [+1.70, +3.40]
both fit+1.57 [+0.18, +2.99]+3.21 [+0.67, +5.69]+8.57 [+5.57, +11.48]+2.56 [+1.70, +3.40]

On VideoOdyssey and MMOU every question fits on both sides, so the two rows cover the same questions and give the same numbers.

The gain also survives removing answer training from both sides:

LEAP windows − official recipe's inputTraceAVLVOmniBenchVideoOdysseyMMOU
base weights and the same outline on both+3.36 [+1.53, +5.18]+6.21 [+3.57, +8.89]+4.61 [+2.12, +7.16]+2.52 [+1.56, +3.48]

LEAP keeps a significant retrieval gain with the same answer weights and outline on both sides, including when both sides answer with base weights.

The revised PDF adds this control to Table 8, and we will move Table 8 into §4.4. The revised PDF states the abstract's gains as "4.5–16.8 pp over the same-stack official recipe".

W11 + Q6 (the "without answer training" control)

"LEAP without answer training" answers with the localization LoRA mounted. That adapter was trained only to pick time-window letters and is not a neutral control. The base weights should be used.

Do the "without answer training" rows change if answered with base weights instead of the localization LoRA?

A2: Thank you for checking which weights answer that row. Answered by base weights on the same windows and outline, the row rises on TraceAV† and moves by less than 1.2 pp on the other three benchmarks, none significant. Against base weights on the same windows and outline, the answer adapter is significantly better on VideoOdyssey and MMOU:

Answer adapter − base weights, same windows and outlineVideoOdysseyMMOU
accuracy (pp)+5.84 [+3.33, +8.20]+5.84 [+4.82, +6.87]

On TraceAV, LEAP's lead over the official recipe comes from retrieval: it holds with base weights on both sides. Retrieval alone thus gives LEAP a significant lead on every benchmark (A1), and the answer adapter adds to it on VideoOdyssey and MMOU. We therefore use one answer LoRA for the four headline benchmarks rather than choosing weights per benchmark on test accuracy.

W1 + Q2 (what the learned localizer contributes)

The paper's own data undercuts the "learned" localization claim. [...] A fair reading is that most of the gain comes from bounded multi-window re-reading, the answer LoRA, and transcript-derived text, not from learning to localize. The paper should reframe accordingly. It should also explain why it deploys the media channel "throughout" when the cheaper transcript channel is equivalent on all four headline benchmarks [...]

Why is the media channel deployed when the base-selector transcript channel matches it on all four main benchmarks? On which benchmark or question types does the learned media localizer clearly win?

A3: Thank you for pressing on what the trained localizer adds. The trained localizer adds where evidence coverage is left to gain. With the outline and the answer adapter fixed†, replacing the base selector (the same selection run by the base model, no adapter mounted) by the localization adapter raises accuracy significantly on VideoOdyssey and MMOU, the benchmarks where it adds the most final-window coverage:

Localization adapter − base selectorVideoOdysseyMMOU
accuracy (pp)+3.11 [+0.58, +5.62]+0.84 [+0.18, +1.49]
final-window coverage (pp)+16.57+12.48

Against off-the-shelf retrievers it leads in the same place: MMOU accuracy +2.98 [+2.06, +3.89] over bge-m3, and higher coverage on VideoOdyssey and MMOU (Table 11).

Why the media channel. (i) It needs no speech. The transcript channel renders a window without speech as "(no speech)", leaving nothing from that window to rank it on, and a recording without speech has no transcript at all. (ii) Where evidence is visual, the media localizer wins. On CG-Bench, whose human clue intervals mark visual evidence, the transcript channel retains less clue time and answers less accurately, most where the clue window holds under 20% speech:

CG-BenchTranscript channel − media channel (pp)
clue time retained−8.40 [−10.54, −6.31]
accuracy−1.80 [−2.97, −0.64]
accuracy, clue window under 20% speech−2.79 [−4.87, −0.77]

(iii) Where evidence is spoken, the transcript channel is the cheaper instance of the same selector. The revised PDF presents the two channels as two instances of one selector, chosen by whether the recording's evidence is spoken (§4.5).

W3 + Q8 (relation to SeViLA; trained selectors)

Training a localizer and an answerer as two parameter-efficient adapters on one shared frozen backbone is the design of SeViLA (Yu et al., NeurIPS 2023), which is not cited. [...] The incremental pieces are the extension to omni-modal backbones with full-rate audio and the specific within-block letter-logit scoring. These are useful engineering, but the paper should say plainly that this is its contribution.

How does LEAP relate to SeViLA's localizer/answerer design? How does it compare with a trained selector (e.g., TimeSearch-R or Zoom-Zero) on the same backbone?

A4: Thank you for the SeViLA reference. We will cite SeViLA; what differs is the regime its localizer and answerer work in. SeViLA scores each of 32 uniformly sampled frames independently, asking whether the frame answers the question, and passes four keyframes to the answerer, which reads the same frame features the localizer saw; its benchmarks' clips average 12–76 s, and it reads no audio. LEAP's localization pass reads a whole 600 s block, native-rate audio plus sparse frames, and scores its eight windows jointly in one prompt; blocks are ranked on those logits across passes, and on Qwen3-Omni the answer pass re-encodes the chosen windows from the source at a denser per-window context. This keeps every pass bounded at four hours while evidence from different blocks is still compared and answered jointly. The localizer is supervised by annotated evidence spans inside one block, not by pseudo-labels from the answerer.

Trained selectors on the same backbone. TimeSearch-R and Zoom-Zero are trained by GRPO on Qwen2.5-VL-7B and are visual-only; Zoom-Zero has released no code or weights, and video-SALMONN-R3 has not released its weights. Running them on Qwen3-Omni would mean retraining their RL policies on that backbone with audio. What we can hold fixed is the backbone: Table 11 drops Zoom-Zero's trial-answer score into our pipeline, and our trained adapter answers more accurately than it on all four benchmarks while covering about 23 points more of VideoOdyssey's annotated evidence. We will fully opensource code.

W4 + Q3 (cross-block ranking has no training signal)

Blocks are ranked by max_k ℓ_{m,k} compared across independent passes. The localization loss (Eq. 3) is a within-pass softmax cross-entropy, which is invariant to adding a constant to all logits in a pass [...] The method would be more principled with negative (evidence-free) blocks and either a cross-block objective or an explicit "no evidence" option.

Can you add evidence-free blocks and a cross-block or "no evidence" objective to localizer training, and report the effect on block ranking?

A5: Thank you for the close reading of Eq. 3. The block ranking draws on the within-pass margin that Eq. 3 trains: in Table 15, dropping the margin cuts VideoOdyssey evidence coverage from 70.2 to 33.8, while subtracting the average logit does not lower it.

Evidence-free blocks and a cross-block objective. We trained this variant: each training clip is joined by three evidence-free donor blocks from other recordings; each block is scanned in its own pass, as at inference, and one cross-entropy runs over the windows of all four. It raises cross-block top-1 accuracy on a 247-item held-out validation set from 0.57 to 0.83, and on VideoOdyssey, where the competing blocks come from the same recording, it raises evidence coverage at three blocks by +9.04 pp [+5.16, +13.00]. Both variants were answered by the answer adapter without the outline, so the chosen windows are the only media the answer pass reads. Neither final-accuracy change is significant, and the intervals exclude gains above 1.6 and 2.6 pp:

Cross-block − within-pass objectiveTraceAVVideoOdyssey
final accuracy (pp)+0.55 [−0.49, +1.57]+0.56 [−1.48, +2.61]

We therefore keep the simpler within-pass objective.

Reviewer_F4vsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LEAP, a framework for long audio-video question answering based on scanning of fixed-size chunks and re-reading of the audio-video chunks to answer the questions under a fixed context budget. The framework uses two LoRA adapters separately trained for the tasks of localization and answering, where the transcripts can also be used as an alternative retrieval method. Results obtained by Qwen3-Omni and MiniCPM-o 4.5 on several benchmarks show improvements in performance.

Critical strengths
  1. LEAP achieves strong results across multiple benchmarks on both Qwen3-Omni and MiniCPM-o 4.5, with consistent improvements over the backbone baselines.
  2. LEAP uses a fixed context budget for each model call and trains the localizer on individual blocks, making long recordings easier to handle.
  3. The ablations provide clear evidence that localization training improves both evidence coverage and answer accuracy.
Critical weaknesses
  1. LEAP does support retrieval without a whole-recording context, but this property is also present in Zoom-Zero’s divide-and-conquer variant, leaving the novelty of the first contribution unclear.
  2. The streaming results show that LEAP works on video prefixes, but it is unclear what new streaming capability the framework adds.
  3. The writing is hard to follow, as readers need to check different parts of the paper to understand the adapter choices and outline settings.
Questions for authors
  1. What does "native support for causal queries" add beyond applying the same media pipeline to the recording prefix available at query time?
  2. How does accuracy change when the retrieved audio-video windows are removed, keeping the same answer LoRA, transcript outline, question, and answer options?
  3. For recordings that both approaches can process, how does LEAP compare with direct whole-recording inference in total query-to-answer time and measured peak GPU memory on the same hardware?
8 comments

We thank the reviewer for recognizing LEAP's consistent gains on both backbones and the ablations on localization training. Table, section and appendix numbers refer to the revised PDF, where the changes are marked in red.

Conventions. pp = percentage points. Baselines are named as in Table 6. A paired difference is read on the same questions as LEAP-side minus comparator, with a two-sided 95% bootstrap interval resampled over source videos (at least 10,000 draws); significant means the interval excludes zero. Denominators are the full official question sets with failures scored wrong. TraceAV accuracy is over all questions unless marked general (the mean of its twelve general sub-tasks used in Table 1). Results marked † are new in this rebuttal.

W1

LEAP does support retrieval without a whole-recording context, but this property is also present in Zoom-Zero's divide-and-conquer variant, leaving the novelty of the first contribution unclear.

A1: Thank you for naming the closest variant. LEAP couples block-local evidence supervision to recording-wide audio-visual retrieval: the same trained window logits (one logit per window letter) select windows inside each block and rank blocks across the recording, so cross-block ranking needs no separately trained scorer, no added parameters and no extra pass (App. A.5). It differs from Zoom-Zero's divide-and-conquer variant in how the scan is trained, what it scores, and what it hears.

The training. Zoom-Zero trains its policy by reinforcement learning. LEAP's localization objective is supervised by annotated evidence spans inside one block, so no training step reads more than one block.

The score. Zoom-Zero ranks a window by the confidence of a trial answer on it, so every scored window costs an answering pass. LEAP ranks windows, and blocks by their best window, on the window logits the localization pass already produced, so ranking adds no pass. Trial-answer confidence also varies mostly between questions: on TraceAV's 30 s-window scan, 71% of its variance lies between questions, which a within-question ranking cannot use, against 29% for our window logit (App. C.6).

Table 11 compares the two scores directly on the Qwen3-Omni backbone and all four benchmarks: every row keeps the same blocks, candidate windows and answer pass, and only the ranking score changes (under trial-answer confidence, the base model with no adapter ranks the windows inside a block). Final-window coverage is the share of questions whose annotated evidence lies in a window the answer pass reads.

Ranking score (Table 11)Final-window coverage, VideoOdyssey (%)Final-window coverage, MMOU (%)
Trial-answer confidence48.382.7
Trained window logit (LEAP)71.794.8

With windows ranked by trial-answer confidence, accuracy is also lower on all four benchmarks. The trained window logit is thus both the cheaper and the better ranking score.

The modality. Zoom-Zero's scan is visual. LEAP's localization pass reads each block's audio at the native rate with sparse frames. Muting the audio of that pass alone on TraceAV changes which windows are kept, and the kept windows then cover fewer of the evidence events carried by sound (75.1% → 61.5%, Table 16a).

The revised PDF states these three differences in §1, not only in App. A.5.

W2 + Q1

The streaming results show that LEAP works on video prefixes, but it is unclear what new streaming capability the framework adds.

What does "native support for causal queries" add beyond applying the same media pipeline to the recording prefix available at query time?

A2: Thank you for pressing on this wording. What LEAP adds under causal access is a bounded pass at any history depth. The usual way to answer a causal query is to read the whole prefix available at query time in one pass; as the prefix grows, that pass samples it ever more sparsely and finally stops at the model's position limit (§1).

LEAP replaces that one pass with two steps. Instead of reading the whole prefix at once, it scores each 600 s block of the prefix in its own pass; rather than sampling the prefix more thinly, it re-reads only the retrieved windows from the raw recording, at the same per-window context however far back they lie. No step needs content after the query time, and no pass grows with the prefix. Blocks are fixed by the clock and ranked on raw logits with no normalization over the recording, so the same block grid runs on a prefix unchanged, with no streaming training.

What this buys on StreamArena (Table 3, same backbone and judge): LEAP leads the whole prefix significantly overall, and by the widest margin where the evidence lies more than 30 minutes back.

StreamArena accuracy (%), Table 3LEAPWhole prefix
Overall32.025.4
Evidence more than 30 minutes back23.612.6

Streaming systems that ingest each frame once into a KV cache or a fixed memory answer from what they kept; LEAP re-reads the raw recording.

The revised PDF adds this contrast to §4.6.

W3

The writing is hard to follow, as readers need to check different parts of the paper to understand the adapter choices and outline settings.

A3: Thank you for reading the configurations closely. The paper evaluates LEAP on nine benchmarks, two backbones and under causal access, and ablates each of its components. The ten-page main text therefore presents the core method and the headline results: §3 defines both adapters and the transcript outline, §4.1 states which results read the outline, and Tables 1–3 and Figures 2–4 report the headline comparisons. The appendix holds the ablations, and each states which settings it changes.

Adapter choices. LEAP mounts at most one of its two LoRA adapters on the frozen backbone in each pass (§3.2.2). The localization LoRA scores the candidate windows in the localization pass; the answer LoRA answers from the retained windows in the answer pass. Each is trained from the base weights on its own data. StreamArena asks open-ended questions scored on free text, and there the base weights answer (§4.6), so LEAP's lead in Table 3 comes with no answer training. MiniCPM-o 4.5 carries its own two adapters (Table 5).

Outline settings. In Tables 1–3, LEAP's answer pass over its retrieved windows reads the transcript outline (§4.1). The selector comparisons (Figure 3, Table 11) answer every row with the same weights and no outline. The outline summarizes the recording and helps most where the retained windows miss the evidence (§4.5); without it, the answer pass reads the recording only through the selected windows.

Table 21 of the revised PDF, pointed to from §4.1, collects these choices:

ConfigurationSelectorAnswer weightsOutlineMain results
LEAPlocalization LoRA, mediaanswer LoRAyesTable 1; Figures 2 and 4
without outlinelocalization LoRA, mediaanswer LoRAnoTables 10, 13 and 14
without answer traininglocalization LoRA, medialocalization LoRAyesTables 1 and 13
selector comparisonlocalization LoRA, base model, or another ranking scorelocalization LoRAnoTable 11; Figures 3 and 10
without localization LoRAbase model, mediaanswer LoRAnoTable 10
transcript channelbase model, transcriptanswer LoRAyesTable 2
without retrievalnone: the whole clip, or the montage on VideoOdysseyanswer LoRAno (yes in Table 8)Tables 1, 8 and 10
causal accesslocalization LoRA or base model, media; or base model, transcript indexbase modelyesTables 3 and 9; Figure 6
MiniCPM-o 4.5its own localization LoRA, mediaits own answer LoRA; base model on StreamArenanoTable 17; Figures 2 and 11

Comments on the paper

3 comments
Area_Chair_thLwOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

Dear Reviewers,

Thank you for your thoughtful reviews and constructive suggestions, which have helped us improve our paper!

We have tried our best to address each of your concerns in our point-by-point responses. If you have any remaining concerns or further questions, please let us know. We are fully committed to resolving all of your concerns.

Thank you again for your time and valuable feedback!

Best regards, The Authors

AuthorsOpenReview ↗

Thank You to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

As the discussion period comes to a close, we would like to express our sincere gratitude to you all for your valuable time, excellent guidance, and constructive feedback!

The discussion has significantly helped us improve the clarity and rigor of our work. We have addressed all the concerns with our best efforts. In summary:

  • Novelty. LEAP differs from Zoom-Zero's divide-and-conquer variant in three ways (§1 of the revised PDF). Training: Zoom-Zero trains its policy by reinforcement learning; LEAP's localization objective is supervised by annotated evidence spans inside one block. Score: Zoom-Zero ranks a window by the confidence of a trial answer on it, so every scored window costs an answering pass; LEAP ranks windows, and blocks by their best window, on the logits its localization pass already produced, so ranking adds no pass. Modality: Zoom-Zero's scan is visual; LEAP's reads native-rate audio with sparse frames, and muting that audio lowers the coverage of evidence carried by sound on TraceAV from 75.1% to 61.5% (Table 16a). With the blocks, candidate windows and answer pass fixed, our trained score retains more evidence than trial-answer confidence and ImageBind similarity (final-window coverage 71.7% vs. 48.3% and 56.2% on VideoOdyssey, Table 11), and it answers more accurately than trial-answer confidence on all four benchmarks. Relative to SeViLA, which scores 32 uniformly sampled frames of a short clip one at a time from frames only and whose answerer reuses the frame features its localizer saw, LEAP scores the eight windows of each 600 s block jointly with their audio, ranks blocks across passes, and re-encodes the chosen windows from the source for the answer. On the same backbone, LEAP is also significantly more accurate on all four benchmarks than a MAGNET-style pipeline we built from MAGNET's released and described stages.
  • Retrieval gain with training and outline held fixed. With the same answer adapter and transcript outline on both sides, LEAP significantly leads the whole clip (the context-filled montage on VideoOdyssey) on all four benchmarks (+2.56 to +8.57 pp), and it significantly leads the official recipe's input when both sides answer with base weights.
  • Efficiency. The O(1) bound is stated for per-pass context and peak memory. On one H100 NVL, LEAP uses less peak memory than the official recipe on TraceAV, LVOmniBench and MMOU. Against a whole clip that reads as many frames as fit LEAP's cumulative prefill (capped at 60k tokens), with the same outline and answer adapter, LEAP leads significantly on TraceAV, LVOmniBench and MMOU (+1.86 to +4.27 pp). With transcription and repeated questions counted, LEAP is faster per recording than the retrieval agent OmniAgent-RL-7B on all four benchmarks.
  • Same-recording distractors. Answering all four benchmarks on the deployed windows and outline, adapters trained with same-recording distractors (two seeds) are less accurate than the deployed adapter on every benchmark, and pooled also less accurate than an adapter trained on an otherwise identical pool whose distractors come from other recordings (−3.20 pp [−4.14, −2.24] with the same seed). At inference, the deployed adapter reads up to nine windows of the same recording and still significantly improves accuracy over base weights on VideoOdyssey and MMOU.

We are pleased to have developed LEAP, which keeps the context of every model call bounded at hour scale by learning to localize evidence from native-rate audio and sparse frames, with evaluations against suitable and compatible baselines, consistent gains on two omni-modal backbones, and informative diagnostic analyses.

We hope our responses have clarified the remaining questions. Thank you again for your hard work, and we are devoted to further contributing to the long audio-visual understanding community!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.