acceptodds
Under review as a conference paper at ICLR 2027

Evi-Prefill: Evidence-Aware Prefill for Efficient Long Video Understanding

Abstract

Existing acceleration methods for long video large models adopt a paradigm that uniformly processes video tokens. This ignores the sequential arrival of task-relevant : once sufficient evidence emerges early, models still process the full video stream, leading to redundant computation. To extend video temporal coverage, chunk-wise inference adopts partial observability yet remains . In this paper, we present , an framework allocating subsequent compute from visual evidence accumulated under partial observability. Specifically, we formulate evidence gathering for incoming video chunks as a Partially Observable Markov Decision Process. With this model, we estimate evidence demand via the model’s internal signals and actively switch from inference, reducing redundant compute while preserving performance. Moreover, Evi-Prefill filters structurally redundant frames before visual encoding to boost evidence estimation efficiency. Evi-Prefill requires no training and complements prior methods. Evaluations over 3 benchmarks achieve 1.5–1.9 speedup without average performance degradation, opening a new route to efficient long video understanding via active visual-evidence acquisition. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Model, Long Video Understaning, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TtBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Evi-Prefill, a training-free pipeline for efficient long-video understanding. It combines SSIM-based frame filtering with attention-guided switching from dense to sparse chunk processing to reduce visual encoding and LLM prefill costs. Experiments on two LLaVA-based backbones and three video benchmarks report additional speedups over existing compression methods while maintaining or slightly improving average accuracy.

Critical strengths
  1. Practical inference efficiency. The method reduces both visual encoding and LLM prefill costs through frame filtering and adaptive chunk processing. The reported speedups, with comparable or slightly improved average accuracy, provide a meaningful practical benefit for long-video inference.
  2. Training-free compatibility. The pipeline can complement existing compression methods without additional training. Results across multiple compression methods and two backbones support its usefulness as an add-on optimization.
Critical weaknesses
  1. Limited methodological novelty. The method appears to be a well-engineered combination of existing heuristics rather than a substantial methodological advance.
  2. Insufficient conceptual insight. The paper does not convincingly establish that its attention-based heuristic measures evidence sufficiency rather than serving as a pipeline-specific scheduling rule.
  3. Limited backbone coverage. Evaluation uses only two LLaVA-based backbones, leaving applicability to newer architectures, such as Qwen3-VL and InternVL, unclear.
  4. Limited benchmark coverage. Broader evaluation on long-video(LVBench, InfiniBench) and egocentric benchmarks(EgoLifeQA) is needed to substantiate the method’s generality.
  5. Insufficient fine-grained analysis. Aggregate results do not establish robustness across video categories, evidence locations, and tasks requiring temporally distributed reasoning.

Limited methodological novelty remains my main concern, so addressing the empirical gaps alone may not be sufficient for me to raise my score.

Questions for authors
  1. Novelty: What is the substantive methodological contribution beyond combining existing frame filtering, attention thresholding, and sparse processing heuristics?
  2. Evidence sufficiency: How do you distinguish declining NCM caused by its cumulative normalization from a genuine reduction in the need for additional evidence?
  3. Matched-budget comparison: Does attention-guided scheduling outperform fixed or uniform sampling at the same latency or processed-frame budget?
  4. Generalizability: Do you have evidence that the gains extend to newer backbones and more challenging long-video or egocentric benchmarks?
  5. Failure cases: How does performance vary across video categories, particularly for late-occurring evidence and temporally distributed reasoning?
15 comments
AuthorsOpenReview ↗

Response to W1&Q1

W1&Q1: What is the substantive methodological contribution beyond combining existing frame filtering, attention thresholding, and sparse processing heuristics?

Thank you for your constructive comments. It is worth noting that Evi-Prefill is a simple yet effective extension acceleration method. However, simplicity does not imply limited novelty. Our contribution extends efficient long video understand from compressing acquired visual representations to allocating subsequent computation based on observed evidence. We clarify this contribution in four aspects.

1. Identifying passive prefill redundancy that remains after compression. The evaluated compression methods reduce per-chunk processing costs but still densely traverse the input video. We instead consider the sequential arrival of task-relevant evidence and its uneven contribution across time. The following table (from Tables 1–2) shows that redundant computation in existing methods can be reduced while maintaining or improving performance, demonstrating the complementarity of visual compression and temporal evidence allocation for further acceleration.

MethodVideoMMELongVideoBenchMLVUGen Latency (s)
VidCom263.257.160.312.3
VidCom2+Ours63.857.660.88.2
VisionZip63.157.860.512.7
VisionZip+Ours64.058.961.07.4
FlashVID63.559.862.813.9
FlashVID+Ours64.259.863.09.3
FlexMem64.462.065.922.4
FlexMem+Ours65.262.566.413.6

2. Establishing an empirical link between structural redundancy and internal attention. Figures 2 and 9 show overlap between structurally similar frames and low-attention regions, motivating filtering before visual encoding. Frame Allocation (FA) uses structural similarity, requiring neither an external model nor pre-extracted visual features. We validate this design through evidence retention and matched-budget comparisons. On the 1,415 LVBench questions whose reference intervals contain candidate frames, FA thus maintains high annotated-evidence coverage while reducing the input frames.:

Frame policyQuestionsRetaining ≥1 reference frameMean reference-frame recall
All candidate frames1,415100.0%100.0%
FA1,41599.2%97.3%

We further match uniform sampling to FA’s retained frame count per question, with the host compressor fixed and EA disabled. At identical frame budgets, FA improves overall accuracy over uniform sampling, with gains of 1.3/1.6 points on long videos, supporting its selection value:

MethodFrame policyMean framesOverall / Short / Medium / Long
VisionZipAll candidates343.463.1 / 74.2 / 61.6 / 53.6
VisionZip+OursUniform299.363.4 / 74.4 / 62.2 / 53.4
VisionZip+OursFA299.364.0 / 75.1 / 62.2 / 54.7
FlashVIDAll candidates343.463.4 / 75.9 / 61.3 / 53.0
FlashVID+OursUniform291.763.5 / 75.1 / 62.9 / 52.4
FlashVID+OursFA291.763.8 / 76.2 / 61.2 / 54.0

3. Turning internal observations into a concrete temporal evidence allocation policy. We formulate subsequent evidence acquisition as sequential decision-making under partial observability, using question-conditioned new-chunk attention as a lightweight proxy signal. To assess this policy, we fix FA and the host compressor and match uniform sampling to EA’s actual processed-chunk count per question. At the same processing budget and comparable latency, EA improves overall accuracy by 0.4/0.7 points, including 1.4/1.3 points on long videos. This supports the value of observation-guided allocation beyond simply processing fewer chunks:

MethodConfigurationOverall / Short / Medium / LongGen Latency (s)
VisionZip+OursUniform63.6 / 75.1 / 61.8 / 53.94.4
VisionZip+OursEA64.0 / 75.1 / 61.7 / 55.34.3
FlashVID+OursUniform63.1 / 75.8 / 60.8 / 52.86.1
FlashVID+OursEA63.8 / 76.1 / 61.1 / 54.16.0

4. Providing a perspective complementary to representation compression (the most crucial novelty of this paper). These designs establish two transferable insights:

  • Compute allocation of temporal evidence remains important after compression.
  • Internal observations produced during inference can guide subsequent visual computation.

Evi-Prefill provides a training-free implementation and experimental baseline for this direction, compatible with existing token compressors. We will highlight the innovations of this paper in subsequent updates to the manuscript.

AuthorsOpenReview ↗

Response to W2

W2: The paper does not convincingly establish that its attention-based heuristic measures evidence sufficiency rather than serving as a pipeline-specific scheduling rule.

We appreciate the opportunity to clarify the insights behind our method.

We identify an observation in streaming processing: New Chunk Attention Reveals Evidence Demand (see Figure 2). This phenomenon holds not only for early evidence but also for later evidence (see Figure 7). We leverage this signal as a proxy for proactive evidence acquisition under partial observability (see Section 4.1). Our approach shows promising performance across different methods and datasets (see Table1-2), and extends effectively to additional backbone architectures and datasets (see following response). These results suggest that our approach goes beyond pipeline-specific scheduling. We hope our modeling of streaming evidence acquisition and insights into proactive prefill will offer a new perspective on efficient long video understanding.

AuthorsOpenReview ↗

Response to Q2

Q2: How do you distinguish declining NCM caused by its cumulative normalization from a genuine reduction in the need for additional evidence?

Thank you for this question. The key distinction is between position-dependent decay and evidence-dependent variation. Reviewer q8Ak raised a similar concern. Following the suggestion of q8Ak, we test whether NCM responds to evidence when processing position and prefix history are held fixed.

On LVBench, we construct paired inputs with the same question, identical prefix, and identical chunk position, replacing only the current chunk with either an annotated evidence chunk EE or a visually matched chunk NN from the same video outside the annotated evidence interval. The visual-token budgets and timestamp prompts are also matched.

Let Cb=∑i<bαi+ϵC_b=\sum_{i<b}\alpha_i+\epsilon denote the shared cumulative prefix mass. Then:

mbE=αbECb+αbE,mbN=αbNCb+αbN.m_b^{E}=\frac{\alpha_b^{E}}{C_b+\alpha_b^{E}}, \qquad m_b^{N}=\frac{\alpha_b^{N}}{C_b+\alpha_b^{N}}.

Therefore:

mbE−mbN=Cb(αbE−αbN)(Cb+αbE)(Cb+αbN).m_b^{E}-m_b^{N}=\frac{C_b(\alpha_b^{E}-\alpha_b^{N})}{(C_b+\alpha_b^{E})(C_b+\alpha_b^{N})}.

With the prefix and position controlled, this difference isolates the current chunk’s attention response rather than a change in processing position.

Chunk positionMean NCM: Evidence (×10²)Mean NCM: Control (×10²)Relative increase
b=32b=324.313.73+15.4%
b=64b=642.291.99+14.6%

At matched chunk positions, evidence chunks exhibit 15.4% and 14.6% higher mean NCM than their controls. Their mean unnormalized attention mass is also higher by 2.60 and 2.26, respectively, indicating that the separation is already present before cumulative normalization. These results support that NCM captures evidence-sensitive information beyond the normalization-induced positional trend. Accordingly, Section 4.1 uses its recent trajectory as an observable surrogate.

Reviewer_pq2cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Evi-Prefill, a training-free active prefill framework for efficient long-video understanding. It uses question-conditioned attention to adaptively switch from dense to sparse processing and filters redundant frames before visual encoding. Experiments show improved efficiency with largely preserved performance across several long-video benchmarks.

Critical strengths
  1. The paper presents a clear active-prefill perspective for reducing redundant computation in long-video inference.

  2. The method is training-free and complementary to existing token-compression approaches.

  3. Experiments show consistent efficiency gains while generally preserving or slightly improving accuracy.

Critical weaknesses
  1. The key assumption that low new-chunk attention implies low future evidence demand is not fully validated, especially for late-occurring evidence.

  2. The POMDP formulation appears stronger than the actual implementation, which mainly relies on threshold-based attention heuristics.

  3. Frame filtering based on SSIM may remove semantically important but visually similar frames.

  4. The evaluation is limited to LLaVA-based backbones, so the generality of the attention-based criterion remains unclear.

Questions for authors

see weaknesses

6 comments
AuthorsOpenReview ↗

Response to Q1

Q1: The key assumption that low new-chunk attention implies low future evidence demand is not fully validated, especially for late-occurring evidence.

Thank you for highlighting this point. We provide three complementary analyses to strengthen the our Evi-Prefill:

  1. The observation that new-chunk attention reflects evidence demand (see Figure 2) also extends to late-occurring evidence. Appendix A.1, Figure 7 provides additional examples showing that attention responds when evidence arrives late and remains focused on it as the context grows. This supports using NCM to guide computation allocation while continuing to sample later content.

  2. Evi-Prefill also improves performance on samples with late-occurring evidence. Using LVBench’s official temporal evidence annotations, we obtain the following results:

MethodEarly (680)Middle (367)Late (300)Overall
FlashVID51.544.746.748.8
FlashVID + Ours52.145.048.049.3
Δ+0.6+0.3+1.3+0.5

We see improvements across all three subsets. Encouragingly, the largest gain is on the late-evidence subset (+1.3 percentage points), suggesting that our method remains effective even when the relevant evidence appears late.

  1. Qualitative evidence coverage. Appendix A.4 further provides visual examples showing how our method covers evidence at different stages of a video. Late-occurring evidence is indeed challenging, but our method still performs well in these cases.
AuthorsOpenReview ↗

Response to Q2

Q2: The POMDP formulation appears stronger than the actual implementation, which mainly relies on threshold-based attention heuristics.

Thank you for your insightful comments. As illustrated in Section 4.1, we uses the POMDP perspective to model evidence acquisition under partial observability, with our implementation corresponding to a simplified, training-free policy. The correspondence is as follows:

  • Latent state sbs_b: whether the accumulated evidence is sufficient for the current query. This state is not directly observable.
  • Observation obo_b: the question-conditioned New Chunk Mass obtained from the current prefill pass:

ob≡mb=αb∑i=1bαi+ϵ,o_b \equiv m_b=\frac{\alpha_b}{\sum_{i=1}^{b}\alpha_i+\epsilon},

where αb\alpha_b aggregates question-to-visual attention received by the incoming chunk.

  • Action aba_b: the temporal density of subsequent processing, determining which future chunks undergo visual encoding, prefilling, and memory writing.

Ideally, the action would depend on the evidence-sufficiency belief defined in Section 4.1:

βb=P(sb=sufficient∣o1:b).\beta_b=P(s_b=\mathrm{sufficient}\mid o_{1:b}).

Our implementation avoids estimating this posterior and learning transition and observation models. Instead, it directly maps the recent observation history and temporal context to an allocation action:

ab=πEA(mb−P+1:b,b,B).a_b=\pi_{\mathrm{EA}}\left(m_{b-P+1:b},b,B\right).

The recent NCM trajectory therefore serves as a lightweight surrogate for belief-based decision-making. EA is a restricted deterministic policy within this partially observable setting, rather than a general POMDP solver. This provides a simple training-free baseline for guiding subsequent visual computation.

Despite its simplicity, this policy provides an effective baseline for temporal evidence compute allocation: at matched processed-chunk budgets, EA improves VideoMME accuracy over uniform sampling by 0.4/0.7 points with VisionZip/FlashVID, including 1.4/1.3 points on long videos.

MethodConfigurationOverall / Short / Medium / LongLongVideoBenchGen Latency (s)
VisionZip-63.1 / 74.2 / 61.6 / 53.657.57.2
VisionZip+OursUniform63.6 / 75.1 / 61.8 / 53.958.14.4 (1.6×)
VisionZip+OursEA64.0 / 75.1 / 61.7 / 55.358.94.3 (1.7×)
FlashVid-63.4 / 75.9 / 61.3 / 53.059.88.9
FlashVid+OursUniform63.1 / 75.8 / 60.8 / 52.859.56.1 (1.5×)
FlashVid+OursEA63.8 / 76.1 / 61.1 / 54.159.86.0 (1.5×)

We have clarified this distinction in Section 4.1 and Appendix A.2 in our manuscript. Admittedly, this simplified implementation may not necessarily be optimal, but it opens up opportunities for the community to further explore streaming evidence acquisition. As mentioned in the Conclusion, we also plan to extend our work toward more fine-grained evidence modeling in future research.

AuthorsOpenReview ↗

Response to Q3

Q3: Frame filtering based on SSIM may remove semantically important but visually similar frames.

Thank you for your insightful comments. We assess its impact on semantic understanding through two complementary evaluations.

  1. Retention of annotated evidence. On the 1,415 LVBench questions whose reference intervals contain candidate frames, FA retains at least one reference frame for 99.2% of questions and achieves 97.3% reference-frame recall. These results indicate that FA maintains strong coverage of annotated evidence while reducing the frame budget.
Frame policyQuestionsRetaining ≥1 reference frameMean reference-frame recall
All candidate frames1,415100.0%100.0%
FA1,41599.2%97.3%
  1. Semantic understanding at matched frame budgets. We compare FA with all candidate frames and uniform sampling on VideoMME, matching the retained frame count per question while keeping the host compressor fixed and EA disabled. At matched frame budgets, FA improves overall accuracy over uniform sampling by 0.6/0.3 points with VisionZip/FlashVID, with larger gains on long videos (+1.3/+1.6 points). It also outperforms using all candidate frames, showing that FA preserves useful semantic information while reducing redundancy.
MethodFrame policyMean framesOverall / Short / Medium / Long
VisionZipAll candidates343.463.1 / 74.2 / 61.6 / 53.6
VisionZip+OursUniform299.363.4 / 74.4 / 62.2 / 53.4
VisionZip+OursFA299.364.0 / 75.1 / 62.2 / 54.7
FlashVIDAll candidates343.463.4 / 75.9 / 61.3 / 53.0
FlashVID+OursUniform291.763.5 / 75.1 / 62.9 / 52.4
FlashVID+OursFA291.763.8 / 76.2 / 61.2 / 54.0
Reviewer_q8AkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evi-Prefill, a training-free framework for efficient long-video understanding. It combines SSIM-based frame filtering with an attention-based strategy that switches from dense chunk processing to sparse continuation. Experiments on three benchmarks and two LLaVA backbones show that Evi-Prefill can further accelerate existing compression methods while maintaining or slightly improving average accuracy.

Critical strengths

The paper addresses a practical problem in long-video understanding: avoiding unnecessary visual encoding and prefilling when additional video content provides limited useful information. The proposed framework is conceptually intuitive and requires no additional training or auxiliary models. A particularly appealing aspect is that Evi-Prefill provides further acceleration on top of existing acceleration methods. Its compatibility with several compression approaches suggests that the proposed scheduling strategy offers a complementary source of efficiency and has practical value as an additional component. Experiments across multiple host methods, two backbones, and three benchmarks provide useful evidence for its effectiveness. The component ablations and longer-input experiments also help explain its behavior.

Critical weaknesses
  • The independent benefit of attention-based allocation is not fully established. A comparison with uniform chunk sampling at the same processing budget is missing. The fixed-stride variant in Table 5 still uses attention to determine the dense-to-sparse transition. Keeping Frame Allocation and the host method fixed, comparing EA against uniformly selecting the same number of chunks would clarify whether attention-guided allocation is more effective than simply processing less video content.
  • The interpretation of NCM as an evidence-sufficiency signal needs further validation. In Eq. 3, the current chunk’s attention mass is normalized by cumulative attention mass. Even with constant per-chunk attention, NCM decreases as 1/b1/b. The authors should distinguish this processing-position effect from an actual reduction in evidence demand, for example through position-controlled analysis.
  • The sampling and chunking configurations are insufficiently evaluated. The experiments use a fixed sampling rate of 1 fps and four-frame chunks, while Table 3 varies only the frame-count limit. It remains unclear whether the advantages persist across different sampling rates and chunk sizes, or whether lower-rate sampling or larger chunks could achieve a similar accuracy–efficiency trade-off.
Questions for authors
  1. Could the authors compare EA with uniform chunk sampling at the same processing budget, while keeping Frame Allocation and the host method fixed?
  2. How does NCM distinguish evidence saturation from the decrease introduced by cumulative normalization?
  3. Does Evi-Prefill retain its advantage when the sampling rate and chunk size are varied for both the host baseline and the proposed method?
  4. Do the main generation-latency measurements include frame selection, preprocessing, and attention-statistic extraction? Is the selection overhead reported in Table 6 already included?
5 comments
AuthorsOpenReview ↗

Response to Q1

Q1: Could the authors compare EA with uniform chunk sampling at the same processing budget, while keeping Frame Allocation and the host method fixed?

Thank you for suggesting this control. We evaluate on two H100 GPUs, matching uniform sampling to Evidence Allocation (EA)’s processed-chunk budget for each sample, with Frame Allocation and the host compressor fixed. The results show that EA’s evidence-guided dense-to-sparse transition allocates the same processing budget more effectively than uniform sampling.

MethodConfigurationOverall / Short / Medium / LongLongVideoBenchGen Latency (s)
VisionZip-63.1 / 74.2 / 61.6 / 53.657.57.2
VisionZip+OursUniform63.6 / 75.1 / 61.8 / 53.958.14.4 (1.6×)
VisionZip+OursEA64.0 / 75.1 / 61.7 / 55.358.94.3 (1.7×)
FlashVid-63.4 / 75.9 / 61.3 / 53.059.88.9
FlashVid+OursUniform63.1 / 75.8 / 60.8 / 52.859.56.1 (1.5×)
FlashVid+OursEA63.8 / 76.1 / 61.1 / 54.159.86.0 (1.5×)

Uniform sampling accelerates inference but does not consistently improve accuracy. At comparable latency, EA outperforms uniform sampling by 0.4/0.7 points on VideoMME, including 1.4/1.3 points on long videos, and 0.8/0.3 points on LongVideoBench with VisionZip/FlashVID, respectively.

AuthorsOpenReview ↗

Response to Q2

Q2: How does NCM distinguish evidence saturation from the decrease introduced by cumulative normalization?

Thank you for this question. The key distinction is between position-dependent decay and evidence-dependent variation. We follow your sueegestion and test whether NCM responds to evidence when processing position and prefix history are held fixed.

On LVBench, we construct paired inputs with the same question, identical prefix, and identical chunk position, replacing only the current chunk with either an annotated evidence chunk EE or a visually matched chunk NN from the same video outside the annotated evidence interval. The visual-token budgets and timestamp prompts are also matched.

Let Cb=∑i<bαi+ϵC_b=\sum_{i<b}\alpha_i+\epsilon denote the shared cumulative prefix mass. Then:

mbE=αbECb+αbE,mbN=αbNCb+αbN.m_b^{E}=\frac{\alpha_b^{E}}{C_b+\alpha_b^{E}}, \qquad m_b^{N}=\frac{\alpha_b^{N}}{C_b+\alpha_b^{N}}.

Therefore:

mbE−mbN=Cb(αbE−αbN)(Cb+αbE)(Cb+αbN).m_b^{E}-m_b^{N}=\frac{C_b(\alpha_b^{E}-\alpha_b^{N})}{(C_b+\alpha_b^{E})(C_b+\alpha_b^{N})}.

With the prefix and position controlled, this difference isolates the current chunk’s attention response rather than a change in processing position. Across 80 questions from 55 videos:

Chunk positionMean NCM: Evidence (×10²)Mean NCM: Control (×10²)Relative increase
b=32b=324.313.73+15.4%
b=64b=642.291.99+14.6%

At matched chunk positions, evidence chunks exhibit 15.4% and 14.6% higher mean NCM than their controls. Their mean unnormalized attention mass is also higher by 2.60 and 2.26, respectively, indicating that the separation is already present before cumulative normalization. These results support that NCM captures evidence-sensitive information beyond the normalization-induced positional trend. Accordingly, Section 4.1 uses its recent trajectory as an observable surrogate.

AuthorsOpenReview ↗

Response to Q3

Q3: Does Evi-Prefill retain its advantage when the sampling rate and chunk size are varied for both the host baseline and the proposed method?

We thank the reviewer for this important question regarding robustness. We vary FPS over {0.5, 1, 2} and chunk size over {4, 6, 8}, applying identical settings to each host baseline and its Evi-Prefill counterpart. Evi-Prefill improves overall accuracy in every tested configuration while providing additional acceleration.

MethodFPSOverallShortMediumLongGen Latency (s)
VisionZip0.563.373.862.054.23.7
VisionZip+Ours0.563.673.862.354.72.6(1.4×)
VisionZip163.174.261.653.67.2
VisionZip+Ours164.075.162.254.74.3(1.7×)
VisionZip263.474.961.853.69.5
VisionZip+Ours264.075.462.454.25.1(1.9×)
FlashVid0.562.873.462.053.04.5
FlashVid+Ours0.563.974.363.354.03.5(1.3×)
FlashVid163.475.961.353.08.9
FlashVid+Ours163.876.161.154.16.0(1.5×)
FlashVid263.776.161.054.011.7
FlashVid+Ours263.976.861.953.07.2(1.6×)

As FPS increases from 0.5 to 2, speedups grow from 1.4× to 1.9× for VisionZip and 1.3× to 1.6× for FlashVID, while maintaining accuracy gains. This is consistent with denser sampling introducing more redundant observations: FA filters structural redundancy, while EA reduces continued dense processing once marginal evidence diminishes.

MethodCHUNK_SIZEOverallShortMediumLongGen Latency (s)
VisionZip463.174.261.653.67.2
VisionZip+Ours464.075.162.254.74.3(1.7×)
VisionZip663.474.961.853.66.7
VisionZip+Ours663.974.863.053.93.9(1.7×)
VisionZip864.074.263.454.46.7
VisionZip+Ours864.474.462.856.03.8(1.8×)
FlashVid463.475.961.353.08.9
FlashVid+Ours463.876.261.254.06.0(1.5×)
FlashVid662.975.761.251.88.5
FlashVid+Ours663.776.761.952.65.7(1.5×)
FlashVid863.475.061.254.08.7
FlashVid+Ours864.776.763.354.15.7(1.5×)

Larger chunk_size also shows further potential. At chunk_size 8, Evi-Prefill achieves 64.4/64.7 accuracy with VisionZip/FlashVID, improving their matched baselines by 0.4/1.3 points, with 1.8×/1.5× speedups. Both achieve higher accuracy and lower latency than their Evi-Prefill configurations at chunk size 4. Our original experiments were limited to a chunk_size of 4 by the memory capacity of RTX 3090 GPUs. Using H100 GPUs allows us to explore larger chunk_size, further unlocking the potential of Evi-Prefill. We thank the reviewer for this suggestion, which prompted us to investigate these settings and demonstrate additional performance and efficiency gains.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Comment by Authors

We sincerely thank all the reviewers for their constructive comments. We are delighted to receive positive feedback such as "conceptually intuitive" (q8Ak), "clear active-prefill perspective" (pq2c), "particularly appealing" (q8Ak), "easy to follow" (pq2c, TtBK), "useful evidence for its effectiveness" (q8Ak), "practical value" (q8Ak, TtBK), "meaningful" (TtBK), "further acceleration" (q8Ak), "consistent/additional efficiency gains" (pq2c, q8Ak, TtBK).

We have carefully addressed all the concerns raised by the reviewers point by point in the individual response section. In addition, we have supplemented our work with extensive and detailed ablation experiments to further investigate specific characteristics of our proposed method. We upload the revised paper, with major updates highlighted in yellow.

  1. Methodological Contribution beyond Existing Heuristics. We clarify that Evi-Prefill extends efficient long-video understanding from compressing acquired representations to allocating subsequent computation based on observed evidence. Matched-budget comparisons separately evaluate FA’s frame selection and EA’s temporal allocation, demonstrating benefits beyond simply processing fewer inputs (Appendix A.4).
  2. Evidence Supporting the Attention-Based Assumptions. We address two distinct concerns. To distinguish evidence-sensitive attention from cumulative-normalization decay, we conduct paired experiments with the same question, prefix, chunk position, and token budget; evidence chunks yield 15.4% and 14.6% higher mean NCM than matched controls. To examine whether low NCM risks missing future evidence, we add late-evidence attention visualizations, evidence-location evaluations, and qualitative coverage successes and failures. These analyses support NCM as a useful observable surrogate (Appendix A.1 & A.4).
  3. POMDP Interpretation and Implementation. We explicitly map the latent state, observation, and action to the implementation. We clarify that EA is a simplified, deterministic, training-free policy under partial observability: it uses recent NCM observations to guide allocation without estimating a posterior belief or learning transition and observation models (Section 4.1 & Appendix A.2).
  4. Semantic Evidence Retention under Frame Filtering. We evaluate whether structural filtering removes important evidence. FA retains at least one annotated reference frame for 99.2% of eligible LVBench questions, with 97.3% mean reference-frame recall. Comparisons against uniform sampling at identical frame budgets further assess its semantic selection value (Appendix A.4).
  5. Generalization and Robustness. We extend evaluation to Qwen3-VL-4B, EgoSchema (full), and LVBench, and vary sampling rates over 0.5, 1, and 2 fps and chunk sizes over 4, 6, and 8 frames. These additions assess applicability across architectures, video characteristics, and processing settings (Appendix A.4).

If there are any remaining questions or concerns, we would be most delighted to engage in further discussions and address them to the best of our ability.

Best regards,

The authors

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chairs (ACs),

We sincerely appreciate your time and effort in reviewing our paper.

During the discussion, we made every effort to address the reviewers' concerns. So far, we have received two positive support and one borderline support from reviewers.

  • Reviewer q8Ak maintained a score of 3 (Weak acceptance) after our rebuttal.
  • Reviewer pq2c increased the score from 2 to 3 (Weak acceptance) after we addressed concerns regarding our assumptions, modeling, and generalization.
  • Reviewer TtBK raised their assessment from 2 to 2.5 (Borderline) after we clarified the methodological novelty and provided extensive additional experimental results. As the reviewer stated: “... so I will maintain my score. Nevertheless, I now consider the paper borderline and would not oppose acceptance.” As ICLR does not provide a separate borderline rating, we represent this assessment as 2.5.

By the end of the discussion, our rebuttal and revisions had resulted in an overall score improvement from 2.33 (3-2-2) to 2.83 (3-3-2.5).

ReviewerInitial RatingFinal RatingConfidenceFeedback
q8Ak333Positive
pq2c233Positive
TtBK22.53Positive & Borderline

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.