acceptodds
Under review as a conference paper at ICLR 2027

CoFi: Compress First, Focus Later for Long Video Understanding

Abstract

Long-video understanding requires allocating a limited visual input budget to observations that are both temporally informative and relevant to the question. Existing training-free methods typically address this bottleneck through query-aware frame selection, visual compression, or adaptive combinations of the two. However, adaptive representation methods generally determine where to allocate visual capacity before explicitly accounting for what information a compressed view of the video is estimated to preserve. We introduce CoFi (Compress First, Focus Later), a training-free visual input construction framework that makes compression and focus acquisition sequentially dependent. CoFi first constructs a query-independent compressed temporal context and conditions a state–trajectory relational prior on this context to estimate the residual visual–temporal structure. It then introduces the question and sequentially acquires standalone focus observations that reduce query-relevant residual uncertainty, updating the residual structure after each acquisition. Under a fixed visual-unit budget, CoFi achieves the best performance on MLVU and LongVideoBench while remaining competitive with the best-performing frame-selection baseline on Video-MME. Ablation studies further show that compressed context and context-conditioned sequential focus acquisition play complementary roles, supporting the effectiveness of the proposed compress-first, focus-later design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-video understanding, Video question answering, Multimodal large language models, Training-free visual input construction, Visual compression, Frame selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Vsc4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoFi addresses long-video question answering with a frozen Video-LLM under a limited visual-input budget. It first constructs query-independent temporal panels and conditions a state–trajectory RBF covariance prior on the compressed context, using representation fidelity to set observation noise. It then sequentially selects query-relevant standalone frames from the residual covariance.

Critical strengths
  1. The method gives a coherent way to condition selection on an existing compressed representation. The positive-semidefinite kernel and posterior updates propagate information between related candidate locations and provide sequential redundancy control. This makes the design more substantive than simply concatenating temporal panels and relevance-ranked frames.
  2. On LLaVA-Video-7B, CoFi achieves the highest accuracy among the compared methods on MLVU and LongVideoBench and the second-highest on Video-MME (Table 1). Table 3 shows that sequential updating improves over Static Acquisition by 2.78, 2.62, and 1.42 percentage points, respectively. Importantly, Table 11 also equips existing selectors with the same context panels and focus budget. These controls, together with the analyses of visual budgets, representations, and approximation stability, provide meaningful evidence for the overall construction strategy.
Critical weaknesses
  1. The primary LLaVA-Video-7B evaluation includes multiple established selectors and visual-representation methods. However, the Qwen2.5-VL-7B and InternVL2.5-8B evaluations compare CoFi only with Uniform Sampling. Appendix Table 16 provides duration-wise results but does not add stronger selection baselines. These experiments establish that CoFi’s improvement over uniform sampling transfers across backbones, but they do not establish whether its advantage over strong existing selectors also transfers. This limits the evidence for the broader applicability of its relative performance advantage.
  2. Context conditioning is a central design choice, but the independent gains shown in Table 3 are modest. Additional comparisons with and without conditioning on more Video-LLM backbones would help clarify the practical value of this mechanism.
  3. The outer relevance factor π_j in Equation (5) has not been separately ablated. Its additional contribution beyond relevance-based candidate filtering and query-weighted residual reduction remains unclear.
Questions for authors
  1. Could you evaluate a representative strong selector, such as GIFT or WFS-SB, on at least one additional backbone under the same visual-unit budget? A matched-panel comparison would be particularly informative for assessing whether CoFi’s relative advantage transfers.
  2. Using the existing predictions, could you characterize the question types or video conditions in which conditioning helps or hurts relative to Unconditioned Sequential? Given the modest aggregate differences in Table 3, this would help explain when accounting for the compressed context materially changes evidence selection.
  3. With the panels, candidate set, budget, and sequential updates fixed, what is the effect of removing only the outer π_j factor in Equation (5)?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Vsc4 (1/3): Cross-backbone Comparison with Strong Selectors

We thank the reviewer for the specific experimental suggestions. We ran all three requested experiments. This part addresses Weakness 1 and Question 1; part 2 addresses Weakness 2 and Question 2.

W1 / Q1. Does CoFi's advantage over strong selectors transfer to other backbones?

Setup. We evaluated Qwen2.5-VL-7B and InternVL2.5-8B on Video-MME, MLVU, and LVB under the same 32-VU budget. Baselines are Uniform Sampling, WFS-SB, and GIFT in their original F32 configurations, plus WFS-SB + Panels, which receives exactly the same four context panels as CoFi and selects 28 focus frames.

Qwen2.5-VL-7B

MethodVideo-MMEMLVULVBAverage
Uniform F3259.8164.0358.5660.80
WFS-SB F3261.5267.2561.1863.32
GIFT F3261.1167.6260.2863.00
WFS-SB + Panels (C4+F28)61.1967.7160.4363.11
CoFi (C4+F28)61.5968.5860.5163.56

InternVL2.5-8B

MethodVideo-MMEMLVULVBAverage
Uniform F3261.0766.4259.4662.32
WFS-SB F3262.1970.0663.2865.18
GIFT F3262.0071.0763.8065.62
WFS-SB + Panels (C4+F28)62.0469.6963.6565.13
CoFi (C4+F28)62.9369.8364.7065.82

Accuracy (%). Bold marks the best result in each column.

Framework-level comparison. On both backbones, CoFi's three-benchmark average is higher than that of every external selector evaluated. Against WFS-SB and GIFT in their original F32 configurations, CoFi is more accurate in 9 of 12 benchmark–selector comparisons. In the matched-panel setting, CoFi exceeds WFS-SB + Panels in all six benchmark–backbone comparisons, so its competitiveness is not explained by the panels alone. The margins are modest (0.08–1.05 points in the matched-panel setting), and GIFT remains stronger on MLVU with InternVL2.5-8B.

These experiments show that the complete training-free CoFi framework remains competitive with strong existing selectors across different Video-LLMs, with modest and benchmark-dependent margins. Part 2 examines the contribution of context conditioning itself.

AuthorsOpenReview ↗

Response to Reviewer Vsc4 (2/3): When Context Conditioning Helps or Hurts

W2 / Q2. What is the practical value of conditioning, and under which question types or video conditions does it help or hurt relative to Unconditioned Sequential?

Using the existing predictions, we examined conditioning at three levels: which frames are selected, whether question-relevant evidence is covered, and downstream accuracy. We report favorable and unfavorable conditions alike. We also broke the results down by question type; the differences between CoFi and Unconditioned Sequential were small and inconsistent, with one exception (MLVU Ego Reasoning, where conditioning helped in every setting we examined), so we focus on the two factors that do separate the cases: focus budget and video duration.

1. Accuracy across focus budgets. With panels, candidates, query weights, and sequential updates held fixed, we set the context to four panels and varied the focus budget (LLaVA-Video-7B).

AllocationMethodVideo-MMEMLVULVB
C4+F28Unconditioned Sequential64.5673.9262.23
C4+F28CoFi64.5274.7062.68
C4+F12Unconditioned Sequential63.1571.7661.18
C4+F12CoFi63.1172.8662.23
C4+F4Unconditioned Sequential60.5569.0959.02
C4+F4CoFi60.7470.8859.54

Accuracy (%). Bold marks the better result within each allocation and benchmark.

The three-benchmark average gain from conditioning rises from 0.40 points at F28 to 0.70 at F12 and 0.83 at F4. MLVU improves at all three budgets, by 1.79 points at F4 (95% CI [0.5, 3.0]). Video-MME shows little benefit under these three C4 allocations.

At F28 the effect is similarly small on the two additional backbones (W2): MLVU improves on both (+0.23 and +0.33 points), and the three-benchmark average differs from Unconditioned Sequential by −0.19 points on Qwen2.5-VL-7B and +0.16 on InternVL2.5-8B. Among the allocations we tested, the default one is thus where conditioning matters least; on LLaVA-Video-7B its observed effect increases at the tighter budgets above.

2. Selection and evidence coverage. At C4+F28, conditioning changes only 0–2% of focus selections in videos up to 60 seconds, about 30% in 3–10-minute videos, and 41% in 15–60-minute videos. At F4, about 55–60% of selections change. On the external LVB referred-moment annotations (±2-second tolerance), conditioning improves evidence coverage by 2.22 points (95% CI [0.6, 3.8]) in 3–10-minute videos and 3.07 points (95% CI [0.9, 5.3]) in 15–60-minute videos. There is no coverage gain in videos up to 60 seconds.

3. Accuracy by video duration (LVB).

LVB durationQuestionsC4+F28C4+F12C4+F4Mean [95% CI]
8–15 s1890.000.00+0.53+0.18 [−0.53, 0.90]
15–60 s1720.00−0.58−2.33−0.97 [−2.33, 0.19]
3–10 min412+0.97+1.70+2.67+1.78 [−0.17, 3.78]
15–60 min564+0.35+1.42−0.18+0.53 [−1.24, 2.30]

CoFi − Unconditioned Sequential, in accuracy points; positive values favor CoFi. Duration groups are those of LVB and cover all 1,337 questions. None of the four mean confidence intervals excludes zero.

  • Where it helps most: 3–10-minute videos, with observed gains of 0.97, 1.70, and 2.67 points as the focus budget shrinks. The mean over the three allocations is not statistically distinguishable from zero at the 95% level.
  • Where it does not help or hurts: in 15–60-second videos, conditioning gives no gain at F28 and an observed drop of 2.33 points at F4 (mean −0.97, 95% CI [−2.33, 0.19]), which is not a statistically reliable negative effect. Shorter videos plausibly leave less complementary evidence to acquire; we have not isolated this mechanism experimentally.
  • 15–60-minute videos: evidence coverage improves, but accuracy is essentially unchanged.

4. Why better evidence coverage does not always raise accuracy. On LVB, conditioning changes whether a question-referred moment is acquired in only 107 of 1,337 questions (66 in favor of CoFi, 41 in favor of Unconditioned Sequential). The coverage gain thus affects a small subset of questions, and acquiring a referred moment does not guarantee that the Video-LLM uses it correctly.

Conclusion. Conditioning changes acquisition most when the video is long relative to the focus budget, and there it improves coverage of question-relevant moments that the compressed context does not represent. Its accuracy contribution depends on video duration and focus budget. We will state these benefits and limits in the revision.

AuthorsOpenReview ↗

Response to Reviewer Vsc4 (3/3): Ablation of the Outer Relevance Factor

W3 / Q3. What is the effect of removing only the outer factor πj\pi_j in Eq. (5)?

We ran this isolated ablation: we removed only the outer candidate-level factor πj\pi_j and kept the compressed context, initial conditioning, admissible candidate pool, inner query weights πt\pi_t, focus budget, and sequential covariance updates unchanged.

Downstream accuracy.

Acquisition scoreVideo-MMEMLVULVBAverage
Without outer πj\pi_j65.3073.8761.4866.88
Full CoFi64.5274.7062.6867.30

Accuracy (%) on LLaVA-Video-7B at C4+F28. Bold marks the better result.

Keeping the outer factor improves MLVU and LVB by 0.83 and 1.20 points, while removing it improves Video-MME by 0.78 points. The factor is therefore not uniformly beneficial, although the full formulation has the higher three-benchmark average.

Acquisition behavior. To examine this trade-off, we measured the query-relevance mass of the selected frames and the query-weighted residual that remains after acquisition.

MeasureVariantVideo-MMEMLVULVB
Selected-frame relevance mass ↑Without outer πj\pi_j0.2390.1260.321
Selected-frame relevance mass ↑Full CoFi0.2580.1610.337
Query-weighted residual remaining ↓Without outer πj\pi_j0.2410.2800.187
Query-weighted residual remaining ↓Full CoFi0.2490.2900.195

Bold marks the preferable value for each measure.

With the outer factor, the selected frames are consistently more query-relevant. Without it, acquisition reduces the query-weighted residual more broadly. The difference also appears in evidence acquisition: on LVB, keeping the outer factor improves the recall of externally annotated question-referred moments by 8.1 points (95% CI [6.3, 9.9]).

Interpretation. The inner weights πt\pi_t reward reducing residual variance at question-relevant locations. The outer factor πj\pi_j additionally favors candidates that are themselves relevant to the question. Removing it favors broad residual reduction; keeping it favors directly relevant observations. This trade-off is consistent with the benchmark-dependent accuracy effects in the ablation. We keep it because a single configuration, with no benchmark-specific tuning, then gives the higher macro-average (67.30% versus 66.88%).

We will add this ablation and its interpretation to the revision.

We hope these experiments address the reviewer's concerns, and we are glad to provide any further analysis during the discussion.

Reviewer_mvWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoFi studies how to combine a compressed overview of a long video with detailed frames relevant to a question. It first packs uniformly sampled frames into panels, estimates what information that context preserves, and then selects additional frames intended to supply missing evidence. The main configuration uses four context panels and 28 standalone focus frames under a 32-visual-unit budget. The central claim is that compression should inform subsequent selection, rather than treating the two decisions independently. The final method of CoFi achieves competitive methods on MLVU, LongVideoBench, and VideoMME.

Critical strengths
  • The overall idea is interesting: select additional observations in light of the compressed context already provided.
  • The provided experimental evidence, both quantitative and qualitative ones, suggest the effectiveness of the methods and provide ways to understand the methods.
Critical weaknesses
  • The motivating is not convincingly established. The paper argues that existing methods select additional frames without accounting for what the compressed context already represents. This is a plausible concern, and the same-panel comparisons in Table 11 are valuable; however, several existing selectors already perform close to CoFi once given that context.

On a side note, this paper reminds me of a related work that also conducts compression and frame selection, but in learnable ways. The authors might want to cite it and also see whether any ways of establishing motivation in that paper could help.

One token per highly selective frame: Towards extreme compression for long video understanding. NeurIPS 2025

  • Which experimental comparison most directly supports the claim that compression informs better subsequent selection? This would be the most substantial contribution from this paper.

  • When compression makes an important detail difficult to see, does CoFi select a detailed frame that restores it? I am curious whether it is possible to see whether this method performs well for video captioning, to have a better understanding of what information is compressed

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Response to Reviewer mvWm (1/2): Motivation for Context-conditioned Acquisition

We thank the reviewer for the thoughtful comments. We agree that the paper should explain more clearly why compressed context should inform subsequent acquisition, given that existing selectors also benefit from the same panels.

W1. Why is context-conditioned acquisition needed, when existing selectors perform close to CoFi once given the same context?

Motivation. The role of an additional full-resolution frame is not fixed. Depending on what the context preserves and what the question asks, a focus frame may show a moment the context does not represent, or it may recover fine detail in a scene the context shows only at low resolution. The question is therefore not whether compressed context can help an existing selector, but whether the acquisition step accounts for what that context already provides.

Panel-insertion comparison. We compare F32 with C4+F28 under the same 32-VU budget. Every C4+F28 method receives the same four context panels.

MethodAllocationVideo-MMEMLVULVB
Focus OnlyF3263.9373.2861.85
Unconditioned SequentialC4+F2864.5673.9262.23
CoFiC4+F2864.5274.7062.68
WFS-SBF3263.1174.1561.85
WFS-SB + PanelsC4+F2864.3773.5162.45
GIFTF3263.3074.5661.32
GIFT + PanelsC4+F2863.6774.0662.15
BOLTF3262.7472.3660.00
BOLT + PanelsC4+F2862.7872.4560.51
FOCUSF3262.4871.3961.33
FOCUS + PanelsC4+F2863.2272.1762.23

Accuracy (%). Bold marks the best result in each column. Focus Only is the F32 reference for both CoFi and Unconditioned Sequential.

Replacing four focus frames with panels raises CoFi's three-benchmark average by 0.95 points (95% CI [0.22, 1.65]). The corresponding changes are 0.81 for FOCUS, 0.55 for Unconditioned Sequential, 0.41 for WFS-SB, 0.23 for GIFT, and 0.21 for BOLT. CoFi's gain is the largest. The differences in gain between methods are not individually significant, so we do not claim that panels help CoFi significantly more than they help other selectors.

What conditioning adds beyond panel insertion. With panels, focus budget, candidates, query weights, and sequential updates held fixed, conditioning changes where the focus budget goes.

BenchmarkUnconditioned SequentialCoFi
Video-MME6.815.38
MLVU2.731.14
LVB8.827.91

Mean number of focus frames per question within ±1 second of a context source frame.

CoFi selects fewer focus frames close to a context source frame. This does not come at the expense of the question-referred moments that the context covers: on LVB, CoFi re-observes them at full resolution at a statistically indistinguishable rate (87.8% versus 88.5%; difference −0.65 points, 95% CI [−1.9, 0.65]). The redirected frames raise the recall of question-referred moments that the context misses by 3.02 points (95% CI [1.5, 4.6]).

The contribution is thus not the addition of panels to the input. It is that focus acquisition is conditioned on what those panels represent, which moves part of the focus budget toward question-relevant moments the context misses without a statistically detectable reduction in re-observation of those it covers. The resulting accuracy gain is modest, and we will say so in the revision; the change in acquisition behavior is what supports the design motivation.

AuthorsOpenReview ↗

Response to Reviewer mvWm (2/2): Supporting Comparison, Detail Recovery, and Related Work

W2. Which experimental comparison most directly supports the claim that compression informs better subsequent selection?

CoFi versus Unconditioned Sequential. The two differ only in whether the initial covariance is conditioned on the compressed context; panels, candidates, query weights, focus budget, and sequential updates are identical. This comparison shows:

  • Accuracy: an average gain of 0.40 points at C4+F28 that grows to 0.70 at F12 and 0.83 at F4 (LLaVA-Video-7B), with MLVU improving at every budget and on all three backbones we tested.
  • Selection: fewer focus frames within ±1 second of a context source frame, with no statistically detectable reduction in re-observation of question-referred moments the context covers (part 1).
  • Evidence: +3.02 points recall of externally annotated question-referred moments not covered by the context (95% CI [1.5, 4.6]).

The matched-panel comparisons with external selectors are supporting evidence. Those selectors choose focus frames independently of the panels they are given. With identical panels, CoFi attains the highest recall of uncovered referred moments on LVB (48.1%, against 47.6% for WFS-SB + Panels and 39.8–43.2% for the others; the margin over WFS-SB is small), and it exceeds WFS-SB + Panels in all six benchmark–backbone accuracy comparisons on Qwen2.5-VL-7B and InternVL2.5-8B, by 0.08–1.05 points (tables in our response to Reviewer Vsc4). We agree that the accuracy margins under matched context are sometimes small; similar accuracy does not imply that the methods acquire the same evidence.

W3. When compression makes a detail hard to see, does CoFi select a detailed frame that restores it?

It can. CoFi treats the compressed context as an imperfect observation, so locations that the context covers keep part of their residual and remain eligible for acquisition (Table 12: most of the benefit comes from modeling observation noise at all, not from cell-specific fidelity). The query weights then decide whether focus budget is spent there. CoFi does not detect which detail was lost, and it does not try to restore details that the question does not need.

Re-observation of covered scenes. We identified 773 LVB question-referred moments within ±2 seconds of a context source frame; in about 89% of them, the context frame shows the same shot. CoFi acquires a full-resolution focus frame within ±2 seconds for 87.8% of them (external selectors: 84.9–88.2%; Unconditioned Sequential: 88.5%). The difference from Unconditioned Sequential is −0.65 points (95% CI [−1.9, 0.65]), so conditioning on the context does not stop CoFi from re-observing these scenes.

Qualitative example. In an LVB L1-Perception question, a speaker's on-screen name caption is illegible in the compressed panel. CoFi selects a full-resolution frame about 0.2 seconds from the annotated moment, where the caption is readable, and answers correctly. FOCUS, given the same panels, selects no full-resolution frame within 60 seconds of the moment and answers incorrectly. We selected examples by a fixed criterion (the referred scene appears in a context panel; CoFi re-acquires it and answers correctly; a comparator with the same panels does not acquire it and answers incorrectly). It yields one case against FOCUS and one against BOLT, and none in the reverse direction.

We therefore claim that conditioning does not stop CoFi from re-observing context-covered, question-relevant scenes, which can restore a lost detail as in the example. We do not claim that CoFi identifies lost details or recovers them more often than other selectors.

Video captioning. The re-observation analysis above addresses whether CoFi looks again at content that is hard to see in the compressed context. We see captioning as a possible extension: the same context-then-focus construction could in principle be applied when the goal is a description and not an answer to a question. We have not evaluated this setting and will note it as future work.

Related work. We will cite and discuss One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (XComp, NeurIPS 2025). XComp learns token-level compression and question-conditioned frame filtering, whereas CoFi builds the visual input for a frozen Video-LLM without training, by conditioning acquisition on compressed context. The settings are related but distinct, and the two approaches may be complementary.

The authors have clarified and addressed my concerns. I keep my original positive review.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

We sincerely thank the reviewer for carefully considering our rebuttal and for acknowledging that our responses have addressed their concerns. We particularly appreciate the reviewer’s constructive questions regarding the motivation for context-conditioned acquisition and the recovery of details obscured by compression. These questions helped us strengthen both the experimental evidence and the explanation of CoFi’s contributions.

We are grateful for the reviewer’s thoughtful engagement and continued positive assessment of our work.

Reviewer_8kRiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoFi, a training-free framework for long video understanding that combines compressed temporal context with query-aware frame selection. The key contribution is the use of context-conditioned residual covariance to guide the selection of additional frames based on what the compressed context is estimated to preserve. A sequential acquisition mechanism further selects query-relevant frames while reducing redundant visual information.

Critical strengths
  • The paper identifies a relevant limitation in existing long video understanding methods: frame selection and visual compression are often performed without explicitly considering what information has already been preserved.
  • The paper provides extensive experiments across three benchmarks, multiple video LLM backbones, and several training-free baselines.
Critical weaknesses
  • The improvements attributable to the proposed context-conditioned residual modelling are relatively limited. In Table 3, CoFi improves over Unconditioned Sequential by only 0.78% and 0.45% on MLVU and LongVideoBench and performing slightly worse on Video-MME. Similarly, the fidelity-aware observation model provides only marginal improvements over constant observation noise in Table 12.
  • The proposed residual covariance is estimated in a frozen visual representation space rather than from the downstream video LLM. However, the paper does not directly validate whether this covariance accurately reflects the information lost during compression or the remaining query relevant information. This makes it difficult to assess whether the proposed acquisition mechanism effectively captures the intended residual uncertainty.
Questions for authors
  • In Table 3, CoFi provides only modest improvements over Unconditioned Sequential, despite context conditioning being a central contribution. Could the authors further clarify when context conditioning is beneficial and provide an analysis of how it changes the selected frames compared with the unconditioned variant?
  • The paper uses residual covariance in the frozen representation space to estimate information not preserved by the compressed context. Could the authors provide evidence that the estimated residual covariance accurately reflects what information is missing from the compressed frames and whether that information is relevant to answering the question?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8kRi (1/2): When Context Conditioning Helps

We thank the reviewer for the careful assessment. We agree that the accuracy gain from conditioning is small at the default allocation. Below we show that the observed gain increases as the focus budget shrinks across the three budgets we tested and that conditioning measurably changes the evidence acquired. This part addresses Weakness 1 and Question 1; part 2 addresses Weakness 2 and Question 2.

W1 / Q1. When is context conditioning beneficial, and how does it change the selected frames?

Controlled comparison. Unconditioned Sequential uses the same context panels, candidate frames, query-affinity weights, and sequential covariance updates as CoFi. The only difference is whether the initial acquisition covariance is conditioned on the compressed context.

Accuracy across focus budgets. We fixed the context at four panels (C4) and varied the focus budget on LLaVA-Video-7B.

AllocationMethodVideo-MMEMLVULVB
C4+F28Unconditioned Sequential64.5673.9262.23
C4+F28CoFi64.5274.7062.68
C4+F12Unconditioned Sequential63.1571.7661.18
C4+F12CoFi63.1172.8662.23
C4+F4Unconditioned Sequential60.5569.0959.02
C4+F4CoFi60.7470.8859.54

Accuracy (%). Bold marks the better method within each allocation and benchmark.

The three-benchmark average gain grows from 0.40 points at F28 to 0.70 at F12 and 0.83 at F4. The trend is clearest on MLVU (0.78, 1.10, 1.79 points). This suggests that conditioning on the context matters more when fewer full-resolution observations are available. Video-MME shows essentially no change across the three evaluated C4 allocations.

How the selected frames change. At C4+F28, conditioning changes about 28–30% of the selected focus frames. To test whether these changes bring in question-relevant evidence, we used the external LVB question-referred moment annotations with a ±2-second matching tolerance.

Acquisition methodRecall of referred moments not covered by context (%)
BOLT + Panels39.8
GIFT + Panels42.6
FOCUS + Panels43.2
Unconditioned Sequential45.0
WFS-SB + Panels47.6
CoFi48.1

A moment counts as covered by context if it lies within the tolerance of a source frame rendered in a compressed panel.

Relative to Unconditioned Sequential, conditioning improves this recall by 3.02 points (95% CI [1.5, 4.6]). Overall referred-moment coverage improves by 2.05 points (95% CI [1.0, 3.1]), and the share of questions with all referred moments covered rises by 2.47 points (95% CI [1.2, 3.8]).

Conditioning does not make CoFi avoid scenes that the context already shows: of 773 LVB referred moments temporally covered by context panels, CoFi still re-acquires 87.8% at full resolution.

Where the benefit is limited. At C4+F28, conditioning changes only 0–2% of focus selections in videos up to 60 seconds, against roughly 30% in 3–10-minute videos and 41% in 15–60-minute videos. Evidence coverage improves in the longer groups, but accuracy gains are not universal: Video-MME is flat under all three allocations, and some short-video settings show negative differences.

In short, conditioning redirects acquisition toward question-relevant moments that the compressed context does not cover, and its observed effect on accuracy is larger at the smaller focus budgets we tested. The duration-wise accuracy analysis, including the cases where conditioning does not help, is in our response to Reviewer Vsc4 (Q2).

AuthorsOpenReview ↗

Response to Reviewer 8kRi (2/2): Validity of the Residual

W2 / Q2. Does the residual reflect information that the compressed context does not preserve, and is that information relevant to the question?

We agree that this should be validated directly. The residual is not meant to estimate the Video-LLM's calibrated semantic uncertainty. It is a representation-space proxy for choosing observations that complement the compressed context.

(a) Does the residual track what compression loses? We analyzed 2,502 videos (Video-MME 883, MLVU 1,122, LVB 497). We excluded 273 videos shorter than 32 seconds, whose full 1-FPS candidate sequences already fit in the 32-VU budget.

For each candidate frame tt, we compared the initial context-conditioned residual variance Ctt(0)C^{(0)}_{tt}, computed from frozen SigLIP features, with the dissimilarity between that frame at full resolution and its most similar compressed context observation, measured in the downstream LLaVA-Video vision tower (initialized from SigLIP and fine-tuned with the Video-LLM, so the two spaces are related but not identical):

ut=1−max⁡ccos⁡(xtV,ℓcV).u_t = 1 - \max_c \cos\left(x_t^V, \ell_c^V\right).

Because the initial residual is question-independent, we compute the Spearman correlation within each video and average over videos.

MeasureVideo-MMEMLVULVBOverall
Temporal-distance baseline0.0310.0810.0290.053
Context-conditioned residual0.1070.2440.1110.169

Within-video Spearman correlation rsr_s with utu_t; higher is better.

The residual reaches rs=0.169r_s = 0.169 overall (95% CI [0.160, 0.179]), which is 0.116 above the temporal-distance baseline (95% CI [0.108, 0.124]). The association persists for candidates within ±2 seconds of a context observation (0.132 versus 0.071), so temporal distance alone does not explain it.

The correlation is modest. We therefore describe the residual as a representation-level acquisition proxy, not as a direct estimate of what the Video-LLM does or does not understand.

(b) Is the missing information relevant to the question? This is tested by the external LVB evidence analysis in part 1: conditioning on the residual raises the recall of annotated question-referred moments outside the context by 3.02 points (95% CI [1.5, 4.6]). These moments are the ones that the LVB annotators marked as the scenes each question refers to, so acquiring more of them means acquiring more question-relevant evidence.

(c) Fidelity-aware observation noise (Table 12). The reviewer notes that fidelity-aware noise adds little over constant noise. We agree and will state this more plainly.

Observation-noise modelVideo-MMEMLVULVB
No Fidelity (νc=0\nu_c = 0)64.2274.4762.23
Constant Noise64.4874.6662.53
Fidelity-aware Noise64.5274.7062.68

Accuracy (%) at C4+F28.

Most of the gain comes from treating the compressed context as an imperfect observation at all (No Fidelity → Constant Noise). Cell-specific fidelity is a smaller refinement on top of that. We also tested whether cell-specific fidelity tracks the downstream distortion of the same cell and found no association (rs=0.01r_s = 0.01), so we will describe it as a minor refinement and not as a validated estimate of per-cell information loss.

Summary. The residual has modest but consistent representation-level support as a proxy for what the compressed context under-represents, and conditioning on it measurably improves the acquisition of question-relevant evidence. Its value lies in conditional acquisition of complementary frames, not in calibrated uncertainty estimation, and we will revise the text to claim only this.

I appreciate the authors for their detailed response. Most of my concerns has been addressed thus I have raised my score to 3.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank the reviewer for their thoughtful reconsideration and constructive feedback. We greatly appreciate their time and consideration.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank the three reviewers for their constructive feedback and valuable suggestions.

The reviews raise three shared concerns: (C1) the motivation for context-conditioned acquisition, (C2) the validity of the acquisition mechanism, and (C3) its effectiveness and generalization relative to existing selectors. We ran additional controlled experiments for each.

Summary of our position. At the default C4+F28 allocation, the stand-alone accuracy gain from context conditioning is small (0.40 points on average), and we will state this plainly in the revision. What the evidence supports is that (i) the full training-free framework is competitive with strong selectors on three Video-LLMs, (ii) conditioning measurably changes which evidence is acquired, and (iii) its observed average accuracy gain increases across the three tested focus budgets as the budget shrinks.

ConcernNew evidenceDetailed in
C1. MotivationPanel-insertion comparison, conditioning ablation, acquisition-overlap analysismvWm
C2. Mechanism and validityFixed-context focus-budget analysis, external evidence annotations, residual validation, component ablations8kRi, Vsc4
C3. Effectiveness and generalizationStrong-selector and matched-panel comparisons on two additional Video-LLMs, full-resolution re-observation analysismvWm, Vsc4

C1. Why should compressed context inform subsequent acquisition? Under a fixed visual budget, compressed context supplies broad temporal coverage and full-resolution focus frames supply detail. The value of an additional focus frame therefore depends on what the context already represents: a selector applied independently of the context allocates its budget as if the context were absent. CoFi conditions its residual covariance on the compressed observations before acquiring focus frames. Our panel-insertion and acquisition-overlap analyses separate this mechanism from simply appending panels to an existing selector.

C2. Does conditioning change acquisition in a meaningful way? We compare CoFi with Unconditioned Sequential, holding the panels, candidate pool, query weights, and sequential updates fixed. With four context panels, the three-benchmark average gain from conditioning is 0.40 points at F28, 0.70 at F12, and 0.83 at F4. On external LongVideoBench (LVB) annotations, conditioning raises the recall of question-referred moments not temporally covered by the context by 3.02 points (95% CI [1.5, 4.6]). Across 2,502 videos, the residual also correlates with the downstream vision tower's dissimilarity to the compressed observations (Spearman rs=0.169r_s = 0.169, versus 0.0530.053 for a temporal-distance baseline). We treat the residual as an acquisition proxy, not as calibrated semantic uncertainty.

C3. Does the framework generalize beyond the original backbone? We added strong-selector comparisons on Qwen2.5-VL-7B and InternVL2.5-8B. Under the same 32-VU budget, CoFi obtains a higher three-benchmark average than every external selector we evaluated on both backbones, and it exceeds WFS-SB given identical context panels in all six benchmark–backbone comparisons (by 0.08–1.05 points). The stand-alone accuracy contribution of conditioning, however, is modest and varies with backbone and video condition. We report it separately from the effectiveness of the full framework, including the settings where it does not help.

The individual responses give protocols, results, and interpretation. We will add these analyses and clarifications to the revision.

AuthorsOpenReview ↗

Update on Context Conditioning across Backbones

We have extended the fixed-context comparison between CoFi and Unconditioned Sequential to Qwen2.5-VL-7B and InternVL2.5-8B at three focus budgets. The full table is in our follow-up to Reviewer Vsc4.

Pooled over the three backbones and three budgets, conditioning has a positive effect on accuracy on average: +0.32 points (95% CI [0.02, 0.62]), with a positive mean on each backbone.

This also corrects one statement. Point (iii) of our summary above and our response to Reviewer 8kRi said that the gain increases as the focus budget shrinks. This holds on LLaVA-Video-7B but does not replicate on the other two backbones, so we restrict it to LLaVA-Video-7B. The evidence-level result (recall of question-referred moments that the context does not cover, +3.02 points) depends only on the selected frames and applies to all three backbones.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

We have uploaded the revised manuscript, incorporating the additional experiments, analyses, and clarifications presented in our rebuttal responses.

Please note that the table and section references in our previously posted rebuttal responses follow the numbering of the original submission, as some numbers have changed in the revised manuscript.

All corresponding results and discussions have been incorporated into the revised manuscript.

We thank the reviewers and Area Chair for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.