- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CoFi addresses long-video question answering with a frozen Video-LLM under a limited visual-input budget. It first constructs query-independent temporal panels and conditions a state–trajectory RBF covariance prior on the compressed context, using representation fidelity to set observation noise. It then sequentially selects query-relevant standalone frames from the residual covariance.
Critical strengths
- The method gives a coherent way to condition selection on an existing compressed representation. The positive-semidefinite kernel and posterior updates propagate information between related candidate locations and provide sequential redundancy control. This makes the design more substantive than simply concatenating temporal panels and relevance-ranked frames.
- On LLaVA-Video-7B, CoFi achieves the highest accuracy among the compared methods on MLVU and LongVideoBench and the second-highest on Video-MME (Table 1). Table 3 shows that sequential updating improves over Static Acquisition by 2.78, 2.62, and 1.42 percentage points, respectively. Importantly, Table 11 also equips existing selectors with the same context panels and focus budget. These controls, together with the analyses of visual budgets, representations, and approximation stability, provide meaningful evidence for the overall construction strategy.
Critical weaknesses
- The primary LLaVA-Video-7B evaluation includes multiple established selectors and visual-representation methods. However, the Qwen2.5-VL-7B and InternVL2.5-8B evaluations compare CoFi only with Uniform Sampling. Appendix Table 16 provides duration-wise results but does not add stronger selection baselines. These experiments establish that CoFi’s improvement over uniform sampling transfers across backbones, but they do not establish whether its advantage over strong existing selectors also transfers. This limits the evidence for the broader applicability of its relative performance advantage.
- Context conditioning is a central design choice, but the independent gains shown in Table 3 are modest. Additional comparisons with and without conditioning on more Video-LLM backbones would help clarify the practical value of this mechanism.
- The outer relevance factor π_j in Equation (5) has not been separately ablated. Its additional contribution beyond relevance-based candidate filtering and query-weighted residual reduction remains unclear.
Questions for authors
- Could you evaluate a representative strong selector, such as GIFT or WFS-SB, on at least one additional backbone under the same visual-unit budget? A matched-panel comparison would be particularly informative for assessing whether CoFi’s relative advantage transfers.
- Using the existing predictions, could you characterize the question types or video conditions in which conditioning helps or hurts relative to Unconditioned Sequential? Given the modest aggregate differences in Table 3, this would help explain when accounting for the compressed context materially changes evidence selection.
- With the panels, candidate set, budget, and sequential updates fixed, what is the effect of removing only the outer π_j factor in Equation (5)?
6 comments
Response to Reviewer Vsc4 (2/3): When Context Conditioning Helps or Hurts
W2 / Q2. What is the practical value of conditioning, and under which question types or video conditions does it help or hurt relative to Unconditioned Sequential?
Using the existing predictions, we examined conditioning at three levels: which frames are selected, whether question-relevant evidence is covered, and downstream accuracy. We report favorable and unfavorable conditions alike. We also broke the results down by question type; the differences between CoFi and Unconditioned Sequential were small and inconsistent, with one exception (MLVU Ego Reasoning, where conditioning helped in every setting we examined), so we focus on the two factors that do separate the cases: focus budget and video duration.
1. Accuracy across focus budgets. With panels, candidates, query weights, and sequential updates held fixed, we set the context to four panels and varied the focus budget (LLaVA-Video-7B).
| Allocation | Method | Video-MME | MLVU | LVB |
|---|---|---|---|---|
| C4+F28 | Unconditioned Sequential | 64.56 | 73.92 | 62.23 |
| C4+F28 | CoFi | 64.52 | 74.70 | 62.68 |
| C4+F12 | Unconditioned Sequential | 63.15 | 71.76 | 61.18 |
| C4+F12 | CoFi | 63.11 | 72.86 | 62.23 |
| C4+F4 | Unconditioned Sequential | 60.55 | 69.09 | 59.02 |
| C4+F4 | CoFi | 60.74 | 70.88 | 59.54 |
Accuracy (%). Bold marks the better result within each allocation and benchmark.
The three-benchmark average gain from conditioning rises from 0.40 points at F28 to 0.70 at F12 and 0.83 at F4. MLVU improves at all three budgets, by 1.79 points at F4 (95% CI [0.5, 3.0]). Video-MME shows little benefit under these three C4 allocations.
At F28 the effect is similarly small on the two additional backbones (W2): MLVU improves on both (+0.23 and +0.33 points), and the three-benchmark average differs from Unconditioned Sequential by −0.19 points on Qwen2.5-VL-7B and +0.16 on InternVL2.5-8B. Among the allocations we tested, the default one is thus where conditioning matters least; on LLaVA-Video-7B its observed effect increases at the tighter budgets above.
2. Selection and evidence coverage. At C4+F28, conditioning changes only 0–2% of focus selections in videos up to 60 seconds, about 30% in 3–10-minute videos, and 41% in 15–60-minute videos. At F4, about 55–60% of selections change. On the external LVB referred-moment annotations (±2-second tolerance), conditioning improves evidence coverage by 2.22 points (95% CI [0.6, 3.8]) in 3–10-minute videos and 3.07 points (95% CI [0.9, 5.3]) in 15–60-minute videos. There is no coverage gain in videos up to 60 seconds.
3. Accuracy by video duration (LVB).
| LVB duration | Questions | C4+F28 | C4+F12 | C4+F4 | Mean [95% CI] |
|---|---|---|---|---|---|
| 8–15 s | 189 | 0.00 | 0.00 | +0.53 | +0.18 [−0.53, 0.90] |
| 15–60 s | 172 | 0.00 | −0.58 | −2.33 | −0.97 [−2.33, 0.19] |
| 3–10 min | 412 | +0.97 | +1.70 | +2.67 | +1.78 [−0.17, 3.78] |
| 15–60 min | 564 | +0.35 | +1.42 | −0.18 | +0.53 [−1.24, 2.30] |
CoFi − Unconditioned Sequential, in accuracy points; positive values favor CoFi. Duration groups are those of LVB and cover all 1,337 questions. None of the four mean confidence intervals excludes zero.
- Where it helps most: 3–10-minute videos, with observed gains of 0.97, 1.70, and 2.67 points as the focus budget shrinks. The mean over the three allocations is not statistically distinguishable from zero at the 95% level.
- Where it does not help or hurts: in 15–60-second videos, conditioning gives no gain at F28 and an observed drop of 2.33 points at F4 (mean −0.97, 95% CI [−2.33, 0.19]), which is not a statistically reliable negative effect. Shorter videos plausibly leave less complementary evidence to acquire; we have not isolated this mechanism experimentally.
- 15–60-minute videos: evidence coverage improves, but accuracy is essentially unchanged.
4. Why better evidence coverage does not always raise accuracy. On LVB, conditioning changes whether a question-referred moment is acquired in only 107 of 1,337 questions (66 in favor of CoFi, 41 in favor of Unconditioned Sequential). The coverage gain thus affects a small subset of questions, and acquiring a referred moment does not guarantee that the Video-LLM uses it correctly.
Conclusion. Conditioning changes acquisition most when the video is long relative to the focus budget, and there it improves coverage of question-relevant moments that the compressed context does not represent. Its accuracy contribution depends on video duration and focus budget. We will state these benefits and limits in the revision.
Response to Reviewer Vsc4 (3/3): Ablation of the Outer Relevance Factor
W3 / Q3. What is the effect of removing only the outer factor in Eq. (5)?
We ran this isolated ablation: we removed only the outer candidate-level factor and kept the compressed context, initial conditioning, admissible candidate pool, inner query weights , focus budget, and sequential covariance updates unchanged.
Downstream accuracy.
| Acquisition score | Video-MME | MLVU | LVB | Average |
|---|---|---|---|---|
| Without outer | 65.30 | 73.87 | 61.48 | 66.88 |
| Full CoFi | 64.52 | 74.70 | 62.68 | 67.30 |
Accuracy (%) on LLaVA-Video-7B at C4+F28. Bold marks the better result.
Keeping the outer factor improves MLVU and LVB by 0.83 and 1.20 points, while removing it improves Video-MME by 0.78 points. The factor is therefore not uniformly beneficial, although the full formulation has the higher three-benchmark average.
Acquisition behavior. To examine this trade-off, we measured the query-relevance mass of the selected frames and the query-weighted residual that remains after acquisition.
| Measure | Variant | Video-MME | MLVU | LVB |
|---|---|---|---|---|
| Selected-frame relevance mass ↑ | Without outer | 0.239 | 0.126 | 0.321 |
| Selected-frame relevance mass ↑ | Full CoFi | 0.258 | 0.161 | 0.337 |
| Query-weighted residual remaining ↓ | Without outer | 0.241 | 0.280 | 0.187 |
| Query-weighted residual remaining ↓ | Full CoFi | 0.249 | 0.290 | 0.195 |
Bold marks the preferable value for each measure.
With the outer factor, the selected frames are consistently more query-relevant. Without it, acquisition reduces the query-weighted residual more broadly. The difference also appears in evidence acquisition: on LVB, keeping the outer factor improves the recall of externally annotated question-referred moments by 8.1 points (95% CI [6.3, 9.9]).
Interpretation. The inner weights reward reducing residual variance at question-relevant locations. The outer factor additionally favors candidates that are themselves relevant to the question. Removing it favors broad residual reduction; keeping it favors directly relevant observations. This trade-off is consistent with the benchmark-dependent accuracy effects in the ablation. We keep it because a single configuration, with no benchmark-specific tuning, then gives the higher macro-average (67.30% versus 66.88%).
We will add this ablation and its interpretation to the revision.
We hope these experiments address the reviewer's concerns, and we are glad to provide any further analysis during the discussion.
Response to Reviewer Vsc4 (1/3): Cross-backbone Comparison with Strong Selectors
We thank the reviewer for the specific experimental suggestions. We ran all three requested experiments. This part addresses Weakness 1 and Question 1; part 2 addresses Weakness 2 and Question 2.
W1 / Q1. Does CoFi's advantage over strong selectors transfer to other backbones?
Setup. We evaluated Qwen2.5-VL-7B and InternVL2.5-8B on Video-MME, MLVU, and LVB under the same 32-VU budget. Baselines are Uniform Sampling, WFS-SB, and GIFT in their original F32 configurations, plus WFS-SB + Panels, which receives exactly the same four context panels as CoFi and selects 28 focus frames.
Qwen2.5-VL-7B
InternVL2.5-8B
Accuracy (%). Bold marks the best result in each column.
Framework-level comparison. On both backbones, CoFi's three-benchmark average is higher than that of every external selector evaluated. Against WFS-SB and GIFT in their original F32 configurations, CoFi is more accurate in 9 of 12 benchmark–selector comparisons. In the matched-panel setting, CoFi exceeds WFS-SB + Panels in all six benchmark–backbone comparisons, so its competitiveness is not explained by the panels alone. The margins are modest (0.08–1.05 points in the matched-panel setting), and GIFT remains stronger on MLVU with InternVL2.5-8B.
These experiments show that the complete training-free CoFi framework remains competitive with strong existing selectors across different Video-LLMs, with modest and benchmark-dependent margins. Part 2 examines the contribution of context conditioning itself.