- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a new task, essentially a variant of long-video question answering: given a set of questions, a long video, and a token budget, can you answer all the questions while staying within the budget? They use three datasets, CG-Bench, LVBench, and CinePile, with two backbones (Qwen3-VL-8B and LLaVA-OneVision-7B). There are no trained models, but rather a smarter way of grouping questions and sampling frames. The method has three phases: (1) Cluster (group questions), (2) Greedy Allocation (sample frames), and (3) Refine (move questions between groups). Each group is then answered with one LMM call. Under a strict budget, their method beats the strongest baseline (VisionZip) by 4.0 points on average, cuts LMM calls from 15 to 6.4 per video, and keeps latency comparable to the baselines.
Critical strengths
-
Interesting formulation of the budget problem. Treating "which questions share a context" as a decision, rather than fixing it to per-question or fully joint, is a simple but new idea, and Figure 1 makes the trade-off clear.
-
The controlled grouping comparison is convincing. Random, fixed-size, and text-based grouping all use the same allocation and budget, yet all do worse than grouping by visual evidence overlap. This shows the gain comes from the grouping signal itself, not just from making fewer calls.
-
The grouping is validated beyond accuracy. On LVBench and CG-Bench, the questions QShare groups together actually share the benchmark's annotated evidence intervals.
Critical weaknesses
-
I am not convinced of the importance of the task. In practice, you rarely have many questions per video. The motivation relies on benchmark statistics (e.g., ~15 questions per video in LVBench) rather than real usage.
-
CinePile is not a long-video dataset (clips are a few minutes long), and it shows the smallest gains over the baselines. I am not sure it should be included. I would recommend switching it with MovieRecapsQA or other longer-form datasets.
-
There needs to be more analysis on the dataset themselves. I am worried about possible answer leakage. When related questions are answered in one prompt, the text of one question or its options can hint at the answer to another. This could explain why Fully Joint beats Independent. Suggested Experiment: A text-only control (grouped questions with no video) would help rule this out. Would also rule out any information leakage of the model itself.
-
Further analysis of the datasets is needed. CinePile questions are generated from templates, and CG-Bench is explicitly built around retrieving short clue intervals. It would help to show how much word overlap there is between questions in a batch, and whether the questions truly need long-video context. If the task mostly reduces to retrieve-then-answer, a simple baseline that retrieves the relevant frames for each question and answers directly should be included. CG-Bench's clue-interval setting could also serve as an upper bound.
Questions for authors
-
What is the upper bound? For example, the accuracy when each question gets the full budget, or when using the annotated evidence frames.
-
In my weaknesses, I detailed some questions and extra experiments and analysis section, I believe the work would benefit from. please feel free to respond to the questions mentioned there.
-
Random question, but what happens if you sample from multiple videos? As in you use frames across the dataset rather than the video itself. This might go around the issue of small number of questions assigned to a video.