acceptodds
Withdrawn from ICLR 2027

Multiple Questions, Shared Evidence: Budget-Aware Grouped Inference for Long-Video QA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long videos are frequently asked about using multiple questions instead of a single one, and a fixed visual-token budget must cover the entire batch; naively re-encoding video content for every question wastes this budget as the batch grows, turning batch video question answering into a resource-allocation problem. Existing methods sit at two extremes: answering each question with its own context repeatedly re-encodes overlapping evidence, while answering the whole batch with one shared context discards question-specific details, and neither adapts to how much the questions actually overlap. We introduce QShare, a batch VideoQA framework that treats which questions should share a context and how the shared budget is spent as one joint decision rather than committing to either extreme. QShare forms an initial partition from question–frame evidence overlap, allocates frames across groups with a submodular greedy procedure that provably approximates the optimal coverage under the shared budget, and iteratively refines the partition — moving, splitting, or merging questions — based on the contexts produced by the allocation; it then answers each group with a single multimodal language-model call. Across CinePile, CG-Bench, and LVBench with two backbones, QShare improves average accuracy by 4.0 points over the strongest baseline under matched batch budgets, reduces the number of LMM calls and the amount of repeated visual prefilling, and keeps its accuracy advantage as the number of questions per video grows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Question Answering, Efficient Multimodal Inference, Cross-Question Evidence Sharing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NNWSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new task, essentially a variant of long-video question answering: given a set of questions, a long video, and a token budget, can you answer all the questions while staying within the budget? They use three datasets, CG-Bench, LVBench, and CinePile, with two backbones (Qwen3-VL-8B and LLaVA-OneVision-7B). There are no trained models, but rather a smarter way of grouping questions and sampling frames. The method has three phases: (1) Cluster (group questions), (2) Greedy Allocation (sample frames), and (3) Refine (move questions between groups). Each group is then answered with one LMM call. Under a strict budget, their method beats the strongest baseline (VisionZip) by 4.0 points on average, cuts LMM calls from 15 to 6.4 per video, and keeps latency comparable to the baselines.

Critical strengths
  • Interesting formulation of the budget problem. Treating "which questions share a context" as a decision, rather than fixing it to per-question or fully joint, is a simple but new idea, and Figure 1 makes the trade-off clear.

  • The controlled grouping comparison is convincing. Random, fixed-size, and text-based grouping all use the same allocation and budget, yet all do worse than grouping by visual evidence overlap. This shows the gain comes from the grouping signal itself, not just from making fewer calls.

  • The grouping is validated beyond accuracy. On LVBench and CG-Bench, the questions QShare groups together actually share the benchmark's annotated evidence intervals.

Critical weaknesses
  • I am not convinced of the importance of the task. In practice, you rarely have many questions per video. The motivation relies on benchmark statistics (e.g., ~15 questions per video in LVBench) rather than real usage.

  • CinePile is not a long-video dataset (clips are a few minutes long), and it shows the smallest gains over the baselines. I am not sure it should be included. I would recommend switching it with MovieRecapsQA or other longer-form datasets.

  • There needs to be more analysis on the dataset themselves. I am worried about possible answer leakage. When related questions are answered in one prompt, the text of one question or its options can hint at the answer to another. This could explain why Fully Joint beats Independent. Suggested Experiment: A text-only control (grouped questions with no video) would help rule this out. Would also rule out any information leakage of the model itself.

  • Further analysis of the datasets is needed. CinePile questions are generated from templates, and CG-Bench is explicitly built around retrieving short clue intervals. It would help to show how much word overlap there is between questions in a batch, and whether the questions truly need long-video context. If the task mostly reduces to retrieve-then-answer, a simple baseline that retrieves the relevant frames for each question and answers directly should be included. CG-Bench's clue-interval setting could also serve as an upper bound.

Questions for authors
  • What is the upper bound? For example, the accuracy when each question gets the full budget, or when using the annotated evidence frames.

  • In my weaknesses, I detailed some questions and extra experiments and analysis section, I believe the work would benefit from. please feel free to respond to the questions mentioned there.

  • Random question, but what happens if you sample from multiple videos? As in you use frames across the dataset rather than the video itself. This might go around the issue of small number of questions assigned to a video.

Reviewer_WUpJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper works on answering multiple questions about the same video under a shared visual-token budget. QShare groups questions according to overlap in their relevant frames, allocates frames across groups using a capped coverage objective, and refines the partition through question moves, splits, and merges. Each resulting group is answered in one LMM call. Experiments cover three benchmarks and two backbones. At a batch budget of 12,544 visual tokens, the reported average accuracy is 47.93%, compared with 43.90% for VisionZip. The paper also includes component ablations, budget sweeps, and latency measurements.

Critical strengths

The paper identifies a practical limitation of both independent and fully joint inference. Related questions can reuse visual evidence, while unrelated questions may need different contexts. Making the sharing pattern adaptive is a sensible contribution, and the proposed pipeline implements this idea without additional LMM calls during planning.

The reported accuracy gains are consistent across all six benchmark–backbone combinations in Table 1. The comparisons with Fully Joint and Independent are particularly useful because they retain the proposed relevance scoring and allocation procedure while changing the grouping strategy. Table 2 also provides evidence that refinement and global allocation contribute beyond the initial grouping.

Critical weaknesses

The coverage objective depends critically on an unspecified normalization. Section 3.3 defines Fi(S)=min⁡(1,∑j∈Srij)F_i(S)=\min(1,\sum_{j\in S}r_{ij}), but only states that relevance scores are normalized to [0,1][0,1]. Different normalizations would produce substantially different allocation behavior. For example, with per-question min–max normalization, selecting the highest-scoring frame would already saturate that question’s coverage. Once all questions saturate, further selections have zero marginal gain, although the algorithm continues spending the budget. The paper does not explain how these ties are resolved or how often saturation occurs. This matters because coverage drives both frame allocation and partition refinement.

Some evaluation details leave the reported gains difficult to interpret. Appendix C restricts answers to A–D, whereas CinePile has five answer choices. This may be a documentation error, but the actual prompt and scoring procedure need to be established. CinePile’s official dataset documentation also distinguishes training and test splits, while the submission lists aggregate dataset statistics without clearly identifying the evaluated split and sample count. Separately, Table 3 describes a common token budget, but Separate Execution repeats each group’s context for every question and would generally consume more visual tokens than QShare. The resulting speedup demonstrates the benefit of sharing a fixed context, but does not establish a speedup at equal token cost. Indeed, Independent is faster than QShare in that table, at lower accuracy.

Questions for authors
  1. What is the exact normalization used to obtain rijr_{ij}? How often do all questions reach the coverage cap before the budget is exhausted, and how are subsequent zero-gain selections handled? These details would help determine whether the reported allocation behavior follows from the stated objective.
  2. Which CinePile version, split, and number of videos/questions were evaluated? Did the actual prompts and answer parser support all five choices, including E? Please clarify whether Appendix C is only an abbreviated template or reflects the implementation.
  3. Could you report the actual total visual-token count for each row of Table 3, especially Separate Execution, and identify where cache construction is included in the latency breakdown? This would clarify which comparisons hold token cost fixed and which hold per-question evidence fixed.
  4. For Figure 4, how were videos with fewer than 32 questions handled? Please specify the evaluated video count and whether the same video subset was used at every batch size. This could materially affect the scalability conclusion.
Reviewer_xKLDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces QShare, a framework for answering multiple questions about the same long video under a shared visual-token budget. It initializes question groups using overlap between question-relevant frames, allocates frames across groups using a capped relevance-coverage objective, and refines the partition through question moves, splits, and merges. Each final group is answered in one multimodal model call. Experiments cover CG-Bench, LVBench, and CinePile with two backbones. Table 1 reports average accuracy of 47.93%, compared with 43.90% for the strongest baseline, while ablations examine frame allocation, grouping, and refinement.

Critical strengths

1 Adaptive sharing is a useful formulation for multi-question video inference. Questions about the same video can share evidence without requiring identical contexts. Making both the question partition and frame allocation decision variables addresses this structure directly. The comparisons against Independent, Fully Joint, Random Grouping, and Semantic Grouping help assess whether evidence-aware grouping provides value beyond simply batching questions.

2The reported accuracy improvements are consistent across the main evaluation settings. QShare exceeds the strongest baseline in each of the six dataset–backbone configurations in Table 1, with a 4.03-percentage-point average improvement. Table 2 further shows lower accuracy when global allocation, group-level selection, or refinement is replaced. Together, these results support the empirical usefulness of the complete framework.

Critical weaknesses

The efficiency comparisons do not consistently preserve the stated resource constraint. Section 4.4 defines Separate Execution as retaining QShare’s group contexts but answering each question independently. Under the paper’s visual-token accounting, this changes the cost from �0��0� to �1��1�, which generally exceeds the original budget. This is a meaningful fixed-evidence comparison, but Table 3 presents it alongside equal-budget comparisons without distinguishing the two conditions. Appendix A also excludes cache construction from the visual-token budget for cross-query reuse methods, while Table 3 does not separately identify that full-model computation within the claimed raw-video-to-answers latency. These ambiguities prevent a clear assessment of the efficiency advantage. QShare’s 54.8-second latency is also higher than Independent’s 51.7 seconds and ReKV’s 54.2 seconds, so the evidence supports an accuracy–latency trade-off rather than a uniform speed advantage.

The coverage objective and its theoretical guarantee require more precise specification. Section 3.3 states that relevance scores are normalized to �2��2�, but does not define the transformation. This is consequential for �3��3�: under per-question min–max normalization, one highest-scoring frame would already saturate a question’s coverage. The objective can also accumulate relevance from redundant frames without establishing that distinct evidence requirements are covered. The paper does not explain how zero-gain allocations are handled when filling the remaining budget. Furthermore, the �4��4� argument assumes a fixed partition and a single cardinality constraint, whereas the implemented allocator also imposes per-group context limits. The guarantee therefore needs explicit assumptions about those limits and should remain clearly separated from guarantees about grouping or answer accuracy.

Questions for authors

1 Can Table 3 distinguish equal-budget comparisons from the fixed-evidence Separate Execution comparison? Where is cache-construction time included for the reuse baselines, and what visual-token total does Separate Execution actually consume?

2 What exact normalization produces the relevance scores, and how does allocation proceed after coverage saturates or marginal gains become zero? Are per-group context limits nonbinding in the evaluated configurations, or does the approximation argument account for them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.