acceptodds
Withdrawn from ICLR 2027

QMA-RAG: Query-Conditioned Multi-Granularity Context Construction for Long-Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-granularity retrieval offers local observations and broader context for long-video question answering, but several retrieved nodes may refer to the same source frames. Constructing a limited visual context therefore requires deciding how these semantic views jointly affect which observations are selected. We introduce QMA-RAG, which combines multi-granularity support on shared source clips before selecting the visual context. A reusable graph links atomic, event, and macro descriptions to their supporting clips. Question-conditioned retrieval supplies candidates whose scores are fused on these clips; selection balances the resulting support with temporal needs and repeated coverage. Selected clips are converted into a bounded set of unique frames with original temporal information and retrieved structured support. Across MLVU, Video-MME, and LongVideoBench with three answering backbones, the complete system improves Uniform inference in all nine settings by 3.80 percentage points on average. On MLVU, construction from identical candidates improves over exact frame deduplication by 1.66 points. Removing cross-level fusion while retaining the full selector reduces accuracy by 1.93 points, supporting the role of aggregated semantic support in source selection. Answering-input and component controls examine how the resulting context and its supporting design choices contribute to answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-video understanding, Video question answering, Retrieval-augmented generation, Query-adaptive retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2FSsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents QMA-RAG, a way to help AI models answer questions about long videos, which have way too many frames to look at all at once. A common trick, retrieval-augmented generation (RAG), first writes short descriptions of the video at three zoom levels: atomic clips (about one minute each), events (a single activity), and macro scenes (several events together). It then picks the frames whose descriptions match the question. The problem is that these levels often describe the same moment, so the limited frame slots get filled with repeats. QMA-RAG fixes this by remembering which clips each description came from. A rule-based planner first decides what the question needs: a small detail, the order of events, or the big picture. The method then adds up the scores from all three levels for each clip (cross-level fusion) and picks clips that are useful but not repetitive. Finally, it gives the answering model up to 320 frames in time order, along with timestamps, retrieved text, and a verification step.

The paper tested this on three benchmarks (MLVU, Video-MME, and LongVideoBench) with three different answering models. QMA-RAG beat Uniform sampling every time, by 3.80 points on average. It also beat the graph-based method Vgent by 1.60 points on average while running a bit faster. It did slightly better than the frame selection methods AKS and DIG too. But when you look at the chosen frames alone, it does about as well as AKS and a bit worse than DIG. So much of its edge comes from the extra text and verification it gives the model, not from picking better frames.

Critical strengths

The paper has a few real strengths, mostly in how the problem is set up and how the experiments are run. The idea itself is not very new, but some parts of the work are solid.

A clear problem. The paper points out that when you search at several levels (atomic clips, events, and macro scenes), different results often point to the same moment, which wastes frame slots. This is a real issue, and Figure 1 explains it clearly. Combining scores per clip (cross-level fusion) is a reasonable way to deal with it.

Useful experiments. The authors test their main idea by keeping everything else the same and changing only how frames are picked. In that test, the method does 1.66 points better than simply removing duplicate frames. They also add the extra inputs one at a time (timestamps, retrieved text, verification), which shows how much each one adds. This also shows that the chosen frames alone are not better than the frame selection methods.

Consistent results. QMA-RAG beats Uniform sampling and Vgent in all nine combinations of benchmarks and answering models, though some of the margins over Vgent are small. It needs no training, works with different answering models, and builds each video’s memory once and reuses it. It also runs a bit faster than Vgent.

Critical weaknesses

The paper has some useful experiments, but there are real problems with how fair its comparisons are, how much is left unexplained, and how far its results can be trusted.

  1. When only the chosen frames are given to the model, QMA-RAG does about the same as AKS and worse than DIG (Table 6 vs. Table 2). Its win over these methods comes from extra text and a verification step that they never get, so the comparison isn't really fair (Section 4.1, "Rerun protocol and comparison scope").

  2. The rule-based planner and the scoring weights were built on MLVU, and all the ablations and parameter tests were also run on MLVU (Section 4.5; Section 5 opening; Appendix B.3). Every parameter sweep peaks exactly at the chosen setting (Figure 3), which suggests the method may be fitted to this one benchmark.

  3. Some of the gains are tiny, like 0.30 points over DIG on LongVideoBench, which is only about 4 questions (Table 2; Table 9), or 0.33 points over Vgent with LongVU on Video-MME (Table 1). Without significance tests, which the authors admit are missing (Appendix E), we can't tell if these gains are real or just noise.

  4. Several important parts are never explained. These include the verification step (Section 3.4), the model that writes the video descriptions (Section 3.2), the planner's actual rules, and some terms in the scoring equations (Appendix B.3, Eqs. 4–5). This makes the method hard to reproduce or judge, and no code release is promised (Reproducibility Statement).

  5. Most ablations drop the counting task from about 59% to about 42% (Table 7), and that drop alone explains most of the overall losses. Since counting has only 206 questions (Table 8), many of the paper's conclusions rest on a few dozen questions.

  6. The method mostly combines known ideas: video graphs, weighted scores, picking relevant but non-repetitive clips, and removing duplicates (Sections 2 and 3.3). It also depends on many hand-picked weights and word-based rules (Appendix B.3) that seem fragile. A simpler planner version even scored below plain Uniform sampling (Table 3).

  7. The paper cites close RAG methods like AdaVideoRAG and ViG-RAG, plus VideoTree and A.I.R. (Section 2), but doesn't compare against any of them. It only runs Uniform, Vgent, AKS, and DIG (Section 4.1). It also doesn't test on hour-long videos or on benchmarks built to require several separate pieces of evidence, even though that is exactly what the method claims to handle.

Questions for authors
  1. Do AKS and DIG get the same retrieved text, timestamps, and verification step as QMA-RAG, or only their selected frames? On MLVU, QMA-RAG's frames alone score 72.21%, about the same as AKS (72.22%) and below DIG (72.72%). Is it correct, then, that the gains over these methods come from the extra inputs rather than from better frame choices? This matters for how I judge the paper's main contribution.

  2. What exactly does the verification step do, and what is the "symbolic answering" mentioned in the failure examples? What text evidence is passed to the answering model, and which model writes the clip, event, and scene descriptions? Does the Uniform baseline also receive subtitles? Without these details, it's hard to tell whether the comparisons with the baselines are fair.

  3. Were the planner rules, the scoring weights, and the thresholds chosen by looking at results on the same MLVU questions that were used for the main results and ablations? If so, the parameter tests peaking exactly at the chosen settings, and the larger gains on MLVU than on the other benchmarks, may partly reflect tuning rather than a general improvement.

Reviewer_yY2wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes QMA-RAG, a multi-granularity retrieval framework for long-video QA. It links atomic, event, and macro-level evidence to shared source clips, fuses their relevance before selection, and constructs a bounded visual context with temporal and structured information. Experiments across three benchmarks and three VLM backbones demonstrate consistent accuracy improvements.

Critical strengths

The paper identifies a retrieval-to-context mismatch caused by overlapping evidence from different semantic granularities.

Controlled comparisons using identical candidates, together with detailed component ablations, provide evidence that source-aligned fusion contributes beyond simple deduplication.

Critical weaknesses

Limited methodological novelty. The proposed system relies largely on existing hierarchical retrieval, hand-designed scoring rules, greedy selection, and source-index bookkeeping. The conceptual advance beyond established video RAG systems needs stronger justification.

Limited efficiency evidence under constrained budgets. The main setting allows up to 320 frames, and complete-system gains also depend on structured support and verification. The advantages of the selection mechanism under smaller frame and total computation budgets remain insufficiently established.

Questions for authors

How does QMA-RAG compare with existing retrieval methods under strictly matched end-to-end computational and visual-token budgets, particularly with 32 or 64 frames?

How sensitive are the results to inaccurate event boundaries, missing captions, or errors in the rule-based query router?

Reviewer_BJPmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies frame selection for video QA, and proposes to use multiple levels of granularity to retrieve relevant context from a source video. Specifically, the authors build a graph with three levels (Atomic, Event, and Macro) of memory from a video, and then retrieve frames with text and visual relevance. Then they are fused, and finally constructed to have a set to use as input context to VLMs. In the experiment, the proposed approach outperforms the presented graph-based and non-graph based frame selection baselines on three benchmarks.

Critical strengths
  • The proposed idea of utilizing multiple level of granularity for retrieval is simple yet interesting.
Critical weaknesses
  • Writing is hard to follow and difficult to parse, and it seems AI was extensively used for it. e.g., in Sec. 3.4, "Observations at 424 s and 484 s preserve ..." does not make sense as it looks to refer a specific example, not a expected general statement. Also it is desirable to add mathematical formulation to better convey the proposed methodology.
  • Motivation is unclear. The authors claims retrieval-to-context gap as a motivation and proposes their approach to address it, but it was not clear how it solves. Furthermore, the novelty is questionable. Graph-based RAG for video QA already exist, e.g., Vgent (Shen et al., 2025b), and the novelty compared to the prior work is unclear or rather incremental.
  • Experiment is not comprehensive. The proposed approach is compared with only three baselines along with simple uniform selection, and more recent and relevant baselines (such as [1-4]) needs to be compared as this research area is recently quite active.

References

[1] Zhu et al., FOCUS: Efficient Keyframe Selection for Long Video Understanding, ICLR 2026

[2] Zou et al., A.I.R.: Adaptive, Iterative, and Reasoning-based Frame Selection for Video Question Answering, ICLR 2026

[3] Li et al., Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding, CVPR 2026

[4] Li et al., FrameOracle: Learning What to See and How Much to See in Videos, ICML 2026

Questions for authors
  • Can you clarify the novelty of the proposed approach compared to prior work?
  • Can you add more recent and relevant baselines in the experimental evaluation? Also can you add proprietary VLM backbones as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.