acceptodds
Under review as a conference paper at ICLR 2027

Self-calibrating Agentic Generation and Answering over Multimodal Instructional Collections

Abstract

Instructional recordings such as lectures, tutorials and talks convey information through speech, slides and source documents like research papers. Question an- swering (Q/A) over them therefore requires reasoning across modalities, combin- ing shared with modality-specific information. Existing instructional Q/A bench- marks do not test this together with demanding reasoning constraints such as tem- poral dependencies, multi-hop connections and long-context reasoning. We pro- pose MM-AdvQAGen, a multimodal Q/A generation pipeline that certifies each question’s modality, context and multi-hop requirements through a series of falsi- fiable tests run by blind solvers. It yields MMAv-InstructBench, built from the seed datasets M3AV (Chen et al., 2024) and SlideSpeech (Wang et al., 2024a): 4,624 questions (1,618 and 3,006) for evaluating answering across four dimen- sions, temporal grounding, cross-modal synthesis, long-context reasoning and multi-hop reasoning. For retrieval and answering over such collections we further propose MM-Graph-RAG, a graph-retrieval pipeline that builds one cross-modal graph per recording, linking slide text, speech transcript and related document content, and derives its retrieval settings from the recording itself rather than per- dataset tuning. On MMAv-InstructBench, MM-Graph-RAG improves judge accuracy over the strongest baseline by up to 10.7 percentage points on M3AV and 16.6 on SlideSpeech. On external datasets of other types it likewise wins against the best baselines by good margins.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal RAG, Multimodal collections, Video understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tsEUOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MM-AdvQAGen, an adversarial question-generation pipeline, and MMAv-InstructBench, a 4,624-question benchmark covering temporal grounding, cross-modal synthesis, long-context reasoning, and multi-hop reasoning. It also proposes MM-Graph-RAG, which connects lecture segments, slides, and source documents through a graph and derives retrieval settings from recording-level statistics. Experiments compare the system against multiple baselines using three answerers and two external benchmarks.

Critical strengths

The paper addresses a meaningful problem: distinguishing genuine evidence-dependent answering from shortcut exploitation. Its generation pipeline retains evidence pointers and verification records, making individual questions auditable.The evaluation includes multiple answerers, modality ablations, retrieval measurements, and an evidence-swap experiment with a shared prompt and context budget. These provide stronger attribution than aggregate accuracy alone. Human verification of approximately 20% of the benchmark and an independent audit of the automatic judge are additional strengths.

Critical weaknesses

Incomplete verification of evidence necessity. The blind-solver mechanism tests each modality independently but does not test pairwise combinations. Consequently, failure on three individual modalities does not establish that all three are necessary, despite the claim that 79% of cross-modal questions require all three. Three-hop verification likewise tests individual hops and proper prefixes but omits other subsets, such as the first and final hops together. Stronger leave-one-source-out verification is needed.

Limited generation diversity and difficulty control. P1 drafting uses three predefined template families, potentially narrowing question diversity and introducing recognizable patterns. Only P1 has explicit easy/hard tiers; the remaining dimensions lack comparable difficulty stratification. Benchmark coverage is also uneven: SlideSpeech excludes cross-modal synthesis and hard temporal questions. Thus, the benchmark provides less comprehensive capability assessment than its framing suggests.

Blind-solver failure is not a certificate of necessity. A solver may fail because of limited capability or conservative refusal rather than missing information. The worked OscarEMR example is concerning: the paper passage already names the platform used during the described review, making a paper-only shortcut plausible. Moreover, slides and paper appear sufficient together, weakening the asserted necessity of speech. Human acceptance establishes question quality more directly than unavoidable evidence dependence.

The central self-calibration claim is insufficiently supported. Table 11 contains numerous fixed thresholds, clipping bands, coefficients, and structural limits. The system is therefore partially adaptive rather than free from manually specified settings. More importantly, replacing adaptive quantities with fixed constants changes accuracy by only −0.8 points under gemma and +0.6 under LLaVA-OneVision. These results undermine self-calibration as a central contribution. Portability requires direct adaptive-versus-fixed comparisons on external datasets; successful transfer of the complete system does not isolate calibration’s benefit.

Baseline fidelity and gain attribution remain concerns. Several baselines are author reimplementations with restricted frame sampling or search budgets. Extremely low external-benchmark results warrant implementation and answer-parsing checks. A stronger simple baseline combining hybrid retrieval, temporal filtering, and targeted frames is missing. Removing graph retrieval costs only 2.0 points under gemma, suggesting much of the overall advantage comes from other pipeline components.

Some experimental conclusions are overstated. External gains are small, and the Qwen3-VL SlideSpeech advantage is statistically inconclusive. Controller comparisons involve different question subsets and cannot establish the causal benefit of a second retrieval round. Short-answer tasks also offer limited evidence of broader instructional reasoning

Questions for authors

See above

5 comments
AuthorsOpenReview ↗

Review Answers

We thank the reviewer for this careful review. New experiments use the gemma-3-27B answerer, Nemotron-49B judge and paired exact McNemar p.

1. Necessity: pairwise and hop subsets, solver capability, OscarEMR. We re-attacked all 931 M3AV cross-modal, long-context and multi-hop items with a solver from an unseen model family (Qwen3.8-27B) on every modality pair, leave-one-out segment set and non-prefix hop subset, with abstention allowed and then forbidden.

  • Hops: non-prefix subsets, including first+last, are answerable no more often than the prefixes already attacked (64.3% vs. 64.8%).
  • Modalities: the 79% in §3.1.2 counts items whose gold evidence spans all three sources. Where one source suffices given the gold evidence, which source varies per item, so the best fixed modality covers only 42.7% of answerable items and 133 need two or more: paper+slides is not a general shortcut.
  • OscarEMR: our pipeline restricted to the paper alone loses −11.0 (p=2e-5), speech only −22.2, slides+video only −34.1 (−16.3 to −34.3 even where one modality sufficed in the audit). A paper-only shortcut exists for an oracle handed the right passage, not for a system that must find it.
  • Priors: question only gives 0–0.3% correct (10–22% when forced).

2. Diversity, difficulty, coverage. We grouped items by how many of the 12 baselines (ours excluded) answer them correctly. Hard items, solved by at most 3 baselines, make up 24–51% of each M3AV dimension and 51–59% on SlideSpeech, so every dimension has a hard tier, not only P1. We answer 37.9% (M3AV) and 35.1% (SlideSpeech) of these hard items, about twice the best baseline (18.3% and 18.1%). P1 templates fix only how the time reference is phrased; distinct-3-gram shares are 0.62–0.76 for temporal grounding and 0.82–0.96 for the other dimensions. SlideSpeech recordings come without papers, so it has no cross-modal synthesis items (Limitations (i)); M3AV covers all five dimensions.

3. Self-calibration and change-point segmentation. Table 11's bounds are guardrails for degenerate recordings (App. B); within them, operating values come from each recording's statistics. Freezing all data-derived constants costs −0.8 (n.s.) on M3AV but −2.3 (p=5e-4) on SlideSpeech, whose recordings are 10× longer. The central adaptive step is change-point segmentation: it cuts where the fused speech and on-screen content changes, so each node holds one complete explanation tied to one slide and the aligned/cross_modal edges link whole ideas. Fixed 30 s windows split or merge explanations and break those links. Replacing it is significant under both answerers: −1.9 (gemma, p=0.026) and −2.4 (LLaVA-OV, p=0.002), where freezing the numeric constants moves accuracy only +0.6. Losses concentrate on multi-hop (−7.5 gemma, −4.7 LLaVA-OV) and temporal grounding (−5.2, −4.2).

4. Baseline fidelity and attribution. (a) External lows: they are confined to gemma (e.g., TVQA LLoVi 4.3, VideoTree 3.6 vs. 45.1, 42.9 under LLaVA-OV). Under these pipelines' prompts gemma-3 often answers in free text instead of selecting an option; the identical implementations score normally under LLaVA-OV and Qwen3-VL, where MM-Graph-RAG still leads every TVQA and Video-MME cell (Tables 29–30). (b) Strong baselines, unrestricted budgets: a tool agent with the suggested ingredients (hybrid BM25+dense search, temporal windows, self-chosen 768-px frames, ≤8 steps) trails by 6.4 (p=2e-7) at 3.7 calls/q; a six-agent system (planner, speech/slide/paper agents, critic, answerer, all gemma-3-27B) trails by 8.2 (p=4e-10) at 7.0 calls/q, and by 5.1 on SlideSpeech (p=0.002). We use 3.0. (c) "Graph only −2": that ablation still used our graph's nodes; hybrid retrieval over standard 30 s chunks loses −2.4 (p=0.011; −4.7 multi-hop).

5. Strength of conclusions. We lead all six main-table columns, significant in five (p ≤ 2e-3); for Qwen3-VL/SlideSpeech (within noise, Table 1) our judge-free REM lead is 34.6 vs. 28.8. Controller: the requested paired test gives 66.4 vs. 66.4 with/without it (gemma) and 61.1 vs. 60.8 (LLaVA-OV); on the 985 questions where it requested a second round, +0.2 (n.s.), against −0.5 where it did not fire. The routing table (App. E) records where it fires; the paired test shows no result depends on it. Without it we keep full accuracy at about 2 calls/q, against 3.7 for the tool agent and 7.0 for the multi-agent system. Short answers are deliberate for verifiable scoring (Limitations (ii)); the hard tiers show the benchmark is far from saturated.

6. Ethics. As outlined in our Ethics statement, we use publicly available corpora in accordance with their terms of use. We release only questions, answers, and supporting evidence pointers, without sharing any audio or video or identifying individuals. All verifiers are informed researchers who participate voluntarily, and we retain only their ratings for individual items.

AuthorsOpenReview ↗

Request for review

We have submitted our rebuttal and addressed the questions and concerns raised in your review. We would greatly appreciate it if you could review our responses and, if they adequately address your concerns, kindly reconsider your score.

AuthorsOpenReview ↗

Request to Review the Rebuttal

Dear Reviewer tsEU,

We would like to respectfully follow up regarding our rebuttal, as the rebuttal period is approaching its deadline. We have addressed the questions and concerns raised in your review and would greatly appreciate it if you could review our responses. If you find that our rebuttal adequately addresses your concerns, we kindly request that you reconsider your score.

Thank you very much for your time and consideration.

Reviewer_6fcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates question answering over instructional videos and multimodal educational materials, with two main contributions.

First, the authors propose MM-AdvQAGen, which constructs a multimodal question-answering benchmark, MMAv-InstructBench, through adversarial question generation and verification. The benchmark contains 4,624 question-answer pairs covering four reasoning dimensions: temporal grounding, cross-modal synthesis, long-context reasoning, and multi-hop reasoning. The core idea is to use solvers with access to restricted evidence to filter out questions that may contain unimodal or partial-evidence shortcuts.

Second, the authors introduce MM-Graph-RAG, which constructs a cross-modal graph from speech transcripts, slides, and related documents. The framework integrates adaptive segmentation, graph-based retrieval, evidence sufficiency assessment, and visual verification to improve complex multimodal question answering.

The authors compare their method against 12 baselines on M3 AV and SlideSpeech, evaluating performance with three vision-language models. Furthermore, they evaluate the generalization ability of their method on TVQA and Video-MME, and conduct ablation studies and retrieval analyses to investigate the contributions of different components.

Critical strengths
  • The paper introduces a multimodal question-answering benchmark covering temporal grounding, cross-modal synthesis, long-context reasoning, and multi-hop reasoning. The adversarial data generation process and human quality verification improve the reliability of the dataset, making it a useful contribution to the research community.
  • The paper compares the proposed method against 12 baselines and evaluates its effectiveness across multiple models and datasets. It also provides ablation studies, retrieval quality analyses, and human evaluations, offering relatively comprehensive empirical support for the proposed approach.
Critical weaknesses
  • The paper does not sufficiently discuss existing studies on instructional video evaluation [1][2], cross-modal question construction and shortcut mitigation [3][4][5], and graph-based video retrieval [6]. As a result, the specific contributions beyond prior work are not clearly established. Additionally, the authors are encouraged to discuss high-quality multimodal data construction studies such as Bee [7] and clarify their connections to and differences from the proposed benchmark generation framework.
  • The failure of a solver under restricted evidence does not necessarily prove that the corresponding modalities or reasoning steps are indispensable. The authors are encouraged to evaluate additional combinations of modalities and subsets of reasoning steps to further validate the reliability of the reasoning capability labels in the benchmark.
  • Some baselines use uniformly sampled video frames, whereas the proposed method employs targeted visual evidence selection. This makes it difficult to isolate the improvements attributable to graph-based retrieval. A more controlled comparison using the same visual inputs and verification strategies across different retrieval methods would help clarify the contribution of graph retrieval itself.
  • The paper evaluates generalization on TVQA and Video-MME, but these benchmarks differ from the instructional video scenarios emphasized in this work. If compatible with the input and evaluation settings, additional experiments on instructional video QA benchmarks such as Video-MMLU [1] and Video-MMMU [2] would provide stronger evidence of generalization to the intended application domain.

[1] Enxin Song et al. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark. ICCV Workshops, 2025.

[2] Kairui Hu et al. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. ACL, 2026.

[3] Alon Talmor et al. MultiModalQA: Complex Question Answering over Text, Tables and Images. ICLR, 2021.

[4] Lin Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models? NeurIPS, 2024. (MMStar)

[5] Xiang Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL, 2025.

[6] Honghao Fu et al. VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG. ACL, 2026.

[7] Yi Zhang et al. Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs. ICLR, 2026.

Questions for authors

same with weaknesses.

7 comments
AuthorsOpenReview ↗

Review Answers

We thank the reviewer for the constructive review. The related work is now in the revised PDF (§2), and each other point is answered with new experiments (gemma-3-27B answerer, the paper's Nemotron-49B judge, paired exact McNemar p).

W1. Related work (revised PDF, §2, "Instructional and lecture datasets"). All seven works are now cited and positioned.

  • Video-MMLU [1] and Video-MMMU [2] evaluate lectures from the video alone, and MultiModalQA [3] does multi-hop QA over text, tables and images without a timeline. MMAv grounds every item in a recording, its slides and its source paper, with gold evidence pointers.
  • MMStar [4] and MMMU-Pro [5] remove image-free shortcuts; our blind-solver gates generalize this to every modality, segment and hop.
  • VideoStir [6] structures long-video RAG spatio-temporally; our graph additionally links slides and source documents and calibrates per recording.
  • Bee [7] curates training data, whereas MM-AdvQAGen certifies evaluation items.

W2. Do the labels reflect real necessity? More modality combinations and step subsets. We re-attacked all 931 M3AV cross-modal, long-context and multi-hop items with a solver from a model family never used in generation (Qwen3.8-27B, vision). It saw every modality pair, every leave-one-out segment set and every non-prefix hop subset (e.g., first+last hop), with abstention allowed and then forbidden, so refusal cannot explain failure.

  1. Step subsets: non-prefix subsets are answerable no more often than the prefixes our pipeline already attacks (64.3% vs. 64.8% of views): no additional shortcut, which confirms the existing gate.
  2. Modalities: even where one source suffices once the gold evidence is handed over, which source varies per item, so the best fixed modality covers only 42.7% of answerable items, and 133 items need two or more. A system must locate the right source per question, exactly the capability the cross-modal label denotes.
  3. Real single-source systems fail decisively (499 cross-modal items; video withheld for text-only variants):
SystemAllItems where one gold modality suffices
Full67.578.9
Paper only56.5 (p=2e-5)62.6 (p=1e-6)
Speech only45.3 (p=4e-18)52.6 (p=9e-15)
Slides+video only33.5 (p=6e-36)44.6 (p=8e-21)

On the 133 multi-source items our lead grows to +18.9 vs. CLaMR (p=1e-4), +18.2 vs. ColPali and +22.7 vs. Video-RAG. A question-only control rules out priors: 0–0.3% correct with abstention allowed and 10–22% when forced. The paper's App. A.7 agrees: removing the paper costs −16.4 on paper-labelled items and +0.1 elsewhere, so the labels predict which evidence each item depends on.

W3. Same visual inputs and verification across retrievers. We ran the controlled comparison: only the retriever changes, while frames, visual verification and answer stage are identical. On M3AV, hybrid BM25+dense over standard chunks loses −2.4 (p=0.011), dense-only −2.0 (p=0.020), uniform context −29.4; the loss concentrates on multi-hop (−4.7) and hard temporal grounding (−4.3). In the reverse direction, systems that choose targeted 768-px frames from any window themselves still trail: a tool agent by 6.4 (p=2e-7) and a six-role multi-agent team with a dedicated slide/frame specialist by 8.2 (p=4e-10), by 12.6 and 14.2 on cross-modal synthesis. The gain is attributable to graph retrieval, not to frame selection.

W4. Generalization. On both external benchmarks MM-Graph-RAG is the most accurate system under every answerer (App. F, Tables 29–30):

  • TVQA (no slides or documents): 55.7 / 60.2 / 60.4 under gemma / LLaVA-OV / Qwen3-VL, ahead of the strongest baseline, CLaMR, in every column and +15.9 over zero-shot.
  • Video-MME: 63.9 / 65.0 / 65.0, leading the strongest baseline by +1.2 / +2.7 / +3.0.
  • Video-MMLU / Video-MMMU: both are video-only, so the source documents and slide–paper links our method is built to exploit do not exist there; Video-MMLU also scores sentence-length explanations, while MMAv and our answer stage target short verifiable answers. They would mainly measure answer style rather than retrieval. Given the reviewer's compatibility condition, §2 now positions them as complementary.

References. [1] Song et al., Video-MMLU, ICCV-W 2025. [2] Hu et al., Video-MMMU, ACL 2026. [3] Talmor et al., MultiModalQA, ICLR 2021. [4] Chen et al., MMStar, NeurIPS 2024. [5] Yue et al., MMMU-Pro, ACL 2025. [6] Fu et al., VideoStir, ACL 2026. [7] Zhang et al., Bee, ICLR 2026.

AuthorsOpenReview ↗

Request for review

We have submitted our rebuttal and addressed the questions and concerns raised in your review. We would greatly appreciate it if you could review our responses and, if they adequately address your concerns, kindly reconsider your score.

AuthorsOpenReview ↗

Request to Review the Rebuttal

Dear Reviewer 6fcE,

We would like to respectfully follow up regarding our rebuttal, as the rebuttal period is approaching its deadline. We have addressed the questions and concerns raised in your review and would greatly appreciate it if you could review our responses. If you find that our rebuttal adequately addresses your concerns, we kindly request that you reconsider your score.

Thank you very much for your time and consideration.

Reviewer_iLtZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark and a retrieval framework for question answering over instructional recordings containing speech, slides, and related documents. Its question-generation pipeline, MM-AdvQAGen, filters questions using blind solvers given incomplete evidence, producing 4,624 questions covering temporal grounding, cross-modal synthesis, long-context reasoning, and multi-hop reasoning. The proposed MM-Graph-RAG constructs a graph connecting evidence across modalities and time, adapts retrieval settings to each recording, and uses an LLM to check evidence sufficiency before answering. It outperforms the evaluated baselines across three answerers, although improvement sizes vary considerably.

Critical strengths

The paper introduces a carefully constructed benchmark that explicitly tests for shortcuts by checking whether questions can be answered using individual modalities, isolated segments, or incomplete reasoning chains. Its graph structure captures meaningful relationships among speech, slides, and documents, supporting retrieval of evidence distributed across modalities and time. The evaluation includes multiple answerers, external benchmarks, human audits, and component ablations.

Critical weaknesses

The main weakness is that the paper does not establish the incremental value of graph construction over capable agentic retrieval. An agent equipped with tools to search transcripts, inspect slides, and retrieve document passages could dynamically discover the relationships encoded in the graph, but the restricted agentic baselines do not adequately test this alternative. Comparisons under matched inference budgets are needed to determine whether the graph provides an accuracy or efficiency advantage. Furthermore, with Gemma on M3AV, replacing the graph with flat dense retrieval reduces overall accuracy by approximately two points, while replacing adaptive statistics with fixed settings changes accuracy by less than one point. These modest contributions leave the necessity of the method’s complexity unclear. Improvements are also uneven across answerers, with the Qwen3-VL gain on SlideSpeech failing to reach statistical significance.

Questions for authors
  1. How does MM-Graph-RAG compare with an agent that iteratively searches transcripts, inspects slides, and retrieves document passages under comparable token or latency budgets? Could such an agent recover the same relationships without constructing a graph?
  2. Given that replacing adaptive settings with fixed constants changes accuracy by less than one point, under what conditions does self-calibration provide a measurable advantage, and what evidence supports its claimed portability?
7 comments
AuthorsOpenReview ↗

Response to reviewer iLtZ - Part 1

We thank the reviewer for the constructive feedback and for recognizing the strengths of our benchmark construction, cross-modal graph representation, and comprehensive evaluation. Unless otherwise specified, the new experiments use the gemma-3-27B answerer, the Nemotron-49B judge, and paired exact McNemar tests. Our detailed response is listed as follows:

Q1. How does MM-Graph-RAG compare with capable agentic retrieval under comparable inference budgets?

We consider two additional graph-free agentic baselines, providing both a capable single-agent configuration and a coordinated multi-agent alternative

Single agent : ReAct with hybrid BM25+dense search over 20 s transcript+slide windows, search_paper, read(t0,t1) and look(t0,t1) (targeted 768-px frames).

Multi-agent system (MAS): six cooperating agents, each run on gemma-3-27B-it, the same model as our answerer: a planner writes one query per source; speech, slide (reads frames) and paper agents search their own source in parallel; a critic checks their reports and sends agents back for a second round; an answerer writes the answer. The single agent also uses gemma-3-27B-it, and all search uses our Qwen3-Embedding-8B embedder. Neither baseline constructs or traverses a cross-modal graph.

SystemLLM calls/qPrompt tok/qM3AVSlideSpeech (1k)
Single agent, ≤4 turns3.053.9k58.654.1
Single agent, ≤8 turns3.715.5k59.955.5
MAS, 1 round4.884.1k56.346.4
MAS + critic, 2 rounds7.045.6k58.152.1
MM-Graph-RAG2.966.2k66.457.2

Accuracy vs. inference cost: MM-Graph-RAG consistently outperforms these agentic baselines while requiring fewer LLM calls per question, although the token usage is more. On M3AV, the strongest single agent configuration trails by 6.5 points (p=2e-7), while the two-round multi-agent system trails by 8.2 points (p=4e-10), despite requiring approximately 2.4× as many LLM calls. The latter also trails by 5.1 points on SlideSpeech (p=0.002).

Why explicit cross-modal relationships help. MM-Graph-RAG constructs relationships between spoken segments, the slides visible during those segments (aligned edges), and semantically corresponding paper passages (cross_modal edges). These relationships are established before question answering using embeddings and recording-specific thresholds, without additional LLM calls. Consequently, retrieval can reach a relevant paper passage through the lecture segment that discusses it, even when the passage has limited lexical overlap with the question. Further, agents must find these links at query time, and miss them. Our analysis reveals two limitations of the evaluated graph-free agents:

1. Incomplete source exploration: Across 1,618 questions, the single agent issued 1,827 lecture searches but only 275 paper searches, despite 500 questions requiring paper evidence.

2. Lack of explicit cross-source association: The multi-agent system searches papers for every question, yet trails MM-Graph-RAG by 14.2 points on cross-modal synthesis. Accessing multiple sources independently does not necessarily establish which passages correspond to particular lecture moments.

The largest differences occur on relational reasoning categories: hard temporal grounding (−19.3), multi-hop reasoning (−15.1), and cross-modal synthesis (−14.2) for the full multi-agent team relative to MM-Graph-RAG. Conversely, agents perform better on some questions that explicitly specify the relevant temporal window (+3 to +6 points), where relational discovery is less important.

AuthorsOpenReview ↗

Request for review

We have submitted our rebuttal and addressed the questions and concerns raised in your review. We would greatly appreciate it if you could review our responses and, if they adequately address your concerns, kindly reconsider your score.

AuthorsOpenReview ↗

Request to Review the Rebuttal

Dear Reviewer iLtZ,

We would like to respectfully follow up regarding our rebuttal, as the rebuttal period is approaching its deadline. We have addressed the questions and concerns raised in your review and would greatly appreciate it if you could review our responses. If you find that our rebuttal adequately addresses your concerns, we kindly request that you reconsider your score.

Thank you very much for your time and consideration.

Comments on the paper

2 comments

Dear Reviewer iLtZ and tsEU,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best,
AC

AuthorsOpenReview ↗

Summary of author response

Dear AC and reviewers,

Thank you for your active engagement while overseeing the review process. We have addressed the concerns through additional experiments, analyses, and manuscript revisions (changes highlighted in red). We summarize the key concerns and how we address them.

  • Value of graph construction over agentic retrieval [iLtZ, tsEU, 6fcE]: We evaluated stronger graph-free baselines, including a ReAct agent and a six-role multi-agent team with hybrid retrieval and targeted frame inspection. MM-Graph-RAG achieves 66.4% on M3AV, outperforming the strongest single agent (59.9%, p=2e-7) and multi-agent team (58.1%, p=4e-10), with fewer LLM calls.

  • Reliability of benchmark verification and cross-modal synthesis [6fcE, tsEU]: We re-evaluated 931 questions using an independent solver across modality combinations and reasoning-hop subsets. Importantly, 133 questions require evidence from multiple sources even with gold evidence provided. On this subset, MM-Graph-RAG outperforms CLaMR by 18.9 points (p=1e-4), ColPali by 18.2, and Video-RAG by 22.7, demonstrating improvements beyond single-source retrieval. We also clarify the distinction between strict multi-source necessity and question-dependent source selection.

  • Benefits of self-calibration [iLtZ, tsEU]: Direct adaptive-versus-fixed comparisons show that freezing data-derived constants reduces SlideSpeech accuracy by 2.3 points (p=5e-4), while adaptive change-point segmentation contributes up to 2.4 points (p=0.002). These results demonstrate the benefits of recording-specific adaptation, particularly when collection characteristics differ.

  • Statistical testing and efficiency [iLtZ, tsEU]: MM-Graph-RAG leads all six main-table comparisons, with statistically significant improvements in five (p ≤ 2e-3); the remaining comparison is reported as inconclusive (p=0.079).

  • Generalization to instructional-video datasets [6fcE]: We additionally evaluated SciVideoBench, an unseen benchmark of narrated scientific instructional videos, on 138 questions sampled from 40 videos. MM-Graph-RAG achieves the highest accuracy among 11 systems (50.7%), outperforming VideoRAG (48.6%), CLaMR (47.1%), and Video-RAG (42.0%). Video-level bootstrap analysis (10,000 resamples) confirms positive 95% confidence intervals against several baselines, including Video-RAG (+8.7 [1.6, 15.5]) and Agentic RAG (+15.2 [7.0, 24.6]). Together with TVQA and Video-MME, this provides evidence of transfer across three external video-QA benchmarks. We also clarify that Video-MMLU (Review Notes, Take Quiz) and Video-MMMU (Perception, Comprehension, Adaptation) evaluate complementary instructional-video capabilities rather than the full cross-document retrieval setting of our work.

Overall, these additional results strengthen the evidence for our two central contributions:

(1) an auditable benchmark for evidence-grounded multimodal instructional QA

(2) a self-calibrating graph-based retrieval framework that explicitly connects evidence across modalities and time.

We thank you again for the feedback regarding our work and actively facilitating the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.