- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MM-AdvQAGen, an adversarial question-generation pipeline, and MMAv-InstructBench, a 4,624-question benchmark covering temporal grounding, cross-modal synthesis, long-context reasoning, and multi-hop reasoning. It also proposes MM-Graph-RAG, which connects lecture segments, slides, and source documents through a graph and derives retrieval settings from recording-level statistics. Experiments compare the system against multiple baselines using three answerers and two external benchmarks.
Critical strengths
The paper addresses a meaningful problem: distinguishing genuine evidence-dependent answering from shortcut exploitation. Its generation pipeline retains evidence pointers and verification records, making individual questions auditable.The evaluation includes multiple answerers, modality ablations, retrieval measurements, and an evidence-swap experiment with a shared prompt and context budget. These provide stronger attribution than aggregate accuracy alone. Human verification of approximately 20% of the benchmark and an independent audit of the automatic judge are additional strengths.
Critical weaknesses
Incomplete verification of evidence necessity. The blind-solver mechanism tests each modality independently but does not test pairwise combinations. Consequently, failure on three individual modalities does not establish that all three are necessary, despite the claim that 79% of cross-modal questions require all three. Three-hop verification likewise tests individual hops and proper prefixes but omits other subsets, such as the first and final hops together. Stronger leave-one-source-out verification is needed.
Limited generation diversity and difficulty control. P1 drafting uses three predefined template families, potentially narrowing question diversity and introducing recognizable patterns. Only P1 has explicit easy/hard tiers; the remaining dimensions lack comparable difficulty stratification. Benchmark coverage is also uneven: SlideSpeech excludes cross-modal synthesis and hard temporal questions. Thus, the benchmark provides less comprehensive capability assessment than its framing suggests.
Blind-solver failure is not a certificate of necessity. A solver may fail because of limited capability or conservative refusal rather than missing information. The worked OscarEMR example is concerning: the paper passage already names the platform used during the described review, making a paper-only shortcut plausible. Moreover, slides and paper appear sufficient together, weakening the asserted necessity of speech. Human acceptance establishes question quality more directly than unavoidable evidence dependence.
The central self-calibration claim is insufficiently supported. Table 11 contains numerous fixed thresholds, clipping bands, coefficients, and structural limits. The system is therefore partially adaptive rather than free from manually specified settings. More importantly, replacing adaptive quantities with fixed constants changes accuracy by only −0.8 points under gemma and +0.6 under LLaVA-OneVision. These results undermine self-calibration as a central contribution. Portability requires direct adaptive-versus-fixed comparisons on external datasets; successful transfer of the complete system does not isolate calibration’s benefit.
Baseline fidelity and gain attribution remain concerns. Several baselines are author reimplementations with restricted frame sampling or search budgets. Extremely low external-benchmark results warrant implementation and answer-parsing checks. A stronger simple baseline combining hybrid retrieval, temporal filtering, and targeted frames is missing. Removing graph retrieval costs only 2.0 points under gemma, suggesting much of the overall advantage comes from other pipeline components.
Some experimental conclusions are overstated. External gains are small, and the Qwen3-VL SlideSpeech advantage is statistically inconclusive. Controller comparisons involve different question subsets and cannot establish the causal benefit of a second retrieval round. Short-answer tasks also offer limited evidence of broader instructional reasoning
Questions for authors
See above
5 comments
Request for review
We have submitted our rebuttal and addressed the questions and concerns raised in your review. We would greatly appreciate it if you could review our responses and, if they adequately address your concerns, kindly reconsider your score.
Request to Review the Rebuttal
Dear Reviewer tsEU,
We would like to respectfully follow up regarding our rebuttal, as the rebuttal period is approaching its deadline. We have addressed the questions and concerns raised in your review and would greatly appreciate it if you could review our responses. If you find that our rebuttal adequately addresses your concerns, we kindly request that you reconsider your score.
Thank you very much for your time and consideration.
Review Answers
We thank the reviewer for this careful review. New experiments use the gemma-3-27B answerer, Nemotron-49B judge and paired exact McNemar p.
1. Necessity: pairwise and hop subsets, solver capability, OscarEMR. We re-attacked all 931 M3AV cross-modal, long-context and multi-hop items with a solver from an unseen model family (Qwen3.8-27B) on every modality pair, leave-one-out segment set and non-prefix hop subset, with abstention allowed and then forbidden.
2. Diversity, difficulty, coverage. We grouped items by how many of the 12 baselines (ours excluded) answer them correctly. Hard items, solved by at most 3 baselines, make up 24–51% of each M3AV dimension and 51–59% on SlideSpeech, so every dimension has a hard tier, not only P1. We answer 37.9% (M3AV) and 35.1% (SlideSpeech) of these hard items, about twice the best baseline (18.3% and 18.1%). P1 templates fix only how the time reference is phrased; distinct-3-gram shares are 0.62–0.76 for temporal grounding and 0.82–0.96 for the other dimensions. SlideSpeech recordings come without papers, so it has no cross-modal synthesis items (Limitations (i)); M3AV covers all five dimensions.
3. Self-calibration and change-point segmentation. Table 11's bounds are guardrails for degenerate recordings (App. B); within them, operating values come from each recording's statistics. Freezing all data-derived constants costs −0.8 (n.s.) on M3AV but −2.3 (p=5e-4) on SlideSpeech, whose recordings are 10× longer. The central adaptive step is change-point segmentation: it cuts where the fused speech and on-screen content changes, so each node holds one complete explanation tied to one slide and the
aligned/cross_modaledges link whole ideas. Fixed 30 s windows split or merge explanations and break those links. Replacing it is significant under both answerers: −1.9 (gemma, p=0.026) and −2.4 (LLaVA-OV, p=0.002), where freezing the numeric constants moves accuracy only +0.6. Losses concentrate on multi-hop (−7.5 gemma, −4.7 LLaVA-OV) and temporal grounding (−5.2, −4.2).4. Baseline fidelity and attribution. (a) External lows: they are confined to gemma (e.g., TVQA LLoVi 4.3, VideoTree 3.6 vs. 45.1, 42.9 under LLaVA-OV). Under these pipelines' prompts gemma-3 often answers in free text instead of selecting an option; the identical implementations score normally under LLaVA-OV and Qwen3-VL, where MM-Graph-RAG still leads every TVQA and Video-MME cell (Tables 29–30). (b) Strong baselines, unrestricted budgets: a tool agent with the suggested ingredients (hybrid BM25+dense search, temporal windows, self-chosen 768-px frames, ≤8 steps) trails by 6.4 (p=2e-7) at 3.7 calls/q; a six-agent system (planner, speech/slide/paper agents, critic, answerer, all gemma-3-27B) trails by 8.2 (p=4e-10) at 7.0 calls/q, and by 5.1 on SlideSpeech (p=0.002). We use 3.0. (c) "Graph only −2": that ablation still used our graph's nodes; hybrid retrieval over standard 30 s chunks loses −2.4 (p=0.011; −4.7 multi-hop).
5. Strength of conclusions. We lead all six main-table columns, significant in five (p ≤ 2e-3); for Qwen3-VL/SlideSpeech (within noise, Table 1) our judge-free REM lead is 34.6 vs. 28.8. Controller: the requested paired test gives 66.4 vs. 66.4 with/without it (gemma) and 61.1 vs. 60.8 (LLaVA-OV); on the 985 questions where it requested a second round, +0.2 (n.s.), against −0.5 where it did not fire. The routing table (App. E) records where it fires; the paired test shows no result depends on it. Without it we keep full accuracy at about 2 calls/q, against 3.7 for the tool agent and 7.0 for the multi-agent system. Short answers are deliberate for verifiable scoring (Limitations (ii)); the hard tiers show the benchmark is far from saturated.
6. Ethics. As outlined in our Ethics statement, we use publicly available corpora in accordance with their terms of use. We release only questions, answers, and supporting evidence pointers, without sharing any audio or video or identifying individuals. All verifiers are informed researchers who participate voluntarily, and we retain only their ratings for individual items.