- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ProV-Mem, a training-free wrapper around a frozen Qwen2.5-VL-7B for proactive streaming VQA. It queries the backbone at a fixed stride (K=3 frames) with a silence-recovery trigger, prompts for a concise grounded description of the current window, and suppresses near-duplicate outputs via prefix-LCS similarity to the last emission. A union-bound lemma relates this three-stage factorization to a joint policy, and CAUC extends PAUC with redundancy, frequency, and length penalties. On ProactiveVideoQA, it reports the best CAUC on all four tasks, the best PAUC on WEB/EGO/TV, and 1.3 s per-response latency.
Critical strengths
- The proposed method is training-free on a frozen backbone, and its prompts and per-scenario settings are disclosed. It is strong on EGO against published numbers. Its gains over the same backbone’s published chunked baseline are large on EGO and TV. Broad sweeps (η, K, τ_sup, κ, ω) and a 3B/7B/32B scaling study are included.
- PAUC does not charge for redundant, overly frequent, or verbose emissions. Reporting a cost-aware score and Rep% alongside PAUC is a step in the right direction.
Critical weaknesses
- The baselines do not support the SOTA claims.
- The re-run baselines are much weaker than their published results. The same GPT-4.1 judge is used, yet most offline open-source baselines score far below their numbers in the ProactiveVideoQA paper.
- Qwen2.5-VL-7B is ProV-Mem's own backbone. Its PAUC falls from 52.7/46.5/36.5/29.3 (published) to 35.82/39.27/29.46/23.25 (this paper).
- IXC-2.5 on EGO falls from 50.3 to 13.19. That is below the 25 a model scores by never responding.
- The online models, by contrast, roughly match their published values.
- The "Baseline" in Fig. 3 is never defined and matches no row of Table 1.
- The strongest systems are missing. GPT-4.1 and GPT-4.1-mini are the best published systems, and both are omitted.
- GPT-4.1-mini's published PAUC on TV (59.4) and VAD (47.7) exceeds ProV-Mem's (57.43, 32.10).
- On WEB, ProV-Mem leads the published LLaVA-OV result (55.0) by under 3 points.
- The scales do not match. MMDuet2 is a 3B model, but it is compared against a 7B ProV-Mem. At matched 3B scale (Fig. 3), ProV-Mem leads on CAUC only on EGO.
- The re-run baselines are much weaker than their published results. The same GPT-4.1 judge is used, yet most offline open-source baselines score far below their numbers in the ProactiveVideoQA paper.
- The reported numbers are inconsistent, and some contradict the paper's conclusions.
- VAD CAUC. Under the default setting it is 31.68 in Tables 1–2 but ≈28.6 in Figs. 3–5. At ≈28.6, ProV-Mem would rank below four proprietary baselines. So the claim of "best CAUC across all scenarios" rests on the inconsistent number.
- EGO PAUC without de-duplication. It is reported as 71.78 (Table 2), 70.64 (App. C.5) and 66.5 (Fig. 4). Sec. 4.5 builds its argument on the lowest value, yet Table 2 shows that removing M_ep improves PAUC on all four tasks.
- Silence recovery. Turning it off (κ=∞) beats the default κ=4 on both PAUC and CAUC, yet κ=4 is still presented as optimal.
- Cooldown. Rep% doubles, from 16% to 32%, when the cooldown τ_sup grows from 30 s to 60 s. A longer cooldown can only suppress more, so Rep% should not rise.
- Scaling. EGO PAUC drops from 7B to 32B (70.9 → ≈65), which contradicts the claimed monotonic scaling.
- The theory and the memory design are overstated.
- Lemma 1 is weaker than presented.
- It is a union bound that holds for any two policies, so it does not justify this particular decomposition.
- Its coupling ignores how the system's own outputs change later states. This removes compounding error by construction.
- The ε terms are never estimated.
- Assumption 1 is never used in the proof.
- Sec. 4.5's discussion of η and ρ also contradicts App. B.
- The "three-tier memory" is thin. It amounts to a frame counter, a fixed prompt and a 200-character prefix of the last response. The generator sees only the current 3-frame window and, apparently, not the question. This sits at odds with the introduction's criticism of methods that discard visual history.
- Lemma 1 is weaker than presented.
Questions for authors
- Baselines.
- Why are most re-run offline baselines far below the published ProactiveVideoQA numbers for the same models and judge (up to ~38 PAUC points)?
- Why were GPT-4.1 and GPT-4.1-mini dropped?
- How was Time measured for each baseline?
- Inputs and tuning.
- Is the question q passed to the backbone at all?
- If not, how does a minimal baseline score: the same backbone captioning every K frames with a generic prompt and no gating?
- How were K, η, τ_sup and κ chosen without a held-out split?
- Which numbers are correct?
- For the default VAD CAUC, is it 31.68 or ≈28.6?
- For EGO without de-duplication, is it 71.78, 70.64 or 66.5?
- Why is Rep% non-monotonic in τ_sup?
2 comments
Looking forward to your feedback aXx5
Dear Reviewer aXx5,
We appreciate your thoughtful comments and suggestions for our manuscript.
As the discussion deadline is approaching, we kindly invite you to check our rebuttal. Please feel free to raise any additional questions, and we will respond in a timely manner.
If our responses have addressed your concerns, we would appreciate your consideration in updating your score. Thank you for your time. We look forward to your feedback.
Best Regards,
Authors of Paper 15303
Response to Reviewer aXx5
We would like to express our sincere gratitude to you for your review and detailed suggestions.
W1: Concerns regarding baseline comparability and SOTA claims