- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an agentic framework for VAU that aims to address insufficient visual evidence in existing methods. The framework consists of an Observer that generates temporally localized evidence, a frozen Verifier that evaluates evidence support across four dimensions, and a Mem-Optimizer that maintains a bounded evidence pool. Experiments on HIVAU-70k demonstrate improvements in VAU performance.
Critical strengths
-
This paper introduces a structured evidence memory mechanism that explicitly evaluates and maintains visual evidence across four dimensions
-
The proposed framework achieves competitive performance on HIVAU-70k, with improvements across multiple evaluation metrics.
Critical weaknesses
- The experiments do not directly compare evidence sufficiency between baseline methods and MemVAU. Therefore, it remains unclear whether the performance improvements are primarily attributed to acquiring sufficient visual evidence. Besides, the proposed framework employs Qwen3-VL as its visual backbone, while the comparisons mainly include older models such as Qwen2-VL.
- The framework relies on a frozen Verifier to evaluate evidence sufficiency. The Observer is trainable while the Verifier keeps frozen. And the Verifier does not engage in the optimization of Observer. It remains unclear whether the two components are sufficiently aligned.
- The proposed Mem-Optimizer maintains a bounded evidence pool. However, there are no comparisons with simpler strategy like unbounded evidence accumulation or Top-K selection, which would better demonstrate the necessity of the proposed Mem-Optimizer.
- The Observer is optimized with GRPO, but Table 2(b) shows relatively limited improvements over the SFT Observer
Questions for authors
Could the authors clarify the implementation details of the baselines in Table 2? Table 2 is one of the most important ablation studies supporting the effectiveness of the proposed framework. However, the implementation details of the baseline methods are insufficiently described.
2 comments
Thanks for your response. Howerver, some concerns remain. First, the main baseline results are largely drawn from the Holmes-VAU paper, and comparisons with more recent models would strengthen the evaluation. Second, the effectiveness and necessity of the Mem-Optimizer unclear, particularly without comparisons against simpler evidence accumulation strategies. Overall, my main concerns have not been fully addressed. I will maintain my rating.
Response to Reviewer Y65y
We thank the reviewer for the comments. The existing experiments examine major performance factors through visual-agent scale controls, fixed-Observer comparisons, and training ablations.
1. Backbone differences and evidence-processing gains
O2/V2/R8 reduces both the Observer and Verifier to 2B and still exceeds Holmes-VAU on five metrics. For example, BLEU-V is 0.661 versus 0.566, and METEOR-E is 0.375 versus 0.165. Thus, some gains persist with smaller visual agents.
Table 2 provides a more direct internal comparison. Raw evidence and MemVAU-SFT use the same SFT Observer, and adding verification and structured memory improves all eight Event/Video metrics. Holding the candidate-generation model fixed supports the contribution of the joint evidence-processing pipeline.
The additional diagnostics measure Verifier cross-prompt consistency and Reporter evidence utilization, each interpreted according to its stated definition.
2. Coordination between the Observer and frozen Verifier
During training, the 4B Judge ranks candidate sets by anomaly-type accuracy, four-dimensional coverage, and diversity. During inference, the 8B Verifier scores the corresponding video segments. Its outputs inform pool updates and the next-round snapshot, allowing the Observer to condition subsequent candidates on retained evidence and unmet dimensions.
Freezing the Verifier keeps this feedback reference fixed. Table 2 compares SFT and GRPO Observers while keeping the Verifier, Mem-Optimizer, and Reporter fixed. GRPO improves all four Video metrics, providing task-level evidence of effective coordination through this interface.
3. Rationale for the Mem-Optimizer
An entry’s value depends on the current pool’s coverage. Equation (13) measures its contribution through the gap increase, surplus loss, and secondary-support loss caused by removing it. An entry that fills a How gap can therefore receive explicit pool-level value. The state is recomputed after each replacement, and S3 preserves all threshold passes.
The capacity analysis also shows a concrete trade-off. In Table 6, increasing K from five to seven lowers the full-four-dimensional pass rate from 91.60% to 88.62%, without consistent improvements in text metrics. K=3 achieves the highest text metrics in this sweep, while K=5 achieves the highest operational coverage. These results support controlling and tuning memory capacity.
The capacity sweep does not directly compare Top-K and unrestricted accumulation. The present rationale for the Mem-Optimizer rests on its explicit pool-level objective, update properties, and full-pipeline results.
4. Contribution of GRPO
Table 8 shows improvements on nine of twelve metrics, slight decreases on two, and one unchanged result. All four Video metrics improve. GRPO therefore provides an incremental improvement over the existing pipeline.
The combined pipeline gain is already present with SFT. MemVAU-SFT achieves CIDEr-E/V of 1.628/1.760, compared with 0.384/0.419 for Raw evidence. The two comparisons separately characterize joint evidence processing and Observer-training optimization.
5. Implementation and control design in Table 2
Table 2 contains three types of controls. For direct inference, Single-pass and CoT use one call, Self-correction uses two, and Multi-round uses three, with shared preprocessing. The evidence-processing comparison holds the SFT Observer fixed. The training comparison fixes the Mem-Optimizer and Reporter and changes only Observer training. Tables 7–8 provide complete results at all three granularities.