- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on distinguishing input sensitivity from meaningful audio use in audio-conditioned visual models. Existing evaluation methods often remove or replace audio and check whether the model's prediction changes. However, these changes do not show whether the model uses the correct audio information or whether audio improves its prediction. CIViC proposes four evaluation checks: input activation, audio replacement, held-out performance, and metric validity. The main contribution is a structured framework that connects these checks with specific claims about audio conditioning, rather than introducing a new evaluation method.
Critical strengths
(1) The paper addresses an important problem in audio-visual evaluation by distinguishing input sensitivity from actual predictive benefits.
(2) CIViC introduces a structured evaluation framework with four evidence gates to assess different aspects of audio conditioning rather than relying on a single test.
(3) The experiments show that changes in model predictions do not always improve performance, highlighting the limitations of audio ablation tests.
Critical weaknesses
(1) The novelty over existing work is unclear. Although the authors mention related studies, they provide limited comparison with existing evaluation methods. It is unclear what CIViC adds beyond combining existing techniques into four evidence gates.
(2) The framework design lacks justification. The authors do not explain why they selected these four checks or whether they are enough to evaluate meaningful audio use.
(3) The evaluation metrics need stronger justification. The paper uses MAE, RMSE, and motion and drift proxies, but it is not always clear how these metrics support the evaluation goals. Some metrics also show conflicting results, without a clear explanation of how they should be interpreted.
(4) The experimental scope is limited. The authors mainly use compact renderers, synthetic settings, and proxy tasks rather than existing audio-visual generation models. This makes it difficult to assess how well CIViC works on real models.
(5) The framework lacks independent validation. The authors do not compare CIViC's decisions with human or expert judgments, controlled ground-truth cases, or simpler evaluation protocols. Without this, it's difficult to determine how reliable the framework is. The presentation needs improvement. Several sections are difficult to follow, particularly the experimental settings, evaluation conditions, and metrics. Clearer explanations would improve readability.
Missing Citations:
Chowdhury, Sanjoy, et al. "Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
Yaman, Dogucan, et al. "Audio-visual speech representation expert for enhanced talking face video generation and evaluation." Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. 2024.
Zhou, Yingjie, et al. "Who is a better talker: Subjective and objective quality assessment for ai-generated talking heads." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
Corona, Enric, et al. "Vlogger: Multimodal diffusion for embodied avatar synthesis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
Kong, Zhe, et al. "Let them talk: Audio-driven multi-person conversational video generation." Advances in Neural Information Processing Systems 38 (2026): 70990-71013.
Questions for authors
(1) What does CIViC contribute beyond combining existing evaluation techniques, and how does it improve upon prior methods?
(2) Section 2 introduces four evidence gates. What justifies their selection, and how do they distinguish meaningful audio use from input sensitivity?
(3) Why was CIViC not evaluated on established audio-driven models such as VLOGGER [4] or MultiTalk [5]?
(4) In Table 8, motion RMSE favors audio removal, while drift favors aligned audio. How should CIViC interpret these conflicting results?
(5) Could the authors evaluate CIViC on controlled cases with known audio dependencies and compare its decisions with expert judgments or existing evaluation methods to demonstrate its reliability?
The paper combines four familiar checks so that input sensitivity is paired with a control and an interpretation rule (Section 2 and Figure 1, pp. 2–3). The submitted experiments do not show that this selection is sufficient or more reliable than existing protocols. A controlled expert comparison or known-outcome evaluation is absent, so the gate decisions remain a proposed interpretation framework rather than independently validated judgments.
The conflicting endpoints are visible in the paper: Study II results change with additional training, and Study III's motion-proxy and drift outcomes favor different conditions (Sections 4–5, pp. 5–9). They should be read as separate proxy measurements, not compressed into one conclusion about audio use. No rendered generator or human quality assessment was run.
The current paper cites VLOGGER and Let Them Talk. It does not include AVTrustBench or the suggested Zhou et al. (2025) and Yaman et al. (2024) references. The submission does not compare the four gates against those methods under a shared task, so it provides no comparative performance result.