acceptodds
Under review as a conference paper at ICLR 2027

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction

Abstract

Removing sound from a visual predictor may change its output, but that test leaves the model's use of the intended audio information uncertain. We call this protocol Conditional Input and Visual Interpretation Checks (CIViC). It links a specific interpretation to the intervention, task comparison, or target check needed to support it. We apply the protocol to an actor-disjoint face-image task, a synthetic feature-mixture task, and a fresh source-video-disjoint SpeakerVid proxy task. The static face-image test detects a small response to replacing audio with another actor's; a separate temporal test does not establish an audio benefit on its endpoint. In a post-confirmation comparison on a second actor-disjoint raster target, four constant-audio fits score better than their audio-trained counterparts, despite a large response when audio is removed at test time. In the synthetic feature-mixture task, the added condition changes a 128-coordinate visual proxy but raises withheld-composition error from 0.1707 for audio alone to 0.1807 at 1,000 steps; the gap shrinks with longer fitting. In the source-video-disjoint SpeakerVid task, a same-target audio-zero arm has lower motion-proxy RMSE than aligned audio in all four held-out conditions and all five paired seeds on four accelerator platforms using one source pack. The same direction appears within each of the four held-out source-video groups, but a secondary long-form drift proxy favors aligned audio in all four conditions. Audio-only, dummy-condition, and shuffled-condition arms expose distinct endpoint behavior, showing why an output response alone cannot justify a conditional interpretation.

Primary area:
datasets and benchmarks
Keywords:
audio-conditioned visual prediction, diagnostic evaluation, input interventions, held-out comparisons, proxy endpoints

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5F9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on distinguishing input sensitivity from meaningful audio use in audio-conditioned visual models. Existing evaluation methods often remove or replace audio and check whether the model's prediction changes. However, these changes do not show whether the model uses the correct audio information or whether audio improves its prediction. CIViC proposes four evaluation checks: input activation, audio replacement, held-out performance, and metric validity. The main contribution is a structured framework that connects these checks with specific claims about audio conditioning, rather than introducing a new evaluation method.

Critical strengths

(1) The paper addresses an important problem in audio-visual evaluation by distinguishing input sensitivity from actual predictive benefits.

(2) CIViC introduces a structured evaluation framework with four evidence gates to assess different aspects of audio conditioning rather than relying on a single test.

(3) The experiments show that changes in model predictions do not always improve performance, highlighting the limitations of audio ablation tests.

Critical weaknesses

(1) The novelty over existing work is unclear. Although the authors mention related studies, they provide limited comparison with existing evaluation methods. It is unclear what CIViC adds beyond combining existing techniques into four evidence gates.

(2) The framework design lacks justification. The authors do not explain why they selected these four checks or whether they are enough to evaluate meaningful audio use.

(3) The evaluation metrics need stronger justification. The paper uses MAE, RMSE, and motion and drift proxies, but it is not always clear how these metrics support the evaluation goals. Some metrics also show conflicting results, without a clear explanation of how they should be interpreted.

(4) The experimental scope is limited. The authors mainly use compact renderers, synthetic settings, and proxy tasks rather than existing audio-visual generation models. This makes it difficult to assess how well CIViC works on real models.

(5) The framework lacks independent validation. The authors do not compare CIViC's decisions with human or expert judgments, controlled ground-truth cases, or simpler evaluation protocols. Without this, it's difficult to determine how reliable the framework is. The presentation needs improvement. Several sections are difficult to follow, particularly the experimental settings, evaluation conditions, and metrics. Clearer explanations would improve readability.

Missing Citations:

Chowdhury, Sanjoy, et al. "Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Yaman, Dogucan, et al. "Audio-visual speech representation expert for enhanced talking face video generation and evaluation." Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. 2024.

Zhou, Yingjie, et al. "Who is a better talker: Subjective and objective quality assessment for ai-generated talking heads." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Corona, Enric, et al. "Vlogger: Multimodal diffusion for embodied avatar synthesis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Kong, Zhe, et al. "Let them talk: Audio-driven multi-person conversational video generation." Advances in Neural Information Processing Systems 38 (2026): 70990-71013.

Questions for authors

(1) What does CIViC contribute beyond combining existing evaluation techniques, and how does it improve upon prior methods?

(2) Section 2 introduces four evidence gates. What justifies their selection, and how do they distinguish meaningful audio use from input sensitivity?

(3) Why was CIViC not evaluated on established audio-driven models such as VLOGGER [4] or MultiTalk [5]?

(4) In Table 8, motion RMSE favors audio removal, while drift favors aligned audio. How should CIViC interpret these conflicting results?

(5) Could the authors evaluate CIViC on controlled cases with known audio dependencies and compare its decisions with expert judgments or existing evaluation methods to demonstrate its reliability?

1 comment

The paper combines four familiar checks so that input sensitivity is paired with a control and an interpretation rule (Section 2 and Figure 1, pp. 2–3). The submitted experiments do not show that this selection is sufficient or more reliable than existing protocols. A controlled expert comparison or known-outcome evaluation is absent, so the gate decisions remain a proposed interpretation framework rather than independently validated judgments.

The conflicting endpoints are visible in the paper: Study II results change with additional training, and Study III's motion-proxy and drift outcomes favor different conditions (Sections 4–5, pp. 5–9). They should be read as separate proxy measurements, not compressed into one conclusion about audio use. No rendered generator or human quality assessment was run.

The current paper cites VLOGGER and Let Them Talk. It does not include AVTrustBench or the suggested Zhou et al. (2025) and Yaman et al. (2024) references. The submission does not compare the four gates against those methods under a shared task, so it provides no comparative performance result.

Reviewer_GYezOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CIViC, a four-gate evaluation framework consisting of Activation, Substitution, Benefit, and Endpoint Validity to assess audio-conditioned visual prediction. Experiments on face-raster prediction, synthetic feature mixtures, and SpeakerVid visual proxies demonstrate that model sensitivity to audio inputs does not necessarily imply meaningful audio utilization or improved prediction performance.

Critical strengths
  1. The paper addresses an important evaluation issue by distinguishing input sensitivity from meaningful audio utilization and actual task benefits.

  2. The paper provides transparent analyses of negative results and acknowledges important limitations in experimental settings and statistical interpretation.

Critical weaknesses
  1. The four CIViC gates largely reorganize existing evaluation practices, including audio ablation, input substitution, performance comparison, and shortcut controls. The paper introduces neither a substantially new evaluation methodology nor sufficiently formalized decision criteria, and fails to demonstrate clear advantages over existing evaluation protocols.

  2. Study I focuses on 64×64 static face prediction, while Studies II and III rely on 128-dimensional visual proxies and synthetic environment/music features. The framework is not validated on actual audio-driven video generation models, and evaluations of lip synchronization, motion realism, and perceptual quality are absent. This significantly limits the demonstrated practical applicability of CIViC.

  3. The cross-actor audio replacements in CREMA-D and RAVDESS simultaneously change speaker identity and speech content, making it difficult to isolate the effects of specific audio information. Moreover, the substitution effect is extremely small on CREMA-D (+0.000109 MAE) and statistically inconclusive on RAVDESS. Even the additional metadata-matched experiments lack precise phoneme-level alignment, weakening the evidence for the proposed Substitution Gate.

Questions for authors

See weaknesses above

3 comments

The CREMA-D replacement changes both actor and speech content, so the +0.000109 MAE result in Table 3 (p. 4) is a joint replacement effect. It does not isolate which audio information caused the raster change. The paper gives no registered practical threshold for that small effect. The RAVDESS interval includes zero, and the metadata-matched follow-up still lacks phoneme-level alignment. These experiments therefore provide bounded input-sensitivity observations; they do not isolate a specific audio contribution.

The paper uses raster and feature-space proxies rather than rendered audio-driven videos. Its results do not measure lip synchronization, continuous motion realism, or perceptual quality. An actual-generator evaluation would be needed to establish those properties; none is reported in the submission.

Thank you for the response. I maintain my rating of 1. The substitution experiments leave it unclear which audio cues drove the observed changes, and CIViC’s added value over standard evaluation practices remains untested on audio-driven video generators.

AuthorsOpenReview ↗

Scope of proxy evidence and generator evaluation

Thank you for separating cue attribution from external validity. Cross-actor substitution changes speaker identity and utterance content together, so it does not identify which acoustic cue drives a response.

Study III provides a separate same-target, source-video-disjoint audio-zero comparison (Tables 7–8, pp. 8–9). For its 128-coordinate motion proxy, audio-zero minus aligned-audio RMSE is negative in all 16 platform-by-condition cells. The four-platform means range from −0.0142 to −0.0143 for clean audio, −0.0214 to −0.0216 for environment-like audio, −0.0210 to −0.0213 for music-like audio, and −0.0206 to −0.0212 for mixed audio; all five paired seed differences are negative in each cell. The four platform runs reuse one source pack, so they are compute replications, not independent video samples. Table 8 gives the opposing endpoint: the secondary segment-drift proxy favors aligned audio in all four conditions, with platform means of 0.0057–0.0064, 0.0076–0.0079, 0.0069–0.0070, and 0.0043–0.0050. It compares 10-second segment means in a flattened window array, not a rendered continuous clip. These endpoints therefore support separate, proxy-specific conclusions; they do not establish which audio cue is used or that audio improves generated video.

The contribution we can defend is a structured link between an audio-use claim and the intervention, matched control, held-out endpoint, and metric checks needed to support it. The results show why output sensitivity alone is insufficient and why the motion and drift outcomes cannot be collapsed into a single claim. They do not establish that the four checks are exhaustive or sufficient, or that CIViC outperforms standard evaluation practice. We have not evaluated VLOGGER, MultiTalk, or another rendered audio-driven video generator, and we have no comparison against expert judgments or a known-outcome generator test. Thus the submission does not establish generator-level validity or comparative advantage; that remains open.

Reviewer_fNa2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CIViC, a diagnostic framework for evaluating whether audio-conditioned visual prediction models meaningfully use their conditioning inputs. The framework organizes evaluation into four evidence gates: activation, substitution specificity, held-out benefit, and endpoint validity.

The authors conduct three studies involving actor-disjoint face-image prediction, synthetic audio-feature mixtures, and a source-video-disjoint SpeakerVid proxy task. Their experiments show that models can be highly sensitive to removing an audio input while exhibiting little sensitivity to replacing its content. Moreover, additional conditioning signals do not necessarily improve prediction accuracy, and some proxy metrics can favor models without audio conditioning.

Critical strengths
  1. The paper addresses a meaningful evaluation problem in audio-conditioned visual prediction. The distinction between input sensitivity and meaningful conditional utilization is important.
  2. The experimental analysis includes multiple control conditions, actor/source-disjoint splits, and repeated training runs. The authors are also careful about distinguishing their empirical observations from broader claims.
Critical weaknesses
  1. Limited methodological novelty.

The four evidence gates primarily organize existing evaluation practices, including input ablation, condition swapping, held-out performance comparison, and shortcut controls. While combining these tests into a structured protocol is useful, the paper does not introduce a substantially new evaluation metric, identification method, or diagnostic technique. The distinction from existing evaluation protocols, particularly Yaman et al. (2026), remains relatively modest.

  1. Lack of evaluation on real audio-driven video generation systems.

This is my main concern. Although the experiments use real audiovisual datasets, the evaluation is limited to simplified raster prediction models and low-dimensional visual proxies. The paper does not apply CIViC to actual audio-driven video generators or evaluate rendered video outputs.

This substantially limits the practical significance of the proposed framework. It remains unclear whether the identified issues translate to real generation systems, where audio-visual synchronization, lip movement accuracy, speaker binding, and perceptual quality are important. Evaluating CIViC on at least one representative audio-driven video generation model would significantly strengthen the paper, particularly if the authors could demonstrate that the proposed gates reveal failures missed by conventional evaluation.

  1. Limited conclusiveness and interpretability of the empirical findings.

Several findings depend strongly on the evaluation setting. For example, in Study II, the disadvantage of adding structured conditions decreases substantially with additional training, making it difficult to determine whether the observed effect reflects conditional utilization or optimization dynamics. In Study III, the audio-zero model performs better on motion-proxy RMSE but worse on the secondary drift proxy. These conflicting results raise questions about how reliably the proposed framework supports meaningful conclusions about model behavior beyond the selected endpoints.

Questions for authors

Please see weaknesses

1 comment

The paper's four gates pair familiar interventions with controls and interpretation rules: audio ablation, input substitution, held-out comparison, and shortcut checks (Section 2 and Figure 1, pp. 2–3). The submitted experiments do not establish that this set is sufficient or more reliable than existing evaluation protocols.

The tests use simplified outputs: 64×64 face-raster prediction in Study I, synthetic 128-coordinate features in Study II, and feature-space motion and drift proxies in Study III. The paper reports no audio-driven video generator or rendered-video quality metrics. In Study III, audio-zero has lower motion-proxy RMSE while aligned audio has lower long-form drift, so the two endpoints do not support a single ranking (Section 5, Tables 7–8, pp. 7–9). The results apply to these controlled proxies and do not measure lip synchronization, motion realism, or perceptual quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.