acceptodds
Under review as a conference paper at ICLR 2027

Reasoning with Future Evidence for Streaming Audio–Visual Understanding

Abstract

Streaming audio–visual models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep repeating it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose StreamFE, which expresses unresolved interpretations as claims about future evidence, linked to a verification interval, a verifying modality, and the states that depend on them. When the interval closes, StreamFE checks the claim against evidence from the specified modality. A refutation reduces the influence of the claim and its dependent states, then guides a state update using the new evidence. Separate audio and visual retention preserves the evidence needed for these checks, and an answer gate delays responses while answer-critical claims await review. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, StreamFE outperforms the strongest open baselines on five streaming and audio–visual benchmarks by more than 10% relative on average. We also introduce AudioFaith, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle–speech conflict. StreamFE reaches , compared with at most for open baselines, while reducing vision-induced auditory hallucinations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Streaming Video Understanding, Omni-modal Reasoning, Audio-Visual Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aEWbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an MLLM-based model for streaming audio--visual understanding. The authors define a failure called premature cross-modal commitment, which refers to the conflicts of modality-level evidence and persistent memory across times. In addition to the method, they also bring a benchmark for this issues. The experiments are conducted on several benchmarks.

Critical strengths
  1. The authos provide a lot of details about the benchmark construction and the method development in the appendix.
  2. The experiments on several standard streaming audio--visual understanding show good performance of the proposed method, which could be better with more recent comparison.
Critical weaknesses

Overall, I respectively believe the motivation does not make sense. The pipeline requires a lot of pre-defined decisions and comfirmations, as well as thresholds. In addition, the construction of the benchmark and following experoments cannot verify the claimed solution to the obsearved issue. Here follow the details.

  1. So is the frequency of the conflict across time hugely related to streaming audio-visual scenes? I believe a further clarification would be better on Page 1.
  2. I was also considering the case of gate for departure. As shown in Figure 1, is the information 'gate 6' is only captured by the visual side, such as the first frame. Then how can the model determine the boarding gate as 6? If the woman just passes the gate 6 and heads to the real gate, how can the model know that? In a real scene, a more visual sign could be provided such as a electric board. In addition, I do believe the auditory conflict is not the reason for the claimed issue. When a changing boarding gate is verified, not only audio but also visual information can help determine the correct gate. Anyway, the motivation is not strong, as presented in current version.
  3. I disagree with the first sentence in Section 2.1. Not all multimodal models including audio-visual models are built on LLMs.
  4. Figure 2 includes too much information, while the front size is too small. In addition, how did the authors determine the 'evidence still incomplete'? And for the other decision trees? Are they encoded in the MLLMs? How?
  5. How did the authors obtain the the calibrated score? And what about the confirmation thresholds?
  6. I am confused about the equation 5. Why did the authors use the min function?
  7. How did the authors represent A_s(q)?
  8. Regarding the proposed benchmark, audiofaith, it comes to the motivation we discussed in the beginning. To what extent can the synthesized data represent the challenge the authors claimed to focus?
  9. The performance achieved by StreamFE is quite strong compared to previous works in Table 1. However, the problem is the link to the proposed designs for the conflict. How and why the designs work in standard streaming audio-visual tasks? If a normal case arise, why the model still performs well without making use of the effect of the confirmation and so on?
  10. The availability of the benchmark and training process.
Questions for authors

See the weaknesses above.

Reviewer_mk7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies premature cross-modal commitment: an early interpretation enters streaming memory and persists after later evidence contradicts it. StreamFE represents uncertain interpretations as future-testable claims with a verifying modality, evidence window, and dependency links. When a claim is refuted, the method reduces its influence and that of dependent states, then guides a state update. An answer gate delays responses while answer-critical claims await verification.

StreamFE uses a frozen Qwen3-Omni backbone with lightweight adaptation. The authors also introduce AudioFaith, which holds video fixed while modifying audio, including subtitle–speech conflicts. Results show improvements across five benchmarks and strong AudioFaith performance.

Critical strengths
  1. Clear and relevant problem. The paper identifies a plausible failure of streaming memory: early errors can propagate through later summaries and reasoning even after contrary evidence arrives.

  2. Well-motivated method. Linking uncertain claims to explicit future tests and dependent states provides a structured mechanism for revising accumulated interpretations.

  3. Useful diagnostic benchmark. AudioFaith’s fixed-video interventions and matched audio-only controls help distinguish auditory recognition from the use of audio when visual evidence competes with it.

  4. Mechanism-focused evaluation. The ablations include verification at issue time, claim-only retraction, non-predictive issues, and latency-matched gating. The appendices also examine correction and error recurrence.

Critical weaknesses
  1. AudioFaith comparisons mix adaptation gains with mechanism gains. StreamFE receives training on the same intervention families used in AudioFaith, including soundtrack replacement and subtitle–speech conflicts, whereas the main baselines lack matched adaptation. Evaluation assets are explicitly held out, but transfer beyond the trained intervention types remains unclear. The matched no-loop control already reaches d′ = 2.58, compared with 2.95 for StreamFE. Training without intervention data and adapting a baseline on the same data would better isolate the loop’s contribution.

  2. AudioFaith primarily tests whether answers follow audio. This is appropriate for diagnosing visual dominance, but does not establish that StreamFE selects the appropriate evidence source more generally. For example, CLASH questions always target the edited spoken fact. Complementary tasks requiring visual evidence despite misleading audio, or joint resolution across both modalities, would test whether the method learns evidence-sensitive reasoning beyond stronger audio preference.

  3. The standard-benchmark comparisons need stronger statistical and experimental support. Most baseline results are reproduced from prior work rather than rerun, and input settings vary across methods. Some gains are small, such as 73.1 to 74.7 on StreamingBench-AV, without uncertainty estimates. The OmniVideoBench values in Figure 3 and Table 22 also need reconciliation. The authors should state the exact reference scores and aggregation behind the “more than 10% relative on average” claim.

  4. The efficiency study is too limited to establish streaming practicality. Timing uses only 25 questions answered correctly by all methods and includes a reimplementation of StreamOV. This selected sample may not represent difficult cases with repeated verification and rewriting. Per-chunk processing time, peak memory, verifier-call counts, and accuracy versus response delay on a broader sample would better characterize the method’s cost.

  5. Verifier and provenance reliability need fuller evaluation. The verifier study samples by predicted verdict and reports agreement within those groups. This supports verdict precision but does not directly establish recall of true refutations. Human–human agreement and adjudication details are also unclear. Since retraction reaches only recorded dependencies, an audit of missing or incorrect provenance links would help establish whether erroneous premises can escape correction.

  6. Robustness to configuration choices and backbone changes is untested. The system combines claim admission, verification thresholds, guidance strength, memory limits, and fixed evidence windows. Although some quantities are calibrated, sensitivity to these choices is not reported. Targeted sensitivity analyses and evaluation with another backbone would strengthen the evidence that the method generalizes.

Questions for authors
  1. How does StreamFE perform without intervention-style training data, and how does a baseline adapted on the same data perform?
  2. Can you evaluate complementary cases where visual evidence should determine the answer despite misleading audio?
  3. Can you reconcile Figure 3 with Table 22 and specify the calculation behind the average relative improvement?
  4. Can you report uncertainty estimates and rerun key baselines under matched input and training budgets?
  5. What are the verifier’s recall on true refutations and the accuracy of recorded dependency links? How were human disagreements handled?
  6. What are the per-chunk latency, peak memory, and accuracy–delay trade-offs on a representative sample? How sensitive are results to guidance strength, claim limits, and evidence-window duration?
Reviewer_i3vVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a phenomenon called cross-modal commitment where an early, visually supported interpretation is written to memory as a fact and keeps being reused after later audio contradicts it. The paper proposes StreamFE, a forecast, verify and revise framework built on a frozen Qwen3-Omni-30B-A3B-Instruct backbone that treats unresolved early interpretations as testable claims about future evidence. Each claim specifies a verifying modality (audio, visual, or audio–visual), a future evidence window duration and dependent reasoning states. When the scheduled window closes, a modality-specific verifier verifies the claim, propagates log-reliability attention attenuation to all dependent memory spans, and applies verdict-conditioned contrastive continuation to rewrite the state before an answer gate triggers a response.

Critical strengths
  1. The method addresses persistent errors in streaming memory through an explicit correction mechanism.
  2. The evaluation goes beyond aggregate accuracy and tests plausible alternative explanations.
Critical weaknesses
  1. The AudioFaith comparison mixes the effect of the method with the effect of training on the benchmark’s intervention.
  2. No variance reporting. There are no seeds, confidence intervals, or significance tests. Several ablation differences rest on very small subsets. For example, SOVBench-O Recall has 97 turns, so 82.5 vs. 74.2 is 80 vs. 72 correct answers, and most rows in that column differ by a handful of questions.
  3. The main baselines are taken from the StreamOV paper.
Questions for authors
  1. How does a baseline trained on the same data, for example Qwen3-Omni with the same LoRA SFT including the 10k intervention records and L_pair, but without the forecast–verify–revise loop, perform?
  2. On the standard benchmarks, how often does the mechanism actually fire? Claims per episode, the refutation rate, and the fraction of answers whose support graph contains a refuted claim should be reported.
  3. Can you report variance across seeds or confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.