- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces an MLLM-based model for streaming audio--visual understanding. The authors define a failure called premature cross-modal commitment, which refers to the conflicts of modality-level evidence and persistent memory across times. In addition to the method, they also bring a benchmark for this issues. The experiments are conducted on several benchmarks.
Critical strengths
- The authos provide a lot of details about the benchmark construction and the method development in the appendix.
- The experiments on several standard streaming audio--visual understanding show good performance of the proposed method, which could be better with more recent comparison.
Critical weaknesses
Overall, I respectively believe the motivation does not make sense. The pipeline requires a lot of pre-defined decisions and comfirmations, as well as thresholds. In addition, the construction of the benchmark and following experoments cannot verify the claimed solution to the obsearved issue. Here follow the details.
- So is the frequency of the conflict across time hugely related to streaming audio-visual scenes? I believe a further clarification would be better on Page 1.
- I was also considering the case of gate for departure. As shown in Figure 1, is the information 'gate 6' is only captured by the visual side, such as the first frame. Then how can the model determine the boarding gate as 6? If the woman just passes the gate 6 and heads to the real gate, how can the model know that? In a real scene, a more visual sign could be provided such as a electric board. In addition, I do believe the auditory conflict is not the reason for the claimed issue. When a changing boarding gate is verified, not only audio but also visual information can help determine the correct gate. Anyway, the motivation is not strong, as presented in current version.
- I disagree with the first sentence in Section 2.1. Not all multimodal models including audio-visual models are built on LLMs.
- Figure 2 includes too much information, while the front size is too small. In addition, how did the authors determine the 'evidence still incomplete'? And for the other decision trees? Are they encoded in the MLLMs? How?
- How did the authors obtain the the calibrated score? And what about the confirmation thresholds?
- I am confused about the equation 5. Why did the authors use the min function?
- How did the authors represent A_s(q)?
- Regarding the proposed benchmark, audiofaith, it comes to the motivation we discussed in the beginning. To what extent can the synthesized data represent the challenge the authors claimed to focus?
- The performance achieved by StreamFE is quite strong compared to previous works in Table 1. However, the problem is the link to the proposed designs for the conflict. How and why the designs work in standard streaming audio-visual tasks? If a normal case arise, why the model still performs well without making use of the effect of the confirmation and so on?
- The availability of the benchmark and training process.
Questions for authors
See the weaknesses above.