- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper is very difficult to follow and understand.
But seems like it investigates whether AV LLMs detect acoustic corruptions or artifacts introduced by the editing pipeline. They are using AVE dataset and 5 corruptions (muted, hallucinated, room mismatch, ambience and foley swap) and Qwen2.5 Omni and Gemini API. to evaluate it. Their setup and conclusion looks unclear. However it seems like the AV LLM judges are not reliably detecting actual audio-visual inconsistencies.
Critical strengths
The problem seems to be interesting and valid. The artifacts introduced during benchmark construction may introduce biasness in the evaluation of audio-visual LLMs.
Critical weaknesses
- Clarity: The paper is very difficult to follow. The motivation, experimental setup and outcome is not clearly written.
- Undefined terminology and insufficient explanations: Several symbols, abbreviations and experimental details are not sufficiently defined. For eg. "manifest_v5.jsonl", "speech_content_swap", "speech_desync". What does register-only mean? etc.
- Limited experiments: Would other AVLLMs (other variants of Qwen, Gemma, etc.) give same conclusion? Can we have more dataset (how about VGGsound, audioset etc.) and corruption diversity ?
Questions for authors
Please refer to the weakness section.