- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EGAA studies hallucination in omni-modal LLMs. It identifies four error types and augments AVHBench and CMM with modality-specific evidence (CSM, AO, UC, and WMA), supported statements, grounded answers, and error-targeted negative answers. The proposed method jointly optimizes three formats and a DPO training through LoRA. Experiments across 3 backbones show improved accuracy in bidirectional transfer between the two benchmarks.
Critical strengths
This paper establishes a link between 4 types of cross-modal errors and specific preference pairs, thereby constructing an effective framework for converting diagnostic results into supervisory signals for training. Bidirectional transfer experiments conducted on AVHBench and CMM demonstrate that EGAA possesses generalization capabilities surpassing the original benchmarks.
Critical weaknesses
Existing evidence has does not fully establish the additional value and broader applicability of EGAA. Comparative analyses have employed mismatched settings, evaluations have focused on binary judgments and brief explanations, and the characterization of error coverage across the four categories remains insufficient, particularly regarding different models, datasets, and more complex cross-modal scenarios.
Questions for authors
- Appendix B.1 provides comparisons under different experimental settings. Could the authors compare EGAA with a representative method such as OmniDPO or MoD-DPO using the same backbone, source data, and evaluation, with comparable training budgets while retaining each method’s supervision design? This would help clarify EGAA’s value over existing approaches.
- Although caption generation are included during training, the evaluation focused primarily on binary judgments (Yes/No) and brief descriptions. Can EGAA also effectively mitigate hallucinations in open-ended audiovisual generation tasks, such as dense caption generation, non-binary question answering, speaker attribution and so on?
- These four error categories are identified based on the responses of Qwen2.5-Omni-7B on AVHBench. How are the coverage of these error categories verified for other models and datasets? Are there any cross-modal errors that do not fall into the aforementioned categories or are difficult to classify? (For example, a cross-modal temporal relationship error, that a door closes at 2-second mark of the video while a bell rings at 5-second mark of the audio, the model correctly indentifies both the door closing and the ringing bell but answer “the bell rang before the door closed”) How might such errors impact the evaluation process and the construction of preference data?
2 comments
Thank you for the supplementary comparative experiments and broader evaluations by the author. However, the preference data uniformly using EGAA still makes it difficult to fully assess the advantages of its data structure over the existing supervised design. The hallucination indicators in open-ended generation also show inconsistent performance. Therefore, I keep the original score.
Response to Reviewer sTrJ
We thank the reviewer for recognizing that EGAA turns diagnostic categories into training signals. We have uploaded a revised paper that contains all new results.
1. Controlled comparison with OmniDPO and MoD-DPO (Q1)
Neither method has released official code, and the anonymous repository announced by OmniDPO is no longer accessible. This is why the paper originally listed their published numbers separately. We implemented both from their papers, kept each method's own objective, and trained them on Qwen2.5-Omni-7B with the same grounded answers and negatives as EGAA, the same LoRA configuration, and the same schedule (Sec. 4.2, Table 2, Appendix C). EGAA outperforms both on every metric in both directions. CMM accuracy is 88.8% against 86.3% for OmniDPO and 87.5% for MoD-DPO, and AVHBench overall accuracy is 82.2% against 79.0% and 80.8%.
The largest gap is on AVHBench matching (81.0% against 74.6% and 78.1%), which asks whether the two streams belong together, a question that no CMM probe asks. Label-only fine-tuning drops matching to 52.7%. The two preference methods keep most of it because they learn from our error-targeted pairs, and EGAA keeps the most because its Evidence Trace and Grounded Answer make the model describe each stream and state what follows from their combination. The comparison therefore shows both the value of the pairs and what the evidence targets add on top of them.
2. Open-ended generation and non-binary questions (Q2)
The paper originally reported only the judgment tasks of AVHBench, the same tasks that OmniDPO and MoD-DPO report, to keep the results comparable. We now evaluate all three backbones on captioning and on multiple-choice questions, using models that saw neither task format in training (Sec. 4.4).
On AVHBench captioning, scored with its official metrics on the CMM-trained models, EGAA gives the highest METEOR and CIDEr for all three backbones. GAVIE-A, which penalizes content absent from the reference, rises for MiniCPM-o-4.5 and VideoLLaMA2.1-7B-AV and falls for Qwen2.5-Omni-7B, whose EGAA captions are 24.8 characters longer on average. Among the 398 captions scored below Base, the judge cites content absent from the reference in 73.9%, and a manual check of 100 of them found that almost all add correct detail. For a beehive video, for example, EGAA adds that the man speaks Russian and mentions a queen bee, which the reference omits (Table 4, Appendix D.3).
For non-binary question answering and speaker attribution, AV-SpeakerBench poses 3,212 four-way questions on who speaks, what is said, and when. EGAA raises overall accuracy by 1.7 to 2.3 points for every backbone and training direction, and speaker detection, which links a voice to a visible person, by 1.4 to 4.5 points for every model (Table 5, Appendix D.2). SVHalluc adds three-way questions, discussed under Q3.
3. Coverage and the temporal example (Q3)
The four types were defined by the authors from the relation among the requested modality, the per-modality evidence, and the claim, not from the outputs of one model on one benchmark (Sec. 3.1). CMM played no part in this process and serves as a check. Miss occurs in only 0.33% of CMM questions, against 21.79% with any error (Table 3).
In your example, both events are perceived and assigned to the correct modality, and the error lies in their temporal relation. In evaluation, such a statement is counted as Miss and as unsupported, so it lowers the scores as it should. For preference data, the evidence text already records when each event occurs, so negatives that swap the order of events across the two streams can be built in the same way as the four current types.
SVHalluc (CVPR 2026) tests such errors on narrated video, and we evaluated all three backbones on it (Sec. 4.4, Appendix D.2). On its semantic tasks, which ask whether narrated speech is visible and thus test the counterpart of CSM, EGAA improves every backbone by 2.4 to 9.1 points. On its temporal tasks, which ask whether a narrated event happens before, during, or after the speech, EGAA also improves Qwen2.5-Omni-7B and VideoLLaMA2.1-7B-AV in both training directions and MiniCPM-o-4.5 when trained on CMM. The evidence text states when each event occurs, and the Evidence Trace asks the model to reproduce the evidence of each stream, so training already exposes the model to the timing of both streams. EGAA targets the errors that dominate the explanations of current models, while Miss and statement-level support keep the remaining errors visible.
We would be grateful to know whether any point remains unclear or whether other analyses would be helpful. We hope these responses lead you to view the paper more favorably.