acceptodds
Under review as a conference paper at ICLR 2027

Seen, Heard, or Assumed? Cross-Modal Hallucinations in Audio-Visual Language Models

Abstract

Despite the rapid progress of audio-visual large language models (AVLLMs), hallucination remains an important reliability challenge in joint audio-visual reasoning. Existing audio-visual hallucination benchmarks emphasize final-answer correctness or answer changes after modifying one modality, while it remains unclear whether these evaluation results reflect correct and stable judgments about what is heard and seen. To jointly assess final-answer correctness, modality-judgment correctness, and cross-modal stability, we introduce AV-TRACE, a benchmark with 7,344 test examples that combines explicit modality judgments with matched audio-visual comparisons for modality-specific hallucination detection and cross-modal stability testing. AV-TRACE reveals that correct final answers can mask modality-judgment failures, and that higher modality-level correctness can coexist with poorer cross-modal stability. Supervised fine-tuning improves answer and judgment accuracy, yet correct judgments can still flip when only the other modality changes. We therefore propose Factorial Quartet Fine-Tuning (FQFT), which augments GRPO with a joint answer-judgment reward and factorial regularization of judgment margins across four target-presence states. On Qwen2.5-Omni-7B, compared with supervised fine-tuning, FQFT increases the task joint accuracy of answers and modality judgments from 76.29% to 84.39%, reduces the hallucination rate from 20.49% to 10.32%, and reduces the flip rate from 4.08% to 1.95%.

Primary area:
datasets and benchmarks
Keywords:
Audio-Visual Language Models, Cross-Modal Hallucinations, Multimodal Benchmarks, Counterfactual Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PBSROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-modal hallucinations in audio-visual large language models. It distinguishes three dimensions that should be evaluated separately: whether the final answer is correct, whether the modality judgments are correct, and whether those judgments remain stable when the other modality changes.

The authors introduce AV-TRACE, which contains 7,344 manually verified test examples. It evaluates correctness and cross-modal stability through explicit modality judgments and audio-visual quartet comparisons. Each quartet crosses audio and video containing or not containing the target, allowing one modality to be held fixed while the other is replaced to detect erroneous judgment flips.

The authors also propose FQFT, which combines a joint reward for answers and modality judgments with quartet regularization within GRPO. The experiments cover five base models and several mitigation methods. FQFT training and ablations use Qwen2.5-Omni-7B, with evaluations on AV-TRACE and an adapted AVHBench* showing higher joint accuracy and lower hallucination rates.

Critical strengths
  1. The evaluation separates reliability issues that can otherwise be conflated. AV-TRACE examines both final answers and explicit modality judgments. By holding the same clip fixed and replacing the other modality, its controlled comparisons identify erroneous flips caused by cross-modal interference. This separates cases in which an answer is correct but the judgment is wrong from cases in which a judgment is correct on one input but unstable across inputs, providing a useful complement to answer accuracy.
  2. The training method directly addresses the identified problems and has controlled experimental support. FQFT incorporates judgment correctness and cross-modal stability into training. Starting from the same SFT initialization, it improves joint accuracy from 81.43% to 84.39% and reduces the hallucination rate from 14.23% to 10.32% relative to vanilla GRPO. Results on the adapted AVHBench* also provide some evidence of transfer, supporting the practical value of the method.
Critical weaknesses
  1. Model coverage remains insufficient. Table 2 evaluates five base models but does not cover the Gemini family or more recent Qwen Omni models. I suggest adding Gemini 3.1 Pro Preview, Gemini 3.8 Flash, Qwen3.5-Omni-Plus, and Qwen3.8-Omni-Flash, restricted to versions publicly available at the time of submission. This would test whether the observed hallucination behavior and main conclusions also hold for stronger models and different model families, strengthening their generality.
  2. FQFT has not been validated across backbones. FQFT training and ablations are conducted on Qwen2.5-Omni-7B, leaving it unclear whether the gains depend on this backbone. At least one additional trainable backbone should be used to reproduce the comparison among SFT, vanilla GRPO, and FQFT to support the transferability of the method.
  3. The ablations do not fully explain the roles of the key components. Table 3 separates the contributions of the joint reward and quartet regularization, but target discrimination, non-target modality suppression, interaction suppression, and mean anchoring within the regularizer are still evaluated as a package. Their individual contributions and necessity therefore remain unclear. Moreover, full FQFT achieves higher joint accuracy, whereas quartet-only has a slightly lower Flip score. Targeted ablations, particularly of interaction suppression and mean anchoring, would help explain their different effects on correctness and stability.
  4. Evidence of external generalization remains limited. External validation is concentrated on AVHBench* after adapting its tasks and output protocol, leaving it unclear how much the gains depend on the current structured evaluation setting. This dataset also lacks quartets with shared clips, so it cannot validate transfer of improvements in Flip. Evaluations under the original task protocol, on other datasets, or on more natural in-the-wild audio-visual data would strengthen the evidence for effectiveness under distribution changes, especially the generalization of cross-modal stability.
Questions for authors

Quality of automated prescreening and annotation cost. The 1.4% error rate reported in the appendix comes from an audit after human verification and adjudication. Is there a comparison between the automated prescreening results and the final human labels, particularly the false acceptance and false rejection rates for audio and vision separately, and the amount of human work actually saved? Given that visual prescreening primarily relies on a single large model judging sampled frames, have the authors evaluated strategies such as identifying supporting frames or time spans, cross-checking with different models, or routing uncertain cases for review? Can these strategies reduce the proportion requiring human review while maintaining annotation quality?

6 comments

We thank the reviewer for the constructive comments and questions. We answer each of them below. Tables A–F are provided in our responses, and we will update all new results and tables into the revised PDF of the paper.

W1: Expanding model coverage to Gemini 3.1 Pro Preview, Gemini 3.8 Flash, Qwen3.5-Omni-Plus, and Qwen3.8-Omni-Flash.

Using the versions publicly available at the submission deadline and the same setup as Paper Table 2, we evaluated Gemini 3.8 Flash, Gemini 3.1 Pro Preview, Qwen3.8-Omni-Flash, and Qwen3.5-Omni-Plus on the full AV-TRACE test set (overall results in Table A; breakdowns by task and interference direction in Table B).

Table A. Results on AV-TRACE (%).

Model / methodAA ↑JA ↑JAA ↑Hall. ↓Flip ↓
Qwen2.5-Omni-7B56.9944.9644.2958.9074.82
Qwen3-Omni-30B-A3B56.4253.7253.7220.4414.48
MiniCPM-o-4.567.8055.1455.1026.7827.48
Gemma-4-E4B48.4347.7947.7930.1127.13
Gemma-4-12B49.8249.6549.6234.1623.53
Gemini 3.8 Flash78.2178.1478.1414.925.77
Gemini 3.1 Pro Preview86.8786.6386.4710.813.71
Qwen3.8-Omni-Flash84.8781.8181.8113.688.16
Qwen3.5-Omni-Plus85.7685.6085.5611.794.78
AVCD50.6341.6041.1121.5527.09
MAD62.3255.9454.1143.7935.96
RL-CoMM57.3845.3544.7158.4033.35
ACPO56.8052.2334.7729.8027.51
ASD45.6838.8330.4933.4822.38
Qwen2.5-Omni-7B +SFT+FQFT (ours)85.7584.4784.3910.321.95

Table B. Results on AV-TRACE by task and interference direction (%).

Model / methodAudio det. JAA ↑Visual det. JAA ↑Det. Hall. V→A ↓Det. Hall. A→V ↓Quar. Hall. V→A ↓Quar. Hall. A→V ↓Flip V→A ↓Flip A→V ↓
Qwen2.5-Omni-7B25.0066.5065.5055.6069.9141.3291.2258.42
Qwen3-Omni-30B-A3B28.0067.2029.0017.5023.805.8423.385.57
MiniCPM-o-4.525.5070.5048.007.6037.9511.5346.298.66
Gemini 3.8 Flash56.9099.5034.200.2017.443.2911.230.31
Gemini 3.1 Pro Preview74.2098.3022.900.6014.523.376.411.01
Qwen3.8-Omni-Flash67.5094.2024.601.4027.402.6915.820.49
Qwen3.5-Omni-Plus72.9098.8024.200.7018.042.929.250.31
Qwen2.5-Omni-7B +SFT+FQFT (ours)78.2088.4014.409.8011.452.103.890.00

All main conclusions of the paper hold on these stronger models: although the four closed-source models reach substantially higher JAA than the five open-weight base models, their Hall. remains 10.81%–14.92%, their Flip remains 3.71%–8.16%, and errors remain heavily concentrated in the visual-to-audio (V→A) direction—detection hallucination under audio interference (A→V) is only 0.20%–1.40%, whereas visual interference (V→A) still induces 22.90%–34.20% hallucination and 6.41%–15.82% Flip.

Moreover, high accuracy still does not imply stability: Gemini 3.1 Pro Preview and Qwen3.5-Omni-Plus achieve higher overall JAA than 7B FQFT via near-ceiling visual detection, yet suffer roughly 2× the Flip rate; Qwen3.8-Omni-Flash is within 0.4 JAA points of vanilla GRPO on Qwen2.5-Omni-7B (81.43% JAA, 2.40% Flip) but flips more than 3× as often. In comparison, 7B Qwen2.5-Omni-7B + FQFT achieves the lowest Hall. (10.32%), lowest Flip (1.95%), and highest audio detection JAA (78.20%) across the entire table, outperforming Gemini 3.8 Flash and Qwen3.8-Omni-Flash on all five metrics.

We thank the reviewer for this valuable comment.

W2: Validating the transferability of SFT, vanilla GRPO, and FQFT on an additional trainable backbone.

We reproduced the full comparison on MiniCPM-o 2.6, from a distinct model family, using the same data, objectives, and pipeline as Qwen2.5-Omni-7B (Table C).

Table C. Results of MiniCPM-o 2.6 on AV-TRACE (%).

Model / methodAA ↑JA ↑JAA ↑Hall. ↓Flip ↓
MiniCPM-o 2.661.9247.2245.6142.3360.63
+SFT77.4875.7875.7417.7611.09
+SFT+Vanilla GRPO81.3978.3078.2512.657.29
+SFT+FQFT (ours)83.3280.7480.6610.847.24

As shown in Table C, on MiniCPM-o 2.6, the ordering across variants matches Qwen2.5-Omni-7B: AA, JA, JAA, and Hall. improve consistently from SFT → vanilla GRPO → FQFT (JAA from 45.61% → 75.74% → 78.25% → 80.66%; Hall. from 42.33% → 17.76% → 12.65% → 10.84%), both policy-training variants cut SFT's Flip rate (11.09%) by roughly one-third (to 7.29% and 7.24%), and FQFT raises audio detection JAA from 28.60% in the base model to 69.10% (and visual detection from 63.70% to 90.70%) while cutting Flip in the dominant V→A direction from 72.76% to 9.85% (and A→V Flip from 48.49% to 4.64%). Relative to vanilla GRPO from the same SFT checkpoint, FQFT improves JAA by +2.41 percentage points and lowers Hall. by −1.81 percentage points on MiniCPM-o 2.6, closely matching the gain pattern on Qwen2.5-Omni-7B and confirming transferability across backbones.

We thank the reviewer for this valuable comment.

W3: Component-level ablations inside the quartet regularizer and comparison between quartet-only and full FQFT on accuracy and Flip.

Under the same setup as Table 3 in the paper, we present the module-level comparison of joint reward and quartet regularization (Table D(a)) separately from the ablation of the individual loss terms inside the quartet regularizer (Table D(b)).

Table D(a). Module-level ablation of joint reward and quartet regularization on Qwen2.5-Omni-7B (%).

TrainingJoint rewardQuartet.AA ↑JA ↑JAA ↑Hall. ↓Flip ↓
SFT✗✗77.5676.3276.2920.494.08
Vanilla GRPO✗✗83.0381.4481.4314.232.40
Joint reward✓✗82.6481.3981.3714.582.28
Quartet✗✓85.1983.3083.2910.881.83
FQFT (ours)✓✓85.7584.4784.3910.321.95

As shown in Table D(a), adding Joint reward alone only scores single-input responses without quartet regularization to suppress cross-modal interference in representation space. Under cross-modal conflict, completions in a GRPO group still tend to make the same modality-judgment errors (and when answer-correct completions share the same judgment outcome, group normalization cancels the bonus; Appendix E). Thus, Joint reward alone slightly narrows the AA-to-JAA gap (1.60 to 1.27 points) without improving JAA (81.37% vs. 81.43% under Vanilla GRPO) or Hall. (14.58% vs. 14.23%). Conversely, Quartet alone constrains the four-state log-probability margins, cutting Hall. from 14.23% to 10.88% and Flip from 2.40% to 1.83%, but its policy reward still scores only the final answer rather than generated modality judgments, leaving JA and JAA (83.30% and 83.29%) 1.90 points behind AA (85.19%). When combined in full FQFT, the quartet regularizer suppresses cross-modal interference internally while Joint reward directly rewards responses where the final answer and all modality judgments are simultaneously correct. Relative to Quartet, FQFT raises JA by 1.17 points to 84.47% and JAA by 1.10 points to 84.39%, narrows the AA-to-JAA gap to 1.36 points, and lowers Hall. to 10.32%. Because cross-state stability is governed by the quartet regularizer rather than the single-input reward, Quartet and full FQFT achieve similarly low Flip rates (1.83% and 1.95%, well below 2.40% under Vanilla GRPO and 2.28% under Joint reward). Across all 1,207 eligible V→A pairs in the test set, FQFT flips only 47 times (3.89%) and 0 times on A→V, so the 0.12-point difference reflects only a few examples.

Table D(b). Loss-term ablation inside the quartet regularizer on Qwen2.5-Omni-7B (%).

TrainingTarget disc.Mean anchor.Non-target supp.Interact. supp.AA ↑JA ↑JAA ↑Hall. ↓Flip ↓
FQFT w/o cross-state coupling✓✓✗✗82.0080.8180.8015.212.94
FQFT w/o non-target supp.✓✓✗✓82.1180.8580.8415.072.97
FQFT w/o interaction✓✓✓✗82.0480.9780.9715.152.37
FQFT w/o mean anchor.✓✗✓✓82.3581.2981.2914.912.16
FQFT w/o target disc.✗✓✓✓81.8880.7580.7415.292.41
FQFT (ours)✓✓✓✓85.7584.4784.3910.321.95

To further examine each loss term inside the quartet regularizer, Table D(b) removes both cross-state terms together (w/o cross-state coupling) or removes each of the four terms individually. Notably, all five ablated variants perform slightly below Vanilla GRPO in Table D(a) (80.74%–81.29% vs. 81.43% JAA; 14.91%–15.29% vs. 14.23% Hall.): when both cross-state terms are removed (w/o cross-state coupling, 80.80% JAA, 15.21% Hall., 2.94% Flip), target discrimination only enlarges the average present-minus-absent gap, which the model can widen via co-occurrence cues on concordant states at the cost of more conflict-state errors. Between the two cross-state terms, non-target modality suppression limits average interference from the other modality across present and absent states (removing it raises Flip to 2.97%, the worst among all variants), while interaction suppression removes opposite interference across the two states (removing it leaves JAA at 80.97% and Hall. at 15.15%). Meanwhile, because the first three terms only constrain relative differences among the four states, removing mean anchoring leaves the four-state mean unconstrained (81.29% JAA, 14.91% Hall.), and removing target discrimination fails to separate positive and negative targets, giving the lowest JAA (80.74%) and highest Hall. (15.29%). Only when all four terms are enabled together (full FQFT) does the model reach 84.39% JAA, 10.32% Hall., and 1.95% Flip, confirming that all four loss terms are indispensable.

Reviewer_qodTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the challenge that existing audio-visual hallucination benchmarks mainly evaluate the accuracy of the final answer or the change in the answer when one modality is modified, and therefore cannot show whether the model’s judgments about what it hears and sees are correct and stable. To solve this challenge, the authors propose AV-TRACE. AV-TRACE asks for modality-wise judgments together with the final answer, and evaluates whether the judgments are maintained when only the other modality changes, using inputs that cross-combine audio and video with and without the target. The authors also propose FQFT, which extends GRPO-based training with a reward that considers both the answer and the judgments, and with a regularization for consistency across the four combinations. This was demonstrated on Qwen2.5-Omni-7B through comparisons with existing AVLLMs and hallucination mitigation methods, and an evaluation on AVHBench is also provided.

Critical strengths
  1. The authors propose a benchmark that evaluates not only the accuracy of the final answer but also the modality-wise judgments and their stability under changes in the other modality, and the data is constructed through human verification.

  2. The proposed method demonstrates superior performance on the proposed benchmark compared to existing AVLLMs and hallucination mitigation methods, and the performance improvement is shown through ablation experiments.

Critical weaknesses
  1. Novelty and motivation. The main contribution and the motivation of this paper are unclear.

    1-1. The authors aim to solve the problem where the model answers that an object is visible or audible when it is not actually present. However, there is insufficient evidence and explanation of whether the setting, which only asks the model to judge the presence of what is seen and heard in audio and visual inputs combined from different clips, actually solves this problem. The motivation of the proposed benchmark is similar to existing works, AVHBench and Thud, and it is not clear what makes the benchmark different other than the way it combines and compares audio and visual inputs. In other words, it is difficult to identify the limitations of prior work through this benchmark, and the authors need to clarify what advantage the benchmark provides. It is also unclear what this benchmark can verify. It is not clear whether the benchmark can distinguish whether the model actually understands the context and the relationship between audio and visual inputs when answering the question, or simply recognizes the pattern of the given pairs.

    1-2. The proposed training method also takes the form of the general fine-tuning process with additional constraints, and it is not clear what makes it different from existing fine-tuning or GRPO, other than the additional task-related training.

  2. The benchmark and its evaluation rely on limited settings.

    2-1. Cross-modal hallucination can be seen as a problem that arises when the understanding of the spatial relationship or context between the two modalities is insufficient. It is not clear whether the proposed method improves this understanding so that the model can answer general questions without confusion, or whether it only solves the problem within the limited question and answer rules defined in the benchmark. In other words, it is difficult to distinguish whether the method solves the hallucination itself, or solves it in a way that fits the benchmark.

    2-2. The proposed method is trained and evaluated on data with a similar composition, and the evaluation on another benchmark is also conducted in the same answer format as the training. Therefore, it is not clear whether the proposed method goes beyond answering in the given question and answer format and solves this problem in general questions that are not covered in training.

    2-3. The existing models used for comparison already have rich and generalized knowledge, whereas the proposed benchmark focuses on a limited task that is close to classification for specific situations, such as whether a sound is audible, and the proposed method is trained for this task. Therefore, it is difficult to determine whether the performance improvement demonstrates the mitigation of cross-modal hallucination as an effect of the proposed method, or comes from training on the labeled data of the benchmark to fit the limited task and the given answer format.

  3. The performance improvement of the proposed method over the base model is not clearly attributed to the proposed training method. Most of the improvement seems to be obtained by general fine-tuning alone, and it is also not clear which part of the proposed method contributes to the improvement. Therefore, it is unclear whether the proposed method has clear novelty or whether the improvement mainly comes from fine-tuning on the data.

  4. The authors design the training method so that the judgment on each modality is not affected by the other modality. However, whether the model actually makes judgments based on the information of the corresponding modality is only shown through the output results, and a qualitative analysis such as an attention map is missing.

  5. The proposed method fine-tunes a general-purpose model for a limited task, but there is no evaluation of whether the general audio-visual understanding ability of the base model is preserved after training. Since the goal is to mitigate hallucination in a general-purpose model, verification of whether the general ability is preserved would be needed.

Questions for authors
  1. What is the motivation of the proposed benchmark and training method, and what is the key difference compared to prior work? In particular, a clear explanation of whether the proposed setting actually solves the cross-modal hallucination problem would help clarify the contribution.

  2. Can the proposed method reduce hallucination in general questions beyond the given question and answer format? In addition, comparing the existing models under the condition of training on the same data would help demonstrate the effectiveness of the proposed framework, benchmark, and data.

  3. An analysis or explanation of whether the performance improvement comes from the data or from the proposed method would help clarify the contribution of the proposed method.

8 comments

We thank the reviewer for the constructive comments and questions. We answer each of them below. Tables A–F are provided in our responses, and we will update all new results and tables into the revised PDF of the paper.

W1 / Q1 (Part 1/2): Benchmark motivation, main contribution over AVHBench and Thud, and how AV-TRACE distinguishes genuine audio-visual context understanding from recognizing pairing patterns.

Our benchmark is motivated by the fact that existing evaluations only check the final answer on a single input. This masks cross-modal hallucinations where a model guesses the right answer from co-occurrence priors, but makes wrong judgments about what is actually seen or heard. Compared with prior work, AVHBench only evaluates single-modality QA on a single input, and Thud only asks about audio without holding the queried clip fixed across four states. In contrast, AV-TRACE requires the model to give both the final answer and audio-visual presence judgments in the same response, evaluated by joint accuracy (JAA). It also uses 2×2 quartets that keep the queried clip fixed and replace only the other modality to measure the flip rate (Flip).

To test whether a model truly understands the audio-visual context and relationship rather than just recognizing pairing patterns, AV-TRACE adds constraints in its evaluation design. The benchmark evaluates both modality evidence perception on original audio-visual inputs (modality-specific hallucination detection) and cross-modal consistency under interference from the other modality (cross-modal stability testing). The model must understand the semantic relationship between the audio event and the visual scene to judge whether they match, while also giving presence judgments for both modalities. In modality-specific hallucination detection, we test not only the final answer to a single question, but also the model's judgment of evidence in each modality. This exposes cases where the model picks the right final answer but makes wrong modality judgments. In addition, in cross-modal stability testing, we reveal whether the model answers based on real evidence in the queried modality, or simply guesses from co-occurrence priors triggered by semantic cues (cross-modal distractors) from the other modality. Therefore, only a model that truly understands each modality and their cross-modal relationship can achieve high JAA, low Hall., and low Flip at the same time. Moreover, there is no video overlap across the training, validation, and test splits. On the external benchmark AVHBench*, which has no quartet pairing structure, FQFT also achieves the highest JAA (75.50%) and lowest Hall. (9.38%, original Table 4), further reducing the chance that the model answers by memorizing pairing patterns.

Table A. Results on AV-TRACE (%).

Model / methodAA ↑JA ↑JAA ↑Hall. ↓Flip ↓
Qwen2.5-Omni-7B56.9944.9644.2958.9074.82
Qwen3-Omni-30B-A3B56.4253.7253.7220.4414.48
MiniCPM-o-4.567.8055.1455.1026.7827.48
Gemma-4-E4B48.4347.7947.7930.1127.13
Gemma-4-12B49.8249.6549.6234.1623.53
Gemini 3.8 Flash78.2178.1478.1414.925.77
Gemini 3.1 Pro Preview86.8786.6386.4710.813.71
Qwen3.8-Omni-Flash84.8781.8181.8113.688.16
Qwen3.5-Omni-Plus85.7685.6085.5611.794.78
AVCD50.6341.6041.1121.5527.09
MAD62.3255.9454.1143.7935.96
RL-CoMM57.3845.3544.7158.4033.35
ACPO56.8052.2334.7729.8027.51
ASD45.6838.8330.4933.4822.38
Qwen2.5-Omni-7B +FQFT (ours)85.7584.4784.3910.321.95

As shown in Table A, AV-TRACE reveals failure modes overlooked by prior benchmarks. On open-weight models, answer-only accuracy substantially overstates joint accuracy, with a 12.70-point gap on both Qwen2.5-Omni-7B and MiniCPM-o-4.5, and 14.48%–74.82% of initially correct absent judgments flip to present when the other modality changes. Cross-modal hallucinations and flips also persist on strong closed-source models, including Gemini 3.1 Pro Preview, Gemini 3.8 Flash, Qwen3.5-Omni-Plus, and Qwen3.8-Omni-Flash, where Hall. stays at 10.81%–14.92% and Flip at 3.71%–8.16%. High accuracy still does not guarantee stability: Qwen3.8-Omni-Flash reaches 81.81% JAA yet has an 8.16% Flip rate, and Gemini 3.1 Pro Preview and Qwen3.5-Omni-Plus flip roughly twice as often as our 7B model. In contrast, Qwen2.5-Omni-7B with FQFT achieves the lowest Hall. at 10.32% and the lowest Flip at 1.95% in the table.

We thank the reviewer for this valuable comment.

W1 / Q1 (Part 2/2): Main contribution of the proposed training method (FQFT) and its key differences from standard fine-tuning (SFT), vanilla GRPO, and pairwise contrastive methods.

The main contribution of our training method (FQFT) and its key difference from existing fine-tuning, GRPO, and pairwise contrastive methods lie in how cross-modal invariance is enforced: standard SFT and vanilla GRPO optimize each sample independently, rewarding getting an isolated output right without constraining the model's internal decision boundary for the queried modality to remain invariant when the other modality is replaced; existing pairwise contrastive methods (e.g., ACPO, MoD-DPO) compare only two inputs at a time and therefore cannot simultaneously eliminate asymmetric interference across both present and absent states of the queried modality.

In contrast, FQFT introduces joint four-state regularization over the 2×2 quartet surface: its quartet regularizer directly shapes internal log-probability margins across all four audio-visual combinations—widening the queried modality's positive–negative discrimination gap while explicitly penalizing both the average shift induced by the other modality and state-dependent cross-modal interaction—and pairs this with a strict joint reward granted only when the final answer and both modality judgments are simultaneously correct. Under identical training data and budgets, FQFT consistently outperforms SFT and vanilla GRPO across both Qwen2.5-Omni-7B (raising JAA from 76.29% under SFT and 81.43% under vanilla GRPO to 84.39%, while lowering Hall. to 10.32% and Flip to 1.95% in original Table 3) and MiniCPM-o 2.6 (raising JAA from 75.74% and 78.25% to 80.66%, and lowering Hall. from 17.76% and 12.65% to 10.84%).

We thank the reviewer for this valuable comment.

W2 / Q2 (Part 1/2): Whether the proposed method genuinely improves cross-modal context understanding or merely fits answers within the benchmark's constrained QA rules.

FQFT genuinely improves the model's cross-modal context understanding rather than merely fitting the benchmark's QA rules. Conceptually, cross-modal hallucination arises because pretrained models conflate multimodal context via co-occurrence priors (confusing what is actually seen, what is actually heard, and what is assumed from co-occurrence) instead of resolving what each modality actually conveys and how they relate. FQFT addresses this at the internal representation level in two ways that do not depend on the benchmark's output format: first, cross-modal disentanglement in margin space, where the quartet regularizer operates on internal log-probability margins across all four audio-visual combinations, forcing the model to separate modality-specific evidence from cross-modal interference so it knows when audio and visual cues agree and when a conflicting cue must be ignored; second, template-independent attention grounding, where the margin queries are computed solely from the raw audio-visual inputs and a modality probe without using the structured QA output template, and training freezes the audio/visual encoders while updating only language-side attention projections via LoRA, directly improving how the backbone attends to and integrates audio and visual tokens even when the benchmark's answer format is removed.

Table B. Results under the original AVHBench response protocol (%).

Model / methodADVH Acc.ADVH F1VDAH Acc.VDAH F1AV Match Acc.AV Match F1Overall Acc.Overall F1
AVHBench (Align.+FT)83.90–77.30–55.60–––
AVCD––––––72.15–
ACPO77.7078.0079.9080.40––78.80–
RL-CoMM78.9688.2465.6379.2551.8568.42––
SFFL85.12–80.79–79.58–81.29–
OmniDPO84.4283.4777.5180.85––––
MAD84.40–78.70–––81.60–
ASD84.00–78.90–70.10–77.70–
MoD-DPO87.6687.6182.4880.9869.0758.53––
MoD-DPO++88.1988.1583.4082.1069.6859.7180.42–
Qwen2.5-Omni-7B79.3179.3384.5084.7284.1784.2682.7882.95
+SFT82.8382.9184.6784.6886.3086.3284.0684.09
+Vanilla GRPO83.2783.4284.5984.7585.9386.0284.1584.31
+FQFT (ours)84.1584.2184.6784.6986.2586.2684.5084.53

Both internal mechanistic analysis and external evaluations without our training template support this conclusion. Internally, under cross-modal conflict, FQFT increases late-layer attention to the queried modality to 86.50% / 74.09%, reduces cross-modal attention shift to 5.98% / 11.56%, and strengthens the anti-hallucination logit push to −1.02, outperforming both SFT and vanilla GRPO on all three measures (layer-wise visualizations are added at the very end of the appendix in the revised PDF, Appendix I, Figures 9 and 10). 

Externally, under the original AVHBench response protocol without our prompt template (Table B), SFT provides the initial jump over the base model, and FQFT brings further gains over both SFT and vanilla GRPO, raising Overall accuracy to 84.50% and ADVH by 1.32 points over SFT and 0.88 points over vanilla GRPO while preserving VDAH and unseen AV Matching, confirming that the learned behavior extends beyond our benchmark rules.

Reviewer_kFbxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AV-TRACE uses explicit presence judgments and matched audio-video quartets to separate answer accuracy from cross-modal stability. FQFT combines GRPO with factorial margin regularization. On Qwen2.5-Omni-7B, it improves joint accuracy over vanilla GRPO from 81.43% to 84.39% and reduces hallucination from 14.23% to 10.32%.

Critical strengths
  • S1. Consequential diagnostic contribution. Reusing clips across matched quartets reveals incorrect modality judgments hidden behind correct final answers (§3.1). Source-grouped splits and an independent 1,000-example annotation audit strengthen the resource (Appendix A).
  • S2. Controlled evidence of benefit. Table 3 holds initialization and policy-update count fixed while crossing reward and quartet regularization. The quartet module improves joint accuracy and hallucination under both reward settings.
Critical weaknesses
  • W1. The specific mechanism remains uncertain. Table 3 enables target discrimination, consistency, anchoring, and extra margin queries together. It establishes a module-level gain, but not an advantage of four-state coupling over matched auxiliary presence supervision.
  • W2. Generalization remains protocol-specific. Training is tested on one backbone, and AVHBench* converts captioning to four-choice selection and requests explicit decisions (§4.4, Appendix G). Transfer to native binary or free-form outputs remains unverified.
Questions for authors
  • Clarification (W1): Is there a control retaining the same independent margin queries and target supervision without cross-state coupling?
  • Clarification (W2): Are there existing results under the original AVHBench response protocol?
5 comments

We thank the reviewer for the constructive comments and questions. We answer each of them below. Tables A–C are provided in our responses, and we will update all new results and tables into the revised PDF of the paper.

W1: Mechanism and ablation comparison of four-state coupling versus matched auxiliary presence supervision.

Under the same setup as Table 3 in the paper, we added five ablated variants of the quartet regularizer (Table A). Specifically, w/o cross-state coupling retains the same margin queries, target discrimination, and mean anchoring as full FQFT while removing both cross-state terms (non-target modality suppression and interaction suppression), and the other four variants (w/o non-target supp., w/o interaction, w/o mean anchor., and w/o target disc.) each remove one of the four components individually. All other training settings, the SFT starting checkpoint, and the update budget follow the paper.

Table A. Ablation of the quartet regularizer on Qwen2.5-Omni-7B (%).

TrainingTarget disc.Mean anchor.Non-target supp.Interact. supp.AA ↑JA ↑JAA ↑Hall. ↓Flip ↓
Qwen2.5-Omni-7B✗✗✗✗56.9944.9644.2958.9074.82
SFT✗✗✗✗77.5676.3276.2920.494.08
FQFT w/o cross-state coupling✓✓✗✗82.0080.8180.8015.212.94
FQFT w/o non-target supp.✓✓✗✓82.1180.8580.8415.072.97
FQFT w/o interaction✓✓✓✗82.0480.9780.9715.152.37
FQFT w/o mean anchor.✓✗✓✓82.3581.2981.2914.912.16
FQFT w/o target disc.✗✓✓✓81.8880.7580.7415.292.41
FQFT (ours)✓✓✓✓85.7584.4784.3910.321.95

Removing the two cross-state coupling terms (w/o cross-state coupling) leaves JAA at only 80.80% (with 15.21% Hall. and 2.94% Flip), confirming that matched auxiliary presence supervision alone yields no gain over Vanilla GRPO without the regularizer (81.43% JAA, 14.23% Hall., and 2.40% Flip in Paper Table 3); adding back the two cross-state coupling terms (full FQFT) raises JAA to 84.39%, lowers Hall. to 10.32%, and reduces Flip to 1.95%.

On detection tasks specifically, relative to w/o cross-state coupling, FQFT raises audio and visual detection JAA from 72.50% and 83.90% to 78.20% and 88.40%, while cutting hallucination on cross-modal distractors from 22.70% to 14.40% (audio) and from 14.80% to 9.80% (visual), showing that gains concentrate precisely where the other modality misleads judgments. Meanwhile, removing non-target modality suppression (which raises Flip to 2.97%, the worst among all five ablated variants), interaction suppression, mean anchoring, or target discrimination individually leaves JAA at 80.74%–81.29%, demonstrating that both cross-state coupling terms must work jointly with target discrimination and mean anchoring.

We thank the reviewer for this valuable comment.

W2: Generalization under the original AVHBench response protocol and across trainable backbones.

In the paper, we adapted AVHBench into AVHBench* with a unified response template to ensure a fair comparison, so that all evaluated models and training baselines are compared under the exact same prompt template, joint metrics, and training data. To further verify transfer beyond that template, we evaluated our models across training stages alongside published baselines under the original AVHBench response protocol (Table B).

Table B. Results under the original AVHBench response protocol (%).

Model / methodADVH Acc.ADVH F1VDAH Acc.VDAH F1AV Match Acc.AV Match F1Overall Acc.Overall F1
AVHBench (Align.+FT)83.90–77.30–55.60–––
AVCD––––––72.15–
ACPO77.7078.0079.9080.40––78.80–
RL-CoMM78.9688.2465.6379.2551.8568.42––
SFFL85.12–80.79–79.58–81.29–
OmniDPO84.4283.4777.5180.85––––
MAD84.40–78.70–––81.60–
ASD84.00–78.90–70.10–77.70–
MoD-DPO87.6687.6182.4880.9869.0758.53––
MoD-DPO++88.1988.1583.4082.1069.6859.7180.42–
Qwen2.5-Omni-7B79.3179.3384.5084.7284.1784.2682.7882.95
+SFT82.8382.9184.6784.6886.3086.3284.0684.09
+SFT+Vanilla GRPO83.2783.4284.5984.7585.9386.0284.1584.31
+SFT+FQFT (ours)84.1584.2184.6784.6986.2586.2684.5084.53

Under the original AVHBench response protocol, Overall accuracy and F1 improve across the base model, SFT, vanilla GRPO, and FQFT, reaching 84.50% and 84.53%. While SFT brings the initial gain over the base model, FQFT further improves ADVH over both SFT (1.32 points) and vanilla GRPO (0.88 points) while keeping VDAH and unseen AV Matching stable. Compared with existing baselines, FQFT achieves the highest VDAH, AV Matching, and Overall accuracy, confirming effectiveness beyond our adapted template.

We also trained MiniCPM-o 2.6, from a distinct model family, using the same data, objectives, and pipeline as Qwen2.5-Omni-7B (Table C).

Table C. Results of MiniCPM-o 2.6 on AV-TRACE (%).

Model / methodAA ↑JA ↑JAA ↑Hall. ↓Flip ↓
MiniCPM-o 2.661.9247.2245.6142.3360.63
+SFT77.4875.7875.7417.7611.09
+SFT+Vanilla GRPO81.3978.3078.2512.657.29
+SFT+FQFT (ours)83.3280.7480.6610.847.24

Matching Qwen2.5-Omni-7B, AA, JA, JAA, and Hall. on MiniCPM-o 2.6 improve consistently from SFT to vanilla GRPO and FQFT. Relative to vanilla GRPO, FQFT raises JAA by 2.41 points to 80.66% and lowers Hall. by 1.81 points to 10.84%, while keeping Flip well below SFT. This confirms that the gains do not depend on a single backbone.

Dear Reviewer kFbx,

Thank you again for your careful review and constructive suggestions. We have posted the experiments and analyses addressing your questions, and we will incorporate all of them into the revised manuscript:

  • W1 (Four-state coupling mechanism): Added five quartet-regularizer ablations (Table A), showing that matched auxiliary supervision without cross-state coupling yields no gain over Vanilla GRPO, while all four terms jointly drive the improvement.
  • W2 (Original AVHBench protocol & second backbone): Verified gains under the original AVHBench protocol (Table B, 84.50% Overall) and reproduced consistent gains over SFT and Vanilla GRPO on MiniCPM-o 2.6 (Table C, 80.66% JAA).

We will incorporate all of your suggestions and these new results into the revised version of the paper. As the discussion deadline is approaching, we would be grateful if you could review our responses and let us know if any concerns remain.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.