acceptodds
Under review as a conference paper at ICLR 2027

H-MPO: Hierarchical Multi-Response Preference Optimization with Modality-Guided Decoding for Omni Large Language Models

Abstract

Omni-modal large language models (OLLMs) exhibit capabilities in processing and integrating audio-visual modalities but remain vulnerable to cross-modal hallucinations. To mitigate this, existing methods employ binary preference learning by contrasting relevant responses against completely disconnected ones. This coarse-grained method overlooks intermediate hallucination states, leading to sparse supervision that struggles to address complex cross-modal conflicts. In this paper, we introduce Hierarchical Multi-Response Preference Optimization (H-MPO), which shifts preference learning from a binary comparison to a multi-hierarchy process. To obtain hierarchical preference data, we design an annotation-free Modality-Guided Decoding pipeline that synthesizes a four-level preference ranking set, capturing distinct degrees of modality faithfulness. To leverage the constructed data structure, H-MPO incorporates asymmetric margin constraints to jointly optimize the preference chain. We further propose a token-level reweighting mechanism to strengthen the supervisory signal for modality-sensitive tokens. Extensive experiments on hallucination benchmarks and general tasks demonstrate that H-MPO mitigates cross-modal hallucinations while preserving the general perception and reasoning capabilities of models. Our code and dataset will be released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Omni-modal large language models, Preference Optimization, Hallucination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_24yMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes H-MPO to reduce cross-modal hallucinations in omni-modal language models. It uses modality-guided decoding to construct four ranked responses, then applies a SimPO-based objective with asymmetric margins and token-level reweighting. Experiments on Qwen2.5-Omni-3B and 7B show improvements on hallucination-specific as well as genral benchmarks.

Critical strengths
  • The pipeline requires neither manual preference annotation nor a stronger teacher model. This reduces the cost of constructing preference data and makes the approach applicable where reliable annotations or stronger models are unavailable.
  • The method improves aggregate hallucination metrics at both model sizes, results also show gains in general audio-visual and audio understanding.
Critical weaknesses
  • Limited training queries coverage. Training samples uses only audio-only or video-only description questions, which is not diverse and differ significantly with many downstream apllications. Although results on DailyOmni are encouraging, performance on questions requiring both modalities remains unclear, limiting evidence for broader audio-visual generalization. I would very much like to see the improvements on questions that require both audio and video input.
  • Unverified supervision quality. Though the idea is intuively correct and is supported by some case studies, the intended four-level ordering is not independently validated, especially whether (y_1) is more accurate rather than merely more detailed than (y_2). Likewise, token probability differences measure modality sensitivity but do not establish factual grounding. These uncertainties affect the reliability of the training signals as well as better understanding of the method.
Questions for authors
  • How does H-MPO perform on DailyOmni questions (or other benchmarks) requiring both audio and video? This would clarify whether suppressing irrelevant modalities also preserves the use of complementary evidence.
  • Table 3 reports general-capability results only for 3B. Does the 7B model also preserve DailyOmni and MMAU performance? This is particularly relevant because 7B uses full fine-tuning, whereas 3B uses LoRA. Also, experiments on other model families (beyond Qwen) would improve soundness of this method.
  • The preferred response is generated by the same backbone so it may still contain errors or omit important details. Would generating it with a stronger model, or grounding it with ASR transcripts and stronger visual captions, improve H-MPO?
6 comments

We appreciate the reviewer's recognition of our core strengths, specifically that our pipeline requires neither manual preference annotation nor a stronger teacher model, and that our method consistently improves hallucination metrics across model scales while yielding gains in general audio-visual and audio understanding.

W1 & Q1: Joint Audio-Visual Understanding Evaluation

Thank you for this constructive suggestion. As shown in the table below, H-MPO consistently improves performance on tasks requiring joint understanding, covering both 7B and an additional benchmark (AV-SpeakerBench, an audiovisual QA benchmark featuring speaker-aware questions and aligned clips with trimmed audio, visual, and audiovisual segments to probe fine-grained cross-modal reasoning). On DailyOmni (which evaluates complex daily scenarios requiring joint audio-visual comprehension), H-MPO consistently improves accuracy for both Qwen2.5-Omni-3B and Qwen2.5-Omni-7B. On AV-SpeakerBench, which strictly requires aligned audio-visual cross-modal reasoning rather than single-modality shortcuts, H-MPO further yields performance gains across both model scales. These results demonstrate that while effectively suppressing cross-modal hallucinations, our method successfully preserves and leverages complementary audio-visual evidence.

ModelDailyOmni Acc.AV-SpeakerBench Acc.
Qwen2.5-Omni-3B (Base)45.1244.05
+ H-MPO (Ours)49.15 (+4.03)44.33 (+0.28)
Qwen2.5-Omni-7B (Base)58.1546.70
+ H-MPO (Ours)58.23 (+0.08)47.82 (+1.12)

Q2: General Capability Preservation (7B Full Fine-Tuning) & Cross-Architecture Generalization

Thank you for raising these valuable suggestions regarding general capability preservation and cross-architecture validation:

  • Performance Preservation on 7B Full Fine-Tuning: To address the concern regarding whether the 7B model (which undergoes full fine-tuning) preserves performance, we evaluated H-MPO on DailyOmni and MMAU. As summarized in the first table below, H-MPO consistently maintains and enhances general capabilities across both 3B (LoRA) and 7B (Full) scales without compromising multi-modal reasoning.
  • Generalization to Other Model Families: To further verify the soundness of our method across diverse architectures, we conducted additional experiments on a different model family (video-SALMONN 2+ 3B) on AVHBench under both LoRA and Full Fine-Tuning, as detailed in the second table below.
ModelDailyOmni Acc.MMAU Avg.
Qwen2.5-Omni-3B (Base)45.1246.70
+ H-MPO (Ours)49.15 (+4.03)52.20 (+5.50)
Qwen2.5-Omni-7B (Base)58.1574.90
+ H-MPO (Ours)58.23 (+0.08)75.70 (+0.80)
Model (video-SALMONN 2+)A$\rightarrow$V Acc.A$\rightarrow$V Prec.A$\rightarrow$V Rec.A$\rightarrow$V F1V$\rightarrow$A Acc.V$\rightarrow$A Prec.V$\rightarrow$A Rec.V$\rightarrow$A F1Overall Avg. Acc.
video-SALMONN 2+-3B (Base)75.73%71.97%84.07%77.55%67.92%62.02%92.30%74.19%71.83%
+ H-MPO (LoRA)81.20%83.85%77.17%80.37%67.96%62.25%91.16%73.98%74.58% (+2.75%)
+ H-MPO (Full)81.02%89.77%69.91%78.61%71.15%67.44%81.71%73.89%76.09% (+4.26%)

W2 & Q3: Self-Generated Preference Data vs. Stronger Teacher Data (7B →\rightarrow 3B)

Thank you for proposing this insightful direction and perspective. Our Modality-Guided Decoding (MGD) and H-MPO framework is intentionally designed to be self-contained, constructing multi-level preference data entirely from the model itself without relying on stronger teacher models. Nevertheless, we shared the exact same intuition early in our exploration and conducted an ablation study on Qwen2.5-Omni-3B using preference data synthesized by a stronger Qwen2.5-Omni-7B model. As shown in the table below, using 7B-generated data not only lags far behind self-generated 3B data, but actually degrades performance below the untuned 3B baseline:

Training Data SourceCMM V-LCMM A-LCMM V-A-LAVHBench (A$\rightarrow$V) AccAVHBench (A$\rightarrow$V) F1
Qwen2.5-Omni-3B (Baseline)91.0085.5091.7577.7873.31
H-MPO (7B-Data →\rightarrow 3B)88.0085.7587.5072.9063.84
H-MPO (3B-Data →\rightarrow 3B, Ours)91.2586.5093.0082.0879.22

This empirical finding strongly validates the core design philosophy of our self-synthesized pipeline: the essence of mitigating cross-modal hallucination is not injecting external new knowledge from a stronger teacher, but calibrating the model to suppress its own intrinsic cross-modal interference.

Thank you for the detailed response and additional experiments. The new results on joint audio-visual understanding, general capability preservation for the 7B model, and another model family address most of my concerns and are encouraging.

However, my second weakness regarding supervision quality remains insufficiently addressed. Particularly whether y1y_1 is more factually accurate rather than simply more detailed than y2y_2. Likewise, the connection between token-level probability differences and factual grounding remains unverified.

Overall, I appreciate the additional evidence and remain positive about the paper. I will maintain my original rating, which already reflects a favorable assessment.

Reviewer_7GGfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present Hierarchical Multi-Response Preference Optimization (H-MPO), a novel framework that transitions preference learning from binary comparisons into a fine-grained, multi-hierarchy scheme. Specifically, the method introduces an annotation-free Modality-Guided Decoding (MGD) pipeline to synthesize a four-tier ranking set capturing graded degrees of modality faithfulness. The framework further incorporates asymmetric margin constraints to jointly optimize the preference hierarchy, along with a token-level reweighting mechanism to dynamically amplify supervision on modality-sensitive tokens. Extensive evaluations on both hallucination and general multimodal benchmarks validate that H-MPO effectively mitigates cross-modal hallucinations while well preserving foundational perception and reasoning capabilities. Overall, this work presents a thoughtful and well-motivated approach to fine-grained multimodal preference alignment.

Critical strengths

The proposed Modality-Guided Decoding (MGD) effectively tackles the data bottleneck in multimodal preference learning. By manipulating modality-conditioned logits, it automatically synthesizes a four-tier preference dataset directly from the model without human annotations. This design naturally avoids cross-model vocabulary mismatch and provides dense, cost-effective supervision across intermediate hallucination states.

Critical weaknesses
  1. The token-level reweighting cannot rule out confounding part-of-speech bias. It claims to assign higher weights to tokens sensitive to cross-modal signals, yet the improvement may simply comes from boosting weights for all content words. Content words inherently carry higher hallucination risk regardless of cross-modal dependency. The paper lacks a POS-based reweighting baseline to disentangle modality sensitivity and lexical category effect, making it hard to validate the core claim of this module.
  2. The MGD decoding pipeline is designed to produce four monotonically decreasing fidelity levels, which implies a meaningful gap between y^{(3)} and y^{(4)}. However, the proposed loss discards this ordering signal. The paper does not analyze “why the ordering signal between two low-fidelity samples cannot be safely exploited”, only showing that full pairwise decomposed loss hurts performance. It is unclear whether the harm comes from the y^{(3)}≻y^{(4)} constraint specifically or the total number of pairwise constraints.
Questions for authors
  1. For the token-level reweighting, can the authors implement a POS-based reweighting baseline, where weight is determined purely by lexical category instead of modality log-probability difference? This baseline can verify whether the gain comes from modality-sensitive weighting or simply from up-weighting all content words.
  2. The token weight w_t is precomputed offline on the frozen initial base model and kept static through the whole training process. As model parameters are updated, model probability distribution changes. Could authors analyze the mismatch between static precomputed w_t and the real modality sensitivity of the model after training?
  3. The asymmetric margin design uses multiplier=2 for L_1. Although the base hyperparameter γ is swept in ablation, the multiplier factor itself is not evaluated. The additional ablation results for different multiplier values should be provided to justify the setting.
3 comments

We appreciate the reviewer's recognition of how Modality-Guided Decoding (MGD) tackles the data bottleneck in multimodal preference learning—automatically synthesizing a four-tier preference dataset without human annotations, avoiding cross-model vocabulary mismatch, and providing dense, cost-effective supervision across intermediate hallucination states.

W1 & Q1: Modality-Sensitive Token Weighting vs. POS-Based Reweighting

We thank the reviewer for this insightful question and fully agree that cross-modal hallucinations predominantly manifest on content words. To verify whether our performance gains stem from modality-sensitive weighting rather than generic POS bias, we implemented and trained four pure POS-based reweighting baselines on Qwen2.5-Omni-3B under the exact same training setup:

  1. Uniform Content-Word Weighting (w∈{1.2,1.5,2.0}w \in \{1.2, 1.5, 2.0\}): All content words are assigned a fixed weight, while other words are set to 1.01.0.
  2. Multi-Class POS Fixed Weighting: Distinct pre-specified weights per lexical category (NOUN/PROPN =1.5= 1.5, VERB =1.3= 1.3, ADJ =1.2= 1.2, ADV =1.1= 1.1, others =1.0= 1.0).

As summarized in the table below, while static POS up-weighting improves over baseline performance, it exhibits a clear trade-off between perception and hallucination resistance. Specifically, while static POS baselines achieve competitive overall accuracy on CMM by inflating positive predictions (higher PA), they do so at the significant expense of Hallucination Resistance (HR), where they fall notably short of our modality-sensitive wtw_t (86.83%--87.75% vs. our 89.20%). In contrast, our wtw_t achieves superior performance on AVHBench while maintaining robust hallucination suppression on CMM. Fundamentally, static POS weighting is modality-blind and indiscriminately amplifies all content words, including ungrounded or distractor elements; whereas our wtw_t selectively targets tokens with true cross-modal log-probability divergence, validating our core claim.

Method / StrategyNOUN / PROPNVERBADJADVOthersAVH Audio-drivenAVH Video-drivenAVH Avg. Acc.CMM PACMM HRCMM Acc.
Base Model----------77.78%59.03%68.41%74.8090.0082.40
POS Content-Word (w=1.2w=1.2)1.21.21.21.21.079.70%72.16%75.93%85.1786.8386.00
POS Content-Word (w=1.5w=1.5)1.51.51.51.51.080.58%73.60%77.09%84.7587.2586.00
POS Content-Word (w=2.0w=2.0)2.02.02.02.01.082.08%74.13%78.10%83.9287.7585.83
POS Multi-Class Fixed1.51.31.21.11.081.29%74.43%77.86%84.0087.5085.75
Modality-Sensitive wtw_t (Ours)Dynamicviawt∈[0.5,1.5]w_t \in [0.5, 1.5]82.08%76.62%79.35%80.0089.2084.60

Q2: Rationale for Offline Precomputation of wtw_t

Thank you for raising this question. We address this design choice from the following two aspects:

  • Efficiency via Offline Precomputation: Computing wtw_t online during training would require multiple additional forward passes per sample, introducing a 3×3\times computational overhead and severely increasing training latency. Offline precomputation completely avoids this training-time bottleneck.
  • Stability of Modality-Sensitive Token Positions: The specific tokens carrying cross-modal semantics (such as modality-grounded objects or hallucinated nouns) versus function words are intrinsic properties of the text-modality pair. Moreover, since our training preference data is also generated offline, the precomputed wtw_t seamlessly aligns with the training pipeline, stably pinpointing these critical positions without requiring online recalibration.

W2: Why (y(3),y(4))(y^{(3)}, y^{(4)}) Are Clustered Rather Than Directly Ordered via y(3)≻y(4)y^{(3)} \succ y^{(4)}:

Thank you for raising this insightful point. In pairwise preference optimization, explicitly enforcing y(3)≻y(4)y^{(3)} \succ y^{(4)} does not merely penalize y(4)y^{(4)}; it simultaneously treats y(3)y^{(3)} as a positive winner and assigns it positive gradients. Because both y(3)y^{(3)} and y(4)y^{(4)} reside in the low-fidelity hallucination regime, directly rewarding a hallucinated response (y(3)y^{(3)}) injects false cross-modal correlations and overfits to fine-grained noise between flawed samples. Clustering (y(3),y(4))(y^{(3)}, y^{(4)}) via their mean reward r(3)+r(4)2\frac{r^{(3)}+r^{(4)}}{2} ensures both hallucination tiers receive strictly negative gradients relative to grounded responses (y(1),y(2)y^{(1)}, y^{(2)}) while jointly covering diverse hallucination severities.

Crucially, omitting an explicit y(3)≻y(4)y^{(3)} \succ y^{(4)} loss term does not discard the intrinsic fidelity gap between y(3)y^{(3)} and y(4)y^{(4)}: as shown in the training diagnostics below, although r3r_3 and r4r_4 start nearly identical in Epoch 1, a clear separation naturally emerges across training (r3−r4r_3 - r_4 widens from +0.30+0.30 to +5.10+5.10, and r3>r4r_3 > r_4 consistency rises from 47.36% to 75.91%). Because y(3)y^{(3)} retains partial modality truth while y(4)y^{(4)} is completely disconnected, learning to ground on target-modality evidence via our clustered objective naturally induces the 4-level reward ordering (r1>r2>r3>r4r_1 > r_2 > r_3 > r_4 reaching 72.85%) as an emergent property, without the instability of explicitly rewarding y(3)y^{(3)}.

Training diagnosticEpoch 1Epoch 2Epoch 3
Mean logged reward, r1r_1−3.54-3.54−7.01-7.01−7.58-7.58
Mean logged reward, r2r_2−7.95-7.95−17.74-17.74−20.89-20.89
Mean logged reward, r3r_3−12.28-12.28−28.03-28.03−33.45-33.45
Mean logged reward, r4r_4−12.58-12.58−31.36-31.36−38.56-38.56
Mean gap, r3−r4r_3-r_4 ↑\uparrow+0.30+0.30+3.33+3.33+5.10+5.10
Median gap, r3−r4r_3-r_4 ↑\uparrow−0.13-0.13+3.11+3.11+5.11+5.11
Logged steps with r3>r4r_3>r_4 ↑\uparrow47.36%69.72%75.91%
Logged steps with r1>r2>r3>r4r_1>r_2>r_3>r_4 ↑\uparrow39.60%64.44%72.85%

Q3: Sensitivity Analysis of the Margin Multiplier in L1\mathcal{L}_1

Thank you for this constructive suggestion. To evaluate the impact of the multiplier factor in L1\mathcal{L}_1 (where our default setting uses a multiplier of 2 with γ=0.5\gamma=0.5), we conducted additional ablation experiments by varying the L1\mathcal{L}_1 margin multiplier across values of 1, 2 (default), and 3 on Qwen2.5-Omni-3B.

As summarized in the table below, using a multiplier of 1 makes the margin too lenient, leading to insufficient suppression of severe hallucinations. Conversely, increasing the multiplier to 3 makes the penalty overly conservative, as an excessively large margin forces the model to reject valid descriptions. Our default multiplier of 2 strikes the optimal trade-off, achieving superior overall performance across both AVHBench and CMM benchmarks.

MultiplierCMM PACMM HRCMM Acc.AVH A$\rightarrow$V Acc.AVH A$\rightarrow$V F1AVH V$\rightarrow$A Acc.AVH V$\rightarrow$A F1AVH Avg. Acc.
Multiplier = 184.17%85.58%84.88%81.38%78.88%75.09%77.47%78.24%
Multiplier = 2 (Default)80.00%89.20%84.60%82.08%79.22%76.62%76.67%79.35%
Multiplier = 382.58%87.00%84.79%80.76%77.80%76.49%77.75%78.63%

Addendum to W1 & Q1 (Supporting Diagnostic Evidence for wtw_t):

In addition to the quantitative POS-reweighting ablation in our first response—which directly demonstrates that our modality-sensitive weight wtw_t outperforms pure POS-based weighting—we further highlight our token-level diagnostics in Appendix A.6 (Figures 5–9), which visualize the underlying probability difference δt\delta_t used to compute wtw_t.

These diagnostics verify that the high-weight tokens prioritized by wtw_t indeed originate from genuine cross-modal conditioning divergence (positive δt\delta_t on target-grounded cues in chosen responses and negative δt\delta_t on distractor-driven hallucinations in rejected responses, even within the same mixed sentence in Fig. 9), rather than modality-blind lexical statistics. Moreover, unlike static POS baselines that apply a uniform multiplier to all content words within a lexical category, wtw_t assigns continuous, sensitivity-calibrated magnitudes (e.g., ∣δt∣|\delta_t| spanning from 0.980.98 to 8.538.53 across Figs. 5–6), concentrating supervision on the most cross-modally diagnostic tokens and avoiding the HR degradation observed with static POS reweighting (89.20% vs. 86.83%–87.75%). We will add explicit cross-references to Appendix A.6 in Sections 3.2 and 3.4 to make this mechanistic validation immediately clear.

Reviewer_THBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes H-MPO, a preference optimization framework for mitigating cross-modal hallucinations in omni-modal language models. The method first uses Modality-Guided Decoding to synthesize four responses with decreasing modality faithfulness by combining logits conditioned on the full input, the question-relevant modality, and the irrelevant modality. It then trains on these responses using an asymmetric multi-response preference objective and token-level reweighting based on modality-conditioned log-probability differences. Experiments on Qwen2.5-Omni-3B and 7B show improvements on AVHBench and CMM while maintaining performance on DailyOmni and MMAU.

Critical strengths
  • The paper addresses an important failure mode of omni-modal models and provides an intuitive motivation for moving beyond binary preference pairs to intermediate cross-modal hallucination states.
  • The proposed data construction is annotation-free, and the overall framework combines response-level ranking with token-level modality sensitivity without introducing inference-time overhead. The experiments cover both hallucination-specific and general understanding benchmarks at two model scales.
Critical weaknesses
  1. The validity of the four-level preference hierarchy and its use in the training objective are not sufficiently established.

    • The decoding coefficients do not guarantee that the generated responses strictly satisfy y(1)≻y(2)≻y(3)≻y(4)y^{(1)} \succ y^{(2)} \succ y^{(3)} \succ y^{(4)}, yet the hierarchy is supported only by selected qualitative examples rather than human or independent-judge validation.
    • The objective constrains r1>r2r_1>r_2 and places r1,r2r_1,r_2 above the mean of r3,r4r_3,r_4, but never enforces r3>r4r_3>r_4. It therefore learns a three-level structure with an unordered hallucination cluster rather than the claimed full four-level chain. The authors should validate the synthesized rankings and either adopt a genuine listwise objective or revise the corresponding claims.
  2. The experimental comparisons and model coverage are insufficient to establish the claimed advantage and generality.

    • MoD-DPO is presented as the closest prior work and a direct motivation for H-MPO, but is not included in the experiments. Comparisons with generic SimPO, training-free methods, and OmniDPO only at 7B do not establish an advantage over the most relevant cross-modal preference optimization baseline.
    • All experiments use Qwen2.5-Omni; the 3B and 7B variants share the same architecture and modality interface. Since Modality-Guided Decoding depends on model-specific behavior under audio-only, video-only, and joint inputs, evaluation on another omni-modal model family, together with multi-seed results, is needed to support the generality and stability claims.
Questions for authors

See weakness

2 comments

We sincerely thank the reviewer for recognizing our intuitive motivation beyond binary preference pairs, our annotation-free framework without inference-time overhead, and our comprehensive evaluation across benchmarks and model scales.

W1: Validity of the 4-Level Preference Hierarchy & Training Objective:

Thank you for providing this constructive suggestion. Since our Modality-Guided Decoding relies on automated synthesis without manual annotations, the generated responses satisfy the intended multi-level hierarchy at the macro level for the vast majority of samples, while individual samples might indeed contain some noise. To address this issue, our Cluster-Averaged Loss is specifically designed to be robust against such noise, ensuring stable optimization while preserving the macro-level supervision hierarchy.

  • Why Cluster Mean for (y(3),y(4))(y^{(3)}, y^{(4)}) Instead of Unrolled Pairwise Comparisons: Because both y(3)y^{(3)} and y(4)y^{(4)} reside in the low-quality hallucination regime, treating them as separate pairwise penalty levels forces the model to overfit to minor noise among hallucinated samples. As evaluated in Appendix A.5 (Table 5) via the Decomposed Loss baseline (which unrolls the cluster mean r(3)+r(4)2\frac{r^{(3)}+r^{(4)}}{2} into individual pairwise terms), this unrolled formulation inflates Hallucination Resistance (CMM HR) at the cost of severe perception collapse (degrading CMM PA and AVHBench F1, as shown in the table below). Using the cluster mean jointly covers diverse hallucination severities without over-penalizing negative-pair noise, outperforming both single-negative baselines (Table 4a) and the Decomposed Loss (Appendix A.5, Table 5).
Qwen2.5-Omni-3B VariantAVHBench AccAVHBench F1CMM PACMM HRCMM HM
Base Model68.456.974.890.082.4
y(1)y^{(1)} vs. y(3)y^{(3)} only (Main Paper, Table 4a)77.274.976.790.883.7
y(1)y^{(1)} vs. y(4)y^{(4)} only (Main Paper, Table 4a)76.573.573.791.082.3
Decomposed Loss (Unrolled Pairwise, Appendix A.5 Table 5)74.167.869.393.381.3
H-MPO (Cluster-Averaged, Ours)79.478.080.089.284.6
  • Emergent 4-Level Separation Validates Intrinsic Data Quality Gap: Crucially, as demonstrated by the training diagnostics in the second table below, where r3r_3 and r4r_4 initially start close in Epoch 1 and steadily widen in both their reward gap and ordering consistency across subsequent epochs, a clear separation between Rank 3 and Rank 4 naturally emerges precisely because our synthesized data possesses a genuine intrinsic quality gap between y(3)y^{(3)} and y(4)y^{(4)}, with y(3)y^{(3)} retaining partial truth while y(4)y^{(4)} is completely disconnected. As the model's capability improves and it learns to ground on target-modality evidence, this intrinsic data difference naturally translates into consistent multi-tier reward orderings over time.
Training diagnosticEpoch 1Epoch 2Epoch 3
Mean logged reward, r1r_1−3.54-3.54−7.01-7.01−7.58-7.58
Mean logged reward, r2r_2−7.95-7.95−17.74-17.74−20.89-20.89
Mean logged reward, r3r_3−12.28-12.28−28.03-28.03−33.45-33.45
Mean logged reward, r4r_4−12.58-12.58−31.36-31.36−38.56-38.56
Mean gap, r3−r4r_3-r_4 ↑\uparrow+0.30+0.30+3.33+3.33+5.10+5.10
Median gap, r3−r4r_3-r_4 ↑\uparrow−0.13-0.13+3.11+3.11+5.11+5.11
Logged steps with r3>r4r_3>r_4 ↑\uparrow47.36%69.72%75.91%
Logged steps with r1>r2>r3>r4r_1>r_2>r_3>r_4 ↑\uparrow39.60%64.44%72.85%

Following the reviewer's valuable advice, we will revise the corresponding claims in the manuscript to soften overly absolute statements, accurately clarifying that our objective optimizes a macro-level preference hierarchy with a clustered negative formulation rather than strictly enforcing a rigid four-level chain across every sample.

W2-1: Comparison with MoD-DPO

We completely understand and agree with the reviewer. However, a direct experimental comparison is constrained by resource availability, and the two methods also operate under different data and annotation paradigms:

  • Code/Checkpoint Availability: MoD-DPO was not included in our initial tables because its code, synthesized dataset, and model checkpoints are not publicly released.
  • Data Scale & External-Model Annotation (MoD-DPO) vs. Streamlined Data Construction (H-MPO): Crucially, the two methods differ substantially in training data scale, construction complexity, and external dependencies. Specifically, MoD-DPO relies on 10,854 videos annotated via a multi-stage pipeline involving three external foundation/expert models (GPT-4o, RAM++, and AudioFlamingo 3). In contrast, H-MPO uses only 1,212 videos constructed through a much more concise pipeline without requiring external annotation models. Despite using far less training data and a simpler data construction process, H-MPO achieves strong hallucination mitigation while also eliminating reference-model memory overhead during training.

W2-2: Evaluation on Another Model Family and Multi-Seed Stability

To comprehensively verify both the cross-architecture generality and multi-seed training stability of our framework, we conducted additional experiments from the following two aspects:

  • Cross-Architecture Generality on video-SALMONN 2+: To verify framework agnosticism, we applied our pipeline to video-SALMONN 2+ (3B) under both LoRA and Full Fine-Tuning. As shown below, H-MPO consistently improves performance across subtasks and tuning paradigms, confirming strong generalization across diverse architectures.
Model (video-SALMONN 2+)A$\rightarrow$V Acc.A$\rightarrow$V Prec.A$\rightarrow$V Rec.A$\rightarrow$V F1V$\rightarrow$A Acc.V$\rightarrow$A Prec.V$\rightarrow$A Rec.V$\rightarrow$A F1Overall Avg. Acc.
video-SALMONN 2+-3B (Base)75.73%71.97%84.07%77.55%67.92%62.02%92.30%74.19%71.83%
+ H-MPO (LoRA)81.20%83.85%77.17%80.37%67.96%62.25%91.16%73.98%74.58% (+2.75%)
+ H-MPO (Full)81.02%89.77%69.91%78.61%71.15%67.44%81.71%73.89%76.09% (+4.26%)
  • Multi-Seed Training Stability on Qwen2.5-Omni-7B: To rigorously evaluate stability, we conducted additional runs with different random seeds (Seed 43 and Seed 44). As summarized below, H-MPO exhibits strong stability with minimal variance while consistently outperforming the baseline across all runs.
Configuration (Qwen2.5-Omni-7B)AVHBench A$\rightarrow$V (Acc %)AVHBench V$\rightarrow$A (Acc %)AVHBench Average (Acc %)CMM Overall
Qwen2.5-Omni-7B (Baseline)82.8382.3182.5785.70
H-MPO (Reported in Paper)83.5884.1383.8687.80
H-MPO (Seed 43)82.8883.5783.2387.21
H-MPO (Seed 44)83.0583.8383.4487.50

Overall, evaluations across different model architectures and multiple random seeds demonstrate the strong generality and training stability of H-MPO, consistently yielding robust improvements over the base models.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your constructive feedback.

We have carefully addressed your concerns in our rebuttal and provided additional experimental results and clarifications. As the discussion period is due to end soon, we would greatly appreciate it if you could take a moment to review our responses and let us know if you have any further questions or concerns.

We would be happy to provide additional clarification if needed.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.