acceptodds
Under review as a conference paper at ICLR 2027

Alignment-State-Aware Preference Optimization for Mitigating Hallucinations in Multimodal Large Language Models

Abstract

Direct Preference Optimization (DPO) has emerged as a widely adopted approach to mitigating hallucinations in multimodal large language models (MLLMs), but it relies on response-level preference signals without distinguishing the importance of individual tokens. Recent studies have attempted to achieve fine-grained, token-level preference optimization by leveraging visual relevance; however, these approaches still suffer from two key limitations. First, visual relevance does not necessarily align with the model's optimization needs: initially difficult tokens may become well aligned during training, making continued emphasis less beneficial despite their high visual relevance. Second, fine-grained token weighting alters the aggregated reward scale, causing preference margins to shift uncontrollably and destabilizing the overall optimization strength. To address these issues, we propose Alignment-State-Aware Preference Optimization (AsAPO). Specifically, AsAPO constructs a perceptual boundary from rejected responses to characterize the model's predictive behavior for a given input. Relative to this boundary, it dynamically estimates each token's optimization needs within the preferred response at the current alignment state and assigns weights accordingly, directing supervision toward content that remains insufficiently aligned as training progresses. Furthermore, AsAPO introduces a preference-margin stabilization constraint to suppress excessive margin shifts induced by token weighting, enabling effective fine-grained supervision while maintaining stable preference optimization. Extensive experiments across three MLLMs and four hallucination benchmarks show that AsAPO consistently outperforms standard DPO and existing fine-grained methods, effectively improving visual faithfulness and mitigating hallucinations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination, DPO, Alignment, MLLMs, Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_mPRmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to mitigate hallucinations in MLLMs which relies on response-level preference signals without distinguishing the importance of individual tokens. The authors argued that visual relevance does not necessarily align with the model’s optimization needs and fine-grained token weighting alters the aggregated reward scale. In light of this, the authors proposed Alignment-State-Aware Preference Optimization which dynamically quantifies token importance through the prediction difficulty of preferred responses relative to rejected responses. Experiments across three MLLMs verify the superiority of the proposed method.

Critical strengths
  1. The authors propose a novel perspective on fine-grained preference optimization for mitigating hallucinations in MLLMs, which identifies visual relevance from optimization importance.
  2. The authors introduced AsAPO to quantify the token importance based on the prediction difficulty of preferred response.
  3. The authors designed a preference-margin stabilization constraint to curb excessive margin shifts.
Critical weaknesses
  1. The authors should conduct ablation studies with simpler variants, such as direct NLL-based weighting or preferred-response-based normalization to verify the importance of the perceptual boundary. Meanwhile, they also did not explain the motivation for equation 6.
  2. The authors stated that the token importance is dynamically adjusted according to the model’s evolving alignment state. However, the authors did not provide the token weights changes during training. Visualizing the token weight across the training stages might provide a clearer illustration.
  3. The proposed method introduces additional toke-level difficulty estimation and stabilization objective. Hence, the authors are suggested to provide the computational efficiency analysis, including training time and memory overhead
  4. Relying on the prediction difficulty for the token importance estimation without visual relevance may lead to that the difficult tokens with linguistic reasons instead of the visual alignment receive large weights.
  5. The proposed token importance estimation relies primarily on prediction difficulty without explicitly incorporating visual relevance. As a result, tokens that are difficult to predict for linguistic reasons, rather than insufficient visual grounding, may also receive large weights. It would be valuable to investigate whether combining alignment-state-aware difficulty with visual relevance could provide more targeted supervision for hallucination mitigation.
Questions for authors

See Weaknesses above

5 comments

We sincerely thank the reviewer for the time and valuable comments. Below, we respond to each of them point by point.

Response to W1:

We thank the reviewer for the suggestions on simpler variants and the perceptual boundary.

Direct NLL weighting. The NLL of a preferred token measures its absolute difficulty under the current policy, but the same NLL may correspond to different relative difficulty across samples. AsAPO therefore uses the reference model’s prediction on the rejected response, which is sampled from the model itself, as a sample-specific difficulty reference, and weights each preferred token by its deviation from this reference. At the beginning of the method design, we considered directly using the raw token NLL as the weight (without normalization, but with the stabilization constraint). However, the results were unsatisfactory, as shown below (trained on our own data):

methodAmber-GObj-Hal
CHAIR↓hal↓Cog↓CHAIR$_s$↓CHAIR$_i$↓
llava7.734.74.255.6715.96
DPO6.625.51.224.337.96
AsAPO_NLL4.021.01.224.06.82
AsAPO2.912.30.513.004.37

Direct NLL weighting improves AMBER-G CHAIR and Hal over standard DPO, but Cog is unchanged and Object-Hal CHAIR$_S$ only drops from 24.33 to 24.00, so the gain over DPO is limited. The full AsAPO is better on all five metrics.

Preferred-response normalization. Following the suggestion, we replaced bb and σl\sigma_l in Eq. (6) with the mean and standard deviation of the preferred response’s own NLL. Since other reviewers raised the question of training data, we ran this additional experiment on the RLHF-V dataset, with same data and hyperparameters for both variants. More weighting ablations under this setting are provided in our response to Reviewer J9b4 (response to W1,Q1).

methodAmber-GObj-Hal
CHAIR↓hal↓Cog↓CHAIR$_s$↓CHAIR$_i$↓
llava7.734.74.255.6715.96
AsAPO (preferred-anchored)3.116.11.115.003.66
AsAPO3.012.80.513.674.04

The preferred-anchored variant improves over the LLaVA baseline. AsAPO further improves on four of the five metrics, while the preferred-anchored variant performs slightly better on CHAIRI_I. These results suggest that using the rejected response as the calibration boundary helps assign token weights that better reflect the model’s current alignment needs.

Motivation of Eq. (6). Different images and questions differ in prediction difficulty, so the same token NLL may imply different learning needs. Eq. (6) follows a standardization (z-score) idea: it builds a sample-specific reference coordinate system from the rejected response, which is sampled by the model itself, using the frozen reference model’s mean NLL bb as the origin and its standard deviation σl\sigma_l (plus ϵ\epsilon) as the scale. Each preferred token’s Iw,tI_{w,t}, which evolves with the policy, is then mapped into this coordinate system. Subtracting bb judges whether a token is harder or easier than the reference level, and dividing by σl\sigma_l measures the deviation relative to the sample’s own variability. AsAPO thus assigns higher weights to preferred tokens that are relatively harder to predict.

We thank the reviewer for their insightful and constructive comments, and we will clarify the respective roles of centering and scaling near Eq. (6) in Sec. 3.1 and add these ablations to the revised paper.

Response to W2:

We thank the reviewer for this suggestion. We have added a visualization showing how the weights of tokens in the same preferred response change during training. We held the image, question, and preferred response fixed and computed the final token weights wtw_t from Eq. (8) at multiple training checkpoints. Visualizations and analysis can be found via the anonymous link. :[Anonymous link]

Response to W3: We thank the reviewer for raising the question of training efficiency. AsAPO reuses the token log-probabilities that standard DPO already computes for response rewards. Negating these values gives the NLLs needed for difficulty estimation, so AsAPO requires no additional forward passes through the policy or reference model. The additional operations include computing and normalizing token weights and calculating the stabilization loss from the weighted and unweighted rewards. Their computational cost scales linearly with response length.

We compared total training time and peak memory per GPU for DPO and AsAPO under the same training setup. (training on RLHF-V dataset)

methodTotal training timePeak memory per GPU
DPO3h10m60.19
AsAPO3h21m56.93

AsAPO increased total training time by 11 minutes, which we consider acceptable given the improvements in hallucination mitigation. Peak GPU memory usage was lower for AsAPO than for DPO in the measured runs. This difference may reflect run-to-run variability in training. Under this setup, we observed no increase in peak GPU memory usage.

Response to W4,W5:

We thank the reviewer for this thoughtful point. We agree that prediction difficulty alone may in principle reflect linguistic factors, and we would like to clarify our perspective. The goal of DPO is to align the model with the hallucination-free response. Token weighting only redistributes optimization effort within the preferred response toward content the current model has not yet aligned; visual relevance is one proxy for such importance, a means rather than the objective. Visual alignment alone is not an ideal end goal. Prior work [1,2] shows that models may still hallucinate even when attending to the correct image regions, so treating visual alignment as the final objective is insufficient for hallucination mitigation. Hallucination-free generation depends on whether the model can ultimately produce the correct response, which is what our alignment-state-aware weighting targets.

We appreciate the reviewers’ suggestion that jointly incorporating visual weighting is a promising direction, particularly for training-free approaches, such as hidden-state feature enhancement, where visual information can be explicitly taken into account. We are planning to explore this direction in future work.

[1] Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating. [2] MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs.

Once again, we sincerely thank the reviewers for their valuable comments, which have been very helpful in improving our paper. We will revise the manuscript accordingly. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our work.

AuthorsOpenReview ↗

Kindly following up on our rebuttal

Dear Reviewer mPRm,

Thank you again for your review and constructive suggestions. As the discussion period ends in 2 days, we would like to kindly follow up on our response.

Following your suggestions, we added ablations with direct NLL weighting and preferred-response normalization, a visualization of token weights across training checkpoints, and a training time and memory comparison with DPO. We also explained the motivation of Eq. (6) and our view on visual relevance.

A summary of all responses is posted at the top. If any point remains unclear or you have further suggestions, we would be happy to address them before the deadline. Thank you for your time.

Reviewer_CqoWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a token-level direct preference optimization method, which uses the rejected/reference response as a baseline and the negative log_softmax value as the token importance indicator.

Critical strengths
  • This is a good direction for the development of direct preference optimization.
Critical weaknesses
  • The reviewer thinks that the authors may not fully discuss some related works. For example, works on token-level direct preference optimization [1,2,3,4,5]. It is necessary to discuss the differences between the proposed method and some of these token-level DPO works.
  • In Table 1, the methods are compared with different training data. This makes the data less reliable because we know that, at such a relatively small scale, the performance is sensitive to the data selection.
  • Please also provide fairer comparisons with other baselines rather than pure DPO. It would make the paper more covincing.

[1] Token-level Direct Preference Optimization. https://arxiv.org/pdf/2404.11999

[2] TIS-DPO: TOKEN-LEVEL IMPORTANCE SAMPLING FOR DIRECT PREFERENCE OPTIMIZATION WITH ESTIMATED WEIGHTS. https://arxiv.org/pdf/2410.04350

[3] Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability. https://arxiv.org/pdf/2411.19943

[4] Token-weighted Direct Preference Optimization with Attention. https://arxiv.org/pdf/2605.21883

[5] TOKEN-IMPORTANCE GUIDED DIRECT PREFERENCE OPTIMIZATION. https://arxiv.org/pdf/2505.19653

Questions for authors
  • Please check Critical Weaknesses.
3 comments

We sincerely thank the reviewer for their valuable time and constructive comments. Below, we provide detailed responses to each of the reviewer's concerns.

response to W1:

We thank the reviewer for the valuable suggestion regarding related work. We would like to clarify that our study primarily focuses on mitigating hallucinations in multimodal large language models (MLLMs) through DPO. Therefore, in our discussion of related work and experimental comparisons, we prioritize recent studies that are most directly relevant to multimodal hallucination mitigation, including both sequence-level and token-level preference optimization methods.

We note that the studies mentioned by the reviewer mainly focus on text-only LLMs, with objectives such as improving general preference alignment, generation quality, or reasoning capabilities, which differ from our focus on mitigating multimodal hallucinations. Although these methods also investigate token-level optimization, they primarily estimate token importance using information such as KL divergence, contrastive models, attention, or gradients. In contrast, AsAPO focuses on the model's current alignment state and dynamically adjusts token weights accordingly. Therefore, we believe that existing methods specifically developed for multimodal hallucination mitigation provide more direct and relevant comparisons, and we have prioritized their discussion in our manuscript.

response to W2,W3:

We thank the reviewer for raising concerns regarding the differences in training data. We constructed our own preference dataset because on-policy data sampled from the model itself plays an important role in DPO training [1]. Since AsAPO is evaluated on models with different architectures, each with its own output distribution, the preference data need to be sampled from each model itself, which a single fixed public dataset cannot provide. To fairly assess AsAPO, we compared it with standard DPO under identical data and experimental settings, which isolates the improvement introduced by our method. Similar evaluation protocols have been adopted in prior studies, given the computational cost and practical difficulty of reproducing existing methods.

Nevertheless, we agree that comparisons on the same training data provide more convincing evidence. To address this, we conducted additional experiments using the preference dataset adopted by closely related token-level DPO methods, including UE-DPO and CAT-PO. The results are presented below.

methoddatasetAmber-GObj-Hal
CHAIR↓hal↓Cog↓CHAIR$_s$↓CHAIR$_i$↓
llavaN/A7.734.74.255.6715.96
CAT-PORLHF-V4.823.72.1--
UE-DPORLHF-V2.917.41.013.726.69
AsAPORLHF-V3.012.80.513.674.04

CAT-PO and UE-DPO results are taken from their original papers; all methods are trained on RLHF-V. “-” denotes not reported.

AsAPO achieves the best Hal, Cog, and CHAIR$_i$, and is comparable to UE-DPO on CHAIR (3.0 vs. 2.9) and CHAIR$_s$ (13.67 vs. 13.72). These results further demonstrate the effectiveness of AsAPO in mitigating hallucinations in MLLMs and indicate that its performance gains are not solely attributable to differences in training data.

Finally, we sincerely thank the reviewer again for the valuable comments and suggestions. We will further improve our manuscript accordingly. We hope that our responses have adequately addressed the reviewer's concerns and that the reviewer will kindly reconsider their evaluation of our work.

[1] Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

AuthorsOpenReview ↗

Kindly following up on our rebuttal

Dear Reviewer CqoW,

Thank you again for your review. As the discussion period ends in 2 days, we would like to kindly follow up on our response.

To address your concern about unfair comparison due to different training data, we trained AsAPO on RLHF-V, the same dataset used by CAT-PO and UE-DPO. Under this matched-data setting, AsAPO achieves the best AMBER-G Hal, Cog, and Object-HalBench CHAIR_i, and is comparable to UE-DPO on the other two metrics. We also clarified how AsAPO differs from the text-only token-level DPO methods you listed.

A summary of all responses is posted at the top. If any concern remains unclear, we would be glad to clarify before the deadline. Thank you for your time and consideration.

Dear Reviewer CqoW,

As the discussion period is coming to an end, we would like to kindly follow up on our response, as we do not yet know whether it has addressed your concerns.

Regarding related work, we would like to note that our original submission already cited [5] (TOKEN-IMPORTANCE GUIDED DIRECT PREFERENCE OPTIMIZATION) in Section 1, as evidence that fine-grained preference optimization is effective. Since the token-level DPO works you listed target text-only LLMs, we did not include them as experimental baselines, and instead compared with methods developed for multimodal hallucination mitigation.

Regarding training data, we trained AsAPO on RLHF-V, the dataset used by CAT-PO and UE-DPO. Under this matched setting, AsAPO achieves the best AMBER-G Hal, Cog, and Object-HalBench CHAIR$_i$, and is comparable on the other two metrics.

Thank you for your time and consideration.

Comments on the paper

6 comments

We thank the reviewer for taking the time to evaluate our manuscript and for providing thoughtful feedback. Below, we address each comment in detail.

Response to W1,Q1: We thank the reviewers for this question. The rejected-response calibration is designed to adjust token weights in a sample-adaptive manner, rather than to re-rank tokens within a response. Using the model’s own rejected response as a reference, it provides a sample-specific origin and scale, so that the same NLL value is interpreted relative to the model’s predictive level for that input, and weights are comparable across samples. Raw NLL lacks this property, and the sigmoid in Eq. (8) then acts on non-negative values whose outputs lie in [0.5, 1) and saturate for harder tokens, which may limit how well weights differentiate among them.

To isolate the calibration, we compare NLL-weight, which applies Eq. (8) directly to raw NLL (at=g(Iw,t)a_t=g(I_{w,t})) with the same normalization and stabilization, against AsAPO under identical data (RLHF-V,Given the reviewer’s concern in W2).

methodAmber-GObj-Hal
CHAIR↓hal↓Cog↓CHAIR$_s$↓CHAIR$_i$↓
llava7.734.74.255.6715.96
NLL-weight w/o stab4.521.62.133.009.86
NLL-weight + stab4.117.91.628.678.18
AsAPO (preferred-anchored)3.116.11.115.003.66
AsAPO3.012.80.513.674.04

NLL-weight + stab vs. AsAPO is the direct comparison asked: AsAPO is clearly better, so the gain comes from the rejected-response calibration rather than from difficulty weighting, normalization, or stabilization. Comparing NLL-weight with and without stabilization shows that the constraint helps but does not close the gap. We also tested building the boundary from the preferred response instead (preferred-anchored), as suggested by Reviewer mPRm. It improves substantially over NLL-weight, indicating that sample-level calibration itself is a major source of the gain. AsAPO is best on four of five metrics (the preferred-anchored variant is slightly better on CHAIR$_I$), supporting the use of the rejected response, which is sampled from the model itself and reflects its own predictive level, as the anchor. We will add this ablation to the revised paper.

Response to W2,Q2: We thank the reviewers for raising the matched-comparison issue. We built our own on-policy preference data because samples drawn from the model itself are important for DPO training [1], and we wanted to validate AsAPO across several architectures. The controlled comparison with standard DPO under identical data and settings was meant to isolate AsAPO’s contribution. We agree, however, that a comparison with existing fine-grained methods trained on the same dataset is more convincing. Since re-implementing prior training pipelines is costly and, here, often infeasible (the CAT-PO release is incomplete and the UE-DPO code is not yet public), we instead trained AsAPO on RLHF-V, the dataset on which CAT-PO and UE-DPO are trained, and compare against their reported results

methodAmber-GObj-Hal
CHAIR↓hal↓Cog↓CHAIR$_s$↓CHAIR$_i$↓
llava7.734.74.255.6715.96
CAT-PO4.823.72.1--
UE-DPO2.917.41.013.726.69
AsAPO3.012.80.513.674.04

Under this setting, AsAPO achieves the best Hal, Cog, and CHAIR$_I$, and is comparable to UE-DPO on CHAIR (3.0 vs. 2.9) and CHAIR$_S$ (13.67 vs. 13.72).

Cost. AsAPO reuses the log-probabilities already computed in DPO’s forward passes; token weights and the stabilization term are lightweight operations over sequence length, requiring no extra forward pass or attention extraction. Under identical hardware and settings, AsAPO’s total training time is 3h21m versus 3h10m for DPO , and its peak memory per GPU shows no noticeable increase(Refer to the response to Reviewer mPRm W3). By their descriptions, CAT-PO additionally computes token rewards from global, local, and semantic views, which may incur extra cost.

We would also like to clarify that computational efficiency is not the main advantage we claim. Rather, we regarded avoiding a significant increase in computational cost as a design prerequisite, which helps the practicality of the method. The core contribution of AsAPO is the alignment-state-aware token weighting mechanism. We will add the matched-data comparison and cost analysis to the revised paper.

Response to Q3:

We thank the reviewers for this question. Yes, the token weights are detached. The self-information Iw,tI_{w,t} and all quantities derived from it (dtd_t, ata_t, wtw_t) are computed without gradient and treated as constants in back-propagation, so that wtw_t serves purely as a scalar measure of token importance; the boundary bb and σl\sigma_l come from the frozen reference model and carry no gradient either. In implementation, we first detach the token log-probabilities computed during the DPO forward pass and then negate the detached values to obtain Iw,tI_{w,t}. We will clarify this in Sec. 3.1 and the implementation details in Sec. 4.1 of the revised paper.

Once again, we sincerely thank the reviewer for the valuable comments, which have greatly helped us improve the paper. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our work.

AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank all reviewers and the AC for their time and constructive feedback. Below we summarize the main concerns and our responses. Details are in our individual responses.

1. Matched-data comparison with token-level / fine-grained baselines (CqoW W2/W3; J9b4 W2/Q2)

Concern: Table 1 compares methods trained on different data.
Response: We trained AsAPO on RLHF-V, the dataset used by CAT-PO and UE-DPO, and compared against their reported results. AsAPO achieves the best AMBER-G Hal (12.8 vs. 17.4 for UE-DPO), the best Cog (0.5 vs. 1.0), and the best Object-HalBench CHAIRiCHAIR_i (4.04 vs. 6.69). It is comparable on AMBER-G CHAIR (3.0 vs. 2.9) and Object-HalBench CHAIRsCHAIR_s (13.67 vs. 13.72). With the training data matched, the gains come mainly from the proposed alignment-state-aware weighting rather than from the data.

2. Isolating the benefit of rejected-response calibration (mPRm W1; J9b4 W1/Q1)
Concern: Are simpler variants (direct NLL weighting, preferred-response normalization) sufficient? What is the motivation for Eq. (6)?
Response: We added ablations under identical data and settings (RLHF-V). Direct NLL weighting with the same stabilization is clearly worse than AsAPO (Object-HalBench CHAIRsCHAIR_s 28.67 vs. 13.67; AMBER-G Hal 17.9 vs. 12.8). The preferred-anchored variant is better than NLL weighting but better than AsAPO on only one of five metrics, so AsAPO is best on four of five. On our own data, NLL weighting gives only limited gains over DPO (Object-HalBench CHAIRsCHAIR_s 24.0 vs. 24.33). We also explained that Eq. (6) is a sample-specific standardization (z-score) built from the model’s own rejected response. We will add these ablations and the explanation to the revised paper.

3. Token weight dynamics during training (mPRm W2)
Response: We added a visualization of token weights across training checkpoints (image, question, and preferred response held fixed): Anonymous link

4. Computational overhead (mPRm W3; J9b4 W2/Q2)
Response: AsAPO reuses the log-probabilities from DPO’s forward pass and needs no extra forward pass. Under the same setup, total training time is 3h18m vs. 3h10m for DPO. Peak memory per GPU was 56.93 GB vs. 60.19 GB, so we observed no noticeable increase in memory.

5. Whether the token weights are detached (J9b4 Q3)
Response: Yes. The weights and all quantities derived from them are computed without gradient and treated as constants. We will clarify this in Sec. 3.1 and Sec. 4.1.

We hope these responses and new results have addressed the concerns. We would be glad to answer any further questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.