Alignment-State-Aware Preference Optimization for Mitigating Hallucinations in Multimodal Large Language Models

Direct Preference Optimization (DPO) has emerged as a widely adopted approach to mitigating hallucinations in multimodal large language models (MLLMs), but it relies on response-level preference signals without distinguishing the importance of individual tokens. Recent studies have attempted to achieve fine-grained, token-level preference optimization by leveraging visual relevance; however, these approaches still suffer from two key limitations. First, visual relevance does not necessarily align with the model's optimization needs: initially difficult tokens may become well aligned during training, making continued emphasis less beneficial despite their high visual relevance. Second, fine-grained token weighting alters the aggregated reward scale, causing preference margins to shift uncontrollably and destabilizing the overall optimization strength. To address these issues, we propose Alignment-State-Aware Preference Optimization (AsAPO). Specifically, AsAPO constructs a perceptual boundary from rejected responses to characterize the model's predictive behavior for a given input. Relative to this boundary, it dynamically estimates each token's optimization needs within the preferred response at the current alignment state and assigns weights accordingly, directing supervision toward content that remains insufficiently aligned as training progresses. Furthermore, AsAPO introduces a preference-margin stabilization constraint to suppress excessive margin shifts induced by token weighting, enabling effective fine-grained supervision while maintaining stable preference optimization. Extensive experiments across three MLLMs and four hallucination benchmarks show that AsAPO consistently outperforms standard DPO and existing fine-grained methods, effectively improving visual faithfulness and mitigating hallucinations.