- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to mitigate hallucinations in MLLMs which relies on response-level preference signals without distinguishing the importance of individual tokens. The authors argued that visual relevance does not necessarily align with the model’s optimization needs and fine-grained token weighting alters the aggregated reward scale. In light of this, the authors proposed Alignment-State-Aware Preference Optimization which dynamically quantifies token importance through the prediction difficulty of preferred responses relative to rejected responses. Experiments across three MLLMs verify the superiority of the proposed method.
Critical strengths
- The authors propose a novel perspective on fine-grained preference optimization for mitigating hallucinations in MLLMs, which identifies visual relevance from optimization importance.
- The authors introduced AsAPO to quantify the token importance based on the prediction difficulty of preferred response.
- The authors designed a preference-margin stabilization constraint to curb excessive margin shifts.
Critical weaknesses
- The authors should conduct ablation studies with simpler variants, such as direct NLL-based weighting or preferred-response-based normalization to verify the importance of the perceptual boundary. Meanwhile, they also did not explain the motivation for equation 6.
- The authors stated that the token importance is dynamically adjusted according to the model’s evolving alignment state. However, the authors did not provide the token weights changes during training. Visualizing the token weight across the training stages might provide a clearer illustration.
- The proposed method introduces additional toke-level difficulty estimation and stabilization objective. Hence, the authors are suggested to provide the computational efficiency analysis, including training time and memory overhead
- Relying on the prediction difficulty for the token importance estimation without visual relevance may lead to that the difficult tokens with linguistic reasons instead of the visual alignment receive large weights.
- The proposed token importance estimation relies primarily on prediction difficulty without explicitly incorporating visual relevance. As a result, tokens that are difficult to predict for linguistic reasons, rather than insufficient visual grounding, may also receive large weights. It would be valuable to investigate whether combining alignment-state-aware difficulty with visual relevance could provide more targeted supervision for hallucination mitigation.
Questions for authors
See Weaknesses above
5 comments
Response to W3: We thank the reviewer for raising the question of training efficiency. AsAPO reuses the token log-probabilities that standard DPO already computes for response rewards. Negating these values gives the NLLs needed for difficulty estimation, so AsAPO requires no additional forward passes through the policy or reference model. The additional operations include computing and normalizing token weights and calculating the stabilization loss from the weighted and unweighted rewards. Their computational cost scales linearly with response length.
We compared total training time and peak memory per GPU for DPO and AsAPO under the same training setup. (training on RLHF-V dataset)
| method | Total training time | Peak memory per GPU |
|---|---|---|
| DPO | 3h10m | 60.19 |
| AsAPO | 3h21m | 56.93 |
AsAPO increased total training time by 11 minutes, which we consider acceptable given the improvements in hallucination mitigation. Peak GPU memory usage was lower for AsAPO than for DPO in the measured runs. This difference may reflect run-to-run variability in training. Under this setup, we observed no increase in peak GPU memory usage.
Response to W4,W5:
We thank the reviewer for this thoughtful point. We agree that prediction difficulty alone may in principle reflect linguistic factors, and we would like to clarify our perspective. The goal of DPO is to align the model with the hallucination-free response. Token weighting only redistributes optimization effort within the preferred response toward content the current model has not yet aligned; visual relevance is one proxy for such importance, a means rather than the objective. Visual alignment alone is not an ideal end goal. Prior work [1,2] shows that models may still hallucinate even when attending to the correct image regions, so treating visual alignment as the final objective is insufficient for hallucination mitigation. Hallucination-free generation depends on whether the model can ultimately produce the correct response, which is what our alignment-state-aware weighting targets.
We appreciate the reviewers’ suggestion that jointly incorporating visual weighting is a promising direction, particularly for training-free approaches, such as hidden-state feature enhancement, where visual information can be explicitly taken into account. We are planning to explore this direction in future work.
[1] Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating. [2] MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs.
Once again, we sincerely thank the reviewers for their valuable comments, which have been very helpful in improving our paper. We will revise the manuscript accordingly. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our work.
Kindly following up on our rebuttal
Dear Reviewer mPRm,
Thank you again for your review and constructive suggestions. As the discussion period ends in 2 days, we would like to kindly follow up on our response.
Following your suggestions, we added ablations with direct NLL weighting and preferred-response normalization, a visualization of token weights across training checkpoints, and a training time and memory comparison with DPO. We also explained the motivation of Eq. (6) and our view on visual relevance.
A summary of all responses is posted at the top. If any point remains unclear or you have further suggestions, we would be happy to address them before the deadline. Thank you for your time.
We sincerely thank the reviewer for the time and valuable comments. Below, we respond to each of them point by point.
Response to W1:
We thank the reviewer for the suggestions on simpler variants and the perceptual boundary.
Direct NLL weighting. The NLL of a preferred token measures its absolute difficulty under the current policy, but the same NLL may correspond to different relative difficulty across samples. AsAPO therefore uses the reference model’s prediction on the rejected response, which is sampled from the model itself, as a sample-specific difficulty reference, and weights each preferred token by its deviation from this reference. At the beginning of the method design, we considered directly using the raw token NLL as the weight (without normalization, but with the stabilization constraint). However, the results were unsatisfactory, as shown below (trained on our own data):
Direct NLL weighting improves AMBER-G CHAIR and Hal over standard DPO, but Cog is unchanged and Object-Hal CHAIR$_S$ only drops from 24.33 to 24.00, so the gain over DPO is limited. The full AsAPO is better on all five metrics.
Preferred-response normalization. Following the suggestion, we replaced b and σl in Eq. (6) with the mean and standard deviation of the preferred response’s own NLL. Since other reviewers raised the question of training data, we ran this additional experiment on the RLHF-V dataset, with same data and hyperparameters for both variants. More weighting ablations under this setting are provided in our response to Reviewer J9b4 (response to W1,Q1).
The preferred-anchored variant improves over the LLaVA baseline. AsAPO further improves on four of the five metrics, while the preferred-anchored variant performs slightly better on CHAIRI. These results suggest that using the rejected response as the calibration boundary helps assign token weights that better reflect the model’s current alignment needs.
Motivation of Eq. (6). Different images and questions differ in prediction difficulty, so the same token NLL may imply different learning needs. Eq. (6) follows a standardization (z-score) idea: it builds a sample-specific reference coordinate system from the rejected response, which is sampled by the model itself, using the frozen reference model’s mean NLL b as the origin and its standard deviation σl (plus ϵ) as the scale. Each preferred token’s Iw,t, which evolves with the policy, is then mapped into this coordinate system. Subtracting b judges whether a token is harder or easier than the reference level, and dividing by σl measures the deviation relative to the sample’s own variability. AsAPO thus assigns higher weights to preferred tokens that are relatively harder to predict.
We thank the reviewer for their insightful and constructive comments, and we will clarify the respective roles of centering and scaling near Eq. (6) in Sec. 3.1 and add these ablations to the revised paper.
Response to W2:
We thank the reviewer for this suggestion. We have added a visualization showing how the weights of tokens in the same preferred response change during training. We held the image, question, and preferred response fixed and computed the final token weights wt from Eq. (8) at multiple training checkpoints. Visualizations and analysis can be found via the anonymous link. :[Anonymous link]