acceptodds
Desk rejected at ICLR 2027

Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Object hallucination remains a major challenge for large vision-language models. While off-policy preference optimization proves to be an effective solution, on-policy reinforcement learning provides a more promising direction as it directly targets a model's current failure modes. However, we find that without fine-grained reward formulation and allocation, on-policy optimization often falls into an easy shortcut: reducing hallucinations merely by saying less—making fewer valid claims. To comprehensively resolve this, we propose a fine-grained alignment framework that couples dense reward signals at the data level with precise credit assignment at the algorithmic level. Specifically, we first construct the Dense Object Presence and Absence (DOPA) dataset to address sparse annotations that prevent valid object claims from being verified and rewarded. DOPA exhaustively annotates the deterministic presence and absence of every concept across an expanded vocabulary, significantly increasing the density of reliable reward signals during on-policy rollouts. Second, we propose Subsentence-level Credit Assignment for on-Policy Optimization (SCAPO) to prevent response-level shared advantages from allowing local hallucinations to compromise all other valid outputs within the same response. By assigning credit to each subsentence independently based on its object claims, SCAPO can precisely reinforce faithful generations and penalize hallucinations. Furthermore, we leverage the resulting faithful image descriptions as auxiliary context to transfer generative gains to discriminative tasks. Experiments demonstrate that our method produces highly informative, faithful descriptions in generative tasks while yielding clear performance gains on discriminative evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Object Hallucination, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Hk1EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates a shortcut behavior in hallucination-oriented reinforcement learning for VLMs, where models reduce the number of generated claims instead of improving visual grounding. The authors identify sparse object annotations and coarse response-level credit assignment as two underlying causes. They propose DOPA and SCAPO (a subsentence-level reward assignment method for on-policy optimization) to address these issues and report improvements on multiple hallucination benchmarks.

Critical strengths

The distinction between improving visual faithfulness and merely reducing the number of generated claims is particularly valuable.

The joint consideration of annotation density and reward granularity is well motivated.

The comparisons between different annotation sources and credit-assignment strategies offer useful insights into the behavior of hallucination-oriented reinforcement learning.

Critical weaknesses
  1. GRPO vs. DAPO: How are GRPO and DAPO trained, and how are their rewards configured? Since DAPO is an improvement over GRPO, why does it perform substantially worse in Table 1?

  2. VTI results and baseline configurations: Why does VTI perform considerably worse than other methods on the AMBER captioning metrics in Table 1? Are the hyperparameters consistently tuned across baselines? The paper should provide the detailed configurations.

  3. Reward dynamics: The paper lacks training curves for the individual reward components in Equation (1). How do these components evolve over training steps?

  4. Advantage assignment: Why are local rewards directly broadcast as token-level advantages rather than normalized as in GRPO? What is the underlying motivation, and how is training stability maintained? Both theoretical and empirical evidence would be valuable.

  5. Inference efficiency: The two-stage inference procedure for discriminative tasks introduces additional computation. The authors should quantify its latency and token overhead.

  6. Marginal gains and alternative formats: Table 4 shows only a 0.5-point improvement (86.9% to 87.4%). Could a standard <think>...</think><answer>...</answer> format achieve similar results without an explicit description-generation stage? The authors should also release the training and inference prompts.

Questions for authors

Please refer to the concerns raised in the weaknesses. While I have several concerns regarding the methodology and experimental evaluation, I am willing to reconsider my assessment and potentially raise my score if these issues are convincingly addressed during the rebuttal.

6 comments

1. GRPO versus DAPO: settings, rewards, and performance differences

GRPO and DAPO are trained with the same Qwen2.5-VL-7B-Instruct backbone. Both GRPO and DAPO use the same base response-level reward, computed from the hallucination rate using DOPA annotations. We implemented DAPO by extending GRPO with additional components: Clip-Higher, Token-Level Policy Gradient Loss, and Overlong Reward Shaping, following their original formulations. Further training details are provided in Appendix C.1.

We hypothesize that DAPO's weaker performance may stem from the fact that its optimization strategies were primarily developed for long-form reasoning tasks and may not be equally effective for hallucination mitigation. Figure 5 in the manuscript helps explain the discrepancy: DAPO achieves a higher FaithScore than GRPO but generates fewer atomic facts in several settings, indicating a stronger conservative-output preference. This more conservative generation leads to a lower Cover Rate, thereby explaining DAPO's lower Caption Score compared with GRPO in Table 1 of the manuscript.

2. VTI results and baseline hyperparameters

For the VTI baseline, we followed the configurations provided in its official implementation (https://github.com/shengliu66/VTI). We will provide additional details on the configurations in the revised manuscript.

We observed that VTI substantially shortens the generated descriptions, resulting in low object coverage. This explains VTI's low AMBER Caption Score in Table 1 of the manuscript; it is also consistent with the “saying less” failure mode emphasized in our paper. In contrast, our SCAPO effectively mitigates this failure mode by preserving informativeness while reducing hallucinations.

3. Reward dynamics: How do the individual reward components in Equation (1) evolve over training steps?

Unlike GRPO, SCAPO does not aggregate rewards at the response level. Instead, Equation (1) defines a piecewise subsentence-level reward. Therefore, a single aggregate reward curve would not fully capture the effects of different subsentence-level rewards during training. The quantities most relevant to the reward dynamics are the frequencies of positive and hallucination-penalized entities in Equation (1), which are reflected by the Cover Rate and the Hal. Rate, respectively.

As shown in Figure 4 of the manuscript, SCAPO steadily increases Cover Rate while reducing Hal. Rate during training, consistent with the intended effects of our reward design in Equation (1).

4. Why advantages are not group-normalized? How is training stability maintained?

The central motivation is to preserve the semantic sign of each verified local reward: hallucinated subsentences should receive a negative advantage, while hallucination-free subsentences containing newly supported object claims should receive a positive advantage. Groupwise normalization, such as subtracting the group mean, can change these signs. Training stability is maintained through KL regularization, and the stable training trends in Figure 4 of the manuscript provide empirical evidence of SCAPO's stability.

We additionally conducted an ablation study using groupwise token-level advantage normalization. The results show that token-level advantage normalization consistently degrades performance across all four benchmarks.

MethodAMBER Cap. Score ↑MS COCO Cap. Score ↑MMHal-Bench Score ↑FaithScore ↑
Token Norm79.3877.203.3291.98
SCAPO (Ours)82.7679.633.5292.82

5. Inference efficiency and token overhead

We randomly sampled 1,000 discriminative questions to evaluate inference efficiency on a single NVIDIA A100 GPU. Direct answering takes an average of 0.792 s (40.9 generated tokens) per question, while the two-stage description-augmented inference takes 1.943 s (142.8 generated tokens) per question. Although the two-stage procedure introduces additional computation, we consider this increase in latency an acceptable trade-off given the improvements in discriminative performance (improving AMBER discriminative accuracy from 83.9% to 86.9%).

6. Marginal gains of SCAPO in Table 4

SCAPO is optimized for faithful and informative generation, rather than discriminative tasks. Its benefits to discriminative tasks are therefore indirect, which explains the relatively modest accuracy gain. Nevertheless, SCAPO improves all four discriminative metrics (accuracy, precision, recall, and F1), indicating that its improvements in generation quality can translate into consistent gains in discriminative performance.

7. Experiments of <think>/<answer> format

Thank you for this suggestion. We evaluated the SCAPO-trained model using a standard <think>...</think><answer>...</answer> format. The results below show that simply adjusting the output format cannot achieve comparable performance, with accuracy and F1 decreasing by 2.3 and 2.7 points, respectively.

Inference formatAcc. ↑Pre. ↑Rec. ↑F1 ↑
Think → Answer85.193.083.988.2
Description-Augmented87.487.694.490.9

8. Training and inference prompts

The training and inference prompts are already included in Appendix D. For convenience, we reproduce the key prompts below.

Training / generative-evaluation description prompt:

Describe this image.

Description prompt for discriminative inference:

Describe this image objectively and in detail. Focus only on visible content.

Description-augmented discriminative answering prompt:

The same model previously described this image as follows:
{generated_description}
Now answer the question based on the image and the description above.

Dear Reviewer Hk1E,

Thank you again for your thoughtful review. We have revised both the main manuscript and the appendix to address your concerns, with all revisions highlighted in the updated PDF for your convenience. If any point remains unclear or if there are additional questions we could clarify, we would be happy to provide further clarification.

We greatly appreciate your time and consideration.

Best regards,

The Authors

Reviewer_oTniOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to adopt reinforcement learning to reduce hallucinations in VLMs while avoid making them say less. It introduces DOPA, a re-annotation dataset of 5,000 COCO images with explicit presence/absence labels for 160 object categories. It also proposes SCAPO, which assigns advantages at the subsentence level to provide process-level training signals. Experiments with Qwen2.5-VL-7B show improved scores on AMBER (76.5→82.8) and COCO (73.2→79.6). The ablations validate the effectiveness of both denser supervision and localized credit assignment. Open-vocabulary evaluations indicate that SCAPO preserves or increases factual richness, and generated descriptions improve downstream discrimination.

Critical strengths
  1. The method is well motivated and achieves strong performance. The paper first identifies that directly training VLMs with RL to reduce hallucinations can substantially diminish the informativeness of their responses. It then introduces SCAPO, which addresses this problem through subsentence-level credit assignment. Comparisons with response-level methods trained on the same DOPA annotations further demonstrate that denser supervision alone is insufficient.

  2. The proposed DOPA is another contribution. Much prior research on hallucination has relied on COCO, whose limited object-category coverage and annotation errors have adversely affected these studies. DOPA partially addresses these limitations by expanding the vocabulary and refining the annotations.

Critical weaknesses
  1. I hope the authors could discuss two works with very similar ideas. (1) Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026) proposed Obliviate, which also annotates model rollouts at the clause level and applies gradient descent to correct clauses and gradient ascent to incorrect ones. This is effectively very similar to advantage-based adjustments in RL and shares a similar approach with SCAPO. (2) Improving Vision-language Models with Perception-centric Process Reward Models (CVPR 2026) goes a step further by introducing a PRM to reduce hallucinations. Compared with filtering based on a fixed vocabulary, a PRM can detect perceptual errors made by VLMs across a broader range of domains and can be naturally incorporated into RL training.
  2. It would be better if the evaluation include other perception-centric VQA benchmarks. The evaluation results on these benchmarks can further reflect the SCAPO's improvement on model's perception capabilities.
Questions for authors
  1. Can author provide evaluation results on general perception-centric VQA benchmarks?
3 comments

1. Relation to existing works

Thank you for pointing out these relevant works. We will discuss both works and clarify their differences from SCAPO in the revised manuscript.

Obliviate (AAAI 2026) adopts an offline objective: it suppresses pre-identified hallucinated fragments while retaining supervised autoregressive learning on reference responses. In contrast, SCAPO focuses on on-policy RL, where each update evaluates the current policy's rollouts and directly assigns separate positive or negative advantages to their subsentences. By evaluating newly generated rollouts at each training step, SCAPO can continuously target the current policy's errors, rather than relying on a fixed set of pre-identified hallucinated fragments.

Perceval (CVPR 2026) uses a learned PRM to identify perceptual errors in broader categories. However, its RL update focuses on penalizing detected erroneous tokens. Correct tokens still inherit the response-level advantage and are punished when the advantage of the whole response is negative. In contrast, SCAPO computes each subsentence's reward independently, explicitly reinforcing supported new object claims and suppressing hallucinated claims. This encourages the model to generate more correct information rather than simply reducing its output, thereby mitigating the “saying less” shortcut.

2. Additional perception-centric VQA benchmarks

We have evaluated the model's general multimodal capabilities across five benchmarks, including perception-oriented evaluations, with the results already included in Appendix E.1 (Table 8). For convenience, we reproduce them below. These results suggest that SCAPO preserves the model's perception capabilities and can even yield modest improvements.

MethodMME Percep. ↑MME Cognit. ↑MMBench Circular ↑MMBench Vanilla ↑MMMU Acc. ↑LLaVA-Bench Rel. Score ↑MM-Vet Total ↑
Qwen2.5-VL1717.8625.784.888.053.869.765.9
+ SCAPO1728.2635.484.488.253.870.466.0

Dear Reviewer oTni,

Thank you again for your thoughtful review. We have revised both the main manuscript and the appendix to address your concerns, with all revisions highlighted in the updated PDF for your convenience. If any point remains unclear or if there are additional questions we could clarify, we would be happy to provide further clarification.

We greatly appreciate your time and consideration.

Best regards,

The Authors

Thank you for the detailed response. Most of my questions have been addressed, and I will maintain my positive score.

Reviewer_SoWpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes two complementary contributions. DOPA provides presence/absence annotations for 160 canonical object categories and 894 aliases across 5,000 COCO training images. SCAPO segments responses into clauses, classifies each clause as containing a newly supported object, a repeated object, a hallucinated object, or no verifiable object, and assigns the corresponding reward to the clause’s token span.

On Qwen2.5-VL-7B, SCAPO reports an AMBER Caption Score of 82.8, an MS COCO score of 79.6, and AMBER discriminative accuracy of 87.4; the reported PhD-all score is 72.6. With DOPA annotations, the AMBER hallucination rate decreases from 11.8% to 9.4%, while coverage increases from 50.5% to 69.6%. In the detailed-description setting on FaithScore, SCAPO obtains a score of 96.3 and an average of 50.22 facts, compared with 93.2 and 41.63 for the Qwen2.5-VL baseline.

Critical strengths
  1. The problem diagnosis is concrete, and the proposed components correspond to it. The paper attributes coverage issues to sparse labels and local hallucinations to response-level rewards, then proposes DOPA and clause-level credit assignment, respectively.
  2. The evaluation includes several baselines and ablations. It compares inference-time methods, offline preference optimization, and on-policy methods, and studies annotation sources and credit-assignment granularity.
  3. The evaluation goes beyond hallucination rate. Caption Score incorporates faithfulness and coverage, while FaithScore and fact counts assess descriptive richness, partially addressing the “say less to hallucinate less” issue.
Critical weaknesses
  1. The closed vocabulary may alter the reward objective. The 160 object categories and aliases cannot cover all verifiable objects, attributes, and relations. The treatment and possible suppression of out-of-vocabulary facts remain unclear.
  2. The evaluation metrics are not fully independent. Caption Score depends on object annotations, and FaithScore uses an LLM judge. They are complementary, but do not constitute fully independent verification of factuality.
  3. The contribution of SCAPO is difficult to isolate. All on-policy methods use DOPA, while offline methods use different preference data. Existing comparisons do not fully disentangle annotation density, credit assignment, and training algorithm effects.
Questions for authors
  1. Please evaluate out-of-vocabulary objects, attributes, and relations, and test whether closed-set rewards systematically suppress correct descriptions outside the label set.
  2. Please report the accuracy of clause segmentation, entity normalization, and object matching, and test reward sensitivity to boundary and synonym perturbations.
  3. Please use factorial controls to separate the contributions of DOPA, SCAPO, and the training algorithm—for example, varying credit assignment under a fixed algorithm and varying algorithms under fixed annotations.
5 comments

1. Evaluation on out-of-vocabulary objects, attributes, and relations

Our 160-object vocabulary was constructed on MS COCO, whereas the experiments include benchmarks with different concept coverage. In particular, 223 of AMBER's 340 object categories are not included in our 160-label vocabulary. Furthermore, MMHal-Bench and FaithScore employ LLM-judge-based, open-vocabulary evaluations, rather than restricting assessment to our 160 labels. Improvements on these benchmarks therefore provide evidence of generalization on out-of-vocabulary objects, attributes, and relations.

2. Accuracy of clause segmentation, entity normalization, and object matching, and reward sensitivity to synonym perturbations.

Since gold annotations are unavailable for these intermediate text-processing steps, we randomly sampled 400 Qwen2.5-VL rollouts on MS COCO and used GPT-6.1-Sol as an external judge to assess their reliability. The resulting judge-assessed agreement on these rollouts was 93.75% for clause segmentation and 92.82% for entity normalization and object matching, suggesting that these intermediate text-processing steps are sufficiently reliable to support SCAPO training.

To assess the sensitivity of SCAPO's reward assignment to imperfect synonym matching, we conducted an alias-coverage ablation by randomly removing half of the aliases while keeping all other training settings unchanged. This prevents the removed synonyms from being matched during training. Despite this perturbation, the results below show that SCAPO with half aliases still achieves improvements over the baseline, indicating that our method remains effective and is robust to imperfect synonym matching. Furthermore, the full alias map achieves better performance on three of the four evaluated metrics, suggesting that expanding synonym coverage could further improve performance.

MethodAMBER Cap. Score ↑MS COCO Cap. Score ↑MMHal-Bench Score ↑FaithScore ↑
Qwen2.5-VL (Baseline)76.5173.183.2692.05
Half Aliases (Ours)82.2480.743.4892.65
Full Aliases (Ours)82.7679.633.5292.82

3. Factorial controls for DOPA and SCAPO

We have already included controlled experiments in our manuscript to examine the individual contributions of DOPA and SCAPO, with detailed analyses and discussions provided in Sections 5.3 and 5.4, respectively.

First, fixing SCAPO and varying the annotations: Table 2 in the manuscript compares SCAPO trained with MS COCO annotations against SCAPO trained with DOPA annotations. With the optimization method held fixed, DOPA yields better overall cross-dataset performance, particularly in terms of AMBER coverage and MMHal-Bench scores, demonstrating the benefits of denser annotations.

Second, fixing DOPA and varying the optimization/credit assignment: All on-policy methods in Table 1 of the manuscript use the same DOPA annotations. We compare GRPO, DAPO, Dr. GRPO, and SCAPO under matched training settings where applicable. Figure 4 in the manuscript further contrasts response-level, object-level, and subsentence-level credit assignment, while Figure 5 in the manuscript shows that SCAPO maintains competitive FaithScore while increasing the number of generated facts. These experiments help disentangle the contributions of annotation coverage and credit assignment.

4. The evaluation metrics are not fully independent. Caption Score depends on object annotations, and FaithScore uses an LLM judge. They are complementary, but do not constitute fully independent verification of factuality.

We agree that Caption Score and FaithScore are complementary rather than fully independent. Caption Score is designed to jointly measure hallucination and informativeness on datasets with entity annotations, while FaithScore is a well-established LLM-judge-based benchmark that also allows us to evaluate both factuality and informativeness. We would appreciate it if the reviewer could further clarify the specific concern regarding their partial dependence and why this is considered a weakness of our evaluation, so that we can better understand and address it.

Dear Reviewer SoWp,

Thank you again for your thoughtful review. We have revised both the main manuscript and the appendix to address your concerns, with all revisions highlighted in the updated PDF for your convenience. If any point remains unclear or if there are additional questions we could clarify, we would be happy to provide further clarification.

We greatly appreciate your time and consideration.

Best regards,

The Authors

Thank you for the response. Although Sections 5.3 and 5.4 present results for SCAPO and DOPA under various settings and experiments, respectively, the concern is how each of them contributes in isolation. Specifically, without the DAPO dataset, how would the results change if SCAPO were trained with only a similar reward-computation scheme? As for SCAPO, the contribution is trivial: it computes the reward against object detection result, in addition to the traditional correctness signal. The assumption that greater informativeness is vital for hallucination mitigation, which underlies the reward design, is also not strongly supported by either observations or experiments. Therefore, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.