- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates a shortcut behavior in hallucination-oriented reinforcement learning for VLMs, where models reduce the number of generated claims instead of improving visual grounding. The authors identify sparse object annotations and coarse response-level credit assignment as two underlying causes. They propose DOPA and SCAPO (a subsentence-level reward assignment method for on-policy optimization) to address these issues and report improvements on multiple hallucination benchmarks.
Critical strengths
The distinction between improving visual faithfulness and merely reducing the number of generated claims is particularly valuable.
The joint consideration of annotation density and reward granularity is well motivated.
The comparisons between different annotation sources and credit-assignment strategies offer useful insights into the behavior of hallucination-oriented reinforcement learning.
Critical weaknesses
-
GRPO vs. DAPO: How are GRPO and DAPO trained, and how are their rewards configured? Since DAPO is an improvement over GRPO, why does it perform substantially worse in Table 1?
-
VTI results and baseline configurations: Why does VTI perform considerably worse than other methods on the AMBER captioning metrics in Table 1? Are the hyperparameters consistently tuned across baselines? The paper should provide the detailed configurations.
-
Reward dynamics: The paper lacks training curves for the individual reward components in Equation (1). How do these components evolve over training steps?
-
Advantage assignment: Why are local rewards directly broadcast as token-level advantages rather than normalized as in GRPO? What is the underlying motivation, and how is training stability maintained? Both theoretical and empirical evidence would be valuable.
-
Inference efficiency: The two-stage inference procedure for discriminative tasks introduces additional computation. The authors should quantify its latency and token overhead.
-
Marginal gains and alternative formats: Table 4 shows only a 0.5-point improvement (86.9% to 87.4%). Could a standard <think>...</think><answer>...</answer> format achieve similar results without an explicit description-generation stage? The authors should also release the training and inference prompts.
Questions for authors
Please refer to the concerns raised in the weaknesses. While I have several concerns regarding the methodology and experimental evaluation, I am willing to reconsider my assessment and potentially raise my score if these issues are convincingly addressed during the rebuttal.
6 comments
7. Experiments of <think>/<answer> format
Thank you for this suggestion. We evaluated the SCAPO-trained model using a standard <think>...</think><answer>...</answer> format. The results below show that simply adjusting the output format cannot achieve comparable performance, with accuracy and F1 decreasing by 2.3 and 2.7 points, respectively.
| Inference format | Acc. ↑ | Pre. ↑ | Rec. ↑ | F1 ↑ |
|---|---|---|---|---|
| Think → Answer | 85.1 | 93.0 | 83.9 | 88.2 |
| Description-Augmented | 87.4 | 87.6 | 94.4 | 90.9 |
8. Training and inference prompts
The training and inference prompts are already included in Appendix D. For convenience, we reproduce the key prompts below.
Training / generative-evaluation description prompt:
Describe this image.
Description prompt for discriminative inference:
Describe this image objectively and in detail. Focus only on visible content.
Description-augmented discriminative answering prompt:
The same model previously described this image as follows:
{generated_description}
Now answer the question based on the image and the description above.
Dear Reviewer Hk1E,
Thank you again for your thoughtful review. We have revised both the main manuscript and the appendix to address your concerns, with all revisions highlighted in the updated PDF for your convenience. If any point remains unclear or if there are additional questions we could clarify, we would be happy to provide further clarification.
We greatly appreciate your time and consideration.
Best regards,
The Authors
1. GRPO versus DAPO: settings, rewards, and performance differences
GRPO and DAPO are trained with the same Qwen2.5-VL-7B-Instruct backbone. Both GRPO and DAPO use the same base response-level reward, computed from the hallucination rate using DOPA annotations. We implemented DAPO by extending GRPO with additional components: Clip-Higher, Token-Level Policy Gradient Loss, and Overlong Reward Shaping, following their original formulations. Further training details are provided in Appendix C.1.
We hypothesize that DAPO's weaker performance may stem from the fact that its optimization strategies were primarily developed for long-form reasoning tasks and may not be equally effective for hallucination mitigation. Figure 5 in the manuscript helps explain the discrepancy: DAPO achieves a higher FaithScore than GRPO but generates fewer atomic facts in several settings, indicating a stronger conservative-output preference. This more conservative generation leads to a lower Cover Rate, thereby explaining DAPO's lower Caption Score compared with GRPO in Table 1 of the manuscript.
2. VTI results and baseline hyperparameters
For the VTI baseline, we followed the configurations provided in its official implementation (https://github.com/shengliu66/VTI). We will provide additional details on the configurations in the revised manuscript.
We observed that VTI substantially shortens the generated descriptions, resulting in low object coverage. This explains VTI's low AMBER Caption Score in Table 1 of the manuscript; it is also consistent with the “saying less” failure mode emphasized in our paper. In contrast, our SCAPO effectively mitigates this failure mode by preserving informativeness while reducing hallucinations.
3. Reward dynamics: How do the individual reward components in Equation (1) evolve over training steps?
Unlike GRPO, SCAPO does not aggregate rewards at the response level. Instead, Equation (1) defines a piecewise subsentence-level reward. Therefore, a single aggregate reward curve would not fully capture the effects of different subsentence-level rewards during training. The quantities most relevant to the reward dynamics are the frequencies of positive and hallucination-penalized entities in Equation (1), which are reflected by the Cover Rate and the Hal. Rate, respectively.
As shown in Figure 4 of the manuscript, SCAPO steadily increases Cover Rate while reducing Hal. Rate during training, consistent with the intended effects of our reward design in Equation (1).
4. Why advantages are not group-normalized? How is training stability maintained?
The central motivation is to preserve the semantic sign of each verified local reward: hallucinated subsentences should receive a negative advantage, while hallucination-free subsentences containing newly supported object claims should receive a positive advantage. Groupwise normalization, such as subtracting the group mean, can change these signs. Training stability is maintained through KL regularization, and the stable training trends in Figure 4 of the manuscript provide empirical evidence of SCAPO's stability.
We additionally conducted an ablation study using groupwise token-level advantage normalization. The results show that token-level advantage normalization consistently degrades performance across all four benchmarks.
5. Inference efficiency and token overhead
We randomly sampled 1,000 discriminative questions to evaluate inference efficiency on a single NVIDIA A100 GPU. Direct answering takes an average of 0.792 s (40.9 generated tokens) per question, while the two-stage description-augmented inference takes 1.943 s (142.8 generated tokens) per question. Although the two-stage procedure introduces additional computation, we consider this increase in latency an acceptable trade-off given the improvements in discriminative performance (improving AMBER discriminative accuracy from 83.9% to 86.9%).
6. Marginal gains of SCAPO in Table 4
SCAPO is optimized for faithful and informative generation, rather than discriminative tasks. Its benefits to discriminative tasks are therefore indirect, which explains the relatively modest accuracy gain. Nevertheless, SCAPO improves all four discriminative metrics (accuracy, precision, recall, and F1), indicating that its improvements in generation quality can translate into consistent gains in discriminative performance.