acceptodds
Withdrawn from ICLR 2027

RHDP: A Gradient-Variation-Aware Reward Hacking Detection and Penalization Mechanism for Chest X-ray Report Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chest X-ray report generation aims to produce a detailed and clinically accurate description for a given image. Although reinforcement learning (RL) fine-tuning can optimize sequence-level preferences, an imperfect proxy reward may be over-optimized, leading to template collapse and deterioration of the clinically meaningful gold objective. We introduce RHDP, a reward-hacking detection and penalization framework built around the long-window variance of input gradients (LVoG). LVoG is the temporal variance of the absolute input gradient of the joint CE–RL loss and is used as a sample-wise reliability score: abnormally low values indicate increased proxy–gold misalignment risk. A decomposition analysis shows that LVoG combines CE–RL transition dynamics, reward-side advantage dynamics, and pure RL policy-side image-conditioned sensitivity; the policy-side signal remains predictive after removing instantaneous advantage scaling. RHDP uses lagged LVoG to modulate the reward and the forward KL divergence, suppressing unreliable updates while preserving generation quality. Experiments on IU-Xray and MIMIC-CXR demonstrate improved clinical accuracy, competitive semantic quality, and consistent mitigation of reward hacking. The code is available on GitHub.

Primary area:
reinforcement learning
Keywords:
Chest X-ray report generation, Reinforcement learning, Reward hacking, Input gradients

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cF48OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RHDP to detect and mitigate reward hacking in chest X-ray report generation. Its central statistic, LVoG, measures the temporal variance of spatially averaged absolute input gradients of the joint cross-entropy and reinforcement-learning loss. Reliability scores derived from LVoG modulate rewards and forward KL regularization. Experiments on IU-Xray and MIMIC-CXR subsets examine report quality, gradient dynamics, and individual component contributions.

Critical strengths
  1. The paper studies an important problem, since optimizing automated rewards does not necessarily improve clinically grounded report generation. Its decomposition of the joint gradient provides a useful basis for investigating which training signals are associated with reward exploitation.
Critical weaknesses
  1. The empirical justification for the composite detector is insufficient. Appendix A.2 claims that joint LVoG has the strongest association with proxy–gold alignment, but Figure 9 reports higher correlations for advantage variance in all three displayed windows. The corresponding values are 0.305, 0.286, and 0.288 for advantage variance, compared with 0.206, 0.226, and 0.221 for LVoG. Although this does not establish which statistic produces a better intervention, the paper provides no advantage-variance-based mitigation baseline to demonstrate the additional benefit of computing input gradients.

  2. The controlled ablations do not consistently support the claim in Section 4.4 that the two components jointly improve clinical accuracy. On MIMIC-CXR-10k, the complete method reduces CheXbert F1 from 0.377 to 0.369 and RadGraph F1 from 0.264 to 0.251 relative to Base, which already uses the improved RadCliQ reward. On IU-Xray, the complete method improves over Base but remains below KL-only on both clinical metrics. The trade-off acknowledged in Section 4.3 may explain some of these differences, but the additional value of combining the components remains insufficiently established. Furthermore, the detector evaluation uses the original RadCliQ as the gold reward despite identifying exploitable behavior under that reward, limiting its interpretation as independent evidence of reduced clinically harmful reward hacking.

Questions for authors
  1. How should the claimed superiority of joint LVoG be reconciled with Figure 9? Does LVoG-based modulation outperform advantage-variance-based modulation when average reward scaling and KL settings are matched?

  2. Given the trade-off acknowledged in Section 4.3, what evidence establishes that combining LVoG and forward KL provides additional value over either component alone? Does combining LVoG and forward KL reduce clinical errors in generated reports compared with using either component alone?

  3. The comparison with recent methods appears limited: VSANet (Neural Networks, 2026) is the only 2026 baseline evaluated on IU-Xray, where RHDP trails it on two of the three reported metrics (BLEU-4 and ROUGE-L), and no 2026 baselines are included on MIMIC-CXR-10k. Could the authors clarify the rationale for baseline selection and provide comparisons with recent strong methods under a consistent evaluation protocol, including clinical metrics, to substantiate the claimed clinical-accuracy–fluency trade-off?

Reviewer_cAmAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RHDP to detect and mitigate reward hacking in (automated) chest X-ray report generation. It uses the temporal variance of joint CE–RL input gradients, here termed LVoG, to modulate rewards and forward-KL regularization. Experiments cover two datasets, IU-Xray and MIMIC-CXR, with additional analyses and a preliminary VLM/PPO experiment.

Critical strengths

The paper aims to address an important problem: optimizing automatic report-generation rewards can produce undesirable clinical and linguistic behavior via reward hacking. The analysis considers multiple rewards, component ablations, and training dynamics. An architecture-independent training intervention could be useful as it could generalize to many settings.

Critical weaknesses
  1. The empirical results are mixed (at best), and the benefit of the method is unclear. E.g., Table 1 shows substantial trade-offs across metrics. More importantly, Table 2 does not consistently demonstrate improvement from the proposed combination. On MIMIC, the full method reduces CheXbert F1 from 0.377 to 0.369 and RadGraph F1 from 0.264 to 0.251 relative to the baseline already using the improved reward. On IU-Xray, KL-only outperforms the full method on both clinical metrics. These results need a more balanced interpretation and uncertainty estimates.

  2. The figures and tables are difficult to interpret throughout. Figures 3 and 5 are particularly hard to parse: small text and low effective resolution make legends and plotted details difficult to read. This impedes assessment of evidence central to the paper. The extensive boldface in Table 2 is also confusing. The highlighting convention should be explained and applied consistently, and ideally also sparingly (e.g. half a table being bold loses its meaning).

  3. The evidence for the added value of LVoG is not convincing. The text claims that joint LVoG has the strongest association with proxy–gold alignment, but Figure 9 reports higher correlations for reward-only advantage variance, VoA, at all three displayed windows. This discrepancy should be reconciled. A matched VoA-only modulation baseline would help establish whether the gradient computation contributes useful information.

  4. The optimization procedure needs clearer exposition. Equations (2)–(4) move from an expected-reward objective and policy-gradient estimator to a joint CE–RL loss without explicitly introducing the differentiable surrogate used for backpropagation. Much later, I saw that Appendix A.2 indeed provides that bridge, but introduces token-length normalization absent from Equation (3). The exact implemented objective, sampling procedure, and stop-gradient conventions should be stated clearly, especially because this loss also defines LVoG.

  5. The positioning relative to recent medical VLM research is incomplete. For example, RadVLM-GRPO already studies clinically aligned RL and reward-dependent report-generation failures. Generally, the paper has very few recent 2026 references despite it being a very active field of research. These findings should be discussed when establishing the novelty and relevance of RHDP. More clear distinction from existing input-gradient variance methods and selective forward-KL regularization would also help clarify the contribution.

Questions for authors
  1. How do the authors reconcile the claimed superiority or added value of LVoG with Figure 9’s stronger VoA correlations?

  2. What evidence establishes the proposed method’s benefit over the improved-reward baseline and individual components in Table 2?

  3. Which RL surrogate is actually implemented, and how are length normalization and lagged LVoG handled?

I believe that clarifying this will improve the paper.

Reviewer_TxmXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the reward hacking in medical report generation. It uses the temporal variance of input gradients from a joint CE–RL loss to weight rewards and adjust forward KL regularization. It also modifies the RadCliQ reward. Experiments mainly examine report quality on chest x-ray generation, reward alignment, and output collapse on IU-Xray and MIMIC-CXR subsets.

Critical strengths

The training signal for medical report generation during RL is studied beyond the fixed correlation provided by the automatic evaluator. The gradient decomposition and reversed-weighting experiments provide useful evidence about its behavior. This offers valuable insights into reliable RL in medical AI models.

Critical weaknesses
  1. The contribution of the paper is limited due to its narrow research scope. Reward hacking in medical AI model training is a valuable research question. However, the paper only focuses on 2D chest X-ray generation, which is basically solved by frontier models. In addition, the experiments in this paper do not provide further evidence for more complicated medical report generation tasks (e.g., 3D CT volumes and MRI scans) or further agentic generation. It is hard to judge whether the proposed method can be generalized or extended to more complex medical generation tasks. Therefore, the results in the paper contribute little to the current medical AI community.
  2. The models compared in this paper are weak. They are mainly conventional report generation models, which makes the comparison unconvincing. The improvement on Qwen2-VL-2B is also limited.
  3. The main graphical illustrations are very coarse and too implicit to convey the main idea of the paper (Figs. 1 and 2).
  4. The metrics used in the paper are outdated, and the evaluation lacks annotations from real doctors. Since the paper stresses the unreliability of automatic scorers widely used in evaluation, it should provide a better evaluation method rather than use surface metrics like BLEU.
  5. The ablation study results for the proposed method are questionable. In Table 2, MIMIC CheXbert F1 is 0.369 for the full method, versus 0.377 for Base and 0.399 for LVoG alone. On IU-Xray, KL alone reaches 0.510 versus 0.507 for the full method. The broad claim of joint gains needs qualification.
Questions for authors
  1. Could you provide results on more complex generation task and comparison with stronger models as baselines (e.g. medical large vision-language models or general vision language models)?
  2. Why full method is weak in your ablation study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.