- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Fact-Commitment Space, a structured reward framework for improving factual correctness in chest radiography. It represents clinical statements using entities, assertion states, and attributes, and assigns different penalties to omissions, unsupported findings, attribute mismatches, and contradictions. The authors develop a Clinical Fact Parser (CFP) and integrate the proposed reward with GRPO to train VeraRad-2B. Experiments show improvements in clinical VQA and several report generation metrics, together with analyses of parsing reliability, reward properties, and visual grounding.
Critical strengths
-
Clinical factual errors are not equally harmful, and explicitly distinguishing different types of errors is a meaningful direction for medical VLM training.
-
Representing clinical claims as explicit commitments allows more interpretable and controllable reward computation. The deterministic scoring procedure and theoretical analysis provide useful justification for the design.
-
The authors validate the parser using human-reviewed data and examine reward consistency through controlled interventions. The VQA results are promising, and the transfer to report generation provides additional evidence that the training affects downstream behavior.
Critical weaknesses
-
The main experiments demonstrate that VeraRad improves over its base model after GRPO training, but there is no matched-budget comparison with alternative reward functions. Since existing clinical factuality metrics could also be used for RL training, it remains unclear whether the improvements come specifically from Fact-Commitment Space or from RL post-training more generally.
-
VeraRad achieves a clear improvement in strict accuracy, but the gain is considerably smaller under answer-extraction accuracy. This suggests that improved output formatting may account for part of the reported improvement. A more direct separation of formatting and clinical correctness would make the results more convincing.
-
The proposed model performs well on the primary report quality metric, but the results are mixed across other metrics. Its reports are also substantially shorter, which may improve some similarity scores while omitting clinically relevant findings. More direct evaluation of unsupported and missing clinical information would strengthen the conclusions.
Questions for authors
-
Can the authors compare the proposed reward against simpler correctness-based rewards and existing clinical factuality rewards under the same GRPO training budget?
-
How much of the VQA improvement comes from better format compliance versus improved clinical correctness? Can these two effects be evaluated separately?
-
Can the authors provide a more direct factuality evaluation of the generated reports, distinguishing hallucinations from clinically important omissions?
-
How sensitive is the training outcome to the relative penalties for omissions, unsupported findings, and attribute mismatches?
4 comments
Reply to Reviewer nrFW (2/2): W3 / Q3 — direct factuality evaluation of reports; Q4 — sensitivity to the penalties
W3 / Q3 — direct factuality evaluation of reports. We now evaluate unsupported and missing findings directly with CheXbert labels (positive classes) and RadGraph precision and recall, and we add GREEN. No model was trained on report generation, so these are transferred effects.
| MIMIC-CXR findings, thinking On | False-finding rate | Missed-finding rate | GREEN |
|---|---|---|---|
| Qwen3.5-2B (base) | 48.9% | 38.2% | 0.301 |
| GRPO, correctness reward (B1) | 47.3% | 37.8% | 0.293 |
| GRPO, RadGraph-F1 reward (B2) | 49.3% | 34.7% | 0.294 |
| VeraRad-2B (FCS reward) | 43.7% | 45.9% | 0.280 |
| FCS reward, second seed | 44.4% | 40.8% | 0.288 |
| FCS, entity+assertion only | 47.2% | 40.7% | 0.290 |
| FCS, κ = 2 | 45.4% | 42.6% | 0.282 |
- The base over-reports: 679 characters and 2.76 positive findings per report, against 407 and 2.28 in the reference. VeraRad produces 367 characters and 2.20 findings, lowers the false-finding rate from 48.90% to 43.69% and raises RadGraph precision from 0.221 to 0.271.
- The cost is more omissions: the missed-finding rate rises from 38.17% to 45.88% and RadGraph recall falls from 0.338 to 0.259.
- B1, B2 and the entity+assertion-only reward do not lower the false-finding rate; the second FCS seed reproduces the shift.
- GREEN, which counts omissions and false findings alike, scores VeraRad below its base and below B2.
We therefore report the result as fewer unsupported findings with more omissions, an effect of the full FCS reward, and not as higher scores on every report metric. As a check on the metrics, a fixed three-sentence normal report scores 1/RadCliQ 1.594 on IU X-Ray, above every model, but 0.655 on MIMIC-CXR (VeraRad On: 0.792), so we base this analysis on MIMIC-CXR. Section 4.3 and Appendix D.3 of the revised PDF give all models, both thinking modes and paired intervals.
Q4 — sensitivity to the relative penalties. Beyond the scorer-level ablations in Figure 4, we trained two policies that each change one cost factor of VeraRad (rows 7 and 8 of the table above, and of the VQA table in our first comment):
- Entity+assertion only (attribute terms removed): extracted VQA accuracy falls by 1.60 points On (−3.06 to −0.13) and 2.50 Off (−3.40 to −1.60), and the reduction in false findings largely disappears. The attribute costs matter for the learned policy.
- κ = 2 (insertion cost doubled, τ unchanged): extracted accuracy changes by −0.60 (−2.02 to 0.85) On and −1.23 (−1.88 to −0.58) Off and stays above B2 in both modes (+2.00, 0.48 to 3.53 On; +0.83, 0.11 to 1.60 Off). Report metrics with thinking On are within the intervals of κ = 1 on all three datasets.
The learned policy therefore depends on the attribute costs and is robust to κ between 1 and 2. The costs can also be read as confidence thresholds, which Section 3.3 now states. Asserting a finding has lower expected cost than omitting it when its confidence exceeds κ/(1 + κ), which is 0.5 for κ = 1. Guessing an attribute beats omitting it when its confidence exceeds 1 − 1/[(1 + κ)(τ + (1 − τ)d)], where d is the distance of a wrong guess: 0.375 for an adjacent severity grade and 0.5 for nominal attributes at κ = 1, τ = 0.6. Since κ = 2 did not lower the false-finding rate further, we present these thresholds as properties of the scorer and not as a guarantee about the trained policy.
Thank you for the additional experiments and clarifications. I appreciate the effort the authors have put into addressing my concerns. My remaining reservation is mainly about the broader scientific contribution. While I find the proposed reward design useful and the results encouraging, I feel the work is more of a practical improvement than a deeper conceptual advance in clinical factuality. I will therefore maintain my original rating. Thanks again for the thoughtful discussion!
Dear Reviewer nrFW,
Thank you for the careful discussion and for recognizing the additional experiments. We respect your assessment, and we would like to briefly state what we see as the conceptual contribution, since it may be less visible behind the engineering.
First, the work treats clinical factuality as a set of typed commitments (what is asserted, denied, or left unstated, and with which attributes) rather than as a similarity score. This changes what a reward can express. The controlled comparison suggests why this matters: under the same budget, rewards based on answer correctness or on RadGraph overlap changed the answer format but not the clinical content, while the commitment-based reward was the only one that improved content.
Second, the asymmetric costs are not just tuning knobs. They define explicit decision rules for the policy: a finding should be asserted only when its confidence exceeds κ/(1 + κ), and an attribute should be specified only above a corresponding threshold (Section 3.3). The learned behavior matches this, with fewer unsupported findings at the price of more omissions, and that trade-off is now something one can set and reason about rather than an incidental side effect of the reward.
We will make this framing more explicit in the introduction and discussion of the final version. Thank you again for your time and constructive feedback.
Reply to Reviewer nrFW (1/2): W1 / Q1 — alternative rewards under the same budget; W2 / Q2 — format versus correctness
Thank you for the clear and constructive review. Each of your four questions is addressed by a new controlled experiment. We answer in two comments, and each contains the numbers it relies on.
W1 / Q1 — alternative rewards under the same budget. We trained two GRPO baselines, a format-tolerant correctness reward (B1) and a RadGraph-F1 reward (B2), together with SFT on the gold answers, a second seed of the FCS reward and two ablations. Every trained model uses the same base (Qwen3.5-2B), the same 3,200 training items in the same order, and the same LoRA, sampling and thinking budgets; the GRPO models share the 800 updates and change only the reward or the seed. Intervals are 95% patient-cluster bootstrap intervals (10,000 resamples) on paired items.
W2 / Q2 — format compliance versus clinical correctness. They can be separated, and we now do so with a per-item transition analysis from the base to VeraRad.
The strict gain thus combines a format component, which B1 and B2 also obtain, and a content component that only the FCS reward obtains: +2.47 points On (0.98 to 3.99) and +1.57 Off (0.44 to 2.66) on extracted accuracy, and +3.00 On (1.44 to 4.55) for the second seed. The paper now reports strict and extracted accuracy side by side and separates the two (Section 4.2 and Appendix D.2).