- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the underrepresentation of focal lesions in full-report generation from 3D CT volumes. The authors propose FLARE, a post-training framework that separates lesion-evidence extraction from selective evidence routing. A frozen MedGemma 1.5 model, fine-tuned for full-report generation, provides global predictions. A crop expert extracts local evidence through the next-token logit difference between a lesion ROI and a null visual input, while a learned router determines whether this evidence should modify the global prediction at each decoding step. On the 100-volume ReXGroundingCT held-out test set, FLARE with ground-truth ROIs improves Micro-F1 from 37.94% to 46.64% and Macro-F1 from 20.20% to 31.04% over the global SFT baseline, while maintaining broadly comparable textual similarity. Experiments using automatically localized ROIs on an additional cohort of 3,324 CT scans show smaller improvements in several clinically oriented metrics. These results support the potential of the framework for incorporating focal findings into comprehensive reports, although the independent contribution of selective routing and its advantage over alternative local-augmentation methods in the automatic setting remain insufficiently established.
Critical strengths
-
Consistent improvements in clinically oriented metrics under controlled backbone comparisons. The paper evaluates FLARE against not only off-the-shelf models but also a global SFT baseline, zero-shot crop augmentation, and a trained global-plus-crop baseline using the same model backbone. With ground-truth ROIs, FLARE achieves the best point estimates in Micro-F1, Macro-F1, RadGraph F1, RaTEScore, and RadCliQ, whereas directly adding crops produces mixed results. These comparisons provide meaningful evidence that the complete framework can make better use of local lesion information during full-report generation, which is a central consideration supporting acceptance.
-
Additional analyses test whether the gains depend on lesion-relevant visual content. In Appendix Table 7, replacing the ground-truth ROI with a region from the same CT that does not contain the target lesion reduces FLARE's Micro-F1 from 46.64% to 37.56% and Macro-F1 from 31.04% to 21.66%. This supports a dependence on relevant visual evidence rather than merely the presence of an additional crop. The automatic localization experiments also demonstrate operation without manual ROIs at inference, while patient-level bootstrap analysis quantifies uncertainty on the small primary test set. Together, these analyses strengthen confidence in the empirical findings.
Critical weaknesses
-
The independent contribution of selective routing is not sufficiently established, and comparisons with related decoding-time control methods are missing. DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts, ACL 2021, already adjusts base-model predictions using the difference between expert and anti-expert logits. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, CVPR 2024, similarly uses differences between predictions under different visual inputs to modify generation. FLARE introduces a lesion-specific crop expert and a learned router, but the relationship to these established decoding strategies warrants a clearer account of its incremental contribution. In particular, the paper does not compare selective routing with adding the same ROI-null evidence at every decoding step using a fixed weight, while holding the crop expert unchanged. The positive-only variant in Appendix A.5.2 simultaneously removes the ROI-null difference and unrelated-ROI negative supervision, so it cannot isolate the router's contribution. The results therefore establish the effectiveness of the complete system more clearly than the necessity of learning when to incorporate evidence, despite this being the central claim of the paper.
-
The incremental advantage and clinical relevance of the fully automatic setting remain insufficiently validated. The strongest improvements are obtained with ground-truth ROIs on the 100-volume test set. On the larger cohort of 3,324 scans using predicted ROIs, Micro-F1 increases from 35.83% to 38.34%, Macro-F1 from 27.16% to 28.50%, and the LLM evaluation score from 38.11 to 38.47, while RadCliQ slightly worsens. More importantly, this automatic experiment compares FLARE only with the global-only baseline, without evaluating competing local-augmentation methods using the same predicted ROIs. It therefore remains unclear whether the gains in this setting are specific to selective evidence routing. The reported patient-level bootstrap analysis concerns the 100-volume primary test set and does not directly quantify uncertainty in these smaller gains on the larger cohort. Because the paper emphasizes full-report generation without manual lesion annotations at inference, stronger evidence of the method's incremental advantage under this condition would materially affect the assessment of its practical contribution.
Questions for authors
-
The independent contribution of selective routing. Have the authors compared router-based selective evidence incorporation with adding the same evidence at every decoding step using a fixed weight, while keeping the crop expert, ROI–null logit difference, and other settings unchanged? The positive-only comparison in Appendix A.5.2 removes both the ROI–null difference and unrelated-ROI negative supervision, so it does not isolate the contribution of selective routing. This comparison would help determine whether the gains specifically depend on learning when to use local evidence.
-
The distinction from existing decoding-time control methods. FLARE’s logit adjustment has a formal connection to DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts and Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. Could the authors clarify the key distinctions in lesion-evidence construction and routing, and identify which experimental results support an additional benefit from these distinctions? This would help establish the incremental contribution over existing decoding strategies.
-
The reliability of the gains with automatic localization. In the automatic localization experiment on 3,324 CT scans, have the authors evaluated a Global + Crop baseline using the same predicted ROIs as FLARE? Are patient-level paired confidence intervals available for the differences between methods on this cohort, particularly for the relatively small improvements in Macro-F1 and the LLM evaluation score? These results would help determine whether the improvements are robust and whether they arise from introducing local information or from FLARE’s specific evidence-incorporation mechanism.