- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Variational Finding-Anchored Direct Preference Optimization (VF-DPO) as an improvement over DPO with a focus on improving clinical consistency of radiology reports by introducing finding-specific question-answer pairs as latent anchors in the training process. The method is evaluated on MIMIC-CXR and CT-RATE across eight model backbones and compared against five (MIMIC-CXR) or four (CT-RATE) other preference optimization methods. The additional computations required for the finding-state alignment and conditional report alignment increase the training time by 2.6x compared to standard DPO.
Critical strengths
- The method addresses an important limitation of DPO for radiology report generation (finding awareness)
- Compared against several preference optimization methods
Critical weaknesses
- Small improvements at high costs: The choice of backbone appears to have a substantially larger impact than the alignment method. On REVTAF, VF-DPO improves RadGraph by only 0.0078 over DPO and 0.001 over the original model, while performing worse than the original model on RaTEScore (Table 1). Overall there are slight performance improvements, but they come with substantially higher costs (2.6x runtime and 2.8x memory compared to DPO). A compute-matched comparison would be insightful.
- Unknown robustness of improvements and theoretical justification: The reported significance tests appear to be based on individual checkpoints and not repeated training runs. Given the small improvement, run-to-run variance could exceed the reported differences. There are no information on checkpoint selection and hyperparameter tuning for baselines (nor a reproducibility section or provided code). Furthermore, replacing the log probabilities in the DPO reward margin with individual ELBOs does not guarantee an accurate approximation of that margin, since the approximation gaps may differ. This relationship is not sufficiently justified.
Questions for authors
- How do the baselines perform with a matched computational budget, e.g. when trained for 2.6x longer?
- Are the results based on single training runs? Can the authors report variance across seeds and clarify checkpoint selection and baseline hyperparameter tuning?
- Equation 4 replaces the log probabilities in the DPO reward margin with their respective ELBOs. Since the four approximation gaps may differ, the resulting margin is not neccessarily a lower bound on the original margin and could even have the opposite sign. Have the authors analyzed this approximation error or established conditions, under which preference ordering is preserved?