- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ACRA, which categorizes discrepancies between generated and reference reports into five types and dynamically adjusts their penalty weights during self-critical sequence training. Experiments on MIMIC-CXR and IU-Xray demonstrate improved report consistency.
Critical strengths
Accounting for distinct clinical errors and their evolving training dynamics is clinically meaningful.
Critical weaknesses
- Limited clinical performance and evaluation. The reported micro-F1 scores are below 0.50, substantially lower than those reported by many recent report-generation models. Please also report macro-F1 and additional clinical efficacy metrics, such as GREEN.
- Insufficient evidence for dynamic allocation. The fixed-weight baseline uses only uniform weights. Without comparison against tuned nonuniform weights, it remains unclear whether the gains arise from better weighting or continuous adaptation.
- Limited methodological contribution and validation. The method primarily applies an existing online weighting mechanism to report-generation training and is evaluated only on the older R2GenCMN backbone, without comparisons involving strong VLMs or recent optimization methods.
- Insufficient independent validation of clinical risk reduction. Training and evaluation rely heavily on the same structured extraction tools, without blinded physician assessment or adequate validation of extraction accuracy.
Questions for authors
- Do the gains persist against tuned fixed weights, frozen final ACRA weights, and simpler dynamic weighting schemes?
- Does ACRA provide consistent gains on stronger VLMs?
- Can physician assessment or independent evaluators confirm reductions in actual clinical errors?
3 comments
Response to Reviewer 3HvS (2/2)
Q2. Does the gain carry over to a stronger VLM?
Appendix A.9 (Table 13) compares fixed-risk SCST and ACRA on the same Qwen2.5-VL-7B backbone and MIMIC-CXR test set, using the same five reference-conditioned risk categories.
| Metric | Fixed-risk SCST | ACRA |
|---|---|---|
| Omission ↓ | 0.248 | 0.226 |
| Unsupported positive ↓ | 0.202 | 0.191 |
| Polarity ↓ | 0.150 | 0.149 |
| Laterality ↓ | 0.096 | 0.098 |
| Severity ↓ | 0.195 | 0.191 |
| Mean risk ↓ | 0.178 | 0.171 |
| BLEU-4 ↑ | 0.142 | 0.144 |
| ROUGE-L ↑ | 0.293 | 0.292 |
Mean risk drops by 0.007, with the largest reductions in omission and unsupported positive findings. The laterality rate increases by 0.002. BLEU-4 increases slightly and ROUGE-L decreases slightly. The experiment supports a lower average reference-conditioned risk on this second backbone, with trade-offs at the category and metric level. The controlled evidence covers R2GenCMN and Qwen2.5-VL-7B.
Q3. Independent assessment of clinical errors
We evaluated the R2GenCMN (NLL), fixed uniform risk weighting, and ACRA outputs with GREEN (Ostmeier et al., 2024) on the same MIMIC-CXR test set (Appendix A.10, Table 14). GREEN uses a separately trained language model to score agreement between generated and reference reports, and its scoring function is outside the training reward in Eq. (11).
| Configuration | Mean GREEN ↑ |
|---|---|
| R2GenCMN (NLL) | 0.300 |
| Fixed uniform risk weights | 0.307 |
| ACRA | 0.315 |
ACRA improves mean GREEN by 0.008 over fixed weighting and 0.015 over NLL. This independent automatic evaluation supports stronger clinical agreement with the reference reports, with physician adjudication, image-level verification, and extractor accuracy outside its scope.
Thank you again for your careful review. If the additional results and clarifications address your concerns, we would appreciate your reconsidering the overall rating in light of the revised manuscript.
I thank the authors for their efforts during the rebuttal period. Their responses have addressed some of my concerns, I will therefore keep my original score.
Response to Reviewer 3HvS (1/2)
We thank the reviewer for highlighting the clinical motivation and for requesting stronger controls and independent evaluation. We address the four critical weaknesses first and then answer the three questions in order.
W1. Clinical performance and micro- and macro-F1
We agree that the absolute CheXbert micro-F1 values in Table 4 are below 0.50. Appendix A.8 explains that cross-study comparisons depend on the test cases, reference text, label inventory, positive-label mapping, and aggregation. For example, DDGIP and REVA-PO use CheXpert labels. Table 4 uses CheXbert micro-F1.
We added a separate CheXbert macro-F1 comparison in Table 5. It uses the same MIMIC-CXR test set, label inventory, and positive-label mapping as the micro-F1 comparison, with equal weight for each finding's positive-class F1 (Appendix A.8).
Relative to fixed uniform weighting, ACRA improves micro-F1 by 0.010 and macro-F1 by 0.008 under the shared protocol. Q3 reports the complementary GREEN evaluation.
W2. Evidence for dynamic allocation
Appendix A.6 adds fixed non-uniform controls, including a run that holds ACRA's final weight vector fixed throughout training. Section 4.4 and Appendix A.4.1 report simpler online variants. Q1 compares these controls with full ACRA.
W3. Methodological contribution and recent comparisons
The contribution is the training formulation for five reference-conditioned clinical discrepancies. ACRA aggregates each risk over eligible report–finding pairs from greedy and reference reports, smooths these risks, and updates the allocation with a prior-regularized KL mirror-ascent step (Sections 3.2–3.3, Eqs. (4)–(7)). It then applies the same allocation to sampled and greedy reports in the self-critical score (Sections 3.4–3.5, Eqs. (11)–(12)). These components operate during training. The generator retains its standard inference procedure. Appendix A.6 characterizes the limiting weights when pairwise risk gaps are constant and prior regularization is positive.
Section 2 distinguishes this risk-driven allocation from MPO (Xiao et al., 2025), which trains a preference-conditioned generator and accepts a preference vector at inference. The revised comparison includes published results for MPO, DDGIP, R2GenGPT + SI + SA, and REVA-PO-7B under their source protocols (Section 4.2, Appendix A.8, Table 12). Q2 gives the controlled result on Qwen2.5-VL-7B.
W4. Independent validation of clinical risk reduction
We agree that the structured extractors used in training overlap with parts of the main clinical evaluation. Appendix A.10 adds GREEN, whose scoring function is separate from the training reward. Q3 reports the result and its scope.
Q1. Tuned fixed, frozen-final, and simpler online allocations
The fixed controls use the same five risk terms on MIMIC-CXR (Appendix A.6, Table 9). The initial-risk control uses the NLL checkpoint's greedy risk profile. The validation-tuned control selects the weights using validation performance. For the frozen-final control, we initialized a separate SCST run from the same NLL checkpoint and held the final ACRA weight vector fixed throughout training.
The tuned weights recover much of the improvement over uniform weights. ACRA lowers mean risk by a further 0.005 relative to validation tuning and 0.003 relative to freezing the final weights. All five category risks are lower than in both controls. The frozen-final comparison supports a modest additional benefit from updating the weights during training in this setting.
Both simpler variants retain online updates. Setting τ = 0 removes the prior regularization, and removing EMA uses current mini-batch risks (Table 6 and Appendix A.6). On MIMIC-CXR, mean risk is 0.185 without prior regularization, 0.183 without smoothing, and 0.177 with the full allocator (Table 6). The corresponding IU-Xray values are 0.170, 0.168, and 0.162 (Table 7, Appendix A.4.1). Removing the prior lowers omission and unsupported-positive risk on both datasets and increases polarity, laterality, and severity risks. The full allocator gives the lowest mean risk among these configurations.