- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue of corpus prior bias by proposing a training-free token reweighting strategy based on training-mean evidence reference comparisons, followed by iterative non-autoregressive report generation.
Critical strengths
- The paper’s training-free reweighting strategy for addressing corpus bias is quite interesting and demonstrates a degree of innovation.
- The ablation experiments are relatively comprehensive, including multiple ablations and sensitivity analyses.
Critical weaknesses
- Using the training-mean evidence as reference to solving corpus bias lacks theoretical and experimental support. The paper’s method emphasizes addressing corpus bias and uses the average image features as a reference, but it does not provide the relationship of using the training mean as a reference and corpus bias.
- The experiments are insufficient regarding whether the method effectively reduces the occurrence of bias. There is a lack of a detailed comparison to demonstrate that the method truly improves model performance by reducing bias occurrence.
- Practical value remains unvalidated. The paper notes that its method leads to an increase in model false positives; however, this may increase the risk of misinterpretation of diagnostic reports, and whether it ultimately improves report quality remains to be verified.
- Experimental evaluation is insufficient: the method was evaluated using only the MIMIC-CXR-JPG test set, which does not fully demonstrate its effectiveness.
- Why is it necessary to use non autoregressive methods in this paper, as previous non-autoregressive methods have not been better than autoregressive methods in Table 1. This paper does not explain the motivation clearly, nor does it have experimental support.
Questions for authors
- Does the subtraction of the training-mean reference prediction actually cancel corpus-common predictive support? The subtraction operation in the method implies this assumption, but the paper provides no direct experimental validation. Averaged image features lack interpretability on their own; you need to add the experiments to first clarify this point, then the subsequent logic will be reasonable.
- Does this method truly improve evaluation metrics by effectively reducing the occurrence of prior knowledge? Your method enhances performance by effectively reducing the occurrence of prior knowledge just based on the improvement in evaluation metrics; however, you need to demonstrate that TARING's performance gains stem from reduced reliance on corpus-common content. You need to add comparative experiments with baselines to demonstrate whether prior knowledge has indeed been reduced.
- Could the model’s false positives lead to a misleading impression? You need to add experiments to show that the improvement in metrics outweighs the increase in misleading impressions; otherwise, the method’s contribution to report generation is too low.
2 comments
Response to jvbe (2/2): Clinical Errors and Limitations
False positives (W3, Q3). TARING does add false findings, and the revised abstract states them next to the gain: per report, CheXbert-derived false negatives fall by 0.10 and false positives rise by 0.06 (Table 7); on seed 0, GREEN counts 0.19 more false findings, 0.14 fewer missing findings and 0.09 more significant errors, with a significantly lower GREEN score (Table 11; significant here and below: 95% paired bootstrap over patients). Over all three seeds (2,322 studies with every report parseable), GREEN counts 0.18 more false findings, 0.15 fewer missing findings and 0.03 more severity errors per report, and the change in GREEN score, [, 0.002], is not significant (Appendix D.10). The guidance strength sets this balance (table in our reply to dcME; Appendix D). On our seed-0 re-run of all 2,357 test studies, s = 1.50 gives a gain as large as s = 1.75 (+0.018 and +0.016; Table 5 gives +0.022 for s = 1.75 over three seeds) with 1.24 false findings per report against 1.26 (1.19 without guidance; the two strengths are not tested against each other) and no significant RadGraph or GREEN loss. s = 1.75 stays because Section 4 selected it on validation micro-F1; choosing s = 1.50 now would use test results, so the paper reports it beside s = 1.75 instead. Whether the omissions saved outweigh the false findings added needs radiologists' judgement, which we could not obtain in this period.
Evaluation beyond MIMIC-CXR (W4). One dataset is not enough, we agree, and the added test does not establish transfer: on all 590 IU X-ray test studies (the R2Gen test split) decoded without retuning, averaged over three reveal seeds, micro-F1 is 0.447 without the reference and 0.454 with TARING (+0.007, not significant; whole-report scoring gives +0.003). The claim stays scoped to MIMIC-CXR.
Why a masked generator (W5). The contrast itself does not need one: CAD, VCD and M3ID apply it in autoregressive decoding, and we agree the submission did not say why it uses a masked generator. The reason is the setting of the abstract (lines 13–14): a masked generator fills positions anywhere in the report at every step, from text on both sides, so the reference prediction has to be made on the same partial report as the current one, which TARING specifies. The revised introduction states this reason.
Presentation. The paper should have used plain terms; coined terms are now replaced, the abstract and Section 3 give the decoding rule in one line, proof steps are left to the appendix, and a new Section 4 paragraph says how the image enters LLaDA.
Response to jvbe (1/2): Reference and Corpus Effects
We thank Reviewer jvbe for the questions on the mechanism.
Why the training mean stands for corpus-common predictions (W1, Q1). At 170 decoding states of 100 test studies (steps 10, 40 and 70 of 80), the mean context's top token matches the average prediction of 256 random training studies at 96.4% of positions, with KL 0.017, against 92.0% and 0.080 for a single training study. The submission gave only the motivation; this measures how close the reference is to the average training study's prediction.
A first-order expansion motivates this approximation. Deviations of training image contexts from their mean sum to zero, so the linear terms of each token logit cancel on average. Nonlinear remainder terms remain, and Appendix A.10 retains them in the exact relation between the TARING score and the score based on averaged training-study logits. The measurements quantify predictive agreement at sampled decoding states; they do not directly measure or bound the remainder. Section 3 and Appendix A.10 present this argument and its empirical evaluation.
Whether corpus-common content decreases (W2, Q2). Corpus-common content decreases, but these analyses do not establish that this reduction causes the F1 gain. Corpus-common sentences, those that appear in at least 1% of the 172,608 training reports, make up 25.7% of the no-reference reports and 21.9% of TARING's (a significant drop, 95% paired bootstrap over patients; radiologists' reports: 11.0%), over all 2,357 test studies and three reveal seeds. Atelectasis, which the mean context writes on its own, is the label whose false positives TARING cuts most (498 to 327 in Table 8; its false negatives rise, 465 to 520).
TARING also recovers findings that the training majority would suppress: on the 3,327 label instances whose report-derived label goes against the majority training label (seed 0), recall of the 216 positives doubles, from 0.107 to 0.218 (+0.111 [0.045, 0.173]), while agreement on explicit negatives falls from 0.551 to 0.496, so this is part of a general shift toward positive statements.
At s = 1.75, image removal and a random-study reference also reduce the corpus-common sentence share (20.3% and 21.7%) without larger F1 gains. Image noising gives 25.0% with a micro-F1 gain that does not differ significantly from the training mean's. On seed 0, the association between changes in sentence share and correct-label counts is weakly positive (Spearman rho = 0.06), so studies with larger reductions in common-sentence share gain fewer correct labels. Appendix D.8 reports this analysis.
With one prespecified permutation of patients' images, the micro-F1 gain is +0.013 [0.003, 0.023], compared with +0.019 [0.010, 0.028] using the correct images. The difference between gains is +0.006 [-0.007, 0.019]. These results do not establish that the improvement is caused by more accurate use of the current study's images. Appendix D.10 reports the shuffled-image analysis and fixed-state token-choice diagnostic.