- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper selects auxiliary tasks in multi-task learning for primary-task explanation robustness under input perturbations, rather than accuracy. It defines PEER, the correlation of primary- and auxiliary-task attribution changes under perturbation, and an explanation instability functional (VAR). Assuming positive explanation transfer, a two-task homotopy argument ties PEER’s sign to the effect of deleting a task; postulating this for m tasks, a first-order analysis supports keeping only negative-PEER tasks (PEER-MTL). Experiments pair tuberculosis chest X-ray classification (primary) with 14 MIMIC-CXR labels (auxiliary) under WhatsApp compression. Across three ViTs, PEER-MTL lowers out-of-lung saliency and prediction instability versus TB-only, all-task and TAG baselines; the derived LLaVA-PEER loses less report quality under compression. The authors release radiologist-written reports for public TB datasets.
Critical strengths
-
The paper selects auxiliary tasks for explanation robustness rather than predictive accuracy, under a realistic, non-adversarial perturbation: lossy WhatsApp transmission in tuberculosis screening. PEER requires a single all-task training run plus attribution passes on paired inputs, so it avoids searching over 2^m task subsets. This framing is the paper’s most durable contribution. It asks a question that prior task-affinity work ignores and yields a testable, counter-intuitive hypothesis: retain auxiliary tasks whose attribution shifts oppose those of the primary task. It is the main reason I see value in a revised version.
-
Across three ViT backbones and four attribution methods (12 settings), PEER-MTL narrows the out-of-lung saliency gap between compressed and original images from 7.9–13.5 points (all-task MTL) to 0.8–2.6 points and lowers prediction instability. The released radiologist reports for public TB benchmarks add value independently of the method.
Critical weaknesses
-
The central claim is assumed, not derived. The PEER-deletion principle states that PEER’s sign predicts whether deleting an auxiliary task raises or lowers VAR. The claim to “prove” the selection rule is unsupported, and PEER’s validity rests entirely on the experiments.
-
No experiment compares PEER with the measured effect of deleting each task, for example through leave-one-task-out retraining. The paper never reports VAR, the quantity Theorem 1 bounds. OLS measures something else, and centering V removes the mean attribution shift that OLS captures.
Questions for authors
-
Paper states that PEER is computed on the MIMIC-CXR and TB test sets, and PI and OLS are reported on the same TB test pairs. Please confirm. If so, does S_KEEP change when PEER is computed on a validation split, and do Tables 1, 3 and 4 still hold?
-
Assumption 5 carries the method. For each of the 14 auxiliary tasks, does sign(PEER) agree with the measured change in VAR after leave-one-task-out retraining? Please report the sign-agreement rate and the correlation, over several seeds.
-
Please report VAR_t(θ_S) versus VAR_t(θ_0), the quantity Theorem 1 bounds, and per-image stability (e.g., rank correlation or top-k overlap between original and compressed maps). Why is V centered, given that centering removes the mean attribution shift that drives OLS? Do the conclusions hold for the uncentered error?
-
Which attribution method computes PEER? Please list exact PEER values with confidence intervals and the resulting S_KEEP for each backbone. Several scores lie near zero or differ markedly across backbones. Is S_KEEP stable across the four attribution methods?
-
How many seeds underlie each table, and what are the standard deviations and the test-set size?
-
Is θ_S trained from scratch or fine-tuned from θ_0? How are the TB and MIMIC losses weighted and sampled?
-
How many radiologists wrote the reports, were they blinded to the labels, and what quality control was applied?
2 comments
Response to Reviewer srkS
How many seeds underlie each table, and what are the standard deviations and the test-set size? Tables 1 and 3 are averaged over three seeds and per-entry standard deviations lie in [0.05, 0.89] for Table 1 and [0.05, 0.25] for Table 3. The test set has 1080 original and WhatsApp compressed image pairs
Is θ_S trained from scratch or fine-tuned from θ_0? How are the TB and MIMIC losses weighted and sampled?
θ_S is fine-tuned from θ₀. The objective is a sum of per-task BCE losses with unit coefficients. Auxiliary labels are obtained from MIMIC-CXR, and each image carries a combined label vector over the primary and auxiliary tasks, so batch composition does not depend on the task set and is identical for θ₀ and every θ_S.
How many radiologists wrote the reports, were they blinded to the labels, and what quality control was applied?
The reports were produced specifically for this dataset by three board-certified radiologists through a HIPAA-compliant tele-radiology platform. Each of the studies was reported once, with studies assigned randomly across the readers. Readers were blinded to all labels: they had access to the image alone, without clinical history or prior studies, and produced free-text reports following the platform’s standard reporting template.
Because each study was read once, inter-reader agreement cannot be computed on the dataset as collected. We acknowledge this as a limitation of the dataset contribution and will state it explicitly in the revised paper.
Response to Reviewer srkS
We thank the reviewer for the insightful feedback and suggestions. We report an additional experiment: leave-one-task-out retraining, in which each auxiliary task is removed individually and the model re-trained, with VAR and PEER_tj reported for all 14 tasks. These results have been added to the revised paper
Paper states that PEER is computed on the MIMIC-CXR and TB test sets, and PI and OLS are reported on the same TB test pairs. Please confirm.
The dataset is split into 3,510 / 810 / 1,080 original images for train / validation / test, As mentioned in Line 442 in the paper, PEER is computed using a held-out validation set of 810 original and compressed image pairs. We apologize for the confusion and have clarified this in Appendix F.
Assumption 5. For each of the 14 auxiliary tasks, does sign(PEER) agree with the measured change in VAR after leave-one-task-out retraining? Please report the sign-agreement rate, over several seeds.
We measure ΔVAR_t,j = VAR_t(θ₋ⱼ) − VAR_t(θ₀) across all 14 leave-one-task-out retrainings, where VAR_t is the quantity also reported as EIF in the paper, so that ΔVAR_t,j ≡ ΔEIF_j. The sign of PEER_tj is opposite to that of ΔVAR_t,j in all three seeds. Figure 7(a) in Appendix G3 shows this for all 14 auxiliary tasks: all 14 fall in the predicted quadrants of the (PEER_tj, ΔEIF_j) plane.
Please report VAR_t(θ_S) versus VAR_t(θ_0), and per-image stability (e.g., rank correlation or top-k overlap between original and compressed maps).
We report the quantity Theorem 1 bounds, together with two per-image stability measures between the original and compressed attribution maps: Rank-Biased Overlap (RBO), a rank correlation, and ‖V‖/‖A‖, the relative magnitude of the attribution shift. Here θ₀ is the all-task model (Vanilla-MTL) and θ_S the PEER-selected model.
VAR_t(θ_S) − VAR_t(θ₀) is negative on both backbones — −3.10e−01 for ViT-B/14 and −9.45e−02 for ViT-B/16, reductions of 6.65× and 8.56× — the direction Theorem 1 predicts, and the per-image measures move consistently with it, RBO rising by 0.147 and 0.050 and ‖V‖/‖A‖ falling by 2.43× and 2.33×
Why is V centered, given that centering removes the mean attribution shift that drives OLS? Do the conclusions hold for the uncentered error?
For the theory to go through there is no reason for V to be centred; all the formulas continue to hold, because they are stated in terms of Vt and no step uses the fact that E[Vt]=0.
Which attribution method computes PEER? Please list exact PEER values with confidence intervals and the resulting S_KEEP for each backbone. Several scores lie near zero or differ markedly across backbones. Is S_KEEP stable across the four attribution methods?
PEER is computed with input-gradient saliency and Grad-Attention. We restrict to these two of the four attribution methods because they are the most faithful under MoRF and LeRF deletion curves (Appendix Figure [4,5]).
Resulting S_KEEP = {j : PEER_tj < 0}.
S_KEEP is stable across the two attribution methods - saliency and Grad-Attention. As mentioned in line 455-458, the following tasks exhibit relatively high negative PEER scores (favorable for inclusion) across architectures: Consolidation, Lung Lesion, Pleural Effusion, Pneumonia and Lung Opacity. In contrast, the following tasks show high positive PEER scores (excluded due to expected negative transfer): Cardiomegaly, Edema, Enlarged Cardiomediastinum, and Pleural Other