acceptodds
Under review as a conference paper at ICLR 2027

Keep Your Peers: Task Selection for Robust Explanations in Multi-Task Learning

Abstract

Reliable deployment of large vision-language models (LVLMs) requires explanations that remain stable under realistic input perturbations, not merely accurate predictions. Yet multi-task learning (MTL) and auxiliary-task selection have been studied predominantly through predictive objectives, with comparatively little attention to explainability robustness. This raises a fundamental question: which auxiliary tasks should be selected to enhance the robustness of explanations for a primary task? This work addresses the question by formulating an explanation instability functional and analyzing its first variation under changes in auxiliarytask participation. The resulting variational analysis is generic and attributionagnostic: it operates at the level of the explanation functional and can be applied with any differentiable attribution map, rather than being tied to a particular explanation method, model architecture, or application. Motivated by this analysis, we introduce PEER(Perturbation Explanation Error Relation), a criterion based on the interaction between perturbation-induced explanation errors of the primary and auxiliary tasks. Under suitable regularity conditions, this relation identifies auxiliary tasks whose retention enhances the robustness of primary-task explanations. The framework is realized in LLaVA-PEER, a LLaVa model trained with PEER-MTL, for chest X-ray analysis in resource-constrained mHealth settings, where severe image compression provides a realistic source of perturbation. The study also releases a radiologist-annotated free-text reporting dataset for publicly available tuberculosis CXR benchmarks. Experiments show that PEERMTL reduces explanation instability and yields substantially more reliable LVLMgenerated reports.

Primary area:
interpretability and explainable AI
Keywords:
explainability, vision language model, robustness, medial VLM, chest x-ray

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_srkSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper selects auxiliary tasks in multi-task learning for primary-task explanation robustness under input perturbations, rather than accuracy. It defines PEER, the correlation of primary- and auxiliary-task attribution changes under perturbation, and an explanation instability functional (VAR). Assuming positive explanation transfer, a two-task homotopy argument ties PEER’s sign to the effect of deleting a task; postulating this for m tasks, a first-order analysis supports keeping only negative-PEER tasks (PEER-MTL). Experiments pair tuberculosis chest X-ray classification (primary) with 14 MIMIC-CXR labels (auxiliary) under WhatsApp compression. Across three ViTs, PEER-MTL lowers out-of-lung saliency and prediction instability versus TB-only, all-task and TAG baselines; the derived LLaVA-PEER loses less report quality under compression. The authors release radiologist-written reports for public TB datasets.

Critical strengths
  1. The paper selects auxiliary tasks for explanation robustness rather than predictive accuracy, under a realistic, non-adversarial perturbation: lossy WhatsApp transmission in tuberculosis screening. PEER requires a single all-task training run plus attribution passes on paired inputs, so it avoids searching over 2^m task subsets. This framing is the paper’s most durable contribution. It asks a question that prior task-affinity work ignores and yields a testable, counter-intuitive hypothesis: retain auxiliary tasks whose attribution shifts oppose those of the primary task. It is the main reason I see value in a revised version.

  2. Across three ViT backbones and four attribution methods (12 settings), PEER-MTL narrows the out-of-lung saliency gap between compressed and original images from 7.9–13.5 points (all-task MTL) to 0.8–2.6 points and lowers prediction instability. The released radiologist reports for public TB benchmarks add value independently of the method.

Critical weaknesses
  1. The central claim is assumed, not derived. The PEER-deletion principle states that PEER’s sign predicts whether deleting an auxiliary task raises or lowers VAR. The claim to “prove” the selection rule is unsupported, and PEER’s validity rests entirely on the experiments.

  2. No experiment compares PEER with the measured effect of deleting each task, for example through leave-one-task-out retraining. The paper never reports VAR, the quantity Theorem 1 bounds. OLS measures something else, and centering V removes the mean attribution shift that OLS captures.

Questions for authors
  1. Paper states that PEER is computed on the MIMIC-CXR and TB test sets, and PI and OLS are reported on the same TB test pairs. Please confirm. If so, does S_KEEP change when PEER is computed on a validation split, and do Tables 1, 3 and 4 still hold?

  2. Assumption 5 carries the method. For each of the 14 auxiliary tasks, does sign(PEER) agree with the measured change in VAR after leave-one-task-out retraining? Please report the sign-agreement rate and the correlation, over several seeds.

  3. Please report VAR_t(θ_S) versus VAR_t(θ_0), the quantity Theorem 1 bounds, and per-image stability (e.g., rank correlation or top-k overlap between original and compressed maps). Why is V centered, given that centering removes the mean attribution shift that drives OLS? Do the conclusions hold for the uncentered error?

  4. Which attribution method computes PEER? Please list exact PEER values with confidence intervals and the resulting S_KEEP for each backbone. Several scores lie near zero or differ markedly across backbones. Is S_KEEP stable across the four attribution methods?

  5. How many seeds underlie each table, and what are the standard deviations and the test-set size?

  6. Is θ_S trained from scratch or fine-tuned from θ_0? How are the TB and MIMIC losses weighted and sampled?

  7. How many radiologists wrote the reports, were they blinded to the labels, and what quality control was applied?

2 comments
AuthorsOpenReview ↗

Response to Reviewer srkS

We thank the reviewer for the insightful feedback and suggestions. We report an additional experiment: leave-one-task-out retraining, in which each auxiliary task is removed individually and the model re-trained, with VAR and PEER_tj reported for all 14 tasks. These results have been added to the revised paper

Paper states that PEER is computed on the MIMIC-CXR and TB test sets, and PI and OLS are reported on the same TB test pairs. Please confirm.

The dataset is split into 3,510 / 810 / 1,080 original images for train / validation / test, As mentioned in Line 442 in the paper, PEER is computed using a held-out validation set of 810 original and compressed image pairs. We apologize for the confusion and have clarified this in Appendix F.

Assumption 5. For each of the 14 auxiliary tasks, does sign(PEER) agree with the measured change in VAR after leave-one-task-out retraining? Please report the sign-agreement rate, over several seeds.

We measure ΔVAR_t,j = VAR_t(θ₋ⱼ) − VAR_t(θ₀) across all 14 leave-one-task-out retrainings, where VAR_t is the quantity also reported as EIF in the paper, so that ΔVAR_t,j ≡ ΔEIF_j. The sign of PEER_tj is opposite to that of ΔVAR_t,j in all three seeds. Figure 7(a) in Appendix G3 shows this for all 14 auxiliary tasks: all 14 fall in the predicted quadrants of the (PEER_tj, ΔEIF_j) plane.

Please report VAR_t(θ_S) versus VAR_t(θ_0), and per-image stability (e.g., rank correlation or top-k overlap between original and compressed maps).

We report the quantity Theorem 1 bounds, together with two per-image stability measures between the original and compressed attribution maps: Rank-Biased Overlap (RBO), a rank correlation, and ‖V‖/‖A‖, the relative magnitude of the attribution shift. Here θ₀ is the all-task model (Vanilla-MTL) and θ_S the PEER-selected model.

ModelVAR_t‖V‖/‖A‖ ↓RBO ↑
ViT-B/14
θ₀ (Vanilla-MTL)3.65e−010.28750.7822
θ_S (PEER-MTL)5.49e−020.11810.9287
VAR_t(θ_S) − VAR_t(θ₀)−3.10e−01
ViT-B/16
θ₀ (Vanilla-MTL)1.07e−010.14210.8842
θ_S (PEER-MTL)1.25e−020.06110.9341
VAR_t(θ_S) − VAR_t(θ₀)−9.45e−02

VAR_t(θ_S) − VAR_t(θ₀) is negative on both backbones — −3.10e−01 for ViT-B/14 and −9.45e−02 for ViT-B/16, reductions of 6.65× and 8.56× — the direction Theorem 1 predicts, and the per-image measures move consistently with it, RBO rising by 0.147 and 0.050 and ‖V‖/‖A‖ falling by 2.43× and 2.33×

Why is V centered, given that centering removes the mean attribution shift that drives OLS? Do the conclusions hold for the uncentered error?

For the theory to go through there is no reason for V to be centred; all the formulas continue to hold, because they are stated in terms of VtV_t and no step uses the fact that E[Vt]=0E[V_t] = 0.

Which attribution method computes PEER? Please list exact PEER values with confidence intervals and the resulting S_KEEP for each backbone. Several scores lie near zero or differ markedly across backbones. Is S_KEEP stable across the four attribution methods?

PEER is computed with input-gradient saliency and Grad-Attention. We restrict to these two of the four attribution methods because they are the most faithful under MoRF and LeRF deletion curves (Appendix Figure [4,5]).

TaskViT-B/16ViT-B/14ViT-B/32
Atelectasis-0.020 +- 0.0012-0.050 +- 0.0070-0.069 +- 0.0022
Cardiomegaly+0.410 +- 0.0024+0.480 +- 0.0024+0.103 +- 0.0065
Consolidation-0.200 +- 0.0058-0.950 +- 0.0018-0.608 +- 0.0084
Edema+0.250 +- 0.0109+0.240 +- 0.0040+0.095 +- 0.0093
Enlarged Cardiomediastinum+0.200 +- 0.0069+0.200 +- 0.0027+0.870 +- 0.0023
Fracture-0.010 +- 0.0093-0.020 +- 0.0086+0.338 +- 0.0043
Lung Lesion-0.450 +- 0.0100-0.810 +- 0.0096-0.379 +- 0.0011
Lung Opacity-0.100 +- 0.0078-0.150 +- 0.0043-0.127 +- 0.0036
No Finding-0.040 +- 0.0018+0.020 +- 0.0013+0.044 +- 0.0054
Pleural Effusion-0.140 +- 0.0019-0.760 +- 0.0041-0.465 +- 0.0043
Pleural Other+0.320 +- 0.0198+0.120 +- 0.0054+0.293 +- 0.0051
Pneumonia-0.080 +- 0.0098-0.230 +- 0.0048-0.232 +- 0.0023
Pneumothorax-0.060 +- 0.0085-0.120 +- 0.0034-0.084 +- 0.0035
Support Devices-0.010 +- 0.0016-0.050 +- 0.0093-0.127 +- 0.0034

Resulting S_KEEP = {j : PEER_tj < 0}.

S_KEEP is stable across the two attribution methods - saliency and Grad-Attention. As mentioned in line 455-458, the following tasks exhibit relatively high negative PEER scores (favorable for inclusion) across architectures: Consolidation, Lung Lesion, Pleural Effusion, Pneumonia and Lung Opacity. In contrast, the following tasks show high positive PEER scores (excluded due to expected negative transfer): Cardiomegaly, Edema, Enlarged Cardiomediastinum, and Pleural Other

AuthorsOpenReview ↗

Response to Reviewer srkS

How many seeds underlie each table, and what are the standard deviations and the test-set size? Tables 1 and 3 are averaged over three seeds and per-entry standard deviations lie in [0.05, 0.89] for Table 1 and [0.05, 0.25] for Table 3. The test set has 1080 original and WhatsApp compressed image pairs

Is θ_S trained from scratch or fine-tuned from θ_0? How are the TB and MIMIC losses weighted and sampled?

θ_S is fine-tuned from θ₀. The objective is a sum of per-task BCE losses with unit coefficients. Auxiliary labels are obtained from MIMIC-CXR, and each image carries a combined label vector over the primary and auxiliary tasks, so batch composition does not depend on the task set and is identical for θ₀ and every θ_S.

How many radiologists wrote the reports, were they blinded to the labels, and what quality control was applied?

The reports were produced specifically for this dataset by three board-certified radiologists through a HIPAA-compliant tele-radiology platform. Each of the studies was reported once, with studies assigned randomly across the readers. Readers were blinded to all labels: they had access to the image alone, without clinical history or prior studies, and produced free-text reports following the platform’s standard reporting template.
Because each study was read once, inter-reader agreement cannot be computed on the dataset as collected. We acknowledge this as a limitation of the dataset contribution and will state it explicitly in the revised paper.

Reviewer_QdhGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper claims that auxiliary task selection has been viewed from the lens of predictive accuracy but no one has looked at it from the perspective of explanation robustness. The idea is interesting overall and finds a legitimate gap in current research around auxiliary task selection. Given a perturbation type, model and a set of tasks, the authors propose to use the correlation between the task errors under the defined perturbation using PEER. Tasks with negative PEER are kept while ones with positive PEER are dropped. This idea is tested out on TB screening from chest X-rays with 14 auxiliary labels posed as tasks. Upon training with PEER-selected auxiliary tasks, authors show that out-of-lung (OLS) saliency and prediction instability was lower compared to model trained using primary task only, vanilla MTL and TAG. In addition to this, the author release radiologist-written reports for three datasets.

Critical strengths

I like the premise of this paper in general. Task affinity literature has traditionally focused on prediction accuracy and this paper asks a questions that has not been addressed or asked before according to my knowledge. I like that this is motivated from a real problem due to compression from whatsapp. The metric is cheap to compute as it does not require you to retrain the model for every unique combination of tasks, which helps with scalability.

Empirical results suggest that this helps across three backbones and four attribution methods. PEER-MTL, the model trained with tasks identified by PEER reduces the difference in OLS score between the original and whatsapp compressed data to about ~2 % compared to ~10% by task affinity work. I like that this work stands as a demonstration for showing utility of explanations, which are often critized for lack of useful application other than fulfilling general curiosity of the person using it.

Finally, the report dataset is a nice contribution on its own and would aid other researchers working on report generation.

Critical weaknesses

No evidence to suggest that compression induces prediction instability. Figure 1 shows one failure mode but “can PEER help?” Is not shown in Figure 1 qualitatively and would be nice to add.

Have you measured OLS with models that were fine-tuned with multi-resolution augmentation data? This is a critical baseline that is missing from the paper and is a trivial question one would want to know. How better is PEER than retraining with data augmentation? This is easy to compute under the perturbation chosen in the paper

Another baseline is to pose this as a seach problem and use something like beam search to find the best auxiliary tasks.

Instead of using PEER-deletion, wouldnt it make more sense to do PEER-insertion as there is a chance of getting to minimality faster? The paper should address the complement for completeness

The paper is missing an important baseline of containing size-matched random tasks that are the same size as S_KEEP

Theory predicts that positive-PEER tasks should be worse than vanilla but this was not demonstrated empirically and would be a nice test.

How well does explanations robustness regularizers mentioned in the paper deal with OLS? These numbers are missing.

With 14 tasks, a leave-one-out retraining. A scatter plot of \delta OLS_j vs PEER_{tj} would help more than Figure 6

At a high level, the tasks selected by PEER (consolidation, effusion, opacity, lung lesion) seem to align with what a radiologist would associate with TB and the one it drops (enlarged cardiomediastinum, edema, cardiomegaly) are mostly cardiac associated conditions. I wonder what that the contribution in the paper provides in this domain, as a radiologist could select tasks based on their domain knowledge that can do equally well?

The abstract says the analysis is generic and adaptable, which sounds fine at the methodological level but is not demonstrated with their experiments. The paper deals with a single perturbation type, domain and primary task.

Coming to the theory the paper claims to contribute, Assumption 5 assumes the sign of \delta_j is opposite to sign of PEEER and Theorem 1’s claim one is a restatement of the same and I see no reason why dVAR/dn should be a positive multiple of E[V_t^T V_a]. As it sits I feel the paper is a hypothesis that was supported by the reported experiments

The thorem might not apply to the experiment scale as the, the constants in condition 2 of Theorem need to be defined for the dataset. EIF is not reported for experiments. Also report from similarity metric like maybe say SSIM between original and whatsapp compressed attribution maps?

The paper does not report standard deviations or multiple seeds anywhere and additionally TB-only saliency OLS is exactly the same for all three backbones which is worth double checking as that is an highly unlikely event.

For the report generation dataset, how many radiologists were used, what was the reporting protocol and inter-reporter agreement? These details are missing and are needed when one of the contribution is the dataset.

Questions for authors

Have you measured OLS with models that were fine-tuned with multi-resolution augmentation data? This is a trivial question one would want to know and it is easy to compute under the perturbation chosen in the paper. How much better is PEER than just retraining with data augmentation?

Did you try posing this as a search problem and using something like beam search to find the best auxiliary tasks? How does PEER compare to that in OLS and in compute?

Instead of PEER-deletion, wouldnt it make more sense to do PEER-insertion as there is a chance of getting to minimality faster? Can you address the complement for completeness?

How does PEER-MTL compare against size-matched random tasks that are the same size as S_KEEP? Theory predicts that keeping only the positive-PEER tasks should be worse than vanilla MTL. Have you tested this empirically? It would be a nice test. How well do the explanation robustness regularizers mentioned in the paper deal with OLS? Can you add these numbers?

With 14 tasks, would it be possible to do a leave-one-out retraining and show a scatter plot of \delta OLS_j vs PEER_{tj}? I think this would help more than Figure 6.

The tasks selected by PEER (consolidation, effusion, opacity, lung lesion) seem to align with what a radiologist would associate with TB and the ones it drops (enlarged cardiomediastinum, edema, cardiomegaly) are mostly cardiac associated conditions. If a radiologist selected tasks based on their domain knowledge, would PEER still do better? If not, what does PEER provide in this domain?

The abstract says the analysis is generic, which sounds fine at the methodological level. Do you have results on any other perturbation type, domain or primary task to back this up?

Coming to the theory, Assumption 5 assumes the sign of \delta_j is opposite to the sign of PEER and claim one of Theorem 1 is a restatement of the same. Is there any reason to expect dVAR/d\eta to be a positive multiple of E[V_t^T V_a] beyond the two tasks shown in Figure 6? As it sits the selection rule reads more like a hypothesis that is supported by the experiments.

Can you estimate the constants in condition 2 of Theorem 1 for your dataset? Without them it is hard to tell if the theorem applies at the scale of the experiments.

Can you report EIF for the experiments, along with a similarity metric like SSIM between the attribution maps on original and whatsapp compressed images?

Can you report standard deviations over multiple seeds? Also the TB-only saliency OLS is exactly the same (99.93) for all three backbones, could you double check this as it is a highly unlikely event?

For the report generation dataset, how many radiologists were used, what was the reporting protocol and what was the inter-reporter agreement? These details are needed since the dataset is one of the contributions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QdhG

Thank you for the suggestions and the insightful comments. We report three additional configurations: fine-tuning with multi-resolution augmentation over original and compressed images; training on the complement set {j:PEERtj>0}\{j : \mathrm{PEER}_{tj} > 0\} (Pos-PEER), and training on random auxiliary subsets of the same cardinality as SS. We additionally report EIF for each configuration and the estimated constants of Theorem 1’s condition 2. These results have been added to the revised paper

Have you measured OLS with models that were fine-tuned with multi-resolution augmentation data? How much better is PEER than just retraining with data augmentation?

We fine-tuned with multi-resolution augmentation over original and WhatsApp-compressed TB images, and evaluated OLS (lower is better) under the same protocol as Table 1 in main paper.

Attribution methodmulti-res (Orig)multi-res (WA)TB-only (Orig)TB-only (WA)PEER-MTL (Orig)PEER-MTL (WA)
ViT-B/14
Saliency52.8998.9853.4499.9345.6347.47
Grad-Attn53.1485.3653.1286.4735.8637.62
Gradcam57.1781.2456.2580.5444.0446.21
Legrad56.3185.2356.2884.6143.6245.83
ViT-B/16
Saliency54.8499.1254.6499.9345.6048.22
Grad-Attn54.1283.7853.4583.2844.3245.18
Gradcam56.2481.7856.2481.4444.6446.07
Legrad57.3282.0457.0282.1743.6445.91

Augmentation does not improve explanation stability. PEER-MTL, by contrast, lowers OLS by 7.3–17.3 points on original images and 35.0–51.5 points on compressed ones relative to augmentation.

Did you try posing this as a search problem and using something like beam search to find the best auxiliary tasks? How does PEER compare to that in OLS and in compute?

We thank the reviewer for the suggestion. Posing selection as a search is the natural formulation, and the obstacle is the cost of its evaluation function: scoring a candidate subset requires training a model. Over 14 auxiliary tasks, greedy forward selection to depth 5 requires 60 trainings, beam search with width 3 requires 152, and exhaustive search requires 2¹⁴ − 1 = 16,383. PEER requires one scoring pass and one training of the selected model. At our per-run cost of 12 GPU-hours this is 24 GPU-hours against 1824 GPU-hours for beam search, a factor of 152. Using Peer to prune candidate subset can be regarded as a direction of future work.

Instead of PEER-deletion, wouldnt it make more sense to do PEER-insertion as there is a chance of getting to minimality faster? Can you address the complement for completeness?

PEER estimates are obtained from the full MTL model. If we want to introduce a task that the model has not seen, we do not know its PEER estimate. Unless one makes the stronger assumption that PEER values are available for unseen tasks, PEER-insertion may not be possible. Addressing the complement for completeness can be regarded as directions for future work.

How does PEER-MTL compare against size-matched random tasks that are the same size as S_KEEP? Have you tested this empirically?

KEEP attains the lowest OLS in all 16 configurations (two backbones × four attribution methods × original and compressed images), improving on a random subset of the same size by 6.2–15.5 points and on the reverse rule Pos-PEER by 13.3–16.2 points on average.

Attribution methodRandom (orig)Random (wp)Pos-PEER (orig)Pos-PEER (wp)
ViT-B/14
saliency51.3459.6457.4559.21
gradattn49.2357.7458.2359.95
gradcam51.2261.4558.7361.34
legrad48.9960.2359.1261.53
ViT-B/16
saliency52.6459.4757.3261.83
gradattn48.8358.1755.3260.46
gradcam53.3559.4458.8462.49
legrad48.3161.2859.8262.99

How well do the explanation robustness regularizers mentioned in the paper deal with OLS? Can you add these numbers?

We attempted to train regularization methods under this protocol and encountered out-of-memory at batch size 4 on [NVIDIA A100 GPU, 40GB*4], which is why we do not report it; the respective papers (Chen et al., 2024a; Wang et al.,2020) evaluate on ResNet-18 model for CIFAR-10 and Flower dataset.

With 14 tasks, would it be possible to do a leave-one-out retraining and show a scatter plot of \delta OLS_j vs PEER_{tj}? I think this would help more than Figure 6.

For each auxiliary task j we retrain without that task and measure ΔOLS_j = OLS(θ-ⱼ) − OLS(θ₀), the change in OLS caused by removing j. Figure 7 in Appendix shows that sign(ΔOLS_j) = −sign(PEER_tj), for all 14 auxiliary tasks.

AuthorsOpenReview ↗

Response to Reviewer QdhG

The tasks selected by PEER (consolidation, effusion, opacity, lung lesion) seem to align with what a radiologist would associate with TB and the ones it drops (enlarged cardiomediastinum, edema, cardiomegaly) are mostly cardiac associated conditions. If a radiologist selected tasks based on their domain knowledge, would PEER still do better? If not, what does PEER provide in this domain?

This comparison would require active engagement with clinicians to construct a properly elicited prior, and currently we are unable to answer this.

The abstract says the analysis is generic, which sounds fine at the methodological level. Do you have results on any other perturbation type, domain or primary task to back this up?

No, we are involved in chest x-ray explainability analysis and the analysis: PEER, VAR_t, Assumption 5 and Theorem 1 are stated for an arbitrary perturbation pair (X, X̃) and an arbitrary primary task t, and no step of the derivation uses a property of compression artefacts, of chest radiographs, or of tuberculosis. Chest x-ray explainability analysis on real-world mhealth settings have significant impact in improving health-care delivery in resource-constrained settings. We chose this instantiation because it is a setting where the perturbation is not synthetic: mobile-health workflows transmit diagnostic chest radiographs through consumer messaging platforms, so the degradation we study is the one deployed models actually encounter. That makes it a strong test of the rule.

Coming to the theory, Assumption 5 assumes the sign of \delta_j is opposite to the sign of PEER and claim one of Theorem 1 is a restatement of the same. Is there any reason to expect dVAR/d\eta to be a positive multiple of E[V_t^T V_a] beyond the two tasks shown in Figure 6? As it sits the selection rule reads more like a hypothesis that is supported by the experiments.

Claim 1 is stated for readability, we are happy to remove claim 1 from Theorem 1. In the two-task case the Peer-deletion principle is derived using positive explanation transfer (Assumption 1). For more than two tasks we do not currently have a proof of the PEER-deletion principle; however, there is empirical evidence to suggest the same.

Can you estimate the constants in condition 2 of Theorem 1 for your dataset? Without them it is hard to tell if the theorem applies at the scale of the experiments.

The constants are estimated for Vit-B32 Vanilla MTL model (θ0\theta_0) as follows:

ConstantValue
M1.0649e−01
C1.16785e+00
L8.8196e−04
‖h_S‖2.0518e−01

Can you report EIF for the experiments, along with a similarity metric like SSIM between the attribution maps on original and whatsapp compressed images?

We report EIF for ViT-B/14 and ViT-B/16, together with relative measures - ‖V‖/‖A‖ and Rank-Biased Overlap (RBO). We prefer RBO to SSIM because SSIM is defined in terms of luminance and contrast statistics, which have no interpretation for an attribution map; while RBO depends only on the ordering of the values.

ModelEIF ↓‖V‖/‖A‖ ↓RBO ↑
ViT-B/14
TB-only8.650.64370.5175
Vanilla-MTL3.65e−010.28750.7822
TAG-MTL7.65e−010.32190.7593
PEER-MTL (ours)5.49e−020.11810.9287
ViT-B/16
TB-only2.070.58750.5422
Vanilla-MTL1.07e−010.14210.8842
TAG-MTL3.47e−010.24370.8175
PEER-MTL (ours)1.25e−020.06110.9341

The three measures induce the same ranking on both backbones — PEER-MTL, Vanilla-MTL, TAG-MTL, TB-only. PEER-MTL raises RBO by 0.147 and 0.050 over Vanilla-MTL, by 0.169 and 0.117 over TAG-MTL, and by 0.411 and 0.392 over TB-only (ViT-B/14 and ViT-B/16 respectively).

Can you report standard deviations over multiple seeds? Also the TB-only saliency OLS is exactly the same (99.93) for all three backbones..

The standard deviation is in the range of [0.14, 0.89] over three seeds reported per entry in Table 1. TB-only saliency OLS (99.9307 , 99.9288 , 99.9304 ) is rounded off to be the same for all three backbones.

For the report generation dataset, how many radiologists were used, what was the reporting protocol and what was the inter-reporter agreement?

The reports were produced specifically for this dataset by three board-certified radiologists through a HIPAA-compliant tele-radiology platform. Each of the studies was reported once, with studies assigned randomly across the readers. Readers had access to the image only, without clinical history or prior studies, and produced free-text reports following the platform’s standard reporting template. Because each study was read once, inter-reader agreement cannot be computed on the dataset as collected. We acknowledge this as a limitation of the dataset contribution and state it explicitly in the revised paper.

Reviewer_QdhGOpenReview ↗

Response to author reply

Thank you for your response. I will go through the full response in detail and base my assessment on the author response.

AuthorsOpenReview ↗

Response to Reviewer QdhG

Thank you for the reply, and for taking the time to consider the response in detail.

Reviewer_3kS4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies auxiliary-task selection in multi-task learning from the point of view of explanation stability. For each task, the authors measure how its attribution map changes when the input is perturbed, centred over the data. Explanation instability (EIF) is half the expected squared norm of this centred attribution change for the primary task, and PEER is the normalised inner product between the changes of the primary task and an auxiliary task. After training a model on all tasks, the KEEP rule retains only the auxiliary tasks with negative PEER and retrains on that subset. The rule is supported by a two-task argument along an auxiliary-weight path (Proposition 1, under Assumption 1) and a first-order result for many tasks (Theorem 1, under Assumptions 2 to 5). The experiments use TB classification on Montgomery, Shenzhen and TBX11K (5,400 images) as the primary task, 14 MIMIC-CXR findings as auxiliary tasks, and WhatsApp compression of the test images as the perturbation. With ViT-B/14, B/16 and B/32, PEER-MTL shows lower out-of-lung saliency (OLS) and prediction instability than TB-only, Vanilla-MTL and TAG-MTL, and a LLaVA model trained with the selected tasks achieves the best report-generation scores. The authors also release radiologist-written reports for the three TB datasets.

Critical strengths
  1. Relevant problem and a practical setting. Figure 1 shows compressed X-rays that receive the same prediction but very different attribution maps, which is a real concern for AI-assisted screening over messaging apps. The use of images actually transmitted through WhatsApp, rather than only simulated compression, makes the perturbation setting practically relevant to low-bandwidth medical screening.

  2. A new task-selection signal with consistent empirical gains. Selecting auxiliary tasks by how their explanations respond to perturbation is a fresh angle on a problem usually approached through predictive accuracy. The procedure needs one full multi-task model and one retraining on the selected subset, instead of retraining for every candidate subset. Empirically, the gap between OLS on original and compressed images is 0.8 to 2.6 points for PEER-MTL against 7.6 to 15.1 for Vanilla-MTL and TAG-MTL, across all 12 backbone and attribution-method settings in Table1, and LLaVA-PEER loses 1.40 RadGraph points under compression against 4.34 for LLaVA-VanillaMTL. These results are promising, although OLS is not the same quantity as the EIF used in the theory.

Critical weaknesses
  1. The experiments do not isolate the contribution of PEER. First, the described protocol appears to allow test-set information to influence task selection. Section 5.1 describes transmitting the test images through WhatsApp, while Section 5.2 and Appendix G.2 state that PEER is computed using original and compressed images from "the MIMIC-CXR and TB test datasets". The final robustness results are also reported on the TB test images. The paper does not state which images were used at each stage, so the authors should confirm the split protocol and, if needed, recompute PEER on compressed validation images. Second, the necessary controls are missing. The prominently reported negative-PEER tasks (Consolidation, Lung Lesion, Pleural Effusion and Lung Opacity) are already plausible pulmonary tasks to train alongside TB detection, yet there are no random subsets of the same size, no clinically chosen subset, no reverse rule that keeps positive-PEER tasks, and no variation across seeds. TAG, the only selection baseline, has higher compressed- image OLS than Vanilla-MTL in all 12 cells of Table 1, and most of the reduction in prediction instability already comes from multi-task training itself (12.5 to 14.8 for TB-only, 2.1 to 3.4 for Vanilla-MTL, 1.3 to 2.3 for PEER-MTL). Third, the key quantities are not reported. EIF, the objective defined in Section 3, is not reported for any of the compared models, and OLS does not compare the two attribution maps of an image pair. TB accuracy, AUC, sensitivity and specificity are also absent, so the trade-off between diagnostic performance and explanation robustness cannot be assessed. PEER may well be effective, but with these gaps the reported gains cannot yet be attributed to it.

  2. The theoretical guarantee relies on assumptions that encode the PEER-deletion relationship. Assumption 1 states that the derivative of the instability along the auxiliary-weight path has the same sign as the explanation alignment, and Proposition 1 integrates this assumed relationship. The derivative actually derived in Appendix A.2 (Eq. 17) does not involve the auxiliary task's explanation response VaV_a, and no argument links its sign to the alignment. For many tasks, the authors state that they "will assume that the principle holds", Assumption 5 imposes the sign relationship directly, and Theorem 1 additionally needs an improvement margin that is not checked for any trained model. Proposition 1 also concerns a model trained with one auxiliary task, while PEER is computed on the model trained with all tasks. The theorem provides a conditional result, but its assumptions do not independently establish why PEER should work for general multi-task networks, and the empirical support is narrow. Figure 6 covers two auxiliary tasks (Consolidation and Cardiomegaly) on ViT-B/32 at about ten points of the path. The introduction's claim to "prove" that retaining the negatively scored tasks lowers EIF should therefore be softened. Comparing the sign of PEER with the effect of leave-one-task-outretraining, which requires 14 retraining experiments per backbone and seed, would test the central assumption directly.

Questions for authors
  1. Split protocol. Which images exactly were used to compute PEER and to produce Figure 6? Since the TB labels and the auxiliary labels come from different datasets, on which images are the primary and auxiliary explanation responses paired? If any TB test images were involved, please recompute PEER on a compressed validation split disjoint from the test set and report whether the selected tasks and the gains in Tables 1 and 3 remain the same.

  2. Does PEER predict the effect of removing a task? Assumption 5 states that PEER and the deletion effect have opposite signs. For each auxiliary task, how often is the sign of PEER opposite to (a) the first-order deletion effect Δj\Delta_j of Eq. 8, for example estimated with Hessian-vector products, and (b) the actual change in EIF after leave-one-task-out retraining, measured as the EIF without task j minus the EIF with all tasks? Consistency in (a) would support the first-order analysis in Theorem 1, and consistency in (b) would show that PEER predicts what happens after actual retraining. How does KEEP compare with random subsets of the same size and with the reverse rule, over several seeds?

  3. Justification of Assumption 1. Eq. 17 expresses the derivative of VARt\mathrm{VAR}_t along the path through the change of VtV_t in the direction Hη−1∇R^aH_\eta^{-1}\nabla \hat{R}_a, and it does not involve VaV_a. Under what conditions on the shared representation does its sign match the sign of Cta(η)C_{ta} (\eta)? Since Proposition 1 uses PEER at the two-task model θ1\theta_1 while KEEP uses the all-task model θ0\theta_0, how often does the sign of PEER agree between these two models?

  4. EIF and diagnostic performance. Does PEER-MTL reduce EIF itself, compared with TB-only, Vanilla-MTL and TAG-MTL, and what are the TB AUC, sensitivity and specificity of these models on original and compressed images?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3kS4

We thank the reviewer for the insightful comments and feedback. Following the suggestion, we report four additional configurations : (i) 14 leave-one-task-out retrainings, which test the sign relation of Assumption 5 directly; (ii) training on the complement set {j : PEER_tj > 0}; (iii) training on random subsets of the same cardinality as S; and (iv) the 14 two-task models pairing each auxiliary task with Tuberculosis. We report EIF (VAR_t) and PEER_tj. These results have been added to the revised paper

Split protocol. Which images exactly were used to compute PEER and to produce Figure 6? The dataset is split into 3,510 / 810 / 1,080 original images for train / validation / test, As mentioned in Line 442 in the paper, Figure 6 was generated using a held-out validation set of 810 original and compressed image pairs. We apologize for the confusion and have clarified this in Appendix F.

Since the TB labels and the auxiliary labels come from different datasets, on which images are the primary and auxiliary explanation responses paired?

Primary and auxiliary responses are paired on the same images. Since V_i is a function of head i’s attribution map on an original–perturbed pair (X, X̃), every head can be evaluated on the same image irrespective of which dataset supplied that task’s labels.

Does PEER predict the effect of removing a task? Assumption 5 states that PEER and the deletion effect have opposite signs. For each auxiliary task, how often is the sign of PEER opposite to (a) the first-order deletion effect of Eq. 8, for example estimated with Hessian-vector products, and

Peer predicts effect of removal of task based on Peer Deletion Priciple. Assumption 5 predicts PEERtj⋅Δj≤0\mathrm{PEER}_{tj}\cdot\Delta_j \le 0, where Δj=∇VARt(θ0)⊤H−1∇R^j(θ0)\Delta_j = \nabla \mathrm{VAR}_t(\theta_0)^\top H^{-1}\nabla \hat{R}_j(\theta_0) is the first-order deletion effect of Eq. 8. We evaluate both quantities on the ViT-B/32 model, computing Δj\Delta_j with Hessian-vector products and conjugate gradients at damping λ=1.0\lambda = 1.0. The sign relation holds for 14 of the 14 tasks in Vit-B/32

Auxiliary task jjPEERtj\mathrm{PEER}_{tj}Δj\Delta_j (λ=1.0\lambda = 1.0)PEERtj⋅Δj<0\mathrm{PEER}_{tj}\cdot\Delta_j < 0
Atelectasis−0.0694-0.06948.54×10−48.54\times10^{-4}✓
Cardiomegaly0.10300.1030−1.26×10−3-1.26\times10^{-3}✓
Consolidation−0.6075-0.60753.67×10−43.67\times10^{-4}✓
Edema0.09470.0947−9.46×10−4-9.46\times10^{-4}✓
Enlarged Cardiomediastinum0.87010.8701−1.54×10−3-1.54\times10^{-3}✓
Fracture0.33800.3380−1.11×10−4-1.11\times10^{-4}✓
Lung Lesion−0.3790-0.37903.96×10−43.96\times10^{-4}✓
Lung Opacity−0.1268-0.12685.54×10−45.54\times10^{-4}✓
No Finding0.04400.0440−1.77×10−2-1.77\times10^{-2}✓
Pleural Effusion−0.4646-0.46465.60×10−45.60\times10^{-4}✓
Pleural Other0.29330.2933−3.85×10−5-3.85\times10^{-5}✓
Pneumonia−0.2315-0.23152.06×10−32.06\times10^{-3}✓
Pneumothorax−0.0844-0.08442.98×10−52.98\times10^{-5}✓
Support Devices−0.1266-0.12661.45×10−41.45\times10^{-4}✓

(b) the actual change in EIF after leave-one-task-out retraining, measured as the EIF without task j minus the EIF with all tasks?

For each auxiliary task j we retrain without that task and measure ΔEIF_j = EIF(θ-ⱼ) − EIF(θ₀), the change in explanation instability caused by removing j. Assumption 5 predicts sign(ΔEIF_j) = −sign(PEER_tj). Figure 7 in Appendix shows this for all 14 auxiliary tasks: all 14 fall in the predicted quadrants of the (PEER_tj, ΔEIF_j) plane.

How does KEEP compare with random subsets of the same size and with the reverse rule, over several seeds?

KEEP attains the lowest OLS in all 16 configurations (two backbones × four attribution methods × original and compressed images), improving on a random subset of the same size by 6.2–15.5 points and on the reverse rule Pos-PEER by 13.3–16.2 points on average.

ModelMethodsaliency (orig)saliency (wp)gradattn (orig)gradattn (wp)gradcam (orig)gradcam (wp)legrad (orig)legrad (wp)
ViT-B/14Random51.3459.6449.2357.7451.2261.4548.9960.23
Pos-PEER57.4559.2158.2359.9558.7361.3459.1261.53
ViT-B/16Random52.6459.4748.8358.1753.3559.4448.3161.28
Pos-PEER57.3261.8355.3260.4658.8462.4959.8262.99
AuthorsOpenReview ↗

Response to Reviewer 3kS4

Justification of Assumption 1. Eq. 17 expresses the derivative of VAT_t along the path through the change of V_t in the direction H^{-1} \nabla \hat{R}a, and it does not involve V_a . Under what conditions on the shared representation does its sign match the sign of C{ta}(\eta)?

In Eq. 17 the effect of task aa is reflected through R^a\hat{R}_a. Assumption 1 makes the connection explicit: it posits that the derivative of VARt\mathrm{VAR}_t depends on VaV_a with the same sign as Cta(η)C_{ta}(\eta). The justification is the one stated in line 239 — an auxiliary task that improves performance does so by promoting a useful shared representation, and positive transfer between related tasks is well documented in multi-task learning. We do not claim that all auxiliary tasks have this property; PEER’s role is precisely to identify those that do. In the two-task case we characterise the relation under positive explanation transfer and verify it empirically in Figure 6. A general characterisation for arbitrary task sets remains open, and we regard it as a direction for future work.

Since Proposition 1 uses PEER at the two-task model while KEEP uses the all-task model , how often does the sign of PEER agree between these two models?

For each auxiliary task jj we compute PEER$_{tj}$ in the two-task model trained on {t,j}\{t, j\} and in the all-task model, on the same backbone (ViT-B/14) and the same held-out image pairs. From the result it is evident that the signs agree for all 14 of the 14 auxiliary tasks.

Auxiliary taskPEER (two-task)PEER (all-task)Signs agree
Atelectasis−0.39−0.05yes
Cardiomegaly0.100.48yes
Consolidation−0.61−0.95yes
Edema0.890.24yes
Enlarged Cardiomediastinum0.200.20yes
Fracture-0.34−0.02yes
Lung Lesion−0.94−0.81yes
Lung Opacity−0.90−0.15yes
No Finding0.970.02yes
Pleural Effusion−0.74−0.76yes
Pleural Other0.290.12yes
Pneumonia−0.45−0.23yes
Pneumothorax−0.10−0.12yes
Support Devices−0.16−0.05yes

EIF and diagnostic performance. Does PEER-MTL reduce EIF itself, compared with TB-only, Vanilla-MTL and TAG-MTL What are the TB AUC, sensitivity and specificity of these models on original and compressed images?

PEER-MTL attains the lowest EIF on Vit-B/14 and Vit-B/16 backbones, with the ordering PEER-MTL < Vanilla-MTL < TAG-MTL < TB-only identical across them. Since EIF is an absolute quantity, we also report the scale-free ratio ‖V‖/‖A‖, which yields the same ordering. PEER-MTL reduces ‖V‖/‖A‖ by 2.3–2.4× over Vanilla-MTL, 2.7–4.0× over TAG-MTL and 5.5–9.6× over TB-only.

On ViT-B/14 and ViT-B/16 PEER-MTL attains both the lowest explanation instability and the highest diagnostic performance.

ViT-B/16 ModelEIF‖V‖/‖A‖ConditionAUCAccSensSpec
TB-only2.07e+000.5875original0.7120.6850.7880.632
compressed0.6970.6610.7570.612
Vanilla-MTL1.07e−010.1421original0.9150.8880.8470.909
compressed0.9020.8780.8380.898
TAG-MTL3.47e−010.2437original0.8430.8230.7990.836
compressed0.8190.8020.7750.816
PEER-MTL (ours)1.25e−020.0611original0.9890.9540.9070.978
compressed0.9890.9480.8890.978
ViT-B/14 ModelEIF‖V‖/‖A‖ConditionAUCAccSensSpec
TB-only8.65e+000.6437original0.7230.6940.7890.646
compressed0.6810.6490.7420.602
Vanilla-MTL3.65e−010.2875original0.9210.8910.8510.911
compressed0.9100.8810.8420.901
TAG-MTL7.65e−010.3219original0.8330.8080.7870.819
compressed0.8020.7920.7580.809
PEER-MTL (ours)5.49e−020.1181original0.9890.9620.9210.982
compressed0.9860.9530.9180.972

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.