Keep Your Peers: Task Selection for Robust Explanations in Multi-Task Learning

Reliable deployment of large vision-language models (LVLMs) requires explanations that remain stable under realistic input perturbations, not merely accurate predictions. Yet multi-task learning (MTL) and auxiliary-task selection have been studied predominantly through predictive objectives, with comparatively little attention to explainability robustness. This raises a fundamental question: which auxiliary tasks should be selected to enhance the robustness of explanations for a primary task? This work addresses the question by formulating an explanation instability functional and analyzing its first variation under changes in auxiliarytask participation. The resulting variational analysis is generic and attributionagnostic: it operates at the level of the explanation functional and can be applied with any differentiable attribution map, rather than being tied to a particular explanation method, model architecture, or application. Motivated by this analysis, we introduce PEER(Perturbation Explanation Error Relation), a criterion based on the interaction between perturbation-induced explanation errors of the primary and auxiliary tasks. Under suitable regularity conditions, this relation identifies auxiliary tasks whose retention enhances the robustness of primary-task explanations. The framework is realized in LLaVA-PEER, a LLaVa model trained with PEER-MTL, for chest X-ray analysis in resource-constrained mHealth settings, where severe image compression provides a realistic source of perturbation. The study also releases a radiologist-annotated free-text reporting dataset for publicly available tuberculosis CXR benchmarks. Experiments show that PEERMTL reduces explanation instability and yields substantially more reliable LVLMgenerated reports.