MIST: Minimal Intervention Selective Token Unlearning for Co-Occurring Disease Recognition in Multi-Label Chest X-Rays

Selective machine unlearning removes the influence of a target concept from a trained model without retraining, and is increasingly needed in medical imaging for privacy requests, annotation errors and evolving clinical standards. In multi-label chest X-ray recognition, the task is unusually hard because diseases co-occur and share visual evidence. We observe that existing methods sit at the two ends of a forgetting–retention trade-off: gradient ascent forgets the target but damages retained diseases, whereas distillation-based unlearning preserves retained diseases but barely forgets. We trace this to three causes: (i) disease evidence is entangled in shared representations, (ii) current methods act on parameters or global features and cannot localise what to forget, and (iii) nothing in their objectives protects the evidence that co-occurring diseases share. To address the above causes, we propose MIST, a novel token-level framework for Vision Transformers that is composed of three key components: (1) Disease-specific Token Selection (DTS) uses gradient-based attribution to find tokens that are informative for the target disease and not for retained diseases; (2) Selective Unlearning (SU) suppresses the target only through these tokens; and (3) Co-occurrence-Aware Retention (CAR) combines teacher–student distillation with a shared-token preservation loss on images where target and retained diseases appear together. On NIH ChestX-ray14 with patient-level five-fold cross-validation, MIST lowers Effusion AUROC from 0.870 to 0.403 while retained-disease AUROC falls by only 0.007 and co-occurrence AUROC by 0.008, which is 65% and 69% less collateral degradation than SalUn at slightly stronger forgetting, and it modifies 37% of patch tokens. Our results demonstrate that MIST is the only method ranked in the top two on all six forgetting and retention measures.