MOIRE: Objective-Specific Adaptation and Residual Readout for Multi-Objective Self-Supervised Learning

Multi-objective training in self-supervised learning (SSL) combines several pre-training objectives to learn representations for downstream tasks. However, training these objectives together requires coordinating their losses through a shared backbone, where updates for one objective can impair predictions for others. We therefore introduce two complementary diagnostic protocols to quantify cross-objective damage under isolated and coordinated updates, and use them to examine existing loss-weighting and gradient-coordination methods from multi-task learning. Our measurements show that these methods only modestly reduce cross-objective damage compared with a fixed-weight sum of losses. Moreover, even effective coordination of self-supervised objectives need not improve downstream performance, because pretraining objectives may emphasize different information from what downstream tasks and evaluation metrics reward. To address both challenges, we propose MOIRE (Multi-Objective Interaction and Residual Evidence). MOIRE gives each objective a separate adaptation path, reducing interference with the shared representation. It retains the resulting objective-specific changes as residual evidence. This evidence can then be combined with the shared representation to better align the learned representation with downstream tasks. Mechanistic experiments show that MOIRE has the lowest relative cross-objective damage among the compared methods. Its downstream performance is similarly strong: across eight datasets spanning graph representation learning and sequential recommendation, it achieves the best average rank in both domains.