ISC-LoRA: Intrinsic Spectral Continuation for Multitask Dense Prediction

Multitask learning with low-rank adaptation (LoRA) efficiently adapts a shared visual model to complementary dense prediction tasks, including semantic segmentation, boundary detection, and geometry estimation. Reweighting the visual features based on their spatial distribution adaptive to the specific task and inputs can further improve these predictions. To do this, multitask LoRA methods often separate low-rank updates into shared and task-private branches. The low-dimensional features in each private branch retain the spatial grid of the current input and can reflect task-specific spatial variation. However, the spatial context does not directly indicate which frequencies should be emphasized or how strongly, limiting the adaptation capability to the specific input or task. Building on this structure, we introduce Intrinsic Spectral Continuation LoRA (ISC-LoRA), a subspace-conditioned training scheme that derives a bounded response over the full spatial-frequency grid from each private branch's current features and subspace. This response is recomputed for each input and private branch, adapting the emphasis of spatial frequencies to each task and image without predefined frequency bands. To make the response intrinsic to this subspace rather than to the particular LoRA rank coordinates used to represent it, our method combines frequency preferences along the principal directions of feature variation using geometry-aware regularization and covariance normalization. We further prove that ISC-LoRA can reweight the spatial frequencies of the private output within a controlled range, with the response determined by the current input and task-private subspace, independent of the chosen LoRA rank basis. Notably, its continuation schedule reduces the modulation strength to zero, making the spectral operator the identity while retaining the learned LoRA factors. The resulting adapters can therefore be evaluated and deployed using standard LoRA computation, without spectral operations. Experiments on PASCAL-Context and NYUD-v2 show that ISC-LoRA outperforms representative multitask adaptation methods. Beyond natural images, evaluations on ten two-dimensional Cell Tracking Challenge (CTC) datasets further demonstrate its advantages over representative adaptation baselines in cell segmentation, detection, and tracking. Our code and checkpoints are available at https://anonymous.4open.science/r/ISC-LoRA.