Rethinking Multi-Source Prompt Transfer: A Common-Knowledge-Guided Orthogonal Adaptation Strategy

Multi-source soft prompt transfer reuses prompts learned on existing tasks to provide shared knowledge and complementary information for a target task. However, the optimality of common-knowledge extraction and the relationship between common and task-specific knowledge remain insufficiently understood. We revisit this process from the perspectives of parameter reconstruction risk and optimization geometry and propose COPT, a common-knowledge-guided orthogonal prompt transfer method. Under assumptions of knowledge representability, weighted centering of task-specific offsets, and squared Frobenius reconstruction risk, we show that a truncated singular value decomposition of the mean source prompt yields an optimal common prompt subject to a rank constraint. We then construct a shared subspace from this common prompt and project gradients onto its orthogonal complement during source-specific knowledge learning, suppressing gradient leakage into shared directions. Finally, we utilize attention mechanisms to integrate general and domain-specific knowledge, enabling adaptation to target tasks. With 77K trainable parameters per target task on T5-Base, COPT achieves average scores of 87.00 on GLUE and 78.14 on SuperGLUE, outperforms the SOTA baseline. Controlled preliminary experiments, ablations, and analyses of hyperparameters and model scale further support the effectiveness of common-knowledge extraction and source-side orthogonal constraints.