acceptodds
Withdrawn from ICLR 2027

Rethinking Multi-Source Prompt Transfer: A Common-Knowledge-Guided Orthogonal Adaptation Strategy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-source soft prompt transfer reuses prompts learned on existing tasks to provide shared knowledge and complementary information for a target task. However, the optimality of common-knowledge extraction and the relationship between common and task-specific knowledge remain insufficiently understood. We revisit this process from the perspectives of parameter reconstruction risk and optimization geometry and propose COPT, a common-knowledge-guided orthogonal prompt transfer method. Under assumptions of knowledge representability, weighted centering of task-specific offsets, and squared Frobenius reconstruction risk, we show that a truncated singular value decomposition of the mean source prompt yields an optimal common prompt subject to a rank constraint. We then construct a shared subspace from this common prompt and project gradients onto its orthogonal complement during source-specific knowledge learning, suppressing gradient leakage into shared directions. Finally, we utilize attention mechanisms to integrate general and domain-specific knowledge, enabling adaptation to target tasks. With 77K trainable parameters per target task on T5-Base, COPT achieves average scores of 87.00 on GLUE and 78.14 on SuperGLUE, outperforms the SOTA baseline. Controlled preliminary experiments, ablations, and analyses of hyperparameters and model scale further support the effectiveness of common-knowledge extraction and source-side orthogonal constraints.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-source soft prompt transfer, parameter-efficient adaptation, common knowledge extraction, orthogonal gradient projection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DPEhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies multi-source soft prompt transfer and proposes COPT, a common-knowledge-guided orthogonal prompt transfer method. Assuming that each source prompt decomposes as Pi=C0+TiP_i = C_0 + T_i with weighted-centered task-specific offsets (∑iwiTi=0\sum_{i} w_i T_i = 0), the authors show that minimizing the squared Frobenius reconstruction error \left\\|C - C_0\right\\|_F^2 is equivalent to minimizing the weighted distance to the source prompts, so that under a rank constraint the truncated SVD of the mean source prompt is optimal (via the Eckart-Young-Mirsky theorem). Building on this, COPT (i) extracts a low-rank common prompt C∗C^{\ast} by truncated SVD of the mean source prompt, (ii) learns task-specific matrices MiM_i that reconstruct each source prompt as Mi⊙C∗M_i \odot C^{\ast}, while projecting their gradients onto the orthogonal complement of a shared subspace spanned by the leading left singular vectors BB of C∗C^{\ast}, i.e., (I−BB⊤)Gi(I - BB^{\top})G_i, and (iii) combines the task-specific matrices through input-dependent attention with a trainable target embedding MtM_t to adapt to the target task. The method is evaluated on GLUE and SuperGLUE with T5 backbones.

Critical strengths
  1. The paper points out that existing multi-source prompt transfer methods extract common knowledge through different empirical procedures (e.g., matrix concatenation, low-rank reconstruction, and distillation), without a common criterion for comparing them. The attempt to unify these choices under a single reconstruction-risk criterion is a useful perspective for the field. The view that task-specific knowledge may overlap with the common-knowledge space, and the attempt to quantify this through the leakage ratio ρparam\rho_{param}, are also interesting.

  2. COPT only adds a single SVD of the mean prompt and a gradient projection, so it is easy to implement. It also keeps the number of trainable parameters at 77K per target task, on par with existing prompt transfer methods. The time-complexity analysis as the number of source prompts grows, together with the measured running time and memory, is also valuable from a practical perspective.

Critical weaknesses
  1. The theoretical contribution is limited. For a finite set of sources, the weighted centering assumption always holds if C0C_0 is defined as the weighted mean, so the conclusion that "the mean is the optimal common prompt" is already built into the assumption. Theorems 1 and 2 are also direct applications of a standard decomposition and the Eckart-Young-Mirsky theorem.

  2. The leakage analysis assumes an additive residual, whereas the actual method uses a multiplicative structure, Mi⊙C∗M_i \odot C^{\ast}, so the analysis does not match the method.

  3. There are serious inconsistencies among the reported numbers. The GLUE average of COPT is 87.0 in Table 2 but 86.44 in the ablation (Figure 3a). Similarly, MNLI is 87.8 vs. 85.85 (Figure 3b), and RTE is 83.9 vs. 79.86 (Figure 4a). In addition, the baseline numbers are taken from prior papers, while PCT as reproduced by the authors achieves RTE 74.75 and a GLUE average of 80.49, which are much lower than the cited numbers (82.3 and 86.2). Therefore, it is difficult to regard the improvements over PCT as a comparison under identical conditions.

  4. The novelty is limited, and the discussion of related work is insufficient. The remaining pipeline of COPT (Hadamard reconstruction and attention-based fusion) follows PCT. The key distinguishing component, projecting gradients onto the orthogonal complement of a subspace, has already been studied in the context of prompt tuning [1, 2]. Although these works address continual learning, they share the core mechanism of COPT in that they project prompt gradients onto directions orthogonal to an SVD-based subspace so as not to interfere with shared knowledge. The differences from these works should therefore be discussed.

[1] J. Qiao, Z. Zhang, X. Tan, C. Chen, Y. Qu, Y. Peng, and Y. Xie. Prompt Gradient Projection for Continual Learning. In ICLR, 2024. [2] Y. Lu, S. Zhang, D. Cheng, Y. Xing, N. Wang, P. Wang, and Y. Zhang. Visual Prompt Tuning in Null Space for Continual Learning. In NeurIPS, 2024.

Questions for authors
  1. Why do the reported numbers for COPT differ between Table 2 and Figures 3 and 4? The GLUE average is 87.0 vs. 86.44, MNLI is 87.8 vs. 85.85, and RTE is 83.9 vs. 79.86. Please clarify under which setting (source-task composition, seed, and hyperparameters) each result was obtained.

  2. Please report the results of PCT rerun with the same codebase and setting as COPT, together with the mean and standard deviation over multiple seeds.

  3. The main text describes an SGD-style update, but according to Appendix A.4, Adam is used. Did you verify whether the actual update, after Adam's coordinate-wise normalization, remains orthogonal to the common subspace? Also, how is MiM_i initialized? Since the gradient projection only fixes the common-direction component at its initial value, I am also curious about the effect of the initialization on the leakage ratio.

  4. The leakage analysis assumes an additive residual, whereas the actual method uses a multiplicative structure, Mi⊙C∗M_i \odot C^{\ast}. Could you explain what restricting MiM_i to the orthogonal complement of the common subspace means at the level of the reconstructed prompt under this multiplicative structure?

  5. Compared with existing work, what is the distinguishing aspect of the orthogonal projection in COPT?

Reviewer_zUYuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COPT, a parameter-efficient approach to multi-source soft prompt transfer that addresses two limitations of existing methods: the lack of theoretical justification for extracting shared knowledge and the potential overlap between shared and task-specific representations.

Critical strengths

Principled formulation of common-knowledge extraction and orthogonal adaptation. The paper provides a theoretically grounded perspective on multi-source prompt transfer by establishing the optimality of truncated SVD under explicit reconstruction assumptions and introducing orthogonal gradient projection to reduce interference between shared and task-specific knowledge. This is consequential because it offers a coherent, interpretable framework for addressing knowledge overlap rather than relying solely on empirical prompt-combination strategies.

Critical weaknesses

Limited theoretical guarantees and insufficient justification of the proposed orthogonality mechanism. The theoretical optimality of common-knowledge extraction relies on restrictive assumptions, particularly the exact weighted centering of task-specific offsets, without establishing their validity in practical settings. More importantly, projecting gradients onto the orthogonal complement does not necessarily ensure orthogonality of the learned representations, especially when using adaptive optimizers and element-wise prompt transformations. This weakens the theoretical justification for the paper's central mechanism and its claimed ability to prevent knowledge interference.

Questions for authors
  1. The optimality of truncated SVD relies on the exact weighted-centering assumption for task-specific offsets. To what extent does this assumption hold for real source prompts, and how sensitive is COPT's performance when it is violated? Evidence addressing this would strengthen confidence in the practical relevance of the theoretical contribution.

  2. Since COPT applies gradient projection before optimization and constructs source-specific representations through element-wise multiplication with the common prompt, how does the method ensure that the resulting representations remain orthogonal to the shared subspace? Could the authors clarify the behavior under Adam and provide empirical evidence of orthogonality at the representation level?

Reviewer_44jUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COPT, a multi-source soft prompt transfer method that aims to improve common-knowledge extraction and reduce redundancy between common and task-specific knowledge. COPT applies SVD to the average of source prompts to extract common knowledge, introduces orthogonal gradient projection, and uses attention-based fusion for target task adaptation. Experiments demonstrate improvements over existing prompt transfer and prompt-based methods.

Critical strengths
  1. The paper addresses a problem in multi-source prompt transfer, where reusing information from multiple source tasks is challenging.
  2. The proposed framework is simple and parameter-efficient, fitting the design creed of parameter-efficient transfer learning.
  3. The experiments is thorough and demonstrate improvements on GLUE and SuperGLUE.
Critical weaknesses
  1. The paper does not clearly establish why overlap between common and task-specific knowledge actually harms transfer performance.

  2. The assumption that task-specific components sum to zero essentially assumes that common knowledge equals the weighted average of source prompts. However, there is little justified. Why this average can represent genuinely shared knowledge? Moreover, optimality under Frobenius reconstruction error does not necessarily imply better transferability.

  3. The orthogonal constraint is applied to gradients, which does not guarantee that the learned task-specific representations are orthogonal to common knowledge, especially given the use of Adam and Hadamard multiplication.

  4. The experiments do not clearly show whether the gains come from the claimed knowledge separation or simply from regularization. Comparisons with simpler alternatives, such as random-subspace projection, can be added to strengthen the results.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.