- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies multi-source soft prompt transfer and proposes COPT, a common-knowledge-guided orthogonal prompt transfer method. Assuming that each source prompt decomposes as with weighted-centered task-specific offsets (), the authors show that minimizing the squared Frobenius reconstruction error \left\\|C - C_0\right\\|_F^2 is equivalent to minimizing the weighted distance to the source prompts, so that under a rank constraint the truncated SVD of the mean source prompt is optimal (via the Eckart-Young-Mirsky theorem). Building on this, COPT (i) extracts a low-rank common prompt by truncated SVD of the mean source prompt, (ii) learns task-specific matrices that reconstruct each source prompt as , while projecting their gradients onto the orthogonal complement of a shared subspace spanned by the leading left singular vectors of , i.e., , and (iii) combines the task-specific matrices through input-dependent attention with a trainable target embedding to adapt to the target task. The method is evaluated on GLUE and SuperGLUE with T5 backbones.
Critical strengths
-
The paper points out that existing multi-source prompt transfer methods extract common knowledge through different empirical procedures (e.g., matrix concatenation, low-rank reconstruction, and distillation), without a common criterion for comparing them. The attempt to unify these choices under a single reconstruction-risk criterion is a useful perspective for the field. The view that task-specific knowledge may overlap with the common-knowledge space, and the attempt to quantify this through the leakage ratio , are also interesting.
-
COPT only adds a single SVD of the mean prompt and a gradient projection, so it is easy to implement. It also keeps the number of trainable parameters at 77K per target task, on par with existing prompt transfer methods. The time-complexity analysis as the number of source prompts grows, together with the measured running time and memory, is also valuable from a practical perspective.
Critical weaknesses
-
The theoretical contribution is limited. For a finite set of sources, the weighted centering assumption always holds if is defined as the weighted mean, so the conclusion that "the mean is the optimal common prompt" is already built into the assumption. Theorems 1 and 2 are also direct applications of a standard decomposition and the Eckart-Young-Mirsky theorem.
-
The leakage analysis assumes an additive residual, whereas the actual method uses a multiplicative structure, , so the analysis does not match the method.
-
There are serious inconsistencies among the reported numbers. The GLUE average of COPT is 87.0 in Table 2 but 86.44 in the ablation (Figure 3a). Similarly, MNLI is 87.8 vs. 85.85 (Figure 3b), and RTE is 83.9 vs. 79.86 (Figure 4a). In addition, the baseline numbers are taken from prior papers, while PCT as reproduced by the authors achieves RTE 74.75 and a GLUE average of 80.49, which are much lower than the cited numbers (82.3 and 86.2). Therefore, it is difficult to regard the improvements over PCT as a comparison under identical conditions.
-
The novelty is limited, and the discussion of related work is insufficient. The remaining pipeline of COPT (Hadamard reconstruction and attention-based fusion) follows PCT. The key distinguishing component, projecting gradients onto the orthogonal complement of a subspace, has already been studied in the context of prompt tuning [1, 2]. Although these works address continual learning, they share the core mechanism of COPT in that they project prompt gradients onto directions orthogonal to an SVD-based subspace so as not to interfere with shared knowledge. The differences from these works should therefore be discussed.
[1] J. Qiao, Z. Zhang, X. Tan, C. Chen, Y. Qu, Y. Peng, and Y. Xie. Prompt Gradient Projection for Continual Learning. In ICLR, 2024. [2] Y. Lu, S. Zhang, D. Cheng, Y. Xing, N. Wang, P. Wang, and Y. Zhang. Visual Prompt Tuning in Null Space for Continual Learning. In NeurIPS, 2024.
Questions for authors
-
Why do the reported numbers for COPT differ between Table 2 and Figures 3 and 4? The GLUE average is 87.0 vs. 86.44, MNLI is 87.8 vs. 85.85, and RTE is 83.9 vs. 79.86. Please clarify under which setting (source-task composition, seed, and hyperparameters) each result was obtained.
-
Please report the results of PCT rerun with the same codebase and setting as COPT, together with the mean and standard deviation over multiple seeds.
-
The main text describes an SGD-style update, but according to Appendix A.4, Adam is used. Did you verify whether the actual update, after Adam's coordinate-wise normalization, remains orthogonal to the common subspace? Also, how is initialized? Since the gradient projection only fixes the common-direction component at its initial value, I am also curious about the effect of the initialization on the leakage ratio.
-
The leakage analysis assumes an additive residual, whereas the actual method uses a multiplicative structure, . Could you explain what restricting to the orthogonal complement of the common subspace means at the level of the reconstructed prompt under this multiplicative structure?
-
Compared with existing work, what is the distinguishing aspect of the orthogonal projection in COPT?