- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual LLM adaptation with joint backbone and soft-prompt updates. It realigns historical prompts through self-distillation, learns a Bezier simplex with fixed task prompts as vertices, and searches this space using task-specific validation scores. On eight TRACE tasks, it achieves the highest reported average score across three Qwen3 scales and improves backward transfer over existing baselines.
Critical strengths
- This paper addresses a practical difficulty of updating the backbone while retaining task-specific prompts.
- The evaluation includes component ablations, a convex-interpolation baseline, combinations with replay methods, and a reverse task order. These provide useful evidence beyond the main comparison.
Critical weaknesses
-
In Eq. 6, freely learned control prompts receive replay supervision, including from the target task. Interior coordinates and the PCA plots therefore do not establish which source-task knowledge is transferred. Table 5 changes parameter capacity along with the geometry, while Table 3 changes the optimization and selection procedure. A direct replay-based prompt update or a simple shared residual parameterization, with matched capacity and search budgets, would help establish what the Bezier structure contributes.
-
Section 4.1 averages eight inference runs, which does not measure variation from training, replay sampling, or independent BO runs. This matters for the 0.2-point average improvement over ProMoT on Qwen3-8B in Table 1. The reverse-order experiment is useful, but independent training repeats and uncertainty estimates are still needed to assess the consistency of the advantage.
-
Assumption 1 defines β and M for derivatives with respect to the control prompts C. Eqs. 19-20 then introduce additional Jacobian factors as if these constants bounded derivatives with respect to the constructed prompt Z. That step does not follow from Eqs. 10-11, so the explicit R_t,k-dependent bound needs correction. Defining the regularity assumptions in Z-space may repair the argument. This issue concerns the stated bound, rather than evidence that the empirical method fails.
Questions for authors
See weaknesses
3 comments
Dear Reviewer,
As the discussion period ends soon, we would be grateful if you could take another look at our responses and additional experimental results.
Thank you for your time and consideration.
Best regards,
The Authors
Dear Reviewer aFiQ,
We humbly request for your engagement during this discussion period. We do not have a lot of time left before this discussion period ends. Let us know if you have any further doubts that we need to explanation.
The Authors
Response
We sincerely thank the reviewer for the insightful suggestions, which have helped us improve our work.
(1) Cross-Task Knowledge Transfer.
Although the shared control prompts receive replay supervision from all historical tasks, including the target task, the selected coordinates still provide insights into potentially beneficial cross-task interactions. Specifically, Table 9 in the Appendix reports the final selected coordinates for each task. When a target task assigns substantial weight to another task's direction, it suggests that the corresponding task supervision may provide useful information for improving the target task through the shared prompt space. Conversely, configurations concentrated near the target task's own anchor suggest limited benefits from cross-task composition.
We agree that these coordinates do not directly reveal the specific knowledge transferred between tasks. A more precise attribution would require disentangling task-specific knowledge within the model's representations and establishing causal links between these representations and downstream performance, which remains a challenging problem in representation interpretability.
(2) Bézier vs. Shared Residual Parameterization.
Following the reviewer's suggestion, we additionally implemented a shared residual baseline using a fixed linear projection to combine learnable residual prompts. Both methods use the same historical task anchors, replay-based space-learning objective, and Bayesian search, with the same validation-search budgets.
The Bézier parameterization improves Average by 1.9% and 2.1%, and BWT by 0.5 and 1.7, on Qwen3-1.7B and 4B, respectively. These results indicate that LAPS's improvements are not simply due to adding learnable residual prompts, but also benefit from the structured Bézier mapping between task preferences and the learned prompt space.
We conducted three independent training runs, with each run evaluated eight times. We additionally equipped ProMoT with the same validation-search procedure. The results are reported below.
LAPS consistently outperforms ProMoT across all three model scales in both Average and BWT. These results provide additional evidence of the consistency improvements of LAPS.
We agree that Assumption 1 should impose the gradient bound and smoothness condition with respect to the generated prompt Z, rather than directly on the control parameters C. We will revise the assumption as follows.
For all tasks i and feasible prompts Z,Z′, we assume
∥∇Zℓi(Z)∥≤M,∥∇Zℓi(Z)−∇Zℓi(Z′)∥≤β∥Z−Z′∥.
Since the Bézier mapping Z(λ;C) is affine in C, its Jacobian is independent of C and satisfies
= \rho_{t,k}(\boldsymbol{\lambda}).$$ Applying the chain rule yields $$\lVert \nabla_{\mathbf{C}}\ell_i \rVert \le M\rho_{t,k}(\boldsymbol{\lambda}), \qquad \lVert \nabla_{\mathbf{C}}^2\ell_i \rVert_{\mathrm{op}} \le \beta\rho_{t,k}^2(\boldsymbol{\lambda}).$$ The Hessian bound holds wherever the Hessian exists; the corresponding smoothness bound also follows directly from gradient Lipschitz continuity. Consequently, the explicit smoothness bound in Lemma 2 retains the same form under the corrected assumption, preserving the convergence rate in Theorem 1. We will revise Assumption 1 and the corresponding derivation accordingly. We have incorporated the main additional experiments and analyses into the revised manuscript, with all changes highlighted in blue. We sincerely thank the reviewer for these valuable suggestions.