acceptodds
Under review as a conference paper at ICLR 2027

Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models

Abstract

Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Jointly adapting model parameters and task-specific soft prompts offers a promising solution, but faces two key limitations: historical prompts may become less effective as the model evolves, while their transferable cross-task relationships are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. LAPS first aligns historical prompts with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable B\'ezier control prompts. Once this anchored prompt space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the optimized prompt to draw on knowledge acquired from observed tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms distillation-based, prompt-based, and joint prompt–parameter adaptation baselines, improving average performance while reducing forgetting. Our implementation is available at https://anonymous.4open.science/r/LAPS_.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Adaptation, Soft Prompt Alignment, Cross-Task Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aFiQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual LLM adaptation with joint backbone and soft-prompt updates. It realigns historical prompts through self-distillation, learns a Bezier simplex with fixed task prompts as vertices, and searches this space using task-specific validation scores. On eight TRACE tasks, it achieves the highest reported average score across three Qwen3 scales and improves backward transfer over existing baselines.

Critical strengths
  1. This paper addresses a practical difficulty of updating the backbone while retaining task-specific prompts.
  2. The evaluation includes component ablations, a convex-interpolation baseline, combinations with replay methods, and a reverse task order. These provide useful evidence beyond the main comparison.
Critical weaknesses
  1. In Eq. 6, freely learned control prompts receive replay supervision, including from the target task. Interior coordinates and the PCA plots therefore do not establish which source-task knowledge is transferred. Table 5 changes parameter capacity along with the geometry, while Table 3 changes the optimization and selection procedure. A direct replay-based prompt update or a simple shared residual parameterization, with matched capacity and search budgets, would help establish what the Bezier structure contributes.

  2. Section 4.1 averages eight inference runs, which does not measure variation from training, replay sampling, or independent BO runs. This matters for the 0.2-point average improvement over ProMoT on Qwen3-8B in Table 1. The reverse-order experiment is useful, but independent training repeats and uncertainty estimates are still needed to assess the consistency of the advantage.

  3. Assumption 1 defines β and M for derivatives with respect to the control prompts C. Eqs. 19-20 then introduce additional Jacobian factors as if these constants bounded derivatives with respect to the constructed prompt Z. That step does not follow from Eqs. 10-11, so the explicit R_t,k-dependent bound needs correction. Defining the regularity assumptions in Z-space may repair the argument. This issue concerns the stated bound, rather than evidence that the empirical method fails.

Questions for authors

See weaknesses

3 comments

We sincerely thank the reviewer for the insightful suggestions, which have helped us improve our work.

Weakness 1: Cross-Task Transfer and Bézier Parameterization

(1) Cross-Task Knowledge Transfer.

Although the shared control prompts receive replay supervision from all historical tasks, including the target task, the selected coordinates still provide insights into potentially beneficial cross-task interactions. Specifically, Table 9 in the Appendix reports the final selected coordinates for each task. When a target task assigns substantial weight to another task's direction, it suggests that the corresponding task supervision may provide useful information for improving the target task through the shared prompt space. Conversely, configurations concentrated near the target task's own anchor suggest limited benefits from cross-task composition.

We agree that these coordinates do not directly reveal the specific knowledge transferred between tasks. A more precise attribution would require disentangling task-specific knowledge within the model's representations and establishing causal links between these representations and downstream performance, which remains a challenging problem in representation interpretability.

(2) Bézier vs. Shared Residual Parameterization.

Following the reviewer's suggestion, we additionally implemented a shared residual baseline using a fixed linear projection to combine learnable residual prompts. Both methods use the same historical task anchors, replay-based space-learning objective, and Bayesian search, with the same validation-search budgets.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWT
Linear-Projection Residual58.9 ± 0.09−1.6 ± 0.1364.1 ± 0.07−1.6 ± 0.14
Bézier Residual60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.07+0.1 ± 0.11

The Bézier parameterization improves Average by 1.9% and 2.1%, and BWT by 0.5 and 1.7, on Qwen3-1.7B and 4B, respectively. These results indicate that LAPS's improvements are not simply due to adding learnable residual prompts, but also benefit from the structured Bézier mapping between task preferences and the learned prompt space.

Weakness 2: Statistical Robustness

We conducted three independent training runs, with each run evaluated eight times. We additionally equipped ProMoT with the same validation-search procedure. The results are reported below.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
ProMoT58.1 ± 0.40-4.6 ± 0.3763.2 ± 0.21-2.1 ± 0.2467.0 ± 0.10+0.1 ± 0.13
LAPS (Ours)60.8 ± 0.12-1.1 ± 0.2066.2 ± 0.15+0.1 ± 0.0867.9 ± 0.09+1.6 ± 0.12

LAPS consistently outperforms ProMoT across all three model scales in both Average and BWT. These results provide additional evidence of the consistency improvements of LAPS.

Weakness 3: Correction of Smoothness Assumptions

We agree that Assumption 1 should impose the gradient bound and smoothness condition with respect to the generated prompt Z\mathbf{Z}, rather than directly on the control parameters C\mathbf{C}. We will revise the assumption as follows.

For all tasks ii and feasible prompts Z,Z′\mathbf{Z},\mathbf{Z}', we assume

∥∇Zℓi(Z)∥≤M,∥∇Zℓi(Z)−∇Zℓi(Z′)∥≤β∥Z−Z′∥.\left\Vert\nabla_{\mathbf{Z}}\ell_i(\mathbf{Z})\right\Vert \le M,\qquad \left\Vert\nabla_{\mathbf{Z}}\ell_i(\mathbf{Z})-\nabla_{\mathbf{Z}}\ell_i(\mathbf{Z}')\right\Vert \le \beta\left\Vert\mathbf{Z}-\mathbf{Z}'\right\Vert.

Since the Bézier mapping Z(λ;C)\mathbf{Z}(\boldsymbol{\lambda};\mathbf{C}) is affine in C\mathbf{C}, its Jacobian is independent of C\mathbf{C} and satisfies

= \rho_{t,k}(\boldsymbol{\lambda}).$$ Applying the chain rule yields $$\lVert \nabla_{\mathbf{C}}\ell_i \rVert \le M\rho_{t,k}(\boldsymbol{\lambda}), \qquad \lVert \nabla_{\mathbf{C}}^2\ell_i \rVert_{\mathrm{op}} \le \beta\rho_{t,k}^2(\boldsymbol{\lambda}).$$ The Hessian bound holds wherever the Hessian exists; the corresponding smoothness bound also follows directly from gradient Lipschitz continuity. Consequently, the explicit smoothness bound in Lemma 2 retains the same form under the corrected assumption, preserving the convergence rate in Theorem 1. We will revise Assumption 1 and the corresponding derivation accordingly. We have incorporated the main additional experiments and analyses into the revised manuscript, with all changes highlighted in blue. We sincerely thank the reviewer for these valuable suggestions.

Dear Reviewer,

As the discussion period ends soon, we would be grateful if you could take another look at our responses and additional experimental results.

Thank you for your time and consideration.

Best regards,
The Authors

Dear Reviewer aFiQ,

We humbly request for your engagement during this discussion period. We do not have a lot of time left before this discussion period ends. Let us know if you have any further doubts that we need to explanation.

The Authors

Reviewer_nBfBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies continual adaptation of large language models with joint updates of model parameters and task soft prompts. The proposed method LAPS has three steps. It first aligns historical prompts to the updated model with self-distillation. It then treats task prompts as vertices and builds an anchored prompt space with a degree-k Bezier simplex and learnable control prompts. Finally it selects a prompt for each task on the validation set with Bayesian optimization. Experiments on the eight TRACE tasks and three Qwen3 scales show better average performance and BWT than ProMoT and other baselines.

Critical strengths

The problem is real. Historical prompts do become less effective as the model changes.

The experiments cover three model scales and include baselines, ablations and sensitivity checks.

Alignment plus prompt space composition gives a stable BWT improvement.

Critical weaknesses

The main gain may come from validation search rather than from the method itself.

The comparison with baselines is not fully fair because only LAPS gets the final per-task search.

The method is more complex than the evidence for it.

Questions for authors

Without the final validation search, how much gain remains over ProMoT?

Do baselines also receive a per-task validation search in the comparison?

Does the main ProMoT baseline use replay? Can the replay variant be added to the main table?

How does the number of control prompts grow with tasks, and does the method remain stable at longer task sequences?

4 comments
AuthorsOpenReview ↗

Response (Part I)

We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback, which have helped us strengthen our experimental validation and clarify our contributions.

Weakness 1 and Question 1: Validation-Search Fairness

To disentangle the benefits of space learning from those of validation search, we additionally equipped ProMoT and Progprompt with the same per-task Bayesian search procedure and validation-evaluation budget. The reported results are averaged over three independent runs, with each run evaluated eight times.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
ProMoT58.1 ± 0.40-4.6 ± 0.3763.2 ± 0.21-2.1 ± 0.2467.0 ± 0.10+0.1 ± 0.13
ProMoT + validation-guided search58.7 ± 0.08-3.5 ± 0.1064.9 ± 0.17-1.2 ± 0.1367.4 ± 0.11+0.3 ± 0.14
ProgPrompt52.1 ± 0.29N/A58.2 ± 0.43N/A58.1 ± 0.24N/A
ProgPrompt + validation-guided search53.2 ± 0.19N/A59.6 ± 0.31N/A60.6 ± 0.22N/A
LAPS60.8 ± 0.12-1.1 ± 0.2066.2 ± 0.15+0.1 ± 0.0867.9 ± 0.09+1.6 ± 0.12

Compared with ProMoT and ProgPrompt equipped with validation-guided search, LAPS achieves higher Average performance and better BWT across all three model scales. These results demonstrate that its improvements cannot be attributed solely to additional validation-guided search, but also reflect the benefits of the historical prompt alignment.

Explanation. This difference stems from the prompt spaces being searched, rather than the search budget itself. For ProMoT, Bayesian optimization is performed over convex combinations of fixed historical prompts, which may be misaligned with the final updated backbone. In contrast, LAPS first realigns historical prompts and then learns a task-performance-aware Bézier space through multi-task replay supervision. Consequently, Bayesian search in LAPS explores learned intermediate configurations that capture cross-task interactions, rather than being restricted to combinations of potentially misaligned historical prompts.

To disentangle the benefits of historical prompt alignment from those of final prompt selection, we report the results using only the aligned task anchors (without Bayesian search), averaged over three independent training runs.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
ProMoT58.1 ± 0.40-4.6 ± 0.3763.2 ± 0.21-2.1 ± 0.2467.0 ± 0.10+0.1 ± 0.13
LAPS (Anchors Only)59.8 ± 0.16-2.1 + ± 0.1465.8 ± 0.16-0.7 ± 0.1767.1 ± 0.13+1.4 ± 0.10

These results show that historical prompt alignment improves retention before any final validation search, particularly on Qwen3-1.7B and 4B. This further demonstrates that the benefits of LAPS are not solely introduced by the Bayesian search.

Question 2: Baselines with Replay

We have already evaluated ProMoT + Replay and Progprompt + Replay in Table 3 of the manuscript, where the baseline is also given access to the same replay data. The results are as follows:

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWT
ProgPrompt + Replay50.8N/A55.1N/A
ProMoT + Replay54.2-6.963.9-2.8
LAPS (Anchors Only)59.8-2.165.8-0.7
LAPS (Bayesian search)60.8-1.166.2+0.1

The results show that simply incorporating replay data into task adaptation does not guarantee improved performance or retention. Although replay is intended to mitigate forgetting, directly mixing historical samples into backbone adaptation may introduce interference with learning the current task. Instead, LAPS uses replay data differently: rather than directly using them for backbone task adaptation, it leverages replay supervision to realign historical prompts and learn the task-performance-aware prompt space. This allows LAPS to preserve and reorganize historical prompt knowledge without relying on direct replay-based backbone adaptation.

AuthorsOpenReview ↗

Response (Part II)

Weakness 3: The Method Is More Complex Than the Evidence for It

We clarify that LAPS is designed as a unified framework to address three complementary challenges in continual fine-tuning: historical prompt alignment, task-performance-aware prompt-space learning, and task-specific prompt selection. To justify these design choices, we conducted additional comparisons against simpler alternatives, as described below.

(1) On-Policy Prompt Alignment.

We replaced OPSD with Teacher-Rollout SFT, while keeping the remaining LAPS components unchanged, to examine whether supervised realignment provides comparable benefits.

Alignment MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWT
Teacher-Rollout SFT59.2 ± 0.08−2.4 ± 0.1765.1 ± 0.08−0.9 ± 0.11
OPSD (LAPS)60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.07+0.1 ± 0.11

OPSD improves Average by 1.6% and 1.1% and BWT by 1.3 and 1.0 on Qwen3-1.7B and 4B, respectively, supporting the benefit of aligning historical prompts on student-generated trajectories.

(2) Bézier Prompt-Space Parameterization.

To isolate the contribution of the Bézier structure, we implemented a shared residual baseline using a fixed linear projection to combine learnable residual prompts. Both methods use the same historical task anchors, replay-based space-learning objective, and Bayesian search.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWT
Linear-Projection Residual58.9 ± 0.09−1.6 ± 0.1364.1 ± 0.07−1.6 ± 0.14
Bézier Residual60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.07+0.1 ± 0.11

The Bézier parameterization improves Average by approximately 1.9% and 2.1% on Qwen3-1.7B and 4B, respectively, while also improving BWT. This indicates that the gains are not merely due to introducing learnable residual prompts, but also benefit from the structured Bézier mapping.

(3) Validation-Guided Prompt Selection.

To examine whether Bayesian search is necessary, we compared Bayesian search with a simple normalized-coordinate grid search on the same learned prompt space, under the same validation-evaluation budget.

Search StrategyQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
Grid Search59.5 ± 0.18−2.2 ± 0.1065.7 ± 0.17−0.8 ± 0.1367.21 ± 0.17+0.8 ± 0.10
Bayesian Search60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.17+0.1 ± 0.1167.9 ± 0.18+1.6 ± 0.10

Bayesian search consistently improves over grid search across all three model scales. These additional experiments provide complementary evidence for the three design choices in LAPS. Each component addresses a distinct challenge, and its contribution is supported by comparisons against simpler alternatives.

Question 3: Scalability and Stability on Longer Task Sequences

(1) Control-Prompt Scalability.

For the cubic Bézier construction, the number of control prompts grows as (t+23)−t=O(t3).\binom{t+2}{3}-t=O(t^3).
Despite this cubic growth, the additional parameter storage remains relatively small for the evaluated task scales. For example, on Qwen3-1.7B, the control prompts require approximately 30 MiB for 8 tasks and 204 MiB for 16 tasks. Moreover, the control prompts are only used during offline space learning, while each task uses a single fixed-length prompt at inference time. For substantially longer sequences (e.g., hundreds of tasks), we agree that the dense construction would become increasingly expensive. As discussed in Section 5 of the manuscript, sparse control-point allocation or task-aware routing could restrict optimization to relevant task interactions and improve scalability.

(2) Stability on Longer Task Sequences.

To directly examine the reviewer's concern, we additionally evaluated LAPS on InstrDialog [1], an independent continual instruction-tuning benchmark with 19 tasks, using Qwen3-1.7B. For a fair comparison, we also equipped the prompt-based baselines, ProgPrompt and ProMoT, with prompt search under the same validation-evaluation budget as LAPS.

MethodAverage ↑BWT ↑
SFT44.46−6.40
SFT + Replay47.67−1.32
ProMoT41.67−6.79
ProgPrompt49.87N/A
OPR-SC47.76−3.51
OPR-RU47.38−2.54
LAPS52.13+1.61

LAPS achieves the highest Average ROUGE-L of 52.13, outperforming the strongest baseline by 2.26%, while achieving a positive BWT of +1.61. These results provide direct empirical evidence that LAPS remains effective on a longer 19-task sequence.

Reference

[1] "CITB: A Benchmark for Continual Instruction Tuning," Findings of ACL: EMNLP 2023.

Dear Reviewer,

As the discussion period is approaching its end, we would be grateful if you could revisit our rebuttal and additional experimental results. We hope these clarifications and new evidence have addressed your concerns.

Thank you again for your time and consideration.

Best regards,
The Authors

Dear Reviewer nBfB,

We humbly request for your engagement during this discussion period. We do not have a lot of time left before this discussion period ends. Let us know if you have any further doubts that we need to explanation.

The Authors

Reviewer_jy33OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LAPS, a framework for continual adaptation of large language models under a task-incremental setting with known task identities. Building on alternating prompt and backbone adaptation, LAPS introduces three components: on-policy self-distillation to align historical prompts with the updated backbone, an anchored Bézier prompt space trained using cross-task replay losses, and task-specific Bayesian optimization over validation performance to select final prompts.

Experiments on eight TRACE tasks with Qwen3-1.7B, 4B, and 8B show the highest average performance among the evaluated methods, with improved backward transfer relative to ProMoT. The paper also investigates component contributions, prompt length, Bézier degree, replay ratios, reversed task order, and compatibility with existing replay methods.

Critical strengths
  1. Clear problem formulation and coherent design: The paper identifies the incompatibility between historical prompts and an evolving backbone as a concrete challenge in joint prompt–parameter adaptation. Updating historical prompts through distillation is a sensible response, and the three components have distinguishable roles: recovering historical behavior, learning intermediate prompt configurations, and selecting task-specific solutions.
  2. An interesting structured prompt parameterization: The anchored Bézier construction preserves existing task prompts at the vertices while introducing learnable intermediate configurations. This provides a structured extension of linear prompt interpolation. Once selection is complete, each task still uses a fixed-length prompt, making the resulting inference procedure straightforward.
  3. Consistent aggregate improvements across the evaluated model scales: Relative to ProMoT, LAPS improves Average by 2.5, 2.8, and 0.2 points, and BWT by 3.2, 1.7, and 1.4 points on the three model sizes. The gains at 1.7B and 4B are meaningful, and the reversed-order experiment provides additional evidence that the relative advantage is not restricted to the default sequence.
  4. A reasonably extensive set of diagnostic experiments: The component ablations, degree and prompt-length studies, replay comparisons, and selected-coordinate analyses help characterize the framework. In particular, the comparison against degree-one interpolation provides evidence that learning the intermediate space is useful. The appendix also describes the Bayesian optimization procedure in substantial detail.
Critical weaknesses
  1. The statistical robustness and evaluation-budget contribution remain insufficiently established: Results are averaged over eight inference runs, which do not measure variability across independent training runs. This matters especially at 8B, where the improvement over ProMoT is only 0.2 Average points. Moreover, final performance incorporates substantial task-specific validation search. The paper would benefit from comparisons with baselines given comparable validation-search budgets. Final BWT also includes post-training prompt selection, so it combines sequential retention with subsequent recovery through search.
  2. The practical efficiency claims are supported mainly by parameter counts: LAPS uses full-parameter fine-tuning and additionally requires teacher evaluations, on-policy generation, control-prompt optimization, and repeated validation evaluation. An additional parameter count below 0.5% does not establish a similarly small computational overhead. Training time, search time, peak memory, and total GPU cost are not sufficiently quantified. Furthermore, the cubic construction requires (t+23)−t\binom{t+2}{3}-t control prompts, making scalability beyond eight tasks an unresolved concern.
  3. The ablations do not fully establish the necessity of the specific mechanisms: The removal studies demonstrate that the components help, but historical self-distillation is not directly compared with supervised prompt realignment using the same replay data and a matched budget. Likewise, the degree-one comparison supports learning additional prompt flexibility, but does not establish whether the Bézier parameterization offers advantages over other similarly sized learned spaces. The visualizations and qualitative examples demonstrate useful intermediate prompts, while providing limited evidence about the underlying cross-task knowledge-transfer mechanism.
  4. The explicit smoothness bound appears to require a correction to the assumptions: Assumption 1 imposes gradient bounds and smoothness directly with respect to the control parameters C. However, Lemma 2 introduces additional Bézier Jacobian factors to obtain bounds proportional to ρ and ρ². This chain-rule argument would normally require the corresponding assumptions with respect to the generated prompt Z. As written, the tighter bound does not appear to follow directly from the stated assumptions. This seems potentially fixable, but affects the explicit constant used in the convergence theorem. The theorem itself only establishes stationarity for the isolated control-prompt optimization stage.
  5. Some reporting details limit interpretation and reproducibility: The “all components removed” configuration in Table 2 does not reproduce the ProMoT results in Table 1, and the difference is not clearly explained. In addition, all experiments use one benchmark and one model family. The reversed-order results preserve LAPS’s relative advantage, but the 4B BWT of −5.7 shows that substantial forgetting can remain.
Questions for authors
  1. Can you report results across independent training seeds, particularly for the 8B comparison with ProMoT? Is the 0.2-point Average improvement stable across runs?
  2. Can you separately report retention before final prompt search and improvements introduced by search? How does LAPS compare with simpler search procedures or prompt adaptation baselines under the same validation-evaluation budget?
  3. How does on-policy prompt alignment compare with supervised realignment using the same replay samples? Is there evidence that the Bézier construction is preferable to a similarly sized alternative parameterization?
  4. What are the training, alignment, space-learning, and final-search costs relative to ProMoT? How do memory and runtime scale as the task sequence grows?
  5. Should Assumption 1 be stated with respect to the generated prompt Z, rather than the control parameters C, to justify the bounds in Lemma 2?
  6. What accounts for the difference between the “all components removed” results in Table 2 and the ProMoT results in Table 1?
6 comments
AuthorsOpenReview ↗

Response (Part I)

We sincerely thank the reviewer for the detailed and constructive feedback, which has greatly helped us improve our work.

Question 1 & Weakness 1.1: Statistical Robustness

We conducted three independent training runs, with each run evaluated eight times. We additionally equipped ProMoT with the same validation-search procedure. The results are reported below.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
ProMoT58.1 ± 0.40-4.6 ± 0.3763.2 ± 0.21-2.1 ± 0.2467.0 ± 0.10+0.1 ± 0.13
ProMoT + validation search58.7 ± 0.08-3.5 ± 0.1064.9 ± 0.17-1.2 ± 0.1367.4 ± 0.11+0.3 ± 0.14
ProgPrompt52.1 ± 0.29N/A58.2 ± 0.43N/A58.1 ± 0.24N/A
ProgPrompt + validation search53.2 ± 0.19N/A59.6 ± 0.31N/A60.6 ± 0.22N/A
LAPS60.8 ± 0.12-1.1 ± 0.2066.2 ± 0.15+0.1 ± 0.0867.9 ± 0.09+1.6 ± 0.12

Compared with ProMoT and ProgPrompt equipped with validation-guided search, LAPS achieves higher Average performance and better BWT across all three model scales. These results demonstrate that its improvements cannot be attributed solely to additional validation-guided search, but also reflect the benefits of the historical prompt alignment.

Question 2 & Weakness 1.2: Analysis of Validation-Guided Search

(1) Contribution of Validation-Guided Search

To disentangle the benefits of historical prompt alignment from those of final prompt selection, we report the results using only the aligned task anchors (without validation-guided search), averaged over three independent training runs.

MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
ProMoT58.1 ± 0.40-4.6 ± 0.3763.2 ± 0.21-2.1 ± 0.2467.0 ± 0.10+0.1 ± 0.13
LAPS (Anchors Only)59.8 ± 0.16-2.1 + ± 0.1465.8 ± 0.16-0.7 ± 0.1767.1 ± 0.13+1.4 ± 0.10
LAPS (validation-guided search)60.8 ± 0.12-1.1 ± 0.2066.2 ± 0.15+0.1 ± 0.0867.9 ± 0.09+1.6 ± 0.12

On Qwen3-8B, LAPS (Anchors Only) achieves a comparable Average to ProMoT (67.1 vs. 67.0), while improving BWT by 1.3 (1.4 vs. 0.1), indicating better retention even without validation-guided search. Validation-guided search further improves Average by 1.0%, 0.4%, and 0.8%, respectively, providing additional gains through task-specific prompt selection.

(2) Comparison with a Simpler Search Strategy (Grid Search)

Following the reviewer's suggestion to compare Bayesian search with a simpler search procedure, we additionally evaluated normalized-coordinate grid search on the same learned prompt space under the same validation-evaluation budget. The results are summarized below:

Search StrategyQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWTQwen3-8B Avg.Qwen3-8B BWT
Grid Search59.9 ± 0.18-2.2 ± 0.1065.9 ± 0.17-0.5 ± 0.1367.2± 0.17+0.8 ± 0.10
Bayesian Search60.8 ± 0.13-1.1 ± 0.1366.2 ± 0.17+0.1 ± 0.1167.9 ± 0.18+1.6 ± 0.10

These results show that Bayesian search consistently outperforms grid search across all three model scales.

AuthorsOpenReview ↗

Response (Part II)

Question 3 & Weakness 3: Prompt Alignment and Space Parameterization

(1) On-Policy vs. Supervised Prompt Alignment

Following the reviewer's suggestion to compare on-policy alignment with supervised realignment using the same replay data, we replaced OPSD with Teacher-Rollout SFT, while keeping the remaining LAPS components unchanged.

Alignment MethodQwen3-1.7B Avg.Qwen3-1.7B BWTQwen3-4B Avg.Qwen3-4B BWT
Teacher-Rollout SFT59.2 ± 0.08−2.4 ± 0.1765.1 ± 0.08−0.9 ± 0.11
OPSD (LAPS)60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.07+0.1 ± 0.11

OPSD consistently outperforms Teacher-Rollout SFT, improving Average by 1.6% and 1.1% and BWT by 1.3 and 1.0 on Qwen3-1.7B and 4B, respectively. These results support the advantage of aligning historical prompts on student-generated trajectories rather than relying solely on supervised learning from teacher rollouts.

(2) Bézier vs. Alternative Prompt-Space Parameterization

We implemented a shared residual baseline using a fixed linear projection to combine learnable residual prompts. Both methods use the same historical task anchors, replay-based space learning, and Bayesian search.

MethodQwen3-1.7B Avg.Qwen3-1.7b BWTQwen3-4B Avg.Qwen3-4B BWT
Linear-Projection Residual58.9 ± 0.09-1.6±0.1364.1 ± 0.07-1.6±0.14
Bézier Residual60.8 ± 0.13−1.1 ± 0.1366.2 ± 0.07+0.1 ± 0.11

The Bézier parameterization improves Average by 1.9% and 2.1% on Qwen3-1.7B and Qwen3-4B, respectively. These results suggest that the benefit of LAPS is not simply due to introducing learnable residual prompts, but also arises from the structured Bézier mapping between task preferences and the learned prompt space.

(3) Cross-Task Knowledge Transfer

The selected coordinates λ\boldsymbol{\lambda} provide an interpretable way to analyze potential cross-task interactions. This is enabled by our smooth Tchebycheff optimization (Eq. 8 of the manuscript), which explicitly associates each coordinate with task losses.

Table 9 in the Appendix provides supporting evidence. Some tasks select intermediate configurations involving multiple task directions (e.g., C-STANCE), suggesting potentially beneficial cross-task interactions, while others favor configurations close to their own anchors (e.g., MeetingBank). These results suggest that LAPS can identify beneficial cross-task configurations when available, while allowing tasks to retain their own prompt representations when cross-task interactions offer limited benefits.

AuthorsOpenReview ↗

Response (Part III)

Question 4 & Weakness 2: Efficiency Analysis of LAPS

We profiled the wall-clock runtime and peak GPU memory consumption.

(1) Runtime Analysis.

The measured runtime (in hours) for a complete task sequence is summarized below.

ModelMethodTask AdaptationAlignmentSpace LearningBayesian SearchTotal
Qwen3-1.7BProMoT8.5–––8.5
LAPS8.52.13.52.216.3
Qwen3-4BProMoT12.9–––12.9
LAPS12.92.65.44.125.0
Qwen3-8BProMoT24.3–––24.3
LAPS24.32.97.95.941.0

The alignment stage includes teacher evaluations and student on-policy generation, the space-learning stage involves control-prompt optimization, and Bayesian search accounts for repeated validation evaluations.

(2) Peak GPU Memory.

The measured peak GPU memory consumption (GiB) is reported below.

ModelTask AdaptationAlignmentSpace LearningBayesian Search
Qwen3-1.7B15.919.59.820.6
Qwen3-4B31.140.224.741.5
Qwen3-8B59.560.734.060.1

(3) Training Overhead vs. Inference Cost.

We acknowledge that LAPS introduces additional computational overhead. However, this overhead is incurred during offline prompt learning and selection, rather than at every inference step. Once the task-specific prompt is selected, it can be directly stored and reused for subsequent inference, without repeating space learning, control-prompt optimization, or Bayesian search. Moreover, the additional parameter storage introduced by LAPS remains below 0.5% of the backbone parameters.

The key trade-off is additional offline adaptation cost in exchange for better knowledge retention and the ability to exploit beneficial cross-task prompt configurations at inference time. We believe this trade-off is worthwhile for continual fine-tuning scenarios where mitigating forgetting and enabling positive transfer across tasks are important than minimizing offline adaptation cost alone.

(4) Scalability with Increasing Tasks.

For the cubic Bézier construction, the number of control prompts grows as (t+23)−t=O(t3)\binom{t+2}{3}-t=O(t^3). To quantify the practical overhead, we analyzed the control-prompt storage and Adam optimizer memory as the number of tasks increases from 8 to 16.

Model8-task Control Storage16-task Control Storage8-task Adam States16-task Adam States
Qwen3-1.7B30 MiB204 MiB120 MiB816 MiB
Qwen3-4B37 MiB255 MiB150 MiB1.0 GiB
Qwen3-8B60 MiB408 MiB240 MiB1.59 GiB

Although the control-point count grows cubically, the absolute memory requirements remain manageable for the analyzed 16-task setting.

For substantially longer task sequences (e.g., hundreds of tasks), the cubic growth of control prompts may require further scalability improvements. As discussed in Section 5 of the manuscript, task-aware routing or sparse control-point selection could restrict optimization to relevant task interactions, avoiding the cost of maintaining a fully dense control-prompt space.

Question 5 & Weakness 4: Correction of Smoothness Assumptions

We agree that Assumption 1 should impose the gradient bound and smoothness condition with respect to the generated prompt Z\boldsymbol{\mathbf{Z}}, rather than directly on the control parameters C\boldsymbol{\mathbf{C}}. We correct the assumption as follows:

For all tasks ii and feasible prompts (Z,Z′)(\boldsymbol{\mathbf{Z}},\boldsymbol{\mathbf{Z}^{'}}), we assume

\qquad |\nabla_{\boldsymbol{\mathbf{Z}}}\ell_i(Z)-\nabla_{\boldsymbol{\mathbf{Z}}}\ell_i(\boldsymbol{\mathbf{Z}}^{'})| \le\beta|\boldsymbol{\mathbf{Z}}-\boldsymbol{\mathbf{Z}}^{'}|.$$ Since the Bézier mapping $\boldsymbol{\mathbf{Z}}(\boldsymbol{\lambda};\boldsymbol{\mathbf{C}})$ is affine in $\boldsymbol{\mathbf{C}}$, its Jacobian is independent of $\boldsymbol{\mathbf{C}}$ and satisfies $$\lVert J_{\mathbf{Z},\mathbf{C}}(\boldsymbol{\lambda})\rVert_{\mathrm{op}} = \rho_{t,k}(\boldsymbol{\lambda}).$$ Applying the chain rule yields $$\lVert \nabla_{\mathbf{C}}\ell_i \rVert \le M\rho_{t,k}(\boldsymbol{\lambda}), \qquad \lVert \nabla_{\mathbf{C}}^2\ell_i \rVert_{\mathrm{op}} \le \beta\rho_{t,k}^2(\boldsymbol{\lambda}).$$ where the second inequality holds wherever the Hessian exists. Equivalently, it follows from the Lipschitz continuity of the gradient under the affine Bézier mapping. Consequently, the explicit smoothness bound in Lemma 2 retains the same form under the corrected assumption. Thus, the correction does not effect the convergence rate or the form of the bound in Theorem 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Kind Request for Reviewers' Feedback Before the Discussion Closes

Dear Reviewer,

With only two days remaining in the discussion period, we sincerely hope to hear your thoughts on whether our rebuttal and additional experiments have adequately addressed your concerns, particularly regarding the multiple independent training runs and further ablation studies.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.