- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies lifelong alignment, where an LLM must incorporate alignment requirements that arrive sequentially while retaining previously acquired behaviors. The paper considers both task-incremental alignment, where new tasks and associated requirements arrive over time, and preference-incremental alignment, where multiple preferences are sequentially introduced for a fixed task. The proposed method, Ready2Blend, keeps the LLM backbone frozen and represents each alignment requirement as a fixed-length continuous prompt. An AlignFormer, conditioned on a natural-language description of the requirement, is trained using alignment supervision to produce the corresponding prompt. To make independently learned prompts composable, the method introduces point-wise and pair-wise regularization that attempts to transfer the geometry of a frozen sentence-embedding space into the prompt representation space. Previously learned prompts are stored in a prompt bank and remain fixed. At inference time, an arbitrary subset of prompts can be combined through weighted arithmetic averaging, allowing requirements to be activated or reweighted without retraining the backbone. The paper evaluates Ready2Blend on task- and preference-incremental alignment using Qwen3.5 and Llama backbones. The reported results show competitive final performance relative to continual post-training methods while modifying no backbone parameters, together with lower training time and a fixed four-token inference overhead. The paper further evaluates prompt-space regularization, arithmetic averaging versus concatenation, preference personalization, order sensitivity, prompt length, alternative LLM judges, a smaller backbone, and an SFT variant.
Critical strengths
-
The paper presents a practically meaningful formulation of continual alignment that decouples newly acquired alignment requirements from the backbone parameters. Each requirement is stored as an independently learned prompt, while the backbone and previously stored prompts remain unchanged. This makes requirements modular at deployment: they can be activated, removed, or reweighted at inference time without another round of backbone post-training. This is a consequential advantage over continual alignment methods that consolidate requirements into a shared model state.
-
The proposed composability regularization is a technically meaningful attempt to address a real difficulty of independently learned prompts, and the empirical validation is reasonably broad. Direct arithmetic over independently optimized prompts is not inherently meaningful; Ready2Blend explicitly regularizes both individual prompt–requirement alignment and pairwise relations among prompts. The main experiments show competitive performance with strong post-training baselines while keeping the backbone frozen, and the paper supplements these results with regularization ablations, prompt-space visualization, arithmetic-vs.-concatenation comparisons, order-sensitivity experiments, personalization, alternative evaluators, different backbone sizes, and an SFT variant.
Critical weaknesses
-
The shared-geometry argument requires further clarification under sequential AlignFormer updates.
The paper states that the shared AlignFormer is optimized at every stage, while previously generated prompts are stored and remain fixed. If also changes across stages, an earlier prompt and a later prompt are generally of the form and , with potentially different projection matrices and . In this case, is not generally equivalent to applying a single linear projection to . A related issue applies to cached projected requirement representations if is also updated across stages. Since the paper uses a common semantic geometry to motivate arithmetic prompt composition, the exact treatment of these projection layers and historical representations is important for the technical interpretation of the method.Maybe I have some misunderstanding about this ,but still I would appreciate clarification on this point, or further theoretical/technical justification showing why the claimed shared-geometry interpretation remains valid under the sequential update procedure. -
The current ablations establish that the proposed regularization improves composition, but they do not fully isolate semantic-geometry transfer as the cause of the improvement.
Table 2 uses a cumulative ablation, from no regularization to point-wise consistency and then to point-wise plus pair-wise consistency, without a pair-wise-only condition. In addition, there is no non-semantic control, such as shuffled requirement embeddings or another fixed target geometry, that would separate the effect of semantic structure from a more generic regularization or stabilization effect. Thus, the empirical evidence supports the effectiveness of the proposed regularization, while the stronger mechanism claim that transferring natural-language semantic geometry is specifically what enables composition would benefit from additional clarification or evidence.
Both concerns appear potentially resolvable through clarification or additional evidence. A satisfactory response, particularly regarding the cross-stage projection geometry, would materially improve my assessment.
Questions for authors
-
How are and handled across continual stages?
Are they updated together with the rest of AlignFormer at every stage, or are either of them frozen? If they are updated, historical , , and are produced under earlier projection matrices. Could the authors clarify how the common-geometry interpretation and the claimed equivalence between blending in prompt space and blending before projection should be understood in this case? If the equivalence still holds under the implemented training procedure, additional justification or analysis would be helpful. -
Could the authors clarify the training objective and model-selection protocol used for the main comparisons?
In particular, what optimization objective is used for the adapted L2P and DualPrompt baselines? Do they receive the same pairwise preference/DPO supervision as Ready2Blend, or are they trained with autoregressive supervision? Also, was the reported choice selected using a held-out validation split rather than the reported test evaluation? These details would help assess the fairness of the frozen-backbone comparison and the independence of model selection from the final evaluation.
11 comments
W1-1. Projection Updates and Prompt Composition
Both and are updated at each continual stage. Historical representations are cached when acquired, while the corresponding prompt is permanently stored in the prompt bank. These values are not recomputed after subsequent AlignFormer updates.
For prompts acquired at different stages,
their weighted composition is
=\alpha W_2^{(j)}Z_j+(1-\alpha)W_2^{(t)}Z_t.$$ When $W_2^{(j)}\neq W_2^{(t)}$, this is not generally equivalent to applying a single projection to the blended latent representations. **Importantly, our method does not rely on this equivalence**, as the stored prompts are directly blended in the shared input embedding space of the frozen LLM. The use of learned projection layers to interface compact representations with frozen language models is well established. BLIP-2 [1] employs a learned query-based interface to bridge representation spaces, ATTEMPT [2] demonstrates the composition of soft prompts over a frozen backbone, and Black-Box Tuning [3] adopts a latent-to-prompt projection for parameter-efficient adaptation. These studies provide architectural precedents for projection-based prompt adaptation and composition, although they do not establish exact equivalence across sequentially updated projections. We have clarified the treatment of stored prompts and cached representations in **Sections 4.1 (lines 236–238)** and **4.2 (lines 289–294)**, explicitly distinguishing prompt-space composition from latent-space projection equivalence. **References** [1] Li, et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." ICML, 2023. [2] Asai, et al. "ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts." EMNLP, 2022. [3] Sun, et al. "Black-Box Tuning for Language-Model-as-a-Service." ICML, 2022.W1-2. Empirical Validation of Projection Stability
To examine whether sequential projection updates disrupt the consistency of historical prompt representations, we compare each historical representation under its original and final-stage projection:
Holding fixed isolates the effect of changes in . We evaluate this diagnostic on two backbones and both continual-alignment settings, with and without composability regularization.
Table 1. Cosine similarity between historical representations under their original and final-stage projections.
| Backbone | Dataset | Reg. | Mean Cos ↑ (Std.) | Stage 1 Cos ↑ |
|---|---|---|---|---|
| Llama 3.1 8B | FeedSum | None | 0.9502 (0.0341) | 0.9154 |
| Llama 3.1 8B | FeedSum | Full | 0.99995 (0.00005) | 0.99995 |
| Llama 3.1 8B | LifeAlign | None | 0.9053 (0.0647) | 0.8586 |
| Llama 3.1 8B | LifeAlign | Full | 0.99988 (0.00020) | 0.99970 |
| Qwen 3.5 9B | FeedSum | None | 0.9678 (0.0206) | 0.9453 |
| Qwen 3.5 9B | FeedSum | Full | 0.99996 (0.00005) | 0.99995 |
| Qwen 3.5 9B | LifeAlign | None | 0.9437 (0.0395) | 0.9231 |
| Qwen 3.5 9B | LifeAlign | Full | 0.99992 (0.00013) | 0.99981 |
Without regularization, mean cosine similarity ranges from 0.9053 to 0.9678. With full composability regularization, it increases to 0.99988–0.99996, with first-stage representations maintaining similarities of 0.99970–0.99995.
These consistently high similarities indicate that sequential projection updates introduce negligible directional changes to historical prompt representations, providing empirical evidence that the learned projection maintains a stable representational geometry throughout continual alignment. This stability is particularly important for Ready2Blend, as independently acquired prompts must remain compatible within the shared LLM embedding space for subsequent composition.
While cosine similarity does not establish exact projection equivalence or preservation of vector norms, the results demonstrate that the projection updates do not substantially disrupt the directions of historical representations under the evaluated conditions.
W1. Clarification: Projection Updates and Shared Geometry
Thank you for identifying this important issue. We agree that blending prompts generated under different projection matrices is not mathematically equivalent to blending their latent representations under a single projection. However, Ready2Blend performs composition directly in the frozen LLM's input embedding space, rather than relying on this equivalence. We clarify the implementation, discuss relevant architectural precedents, and provide additional empirical evidence of projection stability.
Specifically, we address this concern in three parts: (W1-1) Projection Updates and Prompt Composition, clarifying the implementation and mathematical distinction; (W1-2) Empirical Validation of Projection Stability, demonstrating limited projection drift across continual stages; and (W1-3) Interpretation of Shared Geometry, explaining how geometric consistency supports prompt composition without requiring exact projection invariance.