acceptodds
Under review as a conference paper at ICLR 2027

Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts

Abstract

Continual alignment requires LLMs to adapt to new requirements without forgetting previously acquired behaviors. Natural-language instructions are flexible and composable but offer only indirect control, whereas post-training provides stronger adaptation at the cost of repeated parameter updates. We introduce Ready2Blend, which combines the flexibility of natural language with learned alignment. AlignFormer maps each requirement to a fixed-length alignment prompt stored in a modular prompt bank, while the backbone and prior prompts remain frozen. Composability regularization transfers the semantic geometry of textual requirements into prompt space, enabling inference-time blending and reweighting. Across two practical continual alignment settings, Ready2Blend is the only frozen-backbone method that matches post-training-based alignment methods, reaching  % of a joint-training reference with competitive retention, while requiring only a few prompt tokens and up to less training time. Its modular design further enables weighted personalization and order-free composition without retraining. Code will be released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Lifelong Learning, Lifeling Alignment, Learned Prompt, Prompt Composability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YJuLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies lifelong alignment, where an LLM must incorporate alignment requirements that arrive sequentially while retaining previously acquired behaviors. The paper considers both task-incremental alignment, where new tasks and associated requirements arrive over time, and preference-incremental alignment, where multiple preferences are sequentially introduced for a fixed task. The proposed method, Ready2Blend, keeps the LLM backbone frozen and represents each alignment requirement as a fixed-length continuous prompt. An AlignFormer, conditioned on a natural-language description of the requirement, is trained using alignment supervision to produce the corresponding prompt. To make independently learned prompts composable, the method introduces point-wise and pair-wise regularization that attempts to transfer the geometry of a frozen sentence-embedding space into the prompt representation space. Previously learned prompts are stored in a prompt bank and remain fixed. At inference time, an arbitrary subset of prompts can be combined through weighted arithmetic averaging, allowing requirements to be activated or reweighted without retraining the backbone. The paper evaluates Ready2Blend on task- and preference-incremental alignment using Qwen3.5 and Llama backbones. The reported results show competitive final performance relative to continual post-training methods while modifying no backbone parameters, together with lower training time and a fixed four-token inference overhead. The paper further evaluates prompt-space regularization, arithmetic averaging versus concatenation, preference personalization, order sensitivity, prompt length, alternative LLM judges, a smaller backbone, and an SFT variant.

Critical strengths
  1. The paper presents a practically meaningful formulation of continual alignment that decouples newly acquired alignment requirements from the backbone parameters. Each requirement is stored as an independently learned prompt, while the backbone and previously stored prompts remain unchanged. This makes requirements modular at deployment: they can be activated, removed, or reweighted at inference time without another round of backbone post-training. This is a consequential advantage over continual alignment methods that consolidate requirements into a shared model state.

  2. The proposed composability regularization is a technically meaningful attempt to address a real difficulty of independently learned prompts, and the empirical validation is reasonably broad. Direct arithmetic over independently optimized prompts is not inherently meaningful; Ready2Blend explicitly regularizes both individual prompt–requirement alignment and pairwise relations among prompts. The main experiments show competitive performance with strong post-training baselines while keeping the backbone frozen, and the paper supplements these results with regularization ablations, prompt-space visualization, arithmetic-vs.-concatenation comparisons, order-sensitivity experiments, personalization, alternative evaluators, different backbone sizes, and an SFT variant.

Critical weaknesses
  1. The shared-geometry argument requires further clarification under sequential AlignFormer updates.
    The paper states that the shared AlignFormer is optimized at every stage, while previously generated prompts are stored and remain fixed. If Proj2\mathrm{Proj}_2 also changes across stages, an earlier prompt and a later prompt are generally of the form Pj=W2(j)ZjP_j = W_2^{(j)} Z_j and Pt=W2(t)ZtP_t = W_2^{(t)} Z_t, with potentially different projection matrices W2(j)W_2^{(j)} and W2(t)W_2^{(t)}. In this case, αPj+(1−α)Pt=αW2(j)Zj+(1−α)W2(t)Zt\alpha P_j + (1-\alpha)P_t = \alpha W_2^{(j)}Z_j + (1-\alpha)W_2^{(t)}Z_t is not generally equivalent to applying a single linear projection to αZj+(1−α)Zt\alpha Z_j + (1-\alpha)Z_t. A related issue applies to cached projected requirement representations if Proj1\mathrm{Proj}_1 is also updated across stages. Since the paper uses a common semantic geometry to motivate arithmetic prompt composition, the exact treatment of these projection layers and historical representations is important for the technical interpretation of the method.Maybe I have some misunderstanding about this ,but still I would appreciate clarification on this point, or further theoretical/technical justification showing why the claimed shared-geometry interpretation remains valid under the sequential update procedure.

  2. The current ablations establish that the proposed regularization improves composition, but they do not fully isolate semantic-geometry transfer as the cause of the improvement.
    Table 2 uses a cumulative ablation, from no regularization to point-wise consistency and then to point-wise plus pair-wise consistency, without a pair-wise-only condition. In addition, there is no non-semantic control, such as shuffled requirement embeddings or another fixed target geometry, that would separate the effect of semantic structure from a more generic regularization or stabilization effect. Thus, the empirical evidence supports the effectiveness of the proposed regularization, while the stronger mechanism claim that transferring natural-language semantic geometry is specifically what enables composition would benefit from additional clarification or evidence.

Both concerns appear potentially resolvable through clarification or additional evidence. A satisfactory response, particularly regarding the cross-stage projection geometry, would materially improve my assessment.

Questions for authors
  1. How are Proj1\mathrm{Proj}_1 and Proj2\mathrm{Proj}_2 handled across continual stages?
    Are they updated together with the rest of AlignFormer at every stage, or are either of them frozen? If they are updated, historical zjz_j, e~j\tilde{e}_j, and PjP_j are produced under earlier projection matrices. Could the authors clarify how the common-geometry interpretation and the claimed equivalence between blending in prompt space and blending before projection should be understood in this case? If the equivalence still holds under the implemented training procedure, additional justification or analysis would be helpful.

  2. Could the authors clarify the training objective and model-selection protocol used for the main comparisons?
    In particular, what optimization objective is used for the adapted L2P and DualPrompt baselines? Do they receive the same pairwise preference/DPO supervision as Ready2Blend, or are they trained with autoregressive supervision? Also, was the reported choice k=4k=4 selected using a held-out validation split rather than the reported test evaluation? These details would help assess the fairness of the frozen-backbone comparison and the independence of model selection from the final evaluation.

11 comments
AuthorsOpenReview ↗

W1. Clarification: Projection Updates and Shared Geometry

Thank you for identifying this important issue. We agree that blending prompts generated under different projection matrices is not mathematically equivalent to blending their latent representations under a single projection. However, Ready2Blend performs composition directly in the frozen LLM's input embedding space, rather than relying on this equivalence. We clarify the implementation, discuss relevant architectural precedents, and provide additional empirical evidence of projection stability.

Specifically, we address this concern in three parts: (W1-1) Projection Updates and Prompt Composition, clarifying the implementation and mathematical distinction; (W1-2) Empirical Validation of Projection Stability, demonstrating limited projection drift across continual stages; and (W1-3) Interpretation of Shared Geometry, explaining how geometric consistency supports prompt composition without requiring exact projection invariance.

AuthorsOpenReview ↗

W1-1. Projection Updates and Prompt Composition

Both Proj1\mathrm{Proj}_1 and Proj2\mathrm{Proj}_2 are updated at each continual stage. Historical representations (zj,e~j)(z_j,\tilde e_j) are cached when acquired, while the corresponding prompt PjP_j is permanently stored in the prompt bank. These values are not recomputed after subsequent AlignFormer updates.

For prompts acquired at different stages,

Pj=W2(j)Zj,Pt=W2(t)Zt,P_j = W_2^{(j)}Z_j, \qquad P_t = W_2^{(t)}Z_t,

their weighted composition is

=\alpha W_2^{(j)}Z_j+(1-\alpha)W_2^{(t)}Z_t.$$ When $W_2^{(j)}\neq W_2^{(t)}$, this is not generally equivalent to applying a single projection to the blended latent representations. **Importantly, our method does not rely on this equivalence**, as the stored prompts are directly blended in the shared input embedding space of the frozen LLM. The use of learned projection layers to interface compact representations with frozen language models is well established. BLIP-2 [1] employs a learned query-based interface to bridge representation spaces, ATTEMPT [2] demonstrates the composition of soft prompts over a frozen backbone, and Black-Box Tuning [3] adopts a latent-to-prompt projection for parameter-efficient adaptation. These studies provide architectural precedents for projection-based prompt adaptation and composition, although they do not establish exact equivalence across sequentially updated projections. We have clarified the treatment of stored prompts and cached representations in **Sections 4.1 (lines 236–238)** and **4.2 (lines 289–294)**, explicitly distinguishing prompt-space composition from latent-space projection equivalence. **References** [1] Li, et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." ICML, 2023. [2] Asai, et al. "ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts." EMNLP, 2022. [3] Sun, et al. "Black-Box Tuning for Language-Model-as-a-Service." ICML, 2022.
AuthorsOpenReview ↗

W1-2. Empirical Validation of Projection Stability

To examine whether sequential projection updates disrupt the consistency of historical prompt representations, we compare each historical representation under its original and final-stage projection:

sj=cos⁡(W2(j)Zj,  W2(T)Zj).s_j=\cos\left(W_2^{(j)}Z_j,\;W_2^{(T)}Z_j\right).

Holding ZjZ_j fixed isolates the effect of changes in Proj2\mathrm{Proj}_2. We evaluate this diagnostic on two backbones and both continual-alignment settings, with and without composability regularization.

Table 1. Cosine similarity between historical representations under their original and final-stage projections.

BackboneDatasetReg.Mean Cos ↑ (Std.)Stage 1 Cos ↑
Llama 3.1 8BFeedSumNone0.9502 (0.0341)0.9154
Llama 3.1 8BFeedSumFull0.99995 (0.00005)0.99995
Llama 3.1 8BLifeAlignNone0.9053 (0.0647)0.8586
Llama 3.1 8BLifeAlignFull0.99988 (0.00020)0.99970
Qwen 3.5 9BFeedSumNone0.9678 (0.0206)0.9453
Qwen 3.5 9BFeedSumFull0.99996 (0.00005)0.99995
Qwen 3.5 9BLifeAlignNone0.9437 (0.0395)0.9231
Qwen 3.5 9BLifeAlignFull0.99992 (0.00013)0.99981

Without regularization, mean cosine similarity ranges from 0.9053 to 0.9678. With full composability regularization, it increases to 0.99988–0.99996, with first-stage representations maintaining similarities of 0.99970–0.99995.

These consistently high similarities indicate that sequential projection updates introduce negligible directional changes to historical prompt representations, providing empirical evidence that the learned projection maintains a stable representational geometry throughout continual alignment. This stability is particularly important for Ready2Blend, as independently acquired prompts must remain compatible within the shared LLM embedding space for subsequent composition.

While cosine similarity does not establish exact projection equivalence or preservation of vector norms, the results demonstrate that the projection updates do not substantially disrupt the directions of historical representations under the evaluated conditions.

Reviewer_DNjpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes Ready2Blend, an approach that learns soft continuous prefixes for natural language requirements, and makes them blendable and mergeable at inference time on demand.

Critical strengths
  • continual alignment is an important research question
  • the concept of compositional soft prompts is interesting
Critical weaknesses
  • a few more analysis would be nice
Questions for authors
  • I'm a little bit unsure about authors calling "prompts" and "prompt banks" modular. What does modular mean here? How is modularity reflected?

  • The authors mention weight-merging methods in line 76-79 as one of the two main background, however no weight merging methods are compared to in the experiments? It should be simple right: RL a model, one for each requirement, then at infenrece time apply the same mixing weights to fuse into a model and generate? That is very efficient too?

  • Composable preferences remind me of things like Personalized Soups [1]. For the authors' reference.

  • I'm curious would opposite requirements (be concise/be detailed) be mapped onto canceling directions in the soft prompt space? Like would their soft prompt tensors cancel out each other? That would be the ideal/expected case if semantic information is indeed mapped onto the soft prompt space like the authors suggested.

  • The learned soft prompt, prompt bank, etc. is specific to the frozen LLM right? Any updates to the LLM means re-training from scratch?

  • I feel like folks would also be interesting in how this compares to prompt optimization in the discrete space: there are many open tools perhaps worth a quick try.

[1] Jang, et al. "Personalized soups: Personalized large language model alignment via post-hoc parameter merging."

9 comments

We sincerely appreciate the reviewer’s thoughtful and constructive feedback. We have carefully examined each concern and provide detailed responses below. We hope our rebuttal addresses the issues raised, and all discussed revisions will be incorporated into the updated manuscript.


W1. What Does Modularity Mean?

I'm a little bit unsure about authors calling "prompts" and "prompt banks" modular. What does modular mean here? How is modularity reflected?

Thank you for identifying this ambiguity. In Ready2Blend, modularity is a key design property for achieving cost-effective continual alignment and flexible inference-time steering. Specifically, it refers to maintaining each acquired alignment requirement as a separate, reusable soft-prompt control, rather than consolidating all requirements into a single updated model state.

This modular representation allows requirements to be independently stored, selectively activated, and reweighted through lightweight prompt composition, without additional backbone updates or retraining. Importantly, modularity refers to the stored prompts rather than independent AlignFormer models: AlignFormer is shared and updated sequentially, while previously acquired prompts remain fixed in the prompt bank.

Specifically, at stage (t), we optimize the shared AlignFormer using the current requirement and its alignment supervision, then store the resulting prompt (P_t) in the prompt bank. Previously stored prompts remain unchanged. At inference, we retrieve the prompts corresponding to the desired requirements and blend them into a single fixed-length prompt, without further training or modifying the backbone.

The modular components are therefore the stored requirement-specific prompts, rather than separate AlignFormer models for different stages. AlignFormer is shared and updated sequentially; the bank preserves its stage-specific outputs so that earlier prompts need not be regenerated using the updated AlignFormer. Moreover, “independent” refers to acquiring each requirement from its own supervision, rather than complete independence of the optimization trajectories: the pair-wise regularizer explicitly accounts for previously acquired representations.

We have clarified this definition in Section 4.1, lines 236–238, and described requirement selection and reweighting. The distinction between sequential training and order-invariant composition is also clarified in the Introduction, lines 95–97. Keeping stored prompts fixed preserves the individual control units, although their joint behavioral effects can still change when additional prompts are blended.

AuthorsOpenReview ↗

W2. Comparison with Weight-Merging Methods

The authors mention weight-merging methods in line 76-79 as one of the two main background, however no weight merging methods are compared to in the experiments? It should be simple right: RL a model, one for each requirement, then at infenrece time apply the same mixing weights to fuse into a model and generate? That is very efficient too? Composable preferences remind me of things like Personalized Soups [1]. For the authors' reference.

Thank you for this valuable suggestion and for pointing us to Personalized Soups [1]. We agree that parameter merging provides a relevant and efficient approach to composing independently acquired preferences.

Importantly, our objective is not simply to replace parameter merging with prompt averaging. Ready2Blend investigates whether continual alignment can be achieved without modifying the LLM backbone, while maintaining competitive alignment quality and enabling lightweight, flexible steering through soft-prompt composition.

Both parameter merging and Ready2Blend support post-hoc composition. However, they differ in where alignment is learned and stored: parameter-merging methods encode alignment in model updates, whereas Ready2Blend maintains an unchanged backbone and represents each acquired requirement as a compact, reusable prompt. This distinction is particularly relevant when alignment requirements arrive sequentially and different combinations must be activated or reweighted at inference.

To directly assess these alternatives, we conducted additional comparisons with Rewarded Soups [2], Personalized Soups [1], and Bone Soups [3]. The results under our task- and preference-incremental evaluation settings are reported below. Higher values indicate better performance for both metrics.

MethodTask-Inc. BWT ↑Task-Inc. Last ↑Pref-Inc. BWT ↑Pref-Inc. Last ↑Avg. Last ↑
Rewarded Soups0.0120.743−0.0410.5970.670
Personalized Soups0.0160.736−0.0500.5880.662
Bone Soups0.0010.729−0.1340.5480.638
Ready2Blend0.0610.755−0.0330.7190.737

The parameter-merging baselines are competitive in the task-incremental setting, with Rewarded Soups achieving a Last score of 0.743, close to Ready2Blend’s 0.755. A larger gap emerges in the preference-incremental setting, where multiple requirements govern outputs for the same summarization inputs. Ready2Blend achieves a Last score of 0.719, compared with 0.597, 0.588, and 0.548 for the three merging baselines, respectively. It also achieves higher BWT than all three additional baselines in both settings. These results support the effectiveness of Ready2Blend under the evaluated continual-alignment conditions, particularly when multiple preferences must be composed within a shared task. They do not establish superiority across all possible preference mixtures.

We also agree with the reviewer’s observation regarding efficiency. Once merged weights are materialized, parameter merging supports generation without additional prompt tokens. Retaining the constituent models or parameter deltas also permits new mixtures without retraining. The following table distinguishes training, storage, composition, and inference costs. For Personalized Soups, these costs depend on whether preference-specific policies are implemented through full-model training or a parameter-efficient approach such as LoRA.

AuthorsOpenReview ↗

W2. Comparison with Weight-Merging Methods-2

Cost componentReady2Blend: AlignFormer with DPOPersonalized Soups
Training unitShared AlignFormer optimized at each stage; the resulting requirement-specific prompt is storedA separately trained policy or LoRA expert for each preference
Base LMFrozenPreference-specific parameter updates, either directly or through adapters
Trainable parametersAlignFormer parameters, including queries and projections; each stored prompt contains 4h4h valuesPolicy or LoRA adapter parameters, depending on the implementation
Training cost per stageAlignFormer optimization through the frozen LM, requiring forward and backward computationSeparate optimization of each preference-specific expert
Optimizer stateProportional to the trainable AlignFormer parametersProportional to the trainable policy or adapter parameters
Checkpoint storageOne 4×h4\times h prompt tensor per requirement for deployment, plus a shared AlignFormer checkpoint for continued trainingA model or adapter checkpoint per preference
Composition costWeighted averaging of stored prompt tensorsWeighted merging of model parameters or parameter deltas
Final inferenceOne generation with the blended prompt and frozen base LM; four additional prompt tokensOne generation with the merged model; no additional prompt tokens required
Evaluating multiple target mixturesConstruct a prompt blend and generate outputs for each mixtureConstruct or retrieve a parameter mixture and generate outputs for each mixture

Importantly, the four-token representation describes the stored prompt and inference overhead, rather than the total number of trainable parameters: Ready2Blend trains the shared AlignFormer. The table characterizes the computational structure of the approaches; a quantitative comparison of training cost, checkpoint size, and recomposition latency against the additional merging baselines remains to be conducted.

Ready2Blend and Personalized Soups share the principle of acquiring requirement-specific controls and composing them after training. Their principal differences concern where alignment is encoded and how the controls are recombined:

AspectPersonalized SoupsReady2Blend
Alignment targetModel parametersContinuous prompts generated by AlignFormer
CompositionParameter-space mergingPrompt-space blending
BackbonePreference-specific model trainingFrozen throughout continual alignment
Composed representationMerged model weightsA fixed-length, four-token prompt
PersonalizationRecompute the parameter mixtureRecompute the weighted prompt average
Continual settingPrimarily studies post-hoc personalization, including the addition of new preferencesExplicitly evaluates sequential acquisition, retention, and composition

Ready2Blend thus supports requirement selection and reweighting by recombining compact prompt tensors around a single unchanged backbone. Its contribution combines this representation with sequential alignment acquisition and regularization designed to facilitate composition.

We have incorporated the Qwen3.5-9B experimental results into Table 1, with corresponding discussions in Section 5.1 (lines 321–329). We have also expanded the related work discussion in Section 2 (lines 122–126) to cover Rewarded Soups, Personalized Soups, and Bone Soups, acknowledging their ability to support post-hoc preference composition without retraining. Additional results and implementation details are provided in Appendix Table 4 and Appendix A, covering LoRA-based merging methods (lines 874–892) and standard soft prompt tuning with averaging (lines 937–951), the latter serving as a direct control for evaluating the proposed design. We further plan to extend these comparisons to Llama-3.1-8B.

References [1] Jang et al. “Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging.” NeurIPS Workshop on Adaptive Foundation Models (AFM), 2024. [2] Ramé et al. “Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.” NeurIPS, 2023. [3] Xie et al. “Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation.” ACL, 2025.

Reviewer_Ct8eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies lifelong alignment for large language models, covering both task-incremental and preference-incremental settings. The authors argue that discrete text prompts lack fine-grained control, while continual post-training (e.g., DPO, CPPO) causes catastrophic forgetting and entangles preferences in model weights. The authors propose Ready2Blend, which maps alignment requirements into composable continuous prompts. AlignFormer uses a 2-layer Transformer decoder to map natural-language requirements into 4-token prompts while keeping the LLM backbone frozen. Composability regularization adds cosine consistency losses to align prompt representations with sentence-encoder geometry. At inference time, stored prompts are combined via convex combination and prepended to the input. The method is evaluated on Qwen3.5-9B, Llama-3.1-8B, and Qwen3.5-4B across task-incremental and preference-incremental benchmarks.

Critical strengths

The submission presents a lightweight and computationally practical approach to lifelong alignment:

  1. Low computational footprint: The approach keeps the base LLM frozen and trains only 4 soft tokens per requirement, reducing training time compared to full parameter post-training (Section 4.1, Table 11).

  2. Direct ablation on prompt aggregation: Table 10 provides a useful comparison showing that arithmetic prompt averaging performs better than simple token concatenation under the tested settings (Section 4.2, Table 10).

Critical weaknesses

However, several empirical and methodological aspects require further clarification:

  1. Missing foundational baselines: The paper omits essential controls: (a) standard prompt tuning per requirement (directly training a k x h tensor) followed by averaging, and (b) per-requirement LoRA with linear weight merging (e.g., Rewarded Soups). Without these, the necessity of AlignFormer and geometric regularization is unproven.

  2. Narrow margins and missing statistical variance: In Table 1, the lead over CPPO is only 0.014 on Qwen3.5-9B and 0.000 on Llama-3.1-8B, with no random seeds, standard deviations, or significance tests.

  3. Manuscript text contradicts experimental tables: Appendix C.2 claims the highest score across all judges, but Table 6 shows LifeAlign beats Ready2Blend under Qwen3.8-Judge (0.69 vs 0.68). The abstract claims matching post-training across all models (93.1–98.5% of MTL), but on Qwen3.5-4B (Table 8), Ready2Blend reaches only 91.5% and falls behind CPPO.

  4. Hidden training path dependence: The method is not truly "order-free" during training. AlignFormer is updated sequentially across stages and relies on cached historical embeddings; Figure 3 confirms non-zero variation across training orders (sigma = 0.015).

  5. Unverified semantic preservation and conflict handling: The paper lacks a prompt-by-requirement cross-evaluation matrix to verify prompt specificity. In Figure 5b, linear blending fails to resolve conflicting objectives, causing abstractiveness to drop as faithfulness and conciseness are added.

Questions for authors

Please refer to the detailed concerns and clarification points listed in the Critical Weaknesses section above.

9 comments
AuthorsOpenReview ↗

Ready2Blend: Response to Reviewer Comments

We sincerely appreciate for the reveiwer's thoughtful comments. We have carefully reviewed your comments and provide detailed responses to each point below. We hope our rebuttal adequately addresses the raised concerns, and all discussed revisions will be incorporated into the updated manuscript.


W1. Missing foundational baselines

The paper omits essential controls: (a) standard prompt tuning per requirement (directly training a k x h tensor) followed by averaging, and (b) per-requirement LoRA with linear weight merging (e.g., Rewarded Soups). Without these, the necessity of AlignFormer and geometric regularization is unproven.

Thank you for identifying these important missing controls. The central objective of Ready2Blend is to enable cost-effective continual alignment through frozen-backbone adaptation and composable soft-prompt steering. Our design therefore seeks to achieve alignment quality and retention competitive with parameter-updating approaches while preserving lightweight, inference-time control over individual requirements.

Our original baseline selection focused on continual adaptation and retention, including CPPO and LifeAlign, alongside prompt-based continual-learning methods such as L2P and DualPrompt. We agree, however, that direct soft prompt tuning and independent parameter merging are important controls for assessing the contributions of AlignFormer and composability regularization. We have therefore conducted additional comparisons with these approaches:

  • Standard Soft Prompt Tuning + Average: Directly optimize a k×hk\times h prompt tensor for each requirement over the frozen backbone, then average the resulting prompts. This provides a comparison without AlignFormer.
  • Rewarded Soups [1]: Compose independently optimized models through weighted parameter merging.
  • Personalized Soups [2]: Perform post-hoc parameter merging according to preference weights.
  • Bone Soups [3]: Include an additional model-merging approach developed for controllable multi-objective generation.

The additional results on Qwen3.5-9B are reported below.

Performance Comparison with Additional Soft Prompt Averaging and Parameter-Merging Baselines on Qwen3.5-9B.

MethodTask-Inc. BWT ↑Task-Inc. Last ↑Pref-Inc. BWT ↑Pref-Inc. Last ↑Avg. BWT ↑Avg. Last ↑
Standard Soft Prompt+Average-0.0370.451-0.0170.562-0.0270.507
Rewarded Soup0.0120.743-0.0410.597-0.0140.670
Personalized Soup0.0160.736-0.0500.588-0.0170.662
Bone Soup0.0010.729-0.1340.548-0.0660.638
Ready2Blend (Ours)0.0610.755-0.0330.7190.0140.737

The additional results show that parameter-merging methods remain competitive in task-incremental alignment (e.g., Rewarded Soups: 0.743 vs. Ready2Blend: 0.755 Last). However, Ready2Blend demonstrates a more substantial advantage in preference-incremental alignment, where multiple requirements must be jointly satisfied within the same task (0.719 vs. 0.597/0.588/0.548 for Rewarded/Personalized/Bone Soups). Furthermore, standard soft prompt averaging performs substantially worse (0.507 vs. 0.737 Avg. Last), highlighting the importance of learning composable alignment representations rather than simply averaging independently optimized prompts. Together with the regularization ablation in Table 2, these findings support the effectiveness of our design for cost-effective continual alignment and preference composition over a frozen backbone.

We have incorporated the Qwen3.5-9B experimental results into Table 1, with corresponding descriptions in Section 5 (lines 321–329). We have also expanded the related work discussion in Section 2 (lines 122–126) and added implementation details and comparative analyses in Appendix Table 4 and Appendix A (lines 874–892 and 937–951). We plan to further extend the comparison to Llama-3.1-8B.

References [1] Ramé et al. “Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.” NeurIPS, 2023. [2] Jang et al. “Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging.” NeurIPS Workshop on Adaptive Foundation Models (AFM), 2024. [3] Xie et al. “Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation.” ACL, 2025.

AuthorsOpenReview ↗

W2. Narrow margins and missing statistical variance

In Table 1, the lead over CPPO is only 0.014 on Qwen3.5-9B and 0.000 on Llama-3.1-8B, with no random seeds, standard deviations, or significance tests. Thank you for highlighting the importance of statistical variability. We agree that the small performance differences relative to CPPO do not establish consistent superiority without repeated runs.

However, Ready2Blend is not designed to outperform parameter-updating methods solely in alignment accuracy. Rather, our central objective is to achieve cost-effective continual alignment with competitive alignment quality and retention, without updating the LLM backbone, while supporting flexible inference-time composition of newly acquired requirements. Our results demonstrate the feasibility of this approach, achieving up to 4.3× faster training than strong post-training baselines with only four soft prompt tokens.

Accordingly, we have revised Section 5.1 (lines 362–363) to emphasize competitive alignment performance and lower adaptation cost, rather than a broad claim of performance superiority. We also repeated the Qwen3.5-9B experiments across three random seeds (42, 1000, and 1234) to quantify performance variability.

Performance Variability Across Three Random Seeds on Qwen3.5-9B (Mean ± Std.).

MethodTask-Inc. BWT ↑Task-Inc. Last ↑Pref-Inc. BWT ↑Pref-Inc. Last ↑Avg. BWT ↑Avg. Last ↑
MTL—0.692 ± 0.047—0.667 ± 0.060—0.680 ± 0.050
SeqFT-0.101 ± 0.0110.484 ± 0.008-0.123 ± 0.0020.576 ± 0.002-0.112 ± 0.0040.530 ± 0.003
CPPO-0.005 ± 0.0060.730 ± 0.003-0.062 ± 0.0070.703 ± 0.007-0.034 ± 0.0030.717 ± 0.005
EWC-0.064 ± 0.0010.528 ± 0.003-0.145 ± 0.0020.535 ± 0.001-0.104 ± 0.0010.531 ± 0.001
GEM-0.065 ± 0.0080.530 ± 0.006-0.145 ± 0.0010.535 ± 0.002-0.105 ± 0.0030.533 ± 0.002
LifeAlign0.004 ± 0.0060.728 ± 0.0330.004 ± 0.0050.603 ± 0.0100.004 ± 0.0050.666 ± 0.072
DualPrompt-0.000 ± 0.0060.728 ± 0.010-0.004 ± 0.0030.535 ± 0.001-0.002 ± 0.0040.631 ± 0.005
L2P0.050 ± 0.0220.652 ± 0.010-0.070 ± 0.0260.500 ± 0.010-0.010 ± 0.0200.576 ± 0.009
Ready2Blend0.102 ± 0.0520.733 ± 0.019-0.026 ± 0.0080.702 ± 0.0160.038 ± 0.0260.718 ± 0.017

Across three seeds, Ready2Blend achieves comparable average Last performance to CPPO (0.718 ± 0.017 vs. 0.717 ± 0.005), with closely matched results in both task-incremental (0.733 vs. 0.730) and preference-incremental alignment (0.702 vs. 0.703). Notably, Ready2Blend achieves higher average BWT (0.038 ± 0.026 vs. −0.034 ± 0.003), indicating better retention under the evaluated settings, although its Last performance exhibits greater variability. These results support competitive alignment performance without establishing statistically significant superiority.

Importantly, Ready2Blend achieves this performance with 4.3× and 2.6× faster training than CPPO in task- and preference-incremental settings, respectively (1.94 vs. 8.34 hours; 4.10 vs. 10.69 hours; Table 11), while requiring only four additional prompt tokens at inference. This supports our central objective of cost-effective continual alignment with flexible, lightweight steering.

Finally, we have incorporated these additional experimental results into Section 5.1 (lines 369–372), with detailed results and statistical analyses provided in Appendix Table 5 and Section C.1 (lines 1051–1070). We further plan to extend the evaluation to Llama-3.1-8B, including the additional LoRA-based merging baselines (Rewarded Soups, Personalized Soups, and Bone Soups) and standard soft prompt tuning with averaging, as suggested by the reviewer.

AuthorsOpenReview ↗

W3. Manuscript text contradicts experimental tables

Appendix C.2 claims the highest score across all judges, but Table 6 shows LifeAlign beats Ready2Blend under Qwen3.8-Judge (0.69 vs 0.68). The abstract claims matching post-training across all models (93.1–98.5% of MTL), but on Qwen3.5-4B (Table 8), Ready2Blend reaches only 91.5% and falls behind CPPO. Thank you for identifying these inconsistencies. We agree that the conclusions must reflect the scope and exceptions shown in the tables.

W3-1. Results across LLM judges.

The statement that Ready2Blend achieves the highest or tied-highest score under every judge was inaccurate. Under Qwen3.8-Flash-Next, LifeAlign scores 0.69, whereas Ready2Blend scores 0.68.

We have removed the universal ranking claim. Appendix C.3, lines 1133–1134, now describes Ready2Blend as competitive across the evaluated judges while acknowledging that the best-performing method varies with the judge. The experiment supports consistency of competitive performance across judges, rather than an identical ranking under every judge.

W3-2. Results on Qwen3.5-4B.

The reported 93.1–98.5% range refers to the ratios of average Last performance to the MTL reference for the main 8B–9B backbones. It does not include Qwen3.5-4B. The revised Abstract explicitly limits this range to the evaluated 8B–9B models in Abstract lines 21–23.

On Qwen3.5-4B, Ready2Blend achieves an average Last score of 0.657, corresponding to approximately 91.5% of the MTL reference of 0.718. It also falls below CPPO’s average Last of 0.679. We therefore do not claim that Ready2Blend matches the strongest post-training method at every model scale.

The supported conclusion for the smaller backbone is narrower: Ready2Blend outperforms the evaluated frozen-backbone prompt baselines in average Last and BWT, as reported in Appendix C.4. We have scoped the interpretation accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.