acceptodds
Under review as a conference paper at ICLR 2027

EchoAlign: Learning Personalized Style From Parts to Whole

Abstract

Users differ in how they prefer responses to be expressed, including their desired levels of style attributes such as formality and concision. Supporting diverse style preferences in production calls for adaptation from limited preference data and compact personalization parameters alongside a shared model. We introduce EchoAlign, a family of two-stage preference-learning methods that progressively encode personalized style guidance into a compact soft prompt while keeping the language-model backbone frozen. The first stage uses attribute-level preference pairs to train dedicated prompt segments, giving each segment a focused learning objective. The second stage uses overall-style preference pairs to adapt their combination so that they jointly guide generation according to the user's complete style preference. Each complete style preference has a separately learned prompt whose 256 token embeddings can be precomputed for reuse across questions, supporting pluggable personalization with shared model weights. Using approximately 100 preference pairs per profile, EchoAlign achieves relative improvements in style alignment of 55.8–62.5% over unadapted base models and 2.9–11.3% over the strongest tested LoRA baselines across three backbones. We further apply EchoAlign to personalized message generation in production, where a randomized A/B test shows a 6.49% relative increase in click-through rate. This real-user behavioral feedback suggests improved satisfaction with the generated messages.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Personalization, Response Style Alignment, Direct Preference Optimization, Multi-Attribute Controllable Generation, Parameter-Efficient Fine-Tuning, Soft Prompt Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QBRdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed EchoAlign, which learns a 256-token-long soft prompt for each style profile. The authors defined 4 style-related attributes: formality, concision, warmth, and creativity, where each attribute has 5 levels. A style profile has a target level for each attribute, resulting in a total of 625 profiles. The soft prompts are trained using DPO, with a small amount of commercial LLM-generated data, in two stages. In the first stage, the author breaks the 256 tokens into four 64-token groups, one for each attribute, and then trains each of the groups separately with the other groups set to "neutral". In the second stage, the concatenated groups are trained together. The authors presented several alternatives for the second stage, including FUSE (only train an affine map), and REFINE (train the soft prompt directly).

Critical strengths
  1. The evaluation covers the full profile grid across three backbones and three seeds, and the result reporting is transparent.

  2. The setup is practical. The soft prompt is compact, and all users share the same frozen backbone.

  3. The paper includes a randomized online A/B test on real users, showing statistically significant improvement.

Critical weaknesses
  1. The methodological contribution is very limited. The method is basically standard DPO with soft prompt tuning. The paper's own results in Table 2 show that segmented and unsegmented training perform similarly, which undercuts the importance of the main design contribution.

  2. Per-profile training does not scale or generalize well. Every profile needs separate training; it does not generalize well to profiles with few or no user data. The soft prompts also need to be retrained when the attribute set expands, or the backbone model is changed.

  3. The offline evaluation only covers one synthetic, specialized, small-scale dataset created by the authors. The baselines are very limited: the only trained baselines are the authors' own per-profile LoRA variants, while every directly relevant research method the paper itself cites is left out, including but not limited to SteerLM, Janus, P-RLHF, Personalized Soups, and Tailor, etc.

  4. The online test setting is opaque and weakly related to the claimed contribution: the authors used a different 35B backbone, with most of its training/deployment hyperparameters not disclosed; the method was only applied to "eligible" users, where "eligible" lacks a precise definition; the analysis is interim and cannot be recomputed.

Questions for authors

Please see weaknesses above.

Reviewer_VVHROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EchoAlign, a two-stage method that learns personalized style prompts while keeping the language-model backbone frozen. It first trains separate soft-prompt segments for formality, concision, warmth, and creativity using DPO, then adapts their combination using overall-style preferences. Each complete style profile receives a reusable 256-vector prompt. The offline experiments use Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Ministral-3-8B. The authors construct a synthetic style-preference dataset using a proprietary LLM and a benchmark to evaluate style alignment on held-out questions. Performance is measured through a model judge’s 0–10 style-fit score, reported as an equally weighted average across all profiles and a user-frequency-weighted average over 64 common profiles. The selected EchoAlign variants outperform explicit prompting, in-context learning, per-profile LoRA, and direct soft-prompt training on the full-profile average. A separate production A/B test uses Qwen3.5-35B-A3B for personalized push-message generation and reports a 6.49% relative increase in impression-based click-through rate over explicit style prompting.

Critical strengths
  • Practical motivation and compact adaptation: Controlling response style is useful for personalized assistants. The frozen backbone and precomputable per-profile prompts provide a practical approach to supporting multiple style preferences with shared model weights.

  • Additional evaluation beyond the primary judge: Three external model judges broadly corroborate the sampled method rankings. A randomized production A/B test reports a 6.49% relative increase in impression-based click-through rate over explicit style prompting, providing evidence of practical value in the tested deployment.

  • Informative ablations and transparent limitations: The paper examines segmentation, data organization, adaptation variants, and cross-attribute coupling. It reports backbone-dependent failures and acknowledges limitations concerning unseen profiles, evaluation breadth, and the interpretation of click engagement.

Critical weaknesses
  • Limited validation of the attribute formulation: The four attributes have precedents in existing datasets and deployed style presets, but these precedents do not establish that this particular selection and five-level discretization adequately capture users’ style preferences. The formulation remains a design choice whose sensitivity is insufficiently explored. What evidence supports this attribute set, and would the method remain effective with alternative or additional attributes?

  • Missing optimized textual-prompt baseline: The paper compares against explicit style instructions and in-context learning, but does not evaluate automatic optimization of textual system prompts using teacher feedback, such as GEPA. This is a relevant alternative given the reliance on teacher-generated supervision and model-based style evaluation. Does EchoAlign outperform an optimized textual prompt under comparable feedback and optimization budgets?

  • Unclear contribution of attribute-specific segmentation: Table 2 shows comparable performance when attribute pairs are shuffled across segments or when segmentation is removed. The authors interpret this as supporting independent training and reusable initialization, but these results do not establish that the semantic decomposition into attributes contributes to the gains. What evidence distinguishes the benefit of attribute-specific segmentation from that of the two-stage optimization procedure itself?

Overall, I find the contribution insufficiently convincing. The ablations do not clearly establish the benefit of attribute-specific segmentation or explain how the learned segments interact. The lack of comparison with text-prompt optimization methods using LLM feedback also leaves the advantage of the proposed approach unclear.

[1] Agrawal et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” ICLR, 2026. https://arxiv.org/abs/2507.19457

Questions for authors
  • How should a practitioner choose between FUSE and REFINE+MIX for a new backbone? Were the highlighted variants selected using a separate validation set or the reported evaluation scores?
  • How sensitive is soft-prompt training to initialization and learning rate? In particular, what causes FUSE’s high response-collapse rate on Ministral, and how can practitioners select a stable configuration for a new backbone?
Reviewer_8eVKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns a separate 256-vector soft prompt for each of 625 style profiles. Each profile specifies one of five levels along four dimensions: formality, concision, warmth, and creativity. The backbone model remains frozen, and the soft prompts are trained with DPO in two stages. In Stage 1, the prompt is divided into four 64-vector blocks, one for each style attribute. Each block is trained independently using 20 single-attribute preference pairs, where the target level is preferred over a contrasting extreme while the remaining attributes are fixed at the neutral level of 3. In Stage 2, the four blocks are concatenated and further adapted using overall-style preference pairs. These pairs compare the target profile against two distant profiles and are ranked by a judge. The paper evaluates five Stage-2 variants, with particular emphasis on FUSE, which learns a (d x d) affine transformation over the fixed blocks, and REFINE+MIX, which further updates the embeddings using a mixture of attribute-level and overall-style pairs.

The offline evaluation covers all 625 profiles across 17 held-out questions on Qwen2.5-7B, Llama-3.1-8B, and Ministral-3-8B, using three random seeds. Taking the better of FUSE and REFINE+MIX for each backbone, the method outperforms the strongest profile-prompted LoRA baseline by 0.69, 0.31, and 0.18 points, respectively, on a 0–10 judge score. The paper also reports a production A/B test for push-message generation, where the method achieves a 6.49% relative improvement in impression CTR over explicit style prompting.

Critical strengths
  • The evaluation is broad. It covers the full profile grid, three backbone models, and three random seeds. The prefix-based methods and LoRA are trained on a verified identical preference corpus, which makes the comparison cleaner. Failed or collapsed generations are assigned a score of zero rather than excluded from evaluation. The main prespecified comparisons are also re-scored by three external judges with question-cluster bootstrap confidence intervals, and the primary judge is evaluated for both repeatability and agreement with human-annotated datasets.
  • The repeated-measures tracking and coupling analysis in Section 4.4 and Figure 5 provides a useful way to characterize multi-attribute control beyond aggregate performance.
  • The randomized production test is real evidence. It is reported over the whole bucket, including fallback users, and includes a stratified analysis and stratum-level heterogeneity.
Critical weaknesses
  • The paper’s own results do not clearly support the “parts-to-whole” premise. Table 2 shows little consistent benefit from grouping examples by attribute or training dedicated segments. On Ministral, the canonical grouped-and-segmented design is actually the worst of the four settings. This weakens the claim that focused, attribute-specific blocks are the key mechanism. FUSE is also less constrained than its description suggests. If EconcatE_{\text{concat}} has full row rank, the learned affine map can represent essentially any (256 ×\times d) prefix, making FUSE closer to an overparameterized reparameterization of prompt tuning than a method that preserves attribute-specific structure. This may also relate to its instability. The coupling results point in the same direction: the decomposed variants show more cross-attribute coupling than PP+LoRA or MONOSP (0.41–0.51 vs. 0.25–0.33). Finally, the main compositional payoff is never tested. Since Stage 1 depends only on attribute and level, a shared library of (4 ×\times 5=20) blocks should in principle suffice, yet the paper trains 2,500 separate blocks per backbone and seed. Sharing blocks across profiles and testing unseen attribute combinations would be a much stronger test of the proposed decomposition.
  • No single variant is consistently robust, and the headline result is selected per backbone using test performance. Table 1 reports whichever of FUSE or REFINE+MIX has the higher Full-625 mean for each backbone, which amounts to test-set selection within a larger family of five variants. This matters because the variants have very different failure modes. FUSE collapses on 23% of Ministral outputs, while FUSE, JOINT, and REFINE all degrade sharply on Llama’s most frequent profile in two of three seeds. REFINE+MIX, despite being chosen for stability and used in production, underperforms all PP+LoRA variants and MONOSP on Llama. REFINE is the only variant that beats PP+LoRA across all three Full-625 columns, but it shares the Llama failure and still collapses on some Ministral outputs.
  • The experimental scope is fairly narrow. The evaluation uses only 20 training questions and 17 test questions from the same scenario categories, so each profile is trained on many preference pairs but little question diversity. The “limited preference data” framing is also somewhat misleading because the data is synthetic and teacher-generated, with roughly 62k pairs overall. The preference construction is quite coarse: targets are compared against extreme or maximally distant profiles. This may help explain the weak separation between adjacent levels and at the extremes in Figure 4. Comparisons against neighboring profiles would provide a more direct test of calibration. Finally, the abstract’s 55.8–62.5% improvement is measured against a model with no style profile at all. ProfilePrompt is the more meaningful headline baseline.
Questions for authors
  • Can Stage-1 blocks be shared across profiles that share an attribute level? How does a 20-block shared library compare with per-profile blocks, and how does it perform on held-out profile combinations?
  • How does a single profile-conditioned LoRA, trained on all profiles’ data with the profile text, compare? What about prompt tuning initialized from the target-profile text?
  • Table 1 bolds the selected member rather than the best result. For example, REFINE (6.94) and JOINT (6.88) beat the bolded FUSE (6.85) on Qwen Top-64. This is easy to misread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.