- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposed EchoAlign, which learns a 256-token-long soft prompt for each style profile. The authors defined 4 style-related attributes: formality, concision, warmth, and creativity, where each attribute has 5 levels. A style profile has a target level for each attribute, resulting in a total of 625 profiles. The soft prompts are trained using DPO, with a small amount of commercial LLM-generated data, in two stages. In the first stage, the author breaks the 256 tokens into four 64-token groups, one for each attribute, and then trains each of the groups separately with the other groups set to "neutral". In the second stage, the concatenated groups are trained together. The authors presented several alternatives for the second stage, including FUSE (only train an affine map), and REFINE (train the soft prompt directly).
Critical strengths
-
The evaluation covers the full profile grid across three backbones and three seeds, and the result reporting is transparent.
-
The setup is practical. The soft prompt is compact, and all users share the same frozen backbone.
-
The paper includes a randomized online A/B test on real users, showing statistically significant improvement.
Critical weaknesses
-
The methodological contribution is very limited. The method is basically standard DPO with soft prompt tuning. The paper's own results in Table 2 show that segmented and unsegmented training perform similarly, which undercuts the importance of the main design contribution.
-
Per-profile training does not scale or generalize well. Every profile needs separate training; it does not generalize well to profiles with few or no user data. The soft prompts also need to be retrained when the attribute set expands, or the backbone model is changed.
-
The offline evaluation only covers one synthetic, specialized, small-scale dataset created by the authors. The baselines are very limited: the only trained baselines are the authors' own per-profile LoRA variants, while every directly relevant research method the paper itself cites is left out, including but not limited to SteerLM, Janus, P-RLHF, Personalized Soups, and Tailor, etc.
-
The online test setting is opaque and weakly related to the claimed contribution: the authors used a different 35B backbone, with most of its training/deployment hyperparameters not disclosed; the method was only applied to "eligible" users, where "eligible" lacks a precise definition; the analysis is interim and cannot be recomputed.
Questions for authors
Please see weaknesses above.