- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies personalized multi-objective alignment with limited client feedback. It proposes APO, which clusters clients by their preference-weighted losses and adjustment vectors, learns a shared aligner initialization within each cluster, and refines it through adaptation-aware meta-training. The method combines gradient descent with controlled ascent to handle competing objectives. Experiments on Fed-ChatbotPA and UltraFeedback, using two 3B backbones, report improved alignment after adaptation with 20 local examples. The paper also provides conditional optimization and adaptation bounds.
Critical strengths
- The problem is relevant: users have different preferences, but collecting enough feedback to train a separate model for each user is costly. Collaborative initialization is a reasonable approach to this setting.
- The method considers both conflicts between clients and trade-offs within each client. Grouping clients by their current optimization needs, rather than preference weights alone, is a useful design choice.
Critical weaknesses
- The construction of the multi-objective training data is underspecified. Appendix B reports client sample counts and data partitioning, but does not explain how client preference vectors are generated. Appendix A illustrates a chosen/rejected pair for one objective without describing how objective-specific pairs are obtained from the source datasets. These details are needed to reproduce the client losses and understand what preference heterogeneity the experiments evaluate.
- The experiments do not fully establish the source of the gains. Table 2 evaluates the clustering criterion, but there is no separate ablation of controlled ascent or direct measurement of reduced gradient interference. The APO--FedAvg comparison also lacks a clearly reported control for the additional training budget of Phase II. Finally, Appendix E describes FSPO as SFT initialization followed by per-client IPO, leaving unclear whether its original cross-user, preference-conditioned training objective is retained. These gaps limit the conclusions that can be drawn about the proposed components and their advantage over existing methods.
- The evidence for data efficiency is limited to a fixed adaptation budget of 20 examples per client. The results demonstrate improved performance at this budget, but do not establish whether APO retains its advantage with less feedback or requires fewer examples to reach comparable performance. Comparing APO with FedAvg and the main few-shot baseline across several adaptation budgets, such as 5, 10, and 20 examples, would better support the paper's emphasis on personalization under data scarcity.
Questions for authors
- Does the FSPO baseline retain the cross-user, preference-conditioned training objective of the original method? Please clarify how this objective corresponds to the SFT initialization and per-client IPO described in Appendix E, and explain any modifications.
- Could the authors isolate the effect of controlled ascent by disabling it while keeping clustering and meta-training unchanged?
- Could the authors compare Phase II against continuing Phase I with a matched additional training budget and identical test-time adaptation? This comparison would help distinguish the benefit of the adaptation-aware update rule from that of additional training.
5 comments
Q2: The experiments do not fully establish the source of the gains. Table 2 evaluates the clustering criterion, but there is no separate ablation of controlled ascent or direct measurement of reduced gradient interference. The APO--FedAvg comparison also lacks a clearly reported control for the additional training budget of Phase II. Finally, Appendix E describes FSPO as SFT initialization followed by per-client IPO, leaving unclear whether its original cross-user, preference-conditioned training objective is retained. These gaps limit the conclusions that can be drawn about the proposed components and their advantage over existing methods.
A2: We thank the reviewer for asking us to separate controlled ascent from the clustering result in Table 2. The experiment below keeps Bottleneck-Adjustment Clustering, FedAvg aggregation, and adaptation-aware meta-training fixed, and removes only the controlled-ascent slack in the local linear program. It is run on UltraFeedback with Llama-3.2-3B-Instruct, the eight-cluster setting of Table 1. APO is the reported method, and its column reproduces the Table 1 entries. APO w/o control ascent is the same pipeline with the constraints below.
How controlled ascent is removed. In Phase I the objective (6a) is unchanged in both regimes: balancing descent when , and uniform descent when . The bottleneck constraint (6c) is also unchanged. Controlled ascent sits in (6b). For ,
The right-hand side is non-positive, so that non-bottleneck loss may increase, up to the balancing direction. The local step is , and this negative slack is an ascent in the loss. We set that right-hand side to 0 and impose the inequality on every non-bottleneck objective,
\qquad \forall\\, j\in[m]\setminus J^*. \tag{6b'}$$ Together with (6c), the two constraints reduce to one common-descent restriction, $$\mu_i^\top G_i^\top g_i^{(j)} \\;\ge\\; 0, \qquad \forall\\, j\in[m].$$ Every objective is then non-increasing along the step. Phase II keeps the same balancing objective and replaces its constraints (6b)–(6c) by the same restriction, $$\mu^{*} =\arg\max_{\mu\in S^{m}} \mu^{\top} G^{\top} G\\,\boldsymbol{A}_{i} \quad\text{s.t.}\quad \mu^{\top} G^{\top} g^{(j)}\\;\ge\\; 0, \\;\\;\forall\\, j\in[m]. \tag{7'}$$ Cluster assignment, server aggregation, the meta-training outer loop, and the 20-example adaptation are unchanged. The only change is that $\mu$ may no longer buy progress on the balancing direction by increasing a non-bottleneck loss. **Protocol.** Each evaluated model is a point in the normalized objective space of helpfulness, honesty, and truthfulness, with every coordinate in $[0,1]$. For each preference pair we take the checkpoint in 140–180 and the adaptation round in $\\{1,2,3\\}$ with the highest preference-weighted normalized score, and we evaluate HV and IGD at that point. HV is the hypervolume of this single point with respect to the origin, the product of its three normalized coordinates. Each cluster entry is the mean of its five preference vectors. IGD uses the nondominated front of the original seven methods. The average row is the unweighted mean of the eight clusters. Entries are rounded independently to two decimals, as in the paper. Higher Score/HV and lower IGD are better, and bold marks the strictly better value. **Table 2: Controlled-ascent ablation on UltraFeedback/Llama-3.2-3B-Instruct** | Cluster | APO Score | APO HV | APO IGD | APO w/o control ascent Score | APO w/o control ascent HV | APO w/o control ascent IGD | | --- | --- | --- | --- | --- | --- | --- | | $C_0$ | **0.78** (+5.4%) | **0.48** (+17.1%) | **0.10** (−23.1%) | 0.74 | 0.41 | 0.13 | | $C_1$ | **0.79** (+6.8%) | **0.48** (+14.3%) | **0.04** (−55.6%) | 0.74 | 0.42 | 0.09 | | $C_2$ | **0.79** (+9.7%) | **0.52** (+33.3%) | **0.04** (−71.4%) | 0.72 | 0.39 | 0.14 | | $C_3$ | **0.79** (+6.8%) | **0.51** (+21.4%) | **0.06** (−53.8%) | 0.74 | 0.42 | 0.13 | | $C_4$ | **0.74** (+4.2%) | **0.40** (+5.3%) | **0.04** (−60.0%) | 0.71 | 0.38 | 0.10 | | $C_5$ | **0.76** (+2.7%) | **0.43** (+16.2%) | **0.10** (−28.6%) | 0.74 | 0.37 | 0.14 | | $C_6$ | **0.84** (+5.0%) | **0.58** (+13.7%) | **0.04** (−60.0%) | 0.80 | 0.51 | 0.10 | | $C_7$ | 0.82 (+0.0%) | **0.47** (+6.8%) | **0.06** (−45.5%) | 0.82 | 0.44 | 0.11 | | Avg. | **0.79** (+5.3%) | **0.48** (+14.3%) | **0.06** (−50.0%) | 0.75 | 0.42 | 0.12 |Compared with APO w/o control ascent, APO improves the average Score by 5.3% (0.75 to 0.79) and HV by 14.3% (0.42 to 0.48), while reducing IGD by 50.0% (0.12 to 0.06). APO achieves a higher Score in seven of the eight clusters and higher HV and lower IGD in all eight. The largest improvement occurs in , where APO increases Score by 9.7% (0.72 to 0.79) and HV by 33.3% (0.39 to 0.52), while reducing IGD by 71.4% (0.14 to 0.04). is an informative boundary case: the rounded Score remains unchanged at 0.82 (0.815 for APO versus 0.816 without controlled ascent), whereas APO still improves HV by 6.8% (0.44 to 0.47) and reduces IGD by 45.5% (0.11 to 0.06). Thus, even when the scalarized Score is nearly unchanged, controlled ascent yields a solution with substantially better three-objective quality and closer coverage of the empirical Pareto front. This result supports the role of the controlled trade-off permitted by constraint (6b), which the common-descent constraint (6b′) removes.
For the FSPO baseline, we preserve the central two-stage and preference-conditioned design of the original method while adapting it to our per-client personalization protocol. Specifically, the original FSPO first performs Fewshot Pref-FT and then initializes Fewshot IPO from the resulting shared checkpoint; both stages train a single model across users, with each prediction conditioned on a user’s few-shot preferences. Personalization to a new user is then performed primarily through this few-shot context rather than through user-specific parameter updates.
Our implementation follows the same two-stage structure. In the first stage, we train one shared SFT initialization by pooling Roleplay revision examples from multiple personas. Each revision prompt is explicitly conditioned on a (Helpful, Honest, Truthful) preference vector, and the SFT objective maximizes the token likelihood of the preferred revision. In the second stage, we initialize from this shared checkpoint and apply IPO using each client’s preference-conditioned support pairs. Preference information remains explicitly present in the prompt throughout both stages.
To make FSPO comparable with the other personalized baselines in our evaluation, which produce an adapted model for each client, we perform the IPO stage separately for each client and evaluate the resulting client-specific checkpoint. Thus, our implementation retains FSPO’s shared SFT initialization, two-stage SFT-to-IPO training, and preference-conditioned optimization, but replaces the original inference-time, context-only personalization with client-specific IPO adaptation. We will clarify this adaptation and its motivation in Appendix E.
A1: Thank you for pointing out that these details were underspecified. We clarify below how client preference vectors and objective-specific preference pairs are constructed.
Client preference vectors
For a dataset with m preference objectives, we independently generate one preference vector for each client as
\qquad \lambda_i^{(j)} > 0,\quad \sum_{j=1}^{m}\lambda_i^{(j)}=1.$$ We use the fixed random seed 2023 for reproducibility. Specifically, $m=2$ for Fed-ChatbotPA, corresponding to helpfulness and safety, and $m=3$ for UltraFeedback, corresponding to helpfulness, honesty, and truthfulness. The symmetric Dirichlet distribution with concentration parameter $\mathbf{1}_m$ is uniform over the $m$-dimensional probability simplex. It therefore generates clients covering different trade-offs without manually favoring any objective. A larger $\lambda_i^{(j)}$ indicates that client $i$ places greater importance on objective $j$, meaning that the corresponding loss should be more tightly controlled. Preference vectors belonging to held-out test clients are excluded from collaborative training. #### Objective-specific chosen/rejected pairs The chosen/rejected pairs are not synthesized by APO and are not obtained by assigning the same pair to every objective. The source data provide preference pairs annotated for individual preference dimensions. For each objective $j$, we retain the corresponding dimension-specific pair $$(x,\\; y_{j}^{+},\\; y_{j}^{-}),$$ where $y_j^{+}$ is preferred over $y_j^{-}$ according to objective $j$. For example, the helpfulness loss is computed from pairs annotated for helpfulness, whereas the safety loss is computed from pairs annotated for safety. For client $i$, the pairs associated with objective $j$ define a separate DPO loss $$l_i^{(j)}(w) = -\mathbb{E}_{(x,y_j^{+},y_j^{-})\sim\mathcal D_i^{(j)}} \left[ \log \sigma \left( \beta \log\frac{\pi_w(y_j^{+}\mid x)} {\pi_{\mathrm{ref}}(y_j^{+}\mid x)} - \beta \log\frac{\pi_w(y_j^{-}\mid x)} {\pi_{\mathrm{ref}}(y_j^{-}\mid x)} \right) \right].$$ This produces the client-specific multi-objective loss vector $$\boldsymbol l_i(w) = \left( l_i^{(1)}(w),\ldots,l_i^{(m)}(w) \right)$$ and the corresponding gradient matrix $$G_i(w) = \left[ \nabla l_i^{(1)}(w),\ldots,\nabla l_i^{(m)}(w) \right].$$ Thus, each column of $G_i$ contains the training information obtained from the chosen/rejected pairs of one objective. #### How APO combines the objectives APO does not simply merge all pairs or minimize an ordinary weighted sum. It combines the objective-specific losses with the client’s preference vector to compute the weighted losses $$\lambda_i^{(j)}l_i^{(j)}(w).$$ These quantities determine the client’s bottleneck objectives and preference-adjustment vector. APO then solves for coefficients $\boldsymbol{\mu}_i$ and constructs the update direction $$d_i=G_i\boldsymbol{\mu}_i.$$ Consequently, the objective-specific chosen/rejected pairs provide the gradient information for each preference dimension, while $\boldsymbol{\lambda}_i$ specifies the client’s desired trade-off. The APO optimization rule determines how these gradients should be combined while protecting bottleneck objectives and moving the model toward the client-specific preference ray. We will add these details to the appendix, including the Dirichlet distribution, random seed, objective ordering for each dataset, dimension-specific pair construction, and the mapping from these pairs to the APO update. #### Text added to the appendix > **Construction of client preferences and objective-specific feedback.** For a dataset with $m$ objectives, we independently sample each client preference vector as $\boldsymbol{\lambda}_i\sim\operatorname{Dirichlet}(\mathbf{1}_m)$, using random seed 2023. Hence, every vector has strictly positive entries summing to one, and the resulting client preferences are uniformly distributed over the probability simplex. We use $m=2$ for Fed-ChatbotPA (helpfulness and safety) and $m=3$ for UltraFeedback (helpfulness, honesty, and truthfulness). The source datasets provide chosen/rejected pairs annotated for individual objectives. For every objective $j$, we use its corresponding pairs to compute a separate DPO loss $l_i^{(j)}$ and gradient $g_i^{(j)}$. APO forms the loss vector $\boldsymbol l_i=(l_i^{(1)},\ldots,l_i^{(m)})$ and gradient matrix $G_i=[g_i^{(1)},\ldots,g_i^{(m)}]$, combines them with $\boldsymbol{\lambda}_i$ to identify the client’s bottleneck and adjustment vector, and updates the model along $G_i\boldsymbol{\mu}_i$, where $\boldsymbol{\mu}_i$ is obtained from the APO optimization rule.