acceptodds
Under review as a conference paper at ICLR 2027

Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

Abstract

Real-world users often exhibit highly heterogeneous preferences over multiple objectives for LLM responses. A lightweight aligner can tailor these responses to individual preferences, but scarce user-specific feedback makes personalized training difficult. Learning shared initializations across users can support few-shot adaptation. However, heterogeneous preferences and competing objectives cause gradient conflicts across users and within each user, hindering effective initialization learning. This raises a central question: **how can we collaboratively learn aligner initializations that support few-shot adaptation to diverse user preferences?** To answer this question, we propose **A**pproximate **P**areto **O**ptimality (APO). We first group users whose updates are compatible, so that their information can be combined with less interference. Within each group, we combine gradient descent with controlled ascent to coordinate competing objectives and move towards preference-specific points on the Pareto front. This produces an initialization that is close to the optima of the users in the group. We then iteratively refine it using updates from few-shot local adaptation, making it more effective for personalization. Furthermore, we establish conditional suboptimality bounds for a one-local-step collaborative update and characterize how initialization error affects subsequent stochastic adaptation. Experiments on Fed-ChatbotPA and UltraFeedback show consistent improvements over existing methods using only 20 local examples.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM preference alignment, Collaborative learning, Data deficiency, Pareto optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WxZmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies personalized multi-objective alignment with limited client feedback. It proposes APO, which clusters clients by their preference-weighted losses and adjustment vectors, learns a shared aligner initialization within each cluster, and refines it through adaptation-aware meta-training. The method combines gradient descent with controlled ascent to handle competing objectives. Experiments on Fed-ChatbotPA and UltraFeedback, using two 3B backbones, report improved alignment after adaptation with 20 local examples. The paper also provides conditional optimization and adaptation bounds.

Critical strengths
  1. The problem is relevant: users have different preferences, but collecting enough feedback to train a separate model for each user is costly. Collaborative initialization is a reasonable approach to this setting.
  2. The method considers both conflicts between clients and trade-offs within each client. Grouping clients by their current optimization needs, rather than preference weights alone, is a useful design choice.
Critical weaknesses
  1. The construction of the multi-objective training data is underspecified. Appendix B reports client sample counts and data partitioning, but does not explain how client preference vectors are generated. Appendix A illustrates a chosen/rejected pair for one objective without describing how objective-specific pairs are obtained from the source datasets. These details are needed to reproduce the client losses and understand what preference heterogeneity the experiments evaluate.
  2. The experiments do not fully establish the source of the gains. Table 2 evaluates the clustering criterion, but there is no separate ablation of controlled ascent or direct measurement of reduced gradient interference. The APO--FedAvg comparison also lacks a clearly reported control for the additional training budget of Phase II. Finally, Appendix E describes FSPO as SFT initialization followed by per-client IPO, leaving unclear whether its original cross-user, preference-conditioned training objective is retained. These gaps limit the conclusions that can be drawn about the proposed components and their advantage over existing methods.
  3. The evidence for data efficiency is limited to a fixed adaptation budget of 20 examples per client. The results demonstrate improved performance at this budget, but do not establish whether APO retains its advantage with less feedback or requires fewer examples to reach comparable performance. Comparing APO with FedAvg and the main few-shot baseline across several adaptation budgets, such as 5, 10, and 20 examples, would better support the paper's emphasis on personalization under data scarcity.
Questions for authors
  1. Does the FSPO baseline retain the cross-user, preference-conditioned training objective of the original method? Please clarify how this objective corresponds to the SFT initialization and per-client IPO described in Appendix E, and explain any modifications.
  2. Could the authors isolate the effect of controlled ascent by disabling it while keeping clustering and meta-training unchanged?
  3. Could the authors compare Phase II against continuing Phase I with a matched additional training budget and identical test-time adaptation? This comparison would help distinguish the benefit of the adaptation-aware update rule from that of additional training.
5 comments

A1: Thank you for pointing out that these details were underspecified. We clarify below how client preference vectors and objective-specific preference pairs are constructed.

Client preference vectors

For a dataset with mm preference objectives, we independently generate one preference vector for each client as

\qquad \lambda_i^{(j)} > 0,\quad \sum_{j=1}^{m}\lambda_i^{(j)}=1.$$ We use the fixed random seed 2023 for reproducibility. Specifically, $m=2$ for Fed-ChatbotPA, corresponding to helpfulness and safety, and $m=3$ for UltraFeedback, corresponding to helpfulness, honesty, and truthfulness. The symmetric Dirichlet distribution with concentration parameter $\mathbf{1}_m$ is uniform over the $m$-dimensional probability simplex. It therefore generates clients covering different trade-offs without manually favoring any objective. A larger $\lambda_i^{(j)}$ indicates that client $i$ places greater importance on objective $j$, meaning that the corresponding loss should be more tightly controlled. Preference vectors belonging to held-out test clients are excluded from collaborative training. #### Objective-specific chosen/rejected pairs The chosen/rejected pairs are not synthesized by APO and are not obtained by assigning the same pair to every objective. The source data provide preference pairs annotated for individual preference dimensions. For each objective $j$, we retain the corresponding dimension-specific pair $$(x,\\; y_{j}^{+},\\; y_{j}^{-}),$$ where $y_j^{+}$ is preferred over $y_j^{-}$ according to objective $j$. For example, the helpfulness loss is computed from pairs annotated for helpfulness, whereas the safety loss is computed from pairs annotated for safety. For client $i$, the pairs associated with objective $j$ define a separate DPO loss $$l_i^{(j)}(w) = -\mathbb{E}_{(x,y_j^{+},y_j^{-})\sim\mathcal D_i^{(j)}} \left[ \log \sigma \left( \beta \log\frac{\pi_w(y_j^{+}\mid x)} {\pi_{\mathrm{ref}}(y_j^{+}\mid x)} - \beta \log\frac{\pi_w(y_j^{-}\mid x)} {\pi_{\mathrm{ref}}(y_j^{-}\mid x)} \right) \right].$$ This produces the client-specific multi-objective loss vector $$\boldsymbol l_i(w) = \left( l_i^{(1)}(w),\ldots,l_i^{(m)}(w) \right)$$ and the corresponding gradient matrix $$G_i(w) = \left[ \nabla l_i^{(1)}(w),\ldots,\nabla l_i^{(m)}(w) \right].$$ Thus, each column of $G_i$ contains the training information obtained from the chosen/rejected pairs of one objective. #### How APO combines the objectives APO does not simply merge all pairs or minimize an ordinary weighted sum. It combines the objective-specific losses with the client’s preference vector to compute the weighted losses $$\lambda_i^{(j)}l_i^{(j)}(w).$$ These quantities determine the client’s bottleneck objectives and preference-adjustment vector. APO then solves for coefficients $\boldsymbol{\mu}_i$ and constructs the update direction $$d_i=G_i\boldsymbol{\mu}_i.$$ Consequently, the objective-specific chosen/rejected pairs provide the gradient information for each preference dimension, while $\boldsymbol{\lambda}_i$ specifies the client’s desired trade-off. The APO optimization rule determines how these gradients should be combined while protecting bottleneck objectives and moving the model toward the client-specific preference ray. We will add these details to the appendix, including the Dirichlet distribution, random seed, objective ordering for each dataset, dimension-specific pair construction, and the mapping from these pairs to the APO update. #### Text added to the appendix > **Construction of client preferences and objective-specific feedback.** For a dataset with $m$ objectives, we independently sample each client preference vector as $\boldsymbol{\lambda}_i\sim\operatorname{Dirichlet}(\mathbf{1}_m)$, using random seed 2023. Hence, every vector has strictly positive entries summing to one, and the resulting client preferences are uniformly distributed over the probability simplex. We use $m=2$ for Fed-ChatbotPA (helpfulness and safety) and $m=3$ for UltraFeedback (helpfulness, honesty, and truthfulness). The source datasets provide chosen/rejected pairs annotated for individual objectives. For every objective $j$, we use its corresponding pairs to compute a separate DPO loss $l_i^{(j)}$ and gradient $g_i^{(j)}$. APO forms the loss vector $\boldsymbol l_i=(l_i^{(1)},\ldots,l_i^{(m)})$ and gradient matrix $G_i=[g_i^{(1)},\ldots,g_i^{(m)}]$, combines them with $\boldsymbol{\lambda}_i$ to identify the client’s bottleneck and adjustment vector, and updates the model along $G_i\boldsymbol{\mu}_i$, where $\boldsymbol{\mu}_i$ is obtained from the APO optimization rule.

Q2: The experiments do not fully establish the source of the gains. Table 2 evaluates the clustering criterion, but there is no separate ablation of controlled ascent or direct measurement of reduced gradient interference. The APO--FedAvg comparison also lacks a clearly reported control for the additional training budget of Phase II. Finally, Appendix E describes FSPO as SFT initialization followed by per-client IPO, leaving unclear whether its original cross-user, preference-conditioned training objective is retained. These gaps limit the conclusions that can be drawn about the proposed components and their advantage over existing methods.

A2: We thank the reviewer for asking us to separate controlled ascent from the clustering result in Table 2. The experiment below keeps Bottleneck-Adjustment Clustering, FedAvg aggregation, and adaptation-aware meta-training fixed, and removes only the controlled-ascent slack in the local linear program. It is run on UltraFeedback with Llama-3.2-3B-Instruct, the eight-cluster setting of Table 1. APO is the reported method, and its column reproduces the Table 1 entries. APO w/o control ascent is the same pipeline with the constraints below.

How controlled ascent is removed. In Phase I the objective (6a) is unchanged in both regimes: balancing descent when 1βi=1\mathbb{1}_{\beta_i}=1, and uniform descent when 1βi=0\mathbb{1}_{\beta_i}=0. The bottleneck constraint (6c) is also unchanged. Controlled ascent sits in (6b). For j∈Jˉ∖J∗j\in\bar J\setminus J^*,

μi⊤Gi⊤gi(j);≥;Ai⊤Gi⊤gi(j),1J.\mu_i^\top G_i^\top g_i^{(j)} \\;\ge\\; \boldsymbol{A}_i^\top G_i^\top g_i^{(j)}\\,\mathbb{1}_J.

The right-hand side is non-positive, so that non-bottleneck loss may increase, up to the balancing direction. The local step is w←w−ηGiμiw\leftarrow w-\eta G_i\mu_i, and this negative slack is an ascent in the loss. We set that right-hand side to 0 and impose the inequality on every non-bottleneck objective,

\qquad \forall\\, j\in[m]\setminus J^*. \tag{6b'}$$ Together with (6c), the two constraints reduce to one common-descent restriction, $$\mu_i^\top G_i^\top g_i^{(j)} \\;\ge\\; 0, \qquad \forall\\, j\in[m].$$ Every objective is then non-increasing along the step. Phase II keeps the same balancing objective and replaces its constraints (6b)–(6c) by the same restriction, $$\mu^{*} =\arg\max_{\mu\in S^{m}} \mu^{\top} G^{\top} G\\,\boldsymbol{A}_{i} \quad\text{s.t.}\quad \mu^{\top} G^{\top} g^{(j)}\\;\ge\\; 0, \\;\\;\forall\\, j\in[m]. \tag{7'}$$ Cluster assignment, server aggregation, the meta-training outer loop, and the 20-example adaptation are unchanged. The only change is that $\mu$ may no longer buy progress on the balancing direction by increasing a non-bottleneck loss. **Protocol.** Each evaluated model is a point in the normalized objective space of helpfulness, honesty, and truthfulness, with every coordinate in $[0,1]$. For each preference pair we take the checkpoint in 140–180 and the adaptation round in $\\{1,2,3\\}$ with the highest preference-weighted normalized score, and we evaluate HV and IGD at that point. HV is the hypervolume of this single point with respect to the origin, the product of its three normalized coordinates. Each cluster entry is the mean of its five preference vectors. IGD uses the nondominated front of the original seven methods. The average row is the unweighted mean of the eight clusters. Entries are rounded independently to two decimals, as in the paper. Higher Score/HV and lower IGD are better, and bold marks the strictly better value. **Table 2: Controlled-ascent ablation on UltraFeedback/Llama-3.2-3B-Instruct** | Cluster | APO Score | APO HV | APO IGD | APO w/o control ascent Score | APO w/o control ascent HV | APO w/o control ascent IGD | | --- | --- | --- | --- | --- | --- | --- | | $C_0$ | **0.78** (+5.4%) | **0.48** (+17.1%) | **0.10** (−23.1%) | 0.74 | 0.41 | 0.13 | | $C_1$ | **0.79** (+6.8%) | **0.48** (+14.3%) | **0.04** (−55.6%) | 0.74 | 0.42 | 0.09 | | $C_2$ | **0.79** (+9.7%) | **0.52** (+33.3%) | **0.04** (−71.4%) | 0.72 | 0.39 | 0.14 | | $C_3$ | **0.79** (+6.8%) | **0.51** (+21.4%) | **0.06** (−53.8%) | 0.74 | 0.42 | 0.13 | | $C_4$ | **0.74** (+4.2%) | **0.40** (+5.3%) | **0.04** (−60.0%) | 0.71 | 0.38 | 0.10 | | $C_5$ | **0.76** (+2.7%) | **0.43** (+16.2%) | **0.10** (−28.6%) | 0.74 | 0.37 | 0.14 | | $C_6$ | **0.84** (+5.0%) | **0.58** (+13.7%) | **0.04** (−60.0%) | 0.80 | 0.51 | 0.10 | | $C_7$ | 0.82 (+0.0%) | **0.47** (+6.8%) | **0.06** (−45.5%) | 0.82 | 0.44 | 0.11 | | Avg. | **0.79** (+5.3%) | **0.48** (+14.3%) | **0.06** (−50.0%) | 0.75 | 0.42 | 0.12 |

Compared with APO w/o control ascent, APO improves the average Score by 5.3% (0.75 to 0.79) and HV by 14.3% (0.42 to 0.48), while reducing IGD by 50.0% (0.12 to 0.06). APO achieves a higher Score in seven of the eight clusters and higher HV and lower IGD in all eight. The largest improvement occurs in C2C_2, where APO increases Score by 9.7% (0.72 to 0.79) and HV by 33.3% (0.39 to 0.52), while reducing IGD by 71.4% (0.14 to 0.04). C7C_7 is an informative boundary case: the rounded Score remains unchanged at 0.82 (0.815 for APO versus 0.816 without controlled ascent), whereas APO still improves HV by 6.8% (0.44 to 0.47) and reduces IGD by 45.5% (0.11 to 0.06). Thus, even when the scalarized Score is nearly unchanged, controlled ascent yields a solution with substantially better three-objective quality and closer coverage of the empirical Pareto front. This result supports the role of the controlled trade-off permitted by constraint (6b), which the common-descent constraint (6b′) removes.

For the FSPO baseline, we preserve the central two-stage and preference-conditioned design of the original method while adapting it to our per-client personalization protocol. Specifically, the original FSPO first performs Fewshot Pref-FT and then initializes Fewshot IPO from the resulting shared checkpoint; both stages train a single model across users, with each prediction conditioned on a user’s few-shot preferences. Personalization to a new user is then performed primarily through this few-shot context rather than through user-specific parameter updates.

Our implementation follows the same two-stage structure. In the first stage, we train one shared SFT initialization by pooling Roleplay revision examples from multiple personas. Each revision prompt is explicitly conditioned on a (Helpful, Honest, Truthful) preference vector, and the SFT objective maximizes the token likelihood of the preferred revision. In the second stage, we initialize from this shared checkpoint and apply IPO using each client’s preference-conditioned support pairs. Preference information remains explicitly present in the prompt throughout both stages.

To make FSPO comparable with the other personalized baselines in our evaluation, which produce an adapted model for each client, we perform the IPO stage separately for each client and evaluate the resulting client-specific checkpoint. Thus, our implementation retains FSPO’s shared SFT initialization, two-stage SFT-to-IPO training, and preference-conditioned optimization, but replaces the original inference-time, context-only personalization with client-specific IPO adaptation. We will clarify this adaptation and its motivation in Appendix E.

Reviewer_XDaVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Approximate Pareto Optimality (APO), a method that collaboratively learns aligner initialisation to support few-shot adaptation to diverse user preferences. To this end, the authors formalise collaborative personalised preference alignment of LLM under data scarcity; introduce bottleneck-adjusted clustering; demonstrate some asymptotic suboptimality bounds; and finally test their approach empirically FedChatbot-PA and UltraFeedback.

Critical strengths

S1) Large suite of baselines: The authors provide, to the best of my knowledge, a large suite of baselines against which they test their method. I appreciate that.

Critical weaknesses

W1) Presentation: I found this paper very hard to follow because it uses mathematical notation that isn't clearly introduced. While I appreciate the authors' attempt to present theoretical work, as a reader, I find it difficult to follow and even harder to verify whether the proposed maths works. To this end, I am also setting my confidence in this assessment to low. While the content might be correct, I strongly encourage the authors to improve the presentation and motivation of the paper.

W2) No confidence: The reported results do not have any kind of confidence intervals, which would be helpful to interpret if the results are statistically significant. While APO appears to achieve the best metrics in most cases, the differences from the second-best metric could be noise. I would thus encourage the authors to provide at least a 95% bootstrapped confidence interval (if compute is an issue), or even better, run it on multiple seeds (I do not expect this, however)

Questions for authors

Q1: Does it perform well also on generation? I might have misunderstood something, but the metrics seem to be on test samples, not generations of the preference-aligned models. How would an APO-trained model perform when actually generating user-aligned preferences after few-shot sampling?

Q2: What is the effect of fewer few-shot samples? What would the effect be when starting with e.g. 0,1,5,10 few-shot samples?

3 comments

Q1: Presentation: I found this paper very hard to follow because it uses mathematical notation that isn't clearly introduced. While I appreciate the authors' attempt to present theoretical work, as a reader, I find it difficult to follow and even harder to verify whether the proposed maths works. To this end, I am also setting my confidence in this assessment to low. While the content might be correct, I strongly encourage the authors to improve the presentation and motivation of the paper.

A1: We thank the reviewer for highlighting the presentation issue. We agree that the current version introduces too much notation before providing sufficient intuition, which makes both the method and its theoretical guarantees difficult to verify. In the revision, we will move secondary derivations to the appendix, break long proofs into clearly stated intermediate lemmas, and explicitly summarize the assumptions, and conclusions of each theorem. These revisions are intended to make the mathematical claims easier to follow and independently verify without changing the proposed method or results.

Q2: No confidence: The reported results do not have any kind of confidence intervals, which would be helpful to interpret if the results are statistically significant. While APO appears to achieve the best metrics in most cases, the differences from the second-best metric could be noise. I would thus encourage the authors to provide at least a 95% bootstrapped confidence interval (if compute is an issue), or even better, run it on multiple seeds (I do not expect this, however)

A2:

We thank the reviewer for this suggestion. We now report percentile 95% bootstrap confidence intervals for the cross-cluster means. For each cluster we resample its preference vectors with replacement. We then average the three metrics (score, HV, and IGD) of each cluster. Each interval uses 10,000 replicates and is the 2.5th to 97.5th percentile of that bootstrap distribution. The column “Remains ahead” is the fraction of replicates in which the leading method is still better than the second-best method. For IGD, lower is better, so a negative difference means that the leading method has the smaller distance.

APO is ahead of the second-best method on the cross-cluster mean for every metric in Llama UltraFeedback, Qwen UltraFeedback, and Qwen-HH, and the corresponding 95% intervals exclude zero. On Llama-HH, the same statement holds for the weighted score. Its HV interval includes zero, and FSPO has the lower IGD point estimate, with an interval that also includes zero.

Llama UltraFeedback

MetricComparison95% CIRemains ahead
ScoreAPO − FedAvg[0.026, 0.057]100%
HVAPO − FedAvg[0.043, 0.101]100%
IGDAPO − FedAvg[−0.082, −0.032]100%

Qwen UltraFeedback

MetricComparison95% CIRemains ahead
ScoreAPO − FedAvg[0.053, 0.082]100%
HVAPO − FedAvg[0.065, 0.119]100%
IGDAPO − FedAvg[−0.136, −0.087]100%

Llama-HH

MetricComparison95% CIRemains ahead
ScoreAPO − FSPO[0.027, 0.061]100%
HVAPO − FSPO[−0.009, 0.041]89.62%
IGDFSPO − APO[−0.024, 0.014]68.71%

Qwen-HH

MetricComparison95% CIRemains ahead
ScoreAPO − FedAvg[0.018, 0.033]100%
HVAPO − FedAvg[0.018, 0.042]100%
IGDAPO − FedAvg[−0.026, −0.006]99.97%

These results support the main comparison. Where the interval excludes zero, the reported advantage of the leading method over the second-best method is not explained by resampling noise across the evaluation pairs. We agree that this does not apply to every cell: on Llama-HH, the HV gap favoring APO and the IGD gap favoring FSPO both have intervals that include zero. We will add the tables above to the revision in the appendix.

Q3: Does it perform well also on generation? I might have misunderstood something, but the metrics seem to be on test samples, not generations of the preference-aligned models. How would an APO-trained model perform when actually generating user-aligned preferences after few-shot sampling?

A3: Thank you for this question. We agree that the current description does not make the evaluation pipeline sufficiently clear.

APO is designed for a plug-and-play response aligner, rather than a standalone language model that generates an answer directly from the original user question. As illustrated in Appendix A (“Input Dataset Structure for Aligner”), the upstream LLM first provides an initial response. The adapted aligner then receives the original prompt, the initial response, and the target user-preference vectors, and autoregressively generates a revised response that preserves the original meaning while better matching the user’s preferred trade-off among objectives.

Therefore, generation in our setting consists of two stages:

  1. An upstream LLM produces an initial response to the user’s question.
  2. The APO-trained aligner generates the final, preference-aligned revision.

Importantly, our reported metrics are computed on these model-generated revised responses, rather than directly on the chosen or rejected responses stored in the test set. For each held-out client, 20 local examples are used only for few-shot adaptation. The adapted aligner is then applied to unseen test prompts and generates a revised answer for each prompt. These generated revisions are scored by ArmoRM-Llama3-8B-v0.1 along the relevant dimensions—helpfulness and safety for Fed-ChatbotPA, and helpfulness, honesty, and truthfulness for UltraFeedback. We then compute the preference-weighted score, hypervolume, and IGD from these generation-level reward scores.

The chosen and rejected responses shown in the appendix are supervision used to train or adapt the aligner; they are not substituted for the model output during evaluation. Thus, the results in the main table measure how well the few-shot-adapted aligner actually generates preference-aligned revisions for unseen test inputs.

We will revise the experimental section to make this generation pipeline explicit. In particular, we will add the following clarification:

APO trains a plug-and-play response aligner rather than replacing the upstream response generator. At inference time, the upstream LLM first produces an initial answer. Given this answer and the client’s preference vectors, the few-shot-adapted aligner autoregressively generates a revised answer. All reported evaluation metrics are computed by applying ArmoRM to these generated revisions on held-out prompts, rather than to the ground-truth chosen or rejected responses in the dataset.

Q4: What is the effect of fewer few-shot samples? What would the effect be when starting with e.g. 0,1,5,10 few-shot samples?

A4: Thank you for this important question. We agree that evaluating different numbers of few-shot samples is necessary to characterize APO's adaptation efficiency. We have completed an additional 10-shot experiment on UltraFeedback with Llama-3.2-3B-Instruct using the same score metric and evaluation protocol as in the main experiments. The results are:

ClusterAPOFedAvgRelative change
00.7798760.762787+2.24%
10.7852180.784181+0.13%
20.7925290.759820+4.30%
30.7951950.724856+9.70%
40.6949820.711368−2.30%
50.7283900.739869−1.55%
60.8611900.827638+4.05%
Average0.7767690.758646+2.39%

With only half of the original adaptation budget, APO outperforms FedAvg in five of the seven clusters. Its preference weighted average score is 0.7768, compared with 0.7586 for FedAvg, corresponding to an absolute improvement of 0.0181 (approximately 2.39% relative). This provides preliminary evidence that APO retains its advantage when the adaptation data are reduced from 20 to 10 samples. We nevertheless acknowledge the regressions in clusters 4 and 5.

Due to the limited time and computational resources available during the rebuttal period, we have not yet completed the 0-, 1-, and 5-shot evaluations. The 5-shot experiment is currently in progress. We will include the completed 10-shot results and report the remaining results in the revised manuscript.

Thank you to the authors for taking the time to answer.

Unfortunately, I received the responses a bit too close to the deadline, which makes it difficult for me to assess all the additional results properly.

I appreciate that the reviewers introduced the confidence intervals, and their methods seem to work stably. Moreover, I appreciate them taking the time to explain the aligner setup.

I really apologise, though, to be this kind of reviewer: my main concern is that the manuscript is still too overengineered and hard to follow. The authors promised to improve it, yet I cannot see any updated version of the manuscript. To this extent, I will have to keep my score (yet with low confidence), as I am admittedly not at all certain what is happening in the method to justify the results. Sorry

Reviewer_8pJKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at personalizing LLMs for users who only have little data. The idea is to use collaborative learning but considering that user's have diverse and often conflicting preferences. The paper proposes an algorithm with 3 main components, first the users are grouped based on similarity, then a model is trained for each group, then the group models are trained for quick personalization.

Critical strengths
  1. The problem is well motivated. Personalizing for users with little data has strong practical motivation.
  2. The proposed method shows good empirical performance beating other baselines across different datasets.
Critical weaknesses
  1. First of all significant effort needs to be put in re-writing the paper, it is significantly hard to follow for a new reader and is sub-par by ICLR standards. The notation is excessive and dense and the writing lacks clarity. Several explanations can be simplified, paragraphs broken up etc.

  2. Clustered federated learning, pareto optimization with controlled ascent, Reptile meta-learning etc. are known. The authors need to highlight what is novel about the paper other than their combination. Otherwise, this largely reads as an engineering effort rather than novel research.

Questions for authors

See weakness.

1 comment

We are truly grateful for the reviewers' dedicated efforts, which greatly improve the quality of APO. Their well-written and clear comments were invaluable, and we especially appreciate their strong affirmation of our experimental results. Additionally, we provide the following answers to the reviewers' concerns:

Q1: First of all significant effort needs to be put in re-writing the paper, it is significantly hard to follow for a new reader and is sub-par by ICLR standards. The notation is excessive and dense and the writing lacks clarity. Several explanations can be simplified, paragraphs broken up etc.

A1: We sincerely thank the reviewer for this constructive feedback. We agree that parts of Sections 3.1–3.4 introduce notation too quickly and would benefit from clearer explanations. We will simplify the notation, break up dense paragraphs, and explain the intuition behind the clustering criterion and optimization procedure before presenting their formal definitions. We will also move intermediate derivations to the appendix and add concise interpretations of the main theoretical results. These targeted revisions will make the presentation easier to follow and bring the central contributions into clearer focus.

Q2: Clustered federated learning, pareto optimization with controlled ascent, Reptile meta-learning etc. are known. The authors need to highlight what is novel about the paper other than their combination. Otherwise, this largely reads as an engineering effort rather than novel research.

A2: We thank the reviewer for raising this important concern. We agree that clustered FL, controlled ascent, and Reptile-style meta-learning are established techniques, and we do not claim these components individually as novel. Our central contribution is to characterize when collaborative updates interfere with individual preference correction and use this characterization to determine which clients should collaborate. We clarify the methodological and theoretical contributions below.

1: We identify an aggregation-level obstacle that local Pareto optimization alone does not resolve. An update that improves a client’s own preference balance can become harmful after aggregation with other clients’ updates. Section 3.2 explicitly decomposes the shared update’s effect into the client’s local correction gain and the interference introduced by aggregation. Theorem 3.2 then establishes a sufficient condition under which the shared update preserves preference-balance improvement for every client in a cluster. This analysis explains why individually valid preference updates need not remain valid under collaboration and identifies update disagreement as a quantity that the grouping procedure should control.

2: This analysis motivates Bottleneck-Adjustment Clustering (BAC), which groups clients by the preference corrections they need. BAC first partitions clients by the ordering of their preference-weighted losses and then refines the groups using their preference-adjustment vectors. These quantities jointly capture which objectives currently require correction and how compatible the required corrections are, incorporating both user preferences and the model’s current losses. Under the stated shared-gradient and common LP-basis assumptions, Proposition H.7 links adjustment-vector dispersion to update-direction dispersion. This gives the clustering criterion an analytical connection to the interference characterized above.

The controlled ablation in Table 2 supports this design: replacing hierarchical clustering on raw preference vectors with BAC, while keeping subsequent training and adaptation fixed, improves the overall average score from 0.70 to 0.72 and the reported best score from 0.77 to 0.79 on UltraFeedback with Llama-3.2-3B-Instruct. Although the improvement is not uniform across all clusters, the comparison isolates the benefit of the proposed clustering criterion.

3: We connect collaboration quality to initialization error and subsequent personalization. Theorem 3.3 establishes a conditional bound on each client’s asymptotic Chebyshev suboptimality in terms of update disagreement, step size, and descent tolerance. Theorem 3.4 then shows how initialization error contributes to the expected adaptation gap. Together, these results trace how interference during collaboration affects the starting point available for personalization and how that error enters subsequent adaptation. We emphasize that these guarantees hold under explicit assumptions, with one-local-step collaborative updates and an idealized fresh-sample adaptation process.

We appreciate the reviewer’s suggestion and will sharpen the contribution statement around the aggregation-interference analysis, the resulting clustering criterion, and the connection to personalization error, while clearly identifying controlled ascent and adaptation-aware meta-training as established tools used in the framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.