acceptodds
Under review as a conference paper at ICLR 2027

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters

11 views

Abstract

Language model deployment often requires different trade-offs between competing objectives as user preferences change, yet standard post-training methods typically produce a single policy for a fixed objective weighting. We introduce LambdaMix, a new parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning. LambdaMix jointly trains one low-rank adapter per objective over a frozen base model. One preference vector selects the adapter mixture that generates responses and weights the separately normalized rewards used to update the adapters. A single training run supports changing preferences at inference without further optimization. We experiment with four models on helpfulness and harmlessness and on solving math problems while following instructions. We also evaluate the two largest models on rewriting comments to preserve wording while reducing clues about personal attributes. On the first two tasks, LambdaMix achieves higher hypervolume than all evaluated conditioned and policy merging baselines across all four models. LambdaMix uses fewer training completions and stores fewer adapter parameters than a set of GRPO specialist policies. With architecture and initialization fixed, varying adapter mixtures and objective weights together gives wider score ranges and higher coverage than varying either alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-objective reinforcement learning, large language models, preference optimization, inference-time steering, group relative policy optimization, low-rank adaptation, safety, privacy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CoBROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper trains a LoRA for each objective, using a preference vector λ to simultaneously control the adapter's mixing ratio and the objective-standardized reward weights. Joint training is performed within the GRPO framework, resulting in a policy family with adjustable trade-offs during inference time. Comparisons with conditionalization, merging, routing, and expert baselines are conducted on three bi-objective tasks and up to four models.

Critical strengths

Strengths

The problem is practical, and the method is simple: one-time training, inference-time adjustable, merging possible with a fixed λ, and no additional overhead.

Large experimental scale: 4 models, 3 tasks, and over ten baselines (conditionalization, merging, routing, expert). The three ablation methods (mixture-only, weight-only, and both fixed) clearly demonstrate the indispensability of both components. Training and storage budgets are reported.

The discussion of the task-specific variation in the effectiveness of GDPO standardization is relatively honest.

Critical weaknesses

Weaknesses

Limited novelty.

The method essentially adapts PaLoRA (a preference mixture of objective-wise LoRA + annealing schedule) to GRPO, adding GDPO's sub-objective standardization. All three are existing techniques, as described by the authors themselves.

CLP itself has a conditional variant with "linear combination of parameters," very similar to LambdaMix, but this paper only compares the prompt variant of CLP.

The same scorer is used for both training and evaluation.

PKU-SafeRLHF uses the same set of Beaver reward/cost models for training and evaluation; SynthPAI uses the same Qwen3-8B referee for training and evaluation.

Large span and high HV may partly stem from overfitting to the scorer (reward hacking), requiring independent evaluation (using RM, GPT-type referees, or manual sampling).

In the examples in Table 2, the harmlessness score does not change monotonically with w (10.6 → 5.8 → 7.8), and the response is directly rejected when w=0, indicating that the behavior under extreme λ is not necessarily ideal.

The variant selection and description are selective.

The default Full version (with GDPO normalization) is inferior to the Norm. off version on both GSM8K Qwen3-4B/27B and the two SynthPAI models.

On SynthPAI Qwen3.5-9B, Full only has a score of 0.3858, lower than HoE (0.4293), linear interpolation (0.4193), and GRPO expert (0.4257), but the main text reports the Norm. off version.

The abstract states "higher than all conditionalized and merged baselines," intentionally excluding the routing method HoE; while HoE has higher HV on both tasks of Olmo 3 7B.

Fairness and baseline strength are questionable.

LambdaMix has two rank-8 LoRAs (13.2M parameters, equivalent to rank 16 at λ=(0.5,0.5)), while each expert and prompt conditional baseline only has one rank-8 LoRA (6.6M).

The statement that "both are better than all 10 experts at λ=(0.5,0.5)" may partly stem from capacity differences and requires a comparison with a rank-16 single adapter.

The conditional baselines have a span of only 0.1–0.3, hardly changing with preference, suggesting they may not be well-trained under a 500-step, 32k completions budget, or that transfer implementation is flawed.

Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer CoBR

Thank you for the thorough review and for recognizing the practicality and simplicity of the method, the scale of the evaluation, and the ablations showing that both components are needed. We address each point below and mark the corresponding changes in blue in the updated paper.

The method essentially adapts PaLoRA to GRPO, adding GDPO's sub-objective standardization.

The main contribution of our paper is a new online RL method in which a single preference vector both selects the policy by mixing per-objective adapters and weights the standardized rewards used to update it, yielding one model whose trade-off can be changed at inference. Unlike PaLoRA, which combines per-objective adapters under supervised losses on fixed data, LambdaMix trains each adapter mixture on its own responses, so each mixture is optimized for the trade-off it is later used for. As you note, our ablations show that both parts are needed (§6.1, Table 1).

CLP itself has a conditional variant with "linear combination of parameters," very similar to LambdaMix, [...] this paper only compares the prompt variant of CLP.

The two methods differ in the training formulation. LambdaMix's main differences from CLP are using one low-rank adapter per objective, group-relative online RL with per-objective reward standardization, and preference annealing. Our contribution is showing that this combination yields strong, steerable trade-offs, and identifying which components are responsible: varying both mixtures and reward weights is necessary, the identity mapping is a strong default, and per-objective normalization is task-dependent (§5).

We added parameter-conditioned CLP for all models and tasks (§4.2, Tables 13–22). We implement its parameter conditioning with the same two r=8r=8 adapters as LambdaMix, and train them with REINFORCE and a learned critic, as in the original method. LambdaMix has higher HV and wider spans in all eight PKU-SafeRLHF and GSM8K-IF settings (e.g. 0.8735 against 0.7052 on Qwen3.5-9B PKU-SafeRLHF).

PKU-SafeRLHF uses the same set of Beaver reward/cost models for training and evaluation; SynthPAI uses the same Qwen3-8B referee for training and evaluation. [...] requiring independent evaluation (using RM, GPT-type referees, or manual sampling).

We added an independent evaluation with GPT-6.1 Terra as an LLM judge on PKU-SafeRLHF and SynthPAI (Appendix F, Table 24), rescoring the same responses. The judge rates helpfulness and harmlessness on PKU-SafeRLHF and replaces the Qwen3-8B evaluator for attribute support on SynthPAI.

The ranking is preserved under the independent judge. LambdaMix has the highest HV on all four PKU-SafeRLHF models (e.g. 0.5483 against 0.4956 for GDPO specialists on Qwen3.6-27B), and its normalization-off variant achieves the highest HV on both SynthPAI models. Table 24 reports bootstrap intervals with each estimate. GSM8K-IF uses verifiable rewards, so its scores cannot be inflated by exploiting a learned scorer.

In the examples in Table 2, the harmlessness score does not change monotonically with w [...]

This example (now Table 4) illustrates how changing the preference changes the content of a single response, and individual response scores need not be monotonic. The updated §6.3 states this explicitly (L482–503). Our claims on preference control rest on mean scores over the full evaluation sets, not a single example. Mean scores move consistently with the preference. On PKU-SafeRLHF, LambdaMix has no reversals in mean harmlessness on Qwen3.5-9B and Qwen3.6-27B, against three and one for prompt conditioning, and leaves 10 of 11 settings non-dominated on each model, against 7 and 4 (Tables 15 and 16).

At λ=(0,1)\lambda = (0, 1), the complete response (now Appendix H, Table 27) declines manipulative tactics and then offers six ethical alternatives, such as transparent pricing and authentic testimonials, consistent with prioritizing harmlessness while remaining helpful. Appendix H also shows the same prompt for Olmo 3 7B and Qwen3.6-27B (Tables 28 and 29). For example, for Olmo 3 7B, helpfulness rises 5.9 → 6.9 → 7.6 while harmlessness falls 9.2 → 4.6 → 4.3 across w = 0, 0.5, 1.

AuthorsOpenReview ↗

Response to Reviewer CoBR

The default Full version (with GDPO normalization) is inferior to the Norm. off version on both GSM8K Qwen3-4B/27B and the two SynthPAI models.

On SynthPAI Qwen3.5-9B, Full only has a score of 0.3858, lower than HoE (0.4293), linear interpolation (0.4193), and GRPO expert (0.4257), but the main text reports the Norm. off version.

The paper does not designate either variant as the default. We report both side by side in every results figure (Figures 2–4), every appendix table, and in the Alignment and judge evaluations (Tables 3 and 24). Each results subsection states where each variant wins: normalization improves HV on all four PKU-SafeRLHF models (§5.1: L341–345), two of four GSM8K-IF models (§5.2: L369–375), and neither SynthPAI model (§5.3: L385–391). The revised §5.3 now states explicitly that normalized Qwen3.5-9B falls below several baselines (L389–391). As the evidence is task-dependent, which you note as an honest discussion, we keep both variants. We treat normalization as a task-dependent choice.

The abstract states "higher than all conditionalized and merged baselines," [...] while HoE has higher HV on both tasks of Olmo 3 7B.

Thank you for catching this. The abstract now covers all evaluated methods: "On the first two tasks, LambdaMix achieves the highest hypervolume among all evaluated methods for three out of four base models", consistent with §5.1 and §5.2.

LambdaMix has two rank-8 LoRAs (13.2M parameters, equivalent to rank 16 at λ=(0.5,0.5)), while each expert and prompt conditional baseline only has one rank-8 LoRA (6.6M).

The statement that "both are better than all 10 experts at λ=(0.5,0.5)" may partly stem from capacity differences and requires a comparison with a rank-16 single adapter.

We retrained prompt conditioning with a single rank-16 adapter, and parameter-conditioned CLP uses two rank-8 adapters, so both match LambdaMix's 13.2M parameters (Table 5). LambdaMix has higher HV than both in all eight PKU-SafeRLHF and GSM8K-IF settings, e.g. 0.8735 against 0.8013 and 0.7052 on Qwen3.5-9B PKU-SafeRLHF.

The conditional baselines have a span of only 0.1–0.3, hardly changing with preference, suggesting they may not be well-trained under a 500-step, 32k completions budget, or that transfer implementation is flawed.

All online methods are trained under the same protocol and budget (500 updates, 32,000 completions), with method-specific settings following the original papers that introduced them (Appendix A.2). Implementations are in our anonymous code repository. This budget allows wide steering: with a rank-16 adapter, prompt conditioning spans 0.910 in helpfulness and 1.389 in harmlessness on Qwen3.6-27B PKU-SafeRLHF, close to LambdaMix (1.108 and 1.684; Table 16).

We hope these additions and clarifications address your concerns about novelty, independent evaluation, the qualitative example, variant reporting, capacity, and baseline strength, and we are happy to clarify any remaining points.

As the discussion period is coming to an end, we would be grateful if you could take another look at our responses and the revised manuscript and share any further thoughts. The revised manuscript includes additional baseline comparisons, equal-parameter-budget experiments, independent evaluation, uncertainty estimates, and clarifications addressing the concerns raised in your review.

We would be happy to address any remaining questions. Thank you again for your time and consideration.

Reviewer_ZJesOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce LambdaMix as a parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning. The method jointly trains one low-rank adapter per objective over a frozen base model. Experiments span four models on helpfulness and harmlessness and solving math problems/following instructions. In summary, the main feature of this paper is the method for learning preference conditioned adapter mixtures with group-relative online RL.

Critical strengths
  • The empirical evidence and evaluation covers four base models on helpfulness–harmlessness and mathematical correctness–instruction compliance, with some additional tasks on the two largest models. LambdaMix shows superior emprical results against baselines across all four models on the first two tasks. Improvements across learned and verifiable rewards provide evidence that the approach is useful beyond one particular evaluation setting.

  • The authors provide rigorous ablation studies, and hold architecture, initialization, data, optimization, and training budget fixed while comparing the complete method with variants that vary only the adapter mixture or only the reward weights. The method consistently improves coverage and objective ranges across the three tasks.

  • One training run produces a family of policies that can be selected by changing λ\lambda, without further optimization or reward evaluation at inference. On Qwen3.5-9B, the method uses 32,000 training completions and 13.2M adapter parameters, compared with 160,000 completions and 33.1M parameters for five specialists. This is practical for applications that need multiple operating preferences.

Critical weaknesses
  • The novelty is modest, and the closest competing approach is not directly evaluated. The paper explicitly builds on PaLoRA’s preference-conditioned adapter construction and GDPO’s reward normalization. Existing parameter-conditioned CLP also couples preference-dependent policy parameters with the corresponding weighted rewards, but the experiments include only its prompt-conditioned variant.

  • LambdaMix uses two adapters with r=8r=8, but prompt-conditioned CLP uses one adapter with r=8r=8, giving LambdaMix twice the trainable parameters in that space. A prompt-conditioned baseline with r=16r=16 would provide a more meaningful control. Several other methods are adapted to the group-relative training setup, without enough evidence that these adaptations and the shared parameters preserve their competitiveness.

  • The paper does not report a repeated-training-seed protocol or uncertainty intervals, despite evaluating GSM8K on 300 examples and rewriting on only 111 comments. This is important for smaller differences. The results do not establish which smaller improvements and control properties are reproducible.

  • The accuracy metric requires the correct answer to appear inside a complete <answer></answer> block, but the compliance objective can impose JSON or other structural requirements. A response containing the correct numerical answer can therefore receive zero accuracy because it lacks the required tags.

Questions for authors
  • What difference separates LambdaMix from parameter-conditioned CLP implemented with low-rank updates and the same group-relative estimator? Can the authors provide a direct comparison that isolates the benefit of the additional mechanism?

  • Do the principal improvements persist against a prompt-conditioned baseline with the same trainable-parameter budget? What validation or tuning was performed to establish that the adapted Panacea, MOC, PRISM, and HoE baselines retain their expected performance and preference responsiveness?

  • How many independent training runs underlie the reported numbers? Can the authors report variability across training seeds and paired uncertainty estimates over evaluation prompts?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZJes

Thank you for the detailed review, and for recognizing the breadth of the evaluation across learned and verifiable rewards, the rigor of the controlled ablations, and the practicality of obtaining a family of policies from a single training run. We address each point below and mark the corresponding changes in blue in the updated paper.

[...]. The paper explicitly builds on PaLoRA's preference-conditioned adapter construction and GDPO's reward normalization.

The main contribution of our paper is a new method that trains a steerable family of LLM policies in a single online RL run, in which one preference vector both selects the adapter mixture that generates each group of responses and weights the per-objective standardized rewards used to update it. Unlike PaLoRA, which mixes per-objective adapters under supervised multi-task losses on fixed data, LambdaMix trains each mixture on-policy, on the responses it generates, toward its own trade-off.

Existing parameter-conditioned CLP also couples preference-dependent policy parameters with the corresponding weighted rewards, but the experiments include only its prompt-conditioned variant.

What difference separates LambdaMix from parameter-conditioned CLP implemented with low-rank updates and the same group-relative estimator? Can the authors provide a direct comparison [...]?

LambdaMix's main differences from CLP are using one low-rank adapter per objective, group-relative online RL with per-objective reward standardization, and preference annealing. Our contribution is showing that this combination yields strong, steerable trade-offs, and identifying which components are responsible: varying both mixtures and reward weights is necessary, the identity mapping is a strong default, and per-objective normalization is task-dependent (§5).

We added parameter-conditioned CLP for all models and tasks (§4.2, Tables 13–22, Appendix A.2). Following the original method, we mix two rank-8 policy adapters, the same architecture and parameter budget as LambdaMix, and train them with REINFORCE and a learned critic. LambdaMix has higher HV in all eight PKU-SafeRLHF and GSM8K-IF settings (e.g. 0.8735 against 0.7052 on Qwen3.5-9B PKU-SafeRLHF).

LambdaMix uses two adapters with r=8, but prompt-conditioned CLP uses one adapter with r=8 [...]. A prompt-conditioned baseline with r=16 would provide a more meaningful control.

We retrained prompt conditioning with a single rank-16 adapter, so both CLP variants now match LambdaMix's 13.2M parameters (Table 5, Appendix A.2). The improvements persist at an equal parameter budget. LambdaMix has higher HV than both CLP variants in all eight PKU-SafeRLHF and GSM8K-IF settings. On Qwen3.5-9B, it reaches 0.8735 on PKU-SafeRLHF against 0.8013 for prompt conditioning, and 0.8099 on GSM8K-IF against 0.5286 (Tables 15 and 19).

Several other methods are adapted to the group-relative training setup [...]

What validation or tuning was performed [...]?

All online methods are trained under the same protocol: the same data and training budget (500 updates, 32,000 completions). Method-specific settings follow the original papers that introduced them (Appendix A.2), and all implementations are in our anonymous code repository.

The paper does not report a repeated-training-seed protocol or uncertainty intervals [...]

How many independent training runs underlie the reported numbers? Can the authors report variability across training seeds and paired uncertainty estimates over evaluation prompts?

Thank you for the suggestion. We have added 95% bootstrap confidence intervals for every HV in the appendix tables, together with the protocol (Appendix D). We use 10,000 resamples, keeping each prompt's scores together across both objectives and all preference settings, and recompute HV for each resample. We use the same resamples for every method within a task and base model, so the intervals are paired across methods.

The main improvements are well separated. On Qwen3.5-9B, LambdaMix reaches HV 0.8735 [0.855, 0.892] on PKU-SafeRLHF against 0.8013 [0.784, 0.821] for prompt conditioning, and 0.8099 [0.771, 0.849] on GSM8K-IF against 0.6947 [0.642, 0.741] for GRPO specialists (Tables 15 and 19).

Each configuration uses a fixed training seed, so these intervals quantify evaluation sample uncertainty conditional on the trained models. We rely on consistency across models and tasks, rather than repeated seeds.

AuthorsOpenReview ↗

Response to Reviewer ZJes

A response containing the correct numerical answer can therefore receive zero accuracy because it lacks the required tags.

We agree. We added relaxed answer extraction for GSM8K (Appendix E, Table 23), which also credits answers in JSON fields, boxed or standalone values, and explicit final-answer statements, while retaining valid XML answers. Absolute scores change, but LambdaMix's HV ordering relative to the baselines is preserved. On Qwen3.5-9B, LambdaMix reaches HV of 0.8134 against 0.7389 for GRPO specialists, and on Qwen3.6-27B, 0.9078 against 0.8565 for GDPO specialists. Therefore, the GSM8K results do not depend on the answer-format requirement.

We hope these additions address your concerns about the closest baseline, parameter budget, baseline fidelity, uncertainty, and answer extraction, and we are happy to clarify any remaining points.

I appreciate the rebuttal from the authors, and apologize for my late response , which I take into consideration. On the basis of the rebuttal, since the grading system this year is 1-4, I am happy to maintain my score and positive attitude towards this work.

Reviewer_KGhHOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LambdaMix, a parameter-efficient approach to steerable multi-objective alignment. It jointly trains one LoRA adapter per objective using group-relative online reinforcement learning. The same preference vector determines both the adapter mixture used to generate responses and the weighting of separately normalized rewards. At inference time, users can change the preference vector without further training. Experiments examine trade-off quality and preference responsiveness across several models and three two-objective tasks.

As I have already informed the AC/SAC, this paper falls outside my main area of expertise; since reviewer reassignment was not possible, my confidence in this assessment is low, and some of my comments may reflect an incomplete understanding of the work.

Critical strengths
  1. Practical motivation and a simple formulation. Supporting different objective trade-offs within one trained model is useful, and the proposed adapter-based implementation is relatively straightforward.
  2. Broad empirical evaluation. The paper evaluates multiple model families and tasks, with comparisons against conditioned policies, specialist policies, routing, and policy merging. Reporting coverage and responsiveness provides a more informative picture than a single operating point.
  3. Useful component analysis. The controlled ablations help establish that allowing both adapter mixtures and objective weights to vary provides benefits over fixing either component. The discussion of task-dependent normalization effects is also helpful.
Critical weaknesses
  1. A train–test mismatch in the ablation controls makes the evidence for the central coupling somewhat difficult to interpret. I may be misunderstanding the intended comparison, but Section 6.1 trains the Weight-only control using a fixed adapter mixture of (0.5, 0.5), then evaluates it across eleven mixtures. Both fixed is evaluated similarly. These controls therefore encounter test-time adapter combinations that were not directly optimized during training, whereas LambdaMix trains with changing mixtures. Their limited steering range could partly reflect this mismatch, rather than specifically establish the benefit of using identical coefficients for adapter mixing and reward weighting. The results seem informative about the importance of training-time mixture exposure, but I am less certain that they isolate the proposed coupling itself. A comparison with matched mixture exposure, while allowing a learned mapping between objective preferences and mixture coefficients, might help distinguish these explanations. This concern is about the interpretation of the ablations, not a demonstrated failure of LambdaMix.

  2. The relationship between the supplied preference and the effective optimization target could be clearer. I am somewhat unsure how to interpret the preference vector after group-dependent reward normalization. The relative coefficients on raw reward deviations depend on reward variability under the current prompt and policy, in addition to the supplied preference. The paper explicitly acknowledges that the update need not optimize a fixed weighted sum of raw rewards, so I do not view this as an overlooked mathematical error. My broader concern is whether the vector represents a stable specification of objective priorities or primarily serves as a coordinate for steering behavior. Higher hypervolume and fewer reversals support useful coverage and directional responsiveness, but do not directly establish fidelity to a specified preference. Clarifying the intended semantics would help me better assess the central steerability claim.

Questions for authors
  1. How should the fixed-mixture controls’ test-time mixture sweeps be interpreted? Could the authors separate the benefit of training-time mixture exposure from the benefit of directly matching mixture coefficients to reward weights, using a control with comparable mixture coverage?
  2. Under group-dependent normalization, what precise notion of preference does λ represent, and how could fidelity to that preference be evaluated beyond hypervolume and directional responsiveness?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KGhH

Thank you for the careful review, and for recognizing the practical motivation, the breadth of the evaluation, and the value of the component analysis. We address each point below and mark the corresponding changes in blue in our updated paper.

A train–test mismatch in the ablation controls…

How should the fixed-mixture controls’ test-time mixture sweeps be interpreted?

The fixed-mixture controls are designed to isolate the contribution of training across mixtures. Weight-only and Both fixed are trained at a single mixture and then evaluated across all mixtures, which shows how much steering is available without training-time mixture variation. This evaluation principle also applies to policy-merging baselines, whose intermediate adapter combinations have not been jointly optimized during training. The updated §6.1 makes this more explicit (L430–431).

Varying the reward weights is necessary. Mixture-only trains across the same varying mixtures as LambdaMix, but fixes the reward weights at (0.5, 0.5). On Qwen3.5-9B, it reaches HV 0.662 against 0.810 for LambdaMix on GSM8K-IF, and 0.8144 against 0.8735 on PKU-SafeRLHF (Table 1). Training across mixtures alone therefore performs well below full LambdaMix, which requires both varying mixtures and correspondingly varying reward weights.

A comparison with matched mixture exposure, while allowing a learned mapping between objective preferences and mixture coefficients, might help distinguish these explanations.

Thank you for this suggestion. We added this comparison (§6.1: L450–463, Table 2, Appendix G). Rewards remain weighted by λ\lambda, while the adapter mixture η\eta is set by one of three mappings: the identity mapping η=λ\eta = \lambda of LambdaMix, a learned single-parameter temperature map that rescales the preference weights (Temperature), and a learned 49-parameter multilayer perceptron (MLP). The learned mappings η=fϕ(λ)\eta = f_\phi(\lambda) are optimized jointly with the adapters, so mixtures still vary during training but need not equal the reward weights.

HV, Qwen3.5-9BLambdaMixTemperatureMLP
PKU-SafeRLHF0.87350.85520.8688
GSM8K-IF0.80990.78680.7827
SynthPAI0.38580.43790.3678

LambdaMix has higher HV on PKU-SafeRLHF and GSM8K-IF, while the temperature map improves SynthPAI. On SynthPAI, LambdaMix without normalization reaches a similar HV (0.4378), so the identity mapping remains competitive there when normalization is removed. These results support identity as a simple, effective choice, while showing that learned alternatives can improve some settings.

My broader concern is whether the vector represents a stable specification of objective priorities or primarily serves as a coordinate for steering behavior.

Under group-dependent normalization, what precise notion of preference does λ represent [...]

Thank you for asking us to clarify this distinction. λ\lambda serves as a specification of objective priorities during training and as a steering coordinate at inference.

During training, λ\lambda specifies priorities in standardized units. The revised §3 (L180–184) states that λ\lambda weights within-group standardized reward deviations. Equal entries assign equal coefficients to these deviations for every prompt and policy. The coefficients on raw deviations are λt/(σt+ϵ)\lambda_t/(\sigma_t + \epsilon). The exchange rates between raw rewards depend on the sampled group. λ\lambda is therefore not a fixed exchange rate between raw rewards, and the update need not optimize a fixed weighted sum of raw rewards.

At inference, λ\lambdaselects the matching policy. The same vector sets the adapter mixture, so requesting λ\lambda returns the policy trained under those standardized priorities. We evaluate how faithfully these policies follow the requested priorities with the Alignment metric, described in our answer to the next point of our response.

AuthorsOpenReview ↗

Response to Reviewer KGhH

Higher hypervolume and fewer reversals support useful coverage and directional responsiveness, but do not directly establish fidelity to a specified preference.

[...] how could fidelity to that preference be evaluated beyond hypervolume and directional responsiveness?

Hypervolume and reversals do not measure fidelity directly. We also report Alignment, following the inner-product metric of Panacea (Zhong et al., 2024). We now define and explain the Alignment metric in the main Evaluation section (§4.4: L255–263, Eq. 4) and discuss its results in a dedicated Analysis subsection (§6.2: L465–480, Table 3). For each requested λ\lambda, Alignment weights the selected policy's objective scores by the entries of λ\lambda and averages over the preference grid, so objectives with larger requested weight count more.

LambdaMix policies perform better under the requested weights. On Qwen3.5-9B, LambdaMix reaches 0.9097 on PKU-SafeRLHF and 0.8668 on GSM8K-IF, against 0.8880 for prompt conditioning and 0.7958 for GRPO specialists. On SynthPAI, LambdaMix without normalization reaches 0.6899, above GRPO specialists at 0.6873 (Table 3). Therefore, LambdaMix's selected policies deliver higher average utility under the requested preferences, complementing coverage and responsiveness. Full results for all models are in Appendix D.

We hope these revisions resolve your concerns, and we are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the reviewers

We thank the reviewers for their constructive feedback. The reviewers find the problem practical and the method simple (KGhH, ZJes, CoBR). They value the breadth of the evaluation across models, tasks, and baselines (KGhH, ZJes, CoBR), and they consider the controlled ablations rigorous and informative (KGhH, ZJes, CoBR). CoBR also describes our discussion of task-dependent normalization as “relatively honest”. On soundness, all three reviewers raise “minor or fixable concerns”, which we address by this revision. The main concerns are (1) the closest baseline, parameter-conditioned CLP; (2) the comparison may favour LambdaMix through extra parameters; (3) the results lack uncertainty estimates; (4) the use of the training scorers for evaluation; and (5) it is unclear what the ablations isolate and what λ means. We summarize how we addressed each one in the updated manuscript (changes in blue).

  1. Comparison with parameter-conditioned CLP [ZJes Weakness 1, Question 1; CoBR Weakness 1]
  • We added parameter-conditioned CLP for all models and tasks (§4.2, Tables 13–22, Appendix A.2). It uses the same two r=8r=8 adapters as LambdaMix. LambdaMix has higher HV in all eight PKU-SafeRLHF and GSM8K-IF settings (e.g., 0.8735 vs. 0.7052 on Qwen3.5-9B PKU-SafeRLHF).
  1. Equal parameter budget and baseline strength [ZJes Weakness 2, Question 2; CoBR Weaknesses 4–5]
  • We retrained prompt conditioning with a single rank-16 adapter, so both CLP variants now match LambdaMix’s 13.2M parameters (Table 5). LambdaMix still has higher HV in all eight PKU-SafeRLHF and GSM8K-IF settings (e.g., 0.8099 vs. 0.5286 on Qwen3.5-9B GSM8K-IF).
  1. Uncertainty estimates [ZJes Weakness 3, Question 3]
  • We added paired 95% bootstrap confidence intervals for every HV, together with the protocol (Appendix D). We use 10,000 resamples and keep each prompt’s scores together across both objectives and all preference settings. We use the same resamples for every method within a task and base model. The main improvements are well separated. For example, on Qwen3.5-9B, LambdaMix reaches 0.8735 [0.855, 0.892] vs. 0.8013 [0.784, 0.821] for prompt conditioning on PKU-SafeRLHF, and 0.8099 [0.771, 0.849] vs. 0.6947 [0.642, 0.741] for GRPO specialists on GSM8K-IF (Tables 15 and 19).
  1. Independent evaluation [CoBR Weakness 2; ZJes Weakness 4]
  • Independent judge. We rescored the PKU-SafeRLHF and SynthPAI responses with GPT-6.1 Terra as an LLM judge (Appendix F, Table 24). The ranking is preserved. LambdaMix has the highest HV on all four PKU-SafeRLHF models (e.g., 0.5483 vs. 0.4956 for GDPO specialists on Qwen3.6-27B), and its normalization-off variant has the highest HV on both SynthPAI models. GSM8K-IF uses verifiable rewards, so a learned scorer cannot be exploited there.

  • Relaxed answer extraction. We added relaxed answer extraction for GSM8K, which also credits JSON fields, boxed values, and explicit final-answer statements (Appendix E, Table 23). LambdaMix’s HV ordering relative to the baselines is preserved (e.g., 0.8134 vs. 0.7389 for GRPO specialists on Qwen3.5-9B).

  1. What the ablations isolate: mixture exposure vs. coupling [KGhH Weakness 1, Question 1]
  • Mixture-only trains across the same varying mixtures as LambdaMix but fixes the reward weights. It remains well below LambdaMix (e.g., HV 0.662 vs. 0.810 on Qwen3.5-9B GSM8K-IF), so exposure to varying mixtures alone does not explain the gain (Table 1, §6.1: L430–431).

  • Following KGhH’s suggestion, we added learned mappings from preferences to mixtures (a single-parameter temperature map and a 49-parameter MLP), trained jointly with the adapters (§6.1: L450–463, Table 2, Appendix G). The identity mapping has the highest HV on PKU-SafeRLHF and GSM8K-IF. The temperature map improves SynthPAI, where LambdaMix without normalization reaches a similar HV.

  1. Meaning of λ and fidelity to the requested preference [KGhH Weakness 2, Question 2]
  • The revised §3 (L180–184) states that, during training, λ weights within-group standardized reward deviations, so it is not a fixed exchange rate between raw rewards. At inference, the same λ selects the policy trained under those priorities.

  • We now define the Alignment metric in the main text (§4.4: L255–263, Eq. 4) and add a dedicated analysis (§6.2: L465–480, Table 3, Appendix D). Policies selected by LambdaMix achieve higher utility under the requested weights (e.g., 0.9097 on Qwen3.5-9B PKU-SafeRLHF).

  1. Presentation [CoBR Weaknesses 2–3]
  • Normalization variants. We report both variants side by side throughout and treat normalization as task-dependent.

  • Qualitative example. The example (now Table 4, §6.3: L482–503) is now fully presented in Appendix H. We also add examples from two more models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.