- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: This paper trains a LoRA for each objective, using a preference vector λ to simultaneously control the adapter's mixing ratio and the objective-standardized reward weights. Joint training is performed within the GRPO framework, resulting in a policy family with adjustable trade-offs during inference time. Comparisons with conditionalization, merging, routing, and expert baselines are conducted on three bi-objective tasks and up to four models.
Critical strengths
Strengths
The problem is practical, and the method is simple: one-time training, inference-time adjustable, merging possible with a fixed λ, and no additional overhead.
Large experimental scale: 4 models, 3 tasks, and over ten baselines (conditionalization, merging, routing, expert). The three ablation methods (mixture-only, weight-only, and both fixed) clearly demonstrate the indispensability of both components. Training and storage budgets are reported.
The discussion of the task-specific variation in the effectiveness of GDPO standardization is relatively honest.
Critical weaknesses
Weaknesses
Limited novelty.
The method essentially adapts PaLoRA (a preference mixture of objective-wise LoRA + annealing schedule) to GRPO, adding GDPO's sub-objective standardization. All three are existing techniques, as described by the authors themselves.
CLP itself has a conditional variant with "linear combination of parameters," very similar to LambdaMix, but this paper only compares the prompt variant of CLP.
The same scorer is used for both training and evaluation.
PKU-SafeRLHF uses the same set of Beaver reward/cost models for training and evaluation; SynthPAI uses the same Qwen3-8B referee for training and evaluation.
Large span and high HV may partly stem from overfitting to the scorer (reward hacking), requiring independent evaluation (using RM, GPT-type referees, or manual sampling).
In the examples in Table 2, the harmlessness score does not change monotonically with w (10.6 → 5.8 → 7.8), and the response is directly rejected when w=0, indicating that the behavior under extreme λ is not necessarily ideal.
The variant selection and description are selective.
The default Full version (with GDPO normalization) is inferior to the Norm. off version on both GSM8K Qwen3-4B/27B and the two SynthPAI models.
On SynthPAI Qwen3.5-9B, Full only has a score of 0.3858, lower than HoE (0.4293), linear interpolation (0.4193), and GRPO expert (0.4257), but the main text reports the Norm. off version.
The abstract states "higher than all conditionalized and merged baselines," intentionally excluding the routing method HoE; while HoE has higher HV on both tasks of Olmo 3 7B.
Fairness and baseline strength are questionable.
LambdaMix has two rank-8 LoRAs (13.2M parameters, equivalent to rank 16 at λ=(0.5,0.5)), while each expert and prompt conditional baseline only has one rank-8 LoRA (6.6M).
The statement that "both are better than all 10 experts at λ=(0.5,0.5)" may partly stem from capacity differences and requires a comparison with a rank-16 single adapter.
The conditional baselines have a span of only 0.1–0.3, hardly changing with preference, suggesting they may not be well-trained under a 500-step, 32k completions budget, or that transfer implementation is flawed.
Questions for authors
See Weaknesses
3 comments
Response to Reviewer CoBR
The default Full version (with GDPO normalization) is inferior to the Norm. off version on both GSM8K Qwen3-4B/27B and the two SynthPAI models.
On SynthPAI Qwen3.5-9B, Full only has a score of 0.3858, lower than HoE (0.4293), linear interpolation (0.4193), and GRPO expert (0.4257), but the main text reports the Norm. off version.
The paper does not designate either variant as the default. We report both side by side in every results figure (Figures 2–4), every appendix table, and in the Alignment and judge evaluations (Tables 3 and 24). Each results subsection states where each variant wins: normalization improves HV on all four PKU-SafeRLHF models (§5.1: L341–345), two of four GSM8K-IF models (§5.2: L369–375), and neither SynthPAI model (§5.3: L385–391). The revised §5.3 now states explicitly that normalized Qwen3.5-9B falls below several baselines (L389–391). As the evidence is task-dependent, which you note as an honest discussion, we keep both variants. We treat normalization as a task-dependent choice.
The abstract states "higher than all conditionalized and merged baselines," [...] while HoE has higher HV on both tasks of Olmo 3 7B.
Thank you for catching this. The abstract now covers all evaluated methods: "On the first two tasks, LambdaMix achieves the highest hypervolume among all evaluated methods for three out of four base models", consistent with §5.1 and §5.2.
LambdaMix has two rank-8 LoRAs (13.2M parameters, equivalent to rank 16 at λ=(0.5,0.5)), while each expert and prompt conditional baseline only has one rank-8 LoRA (6.6M).
The statement that "both are better than all 10 experts at λ=(0.5,0.5)" may partly stem from capacity differences and requires a comparison with a rank-16 single adapter.
We retrained prompt conditioning with a single rank-16 adapter, and parameter-conditioned CLP uses two rank-8 adapters, so both match LambdaMix's 13.2M parameters (Table 5). LambdaMix has higher HV than both in all eight PKU-SafeRLHF and GSM8K-IF settings, e.g. 0.8735 against 0.8013 and 0.7052 on Qwen3.5-9B PKU-SafeRLHF.
The conditional baselines have a span of only 0.1–0.3, hardly changing with preference, suggesting they may not be well-trained under a 500-step, 32k completions budget, or that transfer implementation is flawed.
All online methods are trained under the same protocol and budget (500 updates, 32,000 completions), with method-specific settings following the original papers that introduced them (Appendix A.2). Implementations are in our anonymous code repository. This budget allows wide steering: with a rank-16 adapter, prompt conditioning spans 0.910 in helpfulness and 1.389 in harmlessness on Qwen3.6-27B PKU-SafeRLHF, close to LambdaMix (1.108 and 1.684; Table 16).
We hope these additions and clarifications address your concerns about novelty, independent evaluation, the qualitative example, variant reporting, capacity, and baseline strength, and we are happy to clarify any remaining points.
As the discussion period is coming to an end, we would be grateful if you could take another look at our responses and the revised manuscript and share any further thoughts. The revised manuscript includes additional baseline comparisons, equal-parameter-budget experiments, independent evaluation, uncertainty estimates, and clarifications addressing the concerns raised in your review.
We would be happy to address any remaining questions. Thank you again for your time and consideration.
Response to Reviewer CoBR
Thank you for the thorough review and for recognizing the practicality and simplicity of the method, the scale of the evaluation, and the ablations showing that both components are needed. We address each point below and mark the corresponding changes in blue in the updated paper.
The main contribution of our paper is a new online RL method in which a single preference vector both selects the policy by mixing per-objective adapters and weights the standardized rewards used to update it, yielding one model whose trade-off can be changed at inference. Unlike PaLoRA, which combines per-objective adapters under supervised losses on fixed data, LambdaMix trains each adapter mixture on its own responses, so each mixture is optimized for the trade-off it is later used for. As you note, our ablations show that both parts are needed (§6.1, Table 1).
The two methods differ in the training formulation. LambdaMix's main differences from CLP are using one low-rank adapter per objective, group-relative online RL with per-objective reward standardization, and preference annealing. Our contribution is showing that this combination yields strong, steerable trade-offs, and identifying which components are responsible: varying both mixtures and reward weights is necessary, the identity mapping is a strong default, and per-objective normalization is task-dependent (§5).
We added parameter-conditioned CLP for all models and tasks (§4.2, Tables 13–22). We implement its parameter conditioning with the same two r=8 adapters as LambdaMix, and train them with REINFORCE and a learned critic, as in the original method. LambdaMix has higher HV and wider spans in all eight PKU-SafeRLHF and GSM8K-IF settings (e.g. 0.8735 against 0.7052 on Qwen3.5-9B PKU-SafeRLHF).
We added an independent evaluation with GPT-6.1 Terra as an LLM judge on PKU-SafeRLHF and SynthPAI (Appendix F, Table 24), rescoring the same responses. The judge rates helpfulness and harmlessness on PKU-SafeRLHF and replaces the Qwen3-8B evaluator for attribute support on SynthPAI.
The ranking is preserved under the independent judge. LambdaMix has the highest HV on all four PKU-SafeRLHF models (e.g. 0.5483 against 0.4956 for GDPO specialists on Qwen3.6-27B), and its normalization-off variant achieves the highest HV on both SynthPAI models. Table 24 reports bootstrap intervals with each estimate. GSM8K-IF uses verifiable rewards, so its scores cannot be inflated by exploiting a learned scorer.
This example (now Table 4) illustrates how changing the preference changes the content of a single response, and individual response scores need not be monotonic. The updated §6.3 states this explicitly (L482–503). Our claims on preference control rest on mean scores over the full evaluation sets, not a single example. Mean scores move consistently with the preference. On PKU-SafeRLHF, LambdaMix has no reversals in mean harmlessness on Qwen3.5-9B and Qwen3.6-27B, against three and one for prompt conditioning, and leaves 10 of 11 settings non-dominated on each model, against 7 and 4 (Tables 15 and 16).
At λ=(0,1), the complete response (now Appendix H, Table 27) declines manipulative tactics and then offers six ethical alternatives, such as transparent pricing and authentic testimonials, consistent with prioritizing harmlessness while remaining helpful. Appendix H also shows the same prompt for Olmo 3 7B and Qwen3.6-27B (Tables 28 and 29). For example, for Olmo 3 7B, helpfulness rises 5.9 → 6.9 → 7.6 while harmlessness falls 9.2 → 4.6 → 4.3 across w = 0, 0.5, 1.