acceptodds
Under review as a conference paper at ICLR 2027

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Abstract

Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose MEAL, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM alignment, Multi-objective alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DBHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MEAL is a framework for multi-objective LLM alignment that predicts prompt-dependent preference weights and injects them into the model's hidden states. Its main contribution is a bidirectional training scheme: the policy learns from reward-selected responses, while the preference network learns from both the policy's SFT loss and reward-derived weight supervision. Experiments on summarization and assistant generation compare reward trade-offs with existing methods.

Critical strengths

A practical preference-control mechanism. The injection module lets response loss train the preference network without differentiating through reward models. The results show gains in some trade-off regions, and Figure 4 gives some support for the complete online method.

Critical weaknesses
  1. The predictor's role is unclear. It predicts response-dependent weights from the prompt alone, without explaining how different targets for the same prompt are reconciled. It remains unclear why the predictor and “bi-level” updates help, or why learning from reward-derived weights improves performance when using them directly performs poorly—if that is what the underspecified RMs ablation tests.
  2. Important methodological details are missing. The number of responses per prompt, the scoring and selection procedure, and the definitions of offline and RMs training are unclear, making the method difficult to reproduce and the ablations hard to interpret.
Questions for authors
  1. How many responses are generated per prompt, and how are they ranked and selected? What exactly do offline and RMs training do?
  2. With matched trainable parameters and budgets, how does MEAL compare with direct reward-weight-conditioned SFT and ordinary joint training? What happens when each feedback term to the predictor is removed?
  3. What measurements support faster and more stable convergence? How do you reconcile the Pareto-superiority claim with Figure 3(b)?
Reviewer_etMLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Meta-Aligner (MEAL), a framework for multi-objective LLM alignment that jointly adapts preference weights and model responses during training. The method introduces a prompt-conditioned preference-weight network that predicts objective weights, which are projected into the LLM's hidden representation to control generation. During online training, the policy generates candidate responses, high-scoring samples are selected through rejection sampling, and reward-model scores provide reference preference weights. The policy and preference network are then updated alternately using supervised fine-tuning loss and a KL-based preference supervision loss.

The authors evaluate MEAL on Reddit Summary and Helpful Assistant using multiple reward objectives. The results are mainly presented through empirical reward trade-off curves, which suggest improvements over several existing multi-objective alignment methods. Additional analyses examine the effects of online meta-learning, preference-weight initialization, and hyperparameter sensitivity.

Critical strengths
  1. A simple and implementable framework. The combination of a prompt-conditioned preference network, lightweight latent preference injection, and online rejection-sampling fine-tuning is straightforward to implement. The approach provides a practical mechanism for jointly updating preference guidance and the policy without requiring multiple independently trained policies.

  2. Encouraging empirical trade-offs. Figures 2 and 3 suggest favorable reward trade-offs on Reddit Summary and Helpful Assistant compared with the reported baselines. The comparison of online and offline variants in Figure 4 also provides some evidence that online adaptation may be beneficial, although the strength of this evidence is limited by the evaluation protocol.

Critical weaknesses
  1. Limited novelty relative to closely related work, particularly Preference Orchestrator (major concern).

    The central components of MEAL substantially overlap with established multi-objective alignment techniques. Most notably, Preference Orchestrator (Liu et al., 2025) already introduces a prompt-conditioned preference network that predicts objective weights from input prompts and learns these weights using normalized reward-model scores. This closely resembles MEAL's preference-weight network and its reference preference construction in Eq. (5). Rewards-in-Context (Yang et al., 2024) and Directional Preference Alignment (Wang et al., 2024) also establish preference-conditioned generation and rejection-sampling-based alignment.

    The main remaining distinction appears to be the alternating optimization of the preference network and policy using downstream SFT loss. However, Appendix A explicitly acknowledges that the implementation does not differentiate through the inner optimization trajectory and instead uses a direct first-order gradient with the updated policy treated as fixed. This makes the claimed bilevel meta-learning mechanism substantially less distinctive than suggested by the paper's framing.

    Furthermore, the latent preference injection in Eqs. (2–4) effectively introduces a preference-dependent additive bias to the output logits. The paper does not sufficiently establish why this mechanism, combined with ordinary alternating optimization, represents a substantial methodological advance. A direct comparison with Preference Orchestrator and an ablation isolating the additional task-gradient feedback are particularly important.

  2. Insufficient experimental evidence to establish the claimed superiority (major concern).

    The experimental evaluation is limited primarily to reward trade-off curves on two benchmarks, without numerical hypervolume comparisons, uncertainty estimates, or repeated-seed results for the main methods. More importantly, the same reward-model objectives appear to be used both for optimizing the policy and for assessing the reported trade-offs, making it difficult to determine whether improved proxy rewards translate into better alignment with independent human preferences.

    The fairness of the baseline comparisons also requires clarification. Section 2 states that some reported baseline results are shared with closely related work, while the present study uses Qwen3-based models. The earlier Preference Orchestrator study used different base models in its corresponding experiments. It is therefore important to establish whether all baselines were rerun under comparable models, reward evaluators, and training budgets.

    Finally, the relationship between automatically predicted preference weights and the externally varied preference weights used to construct the evaluation curves is insufficiently explained. Since the proposed preference network generates weights as a function of the prompt alone, the paper should clarify how arbitrary user-specified preferences are supported and evaluated. Together, these issues substantially weaken the empirical evidence for the central claims.

[1] Liu, B., Xu, N., Yang, J., & Geng, X. (2025). Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models. arXiv preprint arXiv:2511.10656. https://arxiv.org/abs/2511.10656

[2] Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., & Chen, J. (2024). Rewards-in-Context: Multi-Objective Alignment of Foundation Models with Dynamic Preference Adjustment. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 56276–56297. https://proceedings.mlr.press/v235/yang24q.html

[3] Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., & Zhang, T. (2024). Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 8642–8655. https://aclanthology.org/2024.acl-long.468/

Questions for authors
  1. Computational fairness: Since Flux-OPD uses four teacher forward passes per step while vanilla OPD uses one, how should the improvements be interpreted under a fixed total training-compute budget? Are any compute-normalized comparisons or total GPU-hour measurements available? In particular, Appendix Table 6 suggests that reducing the number of contexts can substantially lower computation with limited performance degradation.

  2. Hyperparameter selection: How were the values of the conflict scaling factor β selected? Appendix Table 5 uses different values across downstream video generators and tasks, while Table 4 shows that performance can change noticeably with β. Were these values determined using separate validation data?

  3. Interpretation of conflict-aware weighting: The reverse-KL decomposition establishes that the conflict term measures disagreement but contributes no direct gradient under fixed histories. What evidence indicates that this disagreement reliably predicts whether contextual corrections are beneficial? Could the authors clarify why the exponential mapping is preferable to other weighting functions beyond the reported comparison with entropy and JSD?

Reviewer_AtEoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates multi-objective preference alignment for large language models and proposes the MEAL framework, which integrates a learnable preference generation network with a conditioned language model. The preference network generates objective weights based on input prompts, which are projected and injected into the hidden states right before the language model’s output head.

During training, the model generates responses online and samples are filtered via reward models. The policy model fits the retained responses using SFT loss, while the preference network receives both task loss feedback and KL divergence supervision from reference preferences derived by normalizing reward vectors, following a decoupled parameter update paradigm.

Evaluations are conducted on the Reddit summarization task with Qwen3-0.6B and the HH-RLHF dialogue task with Qwen3-4B to benchmark multi-objective trade-offs, comparing against MORLHF, Reward Soups, RiC and PARM. Experiments covering reward trade-off curves, ablation on preference sources, and hyperparameter analysis show that the proposed method delivers competitive performance across various objective combinations and preference ranges.

Critical strengths
  1. The scope of the theoretical analysis and the gradient computation are reasonably well explained.  The paper explicitly distinguishes the direct first-order update from the full gradient obtained by differentiating through the inner update. It also provides a one-step approximation error bound under stated assumptions and explains the connection to the envelope theorem, helping clarify the method’s optimization interpretation and scope of applicability.

  2. The comparative analyses provide informative empirical observations.  The comparison of preference sources shows that preference weights obtained by directly normalizing reward scores underperform random weights in some settings. Comparisons between online and offline training, together with analyses of key hyperparameters, also provide some empirical support for the training design.

Critical weaknesses
  1. The independent contribution of the core task feedback term has not been sufficiently established. Figure 4 compares preference sources and online versus offline training, but does not explicitly report a control that removes only the gradient from the task loss in Eq. (12), while retaining the same trainable preference network, online sampling procedure, and budget. The current results therefore do not sufficiently isolate the additional contribution of this feedback term beyond preference supervision and online training on high-reward samples.

  2. The semantic basis of preference supervision and its correspondence with the evaluation remain unclear. Eq. (5) converts scores from different reward models into preference supervision, but the comparability of these scores and how this mapping reflects prompt-specific objective importance require further explanation and validation. The evaluation mainly presents reward trade-offs under specified weights, without clearly distinguishing the evaluation of external preference control from that of prompt-specific preference generation.

  3. Model and task coverage is limited, and robustness across scales has not been sufficiently validated. The experiments report results only for Qwen3-0.6B on Reddit Summary and Qwen3-4B on HH-RLHF, without evaluations on larger models (e.g., 7B+) or controlled comparisons across tasks. The evidence for the method’s applicability across model scales and tasks therefore remains limited.

Questions for authors
  1. Ablation of the core feedback. What are the results when only the gradient from the task loss to the preference network in Eq. (12) is removed, while retaining the preference supervision loss and keeping the architecture, initialization, online sampling procedure, and budget unchanged? Can this control establish the independent benefit of bidirectional feedback?

  2. Basis of preference supervision. What justifies interpreting the transformed reward scores in Eq. (5) as the relative importance of objectives for a given prompt? Have the score scales and offsets of the reward models been calibrated? Please provide an explanation and validation of this mapping.

  3. Correspondence between training and inference. When weights are specified in Figures 2–6 and Appendix C, is the preference network used during inference? If so, how is its output combined with the externally supplied weights? Please clarify which experiments evaluate automatic preference guidance and which evaluate external preference control.

  4. Comparison protocol and reliability of the results. Which baselines were reproduced in this work, and which results were taken from prior reports? Are the base models, trainable parameters, and training and online sampling budgets comparable across methods? Please also provide numerical results and uncertainty estimates for the main comparisons to assess whether the observed performance differences are robust.

  5. Could you add experiments across models and tasks, such as evaluating Qwen3-4B on Reddit Summary, to further assess generalizability and help disentangle the effects of model scale and task differences?

  6. Could you add a table summarizing the main experimental results (e.g., Table 1) for clearer presentation? If a table is not well suited to the experimental setup, a brief explanation would suffice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.