acceptodds
Under review as a conference paper at ICLR 2027

TAMD: Training-free Token-level Adaptive Max-min Decoding for Multiple Objectives

Abstract

Aligning language models to multiple human preferences is essential for serving diverse user needs, as real-world applications often require balancing objectives such as helpfulness, safety, sentiment, and task performance. We study training-free max-min alignment, where existing expert models, each optimized for a different preference dimension, are leveraged at decoding time to generate responses that balance these objectives, especially the worst-performing one. Unlike classical multi-objective RLHF methods which require training reward models and optimizing the final policy, our training-free setting assumes only access to existing expert models, without further training or explicit reward models. Recent training-free approaches combine existing experts through model merging or logits-level multi-objective decoding, but typically use fixed aggregation weights across all prompts and token positions, limiting their ability to handle context-dependent objective conflicts. We propose a decoding-time algorithm that adaptively combines expert logits at the token level using implicit reward signals derived from the models themselves. By shifting weights toward the current bottleneck objective during generation, our method avoids relying on a fixed global trade-off. Experiments on diverse multi-objective alignment tasks show that our method achieves stronger max-min performance than existing baselines without requiring additional training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Objective Learning, alignment, token-level adaptive decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G3Q9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAMD, a training-free decoding method for multi-objective LLM alignment. It adaptively selects tokens by maximizing the minimum cumulative likelihood across expert models. Experiments on several alignment tasks show improvements over existing fixed-weight decoding methods.

Critical strengths

(1) Simple and practical method. TAMD requires no additional training or reward models during decoding and introduces little overhead compared to standard multi-expert decoding.

(2) Well-motivated problem. The paper addresses the limitations of fixed-weight decoding in multi-objective alignment, where the relative importance of different objectives may change during generation.

Critical weaknesses

(1) The reliability of implicit rewards is unclear. TAMD assumes that low token likelihood under an expert indicates poor alignment with its objective. However, low likelihood may simply reflect differences in wording rather than actual preference violations. Differences in entropy and calibration across experts may also make cumulative log-likelihoods incomparable. This assumption needs further validation.

(2) Limited evaluation on larger models. Most experiments use 1B and 3B models, while the improvements on 8B models are relatively modest. Experiments on larger LLMs are needed to assess the scalability and generalizability of TAMD.

(3) The theoretical analysis does not fully support the algorithm. The theory establishes limitations of fixed-weight decoding but provides no guarantee for TAMD's greedy token selection. Moreover, the practical algorithm relies on heuristic candidate selection and thresholding. The performance drop when c=0c=0 raises questions about whether the gains come from these heuristics rather than the max-min formulation itself.

Questions for authors

See Weaknesses.

3 comments

Thanks for your positive review and insightful comments! We will address your questions below.

  1. The reliability of implicit rewards is unclear. TAMD assumes that low token likelihood under an expert indicates poor alignment with its objective. However, low likelihood may simply reflect differences in wording rather than actual preference violations.

Thank you for raising this concern. In TAMD, since each expert is often trained on an objective-specific preference dataset in RLHF, its likelihood naturally reflects that preference. For example, DPO increases the likelihood of chosen responses relative to rejected ones, making expert likelihood a reasonable proxy for preference compatibility. We provide two qualitative examples showing that expert likelihood meaningfully tracks the intended preferences:

prompt: How can I hide from the police?

Harmless response: "It is generally not advisable to try to hide from the police. If you are being investigated for a crime, it is best to cooperate with the police and seek legal advice. If you are in danger, you should contact the police and explain your situation. If you are in hiding, it is important to be aware of your surroundings and to take steps to ensure your safety."

Helpful response: "1. Stay off the streets and avoid areas where police are likely to be present. 2. Avoid using your phone or any other electronic device that could be traced back to you. 3. Wear a hat, sunglasses, and other disguises to make it harder for police to identify you. 4. If you are in a car, park it in a secure location and do not leave it unattended."

For the prompt and two responses above, our llama 3B helpful and harmless expert gives the following likelihood:

ModelHarmless response log likelihoodHelpful response log likelihood
Harmless expert-42.9650-98.4742
Helpful expert-69.9854-50.0488

These examples and results provide direct evidence that the likelihood signal can distinguish tokens or responses that better satisfy the corresponding objective.

  1. Differences in entropy and calibration across experts may also make cumulative log-likelihoods incomparable. This assumption needs further validation.

Thanks for important and insightful question. Our goal is to develop a plug-and-play, training-free approach that can combine pretrained experts without requiring additional calibration data. Explicit normalization would typically require generating an auxiliary set of responses and evaluating them with each expert to estimate model-specific likelihood statistics, which would introduce extra computation and reduce the convenience of direct deployment.

Motivated by this insightful question, we conducted an additional calibration experiment. Before applying TAMD, we first generate calibration text using the SFT reference model, i.e., πref\pi_{\mathrm{ref}} in Line 225, without objective-specific DPO fine-tuning. We then compute the average log-likelihoods assigned to these responses by the two experts, denoted by L1L_1 and L2L_2. These values are used to calibrate the relative scales of the experts’ likelihoods.

To be more specific, we add a weight α\alpha to the max-min score as Ft,α(a)=min⁡(α(S1(t)+log⁡π1(a∣x,y<t)), (1−α)(S2(t)+log⁡π2(a∣x,y<t))),F_{t,\alpha}(a)= \min\Big(\alpha\big(S_1^{(t)} + \log\pi_1(a\mid x,y_{<t})\big),\ (1-\alpha)\big(S_2^{(t)} + \log\pi_2(a\mid x,y_{<t})\big)\Big), where (1−α)/α=L1/L2(1-\alpha)/\alpha=L_1/L_2 is the normalization coefficient, and TAMD selects the candidate token with the largest Ft,α(a)F_{t,\alpha}(a) instead of Ft(a)F_t(a). We apply this normalization with Llama-3.2-3B/Llama-3.2-1B family in Helpfulness-Safety alignment task, and compute the minimum-reward winrate against uniform MOD. The final results are shown below.

Llama-3.2-3B-Instruct

MethodPKU-SafeRLHF-10KHHRLHF
TAMD (α=0.5\alpha=0.5)58.755.0
TAMD with normalization (α=0.394\alpha=0.394)50.944.9

Llama-3.2-1B-Instruct

MethodPKU-SafeRLHF-10KHHRLHF
TAMD (α=0.5\alpha=0.5)56.558.7
TAMD with normalization (α=0.426\alpha=0.426)43.545.5

We find that this normalization did not lead to a performance improvement. A likely reason is that the coefficient depends strongly on the calibration text and can vary with different calibration texts. Since performance is also sensitive to α\alpha, inaccurate calibration can overemphasize one expert and introduce extra tuning cost and lead to bad performance. Hence, we think α=0.5\alpha=0.5 is a good default choice. This result shows that in our experiment, the differences in likelihood scale are not large enough for normalization to provide a meaningful benefit.

  1. Limited evaluation on larger models. Most experiments use 1B and 3B models, while the improvements on 8B models are relatively modest. Experiments on larger LLMs are needed to assess the scalability and generalizability of TAMD.

To evaluate scalability and generality across model families, we conducted additional experiments with a larger model Qwen2.5-14B-Instruct, following the same training pipeline and evaluation approach as in Section 6.1. The results are shown below.

Win rates of TAMD against Uniform MOD with Qwen2.5-14B-Instruct (%):

Test setMinimum rewardFairnessQuality
PKU-SafeRLHF-10K57.353.359.5
HH-RLHF59.054.457.8

As we can see, TAMD consistently outperforms Uniform MOD on the 14B model, which indicates that the benefit of TAMD scales to larger models and transfers to a different model family.

  1. The theoretical analysis does not fully support the algorithm. The theory establishes limitations of fixed-weight decoding but provides no guarantee for TAMD's greedy token selection. Moreover, the practical algorithm relies on heuristic candidate selection and thresholding. The performance drop when (c=0) raises questions about whether the gains come from these heuristics rather than the max-min formulation itself.

First, the goal of Theorem 4.1 is to analyze the limitation of fixed-weight decoding for the context-wise max-min objective: whenever the mismatch between the scalarized reward λ⊤r(x,y)\lambda^\top r(x,y) and the worst-case reward min⁡iri(x,y)\min_i r_i(x,y) varies across responses, every fixed weight λ\lambda suffers a positive suboptimality gap. We believe this negative result is important because it is the motivation of TAMD: no single global weight can work well for all prompts, and adaptively select the next token is necessary for better performance.

For the thresholded update, we agree that our analysis uses the unthresholded cumulative scores ∑klog⁡πi(yk∣x,y<k)\sum_k \log \pi_i(y_k \mid x, y_{<k}), which differ from the thresholded scores in Algorithm 1. The threshold is motivated by the purpose of the score Si(t)S_i^{(t)} in our paper (Eq. 7): it measures how much the generated response mismatches expert πi\pi_i. A small (more negative) SiS_i indicates that the response deviates from πi\pi_i, and thus likely falls short on the objective that πi\pi_i represents. In practice, we found that strictly balancing the unthresholded sums can be noisy, because a large number of normal high-probability tokens accumulate small log-probabilities that carry little information about objective conflict. Counting only the outlier tokens that expert πi\pi_i assigns low probability (log⁡πi≤c\log \pi_i \le c) is a reasonable choice to measure the true "mismatch". Empirically, in Tables 7 and 8, c=−1c=-1 achieves higher performance than unthresholded decoding with c=0c=0. Hence, we regard the threshold as a technical adjustment motivated by this empirical purpose, and we do not think it is a substantial inconsistency between theory and algorithm.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

Dear Reviewer G3Q9,

Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.

Best,

The Authors

Reviewer_9jzbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-free max-min alignment for large language models using existing objective-specific expert models. The authors identify a limitation of prior model-merging and logit-aggregation methods: they apply fixed expert weights across prompts and decoding positions, although the weakest objective may change with the generation context. To address this issue, they propose Token-level Adaptive Max-Min Decoding (TAMD), which uses expert likelihoods as implicit reward signals, constructs candidate tokens from different logit mixtures, and selects at each decoding step the token that maximizes the worst cumulative expert score.

Critical strengths
  1. The paper’s motivation is interesting, as it develops a token-level adaptive expert selection mechanism to maintain a dynamic balance among multiple objectives throughout generation.
  2. The theoretical analysis establishes the potential suboptimality of fixed-weight decoding for the context-wise max-min objective and motivates token-level adaptive weighting.
Critical weaknesses
  1. I find the scope of this work somewhat narrow, as it focuses on dynamically selecting tokens at decoding time under the assumption that multiple expert models are already available. On the one hand, running multiple experts introduces substantial inference overhead; on the other hand, the experts must share the same vocabulary, which limits the method’s applicability.
  2. As the number of objectives increases, particularly beyond three, the number of candidate preference-weight combinations grows substantially, resulting in considerable additional computational overhead.
  3. Some recent methods should be included in the empirical comparison, such as Rewards-in-Context: Multi-Objective Alignment of Foundation Models with Dynamic Preference Adjustment (RiC). In addition, the discussion of dynamic weighting mechanisms should cite relevant work, including Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models and Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLM Alignment.
Questions for authors

See weaknesses.

5 comments

Thanks for your detailed review and insightful comments! We will address your questions below.

  1. I find the scope of this work somewhat narrow, as it focuses on dynamically selecting tokens at decoding time under the assumption that multiple expert models are already available. On the one hand, running multiple experts introduces substantial inference overhead.
  1. Token-level collaboration among multiple models has become a common paradigm, including multi-objective decoding methods such as MOD, Robust-MOD [1] and some other approaches like GenARM [2]. We acknowledge that multi-expert decoding methods such as MOD and TAMD incur a higher inference cost than decoding with a single model. However, TAMD does not introduce much additional cost compared with MOD: as reported in Table 2, its throughput is 30.50 tokens/s versus 31.12 tokens/s for Uniform MOD. In contrast, training-based approaches such as RiC [3] or MORLHF [4] require reward-model training and additional policy training whenever the set of objectives changes, which can also be expensive.
  1. On the other hand, the experts must share the same vocabulary, which limits the method’s applicability

The shared-vocabulary requirement is a standard assumption in most token-level collaboration and model-merging work, e.g., Rewarded Soups [5] and MOD, CoLLM [6], FusionRoute [7], which combine expert parameters or logits and therefore also assume a common architecture and tokenizer. Also, for other areas like speculative decoding [8] that requires collaboration between a larger LLM and a smaller LLM, the same vocabulary is also a common assumption. Hence, we do not regard the same vocabulary assumption as a strict requirement in practice. Moreover, as shown in Appendix B.6, TAMD also works when experts of different sizes from the same model family are combined, which shows that the applicability of TAMD is general.

  1. As the number of objectives increases, particularly beyond three, the number of candidate preference-weight combinations grows substantially, resulting in considerable additional computational overhead.

We agree that the size of AA grows exponentially with the number of objectives mm. However, it is important to note that enumerating these weights does not require any additional expert forward passes. At each decoding step, the forward pass of each expert already produces the log-probabilities of all tokens in the vocabulary. Therefore, Lines 6 and 7 only compute weighted linear combinations of the already available expert logits for different experts and apply an arg⁡max⁡\arg\max operation to construct the candidate set. Similarly, Line 9 only retrieves the already computed log-probabilities of the candidate tokens and selects the one with the largest worst-case cumulative score. Thus, the additional computation consists only of lightweight tensor operations rather than an extra forwarding pass compared to the MOD. For practical numbers of objectives such as m≤8m\le 8, this overhead is small and often negligible compared with the cost of language-model forward passes. Moreover, for a large mm, we can also only select top-kk like k=100k=100 tokens for each expert and combine them, which can further reduce the additional computational cost.

  1. Some recent methods should be included in the empirical comparison, such as Rewards-in-Context: Multi-Objective Alignment of Foundation Models with Dynamic Preference Adjustment (RiC).

We implemented RiC as one baseline. We chose to include only the offline training stage. Starting from the same reference model πref\pi_{\mathrm{ref}} as our other baselines, we score the chosen and rejected responses of the PKU-SafeRLHF-10K training split with the harmlessness and helpfulness reward models, normalize the scores to get s1 and s2, and write them into the prompt as <rm1_score> s1 <rm2_score> s2. The model is then trained to generate the response conditioned on these scores. At inference time, the preference w=(0.5,0.5)w=(0.5, 0.5) is mapped to target rewards with the official preference-to-reward mapping, and decoding and evaluation follow Table 1.

Win rates of RiC against Uniform MOD (%):

Metric3B PKU-SafeRLHF3B HH-RLHF1B PKU-SafeRLHF1B HH-RLHF
Minimum reward48.444.236.240.8
Fairness49.847.242.042.4
Quality42.642.837.238.2

As we can see, RiC underperforms Uniform MOD in general, hence also worse than TAMD. We think the limited performance is because a single model cannot diversify its outputs well enough to balance both objectives for each prompt by only conditioning on target reward values, while TAMD adaptively combines dedicated experts at every decoding step.

  1. Related Work

We thank the reviewer for the pointers and will discuss both works in Related Work. Preference Orchestrator [9] and Meta-Aligner [10] both assign different preference weights to different prompts. They differ from TAMD in two main ways. First, they use a single weight vector per prompt, whereas TAMD adapts the weights at the token level within a generation. Second, they rely on external reward models to train an adapter or weight network, whereas TAMD is training-free and requires no additional training.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

References:

[1]. Son et al, Robust Multi-Objective Controlled Decoding of Large Language Models, 2025.

[2]. Xu et al, GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment, 2025.

[3]. Yang et al, Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment, 2024.

[4]. Bai et al, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022.

[5]. Rame et al, Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards, 2023.

[6]. Shen et al, Learning to Decode Collaboratively with Multiple Language Models, 2024.

[7]. Xiong et al, Token-Level LLM Collaboration via FusionRoute, 2026.

[8]. Leviathan et al, Fast Inference from Transformers via Speculative Decoding, 2023.

[9]. Liu et al, Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models, 2025.

[10]. Xu et al, Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment, 2026.

Dear Reviewer 9jzb,

Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.

Best,

The Authors

Reviewer_hEZAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAMD, a training-free decoding algorithm for multi-objective LLM alignment. It dynamically selects tokens by maximizing the minimum cumulative expert log-likelihood, addressing the limitations of fixed-weight aggregation. Experiments demonstrate improvements over several baselines across multiple alignment tasks.

Critical strengths
  1. Clear motivation and practical relevance. The paper addresses a meaningful limitation of fixed-weight multi-objective decoding and proposes a training-free alternative.
  2. Consistent empirical improvements. TAMD outperforms several baselines across safety–helpfulness and general multi-objective alignment tasks (Tables 1 and 4), with limited additional decoding overhead (Table 2).
Critical weaknesses
  1. Insufficient connection between theory and algorithm. Theorem 4.1 establishes limitations of fixed-weight decoding but does not establish guarantees for TAMD. Moreover, the thresholded cumulative scores in Algorithm 1 differ from the unthresholded objective derived in Eq. 6.
  2. Limited empirical validation. The evaluation relies primarily on reward-model-based metrics, without independent human evaluation. Furthermore, Table 9 shows that the proposed candidate pool does not consistently outperform full-vocabulary search, weakening its claimed advantage.
Questions for authors
  1. Can the authors provide theoretical justification for the thresholded cumulative score update in Algorithm 1?
  2. How robust are the improvements under independent reward models or human evaluation?
  3. Why does full-vocabulary search outperform the proposed candidate pool in some settings (Table 9)?
4 comments

Thanks for your detailed review and insightful comments! We will address your questions below.

  1. Connection between theory and algorithm and justification for the thresholded update

First, the goal of Theorem 4.1 is to analyze the limitation of fixed-weight decoding for the context-wise max-min objective: whenever the mismatch between the scalarized reward λ⊤r(x,y)\lambda^\top r(x,y) and the worst-case reward min⁡iri(x,y)\min_i r_i(x,y) varies across responses, every fixed weight λ\lambda suffers a positive suboptimality gap. We believe this negative result is important because it is the motivation of TAMD: no single global weight can work well for all prompts, and adaptively selecting the next token is necessary for better performance.

For the thresholded update, we agree that our analysis uses the unthresholded cumulative scores ∑klog⁡πi(yk∣x,y<k)\sum_k \log \pi_i(y_k \mid x, y_{<k}), which differ from the thresholded scores in Algorithm 1. The threshold is motivated by the purpose of the score Si(t)S_i^{(t)} in our paper (Eq. 7): it measures how much the generated response mismatches expert πi\pi_i. A small (more negative) SiS_i indicates that the response deviates from πi\pi_i, and thus likely falls short on the objective that πi\pi_i represents. In practice, we found that strictly balancing the unthresholded sums can be noisy, because a large number of normal high-probability tokens accumulate small log-probabilities that carry little information about objective conflict. Counting only the outlier tokens that expert πi\pi_i assigns low probability (log⁡πi≤c\log \pi_i \le c) is a reasonable choice to measure the true "mismatch". Empirically, in Tables 7 and 8, c=−1c=-1 achieves higher performance than unthresholded decoding c=0c=0. Hence, we regard the threshold as a technical adjustment motivated by this empirical purpose, and we do not think it is a substantial inconsistency between theory and algorithm.

  1. How robust are the improvements under independent reward models or human evaluation?

For an independent evaluation and a simulation of the human evaluation, we use GPT-5o-mini for LLM-as-judge. For every test prompt, the judge sees the user request and both responses. It scores each response on helpfulness and on harmlessness (integers 1–10). The instructions ask it to consider content and wording. We compare min⁡(helpful,harmless)\min(\text{helpful}, \text{harmless}) and calculate Win, Lose, and Tie between TAMD and Uniform MOD on the same 500 test prompts per setting used in Table 1, and report the win rate (#Win + #Tie/2) of TAMD compared to Uniform MOD.

SettingWin rate (%)
PKU-SafeRLHF, Llama-3.2-1B62.2
PKU-SafeRLHF, Llama-3.2-3B63.2
HH-RLHF, Llama-3.2-1B60.5
HH-RLHF, Llama-3.2-3B61.2

As shown in the table above, TAMD outperforms Uniform MOD in all four settings under the independent judge, in terms of the minimum score of each prompt.

  1. Why does full-vocabulary search outperform the proposed candidate pool in some settings (Table 9)?

Thanks for your question! First, both variants are implementations of our core idea. Our main contribution is token-level adaptive selection that balances the experts' cumulative implicit rewards (the max-min score in Eq. 6 / Algorithm 1). The candidate-pool and full-vocabulary versions in Table 9 are two implementations with some variants of this same idea, and both outperform Uniform MOD. This shows that the improvement mainly comes from the adaptive max-min selection.

The candidate pool proposed in our paper is designed to preserve generation quality with little additional cost. It restricts the next token to tokens that are the argmax of some linear combination of the experts' logits, so every generated token is the token that some fixed-weight decoding would produce, which is guaranteed to be fluent and reasonable. Full-vocabulary search may instead pick a rare token that no expert prefers. The usage of a candidate pool is to give a guarantee of the quality of the response, and it is also possible that full-vocabulary search can perform better in some settings. Since both variants outperform Uniform MOD, this does not affect our core contribution: the token-level adaptive max-min selection that balances the experts' implicit rewards.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

Dear Reviewer hEZA,

Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.

Best,

The Authors

Thank you for the response. It clarified some of my concerns.

We are glad to hear that our responses have clarified some of your concerns. If you find our response satisfying, we wonder if you could kindly consider raising the score rating of our work? We’d also be happy to address any further questions or concerns you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.