- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TAMD, a training-free decoding method for multi-objective LLM alignment. It adaptively selects tokens by maximizing the minimum cumulative likelihood across expert models. Experiments on several alignment tasks show improvements over existing fixed-weight decoding methods.
Critical strengths
(1) Simple and practical method. TAMD requires no additional training or reward models during decoding and introduces little overhead compared to standard multi-expert decoding.
(2) Well-motivated problem. The paper addresses the limitations of fixed-weight decoding in multi-objective alignment, where the relative importance of different objectives may change during generation.
Critical weaknesses
(1) The reliability of implicit rewards is unclear. TAMD assumes that low token likelihood under an expert indicates poor alignment with its objective. However, low likelihood may simply reflect differences in wording rather than actual preference violations. Differences in entropy and calibration across experts may also make cumulative log-likelihoods incomparable. This assumption needs further validation.
(2) Limited evaluation on larger models. Most experiments use 1B and 3B models, while the improvements on 8B models are relatively modest. Experiments on larger LLMs are needed to assess the scalability and generalizability of TAMD.
(3) The theoretical analysis does not fully support the algorithm. The theory establishes limitations of fixed-weight decoding but provides no guarantee for TAMD's greedy token selection. Moreover, the practical algorithm relies on heuristic candidate selection and thresholding. The performance drop when raises questions about whether the gains come from these heuristics rather than the max-min formulation itself.
Questions for authors
See Weaknesses.
3 comments
- Limited evaluation on larger models. Most experiments use 1B and 3B models, while the improvements on 8B models are relatively modest. Experiments on larger LLMs are needed to assess the scalability and generalizability of TAMD.
To evaluate scalability and generality across model families, we conducted additional experiments with a larger model Qwen2.5-14B-Instruct, following the same training pipeline and evaluation approach as in Section 6.1. The results are shown below.
Win rates of TAMD against Uniform MOD with Qwen2.5-14B-Instruct (%):
| Test set | Minimum reward | Fairness | Quality |
|---|---|---|---|
| PKU-SafeRLHF-10K | 57.3 | 53.3 | 59.5 |
| HH-RLHF | 59.0 | 54.4 | 57.8 |
As we can see, TAMD consistently outperforms Uniform MOD on the 14B model, which indicates that the benefit of TAMD scales to larger models and transfers to a different model family.
- The theoretical analysis does not fully support the algorithm. The theory establishes limitations of fixed-weight decoding but provides no guarantee for TAMD's greedy token selection. Moreover, the practical algorithm relies on heuristic candidate selection and thresholding. The performance drop when (c=0) raises questions about whether the gains come from these heuristics rather than the max-min formulation itself.
First, the goal of Theorem 4.1 is to analyze the limitation of fixed-weight decoding for the context-wise max-min objective: whenever the mismatch between the scalarized reward and the worst-case reward varies across responses, every fixed weight suffers a positive suboptimality gap. We believe this negative result is important because it is the motivation of TAMD: no single global weight can work well for all prompts, and adaptively select the next token is necessary for better performance.
For the thresholded update, we agree that our analysis uses the unthresholded cumulative scores , which differ from the thresholded scores in Algorithm 1. The threshold is motivated by the purpose of the score in our paper (Eq. 7): it measures how much the generated response mismatches expert . A small (more negative) indicates that the response deviates from , and thus likely falls short on the objective that represents. In practice, we found that strictly balancing the unthresholded sums can be noisy, because a large number of normal high-probability tokens accumulate small log-probabilities that carry little information about objective conflict. Counting only the outlier tokens that expert assigns low probability () is a reasonable choice to measure the true "mismatch". Empirically, in Tables 7 and 8, achieves higher performance than unthresholded decoding with . Hence, we regard the threshold as a technical adjustment motivated by this empirical purpose, and we do not think it is a substantial inconsistency between theory and algorithm.
Thanks again for your time and efforts! We will be happy to answer any further questions you may have.
Follow-Up
Dear Reviewer G3Q9,
Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.
Best,
The Authors
Thanks for your positive review and insightful comments! We will address your questions below.
Thank you for raising this concern. In TAMD, since each expert is often trained on an objective-specific preference dataset in RLHF, its likelihood naturally reflects that preference. For example, DPO increases the likelihood of chosen responses relative to rejected ones, making expert likelihood a reasonable proxy for preference compatibility. We provide two qualitative examples showing that expert likelihood meaningfully tracks the intended preferences:
prompt: How can I hide from the police?
Harmless response: "It is generally not advisable to try to hide from the police. If you are being investigated for a crime, it is best to cooperate with the police and seek legal advice. If you are in danger, you should contact the police and explain your situation. If you are in hiding, it is important to be aware of your surroundings and to take steps to ensure your safety."
Helpful response: "1. Stay off the streets and avoid areas where police are likely to be present. 2. Avoid using your phone or any other electronic device that could be traced back to you. 3. Wear a hat, sunglasses, and other disguises to make it harder for police to identify you. 4. If you are in a car, park it in a secure location and do not leave it unattended."
For the prompt and two responses above, our llama 3B helpful and harmless expert gives the following likelihood:
These examples and results provide direct evidence that the likelihood signal can distinguish tokens or responses that better satisfy the corresponding objective.
Thanks for important and insightful question. Our goal is to develop a plug-and-play, training-free approach that can combine pretrained experts without requiring additional calibration data. Explicit normalization would typically require generating an auxiliary set of responses and evaluating them with each expert to estimate model-specific likelihood statistics, which would introduce extra computation and reduce the convenience of direct deployment.
Motivated by this insightful question, we conducted an additional calibration experiment. Before applying TAMD, we first generate calibration text using the SFT reference model, i.e., πref in Line 225, without objective-specific DPO fine-tuning. We then compute the average log-likelihoods assigned to these responses by the two experts, denoted by L1 and L2. These values are used to calibrate the relative scales of the experts’ likelihoods.
To be more specific, we add a weight α to the max-min score as Ft,α(a)=min(α(S1(t)+logπ1(a∣x,y<t)), (1−α)(S2(t)+logπ2(a∣x,y<t))), where (1−α)/α=L1/L2 is the normalization coefficient, and TAMD selects the candidate token with the largest Ft,α(a) instead of Ft(a). We apply this normalization with Llama-3.2-3B/Llama-3.2-1B family in Helpfulness-Safety alignment task, and compute the minimum-reward winrate against uniform MOD. The final results are shown below.
Llama-3.2-3B-Instruct
Llama-3.2-1B-Instruct
We find that this normalization did not lead to a performance improvement. A likely reason is that the coefficient depends strongly on the calibration text and can vary with different calibration texts. Since performance is also sensitive to α, inaccurate calibration can overemphasize one expert and introduce extra tuning cost and lead to bad performance. Hence, we think α=0.5 is a good default choice. This result shows that in our experiment, the differences in likelihood scale are not large enough for normalization to provide a meaningful benefit.