acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Cross-Objective Interference in Multi-Objective Alignment

Abstract

We study a persistent failure mode in multi-objective alignment for large language models (LLMs), in which scalarized training improves only a subset of objectives while the others degrade. We formalize this phenomenon as cross-objective interference and conduct, to our knowledge, the first systematic study of nine scalarization algorithms, finding it pervasive yet strongly model-dependent. To explain interference, we derive a local covariance law in which an objective rises or falls with the sign of its covariance with the scalarized score. We extend this analysis to clipped surrogate objectives used in modern alignment, demonstrating that the covariance law remains valid under mild conditions despite clipping. Building on this analysis, we propose COVariance-floor Enforced Reweighting (COVER), a plug-and-play controller that raises an objective's weight only when its reward's covariance with the clipped advantage weight falls below a target. Through extensive experiments, we show that COVER effectively mitigates cross-objective interference while matching linear scalarization when objectives already co-improve. Finally, we complement these local improvement conditions with a global convergence analysis that establishes when non-convex scalarized optimization satisfies the Polyak–ojasiewicz condition and how cross-objective interference depends on model geometry.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Alignment, Multi-objective Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f7iSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces COVER, a controller used during multi-objective fine-tuning to choose which objectives are emphasized in the reward so that optimization is not effectively dominated by only a subset of objectives. A theoretical analysis of how covariance between the per-objective rewards and the scalarized reward influences the rate of improvement (or performance loss) for each objective under a first-order approximation is given to justify adapting the reward weights during training based on that covariance. This in accompanied by a PL-type inequality which relates the gradient of the scalarization-induced value to the suboptimality of the solution. Empirically, the value of adapting objective weights throughout training is shown by comparing COVER with different static weight choices. COVER is also compared with a variety of multi-objective fine-tuning baselines across five benchmarks featuring auxiliary objectives that are conflicting or relatively easy to optimize, creating a high risk that training focuses on some objectives at the expense of others.

Critical strengths

The empirical evaluation of COVER is thorough in its scope, although the analysis focuses more on GSM8K than on the other benchmarks. The use of simple auxiliary objectives seems reasonable because the focus is on cases where the optimization process needs to be prevented from focusing on one easy objective at the expense of others.

The intuition behind the covariance law is introduced clearly.

Critical weaknesses

The main weaknesses I see have to do with generalization and practical usage of the COVER algorithm. Sensitivity to the covariance targets appears to be a major roadblock for applying COVER effectively to new tasks and models. Tables 5 and 9 indicate that simply ensuring the target is positive is not enough to achieve a benefit over the Linear or Dynamic baselines in general. Table 8 shows that even when the targets for primary and auxiliary objectives are the same, whether COVER favors the primary objective or a secondary objective relative to the Linear baseline depends on the exact circumstances. The paper does not provide clear guidance on how to choose the exact values of the targets which can be assumed to generalize to models with significantly different capabilities or more complex combinations of objectives. For a user with underlying intentions about which objective is more important and how much they are willing to give up one reward to increase another, it is important for them to know how they can choose the covariance targets according to those preferences.

I also see a potential issue with the conciseness objective which I have addressed in the questions section.

Questions for authors

Was the initial choice of covariance targets (0.15 for the task objective, 0.05 for the auxiliary objectives) set based on the intuition that the task objective is more important or harder to optimize than the auxiliary objectives? Are these hyperparameters meant to reflect the user's intentions for the relative importance of objectives? If so, was the though process that the task objective should be three times as important as each auxiliary objective?

In MATH-500, the conciseness score depends on past training dynamics (specifically the running average response length). How can comparisons for that objective across algorithms be meaningful when the response length dynamics vary so much as seen in Figure 1? For example, if the mean response length decays very slowly for one algorithm throughout training, but for another algorithm it decays quickly and then oscillates, the first algorithm may have a higher conciseness score during the later stages of training despite having longer responses overall because it keeps most of its responses just below the running average. It would be helpful to run an experiment where the conciseness objective is based on fixed thresholds or relates linearly to the actual response length to verify that the resulting conciseness scores still match what is expected.

5 comments

Thank you for the careful review and for the positive comments on our evaluation, the auxiliary objectives and the intuition behind the covariance law. We address your concerns below.

W1. Sensitivity to the covariance targets appears to be a major roadblock, since a positive target is not enough to achieve a benefit over Linear or Dynamic (Tables 5 and 9)

Target sensitivity is not a roadblock. A target set just above the covariance level beats Linear at both 1.5B and 7B (Tables 5 and 9). COVER applies Thm 4.1 directly. The covariance is each objective's first-order improvement rate, so a positive target acts only once it lies above that rate, and otherwise COVER safely falls back to Linear (lines 376–377 and 393). At 1.5B the covariance is usually between 0.05 and 0.16 (line 419), and every target above this range beats Linear and Dynamic in both final accuracy and drop (up to 0.435 vs. 0.359 and 0.361, Tables 4 and 5). At 7B the mild target of 0.10 beats Linear in all three seeds (0.449 vs. 0.299). We honestly report COVER's results in every setting we tested, including those where it only matches Linear. That is expected, because COVER is not intended to beat every baseline but to show that the covariance law works in practice.

W2. Table 8 shows that even with the same targets, whether COVER favors the primary or a secondary objective relative to Linear depends on the exact circumstances

COVER behaves consistently across Table 8. It favors the task objective in every domain, and it matches or beats Linear on that objective in all four. It raises the task objective's weight on 77–100% of steps, more than any auxiliary weight, and the weight's share of the total grows from 33% (50% on MBPP) to 44–71%. On MBPP, where the pass rate declines gradually, COVER cuts the drop from 0.141 to 0.090. On GSM8K and PKU-SafeRLHF, the task objective already plateaus at a high level under Linear (0.90 and 0.999), so COVER matches Linear. On XSum, coverage loses nearly half its value in the first ten steps under both methods, faster than the COVER can respond, and COVER again matches Linear.

W3. The paper gives no clear guidance on choosing the targets for models with different capabilities, more complex combinations of objectives, or a user's preferences

We want to clarify that for a fair comparison, we deliberately kept the targets fixed rather than tuning them per model, objective or task. We use 0.15 for the task objective and 0.05 for each auxiliary objective in every experiment of Tables 4 and 6–8, which cover both Qwen2.5-1.5B models and all six datasets. To adapt COVER to a new model or set of objectives, we recommend setting each target just above the covariance level that COVER logs while that objective declines. Tables 5 and 9 already show in detail how the target affects accuracy at 1.5B and 7B, so they can guide this choice for models of other scales.

Q1. Do the targets 0.15 and 0.05 mean that the task objective is three times as important as each auxiliary objective?

No. The targets are floors, not importance weights. An objective's weight depends on how long and how far its covariance stays below its floor. With the same targets, the final task weight is 1.1–1.4 times the auxiliary weights on IFEval but 3.7–6.8 times on PKU-SafeRLHF.

Q2. With a running-average conciseness score, how can comparisons across algorithms be meaningful when length dynamics vary so much, as seen in Fig. 1?

We chose the running average on purpose. A fixed length threshold does not generalize, because the length that counts as concise is unknown before training and differs across tasks and models. Once all responses in a batch lie on one side of a fixed threshold, they all receive the same score, so the conciseness objective carries no learning signal. The running average keeps the threshold dynamic and at the scale of the policy's own responses, so the reward always separates shorter responses from longer ones. This is a standard practice also used by Dynamic (TACL 2026).

We hope these clarifications address your concerns, and would be grateful if you would reconsider your rating.

Dear Reviewer f7iS,

Thank you again for your valuable comments. We understand this is a busy time, and we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns. If there are any unsolved concerns, we will be very willing to continue the discussion.

Best Regards,

Authors

Reviewer_f7iSOpenReview ↗

Response to Author Rebuttal

I would like to thank the authors for their effort in responding. I am still not convinced about the practicality of COVER. In their rebuttal to W3, the authors seem to be recommending that the target covariance be adjusted during training such that an objective's weight is forced to increase when that objective is observed to be decreasing. However, at that point couldn't a similar effect be achieved by simply increasing an objective's weight when it is observed to be decreasing? On the other hand, if the targets are fixed and they happen to be too large or too small, then COVER may fail to provide a benefit over Linear, especially when there are two heavily-conflicting objectives. Thus there is a risk that practitioners will have to use a lot of trial-and-error to get a satisfying outcome, which makes the algorithm less practical.

Reviewer_SPD6OpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the negative intervention between different rewards during the post-training process. It identifies the covariance as an early signal for interference between different training objectives and proposes COVER as a new method to deal with it. Specifically, COVER would adjust the weight of different rewards in the final scalar rewards to avoid negative covariance and reduce the deterioration during multi-objective training.

Critical strengths
  • It proposes a novel method of adjusting the weight of rewards during multi-objective post-training
  • It run extensive experiments comparing 10 different baseline methods.
  • The paper is well-written and easy to follow.
Critical weaknesses
  • Most experiments are conducted on small language modes less than 9B so it remains unknown whether these findings could transfer to larger language models.
  • The definition of helpfulness for IFeval is unconvincing and it is unclear why the length of response could be a proxy of helpfulness. Does IFEval itself contains any constraints on response length? If so is there any conflict between your definition of helpfulness and instruction following?
  • From Figure 2 it seems Tchebycheff is on par with the proposed COVER. And in Figure 1, COVER seems to underperform Dynamic baseline on Qwen3-1.7B-Base.
Questions for authors

The final accuracy of COVER in Figure 2 seems to conflict with the final line in Table 4. Is it because they have different experiment setup?

2 comments

Thank you for the review and for the positive comments on the method, the breadth of the comparison and the writing. We address each concern below.

W1. Most experiments are conducted on small language models less than 9B, so it remains unknown whether these findings could transfer to larger language models.

Our findings do transfer to larger models. The paper already shows this on five 7–8B models from four families (Fig. 6) and for COVER at 7B (Table 9). Accuracy falls on four of the five models (for example, from 0.67 to 0.51 on Qwen2.5-7B-Base) while responses shorten and clarity rises, the same pattern as at 1.5B. The Fig. 6 caption states that interference "is not a small-model artifact". At 7B, COVER with target of 0.10 roughly halves Linear's accuracy drop (0.187 vs. 0.357) and ends higher in all three seeds (0.449 vs. 0.299).

W2. The definition of helpfulness for IFEval is unconvincing and it is unclear why the length of response could be a proxy of helpfulness. Does IFEval itself contain any constraints on response length, and is there any conflict with instruction following?

The length proxy is grounded in established RLHF evidence, and it rarely conflicts with constraint following. Only 23 of our 414 prompts cap the response below 150 words. Singhal et al. (2024) show that a reward based on length alone recovers most of the improvement of RLHF over supervised fine-tuning (lines 1011–1015), and a rule-based proxy avoids relying on an LLM judge (lines 1000–1002). IFEval does contain length constraints, in 115 of our 414 prompts, but 55 of them ask for a minimum length, which agrees with the helpfulness reward. The training covariance between helpfulness and constraint following is close to zero (mean −0.11×10−2-0.11\times10^{-2}), while the one between helpfulness and conciseness is about 16 times larger in magnitude (−1.72×10−2-1.72\times10^{-2}). Helpfulness therefore conflicts with conciseness, as we designed (lines 1016–1017), and not with constraint following.

W3. From Fig. 2 it seems Tchebycheff is on par with the proposed COVER, and in Fig. 1 COVER seems to underperform the Dynamic baseline on Qwen3-1.7B-Base

We respectfully disagree with both observations. In Fig. 2, Tchebycheff is not on par with COVER. On Qwen2.5-1.5B-Base it loses 0.14 from its peak accuracy against 0.04 for COVER, and on Qwen3-1.7B-Base COVER has the highest final accuracy of all ten methods. In Fig. 1b, COVER does not underperform Dynamic on Qwen3-1.7B-Base. Its accuracy is at or above Dynamic's on 84 of 113 evaluations and ends higher (0.648 vs. 0.620). Tchebycheff also needs responses three to four times longer than COVER's to keep its accuracy (Fig. 1a).

We also want to highlight two merits of COVER. It is harmless by default and safely falls back to Linear when objectives co-improve (lines 376–377, 393 and 405–416), which neither Tchebycheff nor Dynamic does. It is also much more efficient than Dynamic because it needs no per-objective gradients (lines 433–435).

Q. The final accuracy of COVER in Fig. 2 seems to conflict with the final line in Table 4. Is it because they have different experiment setups?

This is not a mismatch, and neither experiment setup is wrong. Fig. 2 compares ten methods on three models in one figure, so it shows one run per method, smoothed with a moving average to make the results more robust to noise. Table 4 supports the quantitative claims, so it reports the mean and standard deviation over three seeds for each Qwen2.5-1.5B-Base experiment. Since the submission, we have been actively revising the paper for better clarity. The next version will state the number of runs and how each result is averaged.

We hope these clarifications address your concerns, and would be grateful if you would reconsider your rating.

Dear Reviewer SPD6,

Thank you again for your valuable comments. We understand this is a busy time, and we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns. If there are any unsolved concerns, we will be very willing to continue the discussion.

Best Regards,

Authors

Reviewer_XsNkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors provide a systematic theoretical analysis of interactions among rewards in multi-objective reinforcement learning and derive COVER, a new reward-weighting method designed to mitigate interference among reward objectives.

Critical strengths
  • The paper includes both theoretical analysis and experiments, with detailed coverage and supporting details clearly documented in the appendices.
Critical weaknesses
  • Section 4 derives a series of results under the assumption that η\eta is sufficiently small. However, when η\eta is sufficiently small, the second term in Equation (2), the KL divergence term, dominates, making the first term negligible. This calls into question the practical relevance of the theoretical analysis in Section 4.
  • There is a disconnect between the theory and the experiments. The experiments in Section 4.4 use a KL coefficient of 0.001, corresponding to η=1000\eta = 1000. It is unclear how this value satisfies the assumption that η\eta is “sufficiently small.”
  • In Section 5, the definition of the proposed method, COVER, is compressed into a single paragraph, while the definition of the key coefficient wk,lclip(θ)w_{k,l}^{\text{clip}}(\theta) is deferred to the appendix. This presentation gives insufficient prominence to the paper’s central methodological contribution.
Questions for authors

Please address the concerns and clarification requests raised in the Critical Weaknesses section.

12 comments

Thank you for the review, and for recognizing the paper's combination of theoretical analysis and experiments. We address your concerns below.

W1. When η\eta is sufficiently small, the KL term in Eq. (2) dominates and the reward term becomes negligible

The KL term never dominates. For every η>0\eta>0, the step in Eq. (2) gains at least as much in the reward term as it pays in the KL term. The KL in Eq. (2) is measured from the current policy, so the optimizer p+∝pθ eηsp^+\propto p_\theta\,e^{\eta s} satisfies Ep+[s]−Epθ[s]≥1ηKL(p+∥pθ)E_{p^+}[s]-E_{p_\theta}[s]\ge\frac1\eta\mathrm{KL}(p^+\Vert p_\theta). To first order, the reward gain on the left is η Var(s)\eta\,\mathrm{Var}(s) and the KL term on the right is η2Var(s)\frac{\eta}{2}\mathrm{Var}(s), both of order η\eta. Your argument would apply to a penalty toward a fixed reference model, such as β KL(pθ∥pref)\beta\,\mathrm{KL}(p_\theta\Vert p_{\rm ref}) in App. C. A small η\eta only shortens each update, and each objective still changes by η Ex[Cov(rm,s)]+O(η2)\eta\,E_x[\mathrm{Cov}(r_m,s)]+O(\eta^2), as Thm 4.1 shows.

Actually the condition "sufficiently small" can be quantified. For rewards and the score in [0,1][0,1], with CmC_m the covariance in Thm 4.1,

∣rm(p+)−rm(pθ)−ηCm∣≤η2/8,\lvert r_m(p^+)-r_m(p_\theta)-\eta C_m\rvert\le\eta^2/8,

Each objective therefore increases when Cm>0C_m>0 and decreases when Cm<0C_m<0 for every 0<η<8∣Cm∣0<\eta<8\lvert C_m\rvert, and the two-mode example of $\S$4.2 holds exactly for every η\eta.

W2. The experiments use a KL coefficient of 0.001, which does not satisfy the assumption that η\eta is sufficiently small

There is no disconnect, because the KL coefficient 0.001 is not η\eta. They are two different quantities, and no run sets the η\eta of Eq. (2) at all. We distinguish the two below.

  • η\eta in Eq. (2) is the step size of a single update from the current policy pθp_\theta (line 207), and nothing more.
  • The 0.001 in Table 11 is β\beta, which weights KL(pθ∥pref)\mathrm{KL}(p_\theta\Vert p_{\rm ref}) toward the frozen reference model in the training loss (App. C, line 1440).

W3. COVER is compressed into a single paragraph and the key coefficient ww is deferred to the appendix

$\S$5 already gives COVER's update rule in the main text (lines 360–375). Only the definition of ww is deferred to the appendix (Cor. C.4), because of the page limit. Per your suggestion, the next version will define ww in $\S$5 and present the method there in full.

We hope these clarifications address your concerns, and would be grateful if you would reconsider your rating.

Dear Reviewer XsNk,

Thank you again for your valuable comments. We understand this is a busy time, and we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns. If there are any unsolved concerns, we will be very willing to continue the discussion.

Best Regards,

Authors

Why is β\beta not shown explicitly in Eq. (2)? Absorbing β\beta into the notation for the KL divergence is misleading, because it obscures the distinction between the divergence itself and its weight. I would also expect the theoretical upper bound to depend explicitly on β\beta, so that readers can assess how the choice of β\beta affects the bound. Finally, does Eq. (2) require β>0\beta>0? In other words, is the KL penalty mandatory, or can it be removed by setting β=0\beta=0?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.