- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces COVER, a controller used during multi-objective fine-tuning to choose which objectives are emphasized in the reward so that optimization is not effectively dominated by only a subset of objectives. A theoretical analysis of how covariance between the per-objective rewards and the scalarized reward influences the rate of improvement (or performance loss) for each objective under a first-order approximation is given to justify adapting the reward weights during training based on that covariance. This in accompanied by a PL-type inequality which relates the gradient of the scalarization-induced value to the suboptimality of the solution. Empirically, the value of adapting objective weights throughout training is shown by comparing COVER with different static weight choices. COVER is also compared with a variety of multi-objective fine-tuning baselines across five benchmarks featuring auxiliary objectives that are conflicting or relatively easy to optimize, creating a high risk that training focuses on some objectives at the expense of others.
Critical strengths
The empirical evaluation of COVER is thorough in its scope, although the analysis focuses more on GSM8K than on the other benchmarks. The use of simple auxiliary objectives seems reasonable because the focus is on cases where the optimization process needs to be prevented from focusing on one easy objective at the expense of others.
The intuition behind the covariance law is introduced clearly.
Critical weaknesses
The main weaknesses I see have to do with generalization and practical usage of the COVER algorithm. Sensitivity to the covariance targets appears to be a major roadblock for applying COVER effectively to new tasks and models. Tables 5 and 9 indicate that simply ensuring the target is positive is not enough to achieve a benefit over the Linear or Dynamic baselines in general. Table 8 shows that even when the targets for primary and auxiliary objectives are the same, whether COVER favors the primary objective or a secondary objective relative to the Linear baseline depends on the exact circumstances. The paper does not provide clear guidance on how to choose the exact values of the targets which can be assumed to generalize to models with significantly different capabilities or more complex combinations of objectives. For a user with underlying intentions about which objective is more important and how much they are willing to give up one reward to increase another, it is important for them to know how they can choose the covariance targets according to those preferences.
I also see a potential issue with the conciseness objective which I have addressed in the questions section.
Questions for authors
Was the initial choice of covariance targets (0.15 for the task objective, 0.05 for the auxiliary objectives) set based on the intuition that the task objective is more important or harder to optimize than the auxiliary objectives? Are these hyperparameters meant to reflect the user's intentions for the relative importance of objectives? If so, was the though process that the task objective should be three times as important as each auxiliary objective?
In MATH-500, the conciseness score depends on past training dynamics (specifically the running average response length). How can comparisons for that objective across algorithms be meaningful when the response length dynamics vary so much as seen in Figure 1? For example, if the mean response length decays very slowly for one algorithm throughout training, but for another algorithm it decays quickly and then oscillates, the first algorithm may have a higher conciseness score during the later stages of training despite having longer responses overall because it keeps most of its responses just below the running average. It would be helpful to run an experiment where the conciseness objective is based on fixed thresholds or relates linearly to the actual response length to verify that the resulting conciseness scores still match what is expected.
5 comments
Dear Reviewer f7iS,
Thank you again for your valuable comments. We understand this is a busy time, and we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns. If there are any unsolved concerns, we will be very willing to continue the discussion.
Best Regards,
Authors
Response to Author Rebuttal
I would like to thank the authors for their effort in responding. I am still not convinced about the practicality of COVER. In their rebuttal to W3, the authors seem to be recommending that the target covariance be adjusted during training such that an objective's weight is forced to increase when that objective is observed to be decreasing. However, at that point couldn't a similar effect be achieved by simply increasing an objective's weight when it is observed to be decreasing? On the other hand, if the targets are fixed and they happen to be too large or too small, then COVER may fail to provide a benefit over Linear, especially when there are two heavily-conflicting objectives. Thus there is a risk that practitioners will have to use a lot of trial-and-error to get a satisfying outcome, which makes the algorithm less practical.
Rebuttal
Thank you for the careful review and for the positive comments on our evaluation, the auxiliary objectives and the intuition behind the covariance law. We address your concerns below.
W1. Sensitivity to the covariance targets appears to be a major roadblock, since a positive target is not enough to achieve a benefit over Linear or Dynamic (Tables 5 and 9)
Target sensitivity is not a roadblock. A target set just above the covariance level beats Linear at both 1.5B and 7B (Tables 5 and 9). COVER applies Thm 4.1 directly. The covariance is each objective's first-order improvement rate, so a positive target acts only once it lies above that rate, and otherwise COVER safely falls back to Linear (lines 376–377 and 393). At 1.5B the covariance is usually between 0.05 and 0.16 (line 419), and every target above this range beats Linear and Dynamic in both final accuracy and drop (up to 0.435 vs. 0.359 and 0.361, Tables 4 and 5). At 7B the mild target of 0.10 beats Linear in all three seeds (0.449 vs. 0.299). We honestly report COVER's results in every setting we tested, including those where it only matches Linear. That is expected, because COVER is not intended to beat every baseline but to show that the covariance law works in practice.
W2. Table 8 shows that even with the same targets, whether COVER favors the primary or a secondary objective relative to Linear depends on the exact circumstances
COVER behaves consistently across Table 8. It favors the task objective in every domain, and it matches or beats Linear on that objective in all four. It raises the task objective's weight on 77–100% of steps, more than any auxiliary weight, and the weight's share of the total grows from 33% (50% on MBPP) to 44–71%. On MBPP, where the pass rate declines gradually, COVER cuts the drop from 0.141 to 0.090. On GSM8K and PKU-SafeRLHF, the task objective already plateaus at a high level under Linear (0.90 and 0.999), so COVER matches Linear. On XSum, coverage loses nearly half its value in the first ten steps under both methods, faster than the COVER can respond, and COVER again matches Linear.
W3. The paper gives no clear guidance on choosing the targets for models with different capabilities, more complex combinations of objectives, or a user's preferences
We want to clarify that for a fair comparison, we deliberately kept the targets fixed rather than tuning them per model, objective or task. We use 0.15 for the task objective and 0.05 for each auxiliary objective in every experiment of Tables 4 and 6–8, which cover both Qwen2.5-1.5B models and all six datasets. To adapt COVER to a new model or set of objectives, we recommend setting each target just above the covariance level that COVER logs while that objective declines. Tables 5 and 9 already show in detail how the target affects accuracy at 1.5B and 7B, so they can guide this choice for models of other scales.
Q1. Do the targets 0.15 and 0.05 mean that the task objective is three times as important as each auxiliary objective?
No. The targets are floors, not importance weights. An objective's weight depends on how long and how far its covariance stays below its floor. With the same targets, the final task weight is 1.1–1.4 times the auxiliary weights on IFEval but 3.7–6.8 times on PKU-SafeRLHF.
Q2. With a running-average conciseness score, how can comparisons across algorithms be meaningful when length dynamics vary so much, as seen in Fig. 1?
We chose the running average on purpose. A fixed length threshold does not generalize, because the length that counts as concise is unknown before training and differs across tasks and models. Once all responses in a batch lie on one side of a fixed threshold, they all receive the same score, so the conciseness objective carries no learning signal. The running average keeps the threshold dynamic and at the scale of the policy's own responses, so the reward always separates shorter responses from longer ones. This is a standard practice also used by Dynamic (TACL 2026).
We hope these clarifications address your concerns, and would be grateful if you would reconsider your rating.