acceptodds
Withdrawn from ICLR 2027

KAGrad: KL-Aware Gradient Alignment in Multitask Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In gradient-based multitask learning with shared parameters, an update that decreases one task loss can increase another. Existing methods mitigate such conflicts using the Euclidean geometry of task gradients. For probabilistic models, however, gradient geometry alone does not determine local changes in output distributions. Tasks with identical gradients can have different Fisher information matrices and therefore assign different local KL costs to the same update. We propose KL-Aware Gradient Alignment (KAGrad), which incorporates this task-specific geometry into the choice of a shared update. Each task gradient and a supplied positive-definite metric define a quadratic model of predicted improvement; with the exact Fisher, its quadratic penalty is proportional to the local KL term. KAGrad maximizes the smallest fraction of each task's independently attainable model improvement, within a Euclidean ball centered at the average-loss gradient. Equivalently, it minimizes the largest normalized squared distance, in each task's supplied metric, from the update preferred by that task's local model. The optimized worst-task fraction is nonnegative if and only if the feasible set contains an update under which every local model used in the optimization predicts non-increase. When the selected direction is applied directly as a gradient step, the constraint gives average-loss descent and an stationarity bound under smoothness, a lower-bounded average loss, and a step-size bound. Because a Fisher metric need not match task-loss curvature, we also derive a sufficient condition under which the local predictions hold for actual losses after a finite step. Against CAGrad, which uses the same average-gradient constraint with a first-order criterion, KAGrad with estimated Fisher or identity metrics attains higher Cityscapes mean intersection over union ( and versus ) at a shared constraint radius, but higher relative depth error ( and versus ). On MT50, eight-term grouped KAGrad attains higher peak mean task success than eight-term CAGrad-Fast ( versus ) over checkpoints shared across runs. We separately assess the supplied metric. In quadratic tasks, matching it to loss curvature improves worst-task actual loss decrease over an identity metric; rotating the Hessians away diminishes and reverses this advantage. In learned models, diagonal Fisher estimates show no consistent benefit over identity metrics: MT10 peak mean success is versus , and ten paired Multi-Fashion+MNIST runs detect no difference. Cityscapes shows a segmentation–depth trade-off. The benefit of a curvature-matched metric in quadratic tasks does not carry over to diagonal Fisher estimates in these learned models.

Primary area:
optimization
Keywords:
multitask learning, multi-objective optimization, gradient conflict, Fisher information, KL divergence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rPbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies gradient conflicts in multitask learning, arguing that task gradients alone do not capture the different local distributional changes induced by a shared update. It proposes KAGrad, which retains CAGrad’s average-gradient feasible set but replaces its first-order criterion with a normalized quadratic improvement criterion using task-specific metrics, including Fisher estimates. The paper provides local-model certificates and conditional finite-step guarantees, while inheriting average-loss descent guarantees from the shared feasible set. Experiments show benefits with curvature-matched metrics on quadratic tasks, but learned-model results provide mixed improvements over CAGrad and no consistent advantage of Fisher metrics over identity metrics.

Critical strengths
  1. The motivation of involving KL information for assessing task improvements is reasonable and intuitive.

  2. The derivations provide a feasible implementation, although its computational overhead is substantial.

  3. The authors have appeared to have honestly reported their experimental results.

Critical weaknesses
  1. Incorporating Fisher geometry is a straightforward extension of established natural-gradient ideas, while the feasible set and average-loss convergence argument are inherited from CAGrad. Although the normalized quadratic criterion introduces a substantive modification, the overall method remains an incremental extension whose contribution beyond existing gradient-aggregation frameworks is limited.

  2. Fisher KAGrad does not consistently outperform Identity KAGrad; cityscapes segmentation gains come with worse relative depth error, and the classification experiments detect no significant improvement over CAGrad. These mixed results are particularly unconvincing given the roughly twelvefold runtime increase over CAGrad on Multi-Fashion+MNIST.

  3. The learned-model evaluation primarily compares against CAGrad and internal identity-metric variants, without establishing competitiveness against methods such as Nash-MTL, FAMO, or Aligned-MTL. Furthermore, the MT50 comparison uses different task-selection rules, making it difficult to attribute the improvement specifically to the proposed criterion or Fisher geometry.

  4. I suggest the authors rewrite the abstract and the introduction to highlight the contribution of this paper in a logical flow.

Questions for authors

Please refer to the listed Weaknesses.

Reviewer_8aUhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for balancing gradients in multi-task learning using task-specific metrics based on the Fisher information matrix. Inspired by CAGrad, the update direction is chosen using a local improvement model with a linear term involving the task gradient and a quadratic penalty involving the task’s metric. Within a feasible set centered at the average gradient, the authors maximize the worst-task improvement, normalized by the maximum improvement each task’s model can achieve independently. The authors also present a method for grouping tasks when the number of tasks is large. They provide guarantees for their method, most notably average-loss descent under smoothness and step-size conditions, and non-increase of each represented task’s loss when the optimized model improvements are nonnegative and an additional metric–curvature condition holds. The authors demonstrate their method on several benchmarks.

Critical strengths
  • Using the fisher information matrix as a metric in MTL setup is novel as far as I am aware.
Critical weaknesses
  • Clarity: The paper is hard to follow, lacks intuitive explanations, and has several formatting issues. In particular, the abstract is too long and too technical. Similarly, the introduction is overly technical and would benefit from clearer motivation before introducing the details of the proposed approach. Some technical terms appear before they are adequately explained. Additional derivation steps would improve readability, particularly for Eq. (5), while the derivation of Eq. (11), provided in Appendix A.1, could be referenced explicitly. Section 3.1 would benefit from a clearer explanation of the motivation for grouping and its implications for task-level guarantees. The experimental presentation is also difficult to follow. For instance, lines 348–352 report several numerical results that would be easier to interpret in a table, with clearer explanations of the quantities being compared. In Fig. 4, some text extends beyond its bounding boxes.

  • Method: The practical motivation for using the Fisher information matrix is not sufficiently justified. Although the Fisher provides a local KL interpretation, Fisher geometry need not match loss curvature. Therefore, a favorable Fisher-based score does not automatically imply actual task-loss improvement. The authors acknowledge this limitation and provide an additional metric–curvature condition, but the learned-model experiments do not establish a consistent advantage of the estimated Fisher metrics over identity metrics. This leaves the practical benefit of the Fisher-based formulation unclear.

  • Experiments and related work: Although the paper discusses A few related methods, the empirical comparisons are largely restricted to CAGrad and Identity KAGrad. Broader comparisons with established gradient aggregation methods are needed, particularly given the use of standard benchmarks. Relevant work such as BayesAgg-MTL [1] is also missing from the discussion and comparisons. Lastly, even relative to CAGrad, the results do not convincingly establish a consistent advantage. See the related work and comparisons in [1] and the references therein for a comprehensive list.

[1] Achituve, I., Diamant, I., Netzer, A., Chechik, G., & Fetaya, E. (2024). Bayesian uncertainty for gradient aggregation in multi-task learning. arXiv preprint arXiv:2402.04005.

Questions for authors

No questions.

Reviewer_pheEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a KL-aware gradient alignment method to improve multi-task learning.

Critical strengths

This paper uses the Fisher of each task to measure the improvement it predicts for each task as a retained-improvement fraction of that task’s maximum.

Critical weaknesses
  1. The proposed method is similar to Safe and Balanced (TPAMI 2025) but not cited. The main difference is that this paper's method calculates the Fisher's algorithm for each task and updates the range using CAGrad to ensure convergence.

  2. The comparison objects are too few, incomplete, and unfair. The comparison settings for each table are different, especially the use of different variants of CAGrad in each experiment, which is strange. Commonly compared tasks and metrics for these datasets are missing. Commonly used data such as delta m% and average ranking are not presented. Compared to the main baseline CAGrad, the proposed method requires more than ten times the training time.

  3. How to ensure that the gradient of small batches of training data can represent the improvement ratio for each task?

  4. The formatting is poor. For example, the text in Figure 4 exceeds the border.

  5. Only one reference in the last three years.

Questions for authors

Please address weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.