- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies gradient conflicts in multitask learning, arguing that task gradients alone do not capture the different local distributional changes induced by a shared update. It proposes KAGrad, which retains CAGrad’s average-gradient feasible set but replaces its first-order criterion with a normalized quadratic improvement criterion using task-specific metrics, including Fisher estimates. The paper provides local-model certificates and conditional finite-step guarantees, while inheriting average-loss descent guarantees from the shared feasible set. Experiments show benefits with curvature-matched metrics on quadratic tasks, but learned-model results provide mixed improvements over CAGrad and no consistent advantage of Fisher metrics over identity metrics.
Critical strengths
-
The motivation of involving KL information for assessing task improvements is reasonable and intuitive.
-
The derivations provide a feasible implementation, although its computational overhead is substantial.
-
The authors have appeared to have honestly reported their experimental results.
Critical weaknesses
-
Incorporating Fisher geometry is a straightforward extension of established natural-gradient ideas, while the feasible set and average-loss convergence argument are inherited from CAGrad. Although the normalized quadratic criterion introduces a substantive modification, the overall method remains an incremental extension whose contribution beyond existing gradient-aggregation frameworks is limited.
-
Fisher KAGrad does not consistently outperform Identity KAGrad; cityscapes segmentation gains come with worse relative depth error, and the classification experiments detect no significant improvement over CAGrad. These mixed results are particularly unconvincing given the roughly twelvefold runtime increase over CAGrad on Multi-Fashion+MNIST.
-
The learned-model evaluation primarily compares against CAGrad and internal identity-metric variants, without establishing competitiveness against methods such as Nash-MTL, FAMO, or Aligned-MTL. Furthermore, the MT50 comparison uses different task-selection rules, making it difficult to attribute the improvement specifically to the proposed criterion or Fisher geometry.
-
I suggest the authors rewrite the abstract and the introduction to highlight the contribution of this paper in a logical flow.
Questions for authors
Please refer to the listed Weaknesses.