- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoGrad, which uses genetic programming to search for gradient aggregation expressions in multi-task learning. The method first screens candidate expressions based on the expected loss reduction for each task, then selects the final rule by temporarily updating the model and comparing task metrics on the current training batch. Experiments cover four multi-task datasets and synthetic problems, achieving good results on several metrics.
Critical strengths
The experiment in the paper covers different numbers of tasks and prediction types, and the appendix provides results for each task, facilitating observation of performance trade-offs behind the overall metrics. The authors also report the case where the model performs well on training but poorly on testing on NYU-v2, which helps understand the scope of applicability of the method.
Critical weaknesses
-
The contribution of automatically discovered non-linear rules has not been sufficiently validated. Table 3 lacks a control group that only searches linear combinations, while Table 2 statistically reports the frequency of multiplication and division operations, yet it does not demonstrate that these operations bring performance gains. The paper discusses GradDrop and BayesAgg-MTL, but without experimental comparisons, it remains unclear what additional value automatic search offers over existing aggregation rules.
-
The paper does not demonstrate whether the performance gains justify the additional training cost. The method requires evolutionary search and repeated updates, yet Table 3 only compares training times among its own variants. The advantages shown in Table 1 vary depending on the dataset and metric, and the performance trajectories are close to those of FairGrad and SVFair. If a comparison under identical training times were provided, it would verify whether the added complexity is justified.
-
The scoring in Equation 5 is influenced by the magnitude of candidate directions. However, the actual updates in the code still undergo gradient clipping and Adam processing. A higher score does not necessarily imply a greater actual reduction. The paper needs to validate the relationship between scoring and actual loss changes, and verify whether the results still hold after controlling for direction scale.
Questions for authors
-
When maintaining the same search process and candidate evaluation count, what results do we get from only searching linear weight combinations? How does the gain from automatic search compare to existing coordinate-wise gradient aggregation methods?
-
Under the same training time, does EvoGrad still maintain its advantage over SVFair, FairGrad, and FAMO?
-
How consistent is the proxy score with the actual loss change after update? Does normalizing the candidate direction's norm significantly affect the program's selection and final performance?