- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose to use LISC as a sample reweighting method to emphasize less pronounced yet potentially relevant gradient directions, which is especially relevant for the setting of heterogeneous data. They show that this reweighting is equivalent to preconditioning with a local gradient second moment, whilst avoiding the construction of the parameter-space preconditioning matrix. Empirically, LISC improves early-training test performance and is on par with the provided baselines in final performance.
Critical strengths
- Methodological approach: I really like the general idea and the corresponding methodological depth, analyzing the relationship between eigenvalues and the contribution to the gradient, as well as the analysis of the effect of clipping and normalization.
- Versatility of LISC
- The theory section is very nice to read and tidy and states the scope of the approach pretty clearly.
Critical weaknesses
To me, it feels like the empirical findings do not fully match the story:
-
A major claim is LISC's advantage in the heterogeneous data setting. Yet I am not sure how heterogeneous the selected datasets are, so I feel the evaluation setup doesn't match the story and might even overshadow a strength of LISC. See question 1).
-
I feel like the claim about optimization budgets and "how quickly (the model) acquires useful predictive structures" might collapse when taking into account the results from B.9 about the computational overhead of LISC. See question 2).
-
I see how the generalization claim holds up in early epochs, but its advantage doesn't consistently hold for the final model. Thus, with the results presented as is, together with the concerns stated in weakness 2), I am doubtful about the actual advantage of LISC. See questions 1) and 4) for some potentially underexplored advantages of LISC.
-
Further comparison baselines are missing. See question 3).
Questions for authors
-
A major claim is the heterogeneous data setting. In what way are the selected evaluation datasets heterogeneous? Consider adding a more obviously heterogeneous dataset to the evaluation.
-
Please add test accuracy vs. wall-clock time curves.
-
Any thoughts on why Aligned-MTL performs comparatively worse than both the base and LISC? What are the chosen hyperparameters of Aligned-MTL? Also, please add further baseline methods.
-
What is the effect of LISC on the predicted output distribution? Does LISC lead to better or worse calibration than other methods? Does LISC end up in a different local optimum compared to other methods, leading to similar performance yet different strengths?
3 comments
Thank you for the thorough response, the additional experiments, and the transparent communication of the results. Overall, these experiments answer my questions. However, my general concern about the practical benefit in the current evaluation setting remains, and it is partly even reinforced by the warm-up ablation in Table 10. I am also still doubtful about the claim/usefulness on heterogeneous data. While it is now addressed with PACS, the benefit there does not seem substantially different from that on the original datasets. I will therefore keep my score.
Furthermore, from how I’ve understood Table 11 and B.10, LISC yields a model with worse calibration (at least for MMLU). I think this is a very interesting insight. Whilst it might not favour LISC at first glance, understanding the underlying mechanism, together with the different convergence optimum of LISC, could give LISC a potential that the current accuracy evaluation does not show (e.g., for noisy label data sets?). Given how principled the theoretical approach is, I think a more holistic analysis of the solutions LISC converges to could reveal use cases where the method really shines. I am looking forward to seeing what this direction holds, and I believe that convincing results there could turn LISC into a very meaningful and relevant contribution.
Thanks a lot for the follow-up and for appreciating our theoretical approach. We would like to clarify the following points.
Heterogeneous data. On PACS, LISC achieves higher mean accuracy than the baseline at every evaluated checkpoint, including 35.33 ± 2.20 vs 34.71 ± 1.24 at the final checkpoint. The final gain is modest, and the consistent improvements support LISC’s usefulness when training across styles and evaluating on an unseen style.
| Epoch | SGD | SGD + LISC |
|---|---|---|
| 5 | 22.83 ± 1.02 | 26.99 ± 1.75 |
| 20 | 27.26 ± 4.25 | 31.15 ± 2.49 |
| 50 | 34.71 ± 1.24 | 35.33 ± 2.20 |
Practical benefit and warm-up. Table 10 shows that warm-up improves performance in the image settings and also benefits LISC, suggesting that the two are complementary. Warm-up hurts performance in the evaluated language settings, whereas LISC’s early improvements are consistent across vision and language tasks. The wall-clock results also demonstrate practical gains in the reported settings despite LISC's additional computational cost.
Calibration and learned solutions. We agree that calibration results are not positive for LISC. Our claim about the regularizing effect of spectral balancing does not imply improved calibration. The prediction disagreement despite similar accuracy also motivates us to examine the learned solutions beyond accuracy.
We appreciate this constructive suggestion. We believe LISC offers a useful contribution through both its principled approach to gradient preconditioning using data geometry and its demonstrated gains in early performance across various settings.
We thank Reviewer CaeD for highlighting the analysis of gradient directions and the effects of clipping and normalization.
Q1. Data Heterogeneity
Our experiments measure how strongly examples reinforce the same gradient directions. Our claim about heterogeneity is based on gradient energy distributed to different bases, which can be one proxy to measure heterogeneity in terms of concepts, structures, or patterns.
Following your suggestion, we evaluated LISC on PACS [1]. We pooled 3 styles of a class for training, allowing a minibatch to contain several styles, and tested on the fourth. We repeated this for each heldout style. Averaged across heldout styles, LISC gains 4.16 pp at epoch 5 and 3.89 pp at epoch 20. The final difference is 0.6 pp. This gives us evidence of the gain due to LISC on unseen styles. We have added this result in Table 2.
Q2. Equal wall-clock time comparison
We agree that LISC adds a computational overhead because of the spectral transformation. To assess the acceleration in terms of the wall-clock, we have added a test accuracy vs wall-clock plot. The plot (Fig 4) shows that LISC can also be pareto optimal compared to the baseline even when we compare against the wall-clock. For instance, for MMLU, LISC achieves 58.78% in 7.11 min compared to AdamW which achieves 58.41% in 7.69 min. Similarly, for CIFAR-10, LISC reaches 83.73% in 1.61 min, whereas SGD reaches 81.52% at the same wall-clock time.
W3. Early and final gains consistency
Across 11 settings in Table 1, we observe that LISC has higher mean heldout performance vs Base at the first reported checkpoint in all 11, at the middle checkpoint in 8, and at the final checkpoint in 8. Its mean AUC is also higher in all 11 settings. The three lower final means occur on C10, by 0.03, 0.09, and 0.20 pp. On C100 with SGD, the gain narrows from 5.02 pp at epoch 5 to 0.71 pp at epoch 200. We agree that the most consistent benefit is earlier improvement at a fixed update budget. We acknowledge that the final gains are smaller and depend on the setting.
Q3. Aligned-MTL performance and configuration
Aligned-MTL was proposed to balance task gradients, while we treat each example as an objective. To make the evaluation feasible, we applied Aligned-MTL’s spectral rule also in the same measurement coordinates used by LISC (Appendix B.2). Aligned-MTL’s transformation brings every singular value down to the smallest retained value. Consequently, a single small retained value can reduce the aggregate gradient in the classifier-head parameters. During epochs 1–5, the median ratio of the Aligned-MTL gradient norm to the ordinary gradient norm was 0.42% on CIFAR-10. In addition, Aligned-MTL was primarily proposed for scene understanding, and the authors didn’t evaluate their performance in the image classification settings.
We would also like to clarify that Aligned-MTL does have competitive results in some cases. For instance, on CIFAR-10/SGD, it exceeds Base at epochs 5 and 20. On GSM8K, its final exact match is 49.89% vs 49.33% for AdamW and 50.59% for LISC (Table 1).
In addition, our Aligned-MTL runs use the Base optimizer settings, without an independent Aligned-MTL search. We have clarified this in Appendix B.2 in the revised manuscript.
Additional baselines.
We evaluated lagged-loss weighting, in-batch class balancing, and adaptations of GradNorm, GradVac, CAGrad, and IMTL-G [2–5]. LISC has the highest mean epoch-5 accuracy on both datasets and the highest mean final accuracy on C-100. We have provided the complete result in Table 9 in the revised manuscript.
Output Distribution and Calibration
On MMLU, we observed that the paired prediction analysis shows a trade-off. At the final checkpoint, AdamW and LISC differ by only 0.32 pp in accuracy, yet their answers differ on 23.38% of questions. When wrong, LISC assigns its chosen answer 61.86% confidence on average, compared to 52.72% for AdamW. Expected calibration error (ECE) [6], NLL, and Brier score are all higher with LISC at checkpoints examined. We have summarized the results in Table 11.
Yes, LISC converges to a different optimum compared to base. We have added a plot (Fig 9) that depicts this.
[1] Li et al. Deeper, Broader and Artier Domain Generalization. ICCV 2017.
[2] Chen et al. GradNorm. ICML 2018.
[3] Wang et al. Gradient Vaccine. ICLR 2021.
[4] B. Liu et al. Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS 2021.
[5] L. Liu et al. Towards Impartial Multi-task Learning. ICLR 2021.
[6] Guo et al. On Calibration of Modern Neural Networks. ICML 2017.