- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether two factors, sharpness and model complexity, are enough to explain the generalization performance of trained models. It studies 16 combinations of different sharpness and complexity metrics and evaluate them on models trained on CIFAR-{10,100} datasets.
Critical strengths
- This paper has a systematic experimental setup comparing different combinations of sharpness and complexity metrics.
- This paper has insightful takeaways: Finite-scale sharpness outperforms infinitesimal sharpness; perturbation scale matters more than the choice of Hessian statistic; function-level complexity explains generalization more robustly than parameter-level metrics.
Critical weaknesses
- The theory in this paper presents motivation for considering sharpness and complexity jointly, but it does not show that (sharpness, complexity) outperforms other two-factor combinations considered in other work. Also, the experiment mainly studies which particular definitions of sharpness and complexity work best, but it does not show that sharpness and complexity are better factor choices.
- The paper evaluates on CIFAR-{10,100} and classical image models (ResNet, VGG, WideResNet, and ViTs). However, readers would be interested in the results on contemporary settings such as harder image tasks, language models, and multimodal models.
Questions for authors
The authors could discuss the novelty of this paper more clearly, because prior work has studied generalization via multiple measures such as representation consistency and perturbation robustness, spectral scale and shape, etc. Could the authors clarify how their proposed two-factor framework differs from these approaches (beyond the specific choice of sharpness and complexity)?
3 comments
Reply to Rebuttal
Thank you for the rebuttal and the additional BERT experiments. I increased the score to 2 because I still remain concerned about the relevance and scale of the experimental settings. BERT was introduced in 2018, and most empirical study papers after 2021 use more contemporary models and are not limited to text classification benchmarks. Given that the paper aims to explain generalization, I would expect some evaluation on contemporary models, such as generative LLMs or vision-language models.
Thank you for the clarification and for reconsidering the score. We appreciate that our response helped clarify the motivation for the two-factor framework and that the added BERT experiments broadened the empirical scope. We understand that your remaining concern is primarily the scale and contemporaneity of the experimental settings. We agree that extending the study to generative LLMs and vision-language models would be an important direction for future work.
We apologize for the delay in our response, as the additional BERT experiments took longer than anticipated under limited GPU availability.
We sincerely thank the reviewers for their understanding and thoughtful comments.
We have uploaded a REVISED PDF, with changes and additions highlighted in yellow. To minimize the reviewer's reading burden, we will try our best to keep our responses here concise. Please refer to the revised PDF for detailed explanations.
Response to Weakness 2
In Appendix C.5, we report the results of BERT fine-tuned on each of three text datasets: SST2, AGNews, and MNLI.
Overall, BERT results are consistent with our observations in Section 5.
The main results in Table 21 show that, with suitable metric choices, the two-factor view can explain a substantial fraction of the variation in generalization among fine-tuned BERT models. These findings further extends the explanatory scope of the two-factor joint view: BERT fine-tuned on SST2/AGNews/MNLI also falls within the “range of settings” described in Observation 1.
Please see Appendix C.5 on page32 for a detailed discussion of the results of BERT.
Response to Weakness 1 and the Question
We thank the reviewer for this important question.
We agree that prior work has proposed effective multi-factor explanations of generalization, including representation consistency, perturbation robustness, and spectral scale and shape.
A key distinction of the sharpness--complexity view considered here is that the two factors are intended to be application-independent. Many other generalization measures can incorporate task-, modality-, or representation-specific prior knowledge.
In contrast, our goal is to ask how much of generalization can already be explained by broadly applicable, prior-independent quantities before introducing such task-specific structure. This is why we do not attempt to include data-augmentation-based measures or those specialized metrics mentioned by the reviewer in this study.
As discussed in Appendix A, our sharpness-complexity-two-factor choice also has theoretical motivation: PAC-Bayes generalization bounds, which apply to a general supervised-learning setting and naturally involve terms corresponding to model complexity and local sensitivity/sharpness.
Therefore, the two factors, sharpness and complexity, have a theory-based motivation while remaining largely independent of a particular data modality, architecture, or loss function. We do not claim that this PAC-Bayes connection itself is new; rather, it motivates our central question: how far these generic factors can go in explaining generalization.
We agree that this distinction is important and have added a clearer discussion of it to the revised manuscript.
We hope these address the reviewer’s question. Please let us know if further clarification would be helpful.