- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the relationship between generalization and curvature in a in a teacher–student tree-committee machine, with a changing data size (P) to feature dimension (N) ratio . The objective is the -regularized empirical loss. This paper characterizes the minimizer set using the replica method with three order parameters Q, q, and r, capturing the norm of a sampled weight, the overlap between two sampled weights, and the teacher-student alignment, respectively. The classification and regression generalization errors can be subsequently computed using these order parameters. The set of minimizers are characterized for both classification and regression tasks. For the former, the replica symmetry (RS) is preserved; while for the latter, the RS symmetry can be broken for sufficiently small as increases. In such cases, the replica symmetry breaking (RBS) and the recovery phases appear where the transition between them can be discontinuous when is small. The paper additionally characterize the Hessian spectrum for these phases, and shows different flatness-generalization relationships for regression and classification. In the former case (where replica symmetry holds), the sharpness (largest eigenvalue) correlates well with generalization error for different regularization strengths. In the latter case, such correlation only holds for small values of (i.e. overparameterized regime), and become uncorrelated and even reverse for high values of . The analysis is further extended to the mean curvature and the left spectral edge.
Critical strengths
The technical contribution of this work seems to be strong. It quantitatively demonstrates that the flatness-generalization relation can be phase-dependent, and hence different for regression and classification tasks in which the phase behaviours are different. This insight is useful for practitioners as it suggests that the positive correlation between flatness and generalization does not generally hold, and in fact depends on the degree of overparameterization.
The experiments seem to agree well with the theory predictions. It maybe better if the authors can provide some additional details on the experiments in the main text, e.g., the step size of GD, and comment how sensitive the results are to various choices of hyperparameters.
Critical weaknesses
For both classification and regression tasks, the theoretical results are under the squared loss. However, for classification, the squared loss is much less commonly used comparing to the logistic loss. This raises the question how practically relevant the results are for classification. Can the authors comment on whether the reported phase behaviours hold for classification with the logistic loss? This is my main concern.
The reported trends are based on changing loss landscapes, where they are characterized by different values of . In a lot scenarios, one maybe interested in the phase behaviour of a fixed loss landscape. The current results do not capture this. This limitation has been acknowledged by the authors.
Questions for authors
What is in eqn. (7)? It seems to be undefined.
We thank the reviewer for the excellent questions and comments.
Weaknesses
W1. We have now updated our analysis to include the logistic loss for classification. We now show that the shape of the curve of the right edge and spectral mean as a function of the generalisation error can completely change when different losses are considered. This leads to a further non-universality in the classification case.
W2. Indeed this setting tackles the flatness–generalisation question from a different angle. Our reason for investigating this was to confirm whether or not this relation held on general grounds, and was robust to this different angle. We find a negative result, at least when it comes to classification, and can thus insert our work in a line of research which has confirmed the absence of a universal link between curvature and generalisation. Since a relation that is supposed to hold in general should also survive changes in the sample-to-parameter ratio, its failure here is informative even though sample size and curvature vary together. We agree that comparing different solutions to the same problem at fixed α is the complementary question.
Questions
Q1. Thanks, it is the Heaviside theta function, we had forgotten to define it. We have now added the definition.