acceptodds
Under review as a conference paper at ICLR 2027

How Far Can Sharpness and Complexity Jointly Explain Generalization?

Abstract

Large-scale empirical studies have revealed that many widely used generalization measures correlate only weakly with generalization, suggesting that relying on a single factor is insufficient to explain it. In this work, we examine sharpness and complexity jointly, placing the reach of the two-factor view at the center: how far can they jointly explain generalization, what determines when they succeed, and where do they fall short? We investigate these questions quantitatively, using sixteen alternative metric pairs as probes and Pareto analysis and linear regression as diagnostic tools. Our results show that suitably defined pairs provide strong explanatory power across a range of settings. The sharpness perturbation scale substantially affects this success, and the complexity representation influences its robustness. Remaining failures leave open whether more refined measurements are adequate or additional factors are needed.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
deep learning, generalization, sharpness, complexity, pareto analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UiytOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether two factors, sharpness and model complexity, are enough to explain the generalization performance of trained models. It studies 16 combinations of different sharpness and complexity metrics and evaluate them on models trained on CIFAR-{10,100} datasets.

Critical strengths
  1. This paper has a systematic experimental setup comparing different combinations of sharpness and complexity metrics.
  2. This paper has insightful takeaways: Finite-scale sharpness outperforms infinitesimal sharpness; perturbation scale matters more than the choice of Hessian statistic; function-level complexity explains generalization more robustly than parameter-level metrics.
Critical weaknesses
  1. The theory in this paper presents motivation for considering sharpness and complexity jointly, but it does not show that (sharpness, complexity) outperforms other two-factor combinations considered in other work. Also, the experiment mainly studies which particular definitions of sharpness and complexity work best, but it does not show that sharpness and complexity are better factor choices.
  2. The paper evaluates on CIFAR-{10,100} and classical image models (ResNet, VGG, WideResNet, and ViTs). However, readers would be interested in the results on contemporary settings such as harder image tasks, language models, and multimodal models.
Questions for authors

The authors could discuss the novelty of this paper more clearly, because prior work has studied generalization via multiple measures such as representation consistency and perturbation robustness, spectral scale and shape, etc. Could the authors clarify how their proposed two-factor framework differs from these approaches (beyond the specific choice of sharpness and complexity)?

3 comments

We apologize for the delay in our response, as the additional BERT experiments took longer than anticipated under limited GPU availability.

We sincerely thank the reviewers for their understanding and thoughtful comments.

We have uploaded a REVISED PDF, with changes and additions highlighted in yellow. To minimize the reviewer's reading burden, we will try our best to keep our responses here concise. Please refer to the revised PDF for detailed explanations.


Response to Weakness 2

In Appendix C.5, we report the results of BERT fine-tuned on each of three text datasets: SST2, AGNews, and MNLI.

Overall, BERT results are consistent with our observations in Section 5.

The main results in Table 21 show that, with suitable metric choices, the two-factor view can explain a substantial fraction of the variation in generalization among fine-tuned BERT models. These findings further extends the explanatory scope of the two-factor joint view: BERT fine-tuned on SST2/AGNews/MNLI also falls within the “range of settings” described in Observation 1.

Please see Appendix C.5 on page32 for a detailed discussion of the results of BERT.


Response to Weakness 1 and the Question

We thank the reviewer for this important question.

We agree that prior work has proposed effective multi-factor explanations of generalization, including representation consistency, perturbation robustness, and spectral scale and shape.

A key distinction of the sharpness--complexity view considered here is that the two factors are intended to be application-independent. Many other generalization measures can incorporate task-, modality-, or representation-specific prior knowledge.

  • Data-augmentation-based measures provide a simple example: what constitutes a meaningful perturbation depends strongly on the application. For instance, image augmentation may exploit translation or cropping invariance, while other tasks may require very different augmentation rules. Other measures may similarly exploit properties specific to classification tasks, a particular loss function, or a particular representation.

In contrast, our goal is to ask how much of generalization can already be explained by broadly applicable, prior-independent quantities before introducing such task-specific structure. This is why we do not attempt to include data-augmentation-based measures or those specialized metrics mentioned by the reviewer in this study.

As discussed in Appendix A, our sharpness-complexity-two-factor choice also has theoretical motivation: PAC-Bayes generalization bounds, which apply to a general supervised-learning setting and naturally involve terms corresponding to model complexity and local sensitivity/sharpness.

Therefore, the two factors, sharpness and complexity, have a theory-based motivation while remaining largely independent of a particular data modality, architecture, or loss function. We do not claim that this PAC-Bayes connection itself is new; rather, it motivates our central question: how far these generic factors can go in explaining generalization.

We agree that this distinction is important and have added a clearer discussion of it to the revised manuscript.


We hope these address the reviewer’s question. Please let us know if further clarification would be helpful.

Reviewer_UiytOpenReview ↗

Reply to Rebuttal

Thank you for the rebuttal and the additional BERT experiments. I increased the score to 2 because I still remain concerned about the relevance and scale of the experimental settings. BERT was introduced in 2018, and most empirical study papers after 2021 use more contemporary models and are not limited to text classification benchmarks. Given that the paper aims to explain generalization, I would expect some evaluation on contemporary models, such as generative LLMs or vision-language models.

Thank you for the clarification and for reconsidering the score. We appreciate that our response helped clarify the motivation for the two-factor framework and that the added BERT experiments broadened the empirical scope. We understand that your remaining concern is primarily the scale and contemporaneity of the experimental settings. We agree that extending the study to generative LLMs and vision-language models would be an important direction for future work.

Reviewer_bhFwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically investigates how far sharpness and complexity can jointly explain variation in neural-network generalization. Rather than prescribing a scalar formula that combines the two factors, the authors treat each model as a point in the two-dimensional sharpness--complexity plane. They introduce the Pareto contradiction rate (PCR): among pairs for which one model has both lower sharpness and lower complexity, PCR measures how often that model nevertheless has a larger generalization gap. The paper complements PCR with linear regression of the generalization gap on the two metrics.

The study evaluates sixteen pairs formed from four adaptive sharpness measures (Hessian trace, top Hessian eigenvalue, and finite-scale versions of each) and four complexity measures (parameter norm, a proposed function-output norm, path norm, and spectral-norm product). Models are trained on CIFAR-10 and CIFAR-100 using several CNN architectures and a ViT, with SGD or AdamW and with or without normalization layers. The principal empirical conclusions are that suitable metric pairs often achieve low PCR; finite-scale sharpness at a relatively large perturbation radius is frequently more explanatory than infinitesimal Hessian statistics; the perturbation scale matters more than choosing trace versus top eigenvalue; and the function-output norm is more robust than the considered parameter-level complexity measures, especially when models from several sources are pooled. The paper also reports failure cases, most notably WideResNet without batch normalization.

Critical strengths
  1. The paper provides a simple, non-scalarized diagnostic for the joint two-factor view.

PCR directly tests the minimal monotonic implication of the hypothesis: if a model is both less sharp and less complex, it should have a smaller generalization gap. This is preferable to declaring the two-factor view unsuccessful merely because one particular weighted sum has low correlation, and it exposes cases in which a global linear trend coexists with many local ordering contradictions.

  1. The empirical comparison is systematic within the studied image-classification regime and includes informative failures.

Holding the complexity measure fixed while varying the sharpness statistic and radius makes the scale effect easy to see. The comparison across normalized and non-normalized networks, SGD and AdamW, and pooled collections also usefully demonstrates that parameter norm, path norm, and spectral-norm product are not uniformly reliable. The explicit WideResNet-no-BN failure prevents the paper from presenting the two-factor view as universally successful.

Critical weaknesses
  1. The novelty and related-work positioning of the finite-scale conclusion are insufficient.

The main qualitative message that a finite neighborhood can be more informative than infinitesimal Hessian curvature predates this work. Entropy-SGD, SAM, ASAM, and prior empirical studies of adaptive average- and worst-case sharpness already operate over finite-radius neighborhoods; A Modern Look at the Relationship between Sharpness and Generalization explicitly studies a range of radii and derives the Hessian trace/eigenvalue limits as the radius approaches zero. More geometrically, Asymmetric Valleys: Beyond Sharp and Flat Local Minima shows that directional and one-sided valley structure is lost by a simple sharp/flat local description, while the river-valley literature studies extended low-loss structure along optimization trajectories. The submission cites some finite-radius sharpness work but does not clearly distinguish its scale observation from these precedents, and it does not cite the asymmetric-valley or river-valley work. Moreover, the proposed finite-scale quantities are symmetric in the perturbations and can average away exactly the one-sided structure highlighted by asymmetric-valley results. As written, the paper risks presenting a known finite-neighborhood perspective as a central new discovery. The contribution appears instead to be the joint sharpness--complexity evaluation and PCR; the claims and related-work discussion should be revised around that narrower novelty.

  1. The empirical evidence is too narrow, and the key diagnostics need robustness checks before supporting the paper's broad explanatory claims.

All experiments are CIFAR-10/100 classification; the ViT experiment is still small-scale vision classification, and the "cross-dataset" setting only pools these two closely related datasets. There is no ImageNet-scale experiment, language modeling, generative task, or OOD evaluation. This is particularly limiting because prior work has reported weak or even reversed sharpness--generalization relationships for ImageNet, CLIP, and BERT settings. Within the current experiments, PCR is reported without candidate-pair coverage, confidence intervals, or multi-seed uncertainty, so a low rate may be difficult to interpret if only a small or unstable subset of pairs is comparable. Finally, the proposed function norm is the magnitude of centered logits on a reference set. It may partly measure confidence, margins, or logit scale and thus be directly coupled to cross-entropy test loss rather than providing an independent measure of complexity. Since retained models have training loss at most 0.01, the reported generalization gap is also nearly test loss. Controls for logit temperature/margin, reference-set choice, and training hyperparameters are needed to establish the claimed complexity interpretation.

Questions for authors
  1. For every reported PCR, can the authors report the number and fraction of all model pairs that are Pareto-comparable, together with bootstrap confidence intervals or stability across independently sampled hyperparameter sweeps? Could the strongest low-PCR results be reproduced when candidate-pair coverage is held approximately fixed across metric pairs?

  2. What remains empirically novel about the perturbation-scale conclusion relative to finite-radius average/worst-case adaptive sharpness studies, especially A Modern Look at the Relationship between Sharpness and Generalization? How would the conclusions change if one-sided finite perturbations were measured to capture the asymmetric valleys documented in prior work?

  3. Can the authors test whether function norm remains predictive after controlling for logit temperature, margin/confidence, training hyperparameters, and the choice and size of the reference set? In particular, what happens if logits are temperature-normalized before computing the norm, or if the reference inputs come from train, test, OOD, or unlabeled data?

  4. Do the main findings extend beyond CIFAR classification to at least one substantially different regime, such as ImageNet-scale training, BERT/LLM fine-tuning or pretraining, or OOD generalization? This would materially affect whether I view the results as a general account of modern deep-network generalization or as a focused CIFAR study.

6 comments

We apologize for the delay in our response, as the additional BERT experiments took longer than anticipated under limited GPU availability. We sincerely thank the reviewers for their understanding and thoughtful comments.

We have uploaded a REVISED PDF, with changes and additions highlighted in yellow. To minimize the reviewer's reading burden, we will try our best to keep our responses here concise. Please refer to the revised PDF for detailed explanations.


Response to Question 1

We thank both Reviwers mD4U and bhFw for pointing this out.

Please see comments above (Q1. Coverage of candidate pairs. https://openreview.net/forum?id=bp6HAjAdWq&noteId=qLuzMif4BW ), and see results in Appendix C.


Response to Weakness 1 and Question 2

We thank the reviewer for highlighting the importance of this observation.

We agree that finite-radius sharpness predates our work. Our main contribution is to develop a systematic methodology for evaluating the explanatory power of the two-factor view and to report the empirical findings that emerge from this analysis.

This distinction matters because conclusions drawn from a single factor need not carry over to a two-factor setting. For example, many existing studies show that sharpness can correlate poorly with generalization. Such findings do not rule out its usefulness when paired with complexity, as our two-factor analysis illustrates. Conversely, a complexity metric such as path norm can be informative on its own in some settings, yet form a pair with weak explanatory power when combined with a particular sharpness metric. These possibilities motivate directly evaluating the joint explanatory power of the two factors rather than inferring it from their individual performance.

In A Modern Look at the Relationship between Sharpness and Generalization, they studied sharpness as a single factor across a range of perturbation radii and found that finite-scale sharpness still does not consistently explain generalization. Their findings motivate our two-factor analysis. Our additional single-factor results in Appendix C.4 also support this motivation. In our two-factor analysis, we observe a different phenomenon: at small finite radii, the sharpness measures behave similarly to their infinitesimal Hessian-based counterparts, while a substantial improvement in joint explanatory power emerges once the perturbation radius becomes sufficiently large. This effect is visible in Figure 4 and is not merely a consequence of replacing an infinitesimal measure with an arbitrary finite-radius one.

We observe the same qualitative behavior in the additional BERT experiments (Appendix C.5 on page32), suggesting that the importance of a sufficiently large perturbation scale in the two-factor analysis is not specific to CNNs or CIFAR data.

We will revise the manuscript to clarify that the novelty is not the use of finite-radius sharpness itself, but rather this scale-dependent finding within the (S,C) joint view.

As for Asymmetric Valleys: Beyond Sharp and Flat Local Minima, they also focused on sharpness as a single factor, studying how a solution within an asymmetric valley affects generalization. They showed that a solution biased toward the flat side of the valley generalizes better under mild assumptions.

  • We agree that our finite-scale sharpness metrics can conceal this asymmetry because they combine loss changes in opposite directions. However, as we emphasize in the Introduction, these metrics serve only as probes for investigating our central question—the explanatory reach of the joint view. We do not claim that the metrics we evaluated are better than those we did not evaluate. Other measures, such as one-sided sharpness, could also serve as probes.
  • Among the metric pairs we evaluated, at least one (S,C) pair shows strong explanatory power across a range of settings, providing an empirical answer to our central question. Due to limited time, we leave the evaluation of the pair (one-sided sharpness, complexity) to future work. A natural next step is to assess this pair’s PCR and examine how it changes with the perturbation radius.

We thank the reviewer for pointing us to these two studies. We will add them to the related work discussion in the Introduction.

Response to Weakness 2 and Question 3

We thank the reviewer for raising these points.

We would like to clarify the interpretation of the function norm and our experimental protocol.

  • Function norm is not directly coupled to cross-entropy loss.

The function norm, defined in Eq(4), measures the magnitude of class-centered logits, equivalently the aggregate pairwise predictive log-odds:

\sum_i\sum_{k<j} \left(\log\frac{p_k(x_i)}{p_j(x_i)}\right)^2.$$ The CE loss, in contrast, is label-dependent: $$L_{\rm CE}=-\sum_i\log p_{y_i}(x_i).$$ Hence there is no algebraic or monotone equivalence between them two. For example, in binary classification, centered logits $(a,-a)^T$ and $(-a,a)^T$ have exactly the same function norm, while for a fixed label (one-hot coded as $(1,0)^T$), their CE losses are $$\log(1+e^{-2a}) \quad\text{and}\quad \log(1+e^{2a}),$$ which can differ arbitrarily. - **Sensitivity to logit scale is intentional** and does not prevent the function norm from serving as a complexity metric. The function norm is a norm-based complexity proxy in function space, so the magnitude of the represented function is precisely part of what it measures. Temperature scaling $$z(x)\mapsto z(x)/T$$ changes the pairwise log-odds and predictive probabilities, and thus changes the represented probabilistic function. If this overall scale were normalized away, the variation that the norm is designed to measure would also be removed; under a normalization fixing the output magnitude, the corresponding norm would be constant by construction and loss information that we are interested. In contrast, $$z(x)\mapsto z(x)+c(x)\mathbf 1$$ leaves the predictive probabilities unchanged, and this genuine functional redundancy (i.e., the additive ambiguity) is what our class-centering removes (see complexity metrics section on page4). Importantly, **sensitivity to the logit scale does not introduce information about generalization performance**. The function norm is computed entirely from the trained model and a fixed subset of the training inputs, _**without**_ using reference labels or any test data. Indeed, all models retained in our analysis achieve near-zero training loss, however, they exhibit widely varying generalization gaps ($\approx$ test losses). Since training loss itself cannot account for these generalization differences, much less does a label-free complexity metric computed from training inputs become equivalent to the test losses. - **Reference data and hyperparameter sweep**. We use the same fixed subset of the **training inputs** as $D_{\rm ref}$ for every model in a comparison. This choice is deliberate: it allows the complexity metric to be computed **without** access to the test set and ensures that variation in the metric comes from the learned models rather than different reference samples. Since the function norm does not need labels, an independent unlabeled in-distribution set could also be used in principle. Our hyperparameter-sweep protocol follows the methodology in *Fantastic Generalization Measures and Where to Find Them*: we generate a diverse collection of well-trained solutions by varying multiple training hyperparameters rather than evaluating models produced by a single intervention. The hyperparameter variation is therefore part of the evaluation design. _**Our question is** whether properties of the resulting trained solutions, summarized by $(S,C)$ metric pair, explain their variation in generalization._ We do not make a causal claim about $S$ or $C$ after conditioning on every training hyperparameter. - **Robustness of function norm's predictability.** We examine the robustness of function-norm-related PCRs with respect to the _**reference set size**_, please see Appendix C.7 on page 36. The results in Table 22 suggest that the PCR of the pair ($\mathrm{tr(H)}_{\mathrm{fs}}$, function norm) _**remains stable**_ when the function norm is estimated using different reference set sizes. ---- ### Response to Weakness 2 and Question 4 We thank both Reviwers bhFw and Uiyt for suggesting to extend our two-factor analysis to settings beyond CNNs on image data. In Appendix C.5, we report the results of BERT fine-tuned on each of three text datasets: SST2, AGNews, and MNLI. **Overall, BERT results are consistent with our observations in Section 5.** The main results in Table 21 show that, with suitable metric choices, the two-factor view can explain a substantial fraction of the variation in generalization among fine-tuned BERT models. These findings further extends the explanatory scope of the two-factor joint view: BERT fine-tuned on SST2/AGNews/MNLI also falls within the “range of settings” described in Observation 1. Please see **Appendix C.5** on page32 for a detailed discussion of the results of BERT. --- We hope these address the reviewer’s question. Please let us know if further clarification would be helpful.

Thanks for the reply. After reading the provided results, I decide to maintain my score.

Reviewer_mD4UOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the extent to which sharpness and complexity jointly explain generalization gaps in neural networks. The authors evaluate sixteen metric pairs formed from four sharpness metrics and four complexity metrics. They use Pareto contradiction rate (PCR) to assess the ordering induced by the two factors, complemented by linear regression. The experiments cover CIFAR-10 and CIFAR-100, CNNs and ViT, SGD and AdamW, and architectures with and without normalization layers, as well as mixed settings that pool models from different sources. The results show that finite-scale sharpness yields more favorable diagnostic results than infinitesimal sharpness in several settings, and that function norm provides more stable explanatory power in some mixed settings. However, some regimes remain inadequately explained.

Critical strengths
  1. The finite-scale sharpness findings are empirically valuable. For SGD-trained ResNet18 no-BN, using parameter norm as the complexity metric, replacing the infinitesimal Hessian trace with the finite-scale Hessian trace reduces PCR from 49.5% to 3.4% and increases R² from 0.21 to 0.68. The magnitude of this change, together with the perturbation-scale sweeps, suggests that measurements restricted to the local Hessian may miss information relevant to generalization. This is one of the most valuable empirical contributions of the paper and provides concrete motivation to reconsider how sharpness is measured.

  2. Diverse settings and failure cases help delimit the conclusions. The experiments cover different architectures, optimizers, and normalization configurations, and explicitly report a challenging regime, WideResNet no-BN. Reporting both successful and unsuccessful settings helps establish the empirical scope and limitations of the evaluated metrics, increasing the information provided by the study.

Critical weaknesses
  1. Unreported candidate-pair coverage limits the interpretation of PCR. PCR is computed only over candidate pairs for which sharpness and complexity induce the same strict ordering. The authors clearly state this condition, but Tables 4, 5, 7, and 8 do not report the numbers of candidate pairs or their proportions among all model pairs. Different metric pairs and perturbation scales may produce different candidate sets. A low PCR alone therefore does not establish how broadly the observed ordering agreement applies, and changes in PCR across metric pairs cannot be fully interpreted without coverage information. Because PCR is the primary diagnostic, this reporting gap limits the evidence supporting the central empirical conclusions.

  2. The incremental benefits of the second factor and of avoiding scalarization remain unclear. The introduction asks whether complexity contributes explanatory information beyond sharpness. However, the reported experiments primarily compare two-factor metric pairs, without corresponding diagnostics using only sharpness (S) or only complexity (C). They also lack representative scalarization baselines evaluated on the same model collections under comparable evaluation conditions. Consequently, the results do not yet isolate how much information the second factor adds, or whether the more positive findings relative to prior studies arise from metric definitions, model collections, evaluation criteria, or avoiding a prescribed scalarization.

Questions for authors
  1. Coverage of candidate pairs. For the metric pairs and perturbation scales supporting the main conclusions, how many candidate pairs and contradictory pairs are there, and what fraction of all unordered model pairs are candidates? Do improvements in PCR coincide with substantial changes in candidate-pair coverage?

  2. Single-factor and scalarization baselines. On the same model collections, how do the diagnostics compare when using S alone, C alone, and both jointly? Are direct comparisons available against the existing scalarized measures discussed in the paper under comparable evaluation conditions?

  3. Sources of the results in mixed settings. The final mixed setting in Table 8 includes 47 WideResNet no-BN models. For this subgroup, pairing function norm with the finite-scale Hessian trace or the finite-scale Hessian top eigenvalue yields PCRs of 31.3% and 31.5%, respectively, whereas the corresponding PCRs for the pooled collection are 2.7% and 2.4%. Could the authors group the models in this pooled collection by dataset and architecture, and report candidate-pair counts and PCR separately for within-group and between-group comparisons?

  4. Interpretation of negative regression coefficients. Table 3 treats a negative coefficient for either sharpness or complexity as directionally inconsistent with the two-factor view. Does this interpretation apply only to the fitted additive linear approximation, or is it intended as evidence against the monotonic ordering implied by the two-factor view? A linear fit to a nonlinear relationship with correlated predictors can yield a negative coefficient even when the underlying relationship is increasing in both factors. Given this possibility, how should the negative coefficients reported in the paper be interpreted?

  5. Perturbation conventions for the two finite-scale metrics. Equation (1) uses a standard Gaussian vector to construct relative parameter perturbations, whereas Equation (2) uses a direction with unit Euclidean norm. The same value of σ therefore does not automatically imply the same relative perturbation magnitude. Does Observation 3 compare perturbation scale and the choice of Hessian statistic under each metric’s own perturbation convention, or is there also supporting evidence under comparable perturbation magnitudes?

4 comments

We apologize for the delay in our response, as the additional BERT experiments took longer than anticipated under limited GPU availability. We sincerely thank the reviewers for their understanding and thoughtful comments.

We have uploaded a REVISED PDF, with changes and additions highlighted in yellow. To minimize the reviewer's reading burden, we will try our best to keep our responses here concise. Please refer to the revised PDF for detailed explanations.

  • Q1. Coverage of candidate pairs.

We provide PCR in fractional form and its same-denominator counterpart PCR~\widetilde{PCR} in Appendix C.

Please see Tables 11, 12, and 13 for results of individual settings; Tables 16, 17, and 18 for results of mixed settings; and Table 21 for results of BERT.

The same-denominator PCR~\widetilde{PCR} results are consistent with the main conclusions drawn from the PCR-based analysis, indicating that our main findings on explanatory power are robust to differences in the number of candidate pairs.

  • Q2. Single-factor and scalarization baselines.

Please see Appendix C.4 for single-factor baselines. Table 9, Table 14, and Table 20 report individual settings, mixed settings, and BERT settings, respectively. Overall, the results show that neither sharpness nor complexity alone is sufficient to explain variations in generalization, which are consistent with the findings in many existing studies and support our motivation for examining the two factors jointly.

Due to time constraints, we have not evaluated the S-C scalarized measures on these same model collections. As discussed in Section 1.1, Jiang et al. (2019) evaluated such scalarized measures, such as the pacbayes-orig and sharpness-orig in their paper, and reported low correlation scores. Their results provide useful context regarding the association between these scalarized measures and generalization and serve as an informative reference of our study.

  • Q3. Sources of the results in mixed settings.

We additionally report the within-group and between-group breakdown of these two particular PCR values (2.7% and 2.4%) in the mixed six setting. Please see Table 19 on page 31.

Based on the results, we can tell that it is possible that a mixed collection achieves low PCR even though it contains an individual collection whose PCR is relatively high. In other words, the decomposition confirms that highly consistent between-group ordering contributes substantially to the strong result of the pooled collection, while the identified difficult subgroup remains challenging in the within-group analysis.

  • Q4. Interpretation of negative regression coefficients.

We agree that a negative regression coefficient does not, by itself, rule out a relationship that is increasing in both sharpness and complexity.

A negative regression coefficient is qualitatively unfavorable: under the fitted additive model, increasing the corresponding sharpness or complexity while holding the other factor fixed is associated with a smaller generalization gap, opposite to the direction expected from the two-factor view. We therefore regard such a fit as directionally inconsistent with the expected two-factor relationship. Since this coefficient describes a conditional additive trend, however, it is not by itself equivalent to a violation of the broader monotonic ordering; the latter is assessed directly by PCR. We have revised Table 3 accordingly.

  • Q5. Perturbation conventions for the two finite-scale metrics.

See below.

  • Q5. Perturbation conventions for the two finite-scale sharpness metrics.
:= \mathbb{E}_{\epsilon} \frac{2\left[ \ell(f_{w+\sigma\epsilon};Z)-\ell(f_w;Z) \right]}{\sigma^2}, \quad \epsilon=|w|\odot z, \quad z\sim\mathcal{N}(0,I).$$ $$\lambda_{\mathrm{max}}(H)_{\mathrm{fs}}(w;\sigma):=\max_{\lVert v\rVert_2=1}\frac{\ell(f_{w+\sigma |w|\odot v};Z)+\ell(f_{w-\sigma |w|\odot v};Z)-2\ell(f_w;Z)}{\sigma^2}.$$ It is true that the two definitions use perturbations with different Euclidean magnitudes. For $tr(H)_{\mathrm{fs}}$, the Gaussian perturbation $\delta=\sigma z$ has typical norm $O(\sigma\sqrt{d})$, whereas for $\lambda_{max}(H)_{\mathrm{fs}}$ the unit-direction perturbation has norm $\sigma$. ### We use these two conventions for two related reasons. **(1)** In the infinitesimal regime, they are the natural perturbations associated with the two target Hessian quantities: $$\lim_{\sigma\to 0}\mathrm{tr}(H)_{\mathrm{fs}}(w;\sigma)=\lim_{\sigma\to 0}\left(\mathbb{E}_z[z^{\top}Hz]+o(1)\right)=\mathrm{tr}(H),$$ $$\lim_{\sigma\to 0}\lambda_{\max}(H)_{\mathrm{fs}}(w;\sigma)=\lim_{\sigma\to 0}\left(\max_{\lVert v\rVert_2=1}v^{\top}Hv+o(1)\right)=\lambda_{\max}(H),$$ where $H=T\nabla^2 \ell(w) T, \quad T=\operatorname{diag}(|w|).$ Thus the Gaussian construction naturally extends the trace, while the unit-direction construction naturally extends the top eigenvalue. See proofs in Appendix A.3 on page15. **(2)** Although their _**raw Euclidean perturbation norms differ**_ by $O(\sqrt d)$, their _**perturbation scales are exactly matched when measured relative to**_ the Hessian quantity that each construction targets. Let $\delta_{\mathrm{tr}}=\sigma z$, $z\sim\mathcal N(0,I)$, then $$\frac{\mathbb{E}[\delta_{\mathrm{tr}}^\top H\delta_{\mathrm{tr}}]}{\mathrm{tr}(H)}=\frac{\sigma^2\mathbb{E}[z^\top H z]}{\mathrm{tr}(H)}=\sigma^2,$$ Let $u_1$ be a unit top-eigenvector of $H$, $$Hu_1=\lambda_{\max}(H)u_1,\qquad\|u_1\|_2=1,$$ and let $\delta_{\lambda}=\sigma u_1,$ then $$\frac{\delta_{\lambda}^\top H\delta_{\lambda}}{\lambda_{\max}(H)}=\frac{\sigma^2u_1^\top H u_1}{\lambda_{\max}(H)}=\sigma^2.$$ Therefore, in the quadratic geometry underlying the two target quantities, $$\frac{\mathbb{E}[\delta_{\mathrm{tr}}^\top H\delta_{\mathrm{tr}}]}{\mathrm{tr}(H)}=\frac{\delta_{\lambda}^\top H\delta_{\lambda}}{\lambda_{\max}(H)}=\sigma^2.$$ In this relative sense, the same $\sigma$ corresponds to the same perturbation scale, even though the ambient Euclidean norms of the perturbation vectors differ substantially. The difference arises because the trace construction aggregates curvature over all directions, whereas the top-eigenvalue construction measures curvature along one extremal direction. Beyond the quadratic regime, the two perturbation conventions need not correspond to exactly the same effective scale, so some mismatch may remain. We therefore do not interpret equal values of $\sigma$ as implying perfectly matched finite-neighborhood perturbations. **We have added this clarification in Appendix A.3** and emphasize that our empirical conclusions are based on how PCR evolves over the range of perturbation scales within each construction, rather than on a pointwise comparison at a single value of $\sigma$. _According to our additional BERT results_ (Appendix C.5 page32), we have **revised the previous Observation 3 and moved it to Obersation (e) in Appendix C.5**. --- We hope these address the reviewer’s question. Please let us know if further clarification would be helpful.
Reviewer_mD4UOpenReview ↗

Post-Rebuttal Comment

Thank you for your response. The revisions address my main concerns through added candidate pair coverage statistics, single-factor baselines, and BERT experiments. Although comparisons with scalarized measures on the same model collections remain absent and differences in candidate sets still limit PCR comparisons, I will raise my rating from 2 to 3.

Thank you very much for reconsidering the score. We appreciate that the revisions address your main concerns. We note that the same-denominator PCR analysis was added specifically to control for differences in candidate-pair coverage, with consistent conclusions.

We also agree that direct comparisons with scalarized measures on the same model collections would strengthen the study. Prior work cited in the paper has already reported limited explanatory power for several scalarized combinations, which partly motivates our non-scalarized approach. Since many scalarization choices are possible, we were not able to conduct a sufficiently comprehensive comparison during the rebuttal period, but we plan to add this in a future revision.

Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.