acceptodds
Under review as a conference paper at ICLR 2027

CRIK: Convolutional Reshaping on Perturbation Intervals Partitioned by Key Points to Enhance Generalization

25 views

Abstract

Generalization is a primary criterion for evaluating deep neural networks. Research shows that sharp minima in the loss landscape often lead to poor generalization. Sharpness-aware minimization (SAM) employs worst-case perturbations around minima to enhance generalization. However, SAM overlooks important geometric information about the loss landscape within the perturbation interval, which can be reflected by the key points we proposed. To address this limitation, we partition the perturbation interval using the key points and employ convolutional operations to adaptively adjust each interval's contribution to generalization performance. Furthermore, our analysis shows the impact of interval numbers on generalization. Based on this analysis, we propose convolutional reshaping on perturbation intervals partitioned by key points (CRIK). We theoretically examine CRIK's convergence, derive its escape time from sharp minima, and demonstrate that it implicitly combines gradient and Hessian information adaptively. Simulations and experiments demonstrate that CRIK improves generalization performance.

Primary area:
optimization
Keywords:
Deep neural networks, generalization, sharp minima, sharpness-aware minimization.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aGCjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes an improvement to sharpness aware minimization (SAM) called CRIK: convolutional reshaping on perturbation intervals partitioned by key points. The method can roughly be summarized as follows:

Instead of only evaluating the gradient at the current parameters and the (approximated) worst-case perturbed parameters as with SAM, the method partitions the interval into several intervals. These are essentially line segments along the parameter space path between the parameters and perturbed parameters. The key points are then selected points defining each interval. The loss is then evaluated at each keypoint, and then combined via a weighted sum of the change in loss between each adjacent point. The gradient of this sum is then added to the original gradient.

Importantly, this is more computationally expensive than SAM (several forward passes and a big backward pass is required): so instead of performing this at each iteration, an update interval scheduler is used such that the perturbed gradient is only calculated for certain updates, otherwise the previously calculated perturbed gradient is used. The scheduler is chosen such that the computational requirements roughly match that of SAM.

The paper then analyzes the algorithm theoretically, before measuring its generalization performance in comparison to baselines using CIFAR10, CIFAR100, ImageNet, self supervised learning, and a time series task.

Critical strengths

The method is very interesting, and the individual elements of the method (interval selection, weighting, loss aggregation) seem well motivated and non trivial. The experimental evidence also cover a wide range of datasets and experimental setups. Furthermore, while I am unable to completely verify the claims, there is considerable theoretical analysis of the algorithm's properties.

Critical weaknesses
  1. The method does not convincingly beat SAM: the gains in performance are very slight and within standard deviation between different runs, and the TSAM baseline outperforms CRIK in several cases. The gap becomes more convincing when CRIK and TSAM is combined, but the paper does not explain how they are combined and whether the combination is still at the same computational cost as SAM. Unfortunately this leaves it uncertain as to whether CRIK is truly an improvement on SAM, and it's unclear whether the increased complexity of the proposed method is justifiable.
  2. No hyperparameter tuning is performed for the baselines. Importantly, the learning rate is held fixed for all the methods. This implies that one method could simply be less optimally tuned than another, instead of necessarily being better/worse.
  3. The paper does not mention the memory requirements of the proposed algorithm.
Questions for authors
  1. How are CRIK and TSAM combined, and is the computational cost still comparable to SAM? How does the cost of CRIK compare to TSAM?
  2. How optimal are the baselines? Judging by the poor performance of vanilla Adam in Table 1, I suspect that there is still significant room for performance improvement with a learning rate sweep and/or learning rate scheduling.
  3. Does the gradient calculation at line 16 of Algorithm 1 not result in a significant increase in memory footprint? The gradient calculation is now w.r.t. several forward passes.
  4. How is checkpoint selection performed? Is the optimal validation performance checkpoint selected per run or only the last checkpoint, and how do the two selection methods compare?
  5. Please elaborate on the experimental setup for the SSL experiment of Section 3.5. Is CRIK used for both the pretraining and probe training phase? How exactly are the linear heads trained?
  6. Why is SAM or TSAM not compared against in Table 3? The sudden switching of baseline comparisons is not motivated.
Reviewer_skw2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRIK, a sharpness-aware optimization method designed to improve generalization by exploiting geometric information within perturbation intervals that conventional SAM may overlook. CRIK partitions the perturbation interval using recursively selected midpoints and applies convolutional reshaping to adaptively weight the loss changes across subintervals. The resulting regularization term is incorporated into the optimization objective, while a dynamic gradient-reuse strategy reduces computational overhead. The paper provides theoretical analyses of the properties of key points, the implicit combination of gradient and Hessian information, convergence, generalization bounds, and escape times from sharp minima. Extensive experiments validate the effectiveness of the proposed method.

Critical strengths
  • The paper presents a clear and well-motivated research problem by highlighting a limitation of SAM, which primarily focuses on the worst-case perturbation while overlooking the loss variations within the perturbation interval. Building on this observation, the authors propose a novel interval-based reshaping approach to capture richer geometric information about the loss landscape. This provides an interesting and potentially valuable perspective.
  • The experimental evaluation is relatively comprehensive, covering a diverse range of tasks, including image classification, self-supervised learning, language modeling, and time-series prediction. The consistent improvements across different tasks and model architectures provide empirical evidence of the effectiveness and general applicability of the proposed method.
Critical weaknesses
  • The design of the convolutional weighting scheme appears somewhat heuristic and lacks sufficient justification. In particular, it is unclear why the weighting function should be monotonically increasing, thereby assigning larger weights to intervals with higher loss values, or how this design specifically contributes to improved generalization. Furthermore, the choice of a tanh-based weighting function also seems largely empirical. Further theoretical or intuitive explanations would help justify these particular design choices.

  • The experimental comparisons are relatively limited outside image classification. While a broad range of baselines are evaluated on image classification tasks, the comparisons on self-supervised learning, language modeling, and time-series prediction are less comprehensive. Including more competitive baselines, particularly recent SAM variants, would help better validate the effectiveness and general applicability of the proposed method.

Questions for authors
  • What is the rationale behind the proposed convolutional weighting scheme? In particular, why should intervals with higher loss values receive larger weights, and why is a tanh-based function preferred? How do these specific choices contribute to better sharpness characterization and generalization?
  • Could the authors provide more comprehensive comparisons on non-vision tasks? The language modeling experiments only compare against Adam, while the time-series experiments mainly compare against Adam and RAD. Comparisons with SAM and its recent variants would be important for assessing whether the proposed method consistently improves upon existing sharpness-aware optimizers across different domains.
Reviewer_1XbDOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRIK, an optimization method that addresses the underutilization of information within the parameter perturbation interval in SAM’s commonly used single-step approximation. The method selects key points along the perturbation direction through recursive bisection, applies nonlinear weighting to the loss increments over the resulting subintervals, and constructs an additional optimization term in the form of a discrete convolution to guide the model toward flatter regions of the loss landscape. To control the computational overhead of evaluating multiple points, the authors further introduce a dynamic gradient reuse mechanism.

Critical strengths
  1. The research motivation is clear, and the method is intuitive.
  2. The method operates by modifying gradient updates, making it a plug-and-play approach with broad applicability.
Critical weaknesses
  1. The source of the performance gains is insufficiently investigated. The method simultaneously introduces multi-point sampling, nonlinear weighting, and dynamic gradient reuse, but the existing ablation studies do not adequately distinguish the contributions of these components or their interactions. More targeted controlled experiments are recommended to verify the necessity of the core design choices.
  2. The empirical improvements are not consistently substantial, with only small gains on some tasks. Further analysis and explanation may be needed.
Questions for authors

Please refer to the questions raised in the weaknesses above.

1 comment

Dear reviewer,

Have the authors address your concerns in their replies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.