- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes an improvement to sharpness aware minimization (SAM) called CRIK: convolutional reshaping on perturbation intervals partitioned by key points. The method can roughly be summarized as follows:
Instead of only evaluating the gradient at the current parameters and the (approximated) worst-case perturbed parameters as with SAM, the method partitions the interval into several intervals. These are essentially line segments along the parameter space path between the parameters and perturbed parameters. The key points are then selected points defining each interval. The loss is then evaluated at each keypoint, and then combined via a weighted sum of the change in loss between each adjacent point. The gradient of this sum is then added to the original gradient.
Importantly, this is more computationally expensive than SAM (several forward passes and a big backward pass is required): so instead of performing this at each iteration, an update interval scheduler is used such that the perturbed gradient is only calculated for certain updates, otherwise the previously calculated perturbed gradient is used. The scheduler is chosen such that the computational requirements roughly match that of SAM.
The paper then analyzes the algorithm theoretically, before measuring its generalization performance in comparison to baselines using CIFAR10, CIFAR100, ImageNet, self supervised learning, and a time series task.
Critical strengths
The method is very interesting, and the individual elements of the method (interval selection, weighting, loss aggregation) seem well motivated and non trivial. The experimental evidence also cover a wide range of datasets and experimental setups. Furthermore, while I am unable to completely verify the claims, there is considerable theoretical analysis of the algorithm's properties.
Critical weaknesses
- The method does not convincingly beat SAM: the gains in performance are very slight and within standard deviation between different runs, and the TSAM baseline outperforms CRIK in several cases. The gap becomes more convincing when CRIK and TSAM is combined, but the paper does not explain how they are combined and whether the combination is still at the same computational cost as SAM. Unfortunately this leaves it uncertain as to whether CRIK is truly an improvement on SAM, and it's unclear whether the increased complexity of the proposed method is justifiable.
- No hyperparameter tuning is performed for the baselines. Importantly, the learning rate is held fixed for all the methods. This implies that one method could simply be less optimally tuned than another, instead of necessarily being better/worse.
- The paper does not mention the memory requirements of the proposed algorithm.
Questions for authors
- How are CRIK and TSAM combined, and is the computational cost still comparable to SAM? How does the cost of CRIK compare to TSAM?
- How optimal are the baselines? Judging by the poor performance of vanilla Adam in Table 1, I suspect that there is still significant room for performance improvement with a learning rate sweep and/or learning rate scheduling.
- Does the gradient calculation at line 16 of Algorithm 1 not result in a significant increase in memory footprint? The gradient calculation is now w.r.t. several forward passes.
- How is checkpoint selection performed? Is the optimal validation performance checkpoint selected per run or only the last checkpoint, and how do the two selection methods compare?
- Please elaborate on the experimental setup for the SSL experiment of Section 3.5. Is CRIK used for both the pretraining and probe training phase? How exactly are the linear heads trained?
- Why is SAM or TSAM not compared against in Table 3? The sudden switching of baseline comparisons is not motivated.
Dear reviewer,
Have the authors address your concerns in their replies?