- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Sombrero Loss, a regularization method inspired by Mixup that operates on the original inputs without performing input interpolation. The authors analyze its expected gradient and noise covariance, establish a connection with label smoothing and curvature, and evaluate the method on several classification and structured prediction tasks.
Critical strengths
I think the paper addresses an interesting problem by investigating whether the regularization benefits of Mixup can be obtained without mixing the inputs. The proposed method is relatively simple, and the theoretical analysis provides some useful insights into its expected gradient and noise structure. Another positive point is that the authors evaluate the method on several real-world datasets, including CIFAR and ImageNet. Some of the experiments also include multiple runs and show promising improvements over Mixup and standard training.
Critical weaknesses
My main concern is about the novelty of the proposed method. Since Sombrero Loss has the same expected gradient as label smoothing and uses randomly sampled targets, I am not fully convinced that the practical method is sufficiently different from existing stochastic label-smoothing approaches. The paper would benefit from stronger comparisons with these methods.
Another concern is the connection between the theoretical analysis and the experimental results. The authors provide a detailed analysis of gradient noise and its relationship with curvature, but this connection relies on several assumptions. I think the paper needs stronger empirical evidence showing that the proposed mechanism actually explains the improvements during training.
I also have some concerns about the experimental evaluation. Although the main classification results are promising, several important ablation experiments are reported using only single runs. This makes it difficult to determine whether the relatively small improvements are consistent. The detection and translation results also show only modest gains without repeated-run statistics.
Another point is the connection with Mixup. The appendix shows that the retained gradient can differ significantly from the actual Mixup gradient, especially at larger mixing strengths. While this does not invalidate the proposed method, I think the authors should clarify what is actually being preserved from Mixup and why this particular component is important.
Finally, I could not find implementation code in the submitted materials so I was not able to reproduce it.
Questions for authors
-
Since the proposed Sombrero Loss has the same expected gradient as label smoothing and uses stochastic targets, could the authors explain more clearly what distinguishes it from existing randomized label-smoothing methods? It would be useful to include a direct comparison with stochastic label smoothing under matched mean and variance.
-
Proposition 4 connects the intrinsic gradient covariance with a modeled GGN matrix under several assumptions. Could the authors provide empirical measurements showing how closely the actual gradient-noise covariance aligns with the actual GGN or Hessian during training?
-
Table 1 provides mean-matched comparisons to support the contribution of stochastic regularization beyond label smoothing, but most results are from single runs. Could the authors repeat these experiments across multiple seeds and report the variation?
-
Appendix D.6 shows that the retained gradient can differ significantly from the actual Mixup gradient. Could the authors explain how this affects the interpretation of the proposed first-order purification and why the retained component is preferable to other possible decompositions?
-
The improvements in detection and translation are relatively small and reported as point estimates. Could the authors provide repeated-run results to establish whether these improvements are consistent?
-
Could the authors provide the implementation code and more complete reproduction instructions, especially for the centered and decoupled variants?
Note: I might not be an expert in this field so I might have some misunderstanding of the method, and I could not review the appendix deeply.
Thank you for your reply. I hope your health is fine now.
I am still not convinced about the novelty since for me it looks like a combination of several known ideas.
Also, without the code, I cannot check the reproducibility. It would be good if you send the code, at least for CIFAR10 to confirm your results.
For now, I am maintaining my score 2 (weak reject) but it is not my last rating, and I am open to change my assessment since the work seems useful.