- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies training ViTs, including PiT and Swin, from scratch on small datasets. Based on an analysis of the local sensitivity of attention to input perturbations, it proposes GRASP, a consistency regularization method to improve ID and OOD generalization. The experimental results show consistent improvements across multiple datasets and architectures, with additional gains when GRASP is combined with existing methods.
Critical strengths
-
The observation that scratch-trained models are more vulnerable to gradient-guided attention perturbations than pre-trained models is interesting and may inspire further research.
-
The experimental results are good, with consistent improvements across multiple datasets and ViT, PiT, and Swin architectures. The experiments also evaluate the combination of GRASP with several existing methods.
Critical weaknesses
- The theoretical support is relatively weak, despite the good experimental results. The proofs establish several properties under restricted conditions, but the resulting explanation of the training mechanism appears stronger than what is actually proved. The following points need clarification:
a) Proposition 3.3: It assumes that A approaches a one-hot distribution and then shows that the Jacobian vanishes. It mainly describes the saturation property of softmax. It does not explain why training would make A approach a one-hot distribution.
b) Proposition 4.1: The implication depends on sufficient separation between token-wise gradients. But, the paper neither proves nor experimentally verifies whether this condition holds during training.
c) Input perturbation analysis: assuming a bounded logit Jacobian and then deriving bounded logit perturbations is a standard conditional argument. It does not establish that GRASP reduces this Jacobian bound.
-
The novelty over existing attention adversarial regularization methods needs clarification. The authors should clearly explain what GRASP adds compared with these methods.
-
The sensitivity comparisons may use different perturbation strengths. According to Eq.11, the perturbation magnitude depends on the magnitude of G. Pre-trained models, GRASP-trained models, and ordinary scratch-trained models may have different CE confidence levels, gradient scales, and downstream parameter scales. Therefore, using the same T_G does not guarantee the same logit perturbation strength. The authors should clarify how this is controlled. Moreover, the perturbation constraint changes during the derivation, and Eq.11 has not been shown to be the worst-case perturbation under the original norm budget.
Questions for authors
-
How does the theoretical analysis support the claimed training mechanism beyond the one-hot limit in Proposition 3.3? In particular, does the gradient separation required by Proposition 4.1 hold during training, and is there theoretical or experimental evidence that GRASP reduces the logit Jacobian bound?
-
What are the main methodological differences between GRASP and existing attention adversarial regularization methods? Please clarify which aspects constitute the new contribution.
-
Are the sensitivity comparisons conducted under the same actual logit perturbation budget across scratch-trained, pre-trained, and GRASP-trained models? Since Eq.11 depends on the magnitude of G, how are differences in gradient scale controlled?
-
Please also clarify whether Eq.11 is a worst-case perturbation under the original norm constraint or only under the subsequently introduced variance constraint.