- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes SimPE - a patch-similarity based positional encoding module to replace standard positional encoding based on patch global or relative positioning in vision transformers. The paper shows SimPE is robust to rotations using Lipschitz continuity bounds. The paper evaluates simPE rotation robustness against other PE variants in a controlled setting and demonstrates overall better robustness.
Critical strengths
- The proposed SimPE shows overall better rotation robustness in the controlled setting against the other evaluated PE, especially in Chest X-Ray benchmark.
Critical weaknesses
-
The relation of the theoretical claims to the evaluated setting is unclear. The motivation of the method is to improve robustness to spatial rotation of 2D inputs, which is also the evaluation setting. The theoretical claims, however, are related to rotations of the embeddings in their latent space, and it is unclear why spatial rotations of the inputs translates to such rotations in the embedding space.
-
The empirical results do not show a consistent advantage for the proposed simPE. In some cases there is a large gap compared to other methods, for example vs. Sin at Arrows in 15 deg and FashionMNIST in 30 deg. Additionally, there are no reported confidence itervals.
-
The method is not evaluated against data augmentations which is one of the most common approaches to improve robustness. The authors justify this as a controlled experimental setup, but this limits the ability to assess practical usability of the proposed method.
Questions for authors
- Can you explain how spatial rotations are translated to rotations of the latent space?
- Can you repeat the evaluation on multiple seeds and report confidence intervals (at least on a single benchmark)?
- Can you repeat the evaluation using rotation augmentations (at least on a single benchmark)?
3 comments
I thank the authors for seriously engaging with my comments and their detailed response. Unfortunately, some of my concerns remain. Regarding the theoretical results, the authors’ clarification suggests that the claimed stability under input rotations follows from a general Lipschitz continuity argument. Rotation enters this argument only through the distance between the original and rotated images; no rotation-specific property of SimPE is used. Under the same assumptions on image rotation, analogous results could therefore be obtained for any alternative positional encoding that is Lipschitz continuous in its input. Consequently, the revised result does not identify a rotation-robustness property particular to SimPE or explain its advantage over competing positional encodings. Second, my concerns regarding evaluation with data augmentation were not sufficiently addressed. While I appreciate the authors’ approach of evaluating the method in a controlled setting, I believe a complementary evaluation with rotation augmentation could have clarified the practical benefits of the proposed method. Due to the above, I maintain my original score.
We respectfully disagree witrh the reviewer conclusions. It is not trivially true that "under the same assumptions on image rotation, analogous results could therefore be obtained for any alternative positional encoding that is Lipschitz continuous in its input". This shoud be formally proved as done by us for the specific formulation of SimPE. We are not aware of any formal proof analogouos to ours for any other PE. Moreover, we are not claiming that only simPE is robust, we are claimimg that, given the specific theoretical properties (formally proved), simPE is robust and experiments suggets that it is more robust than competing approaches. For the augmentation argument, it is honestly true that a complementary evaluation with rotation augmentation would be biased, since adding rotation in the tranining set wil defiinitely inform the model about the rotation process. On the contrary, we have set an experimental framework where the prediction capabilities of the model are challanged in the hardest way.
Rebuttal reviewer UimQ
We thank the reviewer and we acknowledge the important point issued by her/him concerning the mismatch between some theoretical results and experimentation. In particular, we point out that the main theoretical result (Theorem 1) applies also by directly considering the input space. The paper presents the result for any X and Y in the feature/embedding space. If f is the feature extractor, I an input image and Iθ the rotated image through rotation Rθ, then the theorem holds for X=f(I) and Y=f(Iθ) by reconciling this part with experiments (it establishes a bound for the actual transformation used in the experiments starting from an image and the corresponding rotation). In addition, if we also assume that the feature extractor is Lipschitz (with constant Lf), then we directly obtain a corrected version of the theorem relating the result directly to the input space.
Let Δθ(I)=∥\mboxsimPE(f(Iθ))−\mboxsimPE(f(I))∥F, then Δθ(I)≤2LΦLkLfN∥Iθ−I∥F We also acknowledge that Corollary 2 was not properly presented in the paper. It is a conditional corollary that, in order to be adapted to the input space, requires an additional assumption on the rotation and interpolation scheme.
At the image level we will assume the following regularity assumption: ∥Iθ−I∥F≤cI∣θ∣ The assumption is satisfied by rotations and interpolation schemes that produce continuous changes in pixel values as the rotation angle changes. In particular, since we deal with discrete digital images (finite real-valued pixel matrices),
given an image I, its rotated version Iθ follows a specific interpolation scheme of an underlying continuous rotation process from angle t=0 to angle t=θ, and its rotatory derivative is ∂t∂It. Smooth changes can be modeled by the property of local Lipschitz continuity of the rotational derivative that allows us to conclude (Rademacher's theorem) that Iθ−I=∫0θ∂t∂It\mboxdt. Now consider a discrete image I and the rotated image Iθ obtained using a specified interpolation scheme.If the interpolated scheme produces a locally Lipschitz continuous rotational derivative (i.e., interpolated pixel values vary continuously, with a uniformly bounded rate of change almost everywhere, as a function of the rotation angle), and we further assume that rotational derivative is essentially uniformly bounded in the norm by a constant cI (i.e. the rotation rate has a specific finite limit that applies almost everywhere), then we can finally prove that
\Vert\int_0^\theta \frac{\partial}{\partial t} I_t \mbox{ } dt \Vert_F \leq \vert \int_0^\theta \Vert \frac{\partial}{\partial t} I_t \Vert_F \mbox{ } dt \vert \leq c_I |\theta|.$$ The required assumptions are not valid for any rotation, but they are valid for several rotation of interest such as bilinear, bicubic, or suitable band-limited interpolation. In particular, in the experiments we used bilinear interpolation having rotational derivative defined almost everywhere (failing to be differentiable only along grid boundaries} and satisfying the required conditions of the corollary. In the revised verson, we will provide the details of the proof together with the required references justifying any assumption made. As a corollary of the corrected formulation of Theorem 1 we finally get the corrected formulation of the angle bound (Corollary 2): $$\Delta_\theta(I)\leq 2 L_{\Phi} L_{k} L_f \sqrt{N} c_I |\theta|$$ This reconciles theory and experiments. Concerning statistical reliability, we will add in revised version the required means and confidence intervals for a better comparison. Concerning rotation agumentation , this has been deliberately avoided since the main goal of the experiments are to test the robustness to rotations, thus it would make a little sense to add rotations in the training set that would inform the model about the rotation process. We believe that a sound approach is to have ONLY unrotated images in the training, by testing robustness by rotating images in the test set. This is particularly relevant in the tested clinical application where rotations are errors in the image capturing and should not be used tro train the model.