- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The article proposes a method for training a foundation transformer model for remote sensing data, a popular task in the community right now. It is heavily inspired by general self-supervised methods, such as MAE, JEPA, and DINO. These methods provide two versions of the data to a network: one with full context and the other with limited context. They force the network to infer the full context (or its representation) from the limited context.
The article proposes applying these principles to remote sensing time series, specifically Sentinel 2 time series. The article's originality comes from two points. First, the final guess is made at the frame level, not at a collapsed representation. Second, the guess is performed in an autoregressive manner, which is like adding further masking (i.e., masking the future). Otherwise, the architecture is the classic one-student with autograd and limited context, as well as a frozen EMA teacher with full context. On top of the student, a predictor attempts to infer the full context representation from the limited context version. This part of the architecture is causal, using causal masking in the attention matrices. Until now, this is similar to JEPA; however, the loss is similar to DINO rather than MSE or L1. Representations are projected onto prototypes with two heads and a softmax so that cross-entropy loss can be applied. As in DINO, centering is applied to the teacher's scores for stability.
The method is evaluated against a large set of other methods on different downstream tasks in two settings: frozen + trained head and fully fine-tuned. In both settings, the proposed method outperforms the others. The authors also provide an extensive ablation study to justify their choices.
Critical strengths
The article is easy to follow and understand for the most part, except for a few points that I will mention in the section on weaknesses. It uses widely known general tools and applies them to remote sensing, which is probably why it is easy to follow.
The main idea I find interesting is showing that while applying these methods to remote sensing, causal prediction (predicting the future) brings large improvements compared to unconstrained prediction. This is quite interesting because, in video MAE or video JEPA, they find that it degrades the foundation model's ability to do so. This is not illogical, as the structure and change of remote sensing time series are quite different from classical ones. In fact, I think the authors didn't emphasize this point enough and should have.
The evaluation and ablation study are very complete and they demonstrate the superiority of the method in comparison to competing methods.
Critical weaknesses
I think it's a fair criticism to say that the method section of this article is heavily inspired by well-known methods. It's mostly the experimental results that give the article significance. Even though this should not be the only criterion for rejection, the lack of real novelty is a major weakness.
There are also some things that are unclear, which I will ask about in the questions.
Questions for authors
There are important details that I can't find in the article :
-
Does the encoder network process each frame independently, and is time like a batch dimension? Or, is there an inter-frame attention mechanism like in video backbones? If there is inter-frame attention, then the prediction might not be completely causal because the early tokens could contain information about future tokens from the encoding. In that case, the main claim about causality would be greatly weakened.
-
For me, the attention matrix comes after the softmax. Therefore, the mask should be additive and have a value of 1 for current and past tokens and a value of 0 for future tokens. It seems that the mask is added before the softmax to the query-key product, like a bias. I think the authors should clarify how this is done.
-
Figure 4 might be more interesting if you extended the masking above 90% to show that you have reached optimal points. As it is, it could still grow.
-
In the method section, Objective, I gues s that the prototyping and centering of the teacher are comming from DINO, if so it should be cited
Thank you for the detailed response. The new experiments and clarifications address many of my concerns, but I remain unconvinced by the causal interpretation. As the authors explain, the encoder attends bidirectionally across time; only the predictor is causally masked. Consistent masking prevents direct copying of the target patch from another frame, but tokens encoded at or before time t may still carry information from later frames. The predictor’s attention mask therefore does not establish that its prediction at t is independent of future observations. I change the score to 2.