- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a method to jointly pre-train a remote sensing foundation model (RSFM) that is useful for both downstream recognition/understanding tasks and reconstruction/generation. Specifically, MARVEL converts an image to tokens with an 8x8 patch size and no channel groupings, so each token contains information from only 1 channel/band/wavelength. During pre-training, these tokens are randomly masked and encoded with a standard transformer. Then, a stage-1 decoder receives these visible-token encodings and adds placeholder tokens to the sequence which represent the masked tokens (these placeholder tokens include channel and spatial position information). This stage-1 decoder is the same as a standard MAE decoder (i.e. all-to-all attention)—however, it doesn't predict pixels right away. It provides these tokens to an autoregressive (AR) stage-2 decoder which predicts all the masked spatial tokens of one channel/band/wavelength at a time. The AR head receives the stage-1 decoder's output and the previously predicted tokens (predicted by the same AR model in the previous loop). Loss is applied to the AR's predictions and gradients also flow through the loops, so the AR's pixel predictions should be both accurate and useful to future channel predictions.
The paper also introduces wavelength-based RoPE and uses teacher forcing early in pre-training.
Experimentally, the paper shows gains over several RSFMs in common benchmark datasets. Ablations show (i) the combination of a 2-layer MAE-style decoder and a 2-layer AR decoder work best, (ii) gradients flowing through the loop helps, (iii) re-encoded pixel predictions outperform re-using/updating the hidden states from the prior decoding step, and (iv) wavelength-RoPE helps.
Critical strengths
- improving the recognition abilities of a generatively pre-trained encoder is quite interesting, this generative pre-training has advantages over other SSL methods (e.g. more robust than self-distillation and does not need large batches like contrastive methods).
Critical weaknesses
- the paper seems to evaluate generation using PSNR, I would prefer an evaluation that checks if the generations are useful somehow, e.g. can you plug in the generations into a pre-trained encoder to get predictions on some downstream task? Or another practical use case.
- I get that having 1 model for both generation and recognition would be nice but I'm not convinced it is needed
- there are ablations but do not ablate mask ratio or masking strategy (as far as I can tell)—by "masking strategy" I mean something like masking-out entire channels at a time during pre-training
- I have serious concerns about the downstream recognition experiments. For some models, the authors get pre-trained checkpoints and evaluate themselves and for others the authors re-pre-train models, then evaluate them to get results. It seems that many of the results obtained by the authors are lower than results reported by the original papers for which they compare against. Furthermore, SOTA models are missing, most notably IMHO is OlmoEarth. Finally, MARVEL tokenizes channels without grouping so their ViT-Base is actually a lot more expensive / has more capacity than other ViT-Base models that have equal parameters but fewer tokens.
Questions for authors
none