acceptodds
Under review as a conference paper at ICLR 2027

MARVEL: Masked Autoregressive Visual Understanding and Generation for Spectral Remote Sensing

Abstract

Spectral remote sensing observations support both scene understanding and the generation of unobserved spectral responses. Bringing understanding and generation into unified pretraining allows generative supervision to support transferable representation learning. However, more accurate generation does not necessarily yield better features for understanding. To address this challenge, we introduce MARVEL, a masked autoregressive pretraining framework for spectral understanding and generation. A bidirectional encoder integrates all visible observations, and full-context conditional alignment converts the resulting features into conditions for each target wavelength and spatial position. Spectral content autoregression then uses these conditions to generate responses in wavelength order. Each response is supervised against its own target and re-encoded to guide subsequent predictions. Through this reuse, feedback from subsequent predictions also contributes to shared representation learning. MARVEL achieves competitive performance on 14 downstream tasks. Without additional generation training, it also achieves strong results on spectral generation tasks using Sentinel-2 and Landsat-8 data. Controlled experiments further validate the effectiveness of the proposed design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Optical spectral, earth observation, self-supervised learning, pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fefPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a method to jointly pre-train a remote sensing foundation model (RSFM) that is useful for both downstream recognition/understanding tasks and reconstruction/generation. Specifically, MARVEL converts an image to tokens with an 8x8 patch size and no channel groupings, so each token contains information from only 1 channel/band/wavelength. During pre-training, these tokens are randomly masked and encoded with a standard transformer. Then, a stage-1 decoder receives these visible-token encodings and adds placeholder tokens to the sequence which represent the masked tokens (these placeholder tokens include channel and spatial position information). This stage-1 decoder is the same as a standard MAE decoder (i.e. all-to-all attention)—however, it doesn't predict pixels right away. It provides these tokens to an autoregressive (AR) stage-2 decoder which predicts all the masked spatial tokens of one channel/band/wavelength at a time. The AR head receives the stage-1 decoder's output and the previously predicted tokens (predicted by the same AR model in the previous loop). Loss is applied to the AR's predictions and gradients also flow through the loops, so the AR's pixel predictions should be both accurate and useful to future channel predictions.

The paper also introduces wavelength-based RoPE and uses teacher forcing early in pre-training.

Experimentally, the paper shows gains over several RSFMs in common benchmark datasets. Ablations show (i) the combination of a 2-layer MAE-style decoder and a 2-layer AR decoder work best, (ii) gradients flowing through the loop helps, (iii) re-encoded pixel predictions outperform re-using/updating the hidden states from the prior decoding step, and (iv) wavelength-RoPE helps.

Critical strengths
  • improving the recognition abilities of a generatively pre-trained encoder is quite interesting, this generative pre-training has advantages over other SSL methods (e.g. more robust than self-distillation and does not need large batches like contrastive methods).
Critical weaknesses
  • the paper seems to evaluate generation using PSNR, I would prefer an evaluation that checks if the generations are useful somehow, e.g. can you plug in the generations into a pre-trained encoder to get predictions on some downstream task? Or another practical use case.
  • I get that having 1 model for both generation and recognition would be nice but I'm not convinced it is needed
  • there are ablations but do not ablate mask ratio or masking strategy (as far as I can tell)—by "masking strategy" I mean something like masking-out entire channels at a time during pre-training
  • I have serious concerns about the downstream recognition experiments. For some models, the authors get pre-trained checkpoints and evaluate themselves and for others the authors re-pre-train models, then evaluate them to get results. It seems that many of the results obtained by the authors are lower than results reported by the original papers for which they compare against. Furthermore, SOTA models are missing, most notably IMHO is OlmoEarth. Finally, MARVEL tokenizes channels without grouping so their ViT-Base is actually a lot more expensive / has more capacity than other ViT-Base models that have equal parameters but fewer tokens.
Questions for authors

none

Reviewer_kAMxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MARVEL employs masked pre-training on multispectral remote sensing data for the purposes of spectral understanding and the generation of missing spectral bands. Its decoder utilizes all observed bands to make predictions, generating the missing bands in order of wavelength. Previous prediction results are re-encoded for use in subsequent steps; the gradients propagated through this sequential process facilitate the learning of shared features.

Critical strengths

The motivation for this study is clear and well-founded. The paper points out that improvements in generation quality do not necessarily translate into better encoder representations.

The study is supported by extensive experiments, including 6 segmentation tasks from GEO-Bench and 8 additional benchmarks for transfer learning.

Wavelength descriptors, history scheduling, and reduced-observation tests address important differences in spectral inputs.

Critical weaknesses
  1. Strictly speaking, it is difficult to characterize MARVEL as a "unified framework for understanding and generation"; in essence, it is merely a band-regression model.
  2. As shown in Table 3, MARVEL underperforms compared to the authors' own reproduction of AnyBand-Diff. Furthermore, regression models inherently hold an advantage over diffusion models when evaluated using metrics such as PSNR. 3, The design of the model in the paper leans toward an engineering-oriented approach, functioning more as a combination of existing techniques; for instance, λ-RoPE appears quite similar to previously established wavelength-aware positional encodings.
Questions for authors

1,How does the method perform when a causal-observation variant is trained from scratch, and when it is transferred to hyperspectral data or a sensor not seen during pretraining? 2,Could the authors state precisely what is new in each component compared with these prior works? Could they also identify which part of the design they consider the core methodological contribution, beyond combining these elements for multispectral data?

Reviewer_aoM4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks a simple question: does better generation also lead to better understanding? The authors introduce MARVEL to explore this through missing-band recovery in remote sensing. The model first combines information from all visible bands, then predicts missing bands in wavelength order, passing earlier predictions to later steps. Errors from later predictions can also flow back through this path to train the shared encoder. After pretraining, the encoder supports classification and segmentation, while the full model can directly recover missing bands. Ablation studies examine the roles of the generation strategy, prediction reuse, and gradient feedback to explain how generation training can help the model learn better representations.

Critical strengths
  1. The paper asks a clear question: better generation does not necessarily mean better understanding. Designing a method around this question is more convincing to me than simply putting understanding and generation into one model.

  2. The experiments offer some interesting insights. For example, pure autoregressive training improves generation but hurts understanding. The authors also separate the effects of reusing past predictions and keeping gradients through them. These analyses help explain why the method works.

  3. The paper offers a concrete way for generation training to help spectral understanding, allowing feedback from later predictions to reach the shared encoder through differentiable prediction history. After pretraining, the encoder supports classification and segmentation, while the full model can recover missing bands without additional generative training.

Critical weaknesses
  1. The practical value of the generated results is unclear. Each band is normalized separately for each image, so the results mainly show recovery in normalized space. I’m not sure how well this demonstrates recovery of real spectral responses or supports tasks that require accurate reflectance.

  2. Some key gains are small, with no results from repeated runs. Average segmentation performance improves over TerraMind by 0.25 points, and keeping gradients through past predictions adds 0.31 points. It is hard to tell how stable these gains are, especially since the latter supports the paper’s core mechanism.

  3. The extra computational cost is unclear. The method combines full attention, autoregressive prediction, and differentiable reuse of past predictions. Fixing the decoder depth does not account for multiple prediction steps or gradients across them. I’d like to know whether the gains hold under the same compute budget and justify the extra time and memory.

Questions for authors
  1. How can predictions be converted back to real reflectance, and where would the mean and variance of a missing band come from? Evaluation in physical units would help clarify the practical value.

  2. Is the 0.31-point gain from keeping gradients through past predictions consistent across random seeds? Repeated runs or variability estimates for this comparison would strengthen the claim.

  3. Could the authors report training cost, inference time, and peak memory for Full-only, AR-only, and the full model? This would help assess whether the gains justify the extra cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.