acceptodds
Under review as a conference paper at ICLR 2027

PFU: A Universal Interface between Earth Observation Data and Frozen Foundation Models

Abstract

RGB foundation models discard measurements that do not fit their three-channel interface. We study a different reuse strategy: a trainable spectral frontend that maps Sentinel-2 bands to pseudo-RGB while leaving the deployed backbone unchanged. Under one matched protocol and three predetermined seeds, PFU-E improves over the RGB baseline by 3.482 micro-mAP points on BEN-MM and 3.644 accuracy points on So2Sat. A 15-parameter spectral projection is already stronger than RGB on BEN-MM, while capacity-matched and test-time B08 controls show that the spatial frontend uses aligned NIR rather than only additional parameters. Beyond classification, pretrained PFU-E reaches 41.935 mIoU on PASTIS-HD, 19.909 points above a bare linear probe and 3.351 above scratch PFU. Twelve-band PFU-D improves Cashew Plant segmentation with both LVD and SAT backbones, and PFU-I dual-path and S1+S2 experiments demonstrate that the interface extends to wider spectra and multimodal inputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Earth Observation, Foundation Models, Universal Interface

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9dnfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors train a spectral frontend to map multispectral observations into pseudo-RGB images to use in frozen vision models, with extensions to segmentation, additional spectra and multimodal inputs.

Critical strengths

The classification experiments which the authors proposed, capacity RGB controls, and NIR interventions provide reasonable evidence that the frontend exploits additional spectral information.

Critical weaknesses

First, the contribution beyond existing spectral adaptation methods remains insufficient. As per my knowledge, previous established methods and models already combine spectral adapters with frozen visual backbones(many other variations of existing large/small models, including spectraDino). However, the main comparisons are against RGB probing and linear projection. Without matched comparisons to the closest methods. And the results demonstrate the usefulness of additional spectral information and trainable processing more than the actual merits of such design, which is a digression.

Second, the major segmentation gains can potentially be attributed to other architectural changes. Authors claimed a 19.9-point PASTIS-HD improvement introduces both PFU and VPT-Deep, without RGB+VPT-Deep control set. Plus the approximately 23.6-point PFU-I gain with the SAT backbone also replaces a linear decoder with U-Net (Tables 2 and 4). The comparisons cannot isolate the interface’s contribution from these confounds.

Another problem is that the proposed interface’s generality was not sufficiently backecd. PFU-D modifies the patch projection, departing from the external pseudo-RGB interface. Cross-backbone results are also inconsistent: on BEN-MM, the SAT backbone performance drops with PFU. Separately trained variants with different adaptation mechanisms do not demonstrate that one interface transfers reliably across backbones and sensors.

Questions for authors

Do you consider to include the comparisons with some close peer works that developed spectral adaptation methods? Can you fix the decoder and check how much is the improvement?

Reviewer_MPjWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new startegy to leverage frozen fondation model (such as DINOV3 or JEPA) for multispectral Earth Observation Data (such as Sentinel-2). The idea is to learn spectral module that maps Sentinel-2 bands to pseudo-RGB.
The approach is evaluated for several frozen backbones (mainly DINOV3 based) and evaluated on 3 datasets BEN-MM, PASTIS-HD and So2SAT. This module needs to be trained for each target datasets. The proposed approach seem to improved performances compared to simple baselines but comparison to state of the art adapation module is missing.

Critical strengths
  • The idea behind PFU is simple. Yet, its implementation and evaluation require a careful analysis. The subject is timely due to an increasing number of fondation models and their training cost.
  • The paper contains a lot of experiments with differents backbones and different target datasets containing Sentinel 1 and / or Sentinel 2 data.
Critical weaknesses

Costly archtitecture :

  • The proposed PFU spectral Frontend is about 4M parameters with is very costly when comparing to Parameter Efficient Fine Tuning (PEFT) methods such as LoRA [1].

Missing baselines and related work:

  • Related Work about multimodal Fondation models are missing such as AnySat [2], RAMEN [3] or Galileo [4] or Panopticon [5].

  • Reference missing for the PASTIS-HD dataset it should be [6]. Reference missing for Cashew Plant dataset.

  • It is surprising how few references and comparative methods exist on the topic of adapting foundation models.

  • Main limitation: The approach seems only to rely on 4 of the 12 bands of Sentinel 2 data (subfigure 1 of figure 1) and it is not clear how this approach can transfer between datasets with the same input data. The PFU module seems to learn the semantic specific to the target dataset rather than the semantic labels.

Readability :

  • In the abstract B08 should be introduced for people outside of Remote Sensing community.
  • Figure 1 subfigure 4 What do you mean by "Matched Three-Seed Results".
  • In section 3, it is not clear at all how the model is trained and what is the loss function.
  • Figure 2, optical RGB, SAR and time series are mentionned as input data but the capability of the model to be able to handle such different input data is not explicit at all in the text.
  • Clear references to the different Tables are missing in the main text.
  • The analyses of the results seem to be superficial.
  • Figure 4 : the figure is too distorted, and it's hard to know what to make of it.

Formatting problems :

  • There are far too many significant digits (in the abstract, the main text and all the tables).

[1] Lora: Low-rank adaptation of large language models EJ Hu et al. ArXiv 2021 [2] Anysat: One earth observation model for many resolutions, scales, and modalities G Astrucet al. - CVPR 2025 [3] RAMEN: Resolution-adjustable multimodal encoder for Earth observation N Houdré et al. CVPR 2026 [4] Galileo: Learning global and local features in pretrained remote sensing models Tseng G. et al. ICML, 2025 [5] Panopticon: Advancing any-sensor foundation models for earth observation. Waldmann L. et al. CVPR Workshops 2025. [6] Omnisat: Self-supervised modality fusion for earth observation G Astrucet al. - ECCV, 2024

Questions for authors

Clarification requests:

  • How the module is trained ?
  • How the proposed module compared to Parameters efficient adaptation module such as LoRA or fine-tuning only the end of the fondation models ?
  • Figure 1 subfigure 4 What do you mean by "Matched Three-Seed Results"
  • What is "reviewer controls" mentioned line 194 ?
  • The state-of-the-art performances of the fondation models on the different benchmarks should be mentionned for fair comparison (in Tables 1, 2, 3)
  • The Table 4 should be modified because the rows 1 and 2 are a different task of the rows 3 and 4.

Suggestions for future work:

  • Train a target dataset agnostic PFU that only adapt to the input dataset whatever the target task and then evaluate as the other methods in fine-tuning or linear probing on classical benchmark (including the ones already used).
  • Improve the related Work and the comparative baselines.
Reviewer_YioUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an adapter to map multispectral inputs to RGB images. This allows for improved performance with RGB backbones as compared to existing performance. The method is tested only with the DINOv3-LVD backbone. This, along with the lack of any significant error analysis, marks a work that is unaware of its limitations. Furthermore, the writing has been heavily edited by AI, making it an awkward read. While the technique shows promise based on the emperical results, it needs more exploration and analysis before its usage in the real world.

Critical strengths
  • The empirical results show substantial improvement over existing methods.
Critical weaknesses
  • The paper, as written, assumes high familiarity with remote sensing. For example, the Sentinel-2 bands are directly referred to with names (B02, B03, B04, and B08) with little to no explanation. (Line 120). Similar use of technical terms with little to no explanation is prevalent throughout the paper, making it a tough read for domain experts and practically unreadable for non-experts in remote sensing.

  • The paper lacks any significant error analysis. Are there any trends in the errors? Does the adapter fail in certain regions or for certain features? Does it work with other backbones? Would a more complex adapter yield more gains?

Questions for authors
  • What does PFU stand for?

  • Given the compute-hour vs. accuracy gain tradeoff, is it even practical to use this current method? Many applications might use a simpler backbone, which makes the compute difference more significant. Also, in the real world, practitioners would probably have weaker GPUs than the RTX PRO 6000 Blackwell GPU used here; would that not further increase the compute-hour gap?

  • How sensitive is the proposed method to added noise in the input bands? The NIR experiment shows significant degradation compared to 1x1 Conv 4-to-3. Does this also suggest that the proposed method could be more vulnerable to noise?

I would strongly encourage the authors to rewrite the paper in their own words rather than using an LLM. This will help create a coherent continuous narrative instead of an abrupt array of facts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.