acceptodds
Under review as a conference paper at ICLR 2027

Reuse or Relearn? A Spectral View of Earth Observation Foundation Models

Abstract

Foundation models are rarely used as generic, frozen feature extractors; instead, they are fine-tuned for the target downstream application. This practice is particularly prevalent in Earth observation (EO), and it raises a question that downstream accuracy alone cannot answer: does fine-tuning reuse the pretrained representation, or does it relearn a new one? We study this with spectral diagnostics that compare a model before and after adaptation, quantifying how well its dominant singular subspaces are preserved, how broadly the weight update is distributed, and how large it is. Using natural image models such as CLIP and DINO as a reference, we find that, under the evaluated fine-tuning settings, EO models undergo far larger, higher-rank updates and retain much less of their pretrained structure, so their downstream performance is often obtained with substantial changes to the pretrained weight structure. The diagnostics further provide insight into how cheaply a model can be adapted: where the pretrained subspaces are preserved, adapting a small fraction of the parameters can match full fine-tuning, and where they are not, it can fall behind. More broadly, foundation models, and EO foundation models in particular, should be assessed not only by benchmark accuracy, but also by how reusable their pretrained representation is.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Earth observation, foundation models, representation reuse, spectral analysis, transfer learning, parameter-efficient fine-tuning.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eZJUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes three weight-space diagnostics (relative weight change ρ, a top-k singular subspace overlap “SSI”, and an effective-rank measure “ERA” of delta W) to ask whether fine-tuning reuses or rewrites a pretrained encoder. They apply these to ~8 EO foundation models on 5 EO benchmarks and to CLIP/DINO on natural image tasks, with full FT, LoRA and SVF. My key takeway is: EO models show much lower SSI and larger, higher-rank updates than CV models. Benchmarks where SSI is high (MADOS) are where PEFT matches full FT, while low-SSI ones (TimeMatch) are where SVF falls well behind. There is also a random-init control used to screen out benchmarks with little pretraining gain.

Critical strengths

1.The authors spotted a good research question and the tool authors designed is cost-effective. We mostly judge EFM/RSFM by fine-tuned its downstream performance metrics, which can’t tell a reusable representation from a decent initialization. Anything that needs just two checkpoints could actually get used. The random-init screening of benchmarks is also a nice side contribution. 2. The experimental breadth is real, and the authors are upfront about their counterexamples (Tessera, MMEarth, etc.) and about high SSI possibly meaning under adaptation. The SVF comparison is a sensible functional test of the hypothesis.

Critical weaknesses
  1. I understand there's no 100% fair apple-to-apple comparsion of different models in an EO context, but i still think the EO vs CV comparison is confounded. EO runs use 80 epochs and a tiny linear head on heavily shifted tasks while CV runs use 10 epochs, standard heads, and near in-distribution tasks. App. B.5 shows the decoder alone moves SSI by ~0.3, and DINOv2 on PASTIS drops to SSI ~0.4, so CV models rotate too once the task is shifted. SSI with fixed k=32 also isn’t comparable across widths, and the highest-SSI EO models are exactly the narrow ones. The data supports “EO encoders rotate more under these settings”, not the broader claim about representation reuse if im not mistaken here.
  2. I don't see any activation-space measure reported, so weight-space preservation is never tied to feature preservation? The PEFT story holds for dataset averages but not per model (e.g. on MADOS the highest-SSI model loses with SVF and the lowest gains and CROMA on BraDD has SSI ~0.3 yet SVF beats FT), and no correlation is given.
Questions for authors
  1. if authors recompute the diagnostics at the best-val checkpoint actually used for the reported scores. will the tables change a lot?
  2. Im also curious to see under the same EO task, same decoder, same steps,etc. (say, DINOv3 vs an EO encoder) How much of the gap is left?
  3. Are SVF/LoRA LRs tuned on the EO side? also, are DOFA/PASTIS LoRA (8.9) and Prithvi-2/MADOS full FT (50.9 vs 62.1 LoRA) failed runs? what might be the reason for such failures?
  4. In appendix B.5, does SSI go up or down with upernet, per model?
  5. for CROMA/BraDD, full FT rotates early layers heavily, but SVF (which can’t rotate anything) does better. doesn’t that suggest the rotation is an optimization artifact rather than something the task actually needs?

if authors can address the above concerns, i will consider raising my score.

4 comments

Thank you for the constructive review. Main changes in the revision: (i) new Sec. 3.4 with EO and CV encoders on the same EO tasks and protocol; (ii) new Sec. 3.5 relating SSI to feature similarity (CKA); (iii) added width-proportional k (App. B.3) comparison and discussion; (iv) the PEFT claim is removed from the main results and conclusion; (v) Table 2 updated with tuned runs; (vi) checkpoint text corrected (App. F.4).

W1/Q2 (same task, decoder, steps). We fine-tuned all encoders on MADOS and FLAIR-2 with the identical PANGAEA protocol: same splits, augmentation, linear decoder, loss, optimizer, LR schedule, batch size, epochs, and RGB-only input (so MADOS differs from Table 2, where EO encoders use multispectral input).

Encoder      Pretrain   MADOS         FLAIR-2
                        SSI_32  mIoU  SSI_32  mIoU
--------------------------------------------------
DINOv3       natural    0.972   60.2  0.755   46.7
TerraMind    EO         0.898   50.6  0.540   38.9
Prithvi-2    EO         0.886   49.6  0.486   37.4
SSL4EO-MAE   EO         0.839   40.9  0.485   32.8
DOFA         EO         0.815   46.3  0.483   39.7
CROMA        EO         0.792   50.1  0.397   33.2
DINOv2       natural    0.613   40.9  0.121   31.8

DINOv3 has the highest SSI and the best accuracy on both datasets, although this setting favors the EO encoders, which are evaluated in their own pretraining domain. You are right that CV models rotate under task shift: DINOv2 drops from about 0.97 to 1.00 on natural-image tasks to the lowest SSI of all encoders. Task shift lowers SSI for any model, but by how much depends on the model: the high stability is specific to DINOv3, not to natural-image pretraining in general. Aurora shows the same effect (Table 7, App. B.1): SSI stays above 0.99 on the task closest to pretraining and drops to 0.87 to 0.95 on more distant ones. We therefore keep the in-domain analysis and add the matched experiment as its complement.

W1 (fixed k and width). We recomputed SSI with k = 32d/1024 (32/24/12 for ViT-L/B/S). On MADOS, SSI changes by -0.027 on average, and the encoder ordering is largely preserved on every dataset (rank correlation at least 0.90); the only large change is Presto on TimeMatch (-0.52; width 128, so k = 4). A fixed k is also meaningful in itself: the adaptation dimension is set by the task, not by model width (Aghajanyan et al., 2021; Hu et al., 2022). This rationale is now in Sec. 2.2 and the discussion in App. B.3. Width also does not determine SSI: the three ViT-B encoders share the same k, yet range from 0.42 to 0.91 on PASTIS, and CROMA, the widest, has the lowest SSI among EO encoders on every dataset.

References:

[1] Aghajanyan et al. "Intrinsic dimensionality explains the effectiveness of language model fine-tuning.", 2021.

[2] Hu et al. LoRA: Low-rank adaptation of large language models. ICLR, 2022.

W2 (feature space). We computed linear CKA between pretrained and fine-tuned patch tokens on held-out MADOS images, averaged over blocks. SSI_32 / CKA: CROMA 0.62 / 0.44, DOFA 0.72 / 0.51, Prithvi-2 0.79 / 0.59, TerraMind 0.90 / 0.57, SSL4EO-MAE 0.93 / 0.61. Pearson r = 0.92 (p = 0.03), Spearman 0.90 (p = 0.04), n = 5. Weight-space preservation is reflected in the features (Sec. 3.5, Fig. 3).

W2 (PEFT per model). We agree: the link holds for dataset averages, not per model. We removed the claim from the main results and conclusion and keep only a short discussion paragraph that states this limitation and leaves a per-model study (per-method tuning, more seeds) for future work.

Q1. This was an error in our text: scores and diagnostics both use the last checkpoint, since early stopping on small validation sets is noisy (fixed in App. F.4). Recomputed at best-val, mean over the five EO backbones (last minus best-val): MADOS SSI -0.05, ERA +0.06, rho +0.05; PASTIS -0.04, +0.04, +0.04. The direction is the same for every backbone and the ordering is preserved, so the tables would not change much.

Q3. All EO runs shared one untuned LR (1e-4), and both cases were failed runs. With a small per-method LR sweep (selected on validation, 3 seeds), Full / LoRA / SVF mIoU: Prithvi-2/MADOS 64.2+-1.0 / 63.2+-0.6 / 60.9+-2.0; DOFA/PASTIS 39.9+-0.5 / 39.0+-0.3 / 34.3+-0.2. The causes were opposite: 1e-4 was too large for full fine-tuning on the small MADOS (it overwrites pretrained weights early; 1e-5 works) and too small for LoRA and SVF on PASTIS (under-trained; 3e-4 and 1e-3 work). Table 2 is updated.

Q4. Up. On PASTIS, SSI increases with UPerNet for every encoder (App. B.6). There, UPerNet includes a temporal attention encoder and has 40M+ parameters, so the decoder absorbs much of the adaptation, especially for time-series benchmarks. This is why the main results use the 46K-parameter linear decoder: the adaptation has to happen in the encoder. We extended the discussion there.

Q5. We agree. A low SSI shows that full fine-tuning did rotate the weights, not that the task requires it. On BraDD, SVF cannot rotate anything and is still slightly better (57.4 vs. 54.1 mIoU), so the rotation is incidental. On TimeMatch, SVF falls far behind full fine-tuning (65.9 vs. 83.9 on average), so there it is needed.

We hope this addresses your concerns. If any remain, we would be happy to clarify further or run additional experiments during the discussion period.

Thank you to the authors for the detailed response. It has basically resolved all my concerns, so I'm raising my score to 3.

Thank you again for the constructive feedback, and we are glad our response addressed all your concerns.

Reviewer_iF8hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • Raised an interesting question: in the EOFM domain, does fine-tuning reuse the pretrained representation, or does it relearn a new one?
  • Using natural image models such as CLIP and DINO as a reference, EO models undergo far larger, higher-rank updates and retain much less of their pretrained structure under the evaluated fine-tuning settings, so their downstream performance is often obtained with substantial changes to the pretrained weight structure
  • showed that the evaluated EO models generally undergo larger updates with higher effective ranks and weaker subspace preservation, suggesting less mature representation reuse than in the CV reference
Critical strengths
  • The paper is well written and easy to follow
  • The paper raised an interesting question that downstream accuracy alone cannot answer: does fine-tuning reuse the pretrained representation, or does it relearn a new one?
  • The paper proposed a spectral diagnostic toolkit that characterizes adaptation along three complementary dimensions: The Spectral Stability Index (SSI) measures preservation of dominant pretrained subspaces, the Effective Rank of Adaptation (ERA) measures how many directions contribute to the update, and relative weight change measures its magnitude
Critical weaknesses
  • "Foundation models are rarely used as generic, frozen feature extractors; instead, they are fine-tuned for the target downstream application" -> Earth embeddings have been released since last year, and they can be used directly for the target downstream application; for example, AlphaEarth embeddings and TESSERA embeddings have been deployed for applications like cropland mapping, biomass estimation, etc.
  • "For the CV reference, we use three widely established vision backbones: CLIP (Radford et al., 2021), DINOv2 (Oquab et al., 2024), and DINOv3 (Sim´eoni et al., 2025). We evaluate them on standard classification benchmarks (Flowers-102, CIFAR-10, and Oxford-IIIT Pets) Nilsback & Zisserman (2008); Krizhevsky (2009); Parkhi et al. (2012) and on PASCAL VOC Everingham et al. (2010) for semantic segmentation." -> The comparison is not reasonable; the models and datasets are all different, not enough to derive the conclusion that CVFMs are more mature than EOFMs
  • "Together, these examples illustrate the complementary roles of the diagnostics: SSI measures whether dominant pretrained subspaces persist, whereas ERA measures how broadly the update is distributed. High subspace preservation can coexist with a high-dimensional update, as observed for DINOv3 on PASTIS." -> Here, why do you use DINOv3 (CVFMs) to evaluate on PASTIS (EO dataset)? A bit contradictory to the previous experiment designs
  • "Comparing DINOv2 with its stronger successor DINOv3 illustrates the effect of improved pretraining. DINOv3 consistently exhibits a clearer reuse signature, with higher SSI and lower ρ and ERA. On CIFAR-10, for instance, SSI rises from 0.986/0.987 to 0.998/0.999, ρ drops from 0.057/0.057 to 0.019/0.017, and ERA drops from 0.248/0.371 to 0.048/0.055, so the update is concentrated in several times fewer Directions." -> For CVFMs like DINOv2 and DINOv3, CIFAR-10 is too simple to draw the conclusion; also, you never know if they have been included the pretraining recipe
  • Table 4: Tessera was not mentioned before the table
  • Table 1: no downstream task performance
Questions for authors

See above "Critical Weaknesses"

2 comments

We thank the reviewer for the careful reading and the positive assessment. We are glad the reviewer found the paper well written and agrees that it asks an interesting question that downstream accuracy alone cannot answer: does fine-tuning reuse the pretrained representation or relearn a new one? We also appreciate the recognition of our spectral toolkit, which characterizes adaptation along three complementary dimensions. We address each concern below and have revised the paper (PDF) accordingly.

W1 (frozen embeddings). We agree that pre-computed Earth embeddings are an important and growing use case, and we added the following to the introduction: "The same applies to pre-computed Earth embeddings (Brown et al., 2025; Feng et al., 2026), which are often paired with sizeable task heads and can still be outperformed by end-to-end fine-tuning (Herzog et al., 2026)." Two points are worth noting. First, frozen embeddings are often paired with high-capacity heads: the TESSERA paper itself uses a U-Net head for patch-based segmentation and regression, with 30M parameters on PASTIS-R, i.e., on the order of the encoder itself. Much of the task-specific learning then happens after the frozen features, which is the attribution problem we raise in the introduction. Second, how generally frozen embeddings suffice is still open. OlmoEarth (Herzog et al., CVPR 2026) reports that end-to-end fine-tuning outperforms frozen AlphaEarth embeddings with a trained decoder on all five tasks compared (e.g., 82.2 vs. 66.0 accuracy on Nandi, 86.7 vs. 77.5 mIoU on Solar Farm). When it is used with a k-NN classifier instead of decoder fine-tuning, its performance is even worse, e.g., 55.6 on Nandi. Fine-tuning therefore remains common practice for the encoders we study, and our analysis covers an embedding model directly: TESSERA under fine-tuning (Table 4).

W2 (comparison across different models and datasets). The in-domain comparison remains our main analysis, and the revision now explains why (Sec. 3.4). It is the natural setting for a foundation model: each model is evaluated on data close to what it was pretrained for, and transfer is known to degrade with the distance between pretraining and target data (Matsoukas et al., 2022). The paper already shows this effect for Aurora (Table 7, App. B.1), where SSI is highest on the task closest to pretraining and drops on more distant ones. Evaluating each model in its own domain therefore shows how it adapts under the conditions it was built for.

We agree, however, that this comparison alone cannot separate model from benchmark effects. The new section therefore also adds a matched experiment (Sec. 3.4, Table 6). There, the five EO encoders and DINOv2/DINOv3 are fine-tuned on the same EO tasks (MADOS and FLAIR-2) with the same input modality (RGB), the same linear decoder, and the same PANGAEA training protocol. This setting favors the EO encoders, since the target imagery is closer to their pretraining data. Even so, DINOv3 preserves more of its pretrained weight subspaces than every EO encoder (SSI 0.972/0.755 vs. 0.898/0.540 for the best EO model) and outperforms the best EO model by 9.6 mIoU on MADOS and 7.0 on FLAIR-2. We also note that DINOv2 has the lowest SSI in this setting, so the result reflects the strength of pretraining and not natural-image pretraining as such, consistent with our DINOv2 vs. DINOv3 analysis.

The two experiments are complementary and show the same pattern. Our claim is limited to the models we evaluated and is not a general statement about all CV and EO foundation models.

Matsoukas, Christos, et al. "What makes transfer learning work for medical images: Feature reuse & other factors." arXiv preprint arXiv:2203.01825 (2022).

W3 (DINOv3 on PASTIS). Sec. 3.1 has a different purpose from the EO vs. CV comparison: it checks that the diagnostics respond in a meaningful way before we use them. It does so with two controlled comparisons: changing the pretrained model while the task is fixed, and changing the task while the model is fixed. For the first, we need two models that differ mainly in pretraining strength. DINOv2 and DINOv3 are such a pair (predecessor and successor from the same family), and no comparable pair exists among the EO encoders we evaluate. For the second, we fine-tune the same model on two EO tasks of different complexity (MADOS and PASTIS), since EO is the target domain of the paper. The same DINOv2 to DINOv3 trend also appears on the CV benchmarks (Table 5, e.g., SSI_attn 0.996 and 0.966 on Pets), so the observation does not depend on using EO data. We therefore see this section as consistent with the rest of the design: it grounds the interpretation of SSI and ERA, and the EO vs. CV comparison follows in Sec. 3.3 and 3.4. Note that Sec. 3.4 now also evaluates DINOv2/DINOv3 on EO tasks under a matched protocol. We clarified the purpose of Sec. 3.1 in the text.

W4 (CIFAR-10 too simple; possible pretraining overlap). The conclusion is not drawn from CIFAR-10 alone; the text cites it as one example ("On CIFAR-10, for instance"). The same trend holds on all four CV benchmarks in Table 5 (Flowers-102, CIFAR-10, Pets, and PASCAL VOC segmentation): from DINOv2 to DINOv3, ERA and relative weight change decrease on every dataset, and SSI increases or stays at its already very high level.

Regarding pretraining overlap: we agree this cannot be excluded for the CV benchmarks. DINO's data curation retrieves unlabeled web images similar to several standard benchmarks. The EO datasets are therefore the cleaner test, as they are far from natural-image pretraining data, and the same trend appears there (new Sec. 3.4, Table 6): from DINOv2 to DINOv3, SSI rises from 0.613 to 0.972 on MADOS and from 0.121 to 0.755 on FLAIR-2, and mIoU improves from 40.9 to 60.2 and from 31.8 to 46.7, respectively. The effect is therefore not explained by simple benchmarks or by overlap with pretraining data.

W5 (Tessera). Thank you. AnySat, Presto, and Tessera are now introduced in Sec. 3 together with the other encoders, with the selection rationale in App. E.

W6 (Table 1, downstream performance). Thank you. We added the downstream task performance to Table 1.

We hope this addresses your concerns. If any remain, we would be happy to clarify further or run additional experiments during the discussion period.

Reviewer_fprtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether fine-tuning Earth observation (EO) foundation models reuses pretrained weights or relearns them. Authors compare weights before and after full fine-tuning (FT) using three metrics: relative update norm ρ\rho (Frobenius norm of ΔW\Delta W over pretrained WW), Spectral Stability Index (SSI, top-32 singular subspace overlap), and Effective Rank of Adaptation (ERA, normalized effective rank of ΔW\Delta W). Across 5 EO benchmarks and 5-8 encoders, EO fine-tuning gives much lower SSI (0.543/0.548 for attention/MLP) and higher ρ\rho and ERA than 3 CV models on 4 tasks (SSI 0.986/0.990). Authors conclude EO models reuse features less. They also claim these metrics predict when LoRA or SVF can match full FT.

Critical strengths
  1. Sensible task screening and clean descriptive split. Screening tasks against random init (Fig. 2) is good practice. Under paper's setups, SSI and ρ\rho cleanly separate all 26 EO cells from all 12 CV cells (Tabs. 2-5).
  2. Candid appendix controls. Ablations cover LR sweeps, 10x epochs, decoders, and ConvNeXt (App. B). They show DINOv3 keeps SSI > 0.93 across a 50x LR range (Figs. 6, 8).
  3. Useful negative result on EO PEFT. SVF trails full FT on all 7 TimeMatch backbones (-18.0 mIoU mean, Tab. 4), and both PEFT methods drop on PASTIS (Tab. 2).
Critical weaknesses

Fine-tuning changes EO weights much more than CV weights under paper's two setups. However, experiments do not support the main claims that EO models are inherently less reusable or that spectral metrics predict when PEFT works.

W1. EO-vs-CV gap is confounded by task and training recipe differences (Fig. 1, Tabs. 2-5, 9, 11).

  1. Two arms use very different setups. EO and CV setups differ in tasks (dense segmentation / time series vs classification + VOC), heads (~46K linear head vs standard heads), and budgets (80 epochs at batch 8 vs 10 epochs at batch 64). They also use different LRs/wd (1e-4, wd 0.05 vs per-cell LRs, e.g., FT 2e-5, wd 1e-4; Tab. 11) and model widths (128-1024 vs ViT-B; Tab. 9). No model is tested in both setups under the same recipe.
  2. CV models look like EO models under the EO recipe. In Fig. 1, authors run CV models (DINOv2-L, DINOv3-L) on EO tasks (MADOS/PASTIS) with the EO recipe. DINOv2's SSI drops to 0.771 and 0.400 (at or below EO means 0.765/0.544, vs 0.966-0.998 on CV in Tab. 5). DINOv3's ERA jumps to 0.630/0.763 (vs 0.048-0.218 on CV). Also, switching the decoder to UPerNet shifts SSI by 0.29-0.33 (App. B.5).
  3. Hyperparameters shift metrics as much as the EO-CV gap. Sweeping LR shifts AnySat/Presto SSI from ~0.87-0.88 down to ~0.23-0.24 (Fig. 7). Training 10x longer raises VOC ERA from 0.105 to 0.314 (App. B.4). At LR 1e-5, AnySat is within 1.2 pts of its best accuracy (85.8 vs 87.0) and keeps high SSI (SSI ~0.87, ρ\rho ~0.16, ERA ~0.54; Fig. 7).

W2. SSI does not predict when PEFT matches full FT across backbones (Tabs. 2-4).

  1. Correlation with PEFT gap is zero or negative. Across all 26 EO cells, SSI has near-zero correlation with SVF-FT (r=+0.18r = +0.18, pp ~0.39) and LoRA-FT (r=+0.14r = +0.14, pp ~0.50). Inside each dataset, Spearman correlation between SSI and SVF-FT is negative in all 5 datasets (-0.70 MADOS, -0.60 PASTIS, -0.30 FLAIR-2, -0.40 BraDD, -0.18 TimeMatch; demeaned r=−0.42r = -0.42). For example, on BraDD, SVF beats FT most on the lowest-SSI model CROMA (SSI 0.287, +3.3). On TimeMatch, high-SSI Presto and Tessera suffer large SVF drops (-33.3, -15.3).
  2. MADOS result comes from one outlier, and PEFT LR is untuned. On MADOS, PEFT beats FT on average (58.5/57.7 vs 56.9) only because of one bad FT run (Prithvi-2: FT 50.9 vs LoRA 62.1 / SVF 61.0). Without Prithvi-2, FT (58.4) beats both LoRA (57.7) and SVF (56.9). Also, EO PEFT reuses FT's small LR (1e-4, 1 seed), whereas CV SVF needed a 50x larger LR (5e-3, Tab. 11). So EO SVF drops may simply come from a too-small LR.

W3. Spectral metrics lack null baselines and mostly track update norm ρ\rho (Tabs. 2-4). Within EO, SSI is strongly collinear with update norm ρ\rho (Spearman −0.84-0.84, n=26n = 26). Also, two random kk-subspaces in nn-dim space have expected overlap k/nk/n (0.25 for 128-wide Presto/Tessera vs 0.03 for 1024-wide CROMA). Paper reports no null baseline (such as random-init runs or norm-matched random ΔW\Delta W) to calibrate for width or spectral gaps.

W4. Overstated metric novelty and missing prior work. One-sided top-kk SSI is identical to the singular-subspace overlap ϕ(k,k)\phi(k,k) in LoRA [R1] (also used in [R6]). ERA is Roy-Vetterli effective rank, and the rank of ΔW\Delta W was already analyzed in [R5]. Paper also misses key literature on feature reuse vs random init under domain shift [R3], fine-tuning distortion [R2, R4], and EO MAE pretraining vs scratch [R7].

References

[R1] E. J. Hu, Y. Shen, P. Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.

[R2] A. Kumar, A. Raghunathan, R. Jones, et al. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022. arXiv:2202.10054.

[R3] M. Raghu, C. Zhang, J. Kleinberg, S. Bengio. Transfusion: Understanding Transfer Learning for Medical Imaging. NeurIPS 2019. arXiv:1902.07208.

[R4] A. Merchant, E. Rahimtoroghi, E. Pavlick, I. Tenney. What Happens To BERT Embeddings During Fine-tuning? BlackboxNLP 2020. arXiv:2004.14448.

[R5] D. Biderman, J. Portes, J. J. Gonzalez Ortiz, et al. LoRA Learns Less and Forgets Less. TMLR 2024. arXiv:2405.09673.

[R6] Y. Liu, B. Pang, O. Balabanov, et al. LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning. ICML 2025. arXiv:2506.00772.

[R7] J. Sosa, M. Aloulou, D. Rukhovich, et al. How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks? BMVC Workshops 2024. arXiv:2409.18536.

Questions for authors
  1. Matched recipe & iso-accuracy metrics (W1). What are SSI, ERA, and ρ\rho when CV ViT-Bs are fine-tuned on RGB EO tasks (e.g., FLAIR-2 RGB) under the exact EO recipe? Also, what are the metrics at the smallest LR that stays within ~1 pt of each model's best accuracy?
  2. PEFT LR sweep & per-cell correlation (W2). Does a higher LR (e.g., 1e-3 or 5e-3) close the SVF/LoRA gap on TimeMatch and PASTIS? How do you reconcile the negative within-dataset correlation between SSI and SVF-FT?
  3. Null baselines & controlling for ρ\rho (W3). What are SSI and ERA for the random-init runs and for norm-matched random ΔW\Delta W? How much EO-vs-CV SSI gap remains after controlling for ρ\rho?
4 comments

Thank you for the constructive review and the positive assessment. Your suggestions clearly improved the paper. We address each concern you raised below in two parts.

W1 / Q1 (EO vs. CV comparison is confounded; matched recipe). Thank you for this suggestion; the matched experiment complements our results. We keep the in-domain comparison as our main analysis, since it is the natural setting for a foundation model: transfer degrades with the distance between pretraining and target data (Matsoukas et al., 2022), as our Aurora results also show (Sec. 3.1). We agree, however, that it cannot separate model from recipe effects, so we added a matched experiment (new Sec. 3.4, Table 6) and limit our claims to the evaluated models and fine-tuning settings.

Matched recipe. All encoders are fine-tuned on the same EO tasks (MADOS, FLAIR-2) with RGB input, the same linear decoder, and the same PANGAEA protocol. DINOv3 has the highest SSI (0.972/0.755 against 0.898/0.540 for the best EO encoder) and the best accuracy (+9.6 and +7.0 mIoU), although this setting favors the EO encoders. You are right that CV models can look like EO models under the EO recipe: DINOv2 has the lowest SSI in Table 6. High preservation is thus specific to DINOv3, not to natural-image pretraining as such. On your two further points: ERA measures the breadth of the update and depends on the task, so it rises for DINOv3 on EO tasks as well (Sec. 3.1); and the decoder shift occurs on the time-series tasks, where UPerNet adds a temporal encoder and, with about 40M parameters, absorbs much of the downstream adaptation (App. B.6). This is why the main results and Table 6 use the same small linear decoder.

Hyperparameters and iso-accuracy. We agree that LR moves the diagnostics. A smaller LR can keep SSI high at some accuracy cost, as for AnySat, but this is not the typical case in our runs: for most models, the LR that preserves SSI costs clearly more accuracy. An iso-accuracy or area-under-LR-curve analysis is the rigorous answer, but a fair one needs an LR sweep per model and dataset, at least 10x our budget. We state this in the limitations and leave it for future work. Our comparison is still valid in practice: all encoders are ViTs under the same standard recipe (same task, decoder, LR, and schedule, strictly following the literature). Our targeted controls also show that the gap is not an LR artifact. DINOv3 is stable under both perturbations: SSI stays above 0.93 over almost two orders of magnitude in LR (Fig. 7) and at 0.99 after 10x more training steps (App. B.5). The EO encoders are not: their SSI falls from about 0.9 to below 0.25 across the LR sweep (Fig. 8). The LR therefore changes how much the EO encoders relearn, but no LR in the sweep makes DINOv3 relearn.

Reference:

Matsoukas, Christos, et al. "What makes transfer learning work for medical images: Feature reuse & other factors." arXiv preprint arXiv:2203.01825 (2022).

W2 / Q2 (SSI and PEFT). You are right on each point, and we removed this claim.

What we changed. We no longer claim that SSI indicates when PEFT matches full fine-tuning; this is removed from the abstract, the main results, and the conclusion. A short discussion paragraph remains, noting a pattern at the dataset level only (PEFT is mostly on par on the CV benchmarks, close on MADOS, clearly behind on TimeMatch). We agree that it does not hold per backbone and that the within-dataset correlation is negative. A proper analysis needs per-method tuning for every model and dataset, which is beyond our compute budget, so we leave it for future work.

MADOS outlier. Correct: the Prithvi-2 full FT result was a failed run under the shared LR. Tuned (3 seeds), full FT / LoRA / SVF reach 64.2 / 63.2 / 60.9 mIoU (Table 2 updated).

Does a higher LR close the gap? New per-method LR sweeps (LR selected on validation, 3 seeds):

CaseMethodTunedSelected LR
PASTIS, DOFA (mIoU)Full FT39.91e-4
LoRA39.03e-4
SVF34.31e-3 (5e-3: 34.6)
TimeMatch, Presto (acc. %)Full FT78.91e-3
LoRA71.23e-3
SVF38.05e-3 (1e-3: 41.0)

A higher LR closes the LoRA gap on PASTIS (0.9 points) but not the SVF gap (5.6 points). On TimeMatch it closes neither, because full fine-tuning gains the most from tuning. A too-small LR therefore explains part of the LoRA deficit, but not the SVF deficits.

Negative within-dataset correlation. We agree. SSI is measured on the full fine-tuning run and does not capture what limits a PEFT method. For SVF this is probably capacity issue: on Presto it trains only 1,024 singular values.

W3 / Q3 (null baselines, collinearity with ρ\rho). Thank you for this suggestion. The null baseline is a valuable sanity check for the diagnostics, so we ran it, added a new section (App. B.8, Table 9), and now discuss the relation between rho and SSI in the main text (Sec. 2.4).

Norm-matched random ΔW. We added App. B.8, "Null Baseline: Random Updates of the Same Norm". There, each ΔW is replaced by a Gaussian matrix of the same norm, and SSI and ERA are recomputed. On TimeMatch, a random update leaves the dominant subspaces far more intact than fine-tuning does: the null SSI averages 0.67/0.72 (attention/MLP), against 0.39/0.37 observed. The observed updates are also much more concentrated (ERA of 0.61/0.69 against 0.88/0.97 for the null). The only exception is Presto, due to its narrow width (see below). SSI and ERA are therefore not determined by the update norm alone, which is why we interpret the three diagnostics jointly (Sec. 2.4).

Random initialization. We also trained the same encoder from random weights instead of pretrained weights. Example: TerraMind on MADOS with the linear decoder, as in Table 2. Both runs change the weights by a similar amount (rho of 0.19 from pretrained and 0.21 from random weights), but the pretrained run keeps its dominant subspaces (SSI 0.90) while the random-init run does not (SSI 0.67). The pretrained run also reaches 65.2 mIoU against 46.5 from random weights. So at the same update size, SSI still separates the two.

Collinearity with rho. We agree that SSI and rho are correlated: a larger update rotates the subspaces more, so much of the SSI gap between EO and CV models goes together with the gap in rho. We do not see this as a confound. It is the same finding seen through two diagnostics: the EO models change their weights far more than the CV reference, and DINOv3 reaches the best accuracy while changing its weights the least. For this reason, we always report SSI together with rho and ERA (Sec. 2.4).

Width and chance overlap. We agree that a fixed k gives narrower models a higher chance level (k/d). We chose a fixed k because the number of directions needed for adaptation is set by the task, not by model width (Aghajanyan et al., 2021; Hu et al., 2022), so the same number of dominant directions is compared for every model; this rationale is now stated in Sec. 2.2. To control for width, new App. B.3 recomputes SSI with a width-proportional rank, k = 32d/1024, which gives every model the same chance level (0.03). On MADOS, SSI changes by -0.027 on average and the encoder ordering is largely preserved (rank correlation at least 0.90 on every dataset). The exception is Presto (-0.52; width 128, so k = 4), whose SSI at k = 32 is indeed inflated by its width, as you suspected. The EO vs. CV gap is not affected: models of the same width, and hence the same chance level, can have very different SSI. The CV backbones and three of the five main EO encoders are ViT-B, yet their SSI differs widely (Tables 2 and 5).

References:

[1] Aghajanyan et al. "Intrinsic dimensionality explains the effectiveness of language model fine-tuning.", 2021.

[2] Hu et al. LoRA: Low-rank adaptation of large language models. ICLR, 2022.

W4 (metric novelty and missing prior work). Thank you for these pointers. We agree and revised the paper accordingly.

Metrics. We no longer present the measures as new. Sec. 2.2 now states that each term of SSI is the normalized subspace similarity of Hu et al. (2022), also used by Liu et al. (2025). ERA is the effective rank of Roy & Vetterli (2007), normalized by min(m, n); Sec. 2.3 now relates it to the rank analysis of Biderman et al. (2024). The introduction, contributions, and conclusion now say that we combine established measures and use them as diagnostics of representation reuse.

Related work. All seven references are now cited. Sec. 5 has a new paragraph on feature reuse and fine-tuning dynamics (Raghu et al., 2019; Kumar et al., 2022; Merchant et al., 2020) and cites Sosa et al. (2024) among EO benchmark studies. Our results agree with this literature: as in Biderman et al., MLP updates have a higher effective rank than attention updates in all 38 model-dataset pairs (Sec. 3.3), and as in Sosa et al., pretraining gains are small on some EO benchmarks (Fig. 2).

We hope this addresses your concerns. If any remain, we would be happy to clarify further during the discussion period.

Thanks for the careful response. I will keep my rating. Removing the PEFT claim and relating the metrics to prior work resolves my major concern.

P.s. minor suggestion, pls keep the EO-vs-CV claim limited to the evaluated models and settings. DINOv2 shows the lowest SSI under the EO recipe.

Thank you again for the constructive feedback. We are glad our response resolved your major concern.

We fully agree with your suggestion. The revised version already restricts the EO-vs-CV claim to the evaluated models and settings, and for the final version, we will go through the wording very carefully and rephrase it further where needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and AC: Summary of changes in the revised manuscript

We thank all reviewers for the constructive feedback. Main changes in the revised PDF:

New experiments

  1. Matched cross-domain experiment, EO and CV encoders on the same EO tasks and protocol (new Sec. 3.4, Table 6): fprt, iF8h, eZJU.
  2. Feature-space validation with CKA (new Sec. 3.5, Fig. 3): eZJU.
  3. Null baseline with random updates of the same norm (new App. B.8, Table 9): fprt.
  4. Width-proportional rank for SSI (new App. B.3, Table 8): fprt, eZJU.
  5. Tuned runs replacing two failed runs (Table 2, App. F.4): fprt, eZJU.

Revised claims

  1. PEFT claim removed from abstract, main results, and conclusion: fprt, eZJU.

  2. EO vs. CV claim restricted to the evaluated models and fine-tuning settings (abstract, Sec. 4, conclusion): fprt, iF8h, eZJU.

  3. Metric definitions related to prior work, with our contribution framed as their combined use as diagnostics of representation reuse (Sec. 2.2, 2.3): fprt.

Text

  1. New related-work paragraph on feature reuse and fine-tuning dynamics (Sec. 5): fprt.

  2. Pre-computed Earth embeddings discussed in the introduction: iF8h.

  3. Sec. 3.1 purpose clarified, additional encoders introduced in Sec. 3, downstream performance added to Table 1: iF8h.

  4. Checkpoint description corrected (App. F.4), decoder discussion extended (App. B.6): eZJU.

Details are in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.