- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes three weight-space diagnostics (relative weight change ρ, a top-k singular subspace overlap “SSI”, and an effective-rank measure “ERA” of delta W) to ask whether fine-tuning reuses or rewrites a pretrained encoder. They apply these to ~8 EO foundation models on 5 EO benchmarks and to CLIP/DINO on natural image tasks, with full FT, LoRA and SVF. My key takeway is: EO models show much lower SSI and larger, higher-rank updates than CV models. Benchmarks where SSI is high (MADOS) are where PEFT matches full FT, while low-SSI ones (TimeMatch) are where SVF falls well behind. There is also a random-init control used to screen out benchmarks with little pretraining gain.
Critical strengths
1.The authors spotted a good research question and the tool authors designed is cost-effective. We mostly judge EFM/RSFM by fine-tuned its downstream performance metrics, which can’t tell a reusable representation from a decent initialization. Anything that needs just two checkpoints could actually get used. The random-init screening of benchmarks is also a nice side contribution. 2. The experimental breadth is real, and the authors are upfront about their counterexamples (Tessera, MMEarth, etc.) and about high SSI possibly meaning under adaptation. The SVF comparison is a sensible functional test of the hypothesis.
Critical weaknesses
- I understand there's no 100% fair apple-to-apple comparsion of different models in an EO context, but i still think the EO vs CV comparison is confounded. EO runs use 80 epochs and a tiny linear head on heavily shifted tasks while CV runs use 10 epochs, standard heads, and near in-distribution tasks. App. B.5 shows the decoder alone moves SSI by ~0.3, and DINOv2 on PASTIS drops to SSI ~0.4, so CV models rotate too once the task is shifted. SSI with fixed k=32 also isn’t comparable across widths, and the highest-SSI EO models are exactly the narrow ones. The data supports “EO encoders rotate more under these settings”, not the broader claim about representation reuse if im not mistaken here.
- I don't see any activation-space measure reported, so weight-space preservation is never tied to feature preservation? The PEFT story holds for dataset averages but not per model (e.g. on MADOS the highest-SSI model loses with SVF and the lowest gains and CROMA on BraDD has SSI ~0.3 yet SVF beats FT), and no correlation is given.
Questions for authors
- if authors recompute the diagnostics at the best-val checkpoint actually used for the reported scores. will the tables change a lot?
- Im also curious to see under the same EO task, same decoder, same steps,etc. (say, DINOv3 vs an EO encoder) How much of the gap is left?
- Are SVF/LoRA LRs tuned on the EO side? also, are DOFA/PASTIS LoRA (8.9) and Prithvi-2/MADOS full FT (50.9 vs 62.1 LoRA) failed runs? what might be the reason for such failures?
- In appendix B.5, does SSI go up or down with upernet, per model?
- for CROMA/BraDD, full FT rotates early layers heavily, but SVF (which can’t rotate anything) does better. doesn’t that suggest the rotation is an optimization artifact rather than something the task actually needs?
if authors can address the above concerns, i will consider raising my score.
4 comments
PART II
Q4. Up. On PASTIS, SSI increases with UPerNet for every encoder (App. B.6). There, UPerNet includes a temporal attention encoder and has 40M+ parameters, so the decoder absorbs much of the adaptation, especially for time-series benchmarks. This is why the main results use the 46K-parameter linear decoder: the adaptation has to happen in the encoder. We extended the discussion there.
Q5. We agree. A low SSI shows that full fine-tuning did rotate the weights, not that the task requires it. On BraDD, SVF cannot rotate anything and is still slightly better (57.4 vs. 54.1 mIoU), so the rotation is incidental. On TimeMatch, SVF falls far behind full fine-tuning (65.9 vs. 83.9 on average), so there it is needed.
We hope this addresses your concerns. If any remain, we would be happy to clarify further or run additional experiments during the discussion period.
Thank you to the authors for the detailed response. It has basically resolved all my concerns, so I'm raising my score to 3.
Thank you again for the constructive feedback, and we are glad our response addressed all your concerns.
PART I
Thank you for the constructive review. Main changes in the revision: (i) new Sec. 3.4 with EO and CV encoders on the same EO tasks and protocol; (ii) new Sec. 3.5 relating SSI to feature similarity (CKA); (iii) added width-proportional k (App. B.3) comparison and discussion; (iv) the PEFT claim is removed from the main results and conclusion; (v) Table 2 updated with tuned runs; (vi) checkpoint text corrected (App. F.4).
W1/Q2 (same task, decoder, steps). We fine-tuned all encoders on MADOS and FLAIR-2 with the identical PANGAEA protocol: same splits, augmentation, linear decoder, loss, optimizer, LR schedule, batch size, epochs, and RGB-only input (so MADOS differs from Table 2, where EO encoders use multispectral input).
DINOv3 has the highest SSI and the best accuracy on both datasets, although this setting favors the EO encoders, which are evaluated in their own pretraining domain. You are right that CV models rotate under task shift: DINOv2 drops from about 0.97 to 1.00 on natural-image tasks to the lowest SSI of all encoders. Task shift lowers SSI for any model, but by how much depends on the model: the high stability is specific to DINOv3, not to natural-image pretraining in general. Aurora shows the same effect (Table 7, App. B.1): SSI stays above 0.99 on the task closest to pretraining and drops to 0.87 to 0.95 on more distant ones. We therefore keep the in-domain analysis and add the matched experiment as its complement.
W1 (fixed k and width). We recomputed SSI with k = 32d/1024 (32/24/12 for ViT-L/B/S). On MADOS, SSI changes by -0.027 on average, and the encoder ordering is largely preserved on every dataset (rank correlation at least 0.90); the only large change is Presto on TimeMatch (-0.52; width 128, so k = 4). A fixed k is also meaningful in itself: the adaptation dimension is set by the task, not by model width (Aghajanyan et al., 2021; Hu et al., 2022). This rationale is now in Sec. 2.2 and the discussion in App. B.3. Width also does not determine SSI: the three ViT-B encoders share the same k, yet range from 0.42 to 0.91 on PASTIS, and CROMA, the widest, has the lowest SSI among EO encoders on every dataset.
References:
[1] Aghajanyan et al. "Intrinsic dimensionality explains the effectiveness of language model fine-tuning.", 2021.
[2] Hu et al. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
W2 (feature space). We computed linear CKA between pretrained and fine-tuned patch tokens on held-out MADOS images, averaged over blocks. SSI_32 / CKA: CROMA 0.62 / 0.44, DOFA 0.72 / 0.51, Prithvi-2 0.79 / 0.59, TerraMind 0.90 / 0.57, SSL4EO-MAE 0.93 / 0.61. Pearson r = 0.92 (p = 0.03), Spearman 0.90 (p = 0.04), n = 5. Weight-space preservation is reflected in the features (Sec. 3.5, Fig. 3).
W2 (PEFT per model). We agree: the link holds for dataset averages, not per model. We removed the claim from the main results and conclusion and keep only a short discussion paragraph that states this limitation and leaves a per-model study (per-method tuning, more seeds) for future work.
Q1. This was an error in our text: scores and diagnostics both use the last checkpoint, since early stopping on small validation sets is noisy (fixed in App. F.4). Recomputed at best-val, mean over the five EO backbones (last minus best-val): MADOS SSI -0.05, ERA +0.06, rho +0.05; PASTIS -0.04, +0.04, +0.04. The direction is the same for every backbone and the ordering is preserved, so the tables would not change much.
Q3. All EO runs shared one untuned LR (1e-4), and both cases were failed runs. With a small per-method LR sweep (selected on validation, 3 seeds), Full / LoRA / SVF mIoU: Prithvi-2/MADOS 64.2+-1.0 / 63.2+-0.6 / 60.9+-2.0; DOFA/PASTIS 39.9+-0.5 / 39.0+-0.3 / 34.3+-0.2. The causes were opposite: 1e-4 was too large for full fine-tuning on the small MADOS (it overwrites pretrained weights early; 1e-5 works) and too small for LoRA and SVF on PASTIS (under-trained; 3e-4 and 1e-3 work). Table 2 is updated.