acceptodds
Under review as a conference paper at ICLR 2027

Don't Fix What Isn't Broken: Calibrated Repair of Frozen Foundation Models under Product Shift

Abstract

Frozen foundation models are increasingly deployed in Earth observation, where the scene and its label stay the same but the product changes: satellites are replaced, processing baselines are updated, and resolution, radiometry, or band registration shift. We study how such product shifts break frozen pipelines and what it takes to repair them, across eight encoders, two Sentinel-2 benchmarks, 36 physically grounded degradations, and a real Level-2A-to-Level-1C product change. Only about 4% of the feature displacement reaches the readout, yet this small, concentrated component crosses decision margins and lowers micro-F1 under the real product change by 20 points on average. The information is not lost: a readout refit with labels recovers 92% of the loss. Thirteen test-time adaptation methods recover at most 29% of it, and every one that recovers accuracy also harms clean inputs; batch statistics even mistake class composition for shift, costing up to 37 points on clean streams. We introduce CALIPER, which repairs final embeddings only as far as evidence against a clean null supports, and decides for each image how far it moves. CALIPER recovers 46% of the loss while changing 7 of 15,000 clean predictions, passes a pre-registered test on unseen images, comes close to the strongest baseline on the real product change without its clean-input loss, and stays within 0.11 points on clean streams where batch-level adaptation fails.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Foundation models, Remote Sensing, Test Time Adaptation, Distribution Shift, Earth observation, Representation Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5nwZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why frozen foundation models lose performance when remote sensing products change in resolution, radiometry, or processing level, and how to repair these failures without retraining the model. CALIPER uses clean source data to estimate normal feature variation, decides whether a correction is warranted, and determines the correction amount for each image. Its central goal is to recover predictions affected by product shift while preserving those that were already correct.

Critical strengths
  • The problem and evaluation perspective are practically meaningful. Adaptation methods often emphasize improvements on degraded data, although deployed systems may receive both clean and affected images. I appreciate that this paper examines both corrected errors and newly introduced errors, alongside changes in class composition and arrival order. This makes the evaluation relevant to deployment concerns.
  • The experiments support the main design choices. CALIPER uses normal variation in clean data to control correction strength and adjusts individual images rather than applying the same correction to an entire batch. The results show useful recovery with little disruption to clean predictions, while the ablations support the importance of calibration and per-image correction. The real Sentinel-2 product-change experiment further strengthens the practical relevance.
Critical weaknesses
  • Some baseline configurations limit the conclusions drawn from the comparisons. The authors explicitly report that retained thresholds leave SAR, EATA, and TCA nearly inactive in this setting, while some methods use their original ResNet learning rates on Transformer encoders. Although these choices and implementation checks are transparently documented, I still find it difficult to separate limitations of the methods from a mismatch between their configurations and the task. The broader conclusions about existing adaptation methods should therefore be more carefully qualified.
  • The method still struggles to combine protection and recovery under class-correlated arrival. When similar classes arrive consecutively alongside mixed degradation conditions or clean images, CALIPER preserves clean-input performance but no longer improves overall accuracy. The authors acknowledge this limitation, but it directly affects the method’s usefulness for continuous observation. In particular, always using the class-conditional correction performs better in these settings, suggesting that the default strategy for combining correction types needs further explanation.
Questions for authors
  1. Would the main comparative conclusions hold with baseline settings appropriate to the class count and network architecture, without using target labels? I am particularly concerned about the nearly inactive baselines, whose current results may not reflect their recovery capabilities in this setting.
  2. Why does the default strategy underperform the always-class-conditional variant under class-correlated arrival? Does this arise from insufficient detection of the arrival pattern or from how the two corrections are combined? This distinction would affect my assessment of the method’s reliability on real data streams.
Reviewer_D4BcOpenReview ↗
Rating
4
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper empirically studies product shift in frozen Earth observation models. The authors define product shift as the condition when the scene and label associated with the scene stay the same but other acquisition characteristics like resolution, radiometric gain, band misregistration, sensor noise etc) change. With their experiments on multiple frozen encoders, the authors show: 1) Only about 4% of the feature displacement falls in readout’s space while logit change and flipping decisions tracks loss incurred much better; 2) Refitting the readout with labels recovers 92% of the loss, so the information is still in the embedding ; 3) Testing against 13 Test time adaptation methods only recover 29% of it. In response to these observations, the authors propose CALIPER, which whitens embedding, measures drift and aligns each image by an empirical Bayes posterior estimate of its shift. CALIPER can recover about 46% of loss on m-EuroSAT while changing only 7 of 15k changed predictions.

Critical strengths

The problem proposed by this paper is very interesting and relevant for deployment of ai models not only in EO but other domains with similar issues. The evaluation methodology is very clean: counting fixes and overturns separately rather than only the net accuracy change. As most TTA methods reduce overall performance. I would love to see more TTA paper that utilitize a similar evaluation methodology.

Tying the motivation to real data products available is clear with controlled swapping that isolate a specific drift in data. I appreciate the author’s effort to meticulously run official repo while documenting deviations. The paper first spends time on identifying what is wrong and naturally leads into CALIPER’s design which is neat. The method additionally requires no gradients or additional inference, or current batch’s statistics, which makes it practically usable. The ablation and sensitivity analysis is thorough and report negative results transparently. Overall, it was a fun read, the first job of my career had me dealing with this issue from a different perspective and it is refreshing to read a new solution to it.

Critical weaknesses

The main results of recovering 46% is for a synthetic dataset while on real L1C change, CALIPER gains 3.6 micro-F1 from a 20point drop while PEA (3.9).

Can you test natural shift with no product change? The clean null currently comes from the same distribution, can the clean target differ geographically? Can seasonality affect this? Do we consider this as product shift ?

An oracle is usually used to define the best possible result but the authors use oracle to define a translation reference that applies one average shift per degradation while CALIPER applies per-image translation. I feel this might not be fair to compare as an oracle.

The bayes posterior results in a shift probability for each image, I feel it would be better to use AUROC for separating degraded vs clean images broken down by severity would help the paper’s claim alot

I would recommend the author expand table 1 as it is confusing to compare across columns as the “best other” could come from different methods.

The authors do not provide clarification on how CALIPER’s design and settings like whitening shrinkage, 64 PCs, 256 k-means centroids, identity prior strength, NPMLE grid, exchangeability test, entropic regularization chosen. What data was used to design them?

All results in the paper deal with classification while segmentation and regression is not addressed. In EO, these are common tasks and the authors do not address them.

Questions for authors

How much of the real L1C loss does CALIPER recover in percentage terms, and what do you think explains the gap between the synthetic and real results?

Can you expand Table 1 to show the full row for each baseline instead of a single “best other” row?

The translation reference applies one average shift per degradation, while CALIPER applies a per-image translation plus an affine map, so I’m not sure it is fair to call it an oracle?

How were CALIPER’s design choices and settings decided?

Since the empirical Bayes posterior gives a shift probability per image, can you report AUROC for separating degraded vs. clean images in mixed streams, broken down by severity?

Reviewer_1cSxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CALIPER, which is a test-time adaptation (TTA) method specifically for remote-sensing product shift (e.g., Sentinel-2 Level-2A to Level-1C). First, the method computes embeddings of 2K samples and normalizes each dimensions over the population. It then computes class prototypes and measures how much embedding drift happens by chance. At test time, it first computes the prediction residual (prediction minus prototype) and uses it to predict whether or not a drift has taken place. It then predicts per-image shift and applies three corrections before making a prediction.

The method outperforms many SOTA TTA methods that were invented for other applications, e.g., corruptions to ImageNet-1K.

Critical strengths

none

Critical weaknesses

There are two major issues that result in my recommendation:

  1. It appears the authors do not tune hyperparameters for the baseline methods. Hyperparameters, e.g., learning rate, depend on the model initialization and the data—and these differ significantly between original papers and the experiments presented in this submission. I've used one of the baseline methods (SAR) many times and have found it to be quite sensitive to learning rate, thus I do not trust the reported baseline numbers. All baselines must be re-run, which is likely not feasible during the rebuttal period.
  2. I believe that the topic of product shift in remote sensing is too niche for ICLR. It is more suited to the ML4RS workshop or a remote-sensing journal.
Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.