acceptodds
Under review as a conference paper at ICLR 2027

Tessera v2: Scaling Pixel-wise Earth Foundation Models

14 views

Abstract

Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise teachers (0.5B, 1B, and 2B) and distil the largest into compact students for embeddings-as-data deployment. In aggregate, our 44-million-parameter distilled student outperforms every open and proprietary embedding product we test, several of them an order of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. We plan to release global 10 m annual embeddings covering 2017-2025 as version 2 of the Tessera foundation-model embeddings product.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Earth observation, scaling laws, self-supervised learning, geospatial foundation models, representation learning, knowledge distillation, Matryoshka representation learning, Barlow Twins, remote sensing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_E5SAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The key contributions behind the paper are a large-scale scaling law study of pixel-wise Sentinel-1/2 foundation models trained with Barlow Twins. They find that downstream task accuracy is weakly correlated with pre-training loss. Second, the authors release embeddings from their new TESSERAv2 model trained with MRL, showing that very small prefix sizes is still viable with >90% task accuracy.

Critical strengths

The large-scale sweep for their proposed family of models is well done, detailed and thorough. The experimental rigor is very good and extends to most of this paper. The appendix ablations cover a very large set of lingering questions and is well organized.

The largest strength of the work is its impressive engineering contributions and the released data product, which can help the ML for EO community. This final product is easy to use (embedding), and practical for a lot of use cases as highlighted in the authors' evaluations.

Critical weaknesses

I see practical value in the proposed model. But at the moment, I am not convinced that the paper provides any clearly novel contribution to the representation learning for EO community. My weaknesses are organized below by the two main contributions of the paper.

Scaling Laws

While not every scaling law study has to be broad and widely applicable like [1,2], I see the immediate beneficiaries of a (i) Sentinel-1/2 (ii) pixel-wise (ii) Barlow-twins (iv) embedding-as-a-product scaling law analysis to be the authors of the paper themselves. The novel conclusions from the scaling law analysis that would carry over to other EO models is unclear to me. My comments on the scaling law takeaways:

  1. SSL pre-training loss doesn't reliably predict downstream performance: This is widely known and well-established. [3,4], for example come to mind. However, I still see some value in quantifying this for EO. But why the mismatch occurs in this family of specific models or what makes this finding different from standard SSL paradigms remains unclear. The authors hypothesize about this in the appendix a little, but these experiments are not isolated and are not conclusive.

  2. The scaling law analysis helps to choose a model by fixing a downstream performance target, saving 2-5x compute: This makes sense to me although model selection with supervision on downstream tasks is not particularly novel to this paper. Furthermore, the model chosen with the highest downstream score cannot score worse on that same measure than any pretraining loss selected model. My understanding is that the authors both select the model with the lowest pretraining loss, and the model with the highest downstream (composite) score. This composite is defined on 15 tasks. Using this downstream task rule, the large teacher is trained and evaluated on 14 more held-out tasks. I believe an essential missing comparison is how the model selected by minimizing pretraining loss would do on these same 14 held-out tasks. This isn't reported. Comparing both selection rules on those held-out tasks, at matched pretraining budgets, would establish whether the advantage generalizes and provide stronger support for the claimed compute savings.

MRL embeddings and the proposed compact student models.

  1. To the best of my knowledge, I don't believe the proposed distillation to MRL embedding approach is novel. [5], for example is the multi-teacher variant of the exact distillation contribution proposed in this paper. [5] already trains nested student prefixes through separate MLP heads to match the full frozen teacher embeddings using a cosine loss. The proposed distillation approach is [5] reduced to a single teacher. I would encourage the authors to cite related work on nested distillation, and discuss (if any) new insights or new methods they bring to embedding distillation into prefixes.

Evaluation

  1. Evaluating the proposed models on more standard benchmarks such as BigEarthNet, So2Sat, EuroSAT is important, and not performed in the paper. I understand that the model is a pixel-level embedding model, in which case, direct comparisons to image/patch-based models may be murky.

  2. Data volume mismatch: The authors note that the v1 of the proposed model samples a fixed 40 timesteps while v2 retains all valid observations. Compared to this, Prithvi accepts only 3 acquisitions. This massive difference in input types and scales makes me question the claimed improvements. These could be due to data volume and not due to encoder architecture, proposed distillation approach, etc. While the authors acknowledge this, a matched-observation comparison could be useful.

Writing and figures

I generally find the figures to be dense and overcrowded, and would prefer if the authors modified their figures to better align with the core findings and contributions stated at the end of the introduction. Figure 2 contains most of the evidence, but the several curves, small insets, annotations, and callouts are very distracting. There is a fair bit of redundancy in the figures, and I encourage the authors to significantly trim down their figures into necessary componenets only. For example, figure 1(f) contains two radar plots and two bar charts in one overview panel. Even inside the radar plots which cover 15 tasks and several overlapping model curves, the embedding dimension of the same model are also added. Pruning would help future readers significantly. Figure 1c and 3b are both on the same topic. Figure 1c and 4d both cover the MRL embeddings again. The small encoder vs competing products figures are in figure 1f, 4c, and 4e. I appreciate the volume of results and thorough evaluation performed in the paper, but I believe that not all of this needs to be visually communicated multiple times in the main text.

Writing is verbose and can be simplified. Phrases like: "global surface state" (intro), "adaptivity along two axes" (intro), "projector held near its optimum as a training scaffold" (S3), "we therefore treat the frozen teacher... as a fixed representation distribution" (S4) can be modified to clear alternatives.

Evaluation

Given my concerns around the (i) lack of novelty surrounding the primary contributions made in the paper, (ii) discrepancies surrounding evaluation, and (iii) presentation issues with the figures, I recommend rejection at this time. I encourage the authors to clarify/push-back on my novelty concerns and other concerns whenever applicable. I am very open to increasing my score during the rebuttal stage.

[1] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

[2] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[3] Lourie, Nicholas, Michael Y. Hu, and Kyunghyun Cho. "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check." EMNLP (Findings). 2025.

[4] Liu, Hong, et al. "Same pre-training loss, better downstream: Implicit bias matters for language models." International Conference on Machine Learning. PMLR, 2023.

[5] Grashei, Christian, et al. "Pathryoshka: Compressing pathology foundation models via multi-teacher knowledge distillation with nested embeddings." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors
  1. I'm curious if the pretraining losses in the scaling law analysis are actually comparable across different projector widths. I believe Barlow twins' loss is computed on the projector outputs. Therefore, wouldn't the width changes change both the capacity of the network and the number of terms in the loss? I think the loss normalization/rescaling done in the paper doesn't automatically make different projector widths an apples-to-apples comparison of encoder quality.

  2. Does the model selected change depending on what downstream tasks are used? Could the authors also comment on the correlation between the tasks used for model selection and the tasks used for evaluation?

  3. I am still unclear as to who the beneficiaries of the smaller student model are (the main motivation behind distillation). In section 4, the authors claim

Most users cannot run a 2 B pixel-wise encoder over global Sentinel-1/2 even once, let alone recurrently.

But the authors also assert

We expect users to consume our embeddings as data, that is, as precomputed annual pixel embeddings with lightweight task heads, without running or fine-tuning a backbone.

While I agree with the former claim most users cannot run a 2B pixel-wise encoder globally on S1/S2, Wouldn't the computational cost of a smaller student model also be intractable to generate embeddings on a global scale, especially since both models require the same large-scale Sentinel-1/2 inputs? I see that the appendix mentions a 6% preparation overhead which is promising, but further details on what the preparation entails would be helpful. For example, I assume that the storage cost associated with downloading the inputs to generate a global embedding database to be identical (and intractable) regardless of the size of the model.

  1. I would also appreciate the authors' insights on the true computational costs of their approach. While the smaller student model could indeed reduce GPU inference costs, how does this compare to the storage + ingress + egress costs associated with storing global embeddings for the embedding-as-data framework, especially over time? Recent discussions in EO have framed this cost as the primary bottleneck, since even with a smaller model and embeddings, storage costs are not scalable long-term.
5 comments
AuthorsOpenReview ↗

Response to Reviewer E5SA (1/4)

Thank you for the careful review, and for your positive view of the rigour of the sweep and of the released embeddings as a product for the EO community. We number the weaknesses you presented from W1 to W7 in the order raised: who benefits from the scaling laws (W1), the loss/score mismatch (W2), downstream selection (W3), nested distillation (W4), standard benchmarks (W5), input mismatch (W6), and figures and writing (W7). Your questions are Q1 to Q4. This comment covers W1 to W4. Comment 2 covers the held-out test for W3 and W5 to W7, Comment 3 answers Q1 and Q2, and Comment 4 answers Q3 and Q4. Section numbers refer to the revised PDF, and the new analyses are in Appendix Z.

W1. We agree that the direct users of our exponents are groups training pixel-wise Sentinel-1/2 encoders with Barlow Twins or a closely related objective such as VICReg. EO models differ in training data, in patch- versus pixel-level modelling and in pretraining algorithm, and we do not claim that the exponents transfer to other settings. Appendix S.1 now states this scope. We would suggest one modest practice more widely: checking a loss-based fit against a downstream metric before using it to allocate compute.

W2. The mismatch is known, and the revision notes this now (Section 3.2). Section 6 and Appendix S.2 cite your [3] (first author corrected to Lourie) and [4], and no longer contrast EO with language models. Our point is about what the mismatch does to compute allocation. We do not know why the mismatch occurs in this model family. Appendix S.2 lists four candidate causes as untested hypotheses, and the introduction now calls the cloud and orbital sampling explanation a hypothesis. We tested one simpler explanation, pooling projector widths, and it does not account for the mismatch: using only the P = 4096 runs, the mean within-bucket Spearman between loss and score is -0.10 (Appendix Z.1).

W3. We do not claim downstream model selection as new. Our claim is that, in a sweep that separates encoder, projector and data at fixed compute, the two criteria give different allocation advice. Loss selection keeps the encoder roughly fixed as the budget grows (exponent 0.02), whereas score selection grows it (0.36, Appendix S.3). This difference holds at a single projector width too (your Q1, Comment 3). The downstream-optimal split is also data-heavy (data exponent 0.63, encoder 0.36, which sum to about 1 by construction, Section 3.3). The held-out comparison you asked for is at the start of Comment 2.

W4. Thank you for pointing us to Pathryoshka. It appeared on arXiv in November 2025 and at ECCV in September 2026, which overlaps with the period in which we developed and wrote this work. We should still have found it, and not citing it was an oversight on our part. Our Equation 2 has the same form as its CLS-token objective, with one teacher. Sections 2 and 5 now cite [5] and say that we do not claim the objective as new. What we add is evidence for pixel time series. We ran a control that trains the same student with the same teacher, data and schedule, using only the full-dimensional term, and then compresses it by first-k, random-k or PCA-k (Appendix Z.3). Nested prefixes beat the best of these by 0.015 at k = 16, 0.008 at k = 32 and 0.003 at k = 64, and are level with it at k = 128 (-0.001). On the 14 held-out tasks at k = 16, nested scores 0.601 against 0.588 for PCA-16. The nested objective matters mainly for small prefixes, and PCA recovers most of the difference. Two earlier results remain: the distilled 44M student beats the same encoder trained from scratch (0.593 vs. 0.527, Table 22), and, in a preliminary experiment, prefix losses added to Barlow Twins pretraining gave weaker small prefixes than distillation (Appendix W.2).

AuthorsOpenReview ↗

Response to Reviewer E5SA (2/4)

W3 (continued). You are right that the in-sample gap cannot be negative. We ran the comparison you asked for (Appendix Z.1, Table 30, Figure 2e). In each of the 9 compute buckets we took the score-selected and the loss-selected run and evaluated both on the 14 held-out tasks. The score-selected run is better in 8 of 9 buckets, though in one of them only by 0.002. In the remaining bucket the loss-selected run is ahead by 0.006. The mean gap is 0.033 (95% bucket bootstrap 0.018 to 0.047), or about 2.4x compute using the held-out score/compute slope, against 0.049 on the 15 selection tasks.

We keep the 2 to 5x of the submission as a summary of the in-sample 3.5x (bootstrap 2.6 to 5.5x) and the out-of-sample 2.4x. Section 3.2 now shows how the 3.5x is computed (Appendix G, Equations 25 to 27) and reports the held-out value. The checkpoints in this test come from the sweep, so the test covers the rule within the sweep range. We did not train a loss-selected model at the 2B budget.

W5. Thank you for anticipating the difficulty, and we agree with your reading. Our encoder takes a full year of Sentinel-1/2 observations per pixel. BigEarthNet, So2Sat and EuroSAT are single-date patch benchmarks designed for patch-based models, so a comparison on them would mostly measure the difference in input format rather than representation quality. We have therefore not added them. The closest time-series benchmarks we know of are the three Sentinel time-series tasks of GEO-Bench-2 (PASTIS-R, BioMassters, TreeSatAI-TS), and they are already in our held-out suite (Appendix M).

W6. The headline comparison is between products with different inputs, and Section 5.1 ("Input volume") now says so. To separate the encoder from input volume we matched the number of observations on one task, Austrian Crop, where the Tessera v1 paper already reports v1 with Sentinel-2 observations capped at 1 to 60 per pixel. We used that protocol for v2 and Prithvi (Appendix Z.4, Table 33, Figure 11). At every cap v2 is ahead of v1, by 0.085 at 5 observations and about 0.02 from 40 observations onwards (0.019 at 60), so with the same input v2 is at least as good as v1. With 10 or fewer observations both versions vary considerably across random draws of the dates, and v2 varies slightly less than v1 (Figure 11 shows the 5th to 95th percentile bands). Prithvi is a patch model, so we capped whole scenes and used only scenes with less than 50% cloud cover. There are at most 38 such scenes a year, so its curve stops at 30. Its score stays between 0.23 and 0.31 up to that point, and with three scenes, its usual input, it scores 0.262 against 0.402 for v2 with three observations. This test uses one task, and AlphaEarth cannot be rerun on a different input.

W7. Figure 1 no longer has the Matryoshka panel (old 1c), so the prefix schematic appears once (Figure 3b) and the prefix result once (Figure 4d). The later panels of Figure 1 move up one letter. In Figure 2 we removed the 2 to 5x box and added a panel e that shows, per bucket, how far the score-selected run is ahead of the loss-selected run on the selection tasks and on the held-out tasks. The allocation fits and the extrapolation check move to 2f and 2g. The captions of Figures 4b and 4d state the normalization. We kept the radar panel (old 1f, now Figure 1e) as a compact overview, and Figures 4c and 4e give the full comparison. We rewrote the four phrases you listed, and the Figure 1b inset no longer says that loss selection wastes compute.

AuthorsOpenReview ↗

Response to Reviewer E5SA (3/4)

Q1. Per entry, yes, and for encoder quality we also checked comparability directly, as described below. The divisor P+λBTP(P−1)P+\lambda_{\mathrm{BT}}P(P-1) is the loss when every entry of the cross-correlation matrix is off by one, so the normalized loss is a per-entry mean w⋅a+(1−w)⋅bw\cdot a+(1-w)\cdot b, where aa and bb are the mean squared on- and off-diagonal errors and w=1/(1+λBT(P−1))w=1/(1+\lambda_{\mathrm{BT}}(P-1)). This removes the extra entries of a wider projector, but ww falls from 4.7% to 1.2% between P = 4096 and 16384, so it is exact only when a≈ba\approx b. We therefore removed width from the comparison. The whole encoder sub-sweep and most other runs use P = 4096, so we repeated the analysis on those runs alone. The loss still does not rank runs (within-bucket Spearman -0.10 on average, range -0.27 to +0.15), the score gap is 0.058 (about 4.5x, so it does not shrink), and the loss-selected encoder exponent is -0.04 (95% -0.21 to 0.10) against 0.36 by score. At a single width the two criteria still give different allocations, so pooling widths does not create the finding (Appendix Z.1). Across widths, for the same encoder in the same bucket, the loss at another width is 0.78 to 1.23 times the P = 4096 loss with no consistent direction (new, Appendix S.3).

Q2. Somewhat. Leaving out one source dataset at a time changes the best run in 0 to 3 of 9 buckets (3 when LCMAP is dropped), and the encoder exponent stays between 0.33 and 0.40 (Appendix Z.2). On the relation between the suites, we fixed the 14 held-out tasks before any v2 model was scored on them, and chose them to differ from the 15 selection tasks (Table 29). Eight of them predict urban or socioeconomic quantities (population, GDP, night lights, road density, PM2.5, surface temperature, age structure, urban land use), which no selection task does. Four of the six vegetation tasks ask for dense segmentation, multi-label species or biomass, whereas the selection tasks label single points. The other two (CropHarvest Togo and China) are binary crop/non-crop, like the selection suite's Africa crop mask. The held-out labels come from other sources and mostly other countries (Germany, France, Austria, Finland, Togo, China and the CITYREP cities) and are scored with a CNN head on up to 100% of the labels, not k-NN or linear probes on at most 300 samples. Crop mapping is therefore the main overlap, with different label sources and mostly in different countries. The suites also rank systems differently: AlphaEarth is second on the 15 tasks and last of four on the 14.

Reviewer_bBukOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to allocate pretraining compute for Earth-observation models through hundreds of controlled training runs. It finds that downstream performance is a better selection criterion than pretraining loss and recommends scaling encoder capacity and data while keeping the projector roughly fixed. The resulting large teachers are distilled into compact students with Matryoshka embedding dimensions.

Critical strengths

The scaling study is a great contribution. It is well controlled. It separates encoder size, projector size, and data within a fixed model family across hundreds of runs. The within-compute-bucket analysis is interesting. It supports the claim that pretraining loss is a poor model-selection criterion.

The deployment result is also another great contribution. The student reaches outperforms the strongest baselines. I also apprecaited that Appendix N checks that the ranking survives removing CITYREP and changing the evaluation head. The same-size, from-scratch control also supports the value of the large teacher. The compact students and the variable embedding dimensions are useful for real inference and storage constraints. This makes the contribution useful beyond the scaling experiments.

Critical weaknesses

The evidence for a useful scaling trend is stronger than the evidence for compute-optimal allocation at production scale. The fitted sweep stops at 278M parameters. The larger teachers follow the proposed recipe, but there are no nearby off-recipe allocations at the same compute to test whether the predicted choice is actually optimal. Appendix G also derives the data exponent from the compute identity rather than estimating it independently. The bootstrap over fitted vertices does not capture all uncertainty from training noise and curve fitting. These limits matter for the precision and prescriptive strength of the allocation rule.

The Matryoshka objective is less established than the compression result. The paper compares distilled prefixes with naive prefix-BARLOW TWINS, but that changes both the representation-learning procedure and access to a large teacher. It does not isolate what nested prefix supervision adds to ordinary distillation. In Table 21, random coordinates from the distilled representation are already close to the learned prefixes.

Very minor: I wish the figures were more in style with ICLR format than Nature/PNAS/Science format. But you can ignore this criticism.

Questions for authors

At one large model compute budget, how do the predicted encoder/data allocation and nearby smaller encoder with more data or larger-encoder with less data choices compare? How much does the optimum move across downstream task subsets? How feasible is this question to answer without needing to train a lot more combinations of models?

How do the exponent intervals change across random seeds and the within bucket quadratic fits are resampled? It now these experiments are only available at the resulting vertices?

Does the compute advantage of downstream selection remain the same under independent checkpoint selection and evaluation samples?

I want to understand the value of the Matryoshka objective from the teacher's representation quality. For the same student and teacher, how do nested distillation, full-dimensional-only distillation, and post-hoc PCA or random-coordinate compression compare at 16, 32, and 64 dimensions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bBuk (1/2)

Thank you for the review, and for valuing both the controlled sweep and the deployment result, including the Appendix N checks and the same-size from-scratch control. We give short answers to your questions first. Q1 and Q2 are answered in this comment, and Q3 and Q4 in the next. We answer your weaknesses together with the questions: W1 (allocation at production scale, the data exponent, the bootstrap) under Q1 and Q2, W2 (what nested supervision adds) under Q4, and W3 (figure style) at the end of Comment 2. "New" marks analyses run after submission, and locations refer to the revised PDF.

  • Q1: we cannot test the allocation at teacher scale. Within the sweep the optimum is broad, it moves little across task subsets, and a much smaller design recovers the exponent.
  • Q2: with run resampling and seed noise the exponent intervals roughly double. The conclusions hold.
  • Q3: the advantage of downstream selection holds on held-out tasks but is smaller, about 2.4x against 3.5x in sample. We keep the 2-5x.
  • Q4: nested distillation outperforms post-hoc compression of a full-dimension-only student mainly at small k, by 0.015 at k=16 and 0.003 at k=64.

Q1. The direct test needs two off-recipe runs at the 2B teacher budget, a smaller encoder with more data and a larger one with less. This budget is about 6×10216\times10^{21} FLOPs by Equation 1, over 800x our largest bucket (Section 3.3), and we could not train these runs. The revision therefore uses "compute-optimal" only within the sweep, which covers encoders up to 278M (Section 3.3, Section 6, Appendix G), and calls Figure 2g (Figure 2f in the submission) an extrapolation check. The 0.5B/1B/2B teachers lie on the extended trend. This supports the trend but says nothing about whether their allocation is optimal.

Within the sweep the existing runs answer your question (new, Appendix Z.2). A law refit without the largest bucket predicts a 172M encoder for that bucket. The runs on its iso-FLOP curve (P=4096) score 0.507 (95M), 0.523 (151M), 0.526 (175M), 0.520 (194M) and 0.512 (278M). The spread is 0.019, about 2.4 seed s.d. (0.008), so the fit locates a broad region of sizes. Section 3.3 now says so.

We also left out one source dataset at a time (new, Appendix Z.2). The encoder exponent then ranges from 0.33 to 0.40 (0.36 on all tasks), and the best run changes in 0 to 3 of 9 buckets, 3 when LCMAP is dropped.

Our 395 runs are more than the exponent needs (new, Appendix Z.2). Keeping at most 10 encoder runs per bucket recovers the encoder exponent within its 95% interval in 84% of subsamples (35% with 5 runs), and fitting every other bucket gives 0.33 to 0.39. A refit within this model family, for example for a new data release, would therefore need far fewer runs. However, the teacher-scale test still needs new large runs.

On the data exponent, C∝NencDC\propto N_{\mathrm{enc}}D on the vertices fixes D∗D^{\ast} once Nenc∗N_{\mathrm{enc}}^{\ast} is set, so 0.63 is not an independent estimate. Section 3.3 now states this and names the encoder/data split as the informative quantity.

Q2. Yes, Table 20 resampled only the fitted vertices. We now resample runs within each bucket, add Gaussian noise with the measured seed s.d. (0.008) to every run, and refit the quadratics and power laws (new, Table 31, Appendix Z.2). Seed replicates exist for three configurations per bucket, so the noise is simulated for the other runs.

exponentTable 20 (vertices)runs + seed noise
encoder0.36 [0.31, 0.39]0.36 [0.28, 0.44]
data0.63 [0.61, 0.69]0.64 [0.56, 0.72]
projector0.011 [-0.02, 0.04]0.01 [-0.05, 0.07]

The vertex falls outside the sampled range in 4% of bucket refits. Section 3.3 now quotes all three wider intervals.

AuthorsOpenReview ↗

Response to Reviewer bBuk (2/2)

Q3. We scored the selected checkpoints on tasks never used in selection (new, Appendix Z.1, Table 30, Figure 2e). In each of the 9 buckets we evaluated the score-selected and the loss-selected checkpoint on the 14 held-out tasks. The score-selected run is better in 8 of 9 buckets (in bucket 1 only by 0.002), and in bucket 5 the loss-selected run is ahead by 0.006. The mean gap is 0.033 (95% bucket bootstrap 0.018 to 0.047), against 0.049 in sample. This gap corresponds to about 2.4x compute. The held-out tasks differ from the selection tasks in targets, regions, labels and evaluation head (Table 29), so this test is not a re-run of the same measure.

Projector width (raised by E5SA) does not explain the gap. With only the P=4096 runs, the loss still does not rank runs (mean within-bucket Spearman -0.10) and the in-sample gap is 0.058 (about 4.5x, against 0.049 with all widths). The loss-selected encoder exponent is -0.04 against 0.36 by score, so the two criteria give different allocations at a single width too (Appendix Z.1).

We keep the 2-5x as a summary of the in-sample 3.5x (bootstrap 2.6 to 5.5x) and the out-of-sample 2.4x. Sections 1, 3.2 and 6 now also report the out-of-sample value, and Section 3.2 shows how the in-sample 3.5x is computed: a mean gap of 0.0486 at a score-compute slope of 0.0886 per decade gives 100.0486/0.0886≈3.5×10^{0.0486/0.0886}\approx 3.5\times (Appendix G, Equations 25 to 27). This test holds out tasks but does not resample samples within a task, and we chose both suites (Section 6).

Q4. We trained the control requested (new, Appendix Z.3, Table 32, Figure 10). It uses the same 2B teacher and 44M student, the same data and schedule, and only the full-dimensional loss. From this control we read first-k, random-k and PCA-k (PCA fit on a separate sample). The table gives the 15-task composite.

knested first-kfull-only first-kfull-only random-kfull-only PCA-k
160.5460.4680.4760.531
320.5740.5210.5260.566
640.5880.5660.5680.585
1280.5930.5940.5940.594

The nested student leads the best post-hoc compression by 0.015, 0.008 and 0.003 at k=16, 32 and 64, and ties at k=128. On the 14 held-out tasks at k=16 it scores 0.601, against 0.588 (PCA-16) and 0.552 (first-16) for the control. PCA recovers most of the gap, so the benefit of the nested objective is modest and limited to small prefixes. The nested objective adds about 4% training time.

Your reading of Table 21 is correct. For the distilled model, FIRST and RANDOM differ by at most 0.018 at d=16, 32 and 64 and nearly tie at d=64 (0.543 vs. 0.545), and the distilled rows of Tables 21 and 23 come from a preliminary checkpoint. The revision removes the Section 5.1 claim that only distilled prefixes carry a genuine coordinate ordering, labels those rows as preliminary (Section 5.1, Appendices V and W.2), and describes the comparisons in Section 2 and Appendix R as comparisons with naive prefix-Barlow Twins only.

W3. We removed one panel from Figure 1, so its later panels move up one letter. In Figure 2 we removed the 2 to 5x box and added panel e (the held-out selection gap), so the old panels e and f become f and g. The Figure 4 captions now state the normalization.

Reviewer_5muwOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • Presented the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise BARLOW TWINS family, each evaluated on 15 diverse downstream tasks
  • Found that pretraining loss barely predicts downstream performance
  • Found that the encoder and the data should grow together as the training budget grows, while the projector stays fixed, which gives a simple rule for allocating compute
  • Trained distilled student models based on the scaled teacher model, and the student model outperforms every open and proprietary embedding product, also provided different dimension versions of the produced Matryoshka representations to serve flexible downstream deployment
  • Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models
Critical strengths
  • First very large-scale scaling study in the EO foundation model domain -> The sweep ran on 1,024 NVIDIA GH200 superchips, each pairing a Grace CPU with one H100 GPU (96 GB HBM3)
  • The paper gives a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distill into flexible student models
Critical weaknesses
  • The paper mentioned, "Across a 29-task suite (the 15-task ALPHAEARTH suite plus 14 further heldout datasets), the distilled students outperform every open and proprietary embedding product we compare, and even the smallest prefixes keep most of the full-dimensional score." It would make more sense to provide more detail on the 14 additional heldout datasets. If the datasets are chosen by the authors for the first time, how can we ensure fair selection?
  • When will the authors release the promised global 10m annual embeddings (2017-2025) as Tessera Embedding version 2?
  • The main part of the paper contains a lot of findings and conclusions, but readers need to refer back and forth to the appendix to find the experimental results and supporting evidence, which makes the paper more like a technical report. I understand it is because of page limits, but it would be better if the authors could reorganize the paper to make it more coherent
Questions for authors
  1. The notations in Equation 1 are not well explained; for example, what are N_{fus}, M, and N_{red}?
  2. "Moreover, we find, by fitting separate power laws through loss-selected and score-selected bucket peaks (Appendix S.3), that loss-based selection needs roughly 2–5× the compute to reach the same downstream score. EO scaling laws for this family must therefore be fit against downstream metrics rather than pretraining loss." -> The conclusion that "loss-based selection needs roughly 2–5× the compute to reach the same downstream score" is important to solve the claim made in the introduction, but the details and experiment results are only in appendix
  3. The numerical analysis in Section 3.3 does not have enough prior information, making it hard to understand
  4. Could you explain a bit about the "sinusoidal day-of-year positional encoding"?
  5. For Figure 4: the scores are relatively high, close to 1.00, does it mean the tasks are simple that can be completely solved by the EOFMs (both Tessera v2 and some baselines). Similarly, the performance of different-size models is also very close; is it necessary to provide larger-dimension embeddings? To support the necessity, please provide a case where a bigger size works substantially better
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5muw (1/3)

Thanks for the review, and for recognizing both the scale of the sweep and the resulting formula, which is to train large encoders, select by downstream score, and distill. We number your three weaknesses W1 to W3 in the order of your review and answer them here, and your questions Q1 to Q5 are in the two comments following. Section, figure and appendix numbers refer to the revised PDF, and we mark results computed after submission as "new".

W1. We chose them ourselves, so they are not a random sample of EO tasks, and Section 5.1 now acknowledges this. The 14 tasks come from TreeSatAI-TS, PASTIS-R, Austrian Crop, BioMassters, CropHarvest (Togo, China) and CITYREP (8 tasks). All come from published work, and Austrian Crop was introduced with Tessera v1 (Feng et al., 2026). We fixed the list of 14 tasks before any v2 model was scored on them, and none was used to select a v2 architecture, checkpoint or hyperparameter. The suite also differs from the 15 selection tasks in what is predicted, where, and how it is evaluated. Eight held-out tasks are urban or socioeconomic, a target no selection task has, and the labels come from other sources and mostly other countries (new, Table 29, Appendix Z.1). An independent benchmark made similar choices: the three Sentinel time-series tasks in GEO-Bench-2 (PASTIS-R, BioMassters, TreeSatAI-TS) are all in our suite (Section 5.1).

On the 14 tasks at 100% labels, v2-2B-L leads the best baseline by 0.033. It stays first in 98.9% of 5,000 task-bootstrap resamples (95% interval of the margin +0.003 to +0.057) and whenever one source dataset is left out, including CITYREP (0.644 vs. 0.592 for Tessera v1) (Section 5.1, Appendices N and Z.6). At 30% labels, the interval includes zero. This lead is in the composite score. On individual tasks, v2 beats every baseline on only 5 of the 14, and with the original CITYREP linear probe, OlmoEarth-L leads on that subset (0.574 vs. 0.562) while v2 still leads overall (0.597 vs. 0.573). On Austrian Crop, the margin of v2 is 0.006, inside a seed s.d. of about 0.03 (Table 13). None of this shows how v2 would rank on tasks we did not pick, and Section 5.1 states that.

W2. The global 10 m annual embeddings will be released under a permissive CC0 licence and hosted by a non-profit open-data service that lets anyone download them free of charge (name withheld for anonymity). The cloud resources for generating this full product are now in place, and generation begins in mid-November 2026. We expect 2017 to 2025 to take about two months, so we anticipate releasing the full v2 embeddings in January 2027. Before then, we can produce embeddings for specific regions on request, through a channel that we will link in the camera-ready version.

W3. Thank you for this criticism, which we think is fair. This reliance was largely due to the page limit. The revision may use a tenth page, and we used it to move into the main text the parts that the reader had to look up: the notation of Eq. 1 (Section 3.1), how the 3.5x is computed and the held-out test of the selection rule (Section 3.2, new Figure 2e), the vertex fit and the uncertainty of the exponents (Sections 3.1 and 3.3), the day-of-year encoding (Section 4), and the held-out robustness checks and an input-matched comparison (Section 5.1). We also removed one panel from Figure 1 and the 2 to 5x callout box from Figure 2. We hope the paper now reads less like a technical report. Analyses added during review are collected in Appendix Z.

AuthorsOpenReview ↗

Response to Reviewer 5muw (2/3)

Q1. NfusN_{\mathrm{fus}} is the parameter count of the fusion module, the attention layer that combines the MM per-modality summary tokens of a d-pixel. MM is the number of those tokens: 2 for the Sentinel-1/2 model (one Sentinel-2 token, one Sentinel-1 token), and 3 only in one architecture-sweep variant that encoded Sentinel-1 ascending and descending orbits separately (Appendix F), which we did not select. Because the fusion module receives one token per modality, its cost depends on MM and not on sequence length. NredN_{\mathrm{red}} is the parameter count of the reducer module, which maps the fused token to the embedding. Section 3.1 now defines all three immediately after Eq. 1 (full notation in Appendix B.1).

Q2. In each of the nine compute buckets, we compare the best-scoring run with the lowest-loss run. In sample, the mean gap is 0.0486 and the best score rises by 0.0886 per tenfold compute, so closing the gap requires 100.0486/0.0886≈3.5×10^{0.0486/0.0886}\approx 3.5\times the compute (bootstrap 2.6 to 5.5×). The submitted Section 3.2 attributed this multiple to the power-law fits, but those fits give only the exponents (0.36 score-selected, 0.02 loss-selected). The revised Section 3.2 now gives the calculation above, with the steps in Appendix G (Eqs. 25 to 27).

The score-selected run is chosen on the same 15 tasks, so the in-sample figure is optimistic. We therefore scored the 18 selected checkpoints (score- and loss-selected, nine buckets) on the 14 held-out tasks (new, Appendix Z.1, Table 30). The score-selected run is better in 8 of 9 buckets (in one only marginally) and worse in one. On average it is better by 0.033 (95% CI 0.018 to 0.047), against 0.049 in sample, or about 2.4× (Figure 2e). Projector width does not explain the in-sample gap either. With only the P = 4096 runs, the in-sample gap is 0.058 (about 4.5×, against 0.049 with all widths), and the loss-selected encoder still does not grow with compute (Appendix Z.1). We keep the 2-5× as a summary of the in-sample 3.5× and the out-of-sample 2.4×. Sections 1, 3.2 and 6 now also report the out-of-sample value, and Section 3.2 shows the in-sample calculation. This is still one sweep, and we chose both task suites.

AuthorsOpenReview ↗

Response to Reviewer 5muw (3/3)

Q3. At fixed compute, a larger model sees less data (C∝NDC\propto ND), so the score first rises and then falls with model size, and each budget has a best size. Section 3.3 tracks how this best size changes with budget. We group runs into nine buckets of roughly equal compute, fit a quadratic of score against log model size in each (encoder and projector separately), take the vertex as the best size, and fit a line through the vertices in log-log space. The encoder fit uses all nine buckets. The projector fit uses the seven mid-to-high buckets because its optimum is not resolved at low budgets.

The exponents are 0.36 for the encoder and 0.011 for the projector, which is indistinguishable from zero. Within the sweep (encoders up to 278M), a tenfold budget increase raises the best encoder size about 2.3× and the data about 4.3×, and the projector does not grow (Figure 2f). The data exponent (0.63) is derived, because compute fixes data once encoder size is set. Resampling runs and seed noise, in addition to vertices, roughly doubles the intervals (encoder 0.28 to 0.44) without changing the conclusion (new, Appendix Z.2). The optimum is flat: in the largest bucket, among on-curve P = 4096 runs, encoders from 95M to 278M score within 0.019 of each other. Sections 3.1 and 3.3 now give this short version and say that the fitted allocation holds within the swept range. The 0.5 to 2B teachers extrapolate it.

Q4. Satellite observations are irregular in time due to factors such as cloud cover, so the order of observations may not be informative of the time of year in all instances. We therefore encode the day of year tt (1 to 365) with the fixed sines and cosines of the original Transformer, PE2i(t)=sin⁡(t/100002i/h)\mathrm{PE}_{2i}(t)=\sin(t/10000^{2i/h}) and PE2i+1(t)=cos⁡(t/100002i/h)\mathrm{PE}_{2i+1}(t)=\cos(t/10000^{2i/h}) for token width hh, and add this vector to the observation's band embedding before the per-modality Transformer. Section 4 now gives the formula. In our architecture sweep, this encoding outperformed index order (0.699 vs. 0.507) and a learned position embedding (0.658) on a three-task probe, whose scores are not comparable with the composites in Q5 (Appendix F, Table 3).

Q5. Figure 4b divides each task score by the best score on that task, so the best model shows 1.00 by construction. The 15-task composite is chance-adjusted, so 1 would be perfect. The best student reaches 0.593 on this composite and the 2B teacher 0.608. Raw held-out scores are also well below 1 (mIoU 0.666 on PASTIS-R, 0.461 on Austrian Crop, Table 13), and harder tasks separate models clearly: 0.62 to 1.00 on Canada fine crop types in Figure 4b.

Figure 4d normalizes each student to its own d=128, so the curves meet at 1.00 by construction. In absolute terms, the d=128 composite runs from 0.560 for the 1M student to 0.593 for the 44M student (Table 26), a modest but non-negligible effect. The Figure 4 caption and Section 5.1 now state both normalizations and the absolute range.

Embedding dimension matters most on fine-grained tasks. From d=16 to d=128, the deployed 2B-L student goes from 0.75 to 0.97 on US tree genera (normalized, Figure 4b), while the Africa crop mask barely moves (0.98 to 0.99). Thus d=16 (about 22 TiB per global year) is adequate for coarse tasks, d=64 (87 TiB) recovers most of the loss, and d=128 (175 TiB) still improves results on the hardest tasks (0.91 to 0.97 on tree genera from d=64).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the Area Chair and all reviewers

Thank you to all three reviewers. We appreciate the care with which the paper has been reviewed, despite the extensive appendix. We believe that this has significantly improved the quality of the revised paper.

We note that the reviewers agreed on the paper's strong points. 5muw calls the scaling law sweep the first very large-scale scaling study for EO foundation models and the result "a concrete, empirically grounded recipe". For bBuk, the controlled study, which separates encoder, projector and data across hundreds of runs, is "a great contribution", and so is the deployment result. bBuk also points to the robustness checks in Appendix N and the same-size from-scratch control. E5SA finds the sweep "well done, detailed and thorough" and the experimental rigour "very good", and expects the released embeddings to help the ML for EO community.

Here are the aggregated reviewer concerns and the changes we made in response:

1. E5SA (W3) and bBuk (Q3) asked whether selecting models based on downstream performance continued to outperform selection based on pretraining loss, even on tasks that were not used in selection, and 5muw (Q2) asked how the 2 to 5x was obtained. We scored the score-selected and the loss-selected checkpoint of each of the nine compute buckets on the 14 held-out tasks. The score-selected run is better in 8 of 9 buckets, by 0.033 on average (95% CI 0.018 to 0.047) against 0.049 in sample. Expressed as compute, the gain is about 2.4x out of sample and 3.5x in sample. Section 3.2 and the new Figure 2e show both, and Section 3.2 is updated to give the calculation.
2. E5SA (Q1) doubted that pretraining losses are comparable across projector widths. The whole encoder sub-sweep and most other runs use one projector width. On those runs alone, the loss still does not rank runs, the selection gap does not shrink, and the two criteria still give different encoder allocations.
3. bBuk questioned the scope of the allocation rule. We now call it compute-optimal only within the swept range and treat the 0.5B to 2B teachers as an extrapolation check. Resampling runs and seed noise roughly doubles the exponent intervals, but the conclusions hold. The encoder exponent also stays between 0.33 and 0.40 when any one source dataset is left out.
4. E5SA and bBuk asked what nested distillation adds. We now cite Pathryoshka and do not claim the objective as new. In a new control, the same teacher and student are trained with only the full-dimensional loss and compressed afterwards. Nested training helps mainly for small prefixes. It beats the best post-hoc compression by +0.015 at k = 16 and is level with it at k = 128.
5. E5SA pointed out that the products we compare receive different amounts of input. On Austrian Crop, with the number of Sentinel-2 observations matched, v2 scores higher than v1 at every cap.
6. 5muw asked how the 14 held-out tasks were chosen and when the embeddings will be released, and E5SA asked about the actual costs of the product. The replies explain how the task list was fixed and how it differs from the selection tasks, report measured inference cost with list-price storage and egress, and give the release timeline.
7. On presentation (5muw, E5SA, bBuk), we used the tenth page to bring the notation of Eq. 1, the selection calculation, the vertex fit and the day-of-year encoding into the main text. As a result, a panel was removed from Figure 1, the callout box was removed from Figure 2, and the phrases E5SA quoted are rewritten.

In a few places the reviewers showed that we had claimed more than the evidence supports. We have narrowed our claims about the scope of the exponents and about what the prefix-ordering results show, and have also narrowed several cost statements. New analyses are collected in Appendix Z, with two smaller additions in Appendices S.3 and U, and the replies mark all of them as new.

We think the revision answers the main concerns and is easier to read than the submission. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.