acceptodds
Under review as a conference paper at ICLR 2027

VegBench: Quantifying the Ecological Resolution of Geospatial Models with Expert Field Surveys

11 views

Abstract

We introduce *VegBench*, the first geospatial benchmark to focus on how well models recover hierarchical categorization of ecosystems, from coarse groupings (i.e. forest vs. desert) down to fine-scale vegetation types useful for diverse ecological tasks like protecting threatened species and mitigating wildfire. *VegBench* standardizes decades of expert-collected vegetation classification and accuracy assessment field surveys across the state of California, spanning 42,393 surveys of 387 vegetation alliances. We utilize *VegBench* to compare modern geoFMs against remote sensing and environmental covariate baselines, and find that even the best current models can capture broad ecological structure but fall short as ecological resolution approaches the fine-grained distinctions required for conservation decision-making. *VegBench* offers the geospatial AI community a much needed testbed to assess whether and when models capture *ecologically-meaningful* variation.

Primary area:
datasets and benchmarks
Keywords:
Foundation Models, Ecology

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_d7iAOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper evaluates geoFMs (both time-series based and image-based), remote sensing approaches (based on high resolution NAIP images and lower resolution Sentinel-2 data, and Landsat), and feature baselines (based on location, soil type, elevation) against a large new benchmark for hierarchical vegetation classification, where the ground truth data was curated from over 42k expert field (ground-level) surveys across California, US, spanning a 45-year period. The ML-approaches are additionally compared to expert maps produced by the same vegetation survey programme through an expert-in-the-loop process based on high resolution areal images and environmental data. Not only does the paper propose a large and meticulously curated benchmark, and evaluate a large number of models, but additionally based on the detailed experiments and ablations, they are able to draw several well-supported conclusions and insights about geoFMs. Mainly, that good performance on broad ecological categories does not imply that fine-grained structure required for fine-grained classification is encoded (with the geoFM performance on fine-grained classes falling to the level of simple location-based baselines). A conclusion that time-aggregation and improved resolution help this task in complementary ways is also supported. By interpolation, they also conclude that both temporally aggregated models and high-resolution imagery models would need around ~1000 samples per vegetation alliance in order to reach the performance level of expert maps, pointing out how such a task is not feasible and instead proposing alternative directions for exploration and research in geoFMs. Many other smaller conclusions and recommendations are also supported by the performed analysis.

Critical strengths

I recommend accepting this paper because:

  • The newly proposed benchmark is based on a large amount of curated and and newly published data, and also covers a gap in the current suite of benchmarks available for geoFM models (covering fine-grained vegetation classification task, and showing how good performance on coarse vegetation classification is not an indicator of performance on fine-grained veg classification task). Additionally, all the efforts were made and required details shared to make the work easily reproducible and the benchmark accessible.
  • It provides strong evidence about information currently captured and not captured by geoFM models, offers concrete estimates of the effort/cost needed to close the performance gap between experts and geoFM models for fine-grained veg classification and instead makes recommendations for research directions which could make closing this performance gap feasible. Many additional insights are drawn from the exhaustive experiments and ablations and highlighted by the authors.
  • It is excellently written, well-presented and easy to follow. I particularly commend the appendix section containing all acronyms and terminology, making the manuscript which (must) heavily rely on ecological terminology easily accessible for CS/ML researchers regardless of their familiarity with the specific application area.
Critical weaknesses

The weak points noted here are relatively minor, especially in light of the excellent study and strong conclusions highlighted as strengths of the conducted study.

  • Three metrics are used for evaluation, accuracy, balanced accuracy and hierarchical accuracy. While all of these metrics make a lot of sense, there is a small worry that balanced accuracy as a metric is based on recall only, and therefore a good performance on minority classes could mask a large number of large positives, which would not contribute to the balanced accuracy score. Since the relative order of performance between different models and baselines, and other performance trends, are largely preserved between accuracy and balanced accuracy first, I have some confidence that this would hold for any additional selected metrics. Despite balanced accuracy being only an additional metric with the evaluation largely included in the appendix only, I would encourage the authors to consider either replacing it or additionally including other metrics (macro-F1 also gives all classes equal weight, takes both precision and recall into account, however does not take true negatives into account; and/or MCC (Matthews correlation coefficient) which takes the full confusion matrix into account, it's not perfectly balanced per class but still more robustness to class imbalance than plain accuracy)
  • The benchmark split is stratified in such a way that surveys and vegetation alliances from the same vegetation mapping project are assigned both into train, val and test sets. However, performance drop due to both spatial and temporal domain shift is well-documented. This effect may in fact disproportional affect smaller, isolated habitats and vegetation alliances, where the performance already suffers due to the lack of training data. While redesigning the proposed benchmark such that it also gives useful insights of model performance under domain shift would be a large task and out-of-scope for the current submission, it would be nice to see the authors address this as a limitation of the proposed benchmark. (or explain how or why this is not a weakness of the benchmark)
  • The related works provide a good coverage of three key relevant topics: geospatial benchmarks, ecological classification systems, ecological prediction with geospatial information. However, if space permits, the authors may also consider commenting on a body of work where ML is applied to ecological prediction from more fine-grained information. Some recommended papers to consider additionally are listed at the bottom of this comment ([1]-[4]). All of these papers support the argument already made in the discussion, that adding high-resolution imagery, ground level imagery, other ecological information, and falling back to coarser levels of hierarchy can enable ML models to perform better in fine-grained hierarchical vegetation classification task. Specifically, [1] (as well as already cited paper (Chytrý et al. 2020)) show that fine-grained and hierarchical predictions of EUNIS habitat types can be made from field vegetation plots. [2] performs hierarchical habitat classification from ground-level images (also collected from expert field surverys) according to the UKHab Classification (which, coincidentally, would be worth mentioning as another example of a hierarchical ecological classification system, developed in/for the UK); [3] provides further evidence that increasing spatial resolution increases performance of habitat mapping (however, while the habitat units are very fine-grained in this paper, they are limited to dune habitats and only a single hierarchy level is examined). [4] serves as additional evidence that very-high-resolution imagery can contain fine floristic and structural information required to characterise plant communities (however, the actual classification is into 9 tree species). I suggest that a brief coverage of non-geoFM-based ecological prediction models in your related work could better ground the paper, provide better context, and further support your discussion/conclusions.

[1] Leblanc, C., Bonnet, P., Servajean, M., Thuiller, W., Chytrý, M., Aćić, S., Argagnon, O., Biurrun, I., Bonari, G., Bruelheide, H., et al. (2025). Learning the syntax of plant assemblages. Nature Plants, 11, 2026–2040. DOI: 10.1038/s41477-025-02105-7

[2] Shi, H., Norton, L., Ridding, L., Rolph, S., August, T., Wood, C. M., Qie, L., Bosilj, P., & Brown, J. M. (2026). Habitat classification from ground-level imagery using deep neural networks. Ecological Informatics, 95, 103751. DOI: 10.1016/j.ecoinf.2026.103751.

[3] Pafumi, E., Angiolini, C., Bacaro, G., de Simone, L., Fanfarillo, E., Fiaschi, T., Rocchini, D., Thouverai, E., & Maccherini, S. (2026). Pixels and Patterns in the Sand: Decoding Coastal Dune Habitats With Multi-Resolution CNNs. Journal of Vegetation Science, 37, e70173. DOI: 10.1111/jvs.70173.

[4] Nepi, L., Quattrini, G., Pesaresi, S., Mancini, A., & Pierdicca, R. (2025). AI-based estimation of forest plant community composition from UAV imagery. Ecological Informatics, 90, 103199. DOI: 10.1016/j.ecoinf.2025.103199.

Questions for authors

These do not directly impact my evaluation of the paper, however most if not all are small, easy to correct, and would improve the paper if corrected before publication:

  • There are some minor misspellings ("To whit"), unclear figure labels (you refer to Fig1-a,b,c but there are no subfig a-b-c markings in Fig. 1), and incorrectly linked within-document references (especially to some complex composite figures in the appendix, or appendix sections)
  • Again, your summary of terms is excellent and very helpful. I would suggest adding the term "operational map" or "operational vegetation product" to it.
  • It is difficult to relate the English species names in the Results analysis to the Latin names embedded into Fig 5 (not helped by incorrect references to Fig 5 - i.e. Fig 5-aii is never referenced). Could both Latin and English names be included, either in the images, or in the text, or both?
  • What is the red dot displayed at the centre of each expert map shown in Fig 5?
  • Could you more explicitly state how expert maps, which are segmentation masks, are compared to all the classifier models used in your comparisons?
  • To consider for future work: one of the suggested procedures/recommendations for future research is falling back to a coarser hierarchy level when a fine-grained classification, such as vegetation alliance, can not be resolved. However, do you have any insight into whether predictions of all the tested models are consistent across the levels of the hierarchy?
2 comments
AuthorsOpenReview ↗

Overall response and response to 3 weaknesses

We thank the reviewer for the careful and constructive review, and we sincerely appreciate their enthusiasm for our work. We address the reviewer’s three stated weaknesses, plus smaller points below.

1. Reported accuracy metrics only focus on recall, not precision.

We thank the reviewer for highlighting this. We chose to originally report balanced accuracy—the mean of per-class recall—as our main metric because it directly addresses class imbalance. Every alliance counts equally regardless of how many surveys it has, so rare habitats are not overshadowed by performance on common classes. Per the reviewer’s suggestion, we now also report Macro-F1 in our per-model analyses in the Appendix Figure A1 (subpanel b). Across the entire vegetation hierarchy we see almost no relative difference between model accuracies calculated using macro-F1 versus balanced accuracy (Spearman ρ = 0.97–0.99, Appendix A.6.13). Both balanced accuracy and macro-F1 also report AlphaEarth to have the highest performance at alliance-level classification. Interestingly, the correspondence between macro-F1 and balanced accuracy holds even more strongly for the rarest alliances (Spearman ρ = 0.997 on the stratified and 1.000 on the accuracy assessment, Appendix A.6.13). Given this strong correspondence between balanced accuracy and macro-F1, we continue to report accuracy and balanced accuracy instead of macro-F1 in the rest of the manuscript, and state our motivation for using balanced accuracy more explicitly in Section 3.3 (lines 351-354).

2. Effect of spatial and temporal domain shift across projects.

We thank the reviewer for highlighting that some of our benchmark design choices may interact with our ability to evaluate performance for rare alliances under varying amounts of spatial and temporal distribution shift. Since some vegetation alliances are extremely spatially restricted (e.g. found in only one vegetation mapping project across the entire state), we chose to include surveys from all projects in both train, test, and val so as to maximize the number of alliances we could evaluate. This design choice also mirrors VegCAMP’s current expert mapping workflow, where vegetation is classified, mapped, and evaluated on a per-project basis [1]. This does potentially however limit our ability to test the domain generalization of vegetation alliances that span multiple projects, especially rare but geographically-broad alliances. We agree that this is an important capability to test in geoFMs, and we also agree with the reviewer that this is out of the scope of the current version of VegBench. As suggested, we have added this as a limitation in the main paper (lines 526-533), explaining how we prioritized evaluating within mapping projects rather than under large spatial distribution shift, and how this is an important area of future expansion for our benchmark.

3. Adding related work on non-geoFM ecological prediction.

Thank you for these references and their connection to our work! We have added an additional paragraph to the Related Work outlining these additional fine-grained ecological prediction tasks that use deep learning, and included the references the reviewer recommended (lines 166-178). As the reviewer notes, these works support the points raised in our Discussion section that high resolution imagery and ground level images can provide additional ecological context that have potential to improve fine grained vegetation classification.

[1] California Department of Fish and Wildlife. Survey of California Vegetation: Classification and Mapping Standards. 4 Nov. 2022, https://filelib.wildlife.ca.gov/Public/BDB/Vegetation/ClassificationAnalysisRFP/Attachment1_SCV%20Classification%20and%20Mapping%20Standards%2020221104.pdf.

AuthorsOpenReview ↗

Smaller points raised

Q1: We fixed the typos and the references to non-existent Fig. 1a-c panels, and re-checked all cross-references.

Q2. "Operational map" term. We added this to summary/glossary of terms (Appendix A.1)

Q3. Common vs. Latin names in Fig. 5. The caption now gives both common and scientific names, and every panel (including 5a-ii) is referenced in the text.

Q4. Red dot in Fig. 5. It marks the survey location, and the caption now explains this.

Q5. How expert maps are compared to classifiers. For each test survey, we take the alliance of the expert-map polygon containing the survey point and score it exactly like a classifier prediction, so maps and models are evaluated on the same surveys, labels, and metrics. This is clarified in Section 3.1 and Appendix Section A.2.5.

Q6. Consistency of predictions across hierarchy levels. We did test whether directly predicting a coarser level of the hierarchy versus crosswalking a finer-scale classification up the hierarchy was consistent (Appendix A.6.11, Figure A11), providing some insight to the reviewer’s question. Broadly speaking, rolling up an alliance prediction to a coarser level of the hierarchy such as macrogroup led to slightly worse performance on average than predicting the coarser category directly, but this performance drop closed as the categories became more coarse (e.g. formation). Overall though, the performance drops are small (-2-4% on average) and so we believe these results do support that hierarchy-aware prediction with fallback to coarser levels under uncertainty is a promising direction, which we now note in the Discussion (lines 514-515).

Reviewer_WL6jOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VEGBENCH, a geospatial benchmark for categorizing ecosystems from coarse to fine-grained levels of the vegetation hierarchy. It is built from 42,393 expert-collected field surveys across California, covering 387 vegetation alliances from 1978 to 2023. Fine-grained vegetation alliances are often not possible to distinguish from remote sensing because of spectral similarity, and the paper assesses whether geoFMs can perform this kind of classification. It evaluates a range of geoFMs and baselines and finds that they fail on the fine-grained distinctions required for conservation decision-making.

Critical strengths

A real problem, well motivated. The motivation for fine-grained ecosystem categorization is well explained, and the problem is a challenging real-world one. The benchmark provides a testbed for whether models capture ecologically meaningful variation, rather than the coarse land-cover classes most benchmarks focus on.

The data have value. The vegetation classifications are expert-collected field surveys rather than remote-sensing-derived maps, and they span the full vegetation hierarchy. Labels of this kind are rare in geospatial benchmarking.

Critical weaknesses

The dataset is small and geographically limited.

  1. Small for the task: 42,393 surveys across 387 alliances is a small dataset for fine-grained classification.
  2. Long tail: many alliances have only a handful of training samples. The paper’s own analysis (Fig. 4b) shows that alliances with fewer than about 10 training examples are close to zero accuracy for all models, so a large part of the hierarchy cannot be meaningfully evaluated. Tiny modern subset: only 3,996 training surveys were collected in 2017 or later, so the subset that can be properly aligned with modern imagery is even smaller.
  3. Single region: the data cover only California, so conclusions about geoFM capability may not transfer to other regions or vegetation types (based on similar existing studies). Suggested fix: extending the benchmark across at least a few states would make the dataset substantially more valuable and allow the evaluation of geographic generalization.

Temporal misalignment

  1. The surveys were collected from 1978 to 2023, but the imagery comes from 2018. This mismatch should be discussed and measured, since it makes the baseline results vulnerable. Major events such as the 2021–22 drought may have changed vegetation types, so labels from earlier surveys may no longer describe the 2018 imagery.
  2. Most of the data were collected before 2017, while AlphaEarth embeddings are only available from 2017. The comparison across models should be performed on the post-2017 subset.
  3. It is unclear why imagery was not taken from the survey year.

Spatial leakage

  1. The data split only uses a 3.36 km separation, a very small gap relative to the Landsat tiles (6.72 km). Previous studies have shown that spatial splitting causes OOD shifts even for domain-specific methods (e.g., IrriSight, NeurIPS 2025; AG-STELLA, IJCAI 2026), so the reported performance may be optimistic.
  2. Unclear setup: it is not clear how data are fed to the baseline models:
  3. What is the size/dimension of the samples?
  4. Is the task semantic segmentation or classification only?
  5. Metrics: the evaluation should also report F1 score, as the dataset is imbalanced.
  6. Prithvi: Prithvi is also a time-series model and has a checkpoint for crop classification. It should be evaluated in that capacity rather than as a single-image model. Taken together, these issues make it difficult to know whether the reported failures reflect geoFM limitations or the evaluation setup. Therefore, the baseline evaluation seems flawed.

Additional issues:

  1. Related datasets: other datasets are available (WRLU, Washington State, Colorado), and these also include crop classes that can be categorized. Their relation to VEGBENCH should be discussed.
  2. Taxonomy: the clarification of the taxonomy should appear in the main paper.
  3. Decision-making: the key finding refers to conservation decision-making, but the specific type of decision is not described.
  4. Overall the paper's experimentation setup and organization is hard to read and follow.
Questions for authors
  • Most of the surveys are from before 2017, but all the imagery is from 2018. This gap affects the results quite a bit, especially given events like the recent droughts. Have you checked how performance changes with survey age? It would also help to see all models compared only on the post-2017 data, since AlphaEarth isn’t available before that.
  • The Landsat tiles are 6.72 km wide, so train and test inputs could overlap. What happens if you use a stricter split, say holding out whole regions or projects?
  • Experimentation setup clarification: Which bands does each model actually use, and what’s the input size? And is this purely classification, or is segmentation involved anywhere?
  • A lot of alliances seem to have very few training samples. How many of the 387 can actually be evaluated in a meaningful way? I’d also be curious whether the conclusions still hold with F1, or if you only keep alliances with enough samples.
  • Prithvi is a time-series model and even has a crop-classification checkpoint, but it looks like you ran it on a single timepoint. Was there a reason for that?
  • Do you plan to extend VEGBENCH beyond California? Even adding a few more states would make the benchmark much more convincing and show whether the findings generalize.
6 comments
AuthorsOpenReview ↗

Overall response and response to weakness 1

We thank the reviewer for their thoughtful review. In response to the reviewer’s concerns, we have clarified the size and spatial relevance of our benchmark, how we handle temporal misalignment and potential spatial leakage, and added a more fair comparison for Prithvi V2, along with line by line clarifications. We hope that these changes address the reviewers' concerns and that they consider raising their score. We address individual questions and concerns in detail below.

Weakness 1: Dataset

Points 1. and 3: Size & Single Region

The number of surveys and geographic extent of VegBench—23,546/5,003/14,919 train/val/test samples across California— is comparable to or greater than many existing related geospatial benchmark tasks, which frequently focus on individual regions or countries. GEO-Bench [1] and GEO-Bench-2 [3] have no classification task with more than 20,000 training samples, including TreeSatAI (20,000/4,000/4,000 train/val/test samples from Northern Germany) and ForestNet (6,464/989/989 train/val/test samples from Indonesia). PANGAEA [2] includes BioMassters (7820/869/2773 train/val/test samples from Finland). Furthermore, California encompasses about 424,000 km2, exceeding the area of Germany, and encompasses diverse ecosystems spanning coastal wetlands, deserts, grasslands, and montane forests, making it a useful and valuable regional testbed for fine-grained vegetation classification. We clarify this in the main text now (lines 211-217).

Point 2. Classes and Long Tail

The long tail and large number of classes is a fundamental property of ecological systems [4]. Long-tailed distributions are also common in other ecological datasets used for foundation model evaluation. For example, iNaturalist 2018 [5] contains 437,513 training samples with 8,142 classes and exhibits a long tail distribution (see Figure 2 of their original manuscript), as do Auto Arborist [6] and Pl@ntNet-300k [7], which highlight and include explicit analyses for the long tail. Additionally, rare vegetation types are an important conservation target of the state of California [8] and VegCAMP’s accuracy assessments deliberately sample every mapped type, including rare alliances. Including these rare classes allows VegBench to capture the complexity of the real-world task of fine-scaled vegetation classification.

We believe an important role of benchmarks is to identify challenging real-world tasks and reveal new capability frontiers that can drive progress in ML research. We see our benchmark as an opportunity to understand the current and future capabilities of geospatial ML models across the vegetation hierarchy. It enables us to both 1) define opportunities for improvement and 2) determine which levels of habitat granularity are reliably predicted to inform downstream use.

Reliably predicting alliances with fewer than 10 training surveys is certainly a challenging task. However, the ability to generalize with limited labeled data is often highlighted as a key strength of geoFMs, and thus is important to try to assess in this context. Currently, expert maps reach 66% balanced accuracy on alliances in our benchmark with fewer than 10 training surveys, while the median geospatial model scores below 1% (Fig. 4b). This highlights a key gap between current fully-automated geospatial modeling capabilities and expert mapping, and demonstrates a significant opportunity for improvement.

[1] Lacoste, Alexandre, et al. "Geo-bench: Toward foundation models for earth monitoring." Advances in Neural Information Processing Systems 36 (2023).

[2] Marsocci, Valerio, et al. "Pangaea: A global and inclusive benchmark for geospatial foundation models." arXiv preprint arXiv:2412.04204 (2024).

[3] Simumba, Naomi, et al. "Geo-bench-2: From performance to capability, rethinking evaluation in geospatial ai." arXiv preprint arXiv:2511.15658 (2025).

[4] Enquist, Brian J., et al. "The commonness of rarity: Global and future distribution of rarity across land plants." Science advances 5.11 (2019).

[5] Van Horn, Grant, et al. "The inaturalist species classification and detection dataset." 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018.

[6] Beery, Sara, et al. "The auto arborist dataset: a large-scale benchmark for multiview urban forest monitoring under domain shift." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.

[7] Garcin, Camille, et al. "Pl@ ntNet-300K: a plant image dataset with high label ambiguity and a long-tailed distribution." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

[8] California Department of Fish and Wildlife. California Sensitive Natural Communities. Vegetation Classification and Mapping Program (VegCAMP), 12 June 2026, https://nrm.dfg.ca.gov/FileHandler.ashx?DocumentID=153609

AuthorsOpenReview ↗

Response to Weakness 2: Temporal misalignment, and response to Question 1

Weakness 2: temporal misalignment

Points 1. – 3. Mismatch between survey dates and input imagery, comparisons on the 2017+ subset, and why the imagery was not taken from the survey year, and response to Question 1.

We discussed our choices to use fixed 2018 imagery and all the vegetation survey data in section A.3.1, and extensively ablated the tradeoffs between using temporally fixed and co-registered imagery in Appendix Sections A.6.5, A.6.6, and Figures A5 and A6. To briefly summarize these experiments, many of the VegBench surveys do not have imagery available from their survey year (e.g. all surveys taken before 2017 versus Sentinel-2 imagery). As a result, we either had to discard those surveys or not temporally co-register surveys and imagery. To explore this tradeoff, we evaluated all geospatial models under three temporal alignment settings: 2018 fixed date input co-registered to all surveys, temporally co-registered inputs for all surveys taken from 2017 onward, and 2018 fixed date inputs for all surveys taken from 2017 onwards. In short, using all surveys co-registered with fixed 2018 imagery generally produced the best results across geospatial models—including AlphaEarth—(filled circle, Figure A5) compared to using just co-registered 2017+ surveys and imagery. As a result, we chose to use this fixed co-registration strategy for all subsequent analyses. We discuss these choices further in our response to reviewer vpAQ.

With respect to the risk of changed vegetation types over time: The most abundant and most dominant species that typically define vegetation alliances and associations (e.g. coast redwoods) are also the last species to be lost when an ecosystem changes due to environmental stressors. Indeed, these dominant species can often persist for decades beyond when local environmental conditions would normally support that type of ecosystem [9], such as the so-called “zombie forests” of the foothills of the Sierra Nevada [10]. Therefore, while temporal mismatches may introduce some label noise, we expect vegetation classifications to remain relatively stable across many of the survey-imagery time gaps. We now discuss these considerations and limitations explicitly in Section A.3.1.

[9] Wilfahrt, P. A., Seabloom, E. W., Bakker, J. D., Biederman, L., Bugalho, M. N., Cadotte, M. W., ... & Borer, E. T. (2023). Nothing lasts forever: Dominant species decline under rapid environmental change in global grasslands. Journal of Ecology, 111(11), 2472-2482.

[10] Hill, A. P., Nolan, C. J., Hemes, K. S., Cambron, T. W., & Field, C. B. (2023). Low-elevation conifers in California’s Sierra Nevada are out of equilibrium with climate. PNAS nexus, 2(2), pgad004.

AuthorsOpenReview ↗

Response to Weakness 3: Spatial leakage, and response to Question 2

Weakness 3: spatial leakage

Point 1. Buffer size, OOD shifts, and optimistic performance, and response to Question 2

In our spatially stratified test partition, we aimed to avoid a survey point from one partition (train/val/stratified test) appearing within the spatial extent covered by the input data from one of the other partitions to avoid spatial leakage between train and test. 224 x 224 pixel Landsat tiles are the input with the largest spatial extent of all the geospatial models we evaluated, so we increase our spatial buffer from 3.8 km to 4.752 km to ensure no test or validation survey would fall inside of a Landsat train tile. We do note that we see very similar performance across models whether or not we use a strict train/test spatial partition in Figure A7 (the accuracy assessment partition does not enforce a spatial train/test split, and test surveys can fall as close as 50 m to a train survey in this partition). For a detailed description of how the buffer is implemented see A.3.2. All figures and tables in the updated version of the manuscript that report results using the spatially stratified partition now reflect this new buffer size.

As also noted in our response to reviewer vpAQ, for fine-scale vegetation classification, there is an unavoidable tradeoff between using a larger spatial buffer and the number of alliances that are shared between partitions. While a larger spatial buffer (or even a different strategy like spatial blocking or having a geographically held out test project) would potentially better measure geographic generalization, this would by nature reduce the number of alliances that would be possible to evaluate, favoring the most common alliances. The 4.752km buffer is our best attempt to balance this alliance coverage with spatial separation between partitions.

Points 2. and 3. How data are fed to baseline models, and size dimension of inputs

The models evaluated in VegBench have different input types and dimensions, as designed by their original authors, which we follow in our benchmark. We describe the input types and dimensions of each model in detail in Appendix A.5. Most of the image-based models use 224 x 224 pixel center crops (approximately 135 x 135 m for NAIP, 2.24 x 2.24 km for Sentinel-2, and 6.72 x 6.72 km for Landsat). Models with different input requirements are processed according to their respective specifications, as detailed in Appendix A.5.1. We additionally describe our procedures for classifier training/fine-tuning for our benchmark in Appendix A.6.8 and the training and input preprocessing details for our supervised ResNet-50 and ViT-L/16 baselines in Appendix A.5.4.

Reviewer_vpAQOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose VegBench, and benchmark for vegetation classification across a vegetation hierarchy from the formation to the alliance level. The benchmark is built using around 42k expert field surveys conducted in California from 1978 to 2023. and additionally, It proposes a much more fine-grained task than what geospatial foundation models are typically trained and tested on. The authors compare a range of geoFMs on this dataset, and other baseline models, and show the potential of time-series geo-FMs compared to other machine learning methods on this task, but also highlight the huge performance gap between geoFMs and maps that experts obtain from NAIP imagery (which is used in some of these geoFMs). VegBench proposes a unique testbed for developing models capable of capturing fine-grained ecological information.

Critical strengths

The paper is generally clear and reads well. This paper is a step towards understanding what level of granularity foundation models can capture when it comes to ecological tasks, and it highlights important gaps in geoFMs capabilities, and the fact that they are often not designed to support fine-grained ecological tasks requiring high-resolution input data. The experiments and ablations are sound, and I really appreciated the quality of the analysis and in-depth investigations assessing different design choices in the task or models used in the appendix. I haven't run the code, but it looks clear and ready-to-use.

Critical weaknesses

The paper is generally clear, but some details about the data and task setup remain a bit unclear (see questions). More importantly, while this dataset is definitely of interest to investigate the behaviour of geoFMs on fine-grained tasks and to guide the development of methods that can capture fine-grained ecological information, a question one would want to ask is how relevant is it to try to do alliance classification from remotely sensed imagery. The comparison with expert map relying on NAIP is interesting and sets an upper bound on what might expected of ML models. There is indeed a large gap in performance between geoFMs and expert annotations on remote sensing imagery, but the experts' maps from NAIP (+expert knowledge)are also quite prone to errors, and so one could ask to what extent solving this task is relevant in practice for conservationists. What is an acceptable level of uncertainty of alliance classification, and how it is used in practice? I think there could be a little bit more background on why it is of interest in practice to classify vegetation at the alliance level at scale (geographically and across all possible alliances). I assume that conservationists have targeted alliances that in practice, would require detecting particular species for example, and one might approach the problem completely differently than this fine-grained classification task.

Questions for authors
  • Co-registration: I understand the impossibility to do temporal alignment on all surveys. It seems from A.3.1 that dataset size is a determining driver of performance. My first question is, why have the authors not tried a strategy where the imagery was post-2017 was date matched with the surveys, and the rest of the survey were matched with a fixed datae in order to have the full data? (I understand this might be because of what is highlighted in A.6.4. ) In A.6.4, the co-registration experiments on the subset of post-2017 surveys show that on generally using date-matched imagery performs overwhelmingly worse than fixed 2018 imagery. This is quite an interesting finding but I do not understand the argument of autocorrelation-driven overfitting. How does using imagery co-registered enhance this?
  • data splitting: can the authors clarify the data splitting and what is entailed by building train/val/test set such that they "contain similar proportions of surveys and vegetation alliances for each vegetation mapping project" ? How did the authors ensure representation of the different alliances in the different sets given the long-tailed distribution while also respecting geographical blocking to avoid spatial auto-correlation in practice?
  • Were there patterns of failure when using co-registered data due to season the surveys were conducted in?
  • Data resolution and field surveys and expert maps: what spatial extent does an expert survey cover? Could the authors also clarify the following the evaluations and comparisons with expert maps using NAIP imagery: Most geoFMs use imagery covering a larger extent than what the expert survey would typically encompass. In the stratified test and accuracy assessment test, is the task predicting a single alliance? And then in the expert map comparison, are the geoFMs assessed on a multilabel classification task?
    Or does the NAIP crop to compare expert maps with geoFMs cover the extent of the field surveys? (which is then how the expert map accuracy is computed)
  • Perhaps I missed something, but in Fig 4b, how is it that the std becomes larger for bins of alliances with more training examples per alliance for all the FMs?
  • Continuing the point raised in the weaknesses: that being said, in the discussion, the authors highlight the high accuracy of expert maps on rare classes, but this almost points to the relevant task being more targeted rather than classifying many alliances at scale.
  • What does "mirroring expert vegetation mapmaking into geoFMs" look like when domain knowledge might be extremely specific and focused?

While I am recommending acceptance, I believe it would be important to address the weaknesses highlighted in the previous section and these questions to make the work more accessible both to people looking to tackle fine-grained ecological tasks with geoFMs and people looking to improve methods for capturing fine-grained information. I will not maintain my score if these are not addressed.

6 comments
AuthorsOpenReview ↗

Overall response and response to weaknesses

We thank the reviewer for their thorough review and positive assessment of our work. We also acknowledge that some details of our benchmark setup were unclear in our original submission, as the reviewer noted. To help aid understanding and readability, we have significantly expanded the explanation for our co-registration choices (main paper lines 289-302; Appdx. A.3.1, A.6.5, A.6.6; Figs, A5, A6) and our data splitting strategy (main paper lines 247-269; Appdx. A.3.2, A.6.7). We hope that these changes fully address the reviewer’s concerns so that they maintain or raise their score. If the reviewer feels their concerns still haven’t been fully addressed, we would appreciate hearing what is still missing so we may adequately address them.

Response to weaknesses

The reviewer flagged that it was unclear why vegetation alliance classification from remote sensing imagery is a relevant task for conservation. The ease and spatial scale at which remote sensing imagery can be acquired makes it an efficient resource for mapping vegetation alliance at scale. These remote sensing-based vegetation alliance maps are then used in three different ways for conservation management in California: 1) supporting species habitat mapping [1,2], 2) monitoring species habitat quality [3], and 3) delineating rare and endangered communities / species ranges [4,5]. Some of these tasks are alliance-specific as the reviewer pointed out (e.g. delineating pickleweed-dominated habitat for the endangered salt marsh harvest mouse [4]), while others require mapping a broad set of alliances across a large geographic region with reasonable accuracy (e.g. for the vegetation maps that underpin the California’s statewide species distribution maps for imperiled plants and animals [2]).

In fact, mapping fine-scale vegetation is so important to the state’s conservation priorities that in 2007, the state passed legislation requiring the development and maintenance of a statewide vegetation mapping standard to support wildlife habitat conservation, identify wildlife corridors, and prioritize land-use and conservation decisions [6]. To implement these standards, the state adopted the Survey of California Vegetation mapping protocol in 2022 as its official mapping standard [7]. This protocol mandates that an alliance-level vegetation map must have at least 80% accuracy on accuracy assessment surveys to meet these legislatively-mandated standards [6]. Under these new mapping standards, most of the state’s older expert vegetation alliance maps (many of which we use in VegBench) do not meet this quality threshold and are being actively revised or remapped by the state [8]. This makes VegBench an especially exciting benchmark for geoFM development, as any geoFM-based vegetation classification process that meets the state’s 80% accuracy bar on VegBench would be a powerful new tool to help the state streamline its vegetation map revision process. Beyond California, at COP 16, the Kunming-Montreal Global Biodiversity Framework was revised to identify fine-scale ecosystem classifications and maps as critical to achieving the GBF's targets for biodiversity-inclusive spatial planning, ecosystem restoration, and protected area conservation [9].

We have added a new section A.1.3 to the Appendix outlining these use cases for fine-scale vegetation classification and mapping, and added a brief reference in main text to address this (lines 49-53). We address the reviewer’s further individual questions and concerns below.

AuthorsOpenReview ↗

Response to Questions 1-2

Question 1. Co-registration

We do not combine fixed imagery for pre-2017 surveys with co-registered imagery for 2017+ surveys because this would introduce inconsistent temporal relationships between imagery and survey labels in the same training dataset. Furthermore, in our initial ablations (Fig. A4 in our original submission) we saw that using fixed imagery (open circle) performed better than co-registered imagery (orange diamond) when using the same training volume (namely, surveys from 2017 and onward) when testing with fixed imagery, providing little empirical justification for this mixed approach.

Our original hypothesis was that probes were overfitting to spurious temporal signals, since all surveys for a given vegetation project are typically collected all within the same year, leading to strong spatial clustering within surveys collected within a particular year (our autocorrelated-driven overfitting argument). To more fairly compare the fixed and co-registered approach, we expanded our temporal co-registration analysis to also test with co-registered data (subfig. B. in the new Fig. A5) and added an additional analysis (Fig. A6). This new figure tests performance differences between training and testing with co-registered 2017+ imagery and a. training with fixed 2017+ imagery but testing with co-registered 2017+ imagery, b. training and testing with fixed 2017+ imagery, and c. training with all fixed surveys but testing with co-registered 2017+ imagery. Overall, we still see that using all surveys with fixed training imagery performs the best regardless of whether evaluating with fixed or co-registered imagery. We also see that there are no performance differences between using fixed versus co-registered imagery for 2017+ surveys so long as the train and test imagery use the same co-registration strategy. We believe these additional analyses further support that our choice to use all surveys and fixed imagery is a sound one.

Question 2. Data splitting

We have two separate test partitions. First, the accuracy assessment surveys are independently collected as part of the VegCAMP classification and mapping protocol [7]. These surveys allow for direct comparison to expert vegetation mapping approaches, but some accuracy assessment surveys are taken as close as 50 m to training surveys based on the way VegCAMP implements their accuracy assessment protocol. Therefore, we also developed a separate spatially stratified test partition that uses strict spatial blocking to ensure no spatial leakage between train and test. For the spatially stratified partition, we used a simple greedy optimization to ensure a balanced proportion of both surveys and alliances across projects and across the overall partitions for the train, test, and validation splits (aiming for a 70/15/15 ratio for all targets), while maintaining a strict 4.752 km gap between train, test, and validation surveys.

Because rare vegetation alliances can be very spatially localized, there is a tradeoff between maximizing the number of alliances represented across the train and test set and maximizing this geographic blocking to avoid spatial autocorrelation. We chose our current buffer size of 4.752 km to ensure no test survey could fall within the remote sensing imagery tile of a train survey, but this does mean that some rare alliances end up only in the train or test sets (~33%, see Table A1). We realize our motivation for these choices and the distinction between these two test partitions was unclear, and so we have significantly expanded the benchmark splitting description in the main text (lines 247-269) and add details plus discussion on the nuances behind these choices in Appendix section A.3.2.

AuthorsOpenReview ↗

Response to Questions 3-6

Question 3. Were there patterns of failure when using co-registered data due to season the surveys were conducted in?

The extent to which model performance varies with survey season depends on the type of remote sensing data used. Time Series GeoFMs build annualized features that will not be impacted by the specific timing of the survey within the year, while NAIP imagery is always taken in the summer, so timing of the survey data collection similarly will not be influential. However, models that take in spatiotemporally co-registered Sentinel-2 and Landsat data could see variation in performance based on the survey timing, based on seasonal differences in appearance of the vegetation classes that could make those alliances easier or harder to categorize.

To look into this, we have analyzed seasonal performance differences between the 2017+ co-registered and fixed temporal co-registration strategies in Appendix A.6.15 and Appendix Figure A14. The one area where we observe a seasonal difference in Sentinel-2 and Landsat image models performance is the spring, where there is a bigger drop using co-registered data than fixed data compared to the other groups. However, the spring season has the second to fewest number of test surveys, only 7.82% overall, limiting the conclusions that can be drawn. Furthermore, this analysis cannot fully isolate seasonal effects because co-registration changes both the year and season of image acquisition, and survey subsets differ in size, alliance composition, and geography. While the underlying distribution of the available surveys can make it difficult to disentangle these factors, we believe VegBench overall provides a valuable standardized evaluation of geospatial model performance on fine-grained vegetation classification under realistic remote sensing data availability and temporal alignment constraints.

Question 4. Data resolution of vegetation surveys.

The spatial extent of an expert survey can vary. For example, relevé surveys can range from 10m2 to 1km2, while rapid assessments have a minimum mapping unit of half a hectare, but can range up to five hectares in size [7]. We added a new analysis that compares test set performance across the eight different vegetation survey types used in VegBench, and we see no appreciable difference in performance across survey types, especially for the three most common and standardized survey types (Figure A3). As the reviewer correctly points out though, this variance in survey collection protocol means sometimes the actual remote sensing imagery footprint used by geoFMs contains more than just the single vegetation alliance annotated in a vegetation survey. We explicitly test this spectral mixing effect in Appendix A.6.4 and Figure A4, where we compare performance for vegetation alliances where the entire 135 x 135 m area surrounding a survey is mapped to the same vegetation alliance by expert maps (pure surveys) versus multiple vegetation alliances (mixed surveys). We see that while performance is better for NAIP image models on pure surveys compared to mixed, the overall relative ranking of performance across models is very high between pure and mixed surveys (Spearman rank correlation = 0.88), implying that treating the classification task as single label as opposed to multilabel will lead to similar conclusions on how well geoFMs perform at fine-grained vegetation classification.

Question 5. Clarifying expert map evaluation and comparison process.

Vegetation surveys are designed to enable experts on the ground to classify the vegetation at the specific survey point. Each survey point is assigned a single vegetation alliance, and thus the evaluation setup in our benchmark is constructed as a single label classification task at each survey point to match the structure of the survey data. When we evaluate expert maps on our benchmark, we identify which expert map polygon contains the survey point, assign the alliance of that polygon to the point, and then evaluate it as a single label classification prediction. This allows us to compare maps and models on the same surveys, using the same metrics. Details of this process are now outlined in Appendix A.2.5.

Question 6. Standard deviations in Figure 4b.

All models consistently perform poorly on alliances with fewer numbers of training examples, which results in the standard deviation for per-alliance recall being quite low across all models. As the number of training examples increase, the models in general tend to perform significantly better overall on average per-alliance. However, each model still usually will have at least one alliance it struggles to perform well on even after seeing many training examples, leading to larger variance of performance and thus larger intervals for alliances with many examples.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall response to reviewers

We thank the reviewers for their valuable feedback. We were encouraged that they “appreciated the quality of our analyses” and found the paper “reads well” (reviewer vpAQ), that the motivation for our work is ”well explained” and our “data has value” (reviewer WL6j), and that the paper “is excellently written” and “covers a gap in the current suite of benchmarks available for geoFM models” (reviewer d7iA). We also agree with reviewers that several points of our first submission lacked clarity. Points of confusion raised by multiple reviewers are summarized below, along with the changes we made to address them. For the additional points not summarized here, please see individual reviewer responses.

  1. How we developed the spatial train/test splits was not sufficiently clear. We rewrote and heavily expanded upon our train/test split design choices in Appendix section 3.2, and explained the process in detail in our response to reviewer vpAQ. We emphasize that there is no spatial leakage between the train and test set with the spatially stratified test partition.
  2. Our temporal co-registration choices were not sufficiently clear. We have added additional clarification and analyses in Section 3.2 and expanded the details in Appendix Section 3.1 to add additional clarity and justification for those decisions in our benchmarking design, and explain this in detail in our response to reviewer vpAQ.
  3. We did not sufficiently motivate how alliance-level vegetation classification is useful in downstream applications. We have added concrete examples of how these classifications are used for conservation decision-making in the introduction and Appendix Section 1.3, and explain in detail in our response to reviewer vpAQ.
  4. The size of VegBench relative to other geoFM benchmarking tasks. VegBench is equivalent in size and spatial scale to standard geoFM benchmark classification tasks. We have clarified this in the main text, and in our response to reviewer WL6j.
  5. Prithvi’s evaluation setup used a single image Sentinel-2 input rather than a multiple image HLS input. Per reviewer WL6j’s request, we have added an additional Prithvi baseline aggregated over 4 image HLS data to our list of evaluated geoFMs, in addition to the existing single image Sentinel-2 version. We have updated our figures to include this additional Prithvi baseline in the new version of the pdf (see Fig. 3 for per-model comparison). Additionally, we have also added the new public version of OlmoEarth 1.3 embeddings (https://allenai.org/blog/olmoearth-v1-3) to make our baselines fully reproducible.
AuthorsOpenReview ↗

Small benchmark cleaning changes

We also want to bring to the reviewers’ attention that during the process of adding these additional and updated OlmoEarth 1.3 to our benchmark, we discovered and cleaned up a few errors in our vegetation data curation and cleaning pipeline (specifically, slight variations in alliance name punctuation across surveys that impacted how we linked several surveys to the vegetation hierarchy). After updating, we have 1) added back vegetation surveys that were previously unnecessarily dropped, and 2) removed duplicate alliance names. The updated VegBench contains 43,468 surveys for 363 vegetation alliances, compared to our original 42,393 surveys and 387 alliances. 94.7% of the data from VegBench has not changed, nor have any of the trends or conclusions shifted. We explain these small changes in more detail in a comment below.

Our main contribution of a new expert-verified, field-collected benchmark for evaluating geoFMs for fine-scale ecological classification remains unchanged. We also see negligible changes in the absolute performance and rankings of the different geoFMs between the two versions of VegBench. Across all geoFMs we evaluated, the agreement in performance between the old and new version of VegBench is very high. Specifically, between our original and our current version of VegBench, across all geospatial models the Spearman’s rank correlation for alliance-level accuracy is 0.913 on the accuracy assessment test set and 0.898 on the spatially stratified test set, meaning that the relative ordering of which geoFM performed better changed negligibly before and after these cleaning pipeline changes. Additionally, on average the absolute alliance-level accuracy changed only 0.85% across geoFMs on the accuracy assessment test set and 2.26% for the spatially stratified test set. These changes are far smaller than the 7.06% average accuracy gap between time series and NAIP image geoFMs, and the 7.93% gap between NAIP image and Sentinel-2 images. This means that our key takeaways as to what corresponds to geoFM success and failure on this benchmark remain the same. We have updated all figures and tables in the manuscript to reflect these updates to VegBench, and all benchmark processing clarifications and changes are highlighted green in the updated pdf. Especially, the survey data cleaning and label harmonization portion of the Appendix (Section A2) has been significantly expanded to clarify this process. All of these changes will be incorporated into the public codebase and benchmark csv. Lastly, further details about why the cleaning pipeline changed are described in the following OpenReview comment.

We thank the reviewers for taking the time to review these updates and for their continued engagement with our work. We also hope these details highlight the importance of benchmark contributions like ours to the ML community, as harmonizing heterogeneous ecological field surveys requires reconciling decades of highly fragmented ecological data, changing classification systems, and inconsistent nomenclature. By making this benchmark and its preprocessing pipeline publicly available, we hope to lower the barrier for future research evaluating foundation models on ecologically-meaningful tasks.

AuthorsOpenReview ↗

Specific details of benchmark cleaning pipeline changes

In our original vegetation survey cleaning pipeline, since the vegetation surveys that underpin our benchmark only record the vegetation alliance or association name and record them in inconsistent ways across vegetation mapping projects, we used both the California Natural Communities list (CNC, https://nrm.dfg.ca.gov/FileHandler.ashx?DocumentID=153399) and the online Manual of California Vegetation (MCV, https://vegetation.cnps.org/) to source vegetation alliance / association names and their full vegetation hierarchy for crosswalking. Theoretically, since the CNC and MCV are derived from the same vegetation hierarchy, they should have the same alliance names, but between the CNC and MCV there were still some small inconsistencies that slipped through our original alliance de-duplication pipeline and thus introduced a few duplicate alliance names. First, there were sometimes small punctuation differences between the two lists, such as using en vs. normal dash [e.g. Salvia mellifera – (Artemisia californica) in the CNC vs. Salvia mellifera - (Artemisia californica) in the MCV]. Second, the version of the MCV we used was slightly newer (June 4, 2026) than our version of the CNC (Feb. 27, 2025), meaning some alliances that were still marked as provisional (meaning not yet verified and approved as new USNVC-compliant official vegetation alliances) in the CNC but had already been approved and fully integrated into our later copy of the MCV (e.g. “Quercus wislizeni - Quercus chrysolepis (shrub) [pending]” in the CNC vs. “Quercus wislizeni - Quercus chrysolepis (shrub)” in the MCV).

When combining alliance names from the CNC and MCV, our original pipeline used exact string matching to identify duplicates. Consequently, alliance names that differed only slightly in punctuation or provisional status were treated as separate candidate alliances. For the vegetation surveys themselves, given the vegetation survey data from VegCAMP is prone to vegetation alliance typos, spelling changes, and punctuation differences, we manually reviewed these inconsistencies in the survey data to generate an extensive set of vegetation survey alliance name harmonization steps, such as trimming and collapsing whitespace, using all lowercase names, and normalizing common punctuation and equivalent phrases (like and vs &). Our original pipeline then used a fuzzy matching algorithm to help resolve remaining inconsistencies and match vegetation alliance names annotated in the surveys to the full vegetation hierarchy [specifically difflib.get_close_matches(n=3, cutoff=0.9)]. Despite our extensive manual survey correction effort, the small number of inconsistencies between our alliance reference lists mentioned in the above paragraph remained undetected. In the case where more than one alliance name matched a given survey, our policy was to remove that survey on the assumption that this alliance name was not able to be disambiguated. This means that in our original version of VegBench, over one thousand surveys were removed that we have since been able to robustly match and there were originally 12 erroneously duplicated alliances. We have updated our vegetation survey curation pipeline for VegBench to correct these errors. We now use only the MCV as our canonical source of vegetation hierarchy names, have improved our alliance name de-duplication pipeline to better handle punctuation and survey notation inconsistencies, and now use exact string matching instead of fuzzy matching between vegetation surveys and the MCV hierarchy. All of these steps have been thoroughly documented in the updated pdf (see section A.2, green highlights).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.