- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper evaluates geoFMs (both time-series based and image-based), remote sensing approaches (based on high resolution NAIP images and lower resolution Sentinel-2 data, and Landsat), and feature baselines (based on location, soil type, elevation) against a large new benchmark for hierarchical vegetation classification, where the ground truth data was curated from over 42k expert field (ground-level) surveys across California, US, spanning a 45-year period. The ML-approaches are additionally compared to expert maps produced by the same vegetation survey programme through an expert-in-the-loop process based on high resolution areal images and environmental data. Not only does the paper propose a large and meticulously curated benchmark, and evaluate a large number of models, but additionally based on the detailed experiments and ablations, they are able to draw several well-supported conclusions and insights about geoFMs. Mainly, that good performance on broad ecological categories does not imply that fine-grained structure required for fine-grained classification is encoded (with the geoFM performance on fine-grained classes falling to the level of simple location-based baselines). A conclusion that time-aggregation and improved resolution help this task in complementary ways is also supported. By interpolation, they also conclude that both temporally aggregated models and high-resolution imagery models would need around ~1000 samples per vegetation alliance in order to reach the performance level of expert maps, pointing out how such a task is not feasible and instead proposing alternative directions for exploration and research in geoFMs. Many other smaller conclusions and recommendations are also supported by the performed analysis.
Critical strengths
I recommend accepting this paper because:
- The newly proposed benchmark is based on a large amount of curated and and newly published data, and also covers a gap in the current suite of benchmarks available for geoFM models (covering fine-grained vegetation classification task, and showing how good performance on coarse vegetation classification is not an indicator of performance on fine-grained veg classification task). Additionally, all the efforts were made and required details shared to make the work easily reproducible and the benchmark accessible.
- It provides strong evidence about information currently captured and not captured by geoFM models, offers concrete estimates of the effort/cost needed to close the performance gap between experts and geoFM models for fine-grained veg classification and instead makes recommendations for research directions which could make closing this performance gap feasible. Many additional insights are drawn from the exhaustive experiments and ablations and highlighted by the authors.
- It is excellently written, well-presented and easy to follow. I particularly commend the appendix section containing all acronyms and terminology, making the manuscript which (must) heavily rely on ecological terminology easily accessible for CS/ML researchers regardless of their familiarity with the specific application area.
Critical weaknesses
The weak points noted here are relatively minor, especially in light of the excellent study and strong conclusions highlighted as strengths of the conducted study.
- Three metrics are used for evaluation, accuracy, balanced accuracy and hierarchical accuracy. While all of these metrics make a lot of sense, there is a small worry that balanced accuracy as a metric is based on recall only, and therefore a good performance on minority classes could mask a large number of large positives, which would not contribute to the balanced accuracy score. Since the relative order of performance between different models and baselines, and other performance trends, are largely preserved between accuracy and balanced accuracy first, I have some confidence that this would hold for any additional selected metrics. Despite balanced accuracy being only an additional metric with the evaluation largely included in the appendix only, I would encourage the authors to consider either replacing it or additionally including other metrics (macro-F1 also gives all classes equal weight, takes both precision and recall into account, however does not take true negatives into account; and/or MCC (Matthews correlation coefficient) which takes the full confusion matrix into account, it's not perfectly balanced per class but still more robustness to class imbalance than plain accuracy)
- The benchmark split is stratified in such a way that surveys and vegetation alliances from the same vegetation mapping project are assigned both into train, val and test sets. However, performance drop due to both spatial and temporal domain shift is well-documented. This effect may in fact disproportional affect smaller, isolated habitats and vegetation alliances, where the performance already suffers due to the lack of training data. While redesigning the proposed benchmark such that it also gives useful insights of model performance under domain shift would be a large task and out-of-scope for the current submission, it would be nice to see the authors address this as a limitation of the proposed benchmark. (or explain how or why this is not a weakness of the benchmark)
- The related works provide a good coverage of three key relevant topics: geospatial benchmarks, ecological classification systems, ecological prediction with geospatial information. However, if space permits, the authors may also consider commenting on a body of work where ML is applied to ecological prediction from more fine-grained information. Some recommended papers to consider additionally are listed at the bottom of this comment ([1]-[4]). All of these papers support the argument already made in the discussion, that adding high-resolution imagery, ground level imagery, other ecological information, and falling back to coarser levels of hierarchy can enable ML models to perform better in fine-grained hierarchical vegetation classification task. Specifically, [1] (as well as already cited paper (Chytrý et al. 2020)) show that fine-grained and hierarchical predictions of EUNIS habitat types can be made from field vegetation plots. [2] performs hierarchical habitat classification from ground-level images (also collected from expert field surverys) according to the UKHab Classification (which, coincidentally, would be worth mentioning as another example of a hierarchical ecological classification system, developed in/for the UK); [3] provides further evidence that increasing spatial resolution increases performance of habitat mapping (however, while the habitat units are very fine-grained in this paper, they are limited to dune habitats and only a single hierarchy level is examined). [4] serves as additional evidence that very-high-resolution imagery can contain fine floristic and structural information required to characterise plant communities (however, the actual classification is into 9 tree species). I suggest that a brief coverage of non-geoFM-based ecological prediction models in your related work could better ground the paper, provide better context, and further support your discussion/conclusions.
[1] Leblanc, C., Bonnet, P., Servajean, M., Thuiller, W., Chytrý, M., Aćić, S., Argagnon, O., Biurrun, I., Bonari, G., Bruelheide, H., et al. (2025). Learning the syntax of plant assemblages. Nature Plants, 11, 2026–2040. DOI: 10.1038/s41477-025-02105-7
[2] Shi, H., Norton, L., Ridding, L., Rolph, S., August, T., Wood, C. M., Qie, L., Bosilj, P., & Brown, J. M. (2026). Habitat classification from ground-level imagery using deep neural networks. Ecological Informatics, 95, 103751. DOI: 10.1016/j.ecoinf.2026.103751.
[3] Pafumi, E., Angiolini, C., Bacaro, G., de Simone, L., Fanfarillo, E., Fiaschi, T., Rocchini, D., Thouverai, E., & Maccherini, S. (2026). Pixels and Patterns in the Sand: Decoding Coastal Dune Habitats With Multi-Resolution CNNs. Journal of Vegetation Science, 37, e70173. DOI: 10.1111/jvs.70173.
[4] Nepi, L., Quattrini, G., Pesaresi, S., Mancini, A., & Pierdicca, R. (2025). AI-based estimation of forest plant community composition from UAV imagery. Ecological Informatics, 90, 103199. DOI: 10.1016/j.ecoinf.2025.103199.
Questions for authors
These do not directly impact my evaluation of the paper, however most if not all are small, easy to correct, and would improve the paper if corrected before publication:
- There are some minor misspellings ("To whit"), unclear figure labels (you refer to Fig1-a,b,c but there are no subfig a-b-c markings in Fig. 1), and incorrectly linked within-document references (especially to some complex composite figures in the appendix, or appendix sections)
- Again, your summary of terms is excellent and very helpful. I would suggest adding the term "operational map" or "operational vegetation product" to it.
- It is difficult to relate the English species names in the Results analysis to the Latin names embedded into Fig 5 (not helped by incorrect references to Fig 5 - i.e. Fig 5-aii is never referenced). Could both Latin and English names be included, either in the images, or in the text, or both?
- What is the red dot displayed at the centre of each expert map shown in Fig 5?
- Could you more explicitly state how expert maps, which are segmentation masks, are compared to all the classifier models used in your comparisons?
- To consider for future work: one of the suggested procedures/recommendations for future research is falling back to a coarser hierarchy level when a fine-grained classification, such as vegetation alliance, can not be resolved. However, do you have any insight into whether predictions of all the tested models are consistent across the levels of the hierarchy?
2 comments
Smaller points raised
Q1: We fixed the typos and the references to non-existent Fig. 1a-c panels, and re-checked all cross-references.
Q2. "Operational map" term. We added this to summary/glossary of terms (Appendix A.1)
Q3. Common vs. Latin names in Fig. 5. The caption now gives both common and scientific names, and every panel (including 5a-ii) is referenced in the text.
Q4. Red dot in Fig. 5. It marks the survey location, and the caption now explains this.
Q5. How expert maps are compared to classifiers. For each test survey, we take the alliance of the expert-map polygon containing the survey point and score it exactly like a classifier prediction, so maps and models are evaluated on the same surveys, labels, and metrics. This is clarified in Section 3.1 and Appendix Section A.2.5.
Q6. Consistency of predictions across hierarchy levels. We did test whether directly predicting a coarser level of the hierarchy versus crosswalking a finer-scale classification up the hierarchy was consistent (Appendix A.6.11, Figure A11), providing some insight to the reviewer’s question. Broadly speaking, rolling up an alliance prediction to a coarser level of the hierarchy such as macrogroup led to slightly worse performance on average than predicting the coarser category directly, but this performance drop closed as the categories became more coarse (e.g. formation). Overall though, the performance drops are small (-2-4% on average) and so we believe these results do support that hierarchy-aware prediction with fallback to coarser levels under uncertainty is a promising direction, which we now note in the Discussion (lines 514-515).
Overall response and response to 3 weaknesses
We thank the reviewer for the careful and constructive review, and we sincerely appreciate their enthusiasm for our work. We address the reviewer’s three stated weaknesses, plus smaller points below.
1. Reported accuracy metrics only focus on recall, not precision.
We thank the reviewer for highlighting this. We chose to originally report balanced accuracy—the mean of per-class recall—as our main metric because it directly addresses class imbalance. Every alliance counts equally regardless of how many surveys it has, so rare habitats are not overshadowed by performance on common classes. Per the reviewer’s suggestion, we now also report Macro-F1 in our per-model analyses in the Appendix Figure A1 (subpanel b). Across the entire vegetation hierarchy we see almost no relative difference between model accuracies calculated using macro-F1 versus balanced accuracy (Spearman ρ = 0.97–0.99, Appendix A.6.13). Both balanced accuracy and macro-F1 also report AlphaEarth to have the highest performance at alliance-level classification. Interestingly, the correspondence between macro-F1 and balanced accuracy holds even more strongly for the rarest alliances (Spearman ρ = 0.997 on the stratified and 1.000 on the accuracy assessment, Appendix A.6.13). Given this strong correspondence between balanced accuracy and macro-F1, we continue to report accuracy and balanced accuracy instead of macro-F1 in the rest of the manuscript, and state our motivation for using balanced accuracy more explicitly in Section 3.3 (lines 351-354).
2. Effect of spatial and temporal domain shift across projects.
We thank the reviewer for highlighting that some of our benchmark design choices may interact with our ability to evaluate performance for rare alliances under varying amounts of spatial and temporal distribution shift. Since some vegetation alliances are extremely spatially restricted (e.g. found in only one vegetation mapping project across the entire state), we chose to include surveys from all projects in both train, test, and val so as to maximize the number of alliances we could evaluate. This design choice also mirrors VegCAMP’s current expert mapping workflow, where vegetation is classified, mapped, and evaluated on a per-project basis [1]. This does potentially however limit our ability to test the domain generalization of vegetation alliances that span multiple projects, especially rare but geographically-broad alliances. We agree that this is an important capability to test in geoFMs, and we also agree with the reviewer that this is out of the scope of the current version of VegBench. As suggested, we have added this as a limitation in the main paper (lines 526-533), explaining how we prioritized evaluating within mapping projects rather than under large spatial distribution shift, and how this is an important area of future expansion for our benchmark.
3. Adding related work on non-geoFM ecological prediction.
Thank you for these references and their connection to our work! We have added an additional paragraph to the Related Work outlining these additional fine-grained ecological prediction tasks that use deep learning, and included the references the reviewer recommended (lines 166-178). As the reviewer notes, these works support the points raised in our Discussion section that high resolution imagery and ground level images can provide additional ecological context that have potential to improve fine grained vegetation classification.
[1] California Department of Fish and Wildlife. Survey of California Vegetation: Classification and Mapping Standards. 4 Nov. 2022, https://filelib.wildlife.ca.gov/Public/BDB/Vegetation/ClassificationAnalysisRFP/Attachment1_SCV%20Classification%20and%20Mapping%20Standards%2020221104.pdf.