- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose integrating phenological information into taxonomic foundation models by introducing both a new dataset, Pheno3M, and a new multimodal model, PhenoMap. Pheno3M is a new benchmark dataset that pairs citizen-science plant imagery with phenophase annotations and environmental drivers. By leveraging these annotations with a phenological contrastive loss, PhenoMap produces factorized species and phenology embeddings aligned with plant imagery, outperforming other taxonomic foundation models on phenophase retrieval tasks.
Critical strengths
- Considering phenology in taxonomic foundation models is a crucial step to make these models really useful across applications, especially with the known bias toward specific phenophases. The factorized approach of PhenoMap is relevant.
- Proposing both a curated dataset and a well-designed method that outperforms the other taxonomic foundation models. The external evaluation is also commendable.
Critical weaknesses
- I think the authors could clarify how they expect the produced embeddings to be used and for which ecological applications, and consider evaluation setups accordingly. For example, I am not sure I understand where the mapping suggested by the method name and title fits in. The approach works well to retrieve images of a specific species and phenotype, but its utility to other applications is not sufficiently evaluated. Could it effectively map species distributions at different phenological stages, for instance? I would expect additional experiments in this direction to show its potential for different tasks. Figure 5 is a bit disappointing in this regard: the patterns described are elementary, and I am not sure how they can be useful. The lack of clarity about the true applications of this approach also reflects a field where some models have been called foundation models of biology and/or ecology, even though they were mostly restricted to taxonomic imagery. I believe this paper goes in the right direction, but it needs to clarify its scope.
- Otherwise, I noticed a few less important points that still deserve clarification and could improve the paper:
- L22 and L90: “satellite drivers”. These are not the satellites that drive a species' presence or phenophase, but the climate and other factors that satellite imagery may capture.
- Phenological Contrastive Loss: I think its description could be slightly improved by providing an intuitive explanation.
- Figure 4 is not discussed in the text. Also, which 2d projection is used?
- Perhaps I misunderstood something, but in Table 1, I do not expect the retrieval rate to be uniform. In particular, some phenological states make the plant much easier to recognize than others (e.g., at the flowering stage), which would make images from that phenological state more likely to be retrieved. I believe this is rational behavior for the model.
- Does the image→image setup correspond to querying with another image having the same species and phenophase?
- The objective of the approach is to perform well across the phenophases. However, there are no results showing that PhenoMap improves performance across phenophases, particularly for those that other baselines are supposed to neglect.
Questions for authors
- I would like the authors to clarify the intended scope of applications for PhenoMap embeddings and provide additional experiments accordingly.
- I would like to see my additional points considered and addressed, if necessary.
I am strongly willing to increase my rating if the authors address my concerns appropriately, as I see high value in this paper.
4 comments
I strongly thank the authors for considering all my comments and, in turn, adding numerous new results. While I appreciate this effort, I find it difficult to understand the differences between the new PDF and the previous one, since they haven't been highlighted (e.g., by showing the new elements in blue). Would it be possible to highlight the differences? In particular, it seems that most changes have been made to the Appendix, and I see here a missed opportunity to bring up in the main text one of the very interesting experiments you ran (e.g., Figures 16-18), which could easily replace Figure 5 that I find much less interesting and insightful (but I leave this choice to the authors' discretion).
Thank you for these helpful suggestions. We have uploaded a revised PDF that highlights all changes in blue: revised text in existing sections appears in blue, and entirely new sections, tables, and figures are marked by a blue heading or caption.
We have also moved Figure 16 into the main paper as Figure 8 in Sec. 5.4. We keep Figure 5 because it visualizes the field embeddings themselves, while Figure 8 shows how these embeddings can be used for discriminative phenology mapping.
Thank you for considering these suggestions. I acknowledge the substantial effort the authors have made to improve the paper during the rebuttal phase. I will therefore increase my rating from 2 to 3, although I believe a score of 3.5 would better reflect my assessment, should the AC wish to consider this (I find it unfortunate that the rating scale does not allow for finer granularity).
We thank the reviewer for the thoughtful review and for recognizing the value of our factorized approach and external evaluation. We address each point below. All section, figure, and table references refer to the revised PDF.
W1, Q1. Scope and the mapping task.
PhenoMap is the first framework to capture the dynamic visual distribution of thousands of species across space and time. By learning continuous, fine-grained representations that capture intra-phase variance, PhenoMap supports a large set of mapping and predictive tasks related to phenology. We evaluate the per-species phenophase mapping task in our experiments with USA-NPN and Pheno3M, where we show that PhenoMap outperforms all baselines in the per-species phenophase mapping. We also demonstrate PhenoMap's extensive qualitative mapping capabilities across species, phenological features, and locations.
Quantitative evaluation of mapping. For each visit, we query PhenoMap’s field branch with the species name, location, and date to generate an embedding without imagery. A probe predicts phenophase activity from these embeddings, which we evaluate against recorded phenological observations. On this task, PhenoMap outperforms BioCLIP 2, TaxaBind and ProM3E (Table 11). This result is especially significant under spatial holdout (0.495 vs. at most 0.359 macro AP). We also show that PhenoMap is superior on held out data from Pheno3M in newly added Table 12. Table 12 evaluates the same phenophase prediction task over 178K observations (per holdout protocol) across 3.3K species, showing that PhenoMap is the strongest phenological predictor on more than USA-NPN data.
To our best knowledge, dense expert-drawn species-specific phenological range maps do not exist, so we use spatially sparse survey data for quantitative evaluation.
Qualitative mapping capabilities of PhenoMap. Additionally, PhenoMap can produce species-specific phenology maps by querying the same dynamic representation of species+time across a spatial grid (Sec. 5.4, App. C). We've added new Figures 16-18 in App. C that show additional qualitative mapping capabilities on flowering, leaf growth, leaf appearance, and more. Figure 7 demonstrates that PhenoMap predicts accurate per-species distributions of phenophases across multiple regions.
W2a. "Satellite drivers." Agreed. We replaced "climate and satellite drivers" with "climate histories and satellite-derived environmental features" in the paper.
W2b. Intuition for the loss. Sec. 4.2.1 now explains that the loss aligns paired image and field embeddings and penalizes similarity between opposite phenophases more strongly than between adjacent ones. To allow within-phenophase variation, only matched pairs are positives masking other samples sharing a phenophase.
W2c. Fig. 4. We have added a discussion of Fig. 4 in Sec. 5.2. We have added the projection method, t-SNE, to its caption.
W2d. Non-uniform retrieval in Table 1. We agree that flowering plants are easier to recognize. But ecological monitoring also needs to detect species when flowers are absent. Otherwise, plants may appear absent simply because they are not flowering. Table 1 gives each species equal numbers of images per phenophase, yet models preferentially retrieve flowering images. To test whether this preference generates worse recognition outside flowering, we report species-identification accuracy separately for each phenophase (App. H.1, Table 19).
The flowering-to-vegetative gap in species-ID accuracy is:
W2e. Image→image setup. Yes. Sec. 5.2 (Task setup) defines relevance as matching species and phenophase, for both field and image queries.
W2f. Per-phenophase results. Several results are already broken down by phenophase: retrieval rates (Table 1), and retrieval confusions (Table 27). App. H adds two more breakdowns.