- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces BioInteract, a curated multimodal dataset of co-labelled iNaturalist images filtered for known interactions, used to benchmark the capabilities of different models on vision-language relational understanding of biotic interactions. Existing datasets are limited in both their coverage of biotic interactions and the rigor with which they evaluate them. This work addresses these gaps by curating a dataset of 256K co-labelled images that have been filtered to include interactions between entities that are known to interact in specific contexts. The dataset is evaluated on a benchmark, BioInteract100, through a variety of perturbations to relational queries. The authors claim that current models show consistency gaps across different paraphrases, and are brittle to relation-direction changes.
Critical strengths
-
Addressing a data gap: This work addresses a real data gap and is useful in its formulation for testing the relational understanding of different models. The dataset appears well constructed: the filtering protocol, grounding in GloBI records, and manual verification of a sample of images (Appendix D) are all sensible. I'm particularly excited about the potential usefulnesss of the spatiotemporal metadata which could also support more geospatially grounded analyses of biotic interactions beyond the image+language setting.
-
Template construction is sound: Generating queries from structured triplets rather than with an LLM keeps the perturbations controlled and auditable. The data generation system looks to be well controlled.
Critical weaknesses
-
Brittleness claims: While Table 5 supports the claim of brittleness of the MLLMs (image-text contradictory accuracy is 52% - 65%), Table 10 seems to show that these models are not brittle to relation-direction changes (77% - 97% accuracy on the same reversed queries). In addition, many reversed queries are ecologically impossible (e.g., a flower visits a bee) and can be rejected without looking at the image. The text-only control does not isolate this effect, since its ground truth is ``No'' for every query.
-
Geospatial context is not considered: The spatiotemporal metadata is highlighted as a differentiator (Table 1) but is not used in any evaluation. The likelihood of interactions between different entities is greatly determined by their proximity and temporal overlap in activity (timing of flowers blooming vs. bees being active), and this could serve as a useful prior or control.
Questions for authors
- How many images and queries were used for the MLLM evaluations, and what the in Table 5 is computed over?
- In Table 5, Claude Sonnet 4.6 (text-only, Contradictory) has a flip rate of 0.71 alongside an almost perfect accuracy of 0.99. The perfect accuracy suggests consistency in the outputs. Please clarify.
- In Table 5, how should the "All" column should be interpreted, given that equivalent and contradictory queries have different correct answers?
- With regards to (W1), please clarify whether Tables 5 and 10 were computed on the same images and queries, and how to interpret these results in context of the stated brittleness claim?
- Can the authors provide a blind baseline scored with the same labels as the image condition (e.g., "Is it plausible that X pollinates on Y?" with no image), to help separate ecological priors that the MLLMs may know from visual grounding?
- Have the authors considered providing the geotemporal metadata (location and date) to the models as additional context? This could help reveal how much ecological prior information the models rely on, and whether such information helps beyond the imagery/text alone.
4 comments
We thank the reviewer for their constructive feedback and for recognizing BioInteract's contribution to evaluating biotic interactions, the controlled construction of relational queries, and the potential of its spatiotemporal metadata for ecological research. We particularly appreciate the suggestions regarding the interpretation of relation-direction reversal, the role of ecological prior knowledge, and the clarity of our evaluation protocol.
We address each concern below (it took longer to compute our rebuttal with the addressed questions):
Q1
BioInteract100 comprises 281 unique interaction triplets, each with 100 sampled images (28,100 image records). For the MLLM evaluation, we exclude the symmetric interactsWith relation (125 triplets), resulting in 156 interaction triplets and 15.6K images. Each image is evaluated using five non-contradictory query formulations and two direction-reversed formulations, yielding up to 109.2K image–query evaluations per model.
In Table 5, we first compute accuracy independently for each query formulation and then report the mean ± standard deviation across formulations within each group (non-contradictory, contradictory, and all). Thus, the ± values represent variability in accuracy across query formulations; we will clarify the evaluation size and aggregation procedure in the revised manuscript.
Q2
We thank the reviewer for identifying this inconsistency. In Table 5 it is over 15.6K images covering 156 unique interaction triplets with/without images. We confirm that the flip rate of 0.71 reported for Claude Sonnet 4.6 (text-only, Contradictory) in Table 5 is a reporting error.
Below we report these results in detail:
Text-only evaluation:
| Model | Active Acc. | Passive Acc. | Flip ↓ | Active-reverse Acc. | Passive-reverse Acc. | Flip ↓ |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.93 | 0.98 | 0.07 | 0.99 | 1.00 | 0.01 |
| GPT | 0.92 | 1.00 | 0.07 | 1.00 | 1.00 | 0.00 |
| Gemini | 0.00 | 0.07 | 0.00 | 0.73 | 0.97 | 0.28 |
Image–text evaluation:
| Model | Active Acc. | Passive Acc. | Flip ↓ | Active-reverse Acc. | Passive-reverse Acc. | Flip ↓ |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.85 | 0.82 | 0.07 | 0.63 | 0.63 | 0.00 |
| GPT | 0.82 | 0.71 | 0.35 | 0.32 | 0.33 | 0.08 |
| Gemini | 0.91 | 0.85 | 0.19 | 0.44 | 0.46 | 0.07 |
We will correct the erroneous value in Table 5 and clarify the flip-rate calculation and reporting in the revised manuscript.
Q5
To investigate whether MLLMs rely on ecological prior knowledge independently of visual evidence, we conducted an additional blind text-only evaluation using interaction queries from BioInteract100. Specifically, we reformulated the interaction triplets into ecological plausibility questions (e.g., "Is it plausible that species X visits flowers of species Y? Answer with Yes or No and nothing else.") and evaluated the models without providing images. We focused on the ‘visits flowers of’ interaction, covering 123 interaction triplets and 12.3K image records. We report accuracy for active and passive formulations and their direction-reversed counterparts. Flip rate measures prediction disagreement between paired formulations (↓ lower is better):
| Model | Active Acc. | Passive Acc. | Flip ↓ | Active-reverse Acc. | Passive-reverse Acc. | Flip ↓ |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.97 | 0.88 | 0.13 | 1.0 | 0.91 | 0.08 |
| GPT | 1.0 | 0.96 | 0.03 | 0.73 | 0.31 | 0.45 |
| Gemini | 1.0 | 1.0 | 0.0 | 1.0 | 0.04 | 0.95 |
The results show that all three models achieve high accuracy on correctly directed interactions (0.88–1.00), suggesting that they already encode substantial ecological prior knowledge about species–flower visitation relationships. However, their behavior differs considerably for direction-reversed queries. Claude Sonnet 4.6 remains relatively robust, achieving 1.00 and 0.91 accuracy across the two reversed formulations. In contrast, GPT achieves 0.73 and 0.31, while Gemini exhibits a particularly pronounced difference (1.00 versus 0.04), with a flip rate of 0.95.
These findings suggest that although MLLMs can recognize plausible ecological relationships from species identities alone, their judgments can be highly sensitive to linguistic formulation and interaction direction. This reinforces the importance of distinguishing ecological prior knowledge from visually grounded relational understanding. We will incorporate this complementary evaluation into the revised manuscript.
(continued)
Q3
The "All" column in Table 5 aggregates results across all four query formulations: active, passive, active-reverse, and passive-reverse.
For accuracy, each formulation is evaluated against its corresponding ground-truth label: "Yes" for active and passive queries and "No" for direction-reversed queries. We report the mean accuracy ± standard deviation across the four formulations.
For flip rate, we compare the correctness of predictions (0/1) across all four formulations, using the active query as the reference. An instance is counted as a flip if at least one of the other formulations differs in correctness from the active query. Thus, the "All" flip rate measures inconsistency in prediction correctness, rather than changes in the raw Yes/No responses. This distinction is important because contradictory queries intentionally require different answers, and correctly changing from "Yes" to "No" should not be considered inconsistent behavior.
We will clarify this aggregation procedure and the interpretation of the "All" column in the revised manuscript.
Q4
We thank the reviewer for highlighting the apparent discrepancy between Tables 5 and 10. We clarify that the two tables evaluate different image subsets and experimental settings. Table 5 reports MLLM performance on BioInteract100 across meaning-preserving and contradictory query formulations, whereas Table 10 specifically evaluates the 39 images identified during our manual audit (Appendix D) in which the annotated interactions were not visually observable. For these 39 images, we applied the same binary protocol, using the prompt "Does this image show {some query}?", with an expected answer of "No" because the interaction was not visually observable.
Consequently, the higher accuracy in Table 10 does not contradict the lower contradictory-query accuracy in Table 5, as the results are obtained from different image subsets with different expected answers (see Q2,Q5). Table 10 instead investigates whether MLLMs can distinguish documented ecological relationships from interactions actually supported by visual evidence. We agree that this distinction was insufficiently explained in the manuscript and will clarify the evaluation subsets, objectives, and interpretation of both tables in the revised version.
Q6
We acknowledge that spatiotemporal context has not yet been incorporated into our current evaluations, which focus on assessing the capabilities of pretrained vision–language models using image–text representations. Consequently, we have not yet quantified the added value of geographic and temporal information for recognizing biotic interactions. Nevertheless, such metadata could provide valuable ecological priors. Geographic coordinates can help establish whether interacting species occur within compatible geographical ranges and habitats, while observation dates can provide information about seasonal activity and phenological patterns, such as the overlap between insect activity and flowering periods. These signals could help models assess the plausibility of an interaction, particularly when visual evidence is ambiguous or incomplete. An important direction for future work is therefore to compare image–text, spatiotemporal-only, and combined image–text–spatiotemporal settings.
I thank the authors for their response to my questions.
My questions regarding evaluation size, and the conflicts between some tables have been answered (Table 5 v. 10 and the Claude's text-only flip rate) have been addressed.
After reading the other reviews, I agree with Reviewer 6UZM that it remains unclear whether the benchmark measures relational understanding beyond recognizing the species pair. The blind baseline reinforces this concern, since the models already reach 0.88–1.00 accuracy on directed relational queries from only using species name. I am therefore lowering my score to 2. While I'm excited about the dataset and its potential uses, I don't think the paper, or this benchmark is ready for acceptance at this time.
The concerns raised in my initial review remain unaddressed. I would appreciate clarification on these points, particularly regarding the interpretation of the brittleness results and the role of ecological priors.