acceptodds
Withdrawn from ICLR 2027

BioInteract: A Large-Scale Multimodal Dataset for Evaluating Fine-Grained Semantic Understanding of Biotic Interactions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While recent advances in vision-language models (VLMs) have spurred the development of domain-specific datasets and benchmarks, these evaluations often fail to assess fine-grained semantic understanding, allowing models to achieve high scores without robust visual grounding. We address this evaluation gap through the lens of biotic interactions: directional, asymmetric relationships between organisms (e.g., wasp parasitizes caterpillar vs. caterpillar parasitizes wasp). This relational complexity yields naturally adversarial instances that expose superficial reasoning in current VLMs. To this end, we introduce BioInteract, the largest multimodal dataset for evaluating VLM robustness on real-world biodiversity challenges. Curated from iNaturalist and validated against established interaction literature, the dataset contains 15.4K unique interactions spanning 6.5K taxa across 256K images. Each interaction is structured as a source-relation-target triplet, enabling controlled semantic perturbations. We further introduce BioInteract100, an image retrieval benchmark revealing that state-of-the-art VLMs suffer from severe consistency gaps and are highly brittle to relation-direction reversals. BioInteract provides a faithful evaluation of multimodal AI, while encouraging the development of robust systems for ecological research.

Primary area:
datasets and benchmarks
Keywords:
vlm, fine-grained semantic understanding, biotic interactions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yTgGOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces BioInteract, a curated multimodal dataset of co-labelled iNaturalist images filtered for known interactions, used to benchmark the capabilities of different models on vision-language relational understanding of biotic interactions. Existing datasets are limited in both their coverage of biotic interactions and the rigor with which they evaluate them. This work addresses these gaps by curating a dataset of 256K co-labelled images that have been filtered to include interactions between entities that are known to interact in specific contexts. The dataset is evaluated on a benchmark, BioInteract100, through a variety of perturbations to relational queries. The authors claim that current models show consistency gaps across different paraphrases, and are brittle to relation-direction changes.

Critical strengths
  1. Addressing a data gap: This work addresses a real data gap and is useful in its formulation for testing the relational understanding of different models. The dataset appears well constructed: the filtering protocol, grounding in GloBI records, and manual verification of a sample of images (Appendix D) are all sensible. I'm particularly excited about the potential usefulnesss of the spatiotemporal metadata which could also support more geospatially grounded analyses of biotic interactions beyond the image+language setting.

  2. Template construction is sound: Generating queries from structured triplets rather than with an LLM keeps the perturbations controlled and auditable. The data generation system looks to be well controlled.

Critical weaknesses
  1. Brittleness claims: While Table 5 supports the claim of brittleness of the MLLMs (image-text contradictory accuracy is 52% - 65%), Table 10 seems to show that these models are not brittle to relation-direction changes (77% - 97% accuracy on the same reversed queries). In addition, many reversed queries are ecologically impossible (e.g., a flower visits a bee) and can be rejected without looking at the image. The text-only control does not isolate this effect, since its ground truth is ``No'' for every query.

  2. Geospatial context is not considered: The spatiotemporal metadata is highlighted as a differentiator (Table 1) but is not used in any evaluation. The likelihood of interactions between different entities is greatly determined by their proximity and temporal overlap in activity (timing of flowers blooming vs. bees being active), and this could serve as a useful prior or control.

Questions for authors
  1. How many images and queries were used for the MLLM evaluations, and what the ±\pm in Table 5 is computed over?
  2. In Table 5, Claude Sonnet 4.6 (text-only, Contradictory) has a flip rate of 0.71 alongside an almost perfect accuracy of 0.99. The perfect accuracy suggests consistency in the outputs. Please clarify.
  3. In Table 5, how should the "All" column should be interpreted, given that equivalent and contradictory queries have different correct answers?
  4. With regards to (W1), please clarify whether Tables 5 and 10 were computed on the same images and queries, and how to interpret these results in context of the stated brittleness claim?
  5. Can the authors provide a blind baseline scored with the same labels as the image condition (e.g., "Is it plausible that X pollinates on Y?" with no image), to help separate ecological priors that the MLLMs may know from visual grounding?
  6. Have the authors considered providing the geotemporal metadata (location and date) to the models as additional context? This could help reveal how much ecological prior information the models rely on, and whether such information helps beyond the imagery/text alone.
4 comments

The concerns raised in my initial review remain unaddressed. I would appreciate clarification on these points, particularly regarding the interpretation of the brittleness results and the role of ecological priors.

We thank the reviewer for their constructive feedback and for recognizing BioInteract's contribution to evaluating biotic interactions, the controlled construction of relational queries, and the potential of its spatiotemporal metadata for ecological research. We particularly appreciate the suggestions regarding the interpretation of relation-direction reversal, the role of ecological prior knowledge, and the clarity of our evaluation protocol.

We address each concern below (it took longer to compute our rebuttal with the addressed questions):

Q1

BioInteract100 comprises 281 unique interaction triplets, each with 100 sampled images (28,100 image records). For the MLLM evaluation, we exclude the symmetric interactsWith relation (125 triplets), resulting in 156 interaction triplets and 15.6K images. Each image is evaluated using five non-contradictory query formulations and two direction-reversed formulations, yielding up to 109.2K image–query evaluations per model.

In Table 5, we first compute accuracy independently for each query formulation and then report the mean ± standard deviation across formulations within each group (non-contradictory, contradictory, and all). Thus, the ± values represent variability in accuracy across query formulations; we will clarify the evaluation size and aggregation procedure in the revised manuscript.

Q2

We thank the reviewer for identifying this inconsistency. In Table 5 it is over 15.6K images covering 156 unique interaction triplets with/without images. We confirm that the flip rate of 0.71 reported for Claude Sonnet 4.6 (text-only, Contradictory) in Table 5 is a reporting error.

Below we report these results in detail:

Text-only evaluation:

ModelActive Acc.Passive Acc.Flip ↓Active-reverse Acc.Passive-reverse Acc.Flip ↓
Claude Sonnet 4.60.930.980.070.991.000.01
GPT0.921.000.071.001.000.00
Gemini0.000.070.000.730.970.28

Image–text evaluation:

ModelActive Acc.Passive Acc.Flip ↓Active-reverse Acc.Passive-reverse Acc.Flip ↓
Claude Sonnet 4.60.850.820.070.630.630.00
GPT0.820.710.350.320.330.08
Gemini0.910.850.190.440.460.07

We will correct the erroneous value in Table 5 and clarify the flip-rate calculation and reporting in the revised manuscript.

Q5

To investigate whether MLLMs rely on ecological prior knowledge independently of visual evidence, we conducted an additional blind text-only evaluation using interaction queries from BioInteract100. Specifically, we reformulated the interaction triplets into ecological plausibility questions (e.g., "Is it plausible that species X visits flowers of species Y? Answer with Yes or No and nothing else.") and evaluated the models without providing images. We focused on the ‘visits flowers of’ interaction, covering 123 interaction triplets and 12.3K image records. We report accuracy for active and passive formulations and their direction-reversed counterparts. Flip rate measures prediction disagreement between paired formulations (↓ lower is better):

ModelActive Acc.Passive Acc.Flip ↓Active-reverse Acc.Passive-reverse Acc.Flip ↓
Claude Sonnet 4.60.970.880.131.00.910.08
GPT1.00.960.030.730.310.45
Gemini1.01.00.01.00.040.95

The results show that all three models achieve high accuracy on correctly directed interactions (0.88–1.00), suggesting that they already encode substantial ecological prior knowledge about species–flower visitation relationships. However, their behavior differs considerably for direction-reversed queries. Claude Sonnet 4.6 remains relatively robust, achieving 1.00 and 0.91 accuracy across the two reversed formulations. In contrast, GPT achieves 0.73 and 0.31, while Gemini exhibits a particularly pronounced difference (1.00 versus 0.04), with a flip rate of 0.95.

These findings suggest that although MLLMs can recognize plausible ecological relationships from species identities alone, their judgments can be highly sensitive to linguistic formulation and interaction direction. This reinforces the importance of distinguishing ecological prior knowledge from visually grounded relational understanding. We will incorporate this complementary evaluation into the revised manuscript.

(continued)

Q3

The "All" column in Table 5 aggregates results across all four query formulations: active, passive, active-reverse, and passive-reverse.

For accuracy, each formulation is evaluated against its corresponding ground-truth label: "Yes" for active and passive queries and "No" for direction-reversed queries. We report the mean accuracy ± standard deviation across the four formulations.

For flip rate, we compare the correctness of predictions (0/1) across all four formulations, using the active query as the reference. An instance is counted as a flip if at least one of the other formulations differs in correctness from the active query. Thus, the "All" flip rate measures inconsistency in prediction correctness, rather than changes in the raw Yes/No responses. This distinction is important because contradictory queries intentionally require different answers, and correctly changing from "Yes" to "No" should not be considered inconsistent behavior.

We will clarify this aggregation procedure and the interpretation of the "All" column in the revised manuscript.

Q4

We thank the reviewer for highlighting the apparent discrepancy between Tables 5 and 10. We clarify that the two tables evaluate different image subsets and experimental settings. Table 5 reports MLLM performance on BioInteract100 across meaning-preserving and contradictory query formulations, whereas Table 10 specifically evaluates the 39 images identified during our manual audit (Appendix D) in which the annotated interactions were not visually observable. For these 39 images, we applied the same binary protocol, using the prompt "Does this image show {some query}?", with an expected answer of "No" because the interaction was not visually observable.

Consequently, the higher accuracy in Table 10 does not contradict the lower contradictory-query accuracy in Table 5, as the results are obtained from different image subsets with different expected answers (see Q2,Q5). Table 10 instead investigates whether MLLMs can distinguish documented ecological relationships from interactions actually supported by visual evidence. We agree that this distinction was insufficiently explained in the manuscript and will clarify the evaluation subsets, objectives, and interpretation of both tables in the revised version.

Q6

We acknowledge that spatiotemporal context has not yet been incorporated into our current evaluations, which focus on assessing the capabilities of pretrained vision–language models using image–text representations. Consequently, we have not yet quantified the added value of geographic and temporal information for recognizing biotic interactions. Nevertheless, such metadata could provide valuable ecological priors. Geographic coordinates can help establish whether interacting species occur within compatible geographical ranges and habitats, while observation dates can provide information about seasonal activity and phenological patterns, such as the overlap between insect activity and flowering periods. These signals could help models assess the plausibility of an interaction, particularly when visual evidence is ambiguous or incomplete. An important direction for future work is therefore to compare image–text, spatiotemporal-only, and combined image–text–spatiotemporal settings.

I thank the authors for their response to my questions.

My questions regarding evaluation size, and the conflicts between some tables have been answered (Table 5 v. 10 and the Claude's text-only flip rate) have been addressed.

After reading the other reviews, I agree with Reviewer 6UZM that it remains unclear whether the benchmark measures relational understanding beyond recognizing the species pair. The blind baseline reinforces this concern, since the models already reach 0.88–1.00 accuracy on directed relational queries from only using species name. I am therefore lowering my score to 2. While I'm excited about the dataset and its potential uses, I don't think the paper, or this benchmark is ready for acceptance at this time.

Reviewer_9Pi5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces BioInteract, a dataset of 256K iNaturalist research-grade images covering 15.4K unique biotic interactions among 6.5K taxa. Each interaction is a source–relation–target triplet (e.g., a bee “visits flowers of” a plant). The dataset curation is essentially the authors joining and filtering iNaturalist images and GloBI interaction metadata. The authors then create 5 different types of text “captions” on a 281-triplet subset (BioInteract100). The different captions are created by simple rules like omitting the source or target species, or the relationship, from the original caption. The authors then evaluate how well different CLIP variants retrieve images with these caption variants. They also create contradictory, direction-reversed statements, which they test on both the CLIP models and several VLMs. The VLMs are prompted with a binary image-caption matching question (to the effect of “does this image show this caption?”). In both sets of experiments, the authors mostly state the results as their analysis, and the explanations they do offer are not tested.

Critical strengths
  1. The paper shows how the existing iNaturalist dataset can be joined with existing biological knowledge bases to create species interaction captions, which can then be used to study interactions between different species.
Critical weaknesses
  1. The dataset curation essentially joins and filters existing iNaturalist images and GloBI interaction metadata, filtering by research grade and taxonomic resolution, plus a cross-check against one other GloBI source. There is no image-level quality control beyond a small manual audit. The only task studied is image–text matching, either as text-to-image retrieval with CLIP-style models or as a binary yes/no question for VLMs, and only on a 281-triplet subset (BioInteract100). Why it is consequential: for a dataset and benchmark paper, the dataset and benchmark are the main contribution, and here both add little beyond what existing sources already provide.

2 The five query types that the authors use to benchmark the models are called semantically equivalent, but only the active and passive queries are equivalent. Source-only, target-only and no-relation queries remove information and are therefore not semantically equivalent.

Across both sets of experiments (i.e. text-to-image retrieval with CLIP and binary yes/no questions to VLMs), the authors mostly state the results, and the explanations that they do offer are not tested. For example, lines 446-449 attribute the better performance of no-relation queries for specialized CLIP models (eg BioCLIP) to small organisms and images dominated by environmental context, without any supporting analysis. This explanation is likely wrong, and a more likely explanation is that the BioCLIP variants are trained to align species names with images, not relations with images. Queries containing only the source and target names, without the relationship, are therefore matched better.

Why it is consequential: the paper’s central claim is that VLMs are inconsistent across equivalent queries, but the query set mixes paraphrases with less specific queries. Without tested explanations, the reader learns that models fail but not why. Several previous works have shown that generalist VLMs are bad at biological species related tasks (including all the BioCLIP variant papers, VLM4Bio etc), and an even higher amount of papers have shown that VLMs are bad at domain-specific tasks. Therefore, a finding that these models are bad at predicting/aligning interactions between organisms identified with their scientific names is not unexpected.

Questions for authors
  1. The dataset is heavily skewed toward insect–plant interactions. In Table 7, “interacts with” and “visits flowers of” account for about 84% of the images. Could you state this scope in the abstract and introduction?

  2. How many of the 281 triplets in BioInteract100 belong to each relation type? How does performance differ by relation type? And how much of BioInteract100 is removed when “interacts with” is excluded from the MLLM and direction-reversal evaluations?

  3. How does BioInteract relate to the visual relationship detection and scene graph generation literature, which studies predicting directed (subject, predicate, object) relations between entities in images? Could you also discuss how your findings relate to the general-domain compositionality benchmarks cited in the related works section?

  4. Lines 197–198 contrast BioInteract with datasets focused on recognizing individual species. However, Table 1 and Section 2 identify benchmarks with interaction queries (CoralVQA, AgMMU, INQUIRE) as the closest work. Please contrast Biointeract with these works.

  5. How many interactions or interaction types do the other benchmarks in Table 1 cover?

  6. Could you define the evaluations tasks before the Evaluation section? Currently the two problem settings, text-to-image retrieval with CLIP-style models and binary yes/no verification with MLLMs, have to be inferred from the experimental details.

  7. Does “spatiotemporal annotations/context” refer only to the observation date and GPS coordinates in Table 3? If so, could you clarify the terminology. In the MLLM literature, "spatiotemporal" annotations can suggest spatial annotations such as bounding boxes, and temporal annotations can suggestion annotations of how interactions change over time (for example in videos).

Reviewer_6UZMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BioInteract and BioInteract100, image retrieval and re-ranking benchmark datasets focused on biotic interactions. It curates citizen science imagery from iNaturalist with corresponding source-interaction-target tripets that describe biotic interactions visible in the images. It then evaluates and compares retrieval and re-ranking performance of various vision-language-models using natural language prompts derived from the source-interaction-target triplets. Prompts are generated and perturbed in controlled ways (e.g., dropping source, relation, or target, active or passive formulations) to test how model performance changes in response.

Critical strengths

I find the paper to be overall well-motivated and the dataset setup to be clearly written. I really like the idea of using species interactions as a testbed for jointly probing fine-grained and compositional capabilities of VLMs. The pertubation experiments are a useful tool to examine and quantify shortcomings of existing VLMs.

Critical weaknesses

===Significance of interactions=== The evaluation seems to show that including the interaction within prompts does not consistently improve retrieval performance. However, what remains unclear to me is whether this is because current VLMs are unable to take advantage of this information, or whether the benchmark makes it too easy to perform well by simply identifying the involved species. In other words, are the interactions redundant given the species pair, or do they provide useful information for the ranking problem? This is the most critical weakness, since it affects what we can even learn from the analysis that follows.

===Text-only baselines=== The paper does not make sufficiently clear what the purpose of the text-only baselines is and how exactly they are evaluated. The text says "strong performance in this setting reflects correct detection of absent visual input rather than visual recognition capability", which might imply that the model's goal is to identify that no image was provided. I assume what Table 5 is actually trying to show is how good each model performs a priori (e.g., identifying that "a bee visits a flower" is likely, while "a flower visits a bee" is not). If that's the case, this needs to be made much clearer (and I do think that is a very useful experiment).

===Relation-direction reversal queries=== It is unclear to me what Table 6 and the relation-direction reversal analysis is showing. BioCLIP2's AP@50 drops by about 50% compared to Figure 3, while BioCLIP and SigLIP2 do not seem to change much. Why does BioCLIP2 seem much more impacted by contradictory interactions, yet does not seem to be making use of interactions otherwise? Overall, I think this analysis needs to be much more clear and intuitive. One option might be to quantify changes in similarity scores rather than rankings (which might be noisier, especially if a query is designed to produce no matches and thus all images get similar similarity scores).

===BioCLIP prompting (minor)=== BioCLIP models are primarily trained on taxonomic labels rather than freeform text that includes interactions between two species. It is of course still valid to benchmark them in this way, but I think this simple domain shift is a more plausible explaination for why excluding interactions is advantageous, rather than the explanation currently given in the paper ("This is likely because images in BIOINTERACT100 often depict small organisms, with the visual signal dominated by environmental context providing cues of other relevant organisms.").

Questions for authors
  • How is recall@k computed? I would expect the maximum achievable Recall@10 to be 0.1 (10 matches in the top 10 ranked results, divided by 100 relevant examples), yet Table 6 shows Recall@10 around 0.5.
  • Is there a reason to mix scientific and common names for each model rather than deciding a priori? I would expect CLIP to perform much better using common names and BioCLIP to perform much better using scientific names.
  • There seems to be two conflicting definitions of the flip rate. The text says "if the model produces inconsistent predictions across this set—i.e., at least one prediction differs—we count this as a flip", but section C.2 uses the ground-truth label to define the flip rate. Can you clarify which one is being used?
4 comments

We thank the reviewer for their feedback and suggestions, particularly regarding the extent to which BioInteract evaluates interaction understanding beyond species recognition, the clarity of the dataset design, and the value of using biotic interactions to investigate fine-grained recognition and compositional understanding in vision–language models. We also appreciate the constructive feedback, which will help us further strengthen the manuscript.

We address each concern below.

Distinguishing interaction understanding from species recognition is essential for interpreting retrieval performance. BioInteract was designed to investigate precisely this distinction: whether vision–language models (including domain-spoecific models) capture ecological relationships beyond the identities of the organisms involved. Figure 3 explicitly investigates the contribution of interaction information by comparing retrieval performance across five query formulations, including descriptions with and without the relationship (e.g., bee visits flower vs bee and flowers). Results show that including interactions does not consistently improve retrieval, with the no-relation query often performing better.

Our evaluation focuses on assessing the capabilities of existing pretrained models, rather than their potential to learn ecological relationships. Therefore, the results cannot establish whether interactions are inherently redundant or whether current models lack the necessary relational understanding. Addressing this distinction would require few-shot fine-tuning with explicit interaction supervision, followed by species-controlled evaluation.

In the text-only baseline, we use the same interaction queries as in the image-conditioned evaluation, but without providing an image. This allows us to assess whether models rely on prior ecological and linguistic knowledge rather than visual evidence when answering interaction-related questions.

Table 6 examines the best-performing models to assess whether retrieval is sensitive to query formulation and source–target ordering when interaction direction is reversed. Unlike Figure 3, this evaluation excludes the symmetric interacts with relation (~40%, as highlighted in answer to 9Pi5), resulting in a different and more restricted evaluation subset. Therefore, BioCLIP2's lower AP@50 (0.32 vs. 0.62) primarily reflects performance on a different subset and cannot be attributed directly to contradictory interactions. The similar performance across active and passive formulations suggests limited sensitivity to query structure, although this does not establish whether models distinguish interaction direction.

Biology CLIP-style models are primarily trained on taxonomic information without explicit supervision of ecological interactions between species. However, relational understanding may still emerge implicitly from visual and textual representations learned during pretraining, which our benchmark aims to investigate. The observed results suggest that such capabilities remain limited or are not effectively leveraged by interaction-based queries.

Overall, our goal is to assess the interaction understanding capabilities of existing pretrained models. Few-shot fine-tuning with explicit interaction supervision could potentially improve performance and represents an important direction for future work. In this paper, we deliberately evaluate models in a zero-shot to investigate whether an understanding of interactions emerges from their existing learned representations, without explicit tuning on interaction-specific data.

We address the questions below:

Q1

We thank the reviewer for identifying this inconsistency. Upon revisiting our implementation, we identified an unintentional terminology error: the metric reported as Recall@10 was computed as Hit@10 (Success@10), measuring the proportion of queries for which at least one relevant image appears among the top 10 retrieved results. The metric was correctly computed according to this definition but inadvertently mislabeled in the manuscript. We apologize for the confusion and will correct the terminology throughout the manuscript, explicitly define Hit@10 in the Evaluation section, and ensure consistency between the implementation and reported results. This correction does not change the underlying numerical results or the conclusions drawn from them.

Q2

We evaluate both scientific and vernacular names to assess the influence of terminology across general-purpose and biodiversity-specific models (CLIP-style and LLMs), rather than selecting the most favorable formulation for each model. Results are also reported separately in Appendix E (Table 11).

Q3

Flip rate measures prediction consistency across query formulations, independently of ground-truth correctness. Specifically, a flip occurs when at least one prediction differs among the evaluated formulations for the same image. We will update the definition in Appendix C.2.

I thank the authors for answering and addressing my questions. I still think the setup of the text-only baseline experiment and Table 6 needs to be made much more clear in the manuscript.

Our evaluation focuses on assessing the capabilities of existing pretrained models, rather than their potential to learn ecological relationships. Therefore, the results cannot establish whether interactions are inherently redundant or whether current models lack the necessary relational understanding. Addressing this distinction would require few-shot fine-tuning with explicit interaction supervision, followed by species-controlled evaluation.

My concern is less about fine-tuning models, and more about whether the dataset as-is even supports tracking improvements in relational understanding. In other words, if interactions within the current dataset are purely redundant, improvements in relational understanding by newer models won't be measurable. It should be feasible to probe this using an idealized baseline that "retrieves" images (1) based just on the ground-truth species pair, and (2) based on both species and interaction ground-truth. If these yield significantly different "retrieval" performance, that would support the value of the benchmark for tracking these improvements. If there are no significant differences between these two idealized baselines, then the dataset might need to be changed to support this use case (e.g., by including images showing co-occurrence but no interactions).

We thank the reviewer for clarifying this important methodological concern.

Our text-only baseline replaces the input image with a blank image to investigate whether models recognize the absence of meaningful visual evidence or rely on prior knowledge of species interactions. The results reveal substantial differences across models. Gemini 3 Flash Preview exhibits particularly unstable behavior, achieving only 0.21 overall accuracy with a high flip rate of 0.81, suggesting strong sensitivity to query variations and difficulty handling the absence of visual evidence. In contrast, GPT-5.2 mini and Claude Sonnet 4.6 achieve substantially higher overall accuracy (0.98 and 0.93, respectively). Notably, all three models perform relatively well on contradictory queries, suggesting that they can recognize inconsistent interaction descriptions even without meaningful visual input. Contradictory-query accuracy does not necessarily demonstrate visual relational understanding, particularly with blank images. Instead, it suggests that models can reject contradictory statements based on linguistic information or their interpretation of the missing visual evidence. Further research is needed to investigate whether vision–language models can recognize novel, previously unseen biotic interactions and generalize beyond species relationships encountered during pretraining. In particular, evaluating species-controlled examples with unseen interaction triplets would help distinguish genuine relational understanding from reliance on prior ecological knowledge.

Furthermore, in Table 6 we evaluate whether models capture the directionality of biotic interactions rather than relying solely on species identities, we construct a candidate set containing both the correct interaction description and its direction-reversed counterpart. Specifically, for active queries, we compare the original triplet ((s,r,t)) against the reversed triplet ((t,r,s)). For passive queries, we compare the semantically equivalent passive formulation ((t,r',s)) against its reversed counterpart ((s,r',t)), where (r') denotes the passive form of relation (r). This evaluation requires models to distinguish who performs the interaction and who receives it, while keeping the species identities and relation type fixed. For example, distinguishing a butterfly eats a plant from a plant eats a butterfly cannot be achieved through species recognition alone. By including both descriptions in the candidate set, we assess whether models are sensitive to the semantic roles of the interacting organisms rather than merely recognizing their co-occurrence. As shown in Table 6, performance decreases when evaluating passive formulations compared with active formulations, particularly for BioCLIP2 (AP@50: 0.32 vs. 0.28). These results suggest that current vision–language models remain sensitive to linguistic formulation and face challenges in consistently capturing directed ecological relationships. We exclude the symmetric interactsWith relation because reversing its arguments does not produce a meaningful contradiction.

We will ensure we clearly define the evaluation tasks in the revised manuscript.

We agree that evaluating relational understanding independently of species recognition requires determining whether interaction annotations provide additional discriminative information beyond species identities. To address this, we conducted an additional species-controlled analysis.

From the full BioInteract dataset (256K images; 15.4K unique interaction triplets), we identified species pairs with multiple annotated interaction types and substantial image coverage. The top examples are shown below:

Source speciesTarget speciesInteractionsImagesInteraction types
Danaus plexippusAsclepias syriaca42,875eats, hasHost, interactsWith, visitsFlowersOf
Danaus plexippusAsclepias incarnata41,607eats, hasHost, interactsWith, visitsFlowersOf
Apis melliferaSalvia rosmarinus41,086eats, hasHost, interactsWith, visitsFlowersOf
Bombus fervidusTrifolium pratense4851eats, interactsWith, visits, visitsFlowersOf
Elkalyce comyntasTrifolium repens4718eats, hasHost, interactsWith, visitsFlowersOf
Bombus bimaculatusTrifolium pratense4581eats, interactsWith, visits, visitsFlowersOf
Bombus vagansTrifolium pratense4519eats, interactsWith, visits, visitsFlowersOf
Apis melliferaTaraxacum officinale4287eats, interactsWith, visits, visitsFlowersOf
Adelges tsugaeTsuga canadensis4214eats, hasHost, interactsWith, parasiteOf
Apis melliferaFoeniculum vulgare4214eats, interactsWith, visits, visitsFlowersOf

To further assess whether BioInteract can evaluate relational information beyond species recognition, we conducted a species-controlled retrieval experiment by restricting the candidate pool to a single species pair annotated with four interaction types (eats, interactsWith, visits, and visitsFlowersOf). We selected Apis mellifera and Taraxacum officinale because it provides a relatively balanced distribution across interaction categories.

Table: AP@50 for species-controlled text-to-image retrieval. Active queries include the interaction, while no relation queries contain only the species pair.

InteractionImagesBioCAP ActiveBioCAP No-rel.BioCLIP ActiveBioCLIP No-rel.BioCLIP2 ActiveBioCLIP2 No-rel.
eats900.05320.05990.29910.31860.07860.0916
interactsWith640.04870.05910.03520.04640.03740.0385
visits320.00710.02160.05100.04620.00250.0018
visitsFlowersOf1440.23750.31670.23300.24710.40940.3262

We compared interaction-specific queries (active) against descriptions containing only the species pair (no relation). The results show that interaction information can improve retrieval even when species identities are fixed. For example, BioCLIP2 achieves an AP@50 of 0.409 for visitsFlowersOf, compared with 0.326 without the interaction (+0.083). However, these improvements are not consistent across models or interaction types. BioCLIP and BioCAP generally perform better with species-only queries, suggesting that current pretrained models do not reliably exploit relational information.

We additionally examined retrieval consistency across query formulations. BioCLIP2 achieves a mean Consistency@50 of 0.885, indicating substantial overlap between retrieved images despite changes in interaction wording:

Interaction typeBioCAPBioCLIPBioCLIP2
eats0.660.620.92
interactsWith0.720.620.88
visits0.700.720.88
visitsFlowersOf0.580.620.86
Mean Consistency@500.6650.6450.885

These findings demonstrate that interaction-specific queries can affect retrieval performance independently of species identity, while also highlighting limitations in current models’ ability to consistently exploit this information. We acknowledge that some annotated interactions overlap semantically and that more extensive species-controlled evaluations, including visually verified negative examples, are needed to assess relational understanding more rigorously.

We will include these additional analyses in the revised manuscript and extend them to additional species pairs and also release species pairs benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.