acceptodds
Under review as a conference paper at ICLR 2027

WildBioWiki: Benchmarking Knowledge Use in Wildlife Question Answering with Controlled Inputs

Abstract

Wildlife question answering links animal identity to biological knowledge, but aggregate scores leave the benefits of supplied information and the sources of task difficulty unresolved. WildBioWiki aligns images with fixed species-level questions and evidence to support controlled comparisons. It contains 264,563 images of 392 vertebrate species, species-aware captions, and 3,136 unique question–answer pairs, inducing 2,116,504 image-conditioned QA instances. Across twelve zero-shot models, the best species-macro Acc@65 is 54.6%. Matched comparisons show benefits from correct names and further gains from retained knowledge. On balanced claims from 100 species, supplying identity and knowledge jointly raises decision-and-explanation accuracy by 44.0–55.5 percentage points across three models; human review on 20 species supports these gains. Matching required facts leaves an additional combination penalty unresolved, while human checks expose local errors hidden by high automatic agreement. WildBioWiki supports measurement of information benefits alongside checks on what benchmark scores establish.

Primary area:
datasets and benchmarks
Keywords:
Biodiversity benchmark, Wildlife question answering, Vision-language models, Knowledge-grounded evaluation, Controlled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_yG5JOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WildBioWiki, a species-centered benchmark for wildlife question answering. It links 264,563 images of 392 vertebrate species to 3,136 unique species-level question–answer pairs, supporting evidence, and image-specific captions. The benchmark enables controlled comparisons that vary access to images, species names, and knowledge while keeping questions and reference answers fixed. The authors evaluate twelve zero-shot models and conduct targeted analyses using balanced claims, matched facts, and human review.

Critical strengths

The dataset comprises 392 vertebrate species, 264,563 images, and 3,136 distinct species-level question–answer pairs covering species identity, habitat, diet, distribution, and behavior. The reference answers retain supporting text and provenance information, facilitating verification of their evidential basis. This resource provides a foundation for studying question answering that integrates visual recognition with biological knowledge.

Critical weaknesses

(1) The manuscript contains formatting and cross-referencing issues, including unresolved “Appendix ??” references in the Related Work and Reproducibility Statement sections.

(2) I recommend redrawing Figure 1, as many text elements extend beyond the figure boundaries.

(3) Parts of the manuscript’s writing and figure preparation rely on AI assistance. The writing lacks clarity, and the manuscript’s overall structure is poorly organized.

Questions for authors

(1) The paper should provide a more detailed discussion of existing related datasets. A comparison table should also be included to clarify the advantages of the proposed dataset.

(2) Was the entire dataset verified by domain experts? Were the reference answers cross-checked by multiple reviewers? If manual verification was conducted, what was the level of inter-rater agreement?

(3) The dataset primarily covers common vertebrate species with well-documented biological knowledge. Do the reported findings also hold for rare species and species that are easily confused? Additional generalization experiments on long-tail species and complex field scenes would help establish the scope of the conclusions.

(4) The dataset is the paper’s main contribution, yet only a small subset is provided in the appendix. I recommend that the authors make the complete dataset publicly available.

Reviewer_s8NNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce WildBioWiki, a species-centered benchmark that connects 264,563 images of 392 vertebrate species to 3,136 unique species-level question–answer pairs and retained evidence, alongside image-specific silver captions. The aligned records support matched comparisons that vary access to images, species names, and knowledge while preserving the biological target. They establish broad baselines with twelve zero-shot models. In addition they perform human checks.

Critical strengths
  • The idea of keeping the question and answer fixed while changing the information given to the model is useful.
  • Supporting evidence is kept for every answer, so the references can be checked.
  • The statistics are careful (species-level bootstrap, human check of the judge), and the authors are honest about limitations.
  • The code and model settings are shared for reproducibility.
Critical weaknesses
  • The writing is dense and heavily hedged (“does not establish,” “leaves unresolved”), making the main contribution hard to see. “Appendix ??” appears twice, Figure 1 is too small to read, Figure 3 mostly repeats class proportions, and Figure 4 is hard to read with all models and contrasts overlaid.
  • The benchmark is narrow and likely easy: 8 templated questions, taken from Wikipedia, for common and well-documented animals. These species probably appear heavily in pretraining data, and no overlap test is done.
  • Novelty over Encyclopedic-VQA and InfoSeek is limited. Both already link images to Wikipedia evidence and test with oracle evidence. There is no comparison showing what WildBioWiki reveals that they don’t.
  • The main findings are almost obvious. In the KB-oracle condition, the model is given the same text the answers were written from, so a big score jump is expected. The authors admit this is not a retrieval result. The same applies to Balanced Q8 (51.5 → 99.5), where the evidence states the answer directly (Figure 5).
  • It is unclear how much the benchmark really needs images. Questions are about species facts, and giving the species name adds only 5–10 points. The paper does not show that the 264K images or the 2.1M “induced instances” add anything beyond 3,136 text questions plus a species label.
  • The caption judge is text-only, so it cannot check whether captions match the image.
  • Captions are listed as a contribution but barely analyzed.
  • There is no retrieval baseline, so knowledge access is only tested with the answer text handed over. The species-disjoint split has no supervised baseline either.
Questions for authors

Please address the weaknesses above.

Reviewer_KpAzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces WildBioWiki, a species-centered benchmark for wildlife question answering that links 264,563 images from 392 vertebrate species to 3,136 unique species-level question–answer pairs, supporting evidence, and image-specific silver captions. By assigning the same eight questions to multiple images of each species, the dataset produces approximately 2.1 million image-conditioned QA instances while maintaining shared biological reference targets.

The main contribution is a set of matched evaluation protocols that vary access to images, species identities, and external knowledge while keeping the underlying questions and reference answers fixed. The authors evaluate twelve zero-shot models and conduct targeted experiments on three models to examine information dependence, balanced claim verification, and the effects of combining multiple biological facts. Additional human evaluations assess reference quality and agreement with automatic scoring.

Critical strengths

1. Structured alignment between images and biological knowledge.

The benchmark provides a connection between wildlife images, species-level questions, reference answers, and supporting evidence. This organization facilitates controlled comparisons under different information conditions without reconstructing the evaluation targets. The distinction between unique question–answer pairs and image-conditioned instances is also clearly documented.

2. Attention to evaluation reliability and limitations.

The authors conduct reference-quality audits, distinguish different evaluation outcomes, and use species-level aggregation and bootstrap intervals to account for repeated questions across images. The balanced claim-verification and matched-fact experiments also help identify limitations in the original task definitions. These efforts improve the interpretability of the reported results and are valuable aspects of the work.

Critical weaknesses

1. Limited benchmark novelty and insufficient diversity of evaluation targets.

Although WildBioWiki contains more than 264,000 images and approximately 2.1 million image-conditioned QA instances, the actual number of unique species–question pairs is only 3,136, with eight fixed questions per species. The authors correctly distinguish these quantities and account for repeated observations in their statistical analysis. Nevertheless, the diversity of knowledge targets and question formulations remains limited relative to the apparent scale of the benchmark.

This concern is relevant given existing work such as InfoSeek, Encyclopedic-VQA, and VLM4Bio, which already investigate knowledge-intensive visual question answering, image–entity–knowledge alignment, or biologically relevant visual tasks. WildBioWiki introduces a more specialized ecological question schema and matched input protocols, but the paper does not sufficiently establish what new evaluation capabilities or scientific insights these choices enable beyond existing benchmarks.

The limited task complexity further weakens this distinction. Section 6.1 reports that 84.2% of Q7 questions are single-fact lookups, while none requires deriving a relation beyond the stated facts. The matched-fact study also produces near-ceiling automatic accuracy when knowledge is supplied, without establishing a statistically reliable additional penalty for combining facts. Moreover, the large collection of images is not fully exploited in the controlled evaluations. The main targeted experiments use only one image per species for 100 species, leaving limited analysis of whether different visual observations of the same species affect recognition, knowledge retrieval, or answer reliability. Such an analysis would more directly show the value of linking many images to shared biological targets.

Overall, the dataset organization is useful, but I find its novelty and evaluation depth insufficiently established for a major benchmark contribution.

2. Limitations in the validity and informativeness of the core evaluation protocol.

The paper identifies several limitations in its original eight-question schema, but these remain relevant to the default benchmark.

Most notably, Section 5.3 reports that 368 of the 392 original Q8 references (93.9%) begin with No/False. Although Q8 also requires an explanation, this extreme answer imbalance raises concerns about whether the original verification task adequately distinguishes factual verification from response tendencies. The authors introduce a balanced-Q8 extension to address this issue, but it is evaluated on a smaller subset of 100 species and remains separate from the original broad-evaluation protocol. Consequently, the default benchmark retains an important limitation that the authors themselves identify.

The automatic evaluation procedure also warrants further scrutiny. The primary metric relies on a text-only LLM judge comparing model responses against reference answers or silver captions. While the authors provide human validation, the reported binary agreement for the broad evaluation is 81.7% on QA and 86.4% on captioning. These results offer some support for aggregate comparisons, but they do not fully establish the reliability of individual rubric scores or closely ranked models. In particular, text-only reference agreement cannot independently verify whether generated captions are visually faithful to the source images.

The dataset construction also relies on generated annotations and source-derived species labels. Although reference audits are reported, the visual filtering procedure does not independently verify fine-grained species identities. This is relevant because the validity of the species-level QA targets depends on the correctness of the image-to-species association.

These concerns leave important questions about the reliability, difficulty, and practical usefulness of the benchmark as a reusable evaluation resource.

Minor Presentation Issues.

The manuscript contains unresolved references such as "Appendix ??" in the Related Work and Reproducibility Statement. The organization of the original baseline, revised diagnostic tasks, and scoring versions also requires considerable effort to follow.

Questions for authors

1. Benchmark novelty and evaluation depth.

Compared with existing benchmarks such as InfoSeek, Encyclopedic-VQA, and VLM4Bio, what specific evaluation capability does WildBioWiki enable that cannot be studied using existing image–knowledge resources? Beyond the fixed ecological question schema and matched input conditions, what new scientific insights are supported by the dataset? Given the large number of images per species, could the authors provide an analysis of performance variation across different images of the same species?

2. Validity of the original question schema.

Given that 93.9% of the original Q8 references begin with No/False, and that the Q7 audit identifies predominantly direct fact-retrieval questions, how should the original eight-slot benchmark be interpreted as a measure of broader wildlife question-answering capabilities?

Do the authors intend the balanced-Q8 protocol to replace the original Q8 in future benchmark evaluations? If not, how should future users handle the substantial label imbalance and the identified limitations in task complexity?

3. Evaluation reliability and resource validation.

Could the authors provide more detailed evidence about the reliability of the broad LLM-judged scores, particularly for comparisons between closely performing models? How sensitive are the reported results to evaluation using human judgments or alternative scoring procedures?

For caption evaluation, could the authors clarify how visually correct descriptions that differ from the generated silver references are handled, given that the automatic judge does not access the source images?

Finally, is it possible to provide a larger representative evaluation subset, including images, species labels, questions, and reference annotations, to support independent validation of the benchmark during the review period?

Reviewer_V3NGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

WildBioWiki aligns 264,563 images of 392 vertebrate species with 3,136 species-level QA pairs, supporting evidence, and image-specific silver captions. Fixed questions are reused across species images to compare answering under different access to images, species names, and knowledge records. Twelve-model zero-shot baselines are supplemented by three-model studies on 100 species, including balanced claim verification, matched-fact questions, and human checks. Results support benefits from supplying identity and evidence, while an additional fact-combination penalty remains unresolved. The contribution is a reusable resource and controlled-input protocol for species-centered knowledge QA.

Critical strengths

Shared questions, references, and evidence enable matched changes in information access. Explicit annotation counts and species-aware aggregation account for repeated species-level targets.

Balanced claims and separate decision/explanation scoring address key evaluation biases. Human checks support identity-and-evidence benefits for all three tested models, while the authors appropriately leave the combination penalty unresolved.

Critical weaknesses

Source labels and Grounding DINO filtering do not establish that each photograph supports species identification. Where plausible species imply different answers, ambiguity can affect ecological QA; one classification error may also cause multiple QA failures. Expert identifiability audits and classification-conditioned analysis would clarify these effects, without assuming that every ecological question requires species identification.

Reported binary human agreement does not validate the five-band scoring or model-level comparisons. Auditors' biological expertise is insufficiently documented, and targeted audits do not calibrate the broad scores. Expert calibration across models and question types remains necessary.

Habitat answers may be inferred from scenery without identifying the animal. Shuffled-image controls change both animal and background, leaving their contributions unresolved; animal-only and background-only comparisons by question type would clarify what drives performance. Background use can be valid, but does not by itself establish species-level knowledge. Separately, the broad baseline retains Q8's 93.9% negative-answer prior despite the balanced extension; excluding Q8 would test its effect on model comparisons.

Questions for authors

What fraction of images can biological experts identify to species level from the model-visible input, and how are ambiguous cases scored?

What biological expertise did the auditors have, and how well do their judgments agree with the five-band scores across models and question types?

How does ecological QA performance differ between correctly and incorrectly identified species, and how does it compare with a classification-plus-knowledge baseline?

How does performance change with animal-only versus background-only inputs, particularly for habitat questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.