acceptodds
Desk rejected at ICLR 2027

HerBench: Understanding Fine-Grained Visual Recognition in Multimodal Agents through Trait-Level Evaluation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) equipped with active visual tools have significantly advanced fine-grained visual recognition (FGVR) by inspecting local image regions. However, despite access to high-resolution visual evidence, multimodal agents still struggle with reliable specimen identification, leaving a fundamental question unanswered: *Do identification failures arise from selecting the wrong traits, misjudging subtle trait states, or failing to use those states to distinguish candidate categories?* To dissect this visual reasoning chain, we formulate a trait-level evaluation across three dimensions: *trait selection*, *trait judgment*, and *the use of trait evidence for identification*. We introduce **HerBench**, a specialized benchmark linking 851 megapixel-scale herbarium specimens (spanning 239 species) to 5,325 fine-grained trait questions derived from taxonomist-authored dichotomous keys (dKeys). Furthermore, inspired by professional taxonomic workflows, we propose a dKey-guided agentic framework that interleaves active visual inspection with expert key traversal to systematically guide trait selection and evidence integration. Extensive evaluations across nine state-of-the-art multimodal agents reveal a substantial *trait selection gap*: agents frequently answer isolated trait questions correctly yet completely omit or fail to make definite judgments on the same traits during end-to-end identification. Moreover, even when observation targets are explicitly specified, trait judgment accuracy remains modest (25.51%–64.55%). While dKey guidance improves species F1 for eight of nine models, controlled experiments with reference traits demonstrate that its performance gains are substantially amplified when accurate trait states are supplied, highlighting that the true utility of expert guidance depends fundamentally on the fidelity of the agent's perceptual evidence.

Primary area:
datasets and benchmarks
Keywords:
fine-grained visual recognition, multimodal large language models, multimodal agents, benchmark, herbarium specimens, knowledge-guided reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RyjZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HerBench, a benchmark containing 851 high-resolution herbarium specimens from 239 species and 5,325 trait questions derived from expert-written dichotomous keys. It pairs trait question answering with specimen identification to study trait selection, trait judgment, and evidence use. The proposed agent combines image inspection, observation memory, and key traversal. Identification experiments cover nine models, while detailed trait analyses cover seven. Key guidance improves species F1 for eight models, although precision decreases for models with nonzero F1. Reference traits produce larger gains when combined with keys. The authors interpret these findings as evidence that reliable trait perception limits the benefits of expert guidance.

Critical strengths
  1. HerBench links high-resolution specimens with diagnostic traits and expert identification paths. This supports more detailed evaluation than species accuracy alone.

  2. The study evaluates nine models and includes reference traits, oracle taxonomic entries, and beam search. These experiments help examine several possible sources of identification failure.

  3. Reporting coverage, precision, and hierarchical F1 shows that higher species F1 can come with more incorrect predictions. This helps assess the method’s practical value.

Critical weaknesses
  1. Trait labels need stronger validation. The 200-item expert audit reports 83% answerability agreement but no agreement on concrete trait answers. Some evaluated models also help generate labels, leaving label accuracy and independence uncertain.

  2. Report omission does not establish failed trait selection. A trait absent from the final report may still have been inspected. The automatic matching procedure also lacks reported human validation, weakening the interpretation of Figure 3.

  3. Reference traits change more than perception. They provide species-specific observation targets as well as correct values. The large oracle-genus gains also point to entry-selection errors, so the experiments do not isolate perception as the main bottleneck.

  4. Key retrieval depends on reference labels. Appendix B.2 prioritizes specimen-associated key versions selected using the reference species. This could favor the correct species and needs a control where retrieval does not use the specimen label.

  5. The benefit of structured traversal is not isolated. The main baseline is the authors’ unguided agent. A description-only control or retrieval-based agent [a] would clarify whether key traversal explains the gains, especially given the reduced precision.

[a] Junhan Chen et al. Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding. arXiv:2603.03762, 2026.

Questions for authors
  1. Can you report expert agreement on concrete trait answers and repeat the main analyses on independently verified labels?

  2. How often do traits marked as unmentioned in Figure 3 appear in inspection calls or observation logs? Can human review of full trajectories validate the matching procedure and the trait-selection conclusion?

  3. Can you compare reference values for a fixed, independently chosen trait set with the current species-specific list, while controlling the taxonomic entry?

  4. Do the results hold when key versions are selected using only the region and the agent’s proposed taxon, without access to the specimen’s reference species?

  5. How does key guidance compare with description-only or retrieval-based knowledge access under matched budgets? Are the gains stable across repeated runs, and does precision improve at matched coverage?

5 comments
AuthorsOpenReview ↗

Response to W1 (Q1) & W2 (Q2) & W3 (Q3)

W1 / Q1

Trait QA reference states come from the taxonomist-authored dKeys along validated species paths, not from model voting. Qwen3.8-Max and Kimi-K3, with GPT-5.6-Sol resolving disagreements, only label answerability: their responses are reduced to Answerable/Unknown, and their selected states are discarded.

The 200-item blinded expert audit was designed to check visibility, not the expert-authored dKey content, so it was stratified by answerability label and model agreement, with an emphasis on Unknown items. Answerability agreement is 83.0% (166/200), and on the 58 items the expert answered with a concrete state, the answer matches the dKey state in 91.4% (53/58). These 58 items are too few to repeat the main analyses on expert labels, but those analyses rest on the dKey states rather than on model outputs, and end-to-end identification uses no model-derived labels.

Using evaluated models as annotation assistants also has recent precedents: MME-CoT [1] and MMRB [2] build process annotations from GPT-4o drafts that humans review and correct, and both evaluate GPT-4o itself. In HerBench, the reference states are grounded in taxonomist-authored keys, and annotation accuracy is checked by the independent expert audit above.

W2 / Q2

We manually reviewed 36 Baseline trajectories of Qwen3.5-27B, stratified by Figure 3 outcome and covering 134 Figure 3 traits, that is, traits answered correctly in Trait QA. For each trait, we labeled whether the final report and the inspection log, which records every inspection call and its observation, state the reference state, a contradicting state, or no definite judgment. The automatic matching of Figure 3 agrees with these human labels on 94.0% of traits for the report (κ = 0.86), and the same protocol applied to the log agrees on 91.0% (κ = 0.82).

Figure 3 report categories vs. human labels of the inspection log

Figure 3 reportnLog: correctLog: incorrectLog: no definite judgment
Correct2321 (91.3%)0 (0.0%)2 (8.7%)
Incorrect150 (0.0%)11 (73.3%)4 (26.7%)
No definite judgment9614 (14.6%)3 (3.1%)79 (82.3%)

Of the traits marked as unjudged in Figure 3, 17 of 96 are judged in the inspection log, but 82.3% receive no definite judgment in the log either: the agent never assesses them during identification, although it answers them correctly when asked. The trait selection gap in Figure 3 therefore reflects the trajectory, not only the final report. We will add this analysis to the appendix.

W3 / Q3

Reference values for a fixed trait set are not available: a dKey gives a species' states only along its own path, so the other traits would need new expert annotation of every specimen, and filling in only the path traits would reveal the path. We therefore separate targets from values with the entry fixed to the correct genus (Qwen3.5-27B, 300-specimen subset):

  • A (entry only): the Oracle Genus setting of Table 3.
  • B* (fixed trait list): all traits contrasted in the genus key, without states; every specimen of the same genus key receives the same list.
  • B (species-specific list): the traits on the specimen's own key path, without states.
  • C (Reference Traits): the same species-specific list with states.
ConditionSp. F1hF1
A (entry only)52.4981.53
B* (fixed trait list)52.3981.72
B (species-specific list)55.1382.73
C (Reference Traits)94.9894.66
ComparisonAddedΔSp. F1ΔhF1
B* − Afixed list−0.10 [−4.48, +4.65]+0.18 [−1.16, +1.67]
B − Aspecies-specific list+2.64 [−1.34, +6.87]+1.20 [−0.19, +2.69]
B − B*tailoring to the species+2.74 [−2.16, +7.40]+1.02 [−0.58, +2.67]
C − Btrait states+39.85 [+34.44, +45.66]+11.93 [+10.25, +13.67]

Brackets give 95% paired bootstrap confidence intervals.

  • Trait lists add little (B* − A, B − A). Neither the fixed nor the species-specific list changes Sp. F1 (−0.10, +2.64) or hF1 significantly; once the entry is correct, knowing which traits to examine does not help.
  • Tailoring reveals nothing (B − B*). The species-specific list performs like the fixed one (+2.74 Sp. F1, +1.02 hF1, neither significant), so its targets cannot explain the Reference Traits gain.
  • States drive the gain (C − B). With the same list, the states raise Sp. F1 from 55.13 to 94.98 (+39.85) and hF1 by +11.93; what the agent lacks is judging these traits during identification, not knowing where to look.

Entry selection is a separate source of failure (Table 3), but within the correct genus, identification is limited by trait judgment.

AuthorsOpenReview ↗

Response to W4 (Q4) & W5 (Q5)

W4 / Q4

The conclusions of Table 1 hold when keys are selected by region and requested name alone. Such a selection differs from the runtime of Appendix B.2 only in the Australian keys; for every German request, it returns the same key. In Australia, KeyBase provides several regional keys under the same family or genus name (e.g., Flora of Australia, Western Australia, Queensland). When the agent requests the correct name, the runtime returns the regional key that contains the specimen's species (regional key choice). This keeps the evaluation on how agents use a key rather than on whether a regional key covers the species, but it relies on the label.

We replayed every key request in the dKey runs of Table 1 and counted the specimens affected by the case. ΔF1 is the change in species F1 when every affected specimen is counted as a wrong answer.

ModelAffected specimensΔF1 (worst case)
GPT-5.6-Sol6−0.20
Qwen3.8-Max11−0.52
Kimi-K39−0.56
Qwen3.5-27B1−0.01
Qwen3.5-9B1−0.01
Qwen3.5-35B-A3B3−0.02
Gemma4-31B00.00
Gemma4-26B-A4B4−0.19
Gemma4-E4B00.00

Species F1 drops by at most 0.56, and the dKey gain over the Baseline remains for all eight models with a gain. We will add this case in Appendix B.2 and add a label-free lookup mode to the released runtime.

W5 / Q5

We added two controls that give the agent the same dKey knowledge without stepwise traversal, including a retrieval control in the spirit of [a], and compared four conditions with Qwen3.5-27B on a fixed 300-specimen subset:

  • Baseline: no external knowledge.
  • dKey-guided: the agent opens a key and selects one couplet at a time.
  • Description: for a family or genus chosen by the agent, a tool returns all candidate taxa at once, each described by its dKey leads, the contrasting leads it does not satisfy, and the key's terminal description.
  • Retrieval: the agent writes a query from its observations and retrieves the most similar of the same descriptions (Qwen3-Embedding-8B).

All conditions share the tools, the 30-step budget and the thinking mode; neither control uses specimen labels, and the agent chooses the entry itself. Baseline and dKey-guided were each run three times.

Table A: knowledge formats (Baseline and dKey-guided: mean of three runs)

ConditionStepsKnowledge callsCoveragePrecisionSp. F1ΔSp. F1 vs. dKey
Baseline9.8012.648.710.85—
dKey-guided13.54.334.933.417.27—
Description12.23.838.031.617.39+0.12 [−4.53, +4.86]
Retrieval10.32.540.734.419.91+2.64 [−1.86, +7.14]

Table B: repeated runs and precision at matched coverage

RunBaseline Sp. F1dKey Sp. F1ΔSp. F1Precision at matched coverage (Baseline / dKey)
1 (paper)11.3814.68+3.3155.9 / 50.0
28.3118.77+10.4637.8 / 56.5
312.8718.36+5.4952.4 / 56.9
Mean10.8517.27+6.42 [+3.01, +10.04]48.7 / 54.5 (+5.8 [−7.5, +17.8])

Brackets give 95% paired bootstrap confidence intervals. Precision at matched coverage takes dKey's k most confident answers, with k equal to the number of Baseline answers.

  • Knowledge format (Table A). With similar budgets, Description and Retrieval match dKey guidance, so the gain comes from the expert knowledge that dKeys encode; stepwise traversal adds a traceable record of each trait judgment.
  • Repeated runs (Table B). dKey guidance improves species F1 in all three runs (mean +6.42, CI excluding zero), so the gain is stable.
  • Matched coverage (Table B). dKey precision is higher on average (+5.8, not significant), so its top answers are not less accurate; the lower overall precision comes from additional, less confident answers, as with all three knowledge formats.

We will add both controls and report means over repeated runs together with precision at matched coverage.

[1] D. Jiang et al. MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency. ICML 2025.

[2] Z. Cheng et al. Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark. arXiv:2506.04280, 2025.

Dear Reviewer RyjZ,

Thank you again for your review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our response has addressed your concerns. In particular, we added expert agreement on concrete trait answers, human validation of the Figure 3 matching, and controls that separate trait lists from trait states and dKey knowledge from traversal. We would be happy to clarify any remaining points before the discussion closes.

Best regards,

The Authors

Reviewer_7RLkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HerBench, a benchmark designed to evaluate fine-grained visual recognition in multimodal agents. By leveraging taxonomist-authored dichotomous keys (dKeys), the authors connect trait-level question answering with end-to-end species identification. They also propose a dKey-guided agent framework that uses structured taxonomic knowledge to guide visual inspection and step-by-step identification. The experiments show that agents frequently fail to make definite trait judgments during autonomous identification, while dKey guidance improves species-level performance for most models, with its effectiveness depending on the quality of trait evidence.

Critical strengths
  1. The paper addresses a meaningful research problem and offers a useful perspective on fine-grained visual recognition.

I appreciate the paper's attempt to go beyond final identification accuracy by examining three aspects of the agent's recognition process. Connecting trait-level QA with end-to-end identification on the same specimens provides an interesting perspective for understanding where multimodal agents struggle.

  1. A domain-grounded benchmark built on expert knowledge.

HerBench combines real high-resolution herbarium specimens with expert-authored dichotomous keys, linking structured taxonomic knowledge to fine-grained visual trait questions.

Critical weaknesses
  1. Insufficient trajectory-level analysis makes it difficult to identify the actual causes of agent failures.

I noticed that the dataset filtering described in Appendix A.5 only ensures that each morphological decision along the reference path can generate at least one visual-trait question. It does not guarantee that all the necessary diagnostic traits are actually observable in a given specimen image, or that the image contains sufficient evidence for species-level identification. As a result, when an agent fails to identify the correct species, it is unclear whether this reflects a reasonable abstention due to insufficient visual evidence or an actual failure in fine-grained perception or reasoning. A more systematic analysis of agent execution trajectories, together with further validation of the Answerable/Unknown labels and species-level identifiability may be better to understand the underlying causes of these failures.

  1. The dKey retrieval mechanism may depend on ground-truth species labels, potentially affecting the validity of the evaluation.

Appendix B.2 mentions that the retrieval mechanism prioritizes the specimen-associated dKey validated during dataset construction. This suggests that although the agent independently predicts the taxonomic entry, the specific dKey version it receives may have already been selected based on the ground-truth species, potentially making the subsequent identification task easier. I understand that this design might have been intended to simplify the inference process and reduce computational costs, but it raises concerns about the fairness of the evaluation. It would be helpful if the authors could report how often this retrieval preference is triggered, or provide additional results using a retrieval strategy that does not rely on ground-truth labels.

Questions for authors
  1. Does the Reference Traits experiment truly isolate visual perception from taxonomic reasoning?

Similar to the concern in W2, the reference traits are derived from the ground-truth species' dKey path, which may provide useful information for both trait selection and trait judgment. Could the authors clarify how these two effects are disentangled in the current experiments?

  1. How much of the performance improvement comes from external taxonomic knowledge versus structured dKey traversal?

The main comparison is between a baseline without external taxonomic knowledge and the dKey-guided agent. Would it be possible to include a text-based retrieval or RAG baseline that provides the same taxonomic knowledge without structured dKey traversal? This would help clarify the contribution of the structured decision process itself.

5 comments
AuthorsOpenReview ↗

Response W1 & W2

W1

Thank you for your questions. The filtering in Appendix A.5 is not meant to guarantee that every diagnostic trait is visible. Visibility is annotated separately: each Trait QA item is labeled Answerable or Unknown against the image . With these labels, we can separate the three causes of failure:

  • (a) the image lacks the evidence;
  • (b) the evidence is visible, but the model cannot perceive it;
  • (c) the model can perceive the evidence when asked, but does not obtain or use it during identification.

Excluding (a). We evaluate the 351 specimens whose reference-path questions are all Answerable. Every decision on their key path has visible evidence, so no failure on this subset can be a reasonable abstention due to missing evidence.

Table R1: Sp. F1 on all specimens / the Answerable subset (Ans.); QA = Trait QA accuracy on the reference-path traits of failed specimens in the subset

ModelBaseline Sp. F1 (All / Answerable)Baseline Trait QAdKey-guided Sp. F1 (All / Answerable)dKey-guided Trait QA
Kimi-K350.68 / 70.8164.9%52.88 / 69.7765.9%
Qwen3.5-27B9.35 / 19.9567.8%14.20 / 23.4265.6%
Qwen3.5-9B2.31 / 5.4965.2%5.75 / 10.4563.6%
Qwen3.5-35B-A3B4.57 / 10.1666.3%10.85 / 20.5863.9%
Gemma4-31B2.46 / 5.2660.4%4.09 / 7.5359.8%
Gemma4-26B-A4B8.78 / 18.6556.7%10.27 / 20.4554.1%
Gemma4-E4B0.00 / 0.0031.1%0.00 / 0.0031.1%

Scores are higher than on the full benchmark, so missing evidence accounts for part of the failures in Table 1. The failures that remain on this subset are due to (b) or (c).

Separating (b) from (c). In these failures, models answer 54–68% of the reference-path traits correctly in Trait QA (31% for Gemma4-E4B), so most of the required evidence is perceivable. For Qwen3.5-27B, we further check whether the final report judges each perceivable trait, using the report matching of Figure 3. This matching agrees with human labels on 94.0% of traits (response to Reviewer RyjZ, Q2), and traits missing from the report are also missing from the reasoning in 88.5% of cases (response to Reviewer 4gP6, W1). A failure counts as (b) if the report judges every perceivable trait, so the error lies in a wrong state. It counts as (c) if the report leaves a perceivable trait unjudged, or judges every trait correctly but still misses the species.

Table R2: Failed specimens of Qwen3.5-27B in the Answerable subset (perceivable = correct in Trait QA)

Failed specimensBaseline (n = 309)dKey (n = 288)
(b) Perception39 (12.6%)40 (13.9%)
— only non-perceivable traits missed23 (7.4%)23 (8.0%)
— a perceivable trait misjudged in the report16 (5.2%)17 (5.9%)
(c) Evidence seeking or use270 (87.4%)248 (86.1%)
— a perceivable trait unjudged in the report234 (75.7%)213 (74.0%)
— all traits judged correctly36 (11.7%)35 (12.2%)

For 12.6% (Baseline) and 13.9% (dKey) of these failures, the agent judges every perceivable trait and fails on traits it cannot perceive or misjudges. In 75.7% and 74.0%, the report leaves a perceivable trait unjudged. Failures with visible evidence thus mainly reflect the trait selection gap of Figure 3, and dKey guidance does not change this. The answerability labels were audited by a taxonomist (83.00% agreement on 200 items; Appendix A.4). We will add both tables to the appendix.

W2

The mechanism in Appendix B.2 affects only the Australian keys; for every German request, label-based and label-free lookup return the same key. In Australia, KeyBase provides several regional keys under the same family or genus name (e.g., Flora of Australia, Western Australia, Queensland). When the agent requests the correct name, the runtime returns the regional key that contains the specimen's species (regional key choice). This keeps the evaluation on how agents use a key rather than on whether a regional key covers the species, but it relies on the label.

We replayed every key request in the dKey runs of Table 1 and counted the specimens affected by the case. ΔF1 is the change in species F1 when every affected specimen is counted as a wrong answer.

ModelAffected specimensΔF1 (worst case)
GPT-5.6-Sol6−0.20
Qwen3.8-Max11−0.52
Kimi-K39−0.56
Qwen3.5-27B1−0.01
Qwen3.5-9B1−0.01
Qwen3.5-35B-A3B3−0.02
Gemma4-31B00.00
Gemma4-26B-A4B4−0.19
Gemma4-E4B00.00

Species F1 drops by at most 0.56, and the dKey gain over the Baseline remains for all eight models with a gain. We will this case in Appendix B.2 and add a label-free lookup mode to the released runtime.

AuthorsOpenReview ↗

Response Q1 & Q2

Q1

The reference traits in Figure 4 provide both which traits to examine and their states. To separate the two, we compare three conditions with Qwen3.5-27B on a fixed 300-specimen subset. Each condition is run in both the Baseline and dKey-guided settings, and the agent chooses its own entry:

  • Raw: the image only, as in Table 1.
  • B (trait list): the reference trait list without states, i.e., only which traits to examine.
  • C (Reference Traits): the same list with states, as in Figure 4.

B − raw measures the effect of trait selection, and C − B the effect of trait states. B and C differ only in whether the states are given.

SettingMetricraw → B → CB − rawC − B
BaselineSp. F111.38 → 14.58 → 16.54+3.20 [−2.73, +8.87]+1.96 [−3.64, +8.39]
hF131.39 → 40.08 → 39.06+8.70 [+3.97, +13.24]−1.02 [−4.71, +2.83]
dKey-guidedSp. F114.68 → 15.83 → 36.51+1.14 [−3.51, +6.11]+20.69 [+13.71, +27.89]
hF131.10 → 40.13 → 44.12+9.04 [+4.79, +13.51]+3.98 [−0.36, +8.25]

Brackets give 95% paired bootstrap confidence intervals. Knowing which traits to examine raises hF1 by about 9 points in both settings, mostly at the family and genus levels, but does not significantly improve species F1. Supplying the states for the same trait list accounts for +20.69 of the +21.83 species F1 gain under dKey guidance. The conclusion of Section 5.4 therefore holds after separating the two effects: the species-level gain from Reference Traits comes from the trait states, not from the trait list, and the benefit of dKey guidance depends on reliable trait evidence.

Q2

Thanks for this interesting question. We added two controls that give the agent the same dKey knowledge without stepwise traversal, and compared four conditions with Qwen3.5-27B:

  • Baseline: no external knowledge (Table 1).
  • dKey-guided: the agent opens a key and selects one couplet at a time (Table 1).
  • Description: for a family or genus chosen by the agent, a tool returns all candidate taxa at once. Each candidate is described by the dKey leads on its path, the contrasting leads it does not satisfy, and the key's terminal description; candidate order is shuffled.
  • Retrieval: the agent writes a query from its observations and retrieves the most similar of the same candidate descriptions within the chosen taxon (Qwen3-Embedding-8B).

All other settings are identical to the dKey-guided runs, including the tools, the 30-step budget and the thinking mode. Neither control uses specimen labels, and the agent chooses the entry itself.

Qwen3.5-27B on a fixed 300-specimen subset (Baseline and dKey-guided: mean of three runs)

ConditionCoveragePrecisionSp. F1hF1ΔF1 vs. BaselineΔF1 vs. dKey
Baseline12.648.710.8530.54——
dKey-guided34.933.417.2732.84+6.42 [+3.01, +10.04]—
Description38.031.617.3933.82+6.54 [+1.53, +11.81]+0.12 [−4.53, +4.86]
Retrieval40.734.419.9135.38+9.05 [+4.74, +13.73]+2.64 [−1.86, +7.14]

Brackets give 95% paired bootstrap confidence intervals.

  • Same gain as dKey guidance. Both controls improve on the Baseline as much as dKey guidance does (+6.54 and +9.05 vs. +6.42), and neither differs significantly from it. The gain therefore comes from the expert knowledge that dKeys encode, namely the regional candidate set and its contrasting diagnostic traits, which all three formats deliver.
  • Same coverage–precision trade-off. All three knowledge formats raise coverage and lower precision relative to the Baseline, so this trade-off comes with access to the knowledge rather than with a particular format.
  • Same ceiling. However the knowledge is delivered, species F1 stays below 20%. Consistent with our main conclusion, the limiting factor is reliable trait evidence rather than access to knowledge.

We will add both controls to Section 5 in the revision.

Dear Reviewer 7RLk,

Thank you again for your review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our response has addressed your concerns. In particular, we added a trajectory analysis on fully answerable specimens, a worst-case bound on label-based key selection, and controls that separate trait selection from judgment and knowledge from traversal. We would be happy to clarify any remaining points before the discussion closes.

Best regards,

The Authors

Reviewer_4gP6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this work, the authors introduce HerBench, a benchmark containing 851 high-resolution herbarium specimens across 239 species with 5325 morphological trait questions derived from taxonomist-authored dichotomous keys. It evaluates three aspects of identification: selecting diagnostic traits, judging their states, and using that evidence to identify species. The authors propose an agentic framework combining image cropping, persistent trait records, and interactive key traversal. Experiments compare nine models and examine identification with reference traits, correct taxonomic entries, and alternative search strategies. Results show that isolated trait accuracy does not guarantee identification success and that expert keys become substantially more useful when accurate trait states are supplied.

Critical strengths
  • Herbarium identification requires recognizing subtle structures in pressed, damaged, or incomplete specimens. The high-resolution images make this a useful setting for evaluating active visual inspection beyond ordinary image classification.
  • Connecting trait questions and identification on the same specimens provides more insight than a species score alone. The distinction between trait selection, judgment, and evidence use raise valuable research questions that can be tackled.
  • Reference-trait, oracle-entry, and beam-search experiments show several interacting difficulties. The evaluation coverage is also good with coverage, precision, recall, species F1, and hierarchical F1, which also exposes trade-offs that single or aggregate score can hide.
Critical weaknesses
  • Figure 3 shows the traits answered correctly in direct QA and checks whether the same traits receive definite judgments in identification reports. However, an agent may inspect or use a trait without mentioning it in its final report. The framework supports recorded observations and a rationale, neither guarantees a complete account of the model’s reasoning. So, my doubt is that report omissions do not establish that the agent failed to inspect the trait or did not know it was diagnostic. The claim of a visual-attention deficit is stronger than the measurement supports.
  • Table 1 shows that guidance consistently increases coverage, but precision falls for every model. Recall also improves, so the benefit includes additional correct identifications, the strongest models show small species-F1 gains. GPT-5.6-Sol improves from 50.83 to 54.44, while precision falls from 76.90% to 60.69%. Qwen3.8-Max gains only 0.54 F1. Hierarchical F1 decreases for two models. This is a strong weakness for the paper.
  • The recorded species determines a key path, so that species branch provides the answers but it is not guaranteed that their visiblity in the particular specimen is there.
  • The benchmark includes only species covered by the German and Australian identification keys. It does not test what happens when the correct species is missing from those keys.
Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to W1 & W2

W1

We agree that a trait missing from the final report does not by itself show that the model did not inspect it. To test whether traits were judged during reasoning but left out of the report, we repeated the Figure 3 analysis on the full native reasoning of Qwen3.5-27B. Qwen3.5-27B produces reasoning at every step, so its reasoning covers the whole trajectory. We kept the same judge, reference traits and scoring. We sampled 36 specimens stratified by Figure 3 outcome, covering 134 Figure 3 traits (traits answered correctly in Trait QA).

Final Report JudgmentnReasoning: CorrectReasoning: IncorrectReasoning: No Definite Judgment
Correct2322 (95.7%)01 (4.3%)
Incorrect151 (6.7%)13 (86.7%)1 (6.7%)
No Definite Judgment968 (8.3%)3 (3.1%)85 (88.5%)

The reasoning closely matches the report:

  • Judged correctly in the report: 95.7% are also judged correctly in the reasoning.
  • Judged incorrectly in the report: 86.7% are also misjudged in the reasoning.
  • No definite judgment in the report: 88.5% receive no judgment in the reasoning either.

Traits missing from the report were thus, in most cases, never judged during identification, rather than judged and left out of the report. This supports the visual-attention deficit shown in Figure 3. We will add this analysis next to Figure 3 in the revision.

We also validated the judge against human annotation (see our response to Reviewer RyjZ, Q2). On the same 134 traits, the judge agrees with the human annotator on 94.0% of final reports (κ = 0.86) and 91.0% of tool logs (κ = 0.82).

W2

We respectfully disagree that this is a weakness of the paper. The observations the reviewer lists are reported and discussed in Section 5.2, including the coverage–precision trade-off of GPT-5.6-Sol and the cause of the hF1 decrease for two models, and they are consistent with what the paper claims. We do not claim that dKey guidance improves precision. We claim that it reduces omitted traits, improves species F1 for eight of nine models, and that its benefit is bounded by the fidelity of visual evidence (Section 5.4). The modest gains on raw images follow directly from this last point: a dKey turns trait states into an identification, so it can be only as reliable as the states the agent reads from the image, and on raw images these states are often missing or wrong. Supplying reliable states confirms this: the same guidance then becomes far more effective, and the dKey gain for Kimi-K3 rises from +2.20 to +24.92 species F1. The modest gains on raw images therefore reflect a bottleneck in fine-grained visual evidence rather than in key-guided reasoning, which is the main conclusion of our paper.

As for the drop in overall precision, it does not mean that dKey answers are less accurate. Guided by the key, the agent commits to a species answer on many specimens where the Baseline abstains, so the precision values in Table 1 are measured at very different coverage. For a fair comparison, we computed precision at matched coverage: for each model, we take dKey's k most confident species answers, with k equal to the number of Baseline answers.

ModelMatched coverageBaseline precisiondKey precision
GPT-5.6-Sol49.35%76.90%79.86%
Kimi-K356.29%70.35%73.16%
Qwen3.8-Max65.80%72.86%73.91%

At the same coverage, dKey precision is not lower than the Baseline, and is slightly higher for all three models. The lower overall precision in Table 1 therefore comes from the additional, less confident answers that dKey guidance submits, not from less accurate answers. In the revision, we will report precision at matched coverage alongside Table 1.

AuthorsOpenReview ↗

Response to W3 & W4

W3

HerBench does not assume that the traits on the key path are visible on each specimen. The key path determines only the reference state of each trait, and whether that state can be judged from the specimen image is annotated separately (Section 3). Every question includes the option "Cannot determine from the image" to cover missing organs, occlusion or insufficient resolution, and each question is labeled Answerable or Unknown for its specimen. Of the 5,325 questions, 1,593 (29.9%) are labeled Unknown. For these, the correct answer is "Cannot determine from the image", not the state given by the key. Table 1 reports accuracy on the two groups separately, and Figure 3 includes only traits labeled Answerable.

The answerability labels were also audited by a plant taxonomist. For a stratified sample of 200 questions, the expert answered each question from the full-resolution image without seeing the target taxon, the reference answer or the model votes, and agreed with the Answerable/Unknown label on 83.00% of them (Appendix A.4). The species path therefore fixes the reference state, while the answerability label determines whether the specimen supports a visual judgment. We will make this distinction more prominent in Section 3 in the revision.

W4

Restricting HerBench to species covered by the keys follows from our research question and does not weaken the conclusions drawn for these taxa. HerBench asks whether an agent can find, judge and use the visual evidence that taxonomist-authored keys identify as diagnostic. Trait QA is constructed from the validated key path of each specimen, so a species outside the keys has no reference traits and cannot be evaluated in this way. The trait selection gap, the difficulty of trait judgment, and the evidence bottleneck of dKey guidance are all measured on covered taxa, and none of them depends on how agents behave on species outside the keys. The case the reviewer raises is a different question: whether incomplete knowledge misleads the agent. When the correct species is removed, accuracy must fall; the informative outcome is whether the agent then submits a wrong species from the key or abstains. This is robustness to incomplete knowledge, a valuable direction but beyond the scope of this paper.

We nevertheless ran this test as an additional analysis with Qwen3.5-27B. For 150 specimens (DE 88, AU 62), we deleted each specimen's own species from its key and compared the result with the complete key, using the same label-free key lookup and starting from the correct genus.

ConditionCorrect SpeciesOther Species in KeyAbstention
Complete Key32.0%28.0%40.0%
After Deletion0%54.0%46.0%

After deletion, abstention rises from 40.0% to 46.0%, but most specimens (54.0%) are assigned to another species of the key. Of the 48 specimens identified correctly with the complete key, 31 (64.6%) switch to another species and 17 (35.4%) abstain at genus. No answer falls outside the key. Incomplete keys can therefore mislead the agent into a wrong species, and this deletion protocol lets HerBench expose such failures. We will add it to the appendix as an open-set evaluation and state the closed-world scope in the limitations.

Dear Reviewer 4gP6,

Thank you again for your review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our response has addressed your concerns. In particular, we checked traits missing from the report against the agent's full reasoning, compared dKey and Baseline precision at matched coverage, and tested identification with the correct species removed from the key. We would be happy to clarify any remaining points before the discussion closes.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chairs

We sincerely thank the reviewers for their careful reading of our paper and for recognizing the value of HerBench as an expert-grounded benchmark for fine-grained visual recognition.

It is particularly encouraging that Reviewer 7RLk considers our work to address "a meaningful research problem" and to offer "a useful perspective on fine-grained visual recognition," that Reviewer 4gP6 finds the distinction between trait selection, judgment and evidence use to raise "valuable research questions," and that Reviewer RyjZ acknowledges "the value of HerBench."

All three reviewers also highlight the combination of high-resolution specimens with expert knowledge. Reviewer 4gP6 notes that "the high-resolution images make this a useful setting for evaluating active visual inspection beyond ordinary image classification," Reviewer 7RLk describes HerBench as "a domain-grounded benchmark built on expert knowledge" that "combines real high-resolution herbarium specimens with expert-authored dichotomous keys," and Reviewer RyjZ notes that HerBench "links high-resolution specimens with diagnostic traits and expert identification paths."

The design of our evaluation is likewise recognized. Reviewers 4gP6 and RyjZ note that pairing trait questions with identification on the same specimens provides more insight than species accuracy alone, and Reviewer 7RLk appreciates that our analysis goes beyond final identification accuracy. Reviewers 4gP6 and RyjZ further value the reference-trait, oracle-entry and beam-search experiments, together with metrics that expose trade-offs hidden by a single score.

We appreciate the suggestions to further validate our labels and analyses. During the rebuttal, we added expert agreement on concrete trait answers, human validation of the matching behind Figure 3, analyses of the inspection log and reasoning, and an evaluation on specimens whose diagnostic traits are all answerable, together with a trajectory analysis of failures. These addit selection gap reflects the agent'strajectory rather than only its final report**. We also tested identification with the correct species removed from the key, showing that HerBench can exe knowledge. We will incorporate theseanalyses into the revised paper.

We also appreciate the suggestions to examine the dKey runtime and the source of its gains. We replayed every key request to bound the effect of label-basedit lists from trait states with the entryfixed, and added description and retrieval controls with repeated runs and precision at matched coverage. These results show that the gain comes from thys that HerBench curates and validates, and that identification is limited by trait judgment, consistent with the main conclusion of the paper.

We are glad that, during the discussion, all three reviewers acknowledged that most of their concerns have been addressed. Once again, we thank thetive feedback, and we welcome any furtherdiscussion. If the reviewers find that their concerns have been sufficiently addressed, we would be grateful if they could update their scores accordingly

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.