- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HerBench, a benchmark containing 851 high-resolution herbarium specimens from 239 species and 5,325 trait questions derived from expert-written dichotomous keys. It pairs trait question answering with specimen identification to study trait selection, trait judgment, and evidence use. The proposed agent combines image inspection, observation memory, and key traversal. Identification experiments cover nine models, while detailed trait analyses cover seven. Key guidance improves species F1 for eight models, although precision decreases for models with nonzero F1. Reference traits produce larger gains when combined with keys. The authors interpret these findings as evidence that reliable trait perception limits the benefits of expert guidance.
Critical strengths
-
HerBench links high-resolution specimens with diagnostic traits and expert identification paths. This supports more detailed evaluation than species accuracy alone.
-
The study evaluates nine models and includes reference traits, oracle taxonomic entries, and beam search. These experiments help examine several possible sources of identification failure.
-
Reporting coverage, precision, and hierarchical F1 shows that higher species F1 can come with more incorrect predictions. This helps assess the method’s practical value.
Critical weaknesses
-
Trait labels need stronger validation. The 200-item expert audit reports 83% answerability agreement but no agreement on concrete trait answers. Some evaluated models also help generate labels, leaving label accuracy and independence uncertain.
-
Report omission does not establish failed trait selection. A trait absent from the final report may still have been inspected. The automatic matching procedure also lacks reported human validation, weakening the interpretation of Figure 3.
-
Reference traits change more than perception. They provide species-specific observation targets as well as correct values. The large oracle-genus gains also point to entry-selection errors, so the experiments do not isolate perception as the main bottleneck.
-
Key retrieval depends on reference labels. Appendix B.2 prioritizes specimen-associated key versions selected using the reference species. This could favor the correct species and needs a control where retrieval does not use the specimen label.
-
The benefit of structured traversal is not isolated. The main baseline is the authors’ unguided agent. A description-only control or retrieval-based agent [a] would clarify whether key traversal explains the gains, especially given the reduced precision.
[a] Junhan Chen et al. Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding. arXiv:2603.03762, 2026.
Questions for authors
-
Can you report expert agreement on concrete trait answers and repeat the main analyses on independently verified labels?
-
How often do traits marked as unmentioned in Figure 3 appear in inspection calls or observation logs? Can human review of full trajectories validate the matching procedure and the trait-selection conclusion?
-
Can you compare reference values for a fixed, independently chosen trait set with the current species-specific list, while controlling the taxonomic entry?
-
Do the results hold when key versions are selected using only the region and the agent’s proposed taxon, without access to the specimen’s reference species?
-
How does key guidance compare with description-only or retrieval-based knowledge access under matched budgets? Are the gains stable across repeated runs, and does precision improve at matched coverage?
5 comments
Response to W4 (Q4) & W5 (Q5)
W4 / Q4
The conclusions of Table 1 hold when keys are selected by region and requested name alone. Such a selection differs from the runtime of Appendix B.2 only in the Australian keys; for every German request, it returns the same key. In Australia, KeyBase provides several regional keys under the same family or genus name (e.g., Flora of Australia, Western Australia, Queensland). When the agent requests the correct name, the runtime returns the regional key that contains the specimen's species (regional key choice). This keeps the evaluation on how agents use a key rather than on whether a regional key covers the species, but it relies on the label.
We replayed every key request in the dKey runs of Table 1 and counted the specimens affected by the case. ΔF1 is the change in species F1 when every affected specimen is counted as a wrong answer.
| Model | Affected specimens | ΔF1 (worst case) |
|---|---|---|
| GPT-5.6-Sol | 6 | −0.20 |
| Qwen3.8-Max | 11 | −0.52 |
| Kimi-K3 | 9 | −0.56 |
| Qwen3.5-27B | 1 | −0.01 |
| Qwen3.5-9B | 1 | −0.01 |
| Qwen3.5-35B-A3B | 3 | −0.02 |
| Gemma4-31B | 0 | 0.00 |
| Gemma4-26B-A4B | 4 | −0.19 |
| Gemma4-E4B | 0 | 0.00 |
Species F1 drops by at most 0.56, and the dKey gain over the Baseline remains for all eight models with a gain. We will add this case in Appendix B.2 and add a label-free lookup mode to the released runtime.
W5 / Q5
We added two controls that give the agent the same dKey knowledge without stepwise traversal, including a retrieval control in the spirit of [a], and compared four conditions with Qwen3.5-27B on a fixed 300-specimen subset:
- Baseline: no external knowledge.
- dKey-guided: the agent opens a key and selects one couplet at a time.
- Description: for a family or genus chosen by the agent, a tool returns all candidate taxa at once, each described by its dKey leads, the contrasting leads it does not satisfy, and the key's terminal description.
- Retrieval: the agent writes a query from its observations and retrieves the most similar of the same descriptions (Qwen3-Embedding-8B).
All conditions share the tools, the 30-step budget and the thinking mode; neither control uses specimen labels, and the agent chooses the entry itself. Baseline and dKey-guided were each run three times.
Table A: knowledge formats (Baseline and dKey-guided: mean of three runs)
| Condition | Steps | Knowledge calls | Coverage | Precision | Sp. F1 | ΔSp. F1 vs. dKey |
|---|---|---|---|---|---|---|
| Baseline | 9.8 | 0 | 12.6 | 48.7 | 10.85 | — |
| dKey-guided | 13.5 | 4.3 | 34.9 | 33.4 | 17.27 | — |
| Description | 12.2 | 3.8 | 38.0 | 31.6 | 17.39 | +0.12 [−4.53, +4.86] |
| Retrieval | 10.3 | 2.5 | 40.7 | 34.4 | 19.91 | +2.64 [−1.86, +7.14] |
Table B: repeated runs and precision at matched coverage
| Run | Baseline Sp. F1 | dKey Sp. F1 | ΔSp. F1 | Precision at matched coverage (Baseline / dKey) |
|---|---|---|---|---|
| 1 (paper) | 11.38 | 14.68 | +3.31 | 55.9 / 50.0 |
| 2 | 8.31 | 18.77 | +10.46 | 37.8 / 56.5 |
| 3 | 12.87 | 18.36 | +5.49 | 52.4 / 56.9 |
| Mean | 10.85 | 17.27 | +6.42 [+3.01, +10.04] | 48.7 / 54.5 (+5.8 [−7.5, +17.8]) |
Brackets give 95% paired bootstrap confidence intervals. Precision at matched coverage takes dKey's k most confident answers, with k equal to the number of Baseline answers.
- Knowledge format (Table A). With similar budgets, Description and Retrieval match dKey guidance, so the gain comes from the expert knowledge that dKeys encode; stepwise traversal adds a traceable record of each trait judgment.
- Repeated runs (Table B). dKey guidance improves species F1 in all three runs (mean +6.42, CI excluding zero), so the gain is stable.
- Matched coverage (Table B). dKey precision is higher on average (+5.8, not significant), so its top answers are not less accurate; the lower overall precision comes from additional, less confident answers, as with all three knowledge formats.
We will add both controls and report means over repeated runs together with precision at matched coverage.
[1] D. Jiang et al. MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency. ICML 2025.
[2] Z. Cheng et al. Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark. arXiv:2506.04280, 2025.
Dear Reviewer RyjZ,
Thank you again for your review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our response has addressed your concerns. In particular, we added expert agreement on concrete trait answers, human validation of the Figure 3 matching, and controls that separate trait lists from trait states and dKey knowledge from traversal. We would be happy to clarify any remaining points before the discussion closes.
Best regards,
The Authors
Response to W1 (Q1) & W2 (Q2) & W3 (Q3)
W1 / Q1
Trait QA reference states come from the taxonomist-authored dKeys along validated species paths, not from model voting. Qwen3.8-Max and Kimi-K3, with GPT-5.6-Sol resolving disagreements, only label answerability: their responses are reduced to Answerable/Unknown, and their selected states are discarded.
The 200-item blinded expert audit was designed to check visibility, not the expert-authored dKey content, so it was stratified by answerability label and model agreement, with an emphasis on Unknown items. Answerability agreement is 83.0% (166/200), and on the 58 items the expert answered with a concrete state, the answer matches the dKey state in 91.4% (53/58). These 58 items are too few to repeat the main analyses on expert labels, but those analyses rest on the dKey states rather than on model outputs, and end-to-end identification uses no model-derived labels.
Using evaluated models as annotation assistants also has recent precedents: MME-CoT [1] and MMRB [2] build process annotations from GPT-4o drafts that humans review and correct, and both evaluate GPT-4o itself. In HerBench, the reference states are grounded in taxonomist-authored keys, and annotation accuracy is checked by the independent expert audit above.
W2 / Q2
We manually reviewed 36 Baseline trajectories of Qwen3.5-27B, stratified by Figure 3 outcome and covering 134 Figure 3 traits, that is, traits answered correctly in Trait QA. For each trait, we labeled whether the final report and the inspection log, which records every inspection call and its observation, state the reference state, a contradicting state, or no definite judgment. The automatic matching of Figure 3 agrees with these human labels on 94.0% of traits for the report (κ = 0.86), and the same protocol applied to the log agrees on 91.0% (κ = 0.82).
Figure 3 report categories vs. human labels of the inspection log
Of the traits marked as unjudged in Figure 3, 17 of 96 are judged in the inspection log, but 82.3% receive no definite judgment in the log either: the agent never assesses them during identification, although it answers them correctly when asked. The trait selection gap in Figure 3 therefore reflects the trajectory, not only the final report. We will add this analysis to the appendix.
W3 / Q3
Reference values for a fixed trait set are not available: a dKey gives a species' states only along its own path, so the other traits would need new expert annotation of every specimen, and filling in only the path traits would reveal the path. We therefore separate targets from values with the entry fixed to the correct genus (Qwen3.5-27B, 300-specimen subset):
Brackets give 95% paired bootstrap confidence intervals.
Entry selection is a separate source of failure (Table 3), but within the correct genus, identification is limited by trait judgment.