acceptodds
Under review as a conference paper at ICLR 2027

AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

Abstract

Visual grounding, the task of localizing objects described by natural-language expressions, is a foundational capability for agricultural AI systems, enabling applications such as selective weeding, disease monitoring, and targeted harvesting. Reliable evaluation of agricultural visual grounding remains challenging because agricultural targets are often small, repetitive, occluded, or irregularly shaped, and instructions may refer to one, many, or no objects in an image. Evaluating this capability therefore requires jointly testing localization accuracy, target-set completeness, and existence-aware abstention. To address these challenges, we introduce AgroVG, a multi-source benchmark that formulates agricultural grounding as generalized set prediction: given an image and a referring expression, a model must return all matching target instances or abstain when no target is present. AgroVG contains 10,071 annotation-grounded image-query pairs from ten source datasets across six target families: crop/weed, fruit, wheat head, pest, plant disease, and tree canopy. It supports bounding-box grounding (T1) across all six families and instance-mask grounding (T2) on sources with reliable instance-level pixel annotations, with queries covering single-target, multi-target, and target-absent regimes. AgroVG further provides task-specific protocols for box-set matching and query-level mask coverage. Zero-shot evaluation of 26 model configurations spanning closed-source MLLMs, open-source VLMs, and specialized grounding systems reveals persistent gaps: the best multi-target Set- reaches only 0.35, and the best positive-query mask success rate at [email protected] remains below 0.17. Data and code are available at https://anonymous.4open.science/r/AgroVG-5172/.

Primary area:
datasets and benchmarks
Keywords:
Agricultural Visual Grounding, Benchmark Evaluation, Vision-Language Models, Generalized Grounding, Multi-Target Grounding, Instance-Mask Grounding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SfZqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for agricultural visual grounding constructed using existing datasets covering 6 target families (crop, fruit, wheat head, pst, plant disease, tree canopy). The benchmark contains 10k image query pairs. They evaluate bounding box grounding and mask grounding. queries can include single, multi targets or no targets at all. The authores present a pipeline for annotation, quality control, split aware query genration. and evaluation. The paper benchmarks 26 model configs, under zero shot. The multi target F1 is 0.35 and T2 success rate 0.75 and The results show the current models not good at multi target. Primary contribution is a unified grounding benchmark and systematic evaluation of existing models, not a new learning algorithm

Critical strengths

this paper addresses relatively underexplored problem in agricultural VL understanding. existing benchmarks primary emphasize regonition, VQA and reasoning, crop or weed grounding. AgroVG extends the evaluation across several target families and combines bounding box and mask grounding. I consider this a good contribution because agricultural images contain dense, similar, small and irregular targets that is not covered by traditional grounding benchmarks. This standardized eval could support research.

The evaluation covers 26 model configurations and reports results for single target, multi target and target absent as well. The results show model differences and trade off btween positive target localization and abstention. These diagnostis are more detailed than a single accuracy number.

Critical weaknesses

Limited novelty: The generalized set prediction formulation is already established in prior works and agricultural grounding also invesitaged before. Papers novelty therefore lies in dataset integration, annotation processing and expand eval coverage. The benchmark relies on template generated queries. The authors provide human paraphrase eval. But I am not convinced that the benchmark adequately represents the complexity of natural insturctions in agricultural applications.

Questions for authors
  1. How do the authors ensure that template generated queries adequately measure natural language grounding rather than recognition of a limited set of category patterns ? Can the authors provide results on a bigger human written query subset, preferably covering both T1 and T2 ?

  2. Appendix C.2 indicates that output normalization can change reported results. How much of the observed difference in models is caused by coordinate conventions, parsition failures, or model limitations rather than localization ability? Can the authors report the results using documented recovery procedure and distinguish explicit abstention from invalid or unparseable predictions ?

  3. How were the 10 experts selected, trained and assigned to the annotation review ? Was the inter annotator agreement measured?

3 comments
AuthorsOpenReview ↗

Response to Reviewer SfZq

We thank the reviewer for the constructive feedback.

W1: Novelty and contribution.

AgroVG differs from existing agricultural detection/segmentation datasets and crop/weed-focused grounding resources in two key respects:

  • Query–target construction. We reorganize heterogeneous source annotations into a common instance schema, preserve target identities, and construct explicit query-to-instance mappings with expert quality control. This transforms category-based detection and segmentation annotations into image–query–target records, enabling evaluation against the specific target set selected by each query.
  • Broader, unified evaluation. AgroVG provides unified evaluation of bounding-box and query-level mask grounding across diverse agricultural target families. Both tasks include single-target, multi-target, and target-absent queries, supporting diagnostics of box-set completeness, foreground coverage, and empty-output correctness.

The revision clarifies these distinctions; family-wise results are in Appendix D.1.


Q1: Natural-language grounding beyond templates

We evaluated 500 natural-language queries per task, written and reviewed by domain experts and paired with the original queries. Images, target identities, ground truth, and inference/evaluation settings remain fixed; only wording changes, without additional tuning. The subsets cover six T1 and three T2 target families.

Scores follow the original protocol: T1 positive-query [email protected] and T2 query-level mIoU. Differences are natural − original, computed before rounding; 95% CIs use 5,000 source-stratified paired acquisition-group bootstrap replicates.

Task / ModelOriginalNaturalDifference [95% CI]
T1/InternVL3.5-8B0.36870.3563−0.0123 [−0.0358,+0.0119]
T1/Qwen3-VL-8B0.23770.2306−0.0071 [−0.0305,+0.0167]
T1/GPT-5.40.14550.1306−0.0149 [−0.0389,+0.0085]
T1/OWLv2-Large0.13610.1033−0.0328 [−0.0501,−0.0156]
T1/GPT-4o0.01110.0120+0.0009 [−0.0099,+0.0113]
T2/LISA-7B0.31330.3289+0.0156 [−0.0032,+0.0346]
T2/SAM30.08560.0788−0.0069 [−0.0316,+0.0166]
T2/Grounded-SAM0.27810.2224−0.0557 [−0.0821,−0.0292]

The four T1 MLLMs retain their ranking with aggregate changes within 0.015, though GPT-4o remains near the performance floor. OWLv2 and Grounded-SAM show clearer declines, indicating sensitivity to natural phrasing.

These paired evaluations extend beyond fixed templates and quantify model-dependent wording sensitivity with visual targets held constant.


Q2: Output recovery and coordinate sensitivity

We audited cached outputs for 12 auditable T1 configurations on 1,288 original test queries, without rerunning models.

Confirmed correction: the InternVL3.5-8B adapter incorrectly divided two [0,1] outputs by 1,000. The fix raises All-F1 from 0.3963 to 0.3966.

Coordinate-policy sensitivity: Qwen and InternVL3.5-38B inferred units from image size. Their unlabelled outputs leave the intended units ambiguous, so the documented 0–1000 alternative is evaluated as sensitivity rather than a verified correction for every response.

Model[email protected][email protected]E-Acc
Qwen3-VL-32B0.3748→0.46510.2955→0.40320.8090→0.8040
Qwen3-VL-8B0.3271→0.37960.2244→0.28730.8894→0.8844
InternVL3.5-38B0.3978→0.43310.3318→0.37440.7588→0.7538
Gemini 2.5 Pro0.1849→0.19760.1498→0.19140.3769→0.2312
DeepSeek-VL2-small0.3191→0.32720.2765→0.28780.5528→0.5427
InternVL3.5-8B0.3963→0.39660.3622→0.36260.5829→0.5829

InternVL3.5-8B changes only through the confirmed fix; DeepSeek-VL2-small through syntax recovery; other rows include coordinate sensitivity. Six other audited configurations are unchanged. Under the alternative policy, Qwen3-VL-32B leads in overall F1 and multi-target F1.

Gemini’s 75 submitted correct-empty credits comprise 43 explicit empty JSONs, one verbal absence, 29 normalization fallbacks, and two blank responses. Recovery removes the 29 fallback credits; both blank credits remain. E-Acc measures pipeline-level empty-output correctness, not necessarily intentional abstention. Remaining errors mix recognition, language understanding, target selection, generation limits, and localization.

Main results retain the original protocol with only the confirmed InternVL3.5-8B correction. Alternative coordinate policies and output recovery appear as appendix sensitivity analyses.


Q3: Expert review and inter-annotator agreement

Ten reviewers spanning agriculture and computer science were selected for prior annotation experience and relevant research contributions. Before review, the authors trained them on the annotation workflow and platform.

Each candidate image was reviewed by one expert using normalized box or mask overlays to check image integrity, annotation validity, alignment, and target correspondence. No independent overlapping review or separate adjudication was conducted; inter-annotator agreement was not measured.

Thank you for the detailed info and additional exp. The results address several of my main concerns and make the evaluation more convincing. While I still have some concerns about novelty and how model outputs are handled, I believe the benchmark can be useful to the research community. I am updating my rating to Weak Accept.

We have clarified that AgroVG’s contribution lies in query–target construction and unified evaluation across agricultural target families, building on established grounding formulations.

For output handling, the main results retain the original protocol with the confirmed InternVL3.5-8B adapter correction. Alternative coordinate policies and output recovery results are reported separately as sensitivity analyses, making their effects on model comparisons explicit.

We sincerely thank you for your time, careful reassessment, and constructive feedback. Your comments have helped us clarify AgroVG’s contribution and strengthen the evaluation.

Reviewer_XG2TOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgroVG, an agricultural visual grounding benchmark with 10,071 image–query pairs from ten datasets across six target families. It covers single-target, multi-target, and target-absent queries through box grounding (T1) and mask grounding (T2), with T2 covering three families. Construction combines annotation normalization, expert review, and deterministic, annotation-grounded query generation. Evaluation of representative close-source/open-source MLLMs and specialized grounding models reveals limitations in multi-target completeness, mask coverage, and abstention.

Critical strengths
  1. The paper proposes a useful benchmark for developing AI tools for agriculture. AgroVG combines multiple agricultural sources and six target families with single-target, multi-target, and target-absent queries, and evaluates localization, target-set completeness, and abstention, providing a practical framework for diagnosing grounding limitations.

  2. The paper provides thorough validation, including mask audits, source- and query-level breakdowns, output-handling analyses, and paired bootstrap comparisons, which strengthen the empirical contribution. Explicit discussion of ranking uncertainty and coverage limitations also supports careful interpretation.

Critical weaknesses

T2 terminology should match its evaluation. Section 3.1 defines instance-mask sets, whereas evaluation measures query-level foreground coverage over their union. This is reasonable for heterogeneous model interfaces, but does not separately assess instance separation or cardinality accuracy. Aligning the task definition and discussion would clarify its scope.

Questions for authors

I think the paper is generally rigorous and comprehensive in its evaluation. Still, the interpretation of the abstention results could be framed more clearly.

  1. The paper explains that unsuccessful outputs may become empty predictions, and Table A11 shows that output recovery can affect the performance of target-absent queries. Readers may interpret the main E-Acc results as a direct measure of intentional abstention. Therefore, a brief note would help distinguish performance under the evaluation settings from model behavior.

  2. The overall T1 Set-F1 includes target-absent queries, where an empty prediction receives full credit. Could the paper report the score of a trivial baseline that always outputs an empty set? This would give a reference point for the headline results and clarify how much of each model’s score comes from abstention versus localization ability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XG2T

We thank the reviewer for the positive assessment and constructive suggestions.


Weakness: Consistency between T2 terminology and evaluation.

We agree that the terminology should reflect the evaluation scope. T2 uses instance-level annotations to identify the queried targets, but evaluates their union as a query-level foreground mask. We have revised Section 3.1 and the corresponding terminology throughout the manuscript to describe T2 as query-level mask grounding, explicitly clarifying that the current metrics do not separately assess instance separation or cardinality accuracy.


Question 1: Interpretation of E-Acc and intentional abstention.

We agree that E-Acc measures empty-output correctness under the evaluation pipeline, rather than directly measuring intentional abstention. Missing or unparsable outputs can be normalized to empty predictions and receive correct-empty credit on target-absent queries.

Table A11 of the submission quantifies this sensitivity: output recovery increases Gemini 2.5 Pro's positive-query Set-F1 by 0.0423 while reducing E-Acc by 0.1457. We have added this caveat alongside the main metric definitions and result tables, and revised the discussion to distinguish observed empty-output behavior from intentional abstention.


Question 2: Always-empty baseline and score decomposition.

We evaluated an always-empty predictor using the released T1 evaluator. The test split contains 1,089 positive and 199 target-absent queries. Since this predictor receives full credit only on target-absent queries, its overall Set-F1 is 199/1,288 = 0.1545 at both IoU thresholds. Its positive-query Set-F1, single-target accuracy, and multi-target Set-F1 are all zero, while its E-Acc is 1.0.

The table below reports overall and positive-query Set-F1 at IoU 0.5. The final column shows the contribution of target-absent queries to the overall score, calculated as the number receiving correct-empty credit divided by all 1,288 queries. This is a weighted score contribution, not E-Acc itself. All scores follow the original evaluation protocol.

Model[email protected][email protected]Empty contribution
Always-empty0.15450.00000.1545
GPT-4o0.15050.03200.1234
GPT-5.40.27360.20140.1033
Gemini 2.5 Pro0.18490.14980.0582
Claude Sonnet 4.60.19950.14510.0769
DeepSeek-VL2-tiny0.22270.08530.1506
DeepSeek-VL2-small0.31910.27650.0854
InternVL3.5-8B0.39630.36220.0901
InternVL3.5-38B0.39780.33180.1172
Qwen3-VL-8B0.32710.22440.1374
Qwen3-VL-32B0.37480.29550.1250
OWLv2-Large0.29510.18000.1429
Florence-2-Large0.20190.23880.0000
Grounding DINO 1.50.24710.28950.0023

We have added the always-empty baseline and positive-query scores to the main results to clarify the contribution of positive-target localization and empty-query correctness.

Reviewer_toj5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgroVG, a benchmark for evaluating how well vision-language models locate agricultural objects from natural-language instructions. It contains 10,071 image-query pairs from 10 datasets, covering six agricultural target families, and supports both bounding-box localization and instance-mask segmentation.

The main contributions are a unified benchmark, standardized evaluation protocols, and a zero-shot evaluation of 26 model configurations. Expert reviews and experiments validate the benchmark's quality and reveal that existing models still struggle with identifying all relevant objects, avoiding false predictions when targets are absent, and accurately segmenting objects. The results highlight significant challenges in developing reliable vision-language systems for agriculture.

Critical strengths
  1. AgroVG combines 10 datasets and six agricultural target categories, covering both object localization and segmentation. It also tests whether models can find multiple objects or recognize when a target is absent. This is important because it evaluates real challenges that simpler benchmarks may overlook.

  2. The authors evaluate 26 model configurations and use expert-reviewed annotations and separate development and test sets. The results clearly show that existing models still struggle to find all relevant objects and avoid false predictions. This makes the benchmark useful for guiding future research in agricultural AI.

Critical weaknesses
  1. Most queries are generated from predefined templates, so they may not fully reflect how people naturally describe objects in real agricultural scenarios. This limits how well the benchmark can measure models' ability to understand diverse, real-world instructions.

  2. The instance-mask task covers only three of the six agricultural target categories because reliable pixel-level annotations are not available for all categories. This limits the benchmark's ability to evaluate segmentation performance across the full range of agricultural applications.

Questions for authors
  1. Since most queries are generated from templates, how well do the authors expect AgroVG to reflect real-world instructions? The human-paraphrase evaluation covers only a small subset and two models. Could this limitation affect the conclusions?

  2. Could the authors expand the segmentation task to cover the remaining agricultural categories and include more naturally written queries? This would help make the benchmark more representative of real-world agricultural applications.

1 comment
AuthorsOpenReview ↗

Response to Reviewer toj5

We thank the reviewer for the constructive feedback.


W1 & Q1: Natural-language evaluation beyond templates.

We agree that template-based queries do not capture the full diversity of real-world agricultural instructions. We expanded the evaluation to 500 natural-language queries per task (1,000 pairs in total), rewritten and reviewed by domain experts and paired with their original queries. The evaluation covers five T1 and three T2 models. Images, target identities, ground truth, and inference/scoring settings remain fixed; only the wording changes, without additional tuning.

Scores follow the original protocol: T1 positive-query [email protected] and T2 query-level mIoU. Differences are natural − original; 95% CIs use 5,000 source-stratified paired acquisition-group bootstrap replicates.

Task / ModelOriginalNaturalDifference [95% CI]
T1 / InternVL3.5-8B0.36870.3563−0.0123 [−0.0358, +0.0119]
T1 / Qwen3-VL-8B0.23770.2306−0.0071 [−0.0305, +0.0167]
T1 / GPT-5.40.14550.1306−0.0149 [−0.0389, +0.0085]
T1 / OWLv2-Large0.13610.1033−0.0328 [−0.0501, −0.0156]
T1 / GPT-4o0.01110.0120+0.0009 [−0.0099, +0.0113]
T2 / LISA-7B0.31330.3289+0.0156 [−0.0032, +0.0346]
T2 / SAM30.08560.0788−0.0069 [−0.0316, +0.0166]
T2 / Grounded-SAM0.27810.2224−0.0557 [−0.0821, −0.0292]

Point-estimate rankings remain unchanged within both subsets, while the declines for OWLv2 and Grounded-SAM reveal model-dependent sensitivity to natural phrasing.

We have added these results to the Appendix and updated the dataset limitations.


W2 & Q2: Expanding T2 coverage and natural-language queries.

We constructed a supplementary T2 set of 750 positive image–query pairs, with 250 each for pests, wheat heads, and tree canopies. The set contains human-reviewed natural-language queries with verified query–mask alignment.

Together with the original T2 data, this supplementary set extends mask-grounding evaluation to all six agricultural target families.

We evaluated three models using the original zero-shot configurations and query-level T2 metrics. [email protected] and [email protected] denote the proportions of queries whose predicted masks reach the respective IoU thresholds.

FamilyModelmIoU[email protected][email protected]
PestLISA-7B0.48300.48000.2600
PestPixelLM-7B0.36180.35600.1320
PestGrounded-SAM0.16190.14800.0760
Wheat headLISA-7B0.19930.13200.0280
Wheat headPixelLM-7B0.08060.00400.0000
Wheat headGrounded-SAM0.04280.04800.0160
Tree canopyLISA-7B0.31040.31200.1440
Tree canopyPixelLM-7B0.24630.21200.0720
Tree canopyGrounded-SAM0.12700.13600.1040

LISA-7B achieves the highest mIoU in each added family, while all three models obtain their lowest mIoU on the wheat subset. This extension broadens the benchmark’s target coverage, and the results highlight remaining challenges in precise mask grounding under natural-language queries.

We will further release the expanded data through the existing AgroVG dataset repository on Hugging Face.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.