acceptodds
Under review as a conference paper at ICLR 2027

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Abstract

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting challenging, real-world agent examples and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across all models evaluated, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

Primary area:
datasets and benchmarks
Keywords:
agents, vision-language models, offline evaluation, visual question answering (VQA), benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iGD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentVQA, a benchmark comprising 1,000 multiple-choice questions curated from 23 existing datasets and spanning five domains: web interaction, robotics, egocentric video, gaming, and spatial planning. Through a process involving the generation of challenging negative examples alongside manual filtering and revision, it provides a unified framework for the offline evaluation of visual agents. The authors evaluated 22 vision-language models—achieving a peak accuracy of 58.5%—and, by combining ranking correlations from online evaluations with error analysis, discussed model deficiencies in visual grounding, temporal understanding, and spatial state representation. The paper's core contributions lie in the construction of this cross-domain benchmark and the associated diagnostic analysis.

Critical strengths
  1. It offers clear practical value. By consolidating disparate tasks into an automatically scored evaluation—covering diverse visual inputs while preserving essential action histories—it facilitates the rapid comparison of models' cross-domain capabilities.
  2. The data curation is meticulous. Beyond generating challenging distractors, the process involved manual revisions and item-by-item reviews to resolve ambiguities regarding click regions and multiple valid next steps, demonstrating careful attention to the practical challenges of converting interactive tasks into multiple-choice questions.
  3. Preliminary evidence of validity is provided. Experiments encompassed a range of open-source and closed-source models; the rank correlation coefficients between performance in the spatial and gaming domains and the corresponding tasks in SpatialWorld were approximately 0.90 and 0.85, respectively, supporting its utility as a tool for rapid offline diagnosis.
Critical weaknesses

Direct validation of the incremental contribution is lacking. The benchmarks are primarily derived from filtering and rewriting existing datasets, yet they lack comparisons with methods such as direct aggregation or random sampling of the original data. Furthermore, the respective gains from manual filtering and hard negative examples remain unquantified; thus, the extent to which the new construction process enhances evaluation value is unclear. 2. Validation of online validity is limited in scope. Correlation analysis covers only one online benchmark (SpatialWorld) and 12 models, with strong correlations observed primarily in the spatial and gaming domains. Domains such as web interaction and robotics require more tailored online validation to substantiate the broad applicability of the five-domain benchmark suite. 3. Reporting on the reliability of scoring and rankings is insufficient. Appendix E classifies questions with unparseable answers as "skipped/unevaluated" but fails to specify the denominator for accuracy calculations or the skip rates for individual models, complicating the assessment of result comparability. While Appendix F includes experiments on option shuffling, it reports only aggregated average scores, which is inadequate to demonstrate the stability of model rankings. 4. Conclusions regarding error attribution exceed the available evidence. The analysis focuses solely on Gemini 3.1 Pro, utilizing the model itself to assist in classifying its own errors; additionally, the rate of manual spot-checks and annotation consistency are not reported. Consequently, the conclusion that "perception and state construction constitute the primary bottleneck" should be regarded as an exploratory observation requiring cross-model validation.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer iGD4 (1 of 2)

We thank the reviewer for their detailed comments and thoughtful feedback. We address the reviewer's concerns and questions below.

The benchmarks are primarily derived from filtering and rewriting existing datasets, yet they lack comparisons with methods such as direct aggregation or random sampling of the original data.

VLMs score 12.6–22.5 percentage points lower on AgentVQA compared to randomly sampled examples from the original datasets. To compare AgentVQA with random source sampling, we collected a larger pool of examples from the original datasets, removed ambiguous or invalid samples, and randomly selected 200 valid examples (40 per domain). We kept their original answer formats and options and evaluated six models.

ModelSource samples (%)AgentVQA (%)Drop (pp)
GPT-5.6-sol high77.555.022.5
Gemini 3.1 Pro78.055.722.3
Claude Sonnet 556.043.412.6
Llama-4-Maverick43.528.015.5
Qwen3.8-27B66.050.915.1
Qwen3.7-Flash69.555.014.5

All six models perform considerably worse on AgentVQA, showing that our curated benchmark is more challenging than validity-filtered random samples from the original datasets. Domain-wise results for the source samples are included in Appendix K of the revised manuscript.

Apart from difficulty, our manual curation also corrects ambiguous questions and verifies that each question has one correct answer and clearly incorrect alternatives. The common MCQ format allows us to evaluate models consistently across domains without repeated human grading or VLM judges, which can introduce additional cost and errors. This makes evaluation simpler, reproducible, and less expensive.

Thus, the added value includes clearer annotations, more challenging examples, and standardized, reproducible, lower-cost evaluation.

We have included the random source-sampling comparison in Section 4.6 (Table 3), with domain-wise results and sampling details in Appendix I.1 of the revised manuscript.

Furthermore, the respective gains from manual filtering and hard negative examples remain unquantified.

Hard negatives reduce model accuracy by 9.5–24.5 percentage points. To measure their effect, we evaluated seven models on the same 200 AgentVQA questions, replacing the hard distractors with ordinary, plausible wrong answers while keeping the questions and correct answers unchanged.

ModelHard distractors (%)Ordinary distractors (%)Increase (pp)
GPT-5.6-sol high56.069.0+13.0
Gemini 3.1 Pro56.581.0+24.5
Claude Sonnet 542.557.0+14.5
Llama-4-Maverick27.537.0+9.5
Qwen3.8-27B52.074.5+22.5
Qwen3.7-Flash56.571.0+14.5
Gemini 3.8 Flash65.080.0+15.0

All seven models perform better with ordinary distractors, confirming that our hard negatives make the questions substantially more challenging.

Manual filtering and curation. The random-sampling comparison in our preceding response measures the combined effect of our benchmark construction process. While imrpoving difficulty, manual filtering also helps substantially in removing ambiguous questions. As described in Sec. 3.1, VLM screening removes approximately 45% of candidates for being ambiguous, and human review rejects another 60% of the remaining samples for ambiguity and difficulty, retaining about 22% overall. Among the final 1,000 questions, 70.6% of question-answer pairs are modified from their original versions. These changes exclude minor wording and formatting edits. All retained questions undergo final manual verification to correct ambiguities and annotation errors.

We have added the hard-negative ablation in Section 4.6 (Table 3) and Appendix I.2, and quantified the impact of manual curation in Section 3.1 of the revised manuscript.

Validation of online validity is limited in scope. Correlation analysis covers only one online benchmark (SpatialWorld).

We additionally compared AgentVQA's Web Agents rankings with OSWorld-verified, an online web-agent benchmark, across 11 matched models.

Online benchmarkAgentVQA domainMatched modelsSpearman's ρp-value
OSWorld-verifiedWeb Agents110.84550.00105

The high correlation shows that models performing better on AgentVQA's Web Agents questions also tend to perform better on online web tasks. This extends the evidence beyond SpatialWorld and supports using AgentVQA for lower-cost offline comparison in web interaction.

We have incorporated this comparison in Section 4.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer iGD4 (2 of 2)

Appendix E classifies questions with unparseable answers as "skipped/unevaluated" but fails to specify the denominator for accuracy calculations or the skip rates for individual models.

Thank you for pointing this out. All models have 100% evaluation coverage, with a 0% final skip rate across all domains. Approximately 5% of responses initially had parsing failures, but these were resolved through retries. Our evaluation code handles different output formats across models and retries unparseable responses up to three times before marking them as "unevaluated." As a result, no samples were skipped in Table 1. Domain accuracy is calculated over all 200 questions, and overall accuracy over all 1,000 questions. We have clarified this in Appendix E of the revised manuscript.

We have revised Section 4.1 and Appendix E to clarify the retry procedure, zero final skip rate, and accuracy denominators.

While Appendix F includes experiments on option shuffling, it reports only aggregated average scores, which is inadequate to demonstrate the stability of model rankings.

Model rankings remain largely unchanged across different option orders, with a mean Spearman rank correlation of 0.977. We conducted a new option-shuffling evaluation with six models on a subset of 200 questions, using five option orders: the original (P0) and four shuffled versions (P1–P4). The table reports accuracy (%) and standard deviation (SD, in percentage points).

ModelP0P1P2P3P4SD
Gemini 3.8 Flash62.562.063.563.065.51.21
GPT-5.6-sol high58.559.056.560.057.01.29
Qwen3.8-27B53.555.054.551.052.51.44
Qwen3.7-Flash52.549.053.554.052.51.75
Claude Sonnet 545.543.544.042.043.01.16
Llama-4-Maverick27.527.029.532.027.51.86

Across the ten pairs of option orders, Spearman rank correlation ranges from 0.943 to 1.000, with an average of 0.977. Only the two Qwen models exchange positions in P3 and tie in P4, while the other four models maintain their rankings across all five orders. Accuracy also varies relatively little, with SD ranging from 1.16 to 1.86 percentage points. These results show that model rankings are stable under option shuffling on this subset.

We have updated the option-shuffling analysis in Appendix F with the six-model results, including model-wise accuracy variations and pairwise Spearman rank correlations.

The error analysis focuses solely on Gemini 3.1 Pro, utilizing the model itself to assist in classifying its own errors; additionally, the rate of manual spot-checks and annotation consistency are not reported.

Thank you for sharing this concern. The main error patterns remain consistent across all three models, and human verification agrees with 95% of Gemini's error classifications. We extended the original analysis to Qwen3.8-27B and GLM-4.1V-9B-Thinking, using the same error taxonomy as the paper and requiring quoted evidence to match the saved reasoning exactly.

The same error categories remain dominant across all three models. Grounding errors account for 75–87% of Web Agents failures, visual miscomprehension for 57–89% in Games, and spatial confusion for 45–65% in Robotics. Motion misperception and perception gaps remain the two largest categories in Egocentric Videos, while viewpoint gaps account for 50–74% of errors in Spatial Planning. These results show that the main findings of our original error analysis are not limited to Gemini 3.1 Pro.

We also manually checked 100 randomly selected samples from the original Gemini 3.1 Pro error analysis. Human annotations agreed with Gemini's assigned error categories on 95% of these samples, supporting the reliability of the original classifications.

We have incorporated the three-model error analysis in Section 4.5 (Figure 8) and the methodology and 95% manual verification result in Appendix G of the revised manuscript.

The author’s response has largely addressed my concerns. However, taking the overall quality of the paper into account, I think a score in the 2–3 range (i.e., 2.5) is appropriate; I offer this for the AC's consideration.

Dear Reviewer iGD4,

We are glad that our rebuttal addresses your concerns. We really appreciate your constructive feedback, and this has helped us make our paper stronger.

Best regards, Authors.

Reviewer_TYYNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AgentVQA curates 1k multiple-choice questions from 23 existing datasets across five agentic domains to evaluate VLMs' visual understanding and local decision-making capabilities offline.

Critical strengths
  1. Broad source and coverage: 5 domains, 23 datasets.
  2. Extensive experiments on 22 models.
Critical weaknesses
  1. Limited novelty. "Agent" primarily characterizes the source domains, while the evaluation remains offline VQA. The contribution centers on curating and converting existing tasks into MCQs, with insufficient evidence of added evaluation value over the original benchmarks.
  2. Insufficient scoring transparency. Parsing failures are marked unevaluated without clear reporting of per-model evaluation coverage or scoring denominators.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TYYN (1 of 1)

We thank the reviewer for their detailed comments and thoughtful feedback. We address the reviewer's concerns and questions below.

Limited novelty. "Agent" primarily characterizes the source domains, while the evaluation remains offline VQA. The contribution centers on curating and converting existing tasks into MCQs, with insufficient evidence of added evaluation value over the original benchmarks.

The main contribution of AgentVQA is not just the MCQ conversion. It provides a single evaluation across five agentic domains, with manually checked questions, harder answer choices, and consistent scoring. While the examples are drawn from 23 existing datasets, 70.6% of question-answer pairs are modified from their original datasets.

1. Comparison with original source datasets. To measure the added difficulty of AgentVQA, we randomly sampled 200 valid examples from the original datasets after removing ambiguous or invalid samples, keeping their original answer formats and options. We evaluated six models and compared their results with AgentVQA (Table below). Domain-wise results for the source samples are provided in Appendix K of the revised manuscript.

ModelSource samples (%)AgentVQA (%)Drop (pp)
GPT-5.6-sol high77.555.022.5
Gemini 3.1 Pro78.055.722.3
Claude Sonnet 556.043.412.6
Llama-4-Maverick43.528.015.5
Qwen3.8-27B66.050.915.1
Qwen3.7-Flash69.555.014.5

All six models score 12.6–22.5 percentage points lower on AgentVQA than on the validity-filtered source samples. While this comparison does not isolate the effect of MCQ conversion alone, it shows that our curation process produces a substantially more challenging evaluation set than random sampling from the original benchmarks.

2. Manual curation and question modifications. As described in Sec. 3.1, VLM screening removes approximately 45% of candidates, and human review rejects another 60% of the remaining samples due to ambiguity, annotation errors, or insufficient difficulty. Only about 22% of initial candidates are retained. Among the final 1,000 questions, 70.6% of question-answer pairs are modified from their original versions. All retained questions undergo a final manual verification again.

3. Standardized and reproducible evaluation. The MCQ format also makes it possible to evaluate different tasks using the same scoring method. Open-ended click predictions can have multiple valid answers, making automatic grading difficult (Fig. 4). By manually checking the correct answers and hard negatives, we can score subsequent model predictions automatically without requiring additional human or VLM judges. This allows us to compare models across domains and analyze common failure modes under a consistent evaluation protocol.

We have included the curation statistics in Section 3.1, the source-dataset comparison in Section 4.6 (Table 3) and Appendix I.1, and the MCQ evaluation rationale in Appendix H of the updated manuscript.

Insufficient scoring transparency. Parsing failures are marked unevaluated without clear reporting of per-model evaluation coverage or scoring denominators.

Approximately 5% of responses initially had parsing failures, but all were resolved through retries, leaving 0% unevaluated samples in the final results. We use a robust answer extraction system that handles different output formats across models. If a response still cannot be parsed, our evaluation code retries up to three times before marking the question as "unevaluated." This procedure resulted in 100% evaluation coverage for every model and domain in Table 1, with no samples excluded from scoring. Domain accuracy is calculated over all 200 questions, and overall accuracy over all 1,000 questions.

We have revised the wording in Section 4.1 to explicitly clarify that all parsing failures were resolved, with zero unevaluated samples. The retry procedure and scoring denominators are detailed in Appendix E.

Dear Reviewer,

Thank you for your feedback. We hope our responses have addressed your concerns and welcome any further questions.

Sincerely, The Authors

Reviewer_6FM6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AgentVQA, a benchmark of 1,000 multiple-choice questions for agentic visual understanding. The questions come from 23 existing datasets in five domains. The best of 22 models reaches 58.5%. An error analysis points to perception failures.

Critical strengths

(1) The curation is careful. Only about 22% of candidates survive, and humans verify every kept item.

(2) The study is broad. It covers 22 models across five domains and three input types. Spatial Planning rankings match an online benchmark with a Spearman correlation of 0.895.

Critical weaknesses

(1) The benchmark does not measure the agentic process it claims. All 1,000 items are single-turn, four-option questions with no tool calls, zooming, frame selection, or process metric. Even click grounding is reduced to picking one of four coordinates.

(2) The validity claims lack basic controls. There is no image-free or human baseline, and 7 of 22 models score below the 25% chance level on Web Agents. The online check uses only SpatialWorld with 12 models, and Web Agents and Robotics show no significant correlation.

Questions for authors

(1) What do the top models score with the image removed, and what do humans score on a sample from each domain? This would show whether the below-chance Web Agents results come from item errors.

(2) The error modes are labeled by Gemini 3.1 Pro on its own failures, with only a random subset checked by hand. What is the human agreement rate, and do perception errors still dominate when models can zoom or select frames?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6FM6 (1 of 2)

We thank the reviewer for their detailed comments and thoughtful feedback. We address the reviewer's concerns and questions below.

The benchmark does not measure the agentic process it claims. All 1,000 items are single-turn, four-option questions with no tool calls, zooming, frame selection, or process metric.

AgentVQA is designed as a fast, reproducible offline evaluation of visual capabilities needed by agents, not a replacement for online evaluation. It evaluates visual understanding and local decisions without requiring expensive environment setup and repeated rollouts.

We also tested whether allowing models to zoom into images improves their performance. We evaluated 200 Web Agents questions with up to three image crops per question, keeping the questions and answer options unchanged.

ModelOriginal (%)With Zoom (%)Δ
GPT-5.6-sol high47.044.5-2.5
Gemini 3.1 Pro34.534.50.0
Claude Sonnet 538.031.0-7.0
Llama-4-Maverick18.519.0+0.5
Qwen3.8-27B30.031.0+1.0
Qwen3.7-Flash43.039.0-4.0

Allowing zoom did not consistently improve accuracy. Two models improved slightly, one remained unchanged, and three performed worse. These results suggest that adding zoom alone does not substantially improve performance on these questions.

We have clarified the scope of AgentVQA in Section 3 of the revised manuscript and incorporated the zoom experiment in Section 4.6 (Table 4), with additional details and post-zoom error analysis in Appendix I.4 (Table 12).

Even click grounding is reduced to picking one of four coordinates.

We also evaluated six models on 40 Web Agents questions using open-ended answers, where models directly predicted a click coordinate or action instead of selecting from four options. All responses were manually evaluated.

ModelOpen-ended accuracy (%)
GPT-5.6-sol high92.5
Gemini 3.1 Pro87.5
Claude Sonnet 590.0
Llama-4-Maverick50.0
Qwen3.8-27B70.0
Gemini 3.8 Flash90.0

Models generally performed better on open-ended questions than on their MCQ counterparts with hard negatives, while their rankings remained largely unchanged compared to AgentVQA Web Agents. This suggests that the MCQ format with hard negatives makes the questions more challenging without substantially changing the relative performance of models.

We use MCQs primarily for consistent and automatic evaluation. Open-ended click predictions require additional grading, as multiple click locations may be valid. By manually verifying the correct answer and distractors, our MCQ format avoids the need for human or VLM judges during evaluation and provides deterministic scoring.

We have also incorporated the justification for MCQ-based evaluation and the six-model open-ended comparison in Appendix H (Table 9) of the revised manuscript.

What do the top models score with the image removed, and what do humans score on a sample from each domain? This would show whether the below-chance Web Agents results come from item errors.

Human evaluation. We evaluated humans on 100 questions across all five domains, using the same visual inputs, questions, options, and relevant goals/action histories. Humans achieved 90% overall accuracy, including 85% on Web Agents, where models struggle the most.

DomainHuman accuracy (%)
Web Agents85
Games90
Egocentric Videos95
Robotics85
Spatial Planning95
Overall90

Text-only evaluation. We also tested five models on 200 questions (40 per domain), removing the visual inputs while keeping the questions, options, goals, and action histories unchanged. The table reports accuracy with visual inputs / without visual inputs (%).

ModelWebGamesEgo.RoboticsSpatialOverallΔ (pp)
Gemini 3.1 Pro55/2572.5/1575/32.562.5/37.572.5/3567.5/29-38.5
Claude Sonnet 537.5/22.552.5/3565/42.535/3057.5/4049.5/34-15.5
Qwen3.8-27B50/3565/27.587.5/32.555/22.562.5/2564/28.5-35.5
Qwen3.7 Flash52.5/2557.5/27.587.5/42.547.5/2567.5/37.562.5/31.5-31.0
Gemini 3.8 Flash70/3575/22.587.5/4070/22.570/37.574.5/31.5-43.0

All five models performed worse without visual inputs, with overall accuracy dropping by 15.5-43 percentage points. The drop was observed in every domain for every model.

Together with our manual verification of all 1,000 questions, these results suggest that the low Web Agents scores are primarily due to difficulties in visual grounding rather than incorrect or unanswerable questions.

We have included the human evaluation results in Section 4.2 (Table 2) of the revised manuscript, and the full domain-wise text-only evaluation results in Appendix I.3 (Table 11).

AuthorsOpenReview ↗

Response to Reviewer 6FM6 (2 of 2)

The online check uses only SpatialWorld with 12 models, and Web Agents and Robotics show no significant correlation.

We additionally compared model rankings on OSWorld-Verified, an online web-agent benchmark, with rankings on AgentVQA's Web Agents domain across 11 matched models.

Online benchmarkAgentVQA domainMatched modelsSpearman's ρp-value
OSWorld-VerifiedWeb Agents110.84550.00105

The strong positive correlation suggests that models performing better on AgentVQA's Web Agents domain also tend to perform better on online web-agent tasks. This provides additional evidence that AgentVQA can serve as a useful offline complement to interactive evaluations.

We have incorporated this comparison in Section 4.4 of the revised manuscript, alongside the existing SpatialWorld correlation analysis.

The error modes are labeled by Gemini 3.1 Pro on its own failures, with only a random subset checked by hand.

The main error patterns remain consistent across models. To check whether our findings are specific to Gemini 3.1 Pro, we extended the error analysis to Qwen3.8-27B and GLM-4.1V-9B-Thinking using the same error taxonomy. The table compares the main error categories across all three models, including the original Gemini results from Figure 8.

DomainError categoryGemini 3.1 Pro (%)Qwen3.8-27B (%)GLM-4.1V-9B-Thinking (%)
Web AgentsGrounding error788775
GamesVisual miscomprehension718957
Egocentric VideosMotion misperception393529
Egocentric VideosPerception gaps293026
RoboticsSpatial confusion655045
Spatial PlanningViewpoint gap625074

The same error categories remain prominent across all three models. Grounding errors account for 75–87% of Web Agents failures, while visual miscomprehension, motion perception, and spatial understanding remain major sources of errors in the other domains. These results suggest that the failure patterns identified in our original analysis are not specific to Gemini 3.1 Pro.

We have added the updated three-model error analysis in Section 4.5 (Figure 8) of updated manuscript.

What is the human agreement rate, and do perception errors still dominate when models can zoom or select frames?

Human agreement. We manually checked 100 randomly selected samples from the original Gemini 3.1 Pro error analysis. Human annotations agreed with Gemini's error categories on 95% of the samples, supporting the reliability of the error classification.

Grounding error remains the main failure mode when models can zoom into images. Using the zoom experiment described in the first response, we analyzed the incorrect predictions of Gemini 3.1 Pro and Qwen3.8-27B on the 200 Web Agents questions.

Error category after zoomGemini 3.1 Pro (%)Qwen3.8-27B (%)
Grounding error8584
Goal misunderstanding58
Missing inference12
Logical breakdown94
Reasoning–action mismatch01
Visual hallucination01

Grounding errors account for 85% and 84% of incorrect predictions, respectively, showing that they remain the dominant failure mode even when models can inspect image crops.

We have included the 95% human agreement in Section 4.5 and Appendix G, and the post-zoom error analysis in Appendix I.4 (Table 12) of the updated manuscript.

Dear Reviewer,

Thank you for your feedback. We hope our responses have addressed your concerns and welcome any further questions.

Sincerely, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Clarifications and Revisions

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of AgentVQA's broad coverage across 23 datasets and five domains, careful manual curation, extensive evaluation of 22 models, and practical value for rapid offline diagnosis of agentic visual capabilities.

To address the shared concerns regarding benchmark novelty, validity, online relevance, and evaluation reliability, we conducted several additional experiments and provided detailed clarifications in our individual responses. All new results and clarifications have been incorporated into the revised manuscript, with changes highlighted in blue\textcolor{blue}{blue}.

Summary of Main Reviewer Concerns and Our Responses

  • Benchmark novelty and added value (TYYN, iGD4): We quantified the impact of benchmark construction, finding that 70.6% of samples were substantively modified. Six models score 12.6-22.5 percentage points lower on AgentVQA than on randomly sampled valid source examples. A separate ablation shows that hard negatives reduce accuracy by 9.5-24.5 points. These results demonstrate the additional difficulty and better quality introduced by our curation and distractor construction (Sections 3.1 and 4.6).

  • Agreement with online evaluations (6FM6, iGD4): We extended validation beyond SpatialWorld to OSWorld-Verified, obtaining strong rank correlations in two corresponding domains: ρ = 0.895 for Spatial Planning and ρ = 0.846 for Web Agents. This supports AgentVQA's usefulness as a lower-cost complement to interactive evaluations (Section 4.4, Figure 7).

  • Scoring reliability and ranking stability (TYYN, iGD4): We clarified that all 22 models have 100% evaluation coverage, with no samples excluded due to parsing failures. Our new option-shuffling experiment also shows stable model rankings, with a mean Spearman correlation of 0.977 across five option orders (Appendices E-F).

  • Error-analysis generalizability (6FM6, iGD4): We extended the error analysis from one to three models, finding consistent dominant failure patterns across domains. We also report 95% human agreement on 100 manually checked error classifications (Section 4.5, Figure 8, and Appendix G).

  • Scope of agentic evaluation (6FM6): We clarified that AgentVQA evaluates visual understanding and decision-making capabilities required by agents, rather than complete interactive execution. It is designed to complement, not replace, online evaluations, enabling faster and more reproducible assessment without expensive environment setup and repeated rollouts (Section 3).

  • Benchmark validity and additional ablations (6FM6): Humans achieve 90% accuracy, including 85% on Web Agents, suggesting that the questions are answerable and that low model scores primarily reflect model limitations rather than incorrect or unanswerable samples. Removing visual inputs reduces model accuracy by 15.5-43 percentage points. Additional zooming and open-ended evaluation experiments further examine the benchmark's difficulty, evaluation format, and consistent error modes (Sections 4.2, 4.6, and Appendices H-I).

Taken together, these results provide stronger evidence that AgentVQA offers more than a standardized collection of existing tasks: its manual curation and hard negatives produce a challenging, reliable evaluation, its rankings show strong agreement with corresponding online benchmarks, and its error analysis identifies consistent weaknesses across models. We believe these additions directly address the reviewers' central concerns and strengthen the case for AgentVQA as a scalable complement to interactive agent evaluation.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.