- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentVQA, a benchmark comprising 1,000 multiple-choice questions curated from 23 existing datasets and spanning five domains: web interaction, robotics, egocentric video, gaming, and spatial planning. Through a process involving the generation of challenging negative examples alongside manual filtering and revision, it provides a unified framework for the offline evaluation of visual agents. The authors evaluated 22 vision-language models—achieving a peak accuracy of 58.5%—and, by combining ranking correlations from online evaluations with error analysis, discussed model deficiencies in visual grounding, temporal understanding, and spatial state representation. The paper's core contributions lie in the construction of this cross-domain benchmark and the associated diagnostic analysis.
Critical strengths
- It offers clear practical value. By consolidating disparate tasks into an automatically scored evaluation—covering diverse visual inputs while preserving essential action histories—it facilitates the rapid comparison of models' cross-domain capabilities.
- The data curation is meticulous. Beyond generating challenging distractors, the process involved manual revisions and item-by-item reviews to resolve ambiguities regarding click regions and multiple valid next steps, demonstrating careful attention to the practical challenges of converting interactive tasks into multiple-choice questions.
- Preliminary evidence of validity is provided. Experiments encompassed a range of open-source and closed-source models; the rank correlation coefficients between performance in the spatial and gaming domains and the corresponding tasks in SpatialWorld were approximately 0.90 and 0.85, respectively, supporting its utility as a tool for rapid offline diagnosis.
Critical weaknesses
Direct validation of the incremental contribution is lacking. The benchmarks are primarily derived from filtering and rewriting existing datasets, yet they lack comparisons with methods such as direct aggregation or random sampling of the original data. Furthermore, the respective gains from manual filtering and hard negative examples remain unquantified; thus, the extent to which the new construction process enhances evaluation value is unclear. 2. Validation of online validity is limited in scope. Correlation analysis covers only one online benchmark (SpatialWorld) and 12 models, with strong correlations observed primarily in the spatial and gaming domains. Domains such as web interaction and robotics require more tailored online validation to substantiate the broad applicability of the five-domain benchmark suite. 3. Reporting on the reliability of scoring and rankings is insufficient. Appendix E classifies questions with unparseable answers as "skipped/unevaluated" but fails to specify the denominator for accuracy calculations or the skip rates for individual models, complicating the assessment of result comparability. While Appendix F includes experiments on option shuffling, it reports only aggregated average scores, which is inadequate to demonstrate the stability of model rankings. 4. Conclusions regarding error attribution exceed the available evidence. The analysis focuses solely on Gemini 3.1 Pro, utilizing the model itself to assist in classifying its own errors; additionally, the rate of manual spot-checks and annotation consistency are not reported. Consequently, the conclusion that "perception and state construction constitute the primary bottleneck" should be regarded as an exploratory observation requiring cross-model validation.
Questions for authors
See Weaknesses
4 comments
Response to Reviewer iGD4 (2 of 2)
Appendix E classifies questions with unparseable answers as "skipped/unevaluated" but fails to specify the denominator for accuracy calculations or the skip rates for individual models.
Thank you for pointing this out. All models have 100% evaluation coverage, with a 0% final skip rate across all domains. Approximately 5% of responses initially had parsing failures, but these were resolved through retries. Our evaluation code handles different output formats across models and retries unparseable responses up to three times before marking them as "unevaluated." As a result, no samples were skipped in Table 1. Domain accuracy is calculated over all 200 questions, and overall accuracy over all 1,000 questions. We have clarified this in Appendix E of the revised manuscript.
We have revised Section 4.1 and Appendix E to clarify the retry procedure, zero final skip rate, and accuracy denominators.
While Appendix F includes experiments on option shuffling, it reports only aggregated average scores, which is inadequate to demonstrate the stability of model rankings.
Model rankings remain largely unchanged across different option orders, with a mean Spearman rank correlation of 0.977. We conducted a new option-shuffling evaluation with six models on a subset of 200 questions, using five option orders: the original (P0) and four shuffled versions (P1–P4). The table reports accuracy (%) and standard deviation (SD, in percentage points).
| Model | P0 | P1 | P2 | P3 | P4 | SD |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 62.5 | 62.0 | 63.5 | 63.0 | 65.5 | 1.21 |
| GPT-5.6-sol high | 58.5 | 59.0 | 56.5 | 60.0 | 57.0 | 1.29 |
| Qwen3.8-27B | 53.5 | 55.0 | 54.5 | 51.0 | 52.5 | 1.44 |
| Qwen3.7-Flash | 52.5 | 49.0 | 53.5 | 54.0 | 52.5 | 1.75 |
| Claude Sonnet 5 | 45.5 | 43.5 | 44.0 | 42.0 | 43.0 | 1.16 |
| Llama-4-Maverick | 27.5 | 27.0 | 29.5 | 32.0 | 27.5 | 1.86 |
Across the ten pairs of option orders, Spearman rank correlation ranges from 0.943 to 1.000, with an average of 0.977. Only the two Qwen models exchange positions in P3 and tie in P4, while the other four models maintain their rankings across all five orders. Accuracy also varies relatively little, with SD ranging from 1.16 to 1.86 percentage points. These results show that model rankings are stable under option shuffling on this subset.
We have updated the option-shuffling analysis in Appendix F with the six-model results, including model-wise accuracy variations and pairwise Spearman rank correlations.
The error analysis focuses solely on Gemini 3.1 Pro, utilizing the model itself to assist in classifying its own errors; additionally, the rate of manual spot-checks and annotation consistency are not reported.
Thank you for sharing this concern. The main error patterns remain consistent across all three models, and human verification agrees with 95% of Gemini's error classifications. We extended the original analysis to Qwen3.8-27B and GLM-4.1V-9B-Thinking, using the same error taxonomy as the paper and requiring quoted evidence to match the saved reasoning exactly.
The same error categories remain dominant across all three models. Grounding errors account for 75–87% of Web Agents failures, visual miscomprehension for 57–89% in Games, and spatial confusion for 45–65% in Robotics. Motion misperception and perception gaps remain the two largest categories in Egocentric Videos, while viewpoint gaps account for 50–74% of errors in Spatial Planning. These results show that the main findings of our original error analysis are not limited to Gemini 3.1 Pro.
We also manually checked 100 randomly selected samples from the original Gemini 3.1 Pro error analysis. Human annotations agreed with Gemini's assigned error categories on 95% of these samples, supporting the reliability of the original classifications.
We have incorporated the three-model error analysis in Section 4.5 (Figure 8) and the methodology and 95% manual verification result in Appendix G of the revised manuscript.
The author’s response has largely addressed my concerns. However, taking the overall quality of the paper into account, I think a score in the 2–3 range (i.e., 2.5) is appropriate; I offer this for the AC's consideration.
Dear Reviewer iGD4,
We are glad that our rebuttal addresses your concerns. We really appreciate your constructive feedback, and this has helped us make our paper stronger.
Best regards, Authors.
Response to Reviewer iGD4 (1 of 2)
We thank the reviewer for their detailed comments and thoughtful feedback. We address the reviewer's concerns and questions below.
VLMs score 12.6–22.5 percentage points lower on AgentVQA compared to randomly sampled examples from the original datasets. To compare AgentVQA with random source sampling, we collected a larger pool of examples from the original datasets, removed ambiguous or invalid samples, and randomly selected 200 valid examples (40 per domain). We kept their original answer formats and options and evaluated six models.
All six models perform considerably worse on AgentVQA, showing that our curated benchmark is more challenging than validity-filtered random samples from the original datasets. Domain-wise results for the source samples are included in Appendix K of the revised manuscript.
Apart from difficulty, our manual curation also corrects ambiguous questions and verifies that each question has one correct answer and clearly incorrect alternatives. The common MCQ format allows us to evaluate models consistently across domains without repeated human grading or VLM judges, which can introduce additional cost and errors. This makes evaluation simpler, reproducible, and less expensive.
Thus, the added value includes clearer annotations, more challenging examples, and standardized, reproducible, lower-cost evaluation.
We have included the random source-sampling comparison in Section 4.6 (Table 3), with domain-wise results and sampling details in Appendix I.1 of the revised manuscript.
Hard negatives reduce model accuracy by 9.5–24.5 percentage points. To measure their effect, we evaluated seven models on the same 200 AgentVQA questions, replacing the hard distractors with ordinary, plausible wrong answers while keeping the questions and correct answers unchanged.
All seven models perform better with ordinary distractors, confirming that our hard negatives make the questions substantially more challenging.
Manual filtering and curation. The random-sampling comparison in our preceding response measures the combined effect of our benchmark construction process. While imrpoving difficulty, manual filtering also helps substantially in removing ambiguous questions. As described in Sec. 3.1, VLM screening removes approximately 45% of candidates for being ambiguous, and human review rejects another 60% of the remaining samples for ambiguity and difficulty, retaining about 22% overall. Among the final 1,000 questions, 70.6% of question-answer pairs are modified from their original versions. These changes exclude minor wording and formatting edits. All retained questions undergo final manual verification to correct ambiguities and annotation errors.
We have added the hard-negative ablation in Section 4.6 (Table 3) and Appendix I.2, and quantified the impact of manual curation in Section 3.1 of the revised manuscript.
We additionally compared AgentVQA's Web Agents rankings with OSWorld-verified, an online web-agent benchmark, across 11 matched models.
The high correlation shows that models performing better on AgentVQA's Web Agents questions also tend to perform better on online web tasks. This extends the evidence beyond SpatialWorld and supports using AgentVQA for lower-cost offline comparison in web interaction.
We have incorporated this comparison in Section 4.4 of the revised manuscript.