- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper reveals a ranking reversal in visual reasoning: frontier VLMs outperform humans on AI-generated questions but fall behind on human-authored ones. The gap is mainly linked to longer reasoning paths and failures to identify implicit visual cues, rather than wording differences.
Critical strengths
- Important and timely question. The paper challenges whether “human-level” VLM performance generalizes across different benchmark construction procedures.
- Strong diagnostic analysis. The reasoning-path, step-attempt, and component-oracle analyses go beyond simply reporting the ranking reversal.
- Clear empirical finding. The AIGEN–HUMANGEN ranking reversal is consistent across multiple frontier VLM families.
Critical weaknesses
- Small HUMANGEN scale. The human-authored audit contains only 61 questions over 40 images, which limits the strength of broad conclusions.
- Potential generator confound. AIGEN is generated by a single model, Gemini-2.5-Pro, so the observed gap may partly reflect properties specific to this generator rather than AI-generated questions in general.
- Reasoning-path annotations are subjective. The decomposition into reasoning steps and “attempted” visual cues relies on manual annotations and an LLM judge, making the proposed explanation less definitive.
Questions for authors
see above
2 comments
Requesting feedback
Dear Reviewer 4diU,
Thank you for your insightful review. We have provided the rebuttal and revised manuscript to address your comments, including a general response titled Additional AI-generators and prompting strategies. We would greatly appreciate your feedback before the discussion period ends and would be happy to address any further concerns.
Best, Authors
Author Response
Thank you for your helpful feedback. We would like to address some points:
We understand the concern about dataset size. As a mitigation, we have added Appendix L to our submission, where Tabs. 10 and 11 shows our main results with 95% confidence intervals. We observe the following:
Thus, our main result holds up despite limited dataset size.
We have also added confidence intervals to the correlation results in Tabs. 7-9 in Appendix K. In addition, see Tab. 4 in Appendix D, which does leave-one-subject-out analysis on the human performance, in which we report the minimum and maximum of the averages obtained by leaving any one participant out.
Please see our Official Comment on "Additional AI-generators and prompting strategies", in which we experiment with more generators and prompting strategies to show that our observations generalize beyond Gemini 2.5 Pro.