acceptodds
Withdrawn from ICLR 2027

Human-Level on Which Questions? A Ranking Reversal in Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As frontier VLMs increasingly achieve near- or super-human performance on specific benchmarks, there has been increasing speculation that VLMs have superhuman capabilities, based on the assumption that performance should generalize. We think that it is only healthy to question this assumption. We find that 8 frontier VLMs match or exceed human performance on AI-generated visual-reasoning questions (AIGEN) but fall substantially behind humans on human-generated questions (HUMANGEN). This happens because frontier VLMs are sensitive to the difference between AIGEN and HUMANGEN, while humans are robust to it. We carry out an in-depth analysis of both audits to see what causes this trend. We find that surface-level phrasing does not explain it. Instead, the result is due to differences in the reasoning paths required by the two audits. HUMANGEN tends to have longer paths that involve finding visual cues that frontier VLMs are less likely to look for. Providing these cues in oracle form restores performance for frontier VLMs, but to varying degrees, suggesting that some VLMs still lack world knowledge necessary to deduce an answer from these cues.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, audit, evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4diUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper reveals a ranking reversal in visual reasoning: frontier VLMs outperform humans on AI-generated questions but fall behind on human-authored ones. The gap is mainly linked to longer reasoning paths and failures to identify implicit visual cues, rather than wording differences.

Critical strengths
  1. Important and timely question. The paper challenges whether “human-level” VLM performance generalizes across different benchmark construction procedures.
  2. Strong diagnostic analysis. The reasoning-path, step-attempt, and component-oracle analyses go beyond simply reporting the ranking reversal.
  3. Clear empirical finding. The AIGEN–HUMANGEN ranking reversal is consistent across multiple frontier VLM families.
Critical weaknesses
  1. Small HUMANGEN scale. The human-authored audit contains only 61 questions over 40 images, which limits the strength of broad conclusions.
  2. Potential generator confound. AIGEN is generated by a single model, Gemini-2.5-Pro, so the observed gap may partly reflect properties specific to this generator rather than AI-generated questions in general.
  3. Reasoning-path annotations are subjective. The decomposition into reasoning steps and “attempted” visual cues relies on manual annotations and an LLM judge, making the proposed explanation less definitive.
Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your helpful feedback. We would like to address some points:

Small HUMANGEN scale. The human-authored audit contains only 61 questions over 40 images, which limits the strength of broad conclusions.

We understand the concern about dataset size. As a mitigation, we have added Appendix L to our submission, where Tabs. 10 and 11 shows our main results with 95% confidence intervals. We observe the following:

  • All VLMs except Gemini 3.1 Pro, Gemini 3.7 Flash, and GPT 6 Astra are statistically significantly less accurate than human subjects on HumanGen.
  • All VLMs except Qwen3-VL 235B, Claude Sonnet 4.5, and Claude Opus 4.8 are statistically significantly more accurate than human subjects on AIGen.
  • HumanGen is statistically significantly harder than AIGen for all VLMs.
  • All VLMs except GPT-6 Astra have a statistically significantly larger dropoff between AIGen and HumanGen than the human subjects. GPT-6 Astra also has a larger dropoff than human subjects (by 11.5%), but its dropoff difference has a p-value of roughly 0.1.

Thus, our main result holds up despite limited dataset size.

We have also added confidence intervals to the correlation results in Tabs. 7-9 in Appendix K. In addition, see Tab. 4 in Appendix D, which does leave-one-subject-out analysis on the human performance, in which we report the minimum and maximum of the averages obtained by leaving any one participant out.

Potential generator confound. AIGEN is generated by a single model, Gemini-2.5-Pro, so the observed gap may partly reflect properties specific to this generator rather than AI-generated questions in general.

Please see our Official Comment on "Additional AI-generators and prompting strategies", in which we experiment with more generators and prompting strategies to show that our observations generalize beyond Gemini 2.5 Pro.

AuthorsOpenReview ↗

Requesting feedback

Dear Reviewer 4diU,

Thank you for your insightful review. We have provided the rebuttal and revised manuscript to address your comments, including a general response titled Additional AI-generators and prompting strategies. We would greatly appreciate your feedback before the discussion period ends and would be happy to address any further concerns.

Best, Authors

Reviewer_1kGjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines whether frontier VLMs achieve human-level performance by comparing their performance gap on two self-constructed benchmarks, one AI-generated named AIGEN and another human-written named HUMANGEN, for the task of visual question answering. The authors find that VLMs perform substantially worse on HUMANGEN compared to AIGEN, while human performance only drops slightly, and attribute this difference to longer reasoning paths needed for HUMANGEN compared to AIGEN.

Critical strengths
  • The topic is interesting and important, and the constructed benchmarks are well-documented and potentially useful for future work in this field, though I have some concerns about them (see Critical Weakness 1).
  • The experiments are extensive, though there are issues with the claims (see Critical Weakness 2).
Critical weaknesses
  • My first major concern is the validity of the comparison between the two benchmarks. AIGEN is generated by a single model only using cues mentioned in the image captions, and HUMANGEN is written by a single researcher who typically uses their own judgment rather than the captions. Therefore, the performance gap may come from the different usage of captions or from the particular style of one model/one researcher, rather than from whether AI or humans write the questions.
  • My second concern is the evidence for the claim that VLMs perform worse on HUMANGEN because it requires longer reasoning paths. The relationship between path length and accuracy (Table 1 and Figure 3) is computed on AIGEN and HUMANGEN pooled together, not within each dataset. Since HUMANGEN has both longer paths and lower VLM accuracy than AIGEN, pooling the two datasets can produce a negative relationship even if path length has no effect. If the relationship disappears within each dataset, the main conclusion would not hold.
Questions for authors
  1. Have the authors evaluated VLMs on benchmarks constructed under matched conditions (e.g., using captions in the same way for both, and using multiple models and human researchers rather than one), and does the performance gap remain?
  2. Have the authors conducted experiments on paraphrasing HUMANGEN into AI-style wording and examined how the performance of VLMs changes, as well as how wording is correlated with reasoning paths?
  3. Can the authors report the results of Table 1 and Figure 3 separately for AIGEN and HUMANGEN, with p-values? If the negative relationship between path length and accuracy holds within each dataset, the claim would be substantiated."
2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your helpful feedback. We would like to address some points:

AIGEN is generated by a single model only using cues mentioned in the image captions

Please see our Official Comment on "Additional AI-generators and prompting strategies", in which we experiment with more generators and prompting strategies to show that our observations generalize beyond Gemini 2.5 Pro. This includes experiments where the AI-generators are not provided with image captions so they are not restricted to only those cues.

My second concern is the evidence for the claim that VLMs perform worse on HUMANGEN because it requires longer reasoning paths. The relationship between path length and accuracy (Table 1 and Figure 3) is computed on AIGEN and HUMANGEN pooled together, not within each dataset. Since HUMANGEN has both longer paths and lower VLM accuracy than AIGEN, pooling the two datasets can produce a negative relationship even if path length has no effect. If the relationship disappears within each dataset, the main conclusion would not hold.

We understand this concern. If path length correlates with correctness on AIGen U HumanGen, does that mean that path length explains difficulty, or does it just correlate with dataset membership which in turn correlates with some other factor that actually explains difficulty? We actually discussed this question in Section 3.4, where we found that controlling for path length (LwL_w) did not reduce the performance disparity between AIGen and HumanGen, suggesting indirect correlation rather than direct causation. On the other hand we found that controlling for number of unattempted steps (LUL^U) does reduce the disparity, hence our stated conclusion that “VLMs struggle on HumanGen in part because it contains more steps that they are unlikely to attempt”.

Following your suggestion, we have added Appendix K, where we show correlations (with 95% CIs) on AIGen U HumanGen in Tab. 7, HumanGen-only in Tab. 8, and AIGen-only in Tab. 9. Correlations for path-length (LL and LwL_w) reduce on each dataset individually, but correlations for step-attempt-based metrics (LUL^U and LwUL_w^U) remain on HumanGen-only, while reducing on AIGen-only, likely because AIGen elicits zero unattempted steps on 91% of its questions on average, giving it insufficient spread for a strong correlation. This supports our stated conclusion from the previous paragraph.

Have the authors conducted experiments on paraphrasing HUMANGEN into AI-style wording and examined how the performance of VLMs changes, as well as how wording is correlated with reasoning paths?

Our initial submission did the reverse of this. We had a human researcher, independent of the HumanGen question-writing process, rewrite the questions from AIGen while preserving their meaning. As seen in Fig. 2c, this change in wording has only a marginal effect on VLM performance, changing accuracy by less than a percent or two.

Following your suggestion, we have added an experiment where we use Gemini 2.5 Pro to paraphrase HUMANGEN into AI-style wording. We manually reviewed every paraphrasing from this process. We found that none of the VLM accuracies improved by more than 3 points. Details and results of this experiment are in Appendix M (Table 11). These results, along with those of the other rephrasing experiment, indicate that the differences between human-style and AI-style wording are not a significant factor in our observed results.

AuthorsOpenReview ↗

Requesting feedback

Dear Reviewer 1kGj,

Thank you for your insightful review. We have provided the rebuttal and revised manuscript to address your comments, including a general response titled Additional AI-generators and prompting strategies. We would greatly appreciate your feedback before the discussion period ends and would be happy to address any further concerns.

Best, Authors

Reviewer_VFBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether VLMs perform differently on human-written versus AI-generated evaluation questions, and in particular whether models that outperform humans on AI-generated questions also do so on human-written ones. The authors construct two binary-choice VQA sets on the same image source, one generated by VLM and one written by a human author, and evaluate 15 VLMs alongside 12 human participants. Frontier models that match or exceed human accuracy on the AI-generated set fall substantially behind humans on the human-written set, whereas human accuracy remains comparatively stable across the two. Through a series of diagnostics, the authors argue that the gap stems not from a failure to perceive the images, but from an inability to decompose the questions into the relevant visual cues and integrate them with world knowledge.

Critical strengths

The paper addresses a interesting question. As model-generated evaluation data becomes increasingly common, it is important to know whether "human-level" claims established on such data can be transferred to other evaluation settings. Rather than comparing absolute accuracy alone, the paper measures how performance changes between AI-generated and human-written questions relative to a human reference. The finding that models degrade substantially across the two question sets while human subjects do not is a useful observation that can be important for future study.

Critical weaknesses
  1. Lack of controlled experiments. The central claim contrasts human-written with AI-generated questions, but this is never isolated through controls. AIGEN comes from a single generator, HUMANGEN from a single author, and both use one binary-choice format on one image source. The result therefore cannot be attributed to human versus AI authorship rather than to this particular generator, this particular author, or the task format. Natural controls would be including question from a variety of sources e.g. several independent human authors, non-Gemini models under the same guidelines, and external validation on existing evaluation sets, for example applying the AIGEN procedure to images from established human-authored VQA benchmarks, or comparing existing AI-generated and human-written benchmarks. The problem is compounded by the small scale and the absence of confidence intervals or significance tests, so it remains unclear how robust the claim is and how far it generalizes.

  2. The proposed explanation is not convincingly supported, and no remedy is offered. The authors attribute the models' drop on human-written questions to the models' inability to decompose questions into the relevant cues and integrate them with world knowledge. The supporting analyses, however, are relatively coarse and have multiple confounding factors. The reasoning paths that define the key variables are annotated by the authors without reported inter-annotator agreement; the step-attempt judge is itself one of the evaluated models; and the component probes are derived from the reference reasoning, so the oracle partly supplies the solution path rather than isolating world knowledge. Finally, the paper proposes no remedy, either for the models or for the construction of evaluation sets.

Questions for authors
  1. Could the authors report confidence intervals, standard deviations and significance tests for the main results, in particular for the model–human differences on each question set?
  2. Could the authors provide more details on the human study, such as participant demographics and relevant expertise?
  3. Have both question sets been audited for labeling errors or ambiguity, e.g., by examining items on which the human majority disagrees with the reference answer? How many such items are there, and how do the results change once they are corrected or removed?
  4. Who annotated the reasoning paths? Were the annotators blind to the source of each question, and what is the inter-annotator agreement? How well do the Opus 4.8 judgments agree with human judgments?
2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your helpful feedback. We would like to address some points:

AIGEN comes from a single generator

Please see our Official Comment on AI-generators, in which we experiment with more generators and prompting strategies to show that our observations generalize beyond Gemini 2.5 Pro.

Could the authors report confidence intervals, standard deviations and significance tests for the main results, in particular for the model–human differences on each question set?

We have added Appendix L to our submission, where Tabs. 10 and 11 show our main results with 95% confidence intervals. We observe the following:

  • All VLMs except Gemini 3.1 Pro, Gemini 3.7 Flash, and GPT 6 Astra are statistically significantly less accurate than human subjects on HumanGen.
  • All VLMs except Qwen3-VL 235B, Claude Sonnet 4.5, and Claude Opus 4.8 are statistically significantly more accurate than human subjects on AIGen.
  • HumanGen is statistically significantly harder than AIGen for all VLMs.
  • All VLMs except GPT-6 Astra have a statistically significantly larger dropoff between AIGen and HumanGen than the human subjects. GPT-6 Astra also has a larger dropoff than human subjects (by 11.5%), but its dropoff difference has a p-value of roughly 0.1.

Thus, our main result holds up despite any limitations in dataset size.

We have also added confidence intervals to the correlation results in Tabs. 7-9 in Appendix K. In addition, see Tab. 4 in Appendix D, which does leave-one-subject-out analysis on the human performance, in which we report the minimum and maximum of the averages obtained by leaving any one participant out.

Could the authors provide more details on the human study, such as participant demographics and relevant expertise?

We appreciate your question. We humbly think that our Section 2.2.2 has mostly addressed this, and we will improve the text to make it more clear to the reader. As we mentioned in that section, the human study subjects consist of undergraduate and graduate students as well as faculty members and industry professionals. We will additionally note here that all subjects were recruited from a local university, and all subjects come from a STEM background.

Have both question sets been audited for labeling errors or ambiguity, e.g., by examining items on which the human majority disagrees with the reference answer? How many such items are there, and how do the results change once they are corrected or removed?

We generated the questions in HumanGen entirely by hand without any involvement of AI or automation. Regarding AIGen, we humbly think that our Section 2.2.1 adequately describes our audit process for that dataset, and we will improve the text to make it more clear to the reader. In summary, we manually reviewed every AI-generated question and discarded any that were erroneous or ambiguous or did not require multiple visual cues to answer. This manual review step had a pass rate of roughly 10%, as the underlying PixMo dataset has many images with just one subject or mostly text.

With that being said, we appreciate your suggestion of examining items where the human majority disagrees with the reference. We have manually gone through each question that was answered by multiple humans and manually re-examined any where more than half of the humans disagreed with the reference answer. There were 11 such questions in AIGen and 9 in HumanGen. Our manual re-examination found that all reference answers on these questions were indeed correct, and the human majorities were genuinely incorrect.

Who annotated the reasoning paths? Were the annotators blind to the source of each question

We humbly think that our Appendix E addresses this, and we will make it clearer to the reader. In summary, a head annotator and 3 assistant annotators each made the reasoning path annotations, with the head annotator reviewing all assistant annotations and revising them for consistency as necessary. Fig. 9 in Appendix E shows the WebUI used for annotation, which does not display the source of the question (the “Show L3 ID” button was for debugging purposes only).

AuthorsOpenReview ↗

Requesting feedback

Dear Reviewer VFBJ,

Thank you for your insightful review. We have provided the rebuttal and revised manuscript to address your comments, including a general response titled Additional AI-generators and prompting strategies. We would greatly appreciate your feedback before the discussion period ends and would be happy to address any further concerns.

Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional AI-generators and prompting strategies

All three reviewers raised concerns about using a single AI model to generate the questions for AIGen, and asked whether the ranking reversal between VLMs and human subjects would generalize to other AI generators. We would like to note that our paper aims to highlight the importance of specifying “who wrote the questions” when claiming that VLMs achieve human-level performance. It serves as a cautionary tale about relying solely on AI-generated questions: just as a single counterexample can disprove a universal mathematical statement, a ranking reversal between HumanGen and AIGen, even with a single AI generator, can expose the limitations of generalizing performance on AI-generated questions to broader claims of human-level capability. However, it is still worth considering whether the patterns we observed with Gemini 2.5 Pro as a generator hold across other AI generators and prompting strategies.

To that end, we have taken a set of 40 images and generated questions for them using all 8 models from the main paper. We have also tried 3 generator prompting strategies: (1) the current prompt, which was used to generate AIGen; (2) the current prompt without the provided caption; (3) an “enhanced” prompt that is more aggressive in instructing the generator to make sure its questions require multiple cues to answer. We have manually reviewed every one of these generated questions and removed any unanswerable ones, leaving on average 25 questions per (generator model, prompt version).

We then ran VLM inference on these newly generated datasets using each of the 8 models, resulting in around 200 accuracies, one for each “VLM vs. AI-generator” pair. We included these results in our updated manuscript as detailed below. Overall, by comparing these results with those in the original manuscript, we still see a bigger drop in accuracy for VLMs than for human subjects when going from AI-generated questions to human-generated ones. The pattern is not just a quirk of Gemini 2.5 Pro or our current prompt or our use of captions.

Details about the newly added results.

We’ve updated our submission with Appendix J, where Tabs. 5 and 6 show the results of this inference. Each row in Tab. 5 is an inference VLM. The first two columns show accuracy on HumanGen and AIGen, from the paper. Subsequent columns show the mean and SD of accuracy on the newly generated datasets, aggregated over generating models. The last column shows accuracy specifically over the new dataset generated with Gemini 2.5 Pro with the current prompt, which matches the generation setup for AIGen. By comparing it with the AIGen column, we can isolate any shifts in accuracy due to different image sets or manual filtering protocols.

Tab. 6 shows the accuracy elicited by each generating model and prompt, averaged over inference models. We see that some generator setups elicit a lower average VLM accuracy than the (Gemini 2.5 Pro, current prompt) setup that produced AIGen, but there are others, including all the GPT-6 Astra and Claude Opus 4.8 setups, that elicit higher accuracies, suggesting that the newest models don’t always produce the most challenging questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.