acceptodds
Under review as a conference paper at ICLR 2027

Do you see what I see? Probing visual perception in artificial intelligence with a Gestalt-theory driven Optical Illusion Dataset

Abstract

Machine learning algorithms are increasingly deployed in safety-critical domains such as autonomous driving and medical diagnostics, where ambiguous data poses a fundamental challenge. Optical illusions offer a compelling window into human perceptual processing, yet remain largely absent from machine learning benchmarks. We introduce Ambivision, a dataset of optical illusions featuring intermingled animal pairs grounded in Gestalt psychological principles such as proximity, similarity, closure, etc., designed to evoke genuine perceptual ambiguity rather than superficial visual difficulty. Using Ambivision as a diagnostic probe, we show that widely used attribution methods (Grad-CAM, Integrated Gradients, PipNet, ACE) cannot produce discriminative, human interpretable explanations when both readings rest on the same pixels. We further find, across seven architectures, that supplying an explicit cue to the discriminative region (gaze direction or the eye) improves classification on ambiguous data over both no-cue and random-location baselines, whereas appending the same information as coordinates before the classifier head does not, which is an observation about where the usable signal lives, not a general-purpose training recipe. We release the dataset and the generation and annotation scripts. Ambivision offers a replicable, theoretically grounded methodology for studying where computational and human perception diverge. The dataset is accessible via a kaggle link which will be provided upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Uncertainties, Explainable AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9NpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Ambivision, a dataset of over 200 generated images in which two animals share visual structure. It provides annotations for animal categories, eye positions, gaze directions, and bounding boxes.

Critical strengths

The paper raises an interesting question about what an explanation should tell us when an image has multiple valid interpretations. The annotations could make Ambivision a useful resource for studying this question.

Critical weaknesses
  1. My main concern is both interventions are anchored at the eye, so they probe closely related information. They provide little evidence about how the model uses contour shape, the arrangement of animal parts, texture, or figure–ground relationships, the kinds of structure that motivate the paper in the first place. An improvement after adding these annotations shows that the annotations are useful, but does not tell us much about how the model interprets the original image. The fact that eye-only annotations often outperform gaze arrows is important here. As the authors acknowledge, much of the benefit may come from locating an informative region. I would find comparisons with other landmarks or contour cues, and with arrows whose direction is randomized while their starting point is fixed, more informative about the role of gaze itself. A focused study of eye-based cues is reasonable, but the conclusions should stay within that scope. The current experiments do not establish that the models have learned the relational concepts discussed in the paper. The unsuccessful coordinate-concatenation experiment also does little to settle this question without a clearer account of its implementation and tuning.

  2. Section 5 counts a prediction as correct if it matches either animal. Under this rule, a model can consistently recognize the more salient animal and receive credit without ever recovering the alternative interpretation. I therefore find it difficult to interpret higher accuracy as evidence of better ambiguity resolution. This is especially relevant to the cue experiments. For a fixed image containing animals A and B, what happens when the cue changes from A to B? Does the prediction follow that change, or does the model continue to recognize the same animal more reliably? That would be a more direct test of cue-guided interpretation. The paper should also state clearly whether the evaluation images contain the annotations, since this substantially changes the meaning of the reported improvements.

  3. With only a few hundred base images, the split and labeling procedures are particularly important. I could not find a clear description of the train/validation/test sizes, the number of categories, or how the two valid labels enter the training loss. It is also unclear how the dataset labels map to the ImageNet classifier outputs.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Adressing Weaknesses

Thankyou so much for your feedback. The proposal of the additional baseline with a randomized arrow is useful, thankyou so much. However, we do include a test where a random "eye" area is marked. So the paper definitely concludes that this helps. You justly ask why it is important to talk about more than just the eye area, and in particular gaze direction: For the illusions we note by observation that the animals become distinguishable for humans if we include the gaze direction. The idea is to incorporate what makes the problem easier for humans rather than only AI. We argue, and show by accuracy improvement, that human interpretable concepts that lead to humans being able to distinguish the situation might help in general. We dont claim to know all about the most meaningful concepts, we just propose a dataset that allows for one such new angle.

W2: Counting either animal as correct is too lenient: We test both: on single class, and on allowing both animals to be correct, we will be more sure to highlight this better in a revised version. The reason why this might have failed was due to the page limit restrictions. We tested both so that it would be clear that neither is an issue and both lead to an accuracy improvement. This is the case.

W3: The training-test split was roughly 35% test, 65% training. We apologize and will include this in the revision. The training allows for all 1000 imagenet classes since we use pretrained models for Imagenet, so we use the normal classification problem and then just test whether one of the correct labels, or both, is chosen.

Reviewer_RDYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a dataset of AI-generated animal optical illusions in which two animal interpretations share largely the same visual structure. The dataset is motivated by Gestalt principles and includes additional annotations such as animal labels, eye coordinates, gaze-direction vectors, and bounding boxes. The main motivation is to study ambiguous visual perception and whether current vision and explainability methods can distinguish between two valid interpretations of the same image.

The paper investigates two main questions. First, the authors test whether explicitly highlighting the eye or gaze direction can improve classification. They evaluate several CNN and transformer architectures and compare gaze-direction markings, eye markings, no annotation, and random markings. They find that explicitly marking the relevant image region substantially improves classification, while concatenating gaze/eye coordinates before the final classifier does not. Interestingly, marking only the eye is often at least as effective as the gaze arrow itself. Second, the authors use Ambivision to qualitatively investigate existing explanation approaches including Grad-CAM, Integrated Gradients, PipNet and ACE. They argue that these approaches struggle to give discriminative explanations when the two interpretations depend on overlapping image regions. They complement this analysis with a ROAR experiment investigating whether performance is driven by local shortcuts.

Critical strengths

I think the strongest aspect of the work is the problem itself. Using deliberately ambiguous visual stimuli to test whether ML explanation methods can distinguish between competing interpretations is interesting and somewhat different from conventional saliency benchmarks. The shared-region setup exposes a limitation that is difficult to study using ordinary object-classification datasets. This is consequential to my assessment because I can see Ambivision being useful beyond the particular experiments in this paper if the dataset is validated more rigorously.

The dataset provides more than class labels, including eye coordinates, gaze vectors and bounding boxes, and the authors test eye cues, gaze cues, random cues and no cues rather than showing only the positive intervention. The consistency of the broad cueing effect across multiple CNN and transformer architectures is encouraging.

Critical weaknesses

The central claim of perceptual ambiguity is not sufficiently validated. This is my main concern. The paper is fundamentally motivated by human perceptual ambiguity, Gestalt organization, and a claimed divergence between human and machine perception. However, there is no systematic human evaluation establishing that the generated images actually have two recognizable interpretations, how consistently both interpretations are perceived, or that gaze direction is the cue responsible for switching between interpretations. The paper only provides informal observations that some users did not immediately identify the second animal. This matters because without such validation, it is difficult to know whether Ambivision is really measuring perceptual ambiguity or simply classification difficulty on unusual AI-generated images.

The evidence for a general/structural failure of current XAI methods is too qualitative. The paper makes a fairly strong claim that attribution-based methods cannot produce discriminative, human-interpretable explanations under this type of ambiguity. However, for Grad-CAM, Integrated Gradients, PipNet and ACE, the paper explicitly characterizes the analysis as qualitative and illustrative using selected examples, without estimating a failure rate. The examples are suggestive, but they are not enough for me to conclude that the problem is structurally inherent to these methods rather than being dependent on the models, images or particular explanation configuration. The ROAR analysis is a useful addition, but it answers a somewhat different question, namely whether classification relies on a small localized shortcut. It does not by itself demonstrate that the explanation methods are incapable of providing discriminative explanations.

Questions for authors
  1. Since a central motivation of Ambivision is genuine human perceptual ambiguity, have you conducted any controlled human evaluation measuring whether both animals are actually perceived in each generated image? In particular, what fraction of images are recognized as containing both animals, how consistent are interpretations across participants, and does manipulating/focusing on the annotated gaze or eye cue actually change the perceived interpretation? I think this would considerably strengthen the claim that Ambivision captures perceptual ambiguity rather than simply difficult AI-generated images.

  2. Can the authors quantify how often the explanations for the two competing classes are actually indistinguishable across the dataset? For example, could similarity between class-specific attribution maps be measured and compared with ordinary non-ambiguous image pairs or another suitable control? The current selected qualitative examples support the intuition, but I do not think they are sufficient to establish the claimed structural failure.

  3. How exactly are the train/validation/test splits constructed? Are animal pairs, prompts, or visually related generations shared across splits? Given the relatively small dataset and generative construction process, I think it is important to establish that improvements are not driven by related generated samples or pair-specific regularities.

  4. The eye-only condition appears competitive with the gaze-direction condition. Is the intended conclusion therefore mainly that explicitly cueing the relevant spatial region helps classification, rather than that gaze direction itself contains the crucial disambiguating information?

Reviewer_sYtVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Ambivision, a dataset of images generated to produce perceptual ambiguity. Using Ambivision as a controlled dataset, they probe the degree to which attribution methods applied to models trained on Ambivision can recover known ground truth. In practice, they find that common attribution methods do not produce discriminative explanations. They show that adding explicit cues to the dataset for training improves this discrimination.

Critical strengths
  • The paper is generally clear and easy to follow
Critical weaknesses
  • Dataset
    • Novelty.
      • For XAI. The idea of building a dataset with known ground truth, training a model on it, and evaluating the degree to which heatmaps recover the ground truth as a way to evaluate them is not novel.
        • Sixt, et al. "Do users benefit from interpretable vision? a user study, baseline, and dataset." ICLR (2022).
        • Hesse, et al. "Funnybirds: A synthetic vision dataset for a part-based analysis of explainable ai methods." ICCV (2023).
      • In general, it is not clear how novel the dataset is compared to Rostamkhani et al. (2024).
        • Rostamkhani, et al. "Illusory VQA: Benchmarking and enhancing multimodal models on visual Illusions." CVPR Workshop 2025.
    • Soundness.
      • The dataset is framed as "a dataset of optical illusions [...] grounded in Gestalt psychological principles such as [...] closure [...] designed to evoke genuine perceptual ambiguity" with a motivating Figure 1 example around bistable images.
        • The examples of the dataset presented in Figure 2 are not examples of bistable images (both animals can be perceived at the same time), nor do they evoke a strong perceptual ambiguity. There is no result in the paper that supports this claim (e.g., psychophysics experiments)
        • Similarly, it is questionable to what degree those images respect Gestalt principles. They are closer to a composite containing two animals. Without validation, it seems to be an overclaim
  • XAI results
    • Novelty.
      • The main result that attribution methods sometimes generate heatmaps that are not class-sensitive is not novel
        • Rebuffi, et al. "There and back again: Revisiting backpropagation saliency methods." CVPR (2020)
        • Gu, et al. "Understanding individual decisions of cnns via contrastive backpropagation." ACCV (2018)
        • Williamson, et al. "Case: Contrastive activation for saliency estimation." arXiv (2025)
    • Soundness.
      • The assumption that a controlled dataset can be used to evaluate attribution methods is only true if we can be sure that the model did in fact learn the correct strategy, e.g., we aim for a model near perfect accuracy. In this case, the model performs poorly, about 60%. Therefore, it's impossible to tell apart whether the limitation of heatmaps is the product of the methods or of the model.
Questions for authors
  • How do the authors validate that the dataset adheres to Gestalt principles? Which specific principle generates the ambiguity?
  • Why is the dataset a novel and useful use case for studying attribution methods?
  • How can the authors tell apart whether the limitation of attribution methods observed can be attributed to poor methods rather than poorly performing models?
  • Could you clarify: how are the models trained (what is the target if models can predict both classes), how many classes of animals are considered, to get a sense of chance level.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.