- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Ambivision, a dataset of over 200 generated images in which two animals share visual structure. It provides annotations for animal categories, eye positions, gaze directions, and bounding boxes.
Critical strengths
The paper raises an interesting question about what an explanation should tell us when an image has multiple valid interpretations. The annotations could make Ambivision a useful resource for studying this question.
Critical weaknesses
-
My main concern is both interventions are anchored at the eye, so they probe closely related information. They provide little evidence about how the model uses contour shape, the arrangement of animal parts, texture, or figure–ground relationships, the kinds of structure that motivate the paper in the first place. An improvement after adding these annotations shows that the annotations are useful, but does not tell us much about how the model interprets the original image. The fact that eye-only annotations often outperform gaze arrows is important here. As the authors acknowledge, much of the benefit may come from locating an informative region. I would find comparisons with other landmarks or contour cues, and with arrows whose direction is randomized while their starting point is fixed, more informative about the role of gaze itself. A focused study of eye-based cues is reasonable, but the conclusions should stay within that scope. The current experiments do not establish that the models have learned the relational concepts discussed in the paper. The unsuccessful coordinate-concatenation experiment also does little to settle this question without a clearer account of its implementation and tuning.
-
Section 5 counts a prediction as correct if it matches either animal. Under this rule, a model can consistently recognize the more salient animal and receive credit without ever recovering the alternative interpretation. I therefore find it difficult to interpret higher accuracy as evidence of better ambiguity resolution. This is especially relevant to the cue experiments. For a fixed image containing animals A and B, what happens when the cue changes from A to B? Does the prediction follow that change, or does the model continue to recognize the same animal more reliably? That would be a more direct test of cue-guided interpretation. The paper should also state clearly whether the evaluation images contain the annotations, since this substantially changes the meaning of the reported improvements.
-
With only a few hundred base images, the split and labeling procedures are particularly important. I could not find a clear description of the train/validation/test sizes, the number of categories, or how the two valid labels enter the training loss. It is also unclear how the dataset labels map to the ImageNet classifier outputs.
Questions for authors
See weaknesses.
Adressing Weaknesses
Thankyou so much for your feedback. The proposal of the additional baseline with a randomized arrow is useful, thankyou so much. However, we do include a test where a random "eye" area is marked. So the paper definitely concludes that this helps. You justly ask why it is important to talk about more than just the eye area, and in particular gaze direction: For the illusions we note by observation that the animals become distinguishable for humans if we include the gaze direction. The idea is to incorporate what makes the problem easier for humans rather than only AI. We argue, and show by accuracy improvement, that human interpretable concepts that lead to humans being able to distinguish the situation might help in general. We dont claim to know all about the most meaningful concepts, we just propose a dataset that allows for one such new angle.
W2: Counting either animal as correct is too lenient: We test both: on single class, and on allowing both animals to be correct, we will be more sure to highlight this better in a revised version. The reason why this might have failed was due to the page limit restrictions. We tested both so that it would be clear that neither is an issue and both lead to an accuracy improvement. This is the case.
W3: The training-test split was roughly 35% test, 65% training. We apologize and will include this in the revision. The training allows for all 1000 imagenet classes since we use pretrained models for Imagenet, so we use the normal classification problem and then just test whether one of the correct labels, or both, is chosen.