acceptodds
Under review as a conference paper at ICLR 2027

Impact of color in computer and human-based object recognition under occlusion

Abstract

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to computer vision. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition is negatively impacted by generated camouflaging occluders, with scene-based camouflage performing worse than object-based, and most transformers demonstrate a similar trend. Interestingly, classical CNNs show a strong opposite trend, performing worse on neutral-colored occlusions, suggesting that contextually-related colors, particularly scene-based, benefit object recognition rather than disrupting shape or boundary cues. Additionally, generated camouflaging occluders for training augmentation improves robustness to unseen camouflage for selected CNN model, and occlusion metrics suggest trends about learned mechanisms. Findings have interesting implications for alignment in human and computer vision, model vulnerabilities, and explainability.

Primary area:
interpretability and explainable AI
Keywords:
object recognition, occlusions, color perception, models, human alignment, camouflage, context

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mX1COpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims at improving our understanding of recognition strategies in humans and neural networks. In particular, it compares humans and neural networks (e.g., CNNs, ViTs) on a specific case of partial occlusion, i.e., camouflage occlusion. The occlusion is carried out by vertical, horizontal or diagonal occluding stripes that are either of the dominant color of the background, the dominant color of the object or for control in black. The main results show that performance of humans as well as neural networks (to differing extends) is decreased when images are occluded with camouflage stripes. Additionally, the paper proposes three metrics based on the attribution technique GradCAM++ to further analyze the bahavior of neural networks when exposed to camouflage occlusion.

Critical strengths
  • Relevant and interesting research question: Investigating how contextually colored occlusions affect object recognition in humans and neural networks is a genuinely interesting research question.

  • Exploration of training augmentation with colored-occlusion is a reasonable approach to investigate whether models can learn increased robustness to camouflage occlusions.

  • I also value the reproducibility efforts of the authors by providing the code for the experiments.

Critical weaknesses
  • Human–model comparison uses different tasks: without making the task better comparable, it is hard draw conclusions. Generally accuracies can be a supplement, but models would have to be evaluated on the exact same tasks as the humans were evaluated to compare them in a fair way (potentially with agreement of high-level predictions between the camouflaged image and the original)

  • CNN–ViT training recipes are not controlled: CNN recipes used are very old, the comparison to ViTs may be confounded by weaker training recipes. The ConvNeXt paper [1] experimentally showed that the majority of differences in performance between CNNs and ViT stem from different training recipes used. Moreover, Burgert et al. [2] showed that state-of-the-art training makes CNNs very similarly robust to feature suppression as ViTs, most differences between the architectures are attributable to different training recipes, too.

  • GradCAM-based metrics offer limited additional insights: GradCAM is architecture and layer-dependent. It is hard to attribute GradCAM visualization difficulties for corrupted images with camouflage occlusion to the actual difficulties of the model to recognize the object and properties of the attribution method itself. The observed artifacts may reflect sensitivity to out-of-distribution corruptions or limitations of GradCAM, rather than meaningful differences in the models' internal representations or recognition strategies.

  • Colored-CutOut augmentation experiment evaluation is limited: only one small dataset is used and the difference between CutOut black (normal CutOut) to colored-Cutout with the dominant colors of the image is too small to judge the positive effect on simply one dataset. For reasonable ViT-training and results the dataset size is too small and/or the used standard augmentation pipeline is too weak (shown by the performance which is only about 2/3 of the ResNet). Further, a control via random colors chosen for colored-CutOut is missing. In the current shape the contribution and potential conclusion that can be drawn from that experiment are limited.

  • Limited interpretation and connection to existing literature: While the paper cites some relevant prior work, the discussion and interpretation of results is largely descriptive. Given that the aim of the paper is to improve our understanding of recognition strategies in humans and neural networks, a deeper discussion of potential mechanisms, alternative explanations and connection to the broader literature on comparing humans and models would be beneficial. As an example: Burgert et al. [2] or Subramanian et al. [3].

  • Clarity in writing and formatting concerns: In the current form, the paper does not meet ICLRs standards for clarity in writing, presentation and formatting. To help the authors to improve their manuscript, I list several issues that stood out during the review. This list is not exhaustive, and the manuscript would benefit from a thorough revision.

Miscellaneous

  • I would not list the code as a contribution, in final version code should be accessible, but I do not see it as a major contribution; though a relevant thing to provide

Clarity, Presentation and Formatting

  • Caption of Figure 1 with typo, this needs to be better explained and be more precise, assume the reader sees the Figure before anything else
  • In Introduction, the credit to the original occlusion-based data augmentation should be given to Cutout (you do cite it later, so I would cite it in the introduction too)
  • L067/068: Newer perspectives on feature reliance of DNN should be accounted, too [2]
  • L077-081: Results cannot „aim“, and saying this work recreates partial occlusion is underspecified; please revisit those sentence and improve clarity
  • Contributions do not align well how the abstract is written
  • If the generated views (scene, object, control, original) are going to be separate datasets, please make it explicit, I felt like 3.1 and 3.3 did no use the same and precise wording
  • L223: is it 320 images or 240 or 480? Until now, I thought 80 augmented (each of them has 4 version) + 160 random images.
  • L233: Typo „select models“
  • Equations 1,2,3: You first mention NOP, NOR and then AIO and define them in a reversed way, this is confusing
  • The Equations are introduced in a confusing way, not every variable is fully explained
  • L264/463: If I understood the paper well, I would not write „finetune“ here, better write "manually selected", finetuning is too much associated with training a model
  • Limitations and future work should be presented in the conclusion, at the very end
  • Format: Text is horizontally too close to some Figures
  • Format: not right way of citing the paper in the text, in latex, use e.g., \citep{he_2016} to cite properly
  • Typo: e.g., L087 and L102 missing full dots and in the conclusion, too, L399 classical written capitalized

References

[1] Liu et al., "A ConvNet for the 2020s", CVPR 2022.

[2] Burgert et al., "ImageNet-tained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression", NeurIPS 2025.

[3] Subramanian et al., "Spatial-frequency channels, shape bias, and adversarial robustness", NeurIPS 2023.

Questions for authors

Please see critical weaknesses.

Reviewer_MdgSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This research investigates how partial occlusions and camouflage color properties impact object recognition in both human participants and computer vision models, including convolutional neural networks (CNNs) and transformers. The authors created a custom dataset featuring scene-based and object-based color occlusions to test recognition performance across various orientations and developed novel metrics—such as non-occlusion precision (NOP), non-occlusion recall (NOR), and activation in occlusion (AIO)—to quantify model activation and internal behavior. The findings reveal that while human recognition and transformers are generally hindered by camouflaging occluders, classical CNNs perform worst with neutral colors, suggesting that contextually relevant colors can actually aid machine classification. Furthermore, integrating image-relevant color augmentation during training significantly enhances CNN robustness against unseen camouflage.

Critical strengths

Cross-Paradigm Benchmark: Establishes a direct empirical bridge comparing human visual perception against multiple classes of computer vision models under controlled color occlusion.

Formalized Interpretability Metrics: Introduces quantitative metrics (NOP, NOR, AIO) that allow dataset-wide measurement of feature map alignment under occlusion rather than relying on image-by-image heatmap inspection.

Practical & Effective Augmentation: Demonstrates that training with Cutout-color yields superior generalization against complex visual camouflage compared to standard neutral-patch Cutout

Critical weaknesses

Model-Human Divergence in CNNs: Exposes a notable vulnerability in classical CNNs, which rely heavily on background/context color cues for object recognition rather than processing true occlusion boundaries

Omission of Occluder Texture: The study isolates color properties but does not vary occluder texture, which plays a critical role in real-world camouflage and CNN texture bias

Small Comparative Dataset: The primary human-model comparative dataset contains only 80 images, constrained by participant fatigue and manual fine-tuning of color clusters

Questions for authors

Can you add model results with (a) a luminance-matched gray, (b) random colors, and (c) object/scene colors taken from a mismatched image or class? If CNNs also do well with mismatched colors, the "context" interpretation fails. If they do well only with the image's own colors, a color-shortcut explanation is likely.

Reviewer_9fbqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper gives new analysis on how model and human is activated by diverse occlusion conditions.

Critical strengths

This study topic is interesting and quite new for compariing human activation & neural model activation

Critical weaknesses

The paper topic is interesting and the result is also quite new for neuroscientific aspect. However, the paper seems to be very unpolished and difficult to follow. It is really hard to read the main text as the citation texts are mixed on the main text scripts. Also the figure placement, table placement, caption alignment is very tight and they are not smooth at all.

Also in the overall scripts, the written explanation is too lengthy and is it difficult to follow the experiment protocols. I am bit suspicious that most of the scripts are generated by AI.

The paper need major polishment on its method, and main proposal must be clearer, and supportive figures are required.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.