- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This article proposes CRISP, a method for image classification domain generalization. This method learns visual primitives through the concept bottleneck layer and uses differentiable unary, binary, and ternary predicates to describe the appearance and spatial relationships of the primitives. The model further learns the combination of category related relationships and combines them with local appearance features for classification. The author conducted experiments on the DomainBed and CUB-DG datasets. The results showed that CRISP achieved good performance on some datasets, especially on the CUB-DG dataset using ResNet-50, which showed significant improvement.
Critical strengths
- This article studies domain generalization from the perspective of visual primitives and their spatial relationships, which is different from common feature alignment and data augmentation methods. The research approach has certain value.
- The method design is relatively complete, and the visual element extraction, spatial relationship calculation, and classification modules can be trained end-to-end
- The experiment covered six datasets and was tested using ResNet-50 and ViT-B/14 backbone networks. The author also provides module ablation, comparison of different predicate combinations, and visual analysis.
Critical weaknesses
-
Lack of direct verification of structural invariance. The authors believe that spatial relationships are more likely to remain stable across domains than appearance features, but currently this viewpoint is mainly supported by classification accuracy. There is a lack of quantitative analysis of differences in relational representations across domains.
-
There is still room for discussion on the novelty of the method. There have been related studies on visual primitive extraction, spatial predicates, concept bottlenecks, and sparse weight selection. This article mainly combines these techniques for domain generalization, but the essential differences from existing relational modeling methods are not clear enough.
-
The experimental results are not consistent. Under the ResNet-50 setting, CRISP performs worse than ERM++ on PACS, OfficeHome, and DomainNet. When using ViT-B/14, the performance on CUB-DG is also inferior to ERM++. The article did not fully explain these differences. Although SOTA performance has been achieved on average, there should be more discussion of the datasets where SOTA performance was not achieved.
-
The authors mentioned that the enumeration complexity of ternary predicates is O(K³), but did not provide a systematic comparison with other methods in terms of training time, inference speed, or memory overhead. That is to say, the computational overhead has not been experimentally compared with other methods. In addition, although the authors analyzed the impact of different K values on performance in Figure 4, they did not quantitatively evaluate the corresponding computational overhead.
Questions for authors
-
What are the main differences between CRISP and existing visual relation modeling methods?
-
Is there any empirical evidence to verify the O(K³) enumeration complexity? If K is set too high, will it cause a significant increase in GPU memory overhead?
-
How can the performance degradation of CRISP on PACS, OfficeHome, DomainNet, and under the ViT-B/14 setting be explained?
-
Can the authors provide training time, inference overhead, and GPU memory usage of the model and compare them with the main baselines?
9 comments
Q2. Empirical analysis of enumeration complexity and GPU memory overhead.
Thank you for raising this concern. We agree that an empirical analysis of the computational and memory overhead would help clarify the practical scalability of CRISP's structural scoring mechanism. To this end, we measure the training time per step and GPU memory usage while varying the number of primitives (K) from 8 to 64, using both backbones, ResNet-50 and VIT-B/14, and with the full CRISP training procedure. To ensure that there is no variation, we configure the default allocator with cudnn.benchmark=False, disabling cuDNN autotuning. This gives us a consistent result with no variation in memory measurements across multiple runs. We have also cleared out cached memory between runs, and we have set up GPU synchronization before and after each training iteration to ensure we have an accurate measurement of allocated and reserved memory. We report the results below.
ResNet-50:
| Primitives (K) | Productions | Training time/step (ms) | Peak allocated (GiB) | Peak reserved (GiB) |
| 8 | 464 | 143.4 | 11.666 | 12.463 |
| 16 | 2,144 | 145.1 | 11.672 | 12.461 |
| 32 | 11,200 | 150.9 | 11.701 | 12.625 |
| 64 | 66,432 | 170.0 | 11.864 | 12.783 |
ViT-B/14:
| Primitives (K) | Productions | Training time/step (ms) | Peak allocated (GiB) | Peak reserved (GiB) |
| 8 | 464 | 400.9 | 18.000 | 18.352 |
| 16 | 2,144 | 403.6 | 18.005 | 18.420 |
| 32 | 11,200 | 411.3 | 18.024 | 18.656 |
| 64 | 66,432 | 432.5 | 18.295 | 18.727 |
For both backbones, increasing from 8 to 64 increases the number of productions by approximately 143 times. Over this range of , for ResNet-50, the peak allocated memory introduces additionally 0.320 GB (2.5%), and training time increase modestly by 26.6 ms per step. For the ViT-B/14 backbone, and over the same range, the peak allocated memory increases by 0.295 GiB, or 1.6%, and training time increases by 31.6 ms per step, or 7.9%. The total training footprint therefore grows modestly within the evaluated range, despite the rapid growth in production count.
This relatively modest memory overhead can be attributed in part to our vectorized implementation, which evaluates relations using batched tensor operations over precomputed triplet-index buffers, rather than enumerating individual productions through Python loops. We have added the corresponding explanation of this procedure in Appendix E, which consists of three steps: (1) static enumeration, (2) vectorized predicate computation, and (3) shared parameters between predicate families. For self-containment, we also list these details below.
Q3. How can the performance degradation of CRISP on PACS, OfficeHome, DomainNet, and under the ViT-B/14 setting be explained?
Thank you for raising this point. We agree that analyzing these performance decrease would give a better clarity to the limitation of CRISP: its benefit depends on whether the extracted primitives and their relations provide reliable, class-discriminative information. When primitive localization is ambiguous, or the recovered relations are shared across categories, relational modeling can introduce misleading evidence and thus reduce the overall performance. We list our explanations below:
On PACS, particularly Sketch, abstraction can remove or distort discriminative object parts. As illustrated in Fig.5, correctly classified elephant sketches retain identifiable head and trunk regions, whereas the example misclassified as a giraffe exhibits a less distinct head–trunk structure. This suggests that CRISP becomes less effective when the domain shift obscures the primitives needed to establish informative relations.
On OfficeHome, the challenge also involves semantic ambiguity between otherwise recognizable structures. For example, calculators and calendars both contain regular grids of rectangular elements. The failed calculator example in our figure illustrates how similar local primitives and spatial layouts can support an incorrect category, even when those structures are successfully localized. On DomainNet, our hedgehog example shows highlighted regions spanning multiple drawings and repeated textures. These fragmented localizations can produce relations that reflect incidental patterns rather than the structure of the intended object.
Under ViT-B/14, the degradation further indicates that CRISP’s contribution depends on the backbone representation. Since the visual representation produced by ViT already contains spatial information, providing additional relational representation provides limited complementary information while remaining susceptible to errors in primitive selection and localization. In this case, the benefit of explicitly modeling relations may be outweighed by the noise introduced during their construction.
We have added an extended discussion section and added these explanations.
Q1. What are the main differences between CRISP and existing visual relation modeling methods?
Thank you for this question, we agree that a more explicit comparison with existing visual relation modeling methods would clarify CRISP's positioning. We have therefore expanded the discussion to include the following comparisons.
Santoro et al. [1] introduce Relation Networks, which learn pairwise relational functions over object representations using neural networks. In contrast, CRISP employs predefined geometric predicates operating on learned primitive coordinates and extents, rather than learning arbitrary relational functions over visual features.
Yang et al. [2] propose Graph R-CNN, which models object interactions through relation proposals and graph convolutions for scene graph generation. Similarly, Lu et al. [3] model semantic subject-predicate-object relationships using language priors, while Xu et al. [4] employ iterative message passing between objects and relations, and Zellers et al. [5] exploit recurring relational patterns and global context. Unlike these methods, CRISP does not aim to predict semantic relationships or construct scene graphs. Instead, it learns latent primitives from image-level supervision and evaluates their spatial relationships using a fixed vocabulary of differentiable geometric predicates, without requiring explicit object, part, or relation annotations.
Overall, CRISP differs from these approaches in its primary objective and relational representation. Rather than learning general-purpose relational functions for scene understanding, CRISP uses explicit geometric relationships as a structural inductive bias for domain-generalized classification, aiming to capture structural information that remains informative under appearance shifts.
References [1] Santoro, A., Raposo, D., Barrett, D., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017). A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems. Curran Associates, Inc.
[2] Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D. (2018). Graph R-CNN for Scene Graph Generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol. 11205. Springer, Cham. https://doi.org/10.1007/978-3-030-01246-5_41
[3] Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L. (2016). Visual Relationship Detection with Language Priors. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol. 9905. Springer, Cham. https://doi.org/10.1007/978-3-319-46448-0_51
[4] D. Xu, Y. Zhu, C. B. Choy and L. Fei-Fei, "Scene Graph Generation by Iterative Message Passing," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 3097-3106, doi: 10.1109/CVPR.2017.330.
[5] R. Zellers, M. Yatskar, S. Thomson and Y. Choi, "Neural Motifs: Scene Graph Parsing with Global Context," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 5831-5840, doi: 10.1109/CVPR.2018.00611.