acceptodds
Under review as a conference paper at ICLR 2027

CRISP: Domain Generalization through Compositional Spatial Relations over Visual Primitives

Abstract

Domain generalization requires identifying stable representations that support reliable classification across domains. Domains may differ in low-level attributes, such as color, texture, or visual style, while preserving the same structural relationships among their underlying components. Existing methods primarily address these differences by improving the training process or aligning features across domains. However, since they leave this shared compositional structure implicit, they may overlook a more reliable source of invariance and consequently generalize less effectively to unseen domains. We propose Compositional Relational Invariance from Spatial Primitives (CRISP), an image classification framework that factors visual recognition into visual primitives and their relational composition. We represent these compositions using soft unary, binary, and ternary predicates over primitive locations and appearance, yielding differentiable measures of spatial and visual alignment that can be learned end-to-end. To learn primitives and relational structure jointly, we design an end-to-end architecture with three components: (1) a visual backbone that extracts generalized features, (2) a concept bottleneck layer that maps these features to primitive heatmaps with differentiable spatial coordinates, and (3) a structural scoring layer that evaluates candidate spatial relations among the detected primitives. Finally, we compute class probability from the joint evidence of its class-specific relational compositions and localized primitive appearance. We evaluate CRISP on five real-world image-classification datasets from the widely used DomainBed suite, covering shifts in depiction style, dataset provenance, and camera-trap location. To assess its ability to exploit compositional structure, we additionally evaluate it on CUB-DG, a fine-grained bird-recognition benchmark in which distinguishing species depends on combinations of part-level attributes. Across CUB-DG and the DomainBed benchmark suite, CRISP improves accuracy by a large margin of 7.9 percentage points on CUB-DG and 1.2 percentage points over existing DG methods on DomainBed, achieving the new state-of-the-art on both benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Domain Generalization, Compositional Spatial Relations, Explicit structural recognition pipeline

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vo6hOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This article proposes CRISP, a method for image classification domain generalization. This method learns visual primitives through the concept bottleneck layer and uses differentiable unary, binary, and ternary predicates to describe the appearance and spatial relationships of the primitives. The model further learns the combination of category related relationships and combines them with local appearance features for classification. The author conducted experiments on the DomainBed and CUB-DG datasets. The results showed that CRISP achieved good performance on some datasets, especially on the CUB-DG dataset using ResNet-50, which showed significant improvement.

Critical strengths
  1. This article studies domain generalization from the perspective of visual primitives and their spatial relationships, which is different from common feature alignment and data augmentation methods. The research approach has certain value.
  2. The method design is relatively complete, and the visual element extraction, spatial relationship calculation, and classification modules can be trained end-to-end
  3. The experiment covered six datasets and was tested using ResNet-50 and ViT-B/14 backbone networks. The author also provides module ablation, comparison of different predicate combinations, and visual analysis.
Critical weaknesses
  1. Lack of direct verification of structural invariance. The authors believe that spatial relationships are more likely to remain stable across domains than appearance features, but currently this viewpoint is mainly supported by classification accuracy. There is a lack of quantitative analysis of differences in relational representations across domains.

  2. There is still room for discussion on the novelty of the method. There have been related studies on visual primitive extraction, spatial predicates, concept bottlenecks, and sparse weight selection. This article mainly combines these techniques for domain generalization, but the essential differences from existing relational modeling methods are not clear enough.

  3. The experimental results are not consistent. Under the ResNet-50 setting, CRISP performs worse than ERM++ on PACS, OfficeHome, and DomainNet. When using ViT-B/14, the performance on CUB-DG is also inferior to ERM++. The article did not fully explain these differences. Although SOTA performance has been achieved on average, there should be more discussion of the datasets where SOTA performance was not achieved.

  4. The authors mentioned that the enumeration complexity of ternary predicates is O(K³), but did not provide a systematic comparison with other methods in terms of training time, inference speed, or memory overhead. That is to say, the computational overhead has not been experimentally compared with other methods. In addition, although the authors analyzed the impact of different K values on performance in Figure 4, they did not quantitatively evaluate the corresponding computational overhead.

Questions for authors
  1. What are the main differences between CRISP and existing visual relation modeling methods?

  2. Is there any empirical evidence to verify the O(K³) enumeration complexity? If K is set too high, will it cause a significant increase in GPU memory overhead?

  3. How can the performance degradation of CRISP on PACS, OfficeHome, DomainNet, and under the ViT-B/14 setting be explained?

  4. Can the authors provide training time, inference overhead, and GPU memory usage of the model and compare them with the main baselines?

9 comments
AuthorsOpenReview ↗

Q1. What are the main differences between CRISP and existing visual relation modeling methods?

Thank you for this question, we agree that a more explicit comparison with existing visual relation modeling methods would clarify CRISP's positioning. We have therefore expanded the discussion to include the following comparisons.

Santoro et al. [1] introduce Relation Networks, which learn pairwise relational functions over object representations using neural networks. In contrast, CRISP employs predefined geometric predicates operating on learned primitive coordinates and extents, rather than learning arbitrary relational functions over visual features.

Yang et al. [2] propose Graph R-CNN, which models object interactions through relation proposals and graph convolutions for scene graph generation. Similarly, Lu et al. [3] model semantic subject-predicate-object relationships using language priors, while Xu et al. [4] employ iterative message passing between objects and relations, and Zellers et al. [5] exploit recurring relational patterns and global context. Unlike these methods, CRISP does not aim to predict semantic relationships or construct scene graphs. Instead, it learns latent primitives from image-level supervision and evaluates their spatial relationships using a fixed vocabulary of differentiable geometric predicates, without requiring explicit object, part, or relation annotations.

Overall, CRISP differs from these approaches in its primary objective and relational representation. Rather than learning general-purpose relational functions for scene understanding, CRISP uses explicit geometric relationships as a structural inductive bias for domain-generalized classification, aiming to capture structural information that remains informative under appearance shifts.

References [1] Santoro, A., Raposo, D., Barrett, D., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017). A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems. Curran Associates, Inc.

[2] Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D. (2018). Graph R-CNN for Scene Graph Generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol. 11205. Springer, Cham. https://doi.org/10.1007/978-3-030-01246-5_41

[3] Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L. (2016). Visual Relationship Detection with Language Priors. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol. 9905. Springer, Cham. https://doi.org/10.1007/978-3-319-46448-0_51

[4] D. Xu, Y. Zhu, C. B. Choy and L. Fei-Fei, "Scene Graph Generation by Iterative Message Passing," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 3097-3106, doi: 10.1109/CVPR.2017.330.

[5] R. Zellers, M. Yatskar, S. Thomson and Y. Choi, "Neural Motifs: Scene Graph Parsing with Global Context," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 5831-5840, doi: 10.1109/CVPR.2018.00611.

AuthorsOpenReview ↗

Q2. Empirical analysis of enumeration complexity and GPU memory overhead.

Thank you for raising this concern. We agree that an empirical analysis of the computational and memory overhead would help clarify the practical scalability of CRISP's structural scoring mechanism. To this end, we measure the training time per step and GPU memory usage while varying the number of primitives (K) from 8 to 64, using both backbones, ResNet-50 and VIT-B/14, and with the full CRISP training procedure. To ensure that there is no variation, we configure the default allocator with cudnn.benchmark=False, disabling cuDNN autotuning. This gives us a consistent result with no variation in memory measurements across multiple runs. We have also cleared out cached memory between runs, and we have set up GPU synchronization before and after each training iteration to ensure we have an accurate measurement of allocated and reserved memory. We report the results below.

ResNet-50: 

Primitives (K)ProductionsTraining time/step (ms)Peak allocated (GiB)Peak reserved (GiB)
8464143.411.66612.463
162,144145.111.67212.461
3211,200150.911.70112.625
6466,432170.011.86412.783

ViT-B/14:

Primitives (K)ProductionsTraining time/step (ms)Peak allocated (GiB)Peak reserved (GiB)
8464400.918.00018.352
162,144403.618.00518.420
3211,200411.318.02418.656
6466,432432.518.29518.727

For both backbones, increasing KK from 8 to 64 increases the number of productions by approximately 143 times. Over this range of KK, for ResNet-50, the peak allocated memory introduces additionally 0.320 GB (2.5%), and training time increase modestly by 26.6 ms per step. For the ViT-B/14 backbone, and over the same KK range, the peak allocated memory increases by 0.295 GiB, or 1.6%, and training time increases by 31.6 ms per step, or 7.9%. The total training footprint therefore grows modestly within the evaluated range, despite the rapid growth in production count.

This relatively modest memory overhead can be attributed in part to our vectorized implementation, which evaluates relations using batched tensor operations over precomputed triplet-index buffers, rather than enumerating individual productions through Python loops. We have added the corresponding explanation of this procedure in Appendix E, which consists of three steps: (1) static enumeration, (2) vectorized predicate computation, and (3) shared parameters between predicate families. For self-containment, we also list these details below.

AuthorsOpenReview ↗

Q3. How can the performance degradation of CRISP on PACS, OfficeHome, DomainNet, and under the ViT-B/14 setting be explained?

Thank you for raising this point. We agree that analyzing these performance decrease would give a better clarity to the limitation of CRISP: its benefit depends on whether the extracted primitives and their relations provide reliable, class-discriminative information. When primitive localization is ambiguous, or the recovered relations are shared across categories, relational modeling can introduce misleading evidence and thus reduce the overall performance. We list our explanations below:

On PACS, particularly Sketch, abstraction can remove or distort discriminative object parts. As illustrated in Fig.5, correctly classified elephant sketches retain identifiable head and trunk regions, whereas the example misclassified as a giraffe exhibits a less distinct head–trunk structure. This suggests that CRISP becomes less effective when the domain shift obscures the primitives needed to establish informative relations.

On OfficeHome, the challenge also involves semantic ambiguity between otherwise recognizable structures. For example, calculators and calendars both contain regular grids of rectangular elements. The failed calculator example in our figure illustrates how similar local primitives and spatial layouts can support an incorrect category, even when those structures are successfully localized. On DomainNet, our hedgehog example shows highlighted regions spanning multiple drawings and repeated textures. These fragmented localizations can produce relations that reflect incidental patterns rather than the structure of the intended object.

Under ViT-B/14, the degradation further indicates that CRISP’s contribution depends on the backbone representation. Since the visual representation produced by ViT already contains spatial information, providing additional relational representation provides limited complementary information while remaining susceptible to errors in primitive selection and localization. In this case, the benefit of explicitly modeling relations may be outweighed by the noise introduced during their construction.

We have added an extended discussion section and added these explanations.

Reviewer_Lyv4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRISP for domain-generalized image classification. Its core idea is to explicitly model local visual primitives and their spatial relations to reduce reliance on domain-specific appearance cues. The method learns primitive heatmaps from backbone features and extracts spatial locations, extents, and appearance descriptors. It then constructs class-specific structural scores using unary appearance predicates, binary spatial predicates, and ternary angular predicates, and combines these scores with a localized ROI appearance branch for classification. The model is trained using image-level class labels without additional part or relation annotations. The authors evaluate CRISP on five DomainBed datasets and CUB-DG. The results show good average performance, particularly on CUB-DG, although the gains vary substantially across target domains and backbones.

Critical strengths
  1. Explicitly incorporating spatial relations into domain generalization is a valuable modeling direction. Rather than relying entirely on the backbone to implicitly encode object structure, the method computes relative positions, distances, and angles from primitive coordinates and uses these relations for classification. This introduces an explicit geometric inductive bias without requiring additional part-level annotations. The design is well motivated and has practical value for tasks where appearance varies substantially but local structure remains discriminative.
  2. Some experimental results are compelling, and the ablations support the contribution of the relational module.In Table 1, with ResNet-50, the average accuracy on CUB-DG increases from 61.1% for ERM++ to 69.0%, with a substantial improvement on TerraIncognita as well. Table 2 shows that removing the structural scoring layer reduces average DomainBed performance with both backbones. In Table 3, CUB-DG accuracy progressively improves as binary and ternary predicates are added to the configuration without relations. These results support the usefulness of explicit relational modeling within the proposed architecture and provide empirical evidence beyond the methodological motivation.
Critical weaknesses

1.The central mechanism assumes that the learned heatmaps correspond to comparable object regions across domains, but the paper does not establish this clearly. The heatmap-to-coordinate mapping and the concentration/diversity regularizers encourage spatially distinct primitives, yet they do not prevent the model from relying on background or source-specific cues. Figures 2, 3, and 5 provide useful examples, but there is no quantitative evaluation of cross-domain localization consistency or relation stability. The results therefore show that relational modeling can improve classification, while providing weaker evidence that CRISP learns the interpretable and domain-stable compositions claimed by the paper.

2.With ResNet-50, CRISP improves the average DomainBed score, but the improvement is not uniform across target domains: it is below ERM++ on PACS, OfficeHome, and DomainNet, and the per-domain results include noticeable drops on PACS Sketch and CUB-DG Paint. This does not by itself invalidate the aggregate gain, but it raises the possibility of negative transfer under particular domain shifts. The paper would be more convincing if it characterized which types of domains benefit from the relational representation and which are more vulnerable. The smaller differences obtained with ViT-B/14 also make it important to know whether the reported gains are stable across runs.

Questions for authors

1.Is there quantitative evidence that, within the same class, the same primitive consistently localizes corresponding object regions in source and unseen target domains? Could the authors provide evaluations of part correspondence, foreground coverage, or localization consistency under controlled style changes, rather than only selected examples? This does not require adding part annotations during training; the aim is to validate the cross-domain correspondence assumption underlying the method.

2.Could the authors report how performance changes when removing the concentration regularizer, the diversity regularizer, or both? This ablation would help determine whether the regularization design affects primitive localization or contributes to the lower performance on some target domains. Please also clarify whether the regularization weights were selected using only source-domain validation data.

3.Could the authors characterize which types of domain shifts benefit from CRISP and which are more challenging? In particular, how should the lower performance on PACS Sketch and CUB-DG Paint be interpreted relative to ERM++? This would help clarify the practical scope and failure modes of the proposed method.

5 comments
AuthorsOpenReview ↗

Q1: Adding quantitative evidence that, within the same primitive, consistently localizes object regions in the source and unseen target domains?

Thank you for your suggestions. We agree that quantitative evidence of primitive consistency across domains would strengthen CRISP’s contribution. We have identified that CUB-DG is generated through style transfer [1] and also has corresponding bounding box and part annotations, which can be used as a reliable source for quantitatively evaluating whether the same primitive consistently localizes object regions in the source and unseen target domains. We evaluate two complementary metrics to examine whether the learned primitives remain consistent under domain shifts. For the following experiments, we use the bird’s bounding boxes and anatomical part locations provided per image from the official CUB dataset (https://www.vision.caltech.edu/datasets/cub_200_2011/) as the ground-truth labels to verify the primitives and relation consistency.

Primitive foreground localization. We first measure the proportion of primitive centroids that fall within the annotated bird bounding boxes (centroid-in-box). Using the trained CRISP checkpoint and Art as the held-out target domain and Photo, Cartoon, and Paint as source domains, we obtain the following results:

Art (Target)PhotoCartoonPaint
Centroid-in-box (%)72.273.875.074.4

The results show that 74.4% of primitive centroids fall within the foreground regions across the source domain, compared to 72.2% on the unseen Art domain. This suggests that CRISP maintains similar foreground localization behavior across domains despite being trained with only image-level supervision.

Primitive-to-part consistency. We further examine whether individual primitive indices consistently correspond to the same semantic bird parts. Specifically, we assign each primitive centroid to its nearest annotated part (e.g., beak, breast, tail) using Euclidean distance in normalized image coordinates, with a matching threshold of 0.30. Primitives without a valid match are excluded from the part-assignment counts.

For each primitive index kk, we measure the proportion of valid assignments corresponding to its most frequently associated semantic part. Let nk,pDn_{k,p}^{\mathcal D} denote the number of assignments of primitive kk to part pp over domains D\mathcal D. The agreement is defined as:

\frac{\max_p n_{k,p}^{\mathcal D}} {\sum_p n_{k,p}^{\mathcal D}}.$$ This measures the average consistency of semantic part assignments across primitive indices, where higher agreement indicates that individual primitives tend to localize to the same semantic parts. We evaluate this metric over the three source training domains, over the combined source and held-out target domains, and over each pair of source domains. For example, across all four domains, primitive 3 is assigned to the tail in 80% of its valid assignments, while primitive $9$ is assigned to the breast in 74%. This provides a quantitative measure of primitive consistency without requiring explicit part-level supervision during training. We report the agreement across source domains, across source and held-out target domains, and between individual source-domain pairs. | | | | | | | |---|---|---|---|---|---| ||Training-set agreement|Training and Test set agreement|Photo and Cartoon agreement|Photo and Paint agreement|Paint and Cartoon agreement| |Primitive-to-part (%)|64.35|64.17|82.13|70.83|72.49| The results show that primitive-to-part agreement remains nearly unchanged when extending the evaluation from the source domains (64.35%) to include the unseen Art domain (64.17%). For example, across all four domains, primitive 3 corresponds to the tail in 80% of its valid assignments, while primitive 9 corresponds to the breast in 74%. These findings suggest that the learned primitives retain a degree of semantic consistency across domains. References [1] Min, S., Park, N., Kim, S., Park, S., & Kim, J. (2022, October). Grounding visual representations with texts for domain generalization. In European conference on computer vision (pp. 37-53). Cham: Springer Nature Switzerland.
AuthorsOpenReview ↗

Q2: Ablation of the concentration and diversity regularizers.

Thank you for this suggestion. We agree that evaluating the contribution of each regularizer would help clarify its effect on the learned primitives and overall generalization performance. To this end, we conduct an additional ablation on CUB-DG using the ResNet-50 backbone, independently removing the concentration regularizer LregL_{\mathrm{reg}} and the diversity regularizer LDANNL_{\mathrm{DANN}}, as well as removing both. The results are summarized below.

Regularizer \ TargetPhotoCartoonArtPaintAvg
CECE64.756.570.532.356.0
CE+LregCE + L_{reg} (concentration)73.466.271.651.365.6
CE+LDANNCE + L_{DANN} (diversity)78.060.370.256.166.2
All79.672.372.451.969.1

The results show that both regularizers contribute to the generalization performance of CRISP. Compared to the CE-only baseline, adding the concentration regularizer improves average accuracy by 9.69.6 percentage points, while adding the diversity regularizer improves it by 10.210.2 points. Combining both achieves the highest average accuracy of 69.1%69.1\%, outperforming the CE-only baseline by 13.113.1 points.

Interestingly, the two regularizers exhibit complementary effects across target domains. The concentration regularizer provides a larger improvement on Cartoon when applied individually, while the diversity regularizer is particularly effective on Photo and Paint. Their combination further improves performance on Photo, Cartoon, and Art, although the diversity-only objective performs better on Paint. This suggests that the benefits of the two regularizers vary across domain shifts, but their combination provides the strongest overall generalization performance.

We will incorporate these ablation results into the revised manuscript to clarify the individual and combined contributions of the concentration and diversity objectives.

AuthorsOpenReview ↗

Q3. Could the authors characterize which types of domain shifts benefit from CRISP and which are more challenging?

Thank you; we agree that adding an analysis of which domain would be more challenging for CRISP would improve CRISP’s positioning and applicability. Thus, aside from the existing remarks made in the experiments sections and the appendix figures, we have added the following discussion:

Our results suggest that CRISP’s benefit depends on whether class-discriminative primitives and their spatial relations remain recognizable and can be reliably localized across domains. CRISP is beneficial when appearance changes preserve these structural cues. For example, Fig. 5 shows that correctly classified hedgehog sketches retain recognizable face and spiny-body regions, while correctly classified calculator images preserve keypad structure despite changes in rendering and viewpoint. The TerraIncognita examples similarly suggest that useful relations involving animal regions can remain recoverable under changes in illumination and background.

The lower performance relative to ERM++ on PACS Sketch and CUB-DG Paint highlights limitations of this assumption. On PACS Sketch, abstraction can remove or distort the parts needed to distinguish categories. In the elephant example misclassified as a giraffe, the highlighted regions provide a less distinct head–trunk structure than in the successful examples, making the resulting relations less informative. On CUB-DG Paint, fine-grained recognition requires relations that distinguish visually similar bird species; recovering a coarse anatomical arrangement may be insufficient. The CUB-DG failure visualization illustrates an additional difficulty: the highlighted primitives include substantial background and perch information, which can weaken the relevant anatomical evidence. We therefore interpret these deficits as cases where the additional relational representation may provide insufficient or misleading evidence, limiting its advantage over the strong ERM++ baseline.

Across datasets, the qualitative analysis identifies three recurring challenges: missing or distorted discriminative parts, similar primitives and layouts across different categories, and localization dominated by background structure. The calculator–calendar confusion illustrates the second challenge, since both categories contain regular grids, while the VLCS and TerraIncognita failures illustrate the third. CRISP is therefore most useful when the domain shift preserves discriminative relational structure, and the model can recover it reliably. Its gains are less consistent when either condition fails.

Reviewer_uJMQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new domain generalization method, CRISP, that first extracts visual primitives and then leverages these primitives to learn spatial predicates. Then, a learned sparse score is proposed to aggregate these predicates. The authors conduct experiments on various benchmarks in DomainBed, and the results show that the proposed method outperforms prior arts.

Critical strengths

The idea of explicit spatial predicates is interesting. Besides, the authors make it an end-to-end trainable framework. The evaluation covers multiple benchmarks and backbones.

Critical weaknesses
  1. The experiments do not isolate the claimed compositional mechanism. The structural score in Section 3.2 is a sparse, nonnegative linear average of individual predicate activations: it aggregates their evidence additively but contains no term for predicate-predicate interactions, so one strongly satisfied relation can compensate for others rather than enforcing an explicit conjunction. Moreover, the primitive slots receive only image-level supervision, and Figure 3 is qualitative rather than evidence that a slot or its relations remain stable across domains. Table 3 also adds predicate families—and therefore features and class-specific weights—without a capacity-matched alternative. A decisive test would measure slot/relation consistency on matched-content cross-style images and compare the proposed scorer with an equal-parameter MLP or attention head on exactly the same primitive descriptors.

  2. The CUB-DG protocol is too ambiguous to validate the paper's central 7.9-point gain. Appendix A describes ordinary leave-one-domain-out evaluation with 20% of the source-domain data held out, whereas the original benchmark starts from the official CUB train/test split and assigns disjoint groups of original images to the source styles so that stylized siblings do not overlap across source domains.

  3. The ViT block used in this paper is ViT-B/14, but the exact ERM++ results are evaluated under a ViT-B/16 backbone, and CRISP does not specify its ViT pretraining checkpoint. The claimed gains are only 0.4 points with ViT and 1.2 points with ResNet-50, with the latter losing to ERM++ on three of five datasets, yet no seed count or uncertainty is reported. ERM++ should be rerun with the identical backbone, initialization, augmentation, optimization, and model-selection protocol, and paired multi-seed results should be reported.

Questions for authors

Please see the Weakness part.

7 comments
AuthorsOpenReview ↗

Q1.1. Quantitative evaluation of primitive consistency across domains.

Thank you for your detailed and constructive review. Below, we address your concerns:

Q1. Provide experiment details on primitives and relation consistency across experiments, and measure the contribution of grammar.

Q1.1. Quantitative evaluation of primitive and relation consistency across domains.

We agree that quantitative evidence of primitive and relation consistency across domains would strengthen CRISP’s contribution. To this end, we have conducted additional experiments on CUB-DG using the bounding box and part annotations provided by the original dataset. We evaluate three complementary metrics to examine whether the learned primitives and their spatial relations remain consistent under domain shifts. For the following experiments, we use the bird’s bounding boxes and anatomical part locations provided per image from the official CUB dataset (https://www.vision.caltech.edu/datasets/cub_200_2011/) as the ground-truth labels to verify the primitives and relation consistency.

Primitive foreground localization. We first measure the proportion of primitive centroids that fall within the annotated bird bounding boxes (centroid-in-box). Using the trained CRISP checkpoint and Art as the held-out target domain and Photo, Cartoon, and Paint as source domains, we obtain the following results:

Art (Target)PhotoCartoonPaint
Centroid-in-box (%)72.273.875.074.4

The results show that 74.4% of primitive centroids fall within the foreground regions across the source domain, compared to 72.2% on the unseen Art domain. This suggests that CRISP maintains similar foreground localization behavior across domains despite being trained with only image-level supervision.

Primitive-to-part consistency. We further examine whether individual primitive indices consistently correspond to the same semantic bird parts. Specifically, we assign each primitive centroid to its nearest annotated part (e.g., beak, breast, tail) using Euclidean distance in normalized image coordinates, with a matching threshold of 0.30. Primitives without a valid match are excluded from the part-assignment counts. For each primitive index kk, we measure the proportion of valid assignments corresponding to its most frequently associated semantic part. Let nk,pDn_{k,p}^{\mathcal D} denote the number of assignments of primitive kk to part pp over domains D\mathcal D. The agreement is defined as:

\frac{\max_p n_{k,p}^{\mathcal D}} {\sum_p n_{k,p}^{\mathcal D}}.$$ This measures the average consistency of semantic part assignments across primitive indices, where higher agreement indicates that individual primitives tend to localize to the same semantic parts. We evaluate this metric over the three source training domains, over the combined source and held-out target domains, and over each pair of source domains. For example, across all four domains, primitive 3 is assigned to the tail in 80% of its valid assignments, while primitive 9 is assigned to the breast in 74%. This provides a quantitative measure of primitive consistency without requiring explicit part-level supervision during training. We report the agreement across source domains, across source and held-out target domains, and between individual source-domain pairs. | | | | | | | | --------------------- | ---------------------- | ------------------------------- | --------------------------- | ------------------------- | --------------------------- | | | Training-set agreement | Training and Test set agreement | Photo and Cartoon agreement | Photo and Paint agreement | Paint and Cartoon agreement | | Primitive-to-part (%) | 64.35 | 64.17 | 82.13 | 70.83 | 72.49 | The results show that primitive-to-part agreement remains nearly unchanged when extending the evaluation from the source domains (64.35%) to include the unseen Art domain (64.17%). For example, across all four domains, primitive 3 corresponds to the tail in 80% of its valid assignments, while primitive 9 corresponds to the breast in 74%. These findings suggest that the learned primitives retain a degree of semantic consistency across domains, providing a basis for constructing transferable spatial relations. We further provide experiments on relation consistency below.
AuthorsOpenReview ↗

Q1.1. (2) Quantitative evaluation of relation consistency across domains.

Relation consistency. While CRISP's predicate learned class-specific weights are domain-independent by construction and shared across domains, their activation strengths may vary as primitive locations and extents change across domains. Therefore, consistent primitive correspondence alone does not guarantee that the same relations contribute equally to classification. To evaluate this, we decompose each class’s score into its constituent grammar productions' contributions, accounting for both learned weights and predicate activation strengths, and identify the five highest-contributing productions for each class on the source domains and measure the proportion that remain among the top-5 in the held-out target domain. This evaluates whether the most influential structural relations remain consistent under domain shifts.

Art (Target)PhotoCartoonPaint
98.5095.6098.4093.92

The results show that 96.61% of the top-5 contributing productions are preserved on average across held-out target domains, with the highest consistency observed on Art (98.50%) and Cartoon (98.40%). For example, using the identified primitive-to-part correspondences, the relation above(tail, breast) remains among the top-5 contributing productions across all four domains for one bird class. Similarly, relations such as above(crown, background) and above(back, tail) remain consistently influential across domains for another class. These results provide empirical evidence that CRISP preserves not only primitive correspondences but also the most discriminative structural relations under domain shifts.

Together, these experiments provide quantitative evidence that CRISP learns primitives with relatively consistent semantic correspondences and preserves the contributions of their spatial relations across domains. Although the predicate functions are domain-independent by construction, the observed stability of their contributions is an empirical result rather than an architectural guarantee. These findings further support our motivation for using explicit spatial relations as a structural inductive bias for domain generalization.

AuthorsOpenReview ↗

Q1.2. Comparison with a parameter-matched MLP scorer.

Thank you; we agree that comparing the proposed grammar-based scorer with a parameter-matched MLP, while keeping the remaining architecture unchanged, would help isolate the contribution of explicit structural scoring.

To this end, we replace the grammar scorer with a standard two-layer MLP architecture, with the hidden dimension chosen to closely match both conventional choices of parameters and that of the structural (grammar) scorer. Specifically, the MLP consists of a Linear layer (2048,184)(2048, 184), followed by a ReLU layer, a LayerNorm with 184184 parameters, another Linear layer (184, 184), a ReLU layer, and a classification layer (184,200)(184, 200). Altogether, this constitutes 448,424448,424 parameters, closely matching the 449,311449,311 parameters of the original grammar (structural) scorer. Since the hidden dimension must be an integer, exact parameter matching is not always possible without introducing additional architectural modifications. Nevertheless, the difference in parameter count is less than 0.2%0.2\%, which we consider sufficiently small for a capacity-matched comparison. This allows us to evaluate the contribution of the structural scoring mechanism while maintaining comparable model capacity. We retain the original ROI branch and all other architectural components. We evaluate both CRISP and the MLP variant using the ResNet-50 backbone on CUB-DG, retraining both models from scratch under the same experimental protocol. The results are summarized below.

ModelArtCartoonPhotoPaintAvg
CRISP72.472.379.651.969.1
MLP69.868.875.353.366.8

CRISP outperforms the parameter-matched MLP on three of the four target domains, achieving an average accuracy of 69.169.1%, compared to 66.866.8% for the MLP, an improvement of 2.32.3 percentage points. The largest improvement is observed on Photo (+4.3+4.3 points), followed by Cartoon (+3.5+3.5 points) and Art (+2.6+2.6 points). The MLP performs slightly better on Paint (+1.4+1.4 points), indicating that the benefit of structural scoring varies across target domains.

Since both models use the same primitive descriptors, retain the ROI branch, and have nearly identical parameter counts, these results suggest that the improvement cannot be attributed solely to additional model capacity or the learned primitive representations. Instead, they provide further evidence that explicitly modeling structural relations between primitives offers an advantage over a generic learned scoring function for cross-domain generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.