acceptodds
Under review as a conference paper at ICLR 2027

HIVE: Hierarchical Induction of Visual Embeddings for Part–Whole Representation Learning

Abstract

Visual recognition benefits from representations that reveal how local image evidence is organized into higher-level semantic structure, yet existing hierarchy-aware visual backbones often provide limited quantitative access to the induced part-whole relations. To address this, we propose HIVE, hierarchical induction of visual embeddings for part-whole representation learning, whose cross-level assignments are directly readable as a part-whole composition graph. HIVE maps image patches to fine-level embeddings, recursively organizes them into higher-level structures through graph-based reasoning and hierarchical pooling, and reintegrates multi-level representations into the token space for recognition. We further introduce part-whole contribution analysis to quantify directional fine-to-mid and mid-to-coarse composition and visualize image-specific parsing trees. Experiments show that HIVE serves as a competitive general-purpose ViT backbone, while producing multilevel embeddings that are class-relevant and structurally coherent.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Structured Representation Learning, Hierarchical Representation Learning, Visual Hierarchy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ti1oOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HIVE, a hierarchy-inducing visual backbone that recursively organizes image patches into fine-, mid-, and coarse-level feature embeddings. The model combines feature binding, graph-based relational reasoning, learned hierarchical pooling, and multi-level projection, while preserving image-conditioned assignments between adjacent hierarchy levels. These assignments make the learned cross-level composition directly accessible for analysis. The paper evaluates HIVE on ImageNet classification, unsupervised part discovery on PartImageNet, and semantic segmentation on ADE20K, and further analyzes class separability and hierarchical mediation of the learned feature states.

Critical strengths
  1. The empirical gains are substantial and are supported by useful controlled comparisons. I found the PartImageNet results particularly convincing. Under the same MPAE pipeline, replacing the raw ViT-S features with HIVE representations leads to sizeable and consistent improvements in both NMI and ARI across datasets and part granularities, without using part annotations for training. The controlled ADE20K comparison also improves the ViT-S baseline under the same decoder and training protocol. These results suggest that HIVE improves the underlying representation rather than merely producing visually appealing hierarchy diagrams.

  2. The paper goes beyond task accuracy and provides evidence that the learned hierarchy is actually used by the model. I especially liked the analyses of feature-state organization and hierarchical mediation. Mid- and coarse-level states become substantially more class-aligned than fine-level states, and perturbing class-relevant fine-level features has a much larger effect when the upper hierarchy is recomputed than when it is held fixed. To me, these experiments provide meaningful evidence that the hierarchy participates in computation rather than serving only as an interpretability overlay.

Critical weaknesses

My main reservation is the degree of architectural novelty. Many individual ingredients of HIVE — competitive feature binding, graph attention, differentiable pooling, and multi-level representation aggregation — are established ideas. I understand the paper’s novelty as the assignment-preserving recursive organization that keeps image-conditioned child–parent relations accessible across levels. This is useful and well implemented, but the distinction from earlier multi-scale, grouping, and structured visual representation methods is more evolutionary than fundamental. Consequently, I find the empirical evidence important for establishing the contribution.

Questions for authors

My main question is whether the benefit specifically requires the recursive assignment-preserving hierarchy, rather than a strong but non-hierarchical multi-level feature aggregation mechanism. The current module and hierarchy-level ablations are useful, but I would be interested in a capacity-matched baseline that retains comparable numbers of feature states and multi-level projections while removing or breaking the recursive child–parent assignments. Such a comparison would help isolate the contribution of the explicit hierarchical organization from generic multi-scale representation enrichment.

As a clarification rather than a criticism, I also appreciate that the paper explicitly states that its model-internal hierarchy need not coincide exactly with human-defined semantic parts. I think this is an appropriate scope for the current evidence.

3 comments

As the discussion period draws to a close, we sincerely thank you again for your constructive and supportive assessment. We hope that our capacity-matched comparison between recursive HIVE and parallel multilevel grouping, together with the additional MS-COCO experiments, has further strengthened the empirical evidence for HIVE's distinctive contributions.

We would greatly appreciate your feedback on whether any concerns remain. If our responses and additional results have been satisfactory, we would be grateful for your consideration of updating your evaluation accordingly.

Thanks for the thoughtful rebuttal and the additional experiments!

I particularly appreciate the recursive-vs-parallel comparison, which directly addresses my main concern. The new COCO and SPIN results are also helpful and make the hierarchical representation analysis more informative.

I still see the architectural novelty as relatively incremental, so I'll keep my overall rating at 3. However, the additional evidence has improved my understanding of the method and its contributions, and I'm happy to raise my Informativeness score from 2 to 3.

Thanks again for the careful revisions!

We sincerely thank you for revisiting our rebuttal and revised manuscript. We particularly appreciate your recognition that our recursive-versus-parallel comparison directly addresses your main concern, as well as your positive assessment of the additional MS-COCO and SPIN experiments.

We are grateful for your thoughtful feedback, acknowledgment of HIVE's contributions, and decision to raise your Informativeness score.

Reviewer_UHbaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HIVE, a new backbone with hierarchy-aware module to better model the relationships between the instances in the image. HIVE has three key components: A slot-attention like fine-grained binding, a GAT with Diff Pooling and a cross-attention that maps the representation into the token space. Results on ImageNet-1K, ADE20K demonstrates the supoeriority of the proposed method, with additional part-whole contribution analysis and tSNE visualization provided to justify the proposed components.

Critical strengths
  1. The problem studied in this paper is important and interesting.
  2. The graph-based structure enables the proposed structure to be natively explainable.
  3. The authors conducted experiments to justify the superiority of the proposed method.
Critical weaknesses

See Questions

Questions for authors

How is the backbone initialized? Appendix B.2.1 states that models are trained for 50 epochs with a learning rate of 5e-5 and batch size 128, which resembles a fine-tuning recipe rather than training from scratch. If a pretrained checkpoint (e.g., ImageNet-21k) is used, the comparison with from-scratch baselines in Table 1 (e.g., DeiT-S, ViT-S) would not be fair. Could the authors clarify the initialization and report a vanilla ViT baseline trained under the same initialization and protocol?

HIVE-T with UperNet achieves 43.6 mIoU, surpassing DependencyViT-T by 3.3 points and even exceeding HIVE-S with Semantic FPN (41.1). This margin is unusually large at the Tiny scale. Could the authors provide more details on how each component is initialized and trained? In addition, the parameter counts suggest the UperNet head is ~4.5M for HIVE-T but ~17M for HIVE-S; please clarify the head configurations.

The paper claims that HIVE induces part–whole structure, but this property appears to be largely guaranteed by construction: since Φ𝑙+1=Φ𝑙𝑆𝑙, each higher-level footprint is by definition a non-negative combination of lower-level footprints, so child-in-parent containment follows from the architecture rather than emerging from learning. Could the authors disentangle what is learned from what is architecturally imposed, e.g., by measuring alignment with the object/part annotations in PartImageNet, or by applying the same contribution metric to baselines such as DependencyViT or hierarchical clustering on ViT features?

4 comments

As the discussion period draws to a close, we sincerely thank you again for your constructive feedback. We hope that our clarifications on backbone initialization and ADE20K settings, together with the additional matched-baseline comparisons, external semantic validation, and MS-COCO experiments, have adequately addressed your concerns.

We would greatly appreciate your feedback on whether any concerns remain. If our responses have been satisfactory, we would be grateful for your consideration of updating your evaluation accordingly.

As today marks the final day of the discussion period, we sincerely thank you again for your constructive feedback. We have made every effort to address your concerns through detailed clarifications, additional experiments, and revisions to our manuscript.

We would be grateful to know whether any concerns remain and would be happy to respond promptly before the deadline. If our responses have adequately addressed your concerns, we would greatly appreciate your consideration of raising your score accordingly.

Thank you again for your time and consideration.

Thanks for the response which have addressed most of my concerns. I raised my score accordingly.

We sincerely thank you for revisiting our rebuttal and revised manuscript, and for raising your score. We appreciate your thoughtful questions, which helped us strengthen the paper, particularly the matched-baseline comparisons and the analysis of learned versus imposed structure.

Reviewer_Jd47OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HIVE, a visual representation framework that organizes image features into a hierarchy of fine-, mid-, and coarse-level embeddings. It first extracts feature states using competitive attention and recurrent refinement, then applies graph attention and differentiable pooling to construct higher levels. The learned assignments are reused to propagate patch-to-feature composition maps across the hierarchy and project multilevel information back into the token representation. Training combines a task objective with structural regularizers. Experiments report competitive ImageNet classification results and improved part discovery when HIVE is added to MPAE. The paper also introduces a spatial contribution analysis to visualize cross-level relations and image-specific parsing structures.

Critical strengths
  • The paper is generally well written and easy to follow. The motivation is clear: learning representations that expose how local features compose into higher-level structures is an interesting and relevant direction.

  • The assignment-preserving formulation provides a useful connection between the model’s computation and its visualization. Reusing the cross-level assignments for both feature aggregation and patch-level projection makes the learned organization directly accessible.

  • The ImageNet results demonstrate competitive recognition performance at both Tiny and Small scales. Reporting the mean and standard error across three seeds is a positive aspect of the evaluation.

  • The part-discovery results are a strong empirical point. Adding HIVE improves both NMI and ARI across both PartImageNet variants and both part counts. For example, at 25 discovered parts on PartImageNet-O, ARI increases from 57.3 to 70.8. The comparison with a fixed frozen backbone and otherwise unchanged MPAE pipeline helps establish the usefulness of the added feature organization.

Critical weaknesses
  1. My main concern is whether the evidence establishes semantic part–whole relations, which are central to the paper’s motivation and contribution. Table 2 evaluates agreement with flat part annotations, but does not assess whether the learned parent–child assignments connect the correct parts to the correct wholes. The contribution analysis uses spatial co-activation and containment between maps whose higher levels are already constructed through (Φ(l+1)=Φ(l)S(l)\Phi^{(l+1)}=\Phi^{(l)}S^{(l)}). These measurements are useful descriptions of the induced grouping, but are not independent validation of its semantic correctness. Furthermore, the main hierarchy examples use a separately trained 8/4/2-state model on 20 classes, rather than the default 128/64/32 model. I would appreciate quantitative evaluation of cross-level relations on the main configuration, for example whether annotated parts are assigned to the appropriate object-level parents.

  2. The classification comparison does not sufficiently isolate the benefit of HIVE under a matched training setup. The baseline results are taken from their respective papers, while the main text does not establish equivalence in initialization, training recipe, or optimization budget. HIVE-S reaches 82.8%, compared with 82.6% for Hi-Mapper, while using somewhat more parameters and computation. These results support competitiveness, but the small margin is difficult to attribute specifically to the proposed hierarchy without a baseline trained using the same recipe. A matched ViT/DeiT comparison would also help interpret the larger improvements over the standard backbone numbers.

  3. The discussion of supervision in the part-discovery comparison needs correction. Section 4.1.2 describes competing methods as using “part supervision” and contrasts this with HIVE’s lack of part annotations. However, PDiscoNet [1] and PDiscoFormer [2] also discover parts without part annotations; they use image-level class labels. Since HIVE itself is trained for supervised ImageNet classification, the distinction between class supervision, part supervision, and backbone pretraining should be made explicit. This does not remove the numerical gains, but it changes the interpretation of the claimed supervision advantage. The paper should clearly state which labels are used at each training stage for HIVE and the comparison methods.

  4. The specific methodological novelty and the necessity of the hierarchy could be established more clearly. The framework combines competitive feature binding, graph attention, differentiable pooling, and projection back to patches, all closely connected to the cited prior work. The distinctive contribution appears to be their integration through recursively preserved assignments. A comparison with a flat feature module or independently learned multilevel groups at a similar parameter budget would help establish what this recursion adds beyond extra representation capacity.

Questions for authors

I would appreciate it if the authors could elaborate further on the weaknesses above.

  1. I also have a clarification question about how the propagated graph structure affects attention. Equation (5) produces strictly positive soft assignments for finite logits. Consequently, (J(l+1)=S(l)⊤A(l)S(l)J^{(l+1)}=S^{(l)\top}A^{(l)}S^{(l)}) appears to have positive entries throughout. Since the neighborhood is defined only by (Jij>0J_{ij}>0), and Equations (2)–(4) do not otherwise use the edge magnitudes, does graph attention become fully connected after the first pooling step? If so, what role do the propagated edge weights play beyond the link-prediction regularizer? Please clarify whether additional sparsification or edge-weighted attention is intended.
  2. I would like to draw the authors’ attention to Hierarchical Slot Attention [3], which seems relevant to this work. It jointly learns decompositions at multiple granularities and evaluates grouping purity and containment. Although its supervision setting differs from HIVE’s, could the authors consider whether a fair benchmark comparison is possible or, at least, discuss the connection to better position their contribution?

[1] Van Der Klis, Robert, et al. "Pdisconet: Semantically consistent part discovery for fine-grained recognition." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[2] Aniraj, Ananthu, et al. "Pdiscoformer: Relaxing part discovery constraints with vision transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[3] Madan, Neelu, et al. "HSA: Hierarchical Slot Attention for Multi-granularity Scene-Decomposition." arXiv preprint arXiv:2607.08249 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.