acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Hierarchical Scene Structure from Human Perceptual Grouping

Abstract

Understanding the hierarchical organization of natural scenes remains a persistent challenge in computer vision. In this work, we present a data-driven framework for recovering multi-scale scene hierarchies directly from human visual grouping judgments without requiring semantic labels, demonstrating that universal grouping principles can be learned independently of semantic category knowledge. We introduce Triple-Seg, a benchmark dataset of 210 natural images paired with segmentations and dense human triplet comparison judgments of the image regions. Using a neural embedding model trained on these judgments, segment-level visual features are mapped into a embedding space that enables the reconstruction of structural scene dendrograms via agglomerative clustering. By relying on perceptual rather than semantic cues, the model generalizes to unseen categories from only a few hundred training images. Code and datasets will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Perceptual Grouping, Hierarchical Organization, Hierarchical Scene Understanding, Scene Hierarchy, Human-AI Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qRDiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Triple-Seg, a dataset of human judgments about which image regions belong together. It trains a region-embedding model on these judgments and uses hierarchical clustering to produce scene trees. The work contributes a behavioral dataset and a way to predict perceptual grouping from visual features. The main evaluation measures grouping prediction on held-out images. The paper also compares model architectures, studies feature choices and annotation volume, and visualizes the trees and embedding space.

Critical strengths

The dataset asks people which regions of natural scenes belong together, providing grouping judgments that category labels alone do not specify. Multiple annotations for a subset of triplets also make it useful for studying agreement between people.

The MLP predictor improves substantially over the raw-feature triplet baseline while using a relatively simple set of visual descriptors. Together with the feature ablations, this gives an interpretable starting point for understanding which cues help predict human grouping judgments.

Critical weaknesses

Figure 2 evaluates triplet prediction, whereas Figures 3–4 show trees produced by clustering, so the reported accuracy doesn't directly establish that the final trees recover human hierarchies. Before making that claim, I'd use the trees' merge heights to predict held-out triplets and compare with raw-feature clustering under the same masks, which would at least test whether the tree preserves the behavioral relationships learned by the embedding.

The claims about semantic independence and unseen categories are also stronger than the evidence. For the claims in the abstract to follow, I'd need evidence that goes beyond holding out images. The test categories may still be familiar, and the human judgments can still draw on semantic knowledge. A category-disjoint split would address transfer, while triplets where appearance and category identity favor different pairings would probe the semantic claim. Otherwise I'd narrow the wording to predicting human grouping on new images.

Questions for authors
  1. Could you evaluate the final trees against held-out triplets and a small set of independently judged group merges, using raw-feature clustering under the same masks as a baseline?

  2. For the semantic claim, can you test triplets where appearance and category membership suggest different groupings?

2 comments

Critical Weaknesses:

Response 1: We thank the reviewer for this insightful critique. Our quantitative evaluation in Figure 2 measures triplet accuracy in the learned embedding space rather than directly validating the hierarchical trees shown in Figures 3–4. We agree that predicting held-out triplets from the dendrograms' merge heights—and comparing those results against trees constructed from raw features and other baselines under the same SAM3 masks is an ideal way to quantify how much behavioral information survives the clustering process. To test this thoroughly in our follow-up research, we plan to pair this exact merge-height evaluation with a dense data collection study in which we capture 100% of all possible triplet judgments for a subset of images. Having complete behavioral annotations will allow us to exhaustively benchmark the induced trees against all ground-truth relationships, systematically comparing merge-height predictions across different clustering algorithms and baseline feature representations.While executing this systematic evaluation across clustering methods and baselines is unfeasible within the timeframe of this rebuttal, we have updated Section 5 (Future Work) to explicitly incorporate this experimental protocol.

Response 2: We agree that an image-level split does not guarantee a category-disjoint evaluation, and that human observers viewing recognizable natural scenes can still draw upon top-down semantic knowledge even when explicit category labels are withheld.Following your recommendation, we have narrowed our claims throughout the manuscript. Specifically, we now frame our contribution as predicting human perceptual grouping on unseen scenes rather than claiming generalization to unseen categories, and we clarify that our model is trained without explicit semantic category labels rather than asserting complete independence from semantic priors. To directly probe the boundary between low-level visual appearance and semantic identity in future work, we plan to conduct a dedicated follow-up study extending our evaluation to abstract artworks and non-figurative compositions. By using stimuli in which recognizable object identities and explicit semantic references are entirely absent, we can eliminate top-down semantic confounds and cleanly isolate how purely low- and mid-level visual cues drive human grouping judgments. Alongside constructing category-disjoint splits on natural scenes, this abstract-art study will allow us to separate appearance-driven grouping from category-driven grouping without the two competing in the same natural image. While collecting a new behavioral dataset on abstract stimuli is unfeasible within the timeframe of this rebuttal, we have outlined this exact experimental direction in Section 5 (Future Work).

Question 1: Thank you for this question, see response to Comment 1.

Question 2: Thank you for this question, see response to Comment 2.

I appreciate the authors’ effort in designing these thoughtful follow-up experiments. At this stage, I will maintain my rating, but I believe the paper will be more thorough once these experiments are included.

Reviewer_NNh5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an experiment on perceptual grouping. Participants choose sequences of tree segments and group them into similar regions. To perform the experiment, a small dataset, dubbed Triple-Seg, formed by 210 images, is created by segmenting them using SAM3, and 6 features (shape, texture,...) are extracted to form a 32-dimensional vector, which is the input to the model. Out of 236 participants, human judgments for 33,257 unique triplets (representing 33.61 % of all possible triplets), achieving an average coverage of 38.49% per image, were collected. On perceptual grouping, the MLP model architecture achieved the highest accuracy at 0.6610, approaching the human agreement upper bound of 0.7034.

Critical strengths

The paper addresses a topic relevant to both computer vision and the psychology of human vision.

The paper presents a potentially useful behavioural-supervision pipeline. The paper’s credible contribution is a behaviourally supervised baseline for ranking grouping affinity among fixed image regions, followed by a standard procedure for turning those affinities into a hierarchy. That is worthwhile, especially because it tests feature contributions and reports a human-agreement ceiling. The paper is well written, and the images are eye-catching.

Critical weaknesses
  • Literature: The literature review should be substantially broadened. The paper positions its contribution mainly against local Gestalt-cue models and similarity learning for isolated stimuli, but does not sufficiently engage with work on hierarchical image segmentation, object/part decomposition, perceptual organization at multiple granularities, and scene parsing. This omission matters because the proposed method starts with a fixed SAM3 segmentation and learns grouping only within those supplied regions. Relevant literature is needed not merely for completeness, but to clarify what is new relative to existing work and to discuss the unresolved question of what constitutes a perceptual part and at which scale a decomposition should stop. The authors should also discuss established object, stuff, and part segmentation resources—such as COCO-Stuff, LVIS, PartImageNet, and PASCAL-Part—and explain why a new 210-image region inventory was required. Even without semantic labels, these datasets could provide candidate masks for the behavioural triplet task. A comparison across mask sources and segmentation granularities would be particularly informative, since the proposed hierarchy necessarily depends on the regions supplied at its leaves. Triple-Seg is not a substitute for established part/scene datasets. A dataset of 210 images can be valuable for collecting a specific kind of behavioral annotation, but it is too small to support claims of general scene-level learning.

  • The experiment: The paper fixes the very ontology it claims to study. This is the most consequential issue. The hierarchy begins from a predetermined image segmentation, so participants do not judge whether an eye, iris, pupil, face, or whole head should be a unit. They choose the strongest grouping among already-defined regions. The experiment thus measures relations among author-specified primitives, not human selection of perceptual parts or the depth of part decomposition. Any claim to recover a human hierarchy must be conditioned on that segmentation: it recovers a hierarchy over the supplied segments. The validation cannot establish full hierarchical fidelity. Five-way triplet choices provide sparse, local comparative preferences. Transforming those judgments into a global dendrogram assumes that pairwise embedding similarity and average-linkage clustering faithfully represent hierarchical perceptual organization. That is plausible as a modelling choice, but it is not demonstrated merely by triplet prediction accuracy. The reported model reaches 0.6610 accuracy versus human agreement of 0.7034, leaving a meaningful gap; moreover, agreement and model accuracy alone do not validate the topology of the output trees.

  • Grouping: Similarity is not automatically grouping. The strongest learned predictors are color and texture, each at roughly 0.60 accuracy, while position and shape are near 0.50. This suggests the model often clusters visually similar regions rather than capture relations such as common region, connectedness, object membership, functional association, or occlusion structure. Their qualitative examples reinforce this ambiguity: yolks cluster with yolks, and similarly colored vessels cluster together, which is sensible similarity-based organization but not necessarily a perceptual part-whole hierarchy. The clustering algorithm imposes the hierarchy. Agglomerative clustering always yields a tree. The relevant scientific question is whether that tree, by average linkage specifically and with the selected linkage distances, reflects human perceptual organization. The paper shows appealing examples but lacks direct evaluation of tree structure against independently collected hierarchical human judgments or comparisons with other clustering/linkage methods. Feature engineering weakens the “purely data-driven” framing. The pipeline uses predefined descriptors for size, position, skeleton, shape, texture, and color. Human labels supervise how the model weights or embeds those descriptors, but the representational vocabulary is chosen in advance. Generalization evidence is limited. The paper shows held-out prediction and cross-image mixing in t-SNE/UMAP, but these do not strongly show that the model learns general perceptual laws. The authors acknowledge that performance has not saturated with more triplets, further indicating that the dataset is still small relative to the breadth of their claims.

Questions for authors

Have you considered why you collected human judgments for 33,257 unique triplets (representing 33.61 % of all possible triplets), achieving an average coverage of 38.49% per image. Triplet given as prior might have been confounding.

Why you did not consider existing datasets?

Did you provide a denotation for the triplet? E.g. iris, pupil eye, or boat, rock, lake..

2 comments

Critical Weaknesses:

Response 1: Thank you for this comment. We agree that our hierarchical clustering depends entirely on initial segmentation granularity and that the manuscript needed stronger engagement with part-whole decomposition and scene-parsing literature. To address this, we have broadened Section 2 (Related Work) to discuss prior hierarchical segmentation research and contextualize our work alongside COCO-Stuff, LVIS, PartImageNet, and PASCAL-Part.

We curated a new 210-image set for Triple-Seg rather than using existing mask sources to strictly control variance in visual complexity and scene content, and to establish stimulus overlap with ongoing lab studies for future cross-domain comparisons. We also agree that 210 images cannot support claims of "general scene-level learning" and have revised the manuscript to emphasize that our primary contribution is the dense behavioral annotations (33,257 triplet judgments). Finally, comparing recovered hierarchies across different mask sources (e.g., SAM3 vs. PASCAL-Part) is an excellent direction that we will incorporate into our future research.

Response 2: We thank the reviewer for this insightful critique and fully agree that relying on predetermined segmentation primitives prevents our framework from being an unconstrained model of human perception. Because no current segmentation model yields 1:1 human perceptual parts out of the box, conditioning on SAM3 segments was necessary to make the grouping problem tractable and isolate higher-order grouping relations.

We also agree that local triplet accuracy does not validate the global dendrogram topology. To address this, we updated Section 5 (Future Work) to outline a follow-up study that will collect 100% of all possible triplet judgments on an image subset, enabling us to evaluate global tree structures by comparing dendrogram merge heights against all ground-truth behavioral relationships.

Response 3: We thank the reviewer for this insight and agree with your assessments. While we acknowledge that color and texture are the strongest individual predictors (~0.60 accuracy), structural features like position and shape still achieve ~0.50 accuracy—well above the 0.33 random chance baseline for our 3-way triplet task. This indicates that the model captures meaningful compositional and part-whole relationships, as seen qualitatively when the avocado seed groups with its flesh based on spatial and object-membership cues rather than surface matching.

Regarding clustering, we agree that agglomerative clustering enforces a hierarchy that our current triplet accuracy does not directly validate. To address this, we have added a follow-up study to Section 5 (Future Work) that will collect 100% of all possible triplet judgments on an image subset to quantitatively evaluate global tree topologies via dendrogram merge heights and compare alternative linkage methods. Finally, we concede that our predefined descriptors represent a hand-crafted vocabulary and that our current results do not establish "general perceptual laws." Accordingly, we have tempered our "purely data-driven" framing and revised our generalization claims to state simply that the network predicts grouping on unseen scenes, removing broader claims of universal structural representations.

Question 1: Thank you for this question. While standard behavioral embedding studies using SPoSE (e.g., Hebart et al., 2020) recover representational spaces from less than 1% of all triplets, our within-image segment comparisons required higher density. Because pilot data showed that ~20% coverage per image yielded a reliable grouping matrix, we targeted ~30% as a safety margin. Finally, the discrepancy between the dataset-wide coverage (33.61%) and the per-image average (38.49%) arises because images with fewer segments naturally reached higher completion rates.

Question 2: Thank you for this question. Unlike existing datasets (e.g., COCO-Stuff, PASCAL-Part) that lack behavioral grouping judgments, Triple-Seg is designed as a targeted behavioral resource rather than a substitute. We curated a new 210-image set rather than sourcing existing masks to strictly control scene content and visual complexity, and to ensure stimulus overlap with ongoing lab studies for future cross-domain comparisons. We have added this rationale to Sections 2 and 3.6, and plan to evaluate alternative mask sources in future work.

Question 3: Thank you for this question. No, Triple-Seg does not include semantic denotations (text or category labels) for the individual segments; each triplet consists strictly of three unlabeled binary masks generated by SAM3 paired with the human grouping judgment, and participants were shown only the highlighted visual regions without any text labels.

Reviewer_NNh5OpenReview ↗

Understand the answer motivations but do not change my rating

I understand the difficulties the authors faced, as explained and justified in their answers, yet the experiment needs better integration with the appropriate vision model.

Reviewer_FPVeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Triple-Seg, containing 210 natural images and 33,257 annotated segment triplets. Images are segmented using SAM3, and each region is represented by handcrafted descriptors covering size, position, color, shape, texture, and skeleton geometry. A neural network maps these descriptors into normalized embeddings, trained to predict which pair within a triplet humans group most strongly. Average-linkage clustering then converts the learned similarities into scene dendrograms. The best model achieves 66.10% triplet-choice accuracy, compared with 40.36% for raw feature distances and a reported human agreement of 70.34%.

Critical strengths
  1. An interesting supervision signal. Within-image grouping judgments provide a useful perspective on visual organization beyond conventional category annotations. The dataset could support research connecting perceptual organization, metric learning, and hierarchical representations.
  2. A clear improvement over the raw-feature baseline. The increase from 40.36% to 66.10% suggests that learning how to combine visual descriptors captures meaningful regularities in human choices.
  3. A simple and inspectable model. The compact feature representation makes controlled ablations possible. The finding that color and texture are particularly predictive is informative, and evaluating all 63 nonempty feature subsets is a useful exploratory analysis.
Critical weaknesses
  1. The central claim of hierarchical recovery is not quantitatively evaluated. The reported accuracy measures whether embedding similarities predict individual triplet choices. The final dendrograms are evaluated only through qualitative examples. Agglomerative clustering produces a hierarchy for any suitable dissimilarity matrix; producing a tree does not establish that its organization matches human perception. A direct evaluation is possible with the existing annotations: for each held-out triplet, use the tree’s merge heights to predict the closest pair, then measure agreement with human judgments. Comparing accuracy before and after clustering would reveal how much behavioral information survives the hierarchical reconstruction. The same evaluation should include trees constructed from raw features and other baseline representations. The paper should also distinguish perceptual similarity hierarchies from part–whole scene hierarchies. For example, grouping water and sky may reflect visual similarity without demonstrating compositional scene structure.

  2. Independence from semantics and generalization to unseen categories are not established. Participants view recognizable natural scenes and receive unrestricted grouping instructions. Avoiding explicit category labels does not prevent their judgments from reflecting object identity, function, or scene knowledge. Furthermore, the input regions come from SAM3, a pretrained segmentation model developed using extensive concept-labeled data. Its published description explicitly reports training data containing millions of concept labels. Consequently, the complete pipeline inherits substantial learned segmentation priors. The evidence supports training the grouping module without additional category labels. Demonstrating semantic independence would require controlled stimuli or interventions that separate semantic information from low-level visual cues. Likewise, an image-level split establishes generalization to unseen images, while the claim about unseen categories requires a category-disjoint evaluation.

  3. The sampling and model-selection protocols need clarification. Sections 3.3 and 4.1 describe an 80/10/10 image split. However, Section 4.3 states that the scaling experiments use triplets “across all 210 images,” with sample sizes extending to 30,000 and the full dataset. It is unclear whether these numbers refer to training-only annotations or to the complete dataset before splitting. Please report the exact image and triplet counts in each split and confirm that all scaling experiments use a fixed, untouched test set. The statement that MLP was selected based on the displayed test accuracies also leaves unclear whether architecture selection used validation results. These are protocol ambiguities, rather than demonstrated leakage, but they directly affect the interpretation of generalization and scaling.

  4. The methodological contribution and baseline coverage remain limited. The pipeline combines established descriptors, a conventional embedding network, a triplet-choice objective, and standard hierarchical clustering. Its strongest potential contribution is the dataset and the application of behavioral supervision to within-scene grouping. The relationship to SPoSE needs greater precision. Original SPoSE explicitly uses sparse, nonnegative representations, whereas the described model specifies normalized embeddings without corresponding sparsity or nonnegativity constraints. The associated interpretability properties therefore require separate justification. The related-work discussion should also acknowledge earlier learning of grouping affinities from human segmentations, rather than broadly characterizing earlier grouping approaches as relying on manually specified criteria. Fowlkes, Martin, and Malik, for example, learned affinities from human segmentation data using brightness, color, and texture cues. Useful additional comparisons would include pretrained region features with the same prediction head and quantitative comparisons of the final hierarchies under identical input masks.

Technical and presentation issues. Several details warrant correction or clarification:

  • Grouping probabilities: Neither clipped cosine similarity nor M/max⁡(M)M/\max(M) is automatically a calibrated grouping probability. These quantities should be described as affinities unless a probabilistic interpretation is established. The conversion from affinity to the dissimilarity supplied to average linkage also needs specification.
  • Feature normalization: “Each sub-vector was normalized” is ambiguous. If this means per-segment L2 normalization, the positive scalar size feature becomes constant. Explain how size information is preserved.
  • Scaling claims: A linear fit over seven dataset sizes with R2=0.969R^2=0.969 supports a trend over the observed range; it does not establish a general scaling law or demonstrate that performance will remain unsaturated.
  • Embedding interpretation: Mixing image identities in a t-SNE plot does not establish that the model has learned general structural representations.
  • Missing supporting material: Appendices A.2 and A.5 and the complete feature-ablation table are referenced but unavailable in the supplied PDF.
Questions for authors
  1. What is the held-out triplet accuracy when predictions are derived from the final dendrograms?
  2. Were validation and test images excluded from every training subset in the scaling experiment?
  3. How was the human agreement ceiling estimated, and what uncertainty accompanies it?
  4. Do the conclusions persist with category-disjoint evaluation and segmentation that does not depend on a pretrained foundation model?
1 comment

Critical Weaknesses:

Response 1: We thank the reviewer for this insightful critique. We think your proposed evaluation using dendrogram merge heights on held-out triplets to measure how much behavioral information survives clustering relative to raw-feature and baseline trees—is an excellent methodology for quantitatively validating the induced hierarchies. Pairing this merge-height analysis with alternative clustering algorithms and our planned follow-up study (collecting 100% of triplet annotations on a subset of images so all structural relationships can be exhaustively tested against the tree) will provide a comprehensive benchmark of hierarchical fidelity. While running this systematic evaluation across multiple clustering methods and baselines is unfeasible within the timeframe of this rebuttal, we have updated Section 5 (Future Work) to explicitly incorporate this exact evaluation protocol.

Response 2: We appreciate this critique. You raise an good point regarding the inherent semantic priors present in both human top-down processing and the SAM3 segmentation boundaries. We agree that despite our efforts to exclude explicit semantic labels from the training pipeline and experimental instructions, it is not possible in the current configuration to prevent participants from utilizing category knowledge when grouping recognizable elements in natural scenes.

To accurately reflect this, we have revised our claims throughout the manuscript. We now specify that our method operates 'without explicit semantic category labels during training' rather than claiming pure semantic independence. We have also corrected our generalization claims, replacing 'unseen categories' with 'unseen scenes,' as we agree our random image-level split does not guarantee a category-disjoint evaluation.

Regarding your suggestion to use controlled stimuli or interventions to separate semantic information from low-level visual cues, we completely agree that this is the necessary next step. Our plan to address this is to extend the dataset using abstract artworks and non-figurative compositions. By evaluating stimuli where explicit semantic references and recognizable object identities are entirely absent, we can cleanly isolate and observe the interplay of purely low- and mid-level visual features on grouping behavior. While conducting a new, large-scale behavioral data collection campaign on these controlled stimuli is unfeasible within the short timeframe of this rebuttal, we recognize its importance.

Response 3: Thank you for carefully examining our experimental protocol and catching this issue. You are right to flag the scaling setup in Section 4.3: to simulate smaller dataset regimes, our original experiment proportionally sub sampled the validation and test splits alongside the training split, rather than evaluating against a single, fixed test set across all runs.

We fully agree that keeping the validation and test sets fixed and untouched while sub sampling strictly from the training split is the proper protocol for a scaling analysis. We are currently re-running these scaling experiments under this corrected setup and hope to complete them and share the updated results before the rebuttal period ends.

Response 4: We sincerely thank the reviewer for this constructive assessment. We agree that the primary contribution of this work lies in the Triple-Seg dataset and the application of behavioral human triplet supervision to within-scene perceptual grouping, rather than in proposing a novel network architecture.

We also appreciate you pointing out the oversight regarding Fowlkes, Martin, and Malik (2003). We agree that our Related Work section should properly acknowledge early data-driven approaches that learned grouping affinities from human segmentations, rather than characterizing all early work as relying on manually specified criteria. We have added this citation and discussion to Section 2.

Finally, we thank you for the excellent experimental suggestion specifically, evaluating pretrained region features with the same prediction head and quantitatively comparing the resulting hierarchies under identical input masks. While running these additional baselines and tree-level evaluations is unfeasible within the timeframe of this rebuttal, they provide a clear roadmap for strengthening the work, and we have explicitly incorporated them into Section 5 (Future Work).

Questions For Authors:

1: Response 1 2: Response 3 3: Thank you for requesting this methodological clarification. We estimated the human agreement ceiling across 951 multi-evaluated triplet instances by averaging each triplet’s majority-choice agreement rate (the proportion of participants selecting the most frequent option). Non-parametric bootstrapping (10,000 resamples with replacement) yielded a mean agreement ceiling of 0.7034 (95% CI: [0.6874, 0.7194]). We have updated Section 4.1 of the manuscript to include this confidence interval. 4: Response 4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.