- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Triple-Seg, a dataset of human judgments about which image regions belong together. It trains a region-embedding model on these judgments and uses hierarchical clustering to produce scene trees. The work contributes a behavioral dataset and a way to predict perceptual grouping from visual features. The main evaluation measures grouping prediction on held-out images. The paper also compares model architectures, studies feature choices and annotation volume, and visualizes the trees and embedding space.
Critical strengths
The dataset asks people which regions of natural scenes belong together, providing grouping judgments that category labels alone do not specify. Multiple annotations for a subset of triplets also make it useful for studying agreement between people.
The MLP predictor improves substantially over the raw-feature triplet baseline while using a relatively simple set of visual descriptors. Together with the feature ablations, this gives an interpretable starting point for understanding which cues help predict human grouping judgments.
Critical weaknesses
Figure 2 evaluates triplet prediction, whereas Figures 3–4 show trees produced by clustering, so the reported accuracy doesn't directly establish that the final trees recover human hierarchies. Before making that claim, I'd use the trees' merge heights to predict held-out triplets and compare with raw-feature clustering under the same masks, which would at least test whether the tree preserves the behavioral relationships learned by the embedding.
The claims about semantic independence and unseen categories are also stronger than the evidence. For the claims in the abstract to follow, I'd need evidence that goes beyond holding out images. The test categories may still be familiar, and the human judgments can still draw on semantic knowledge. A category-disjoint split would address transfer, while triplets where appearance and category identity favor different pairings would probe the semantic claim. Otherwise I'd narrow the wording to predicting human grouping on new images.
Questions for authors
-
Could you evaluate the final trees against held-out triplets and a small set of independently judged group merges, using raw-feature clustering under the same masks as a baseline?
-
For the semantic claim, can you test triplets where appearance and category membership suggest different groupings?
2 comments
I appreciate the authors’ effort in designing these thoughtful follow-up experiments. At this stage, I will maintain my rating, but I believe the paper will be more thorough once these experiments are included.
Critical Weaknesses:
Response 1: We thank the reviewer for this insightful critique. Our quantitative evaluation in Figure 2 measures triplet accuracy in the learned embedding space rather than directly validating the hierarchical trees shown in Figures 3–4. We agree that predicting held-out triplets from the dendrograms' merge heights—and comparing those results against trees constructed from raw features and other baselines under the same SAM3 masks is an ideal way to quantify how much behavioral information survives the clustering process. To test this thoroughly in our follow-up research, we plan to pair this exact merge-height evaluation with a dense data collection study in which we capture 100% of all possible triplet judgments for a subset of images. Having complete behavioral annotations will allow us to exhaustively benchmark the induced trees against all ground-truth relationships, systematically comparing merge-height predictions across different clustering algorithms and baseline feature representations.While executing this systematic evaluation across clustering methods and baselines is unfeasible within the timeframe of this rebuttal, we have updated Section 5 (Future Work) to explicitly incorporate this experimental protocol.
Response 2: We agree that an image-level split does not guarantee a category-disjoint evaluation, and that human observers viewing recognizable natural scenes can still draw upon top-down semantic knowledge even when explicit category labels are withheld.Following your recommendation, we have narrowed our claims throughout the manuscript. Specifically, we now frame our contribution as predicting human perceptual grouping on unseen scenes rather than claiming generalization to unseen categories, and we clarify that our model is trained without explicit semantic category labels rather than asserting complete independence from semantic priors. To directly probe the boundary between low-level visual appearance and semantic identity in future work, we plan to conduct a dedicated follow-up study extending our evaluation to abstract artworks and non-figurative compositions. By using stimuli in which recognizable object identities and explicit semantic references are entirely absent, we can eliminate top-down semantic confounds and cleanly isolate how purely low- and mid-level visual cues drive human grouping judgments. Alongside constructing category-disjoint splits on natural scenes, this abstract-art study will allow us to separate appearance-driven grouping from category-driven grouping without the two competing in the same natural image. While collecting a new behavioral dataset on abstract stimuli is unfeasible within the timeframe of this rebuttal, we have outlined this exact experimental direction in Section 5 (Future Work).
Question 1: Thank you for this question, see response to Comment 1.
Question 2: Thank you for this question, see response to Comment 2.