acceptodds
Under review as a conference paper at ICLR 2027

SceneParser: Hierarchical Scene Parsing for Visual Semantics Understanding

Abstract

General scene perception has progressed from object recognition toward open-vocabulary grounding, part localization, and affordance prediction. Yet these capabilities are often realized as isolated predictions that localize objects, parts, or interaction points without capturing the structured dependencies needed for interaction-oriented scene understanding. To address this gap, we introduce an interaction-oriented formulation of hierarchical scene parsing that represents scenes as explicit sceneobjectpartaffordance hierarchies with cross-level bindings. We instantiate this task with SceneParser, a VLM-based parser trained for unified hierarchical generation with structural-completion pseudo labels and curriculum learning. To support training and evaluation, we construct SceneParser-Bench, a large-scale benchmark built with a scalable hierarchical data engine, containing 110K training images, a 5K validation split, 777K objects, 1.14M parts, 1.74M affordance annotations, and 1.74M valid object-part-affordance chain instances. We further introduce Level-1 to Level-3 conditional metrics and ParseRate to evaluate localization, cross-level binding, and hierarchical completeness. Experiments show that existing MLLMs and hierarchy assembly pipelines struggle with hierarchical parsing on our SceneParser-Bench, while SceneParser achieves stronger structure-aware performance. Besides, ablations, evaluations on COCO and AGD20K, and a downstream planning probe demonstrate that our SceneParser is compatible with conventional tasks and provides an actionable representation for visual understanding.

Primary area:
datasets and benchmarks
Keywords:
Scene Understanding, Visual Grounding, Part Segmentation, Affordance Grounding, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NTekOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new task that generates the structured dependencies for scene parsing. To support evaluation, they build SceneParser-Bench, which contains 110k train/5k val images with affordance annotations generated by an automated pipeline and introduces Level-1 to Level-3 condition metrics with ParseRate. They also train SceneParser using pseudo-labels and curriculum learning to solve this task. Experiments show better performance on SceneParser-Bench than general MLLMs and demonstrate that SceneParser provides good representation for downstream tasks.

Critical strengths
  1. The built datasets and introduced metrics provide a good benchmark to evaluate the structure-aware performance of models. The hierarchical data engine that generated affordance annotations with 90.4% full-chain accuracy provides an effective solution for reducing manual annotation costs.

  2. The proposed hierarchical scene parsing organizes scenes into explicit structures, and it provides the interaction-relevant scene information for downstream manipulation, filling the gap between isolated predictions and compositional scene understanding.

Critical weaknesses
  1. The introduced metrics are flawed. Appendix A shows that L2 is only computed over matched parent objects, and L3 is computed over matched object-part pairs. Under this rule, although a model fails to detect almost all objects in a scene, it can still get a high L2 score by parsing parts on few matched objects. And the ParseRate only checks whether the structure is complete, rather than whether the child predictions are actually correct.
  2. Although Appendix B audits 1000 validation images with 90.4% full-chain accuracy, humans did not correct wrong samples. And Figure 6 shows spelling errors such as “obj:glaves”, “part:handel” which may happen in the datasets. With these wrong labels, models are penalized although outputting the right answers. The datasets are generated by the automated pipeline with Rex-Omni, and SceneParser is fine-tuned from Rex-Omni, which can naturally get better performance under the strict name-matching rule.
Questions for authors
  1. Why are the scene-level evaluation numbers omitted (-) for Qwen3.5-4B (FT), Rex-Omni Assembly, and Rex-Omni (FT) in Table 1? Could you provide these results?

  2. In L2/L3, how are the GT and predicted descendants under unmatched parent objects/parts counted?

  3. Since Appendix B only reports full-chain accuracy on 1,000 images without incorrect samples, how do the baselines perform if evaluated on a human-corrected validation split?

3 comments
AuthorsOpenReview ↗

Response to Reviewer NTek (Part 1/2)

Dear Reviewer NTek, We provide below our rebuttal response addressing the questions and concerns raised in the review. We thank the reviewer for the careful and constructive assessment. We address the weaknesses (W) and questions (Q) below.

[W1, Q2]: End-to-End Counting and Correctness-Based Hierarchy Recovery

  • Denominators and End-to-End Chain Metrics

Appendix A does not sufficiently distinguish matching constraints from counting scope. An L2 TP requires correctly matched objects and parts; an L3 TP additionally requires a matching action and a spatially valid interaction point. However, all GT and predicted chains remain in the denominators:

At each level, false positives equal the total number of predicted chains minus true positives, and false negatives equal the total number of GT chains minus true positives. L2 counts object–part chains, while L3 counts object–part–affordance chains.

Descendants of unmatched GT parents remain FNs, and descendants of unmatched predicted parents remain FPs. Thus, missing most objects cannot be hidden by accurately parsing a few recovered objects: the reported L2/L3 are global end-to-end chain metrics, not matched-parent conditional scores. The supplementary implementation applies these rules; we will clarify Appendix A accordingly.

  • Correctness-Based Completeness

Object-level ParseRate uses all parse-eligible GT objects as its denominator. Success requires a correctly matched object and at least one correctly recovered downstream chain of the required type, rather than merely generating an expansion. To additionally assess exhaustive coverage, we report:

MetricDefinition
Per-object Part RecallCorrectly matched parts / GT parts per object, averaged over GT objects with parts
Per-object Affordance RecallCorrectly recovered affordance chains / GT affordance chains per object, averaged over GT objects with affordances
Complete RecoveryFraction of objects with every GT part and affordance chain correctly recovered; extra children are allowed
Exact RecoveryComplete Recovery with no extra predicted parts or affordances within the object

Missed GT objects receive zero recall. Complete/Exact Recovery use the same denominator as ParseRate: all parse-eligible GT objects. All correctness decisions follow the same hierarchical matching and spatial criteria.

  • Completeness Ablation
SettingParseRatePart RecallAffordance RecallComplete RecoveryExact Recovery
No pseudo, no curriculum39.6341.3326.3525.0120.27
Pseudo, no curriculum40.5042.0727.2325.8120.68
Pseudo + curriculum43.7544.3429.1627.7522.30

Compared with pseudo-only training, the curriculum setting improves Part/Affordance Recall by 2.27/1.94 points and Complete/Exact Recovery by 1.94/1.62 points. These results support improved correct child coverage and full-reference hierarchy recovery.

AuthorsOpenReview ↗

Response to Reviewer NTek (Part 2/2)

[W2、Q3] Human-Corrected Evaluation and Name-Matching Robustness

Thank you for highlighting potential bias from annotation errors and strict name matching. We address these concerns through two complementary evaluations.

1. Human-corrected evaluation. Beyond estimating annotation error rates, we corrected identified errors in a 500-image validation subset and re-evaluated all methods, keeping predictions, prompts, and evaluation rules unchanged.

MethodL1 F1L2 F1L3 F1ParseRate
Ours-3B (FT)75.24 → 75.6042.67 → 43.1327.28 → 27.3243.76 → 44.71
Qwen3.5-4B (FT)49.62 → 50.0425.20 → 25.7414.53 → 15.0518.79 → 19.14
Rex-Omni (FT)32.09 → 32.4119.85 → 19.9524.49 → 24.5515.71 → 15.86

Values are percentages; arrows indicate original → human-corrected GT. All three models improve slightly, while SceneParser retains its lead across all metrics. This provides evidence that its advantage on this subset is not merely due to reproducing automated annotation errors.

2. Name-matching robustness. We additionally evaluated three settings on the full validation set:

  • Strict: identical names after lowercasing and whitespace trimming.
  • GT-Vocab: a shared normalization and synonym mapping constructed exclusively from training and validation GT vocabularies.
  • GT+Pred-Vocab: an expanded mapping additionally covering names generated by all evaluated models, testing aliases outside the GT vocabulary.

The mappings cover punctuation, conservative singularization, action inflections, and an auditable synonym list. In both normalized settings, the same mapping is applied to GT and predictions for every model; all other evaluation rules remain unchanged. Representative results are:

MethodL1 F1L2 F1L3 F1
GPT-5.521.47 / 21.73 / 22.131.80 / 1.90 / 2.050.57 / 0.67 / 0.72
Gemini 3.1 Pro56.29 / 56.29 / 56.298.07 / 8.69 / 9.093.92 / 5.05 / 5.25
Qwen2.5-VL-72B42.13 / 42.14 / 42.143.24 / 3.34 / 3.440.95 / 1.12 / 1.16
Ours-3B (FT)73.82 / 73.82 / 73.8243.61 / 44.03 / 44.0329.40 / 30.53 / 30.61

Each cell reports Strict / GT-Vocab / GT+Pred-Vocab, in percentages. Normalization recovers additional valid matches, particularly at L3, while SceneParser’s substantial advantage over these baselines remains.

We will document the human-correction protocol and release the corrected subset, complete evaluation results, and canonical name mappings.

[Q1]: Scene-Level Baselines and Assembly Evaluation Protocol

Thank you for highlighting this omission. We now report both fine-tuned baselines on the same validation set under the same full-scene evaluation protocol, without GT category or box prompts.

MethodL1 F1L2 F1L3 F1ParseRate
Qwen3.5-4B (FT)50.937.025.943.7
Rex-Omni (FT)32.723.225.426.1
Ours-3B (FT)54.637.526.348.4

Rex-Omni Assembly uses GT-provided queries and parent–child identities, together with GT-assisted candidate selection. It therefore measures assisted grounding rather than autonomous full-scene parsing. We will explicitly label this distinction and add the above results to Table 1.

Thanks for the author's response. Most of my concerns are solved.

Reviewer_2SpVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SceneParser, a hierarchical scene parsing framework that represents an image as a scene–object–part–affordance hierarchy. The authors construct SceneParser-Bench, containing 110K training images and 5K validation images, with object, part, and affordance annotations linked through explicit parent–child relationships. They further propose Level-1 to Level-3 conditional evaluation metrics and ParseRate to assess localization, cross-level binding, and structural completeness. The proposed SceneParser model is built upon Rex-Omni and trained using structural-completion pseudo-labels and curriculum learning. Experiments compare the method with existing multimodal large language models and hierarchy assembly baselines, together with ablations, cross-dataset evaluations, and a downstream planning probe. The results suggest improved hierarchical parsing performance and potential benefits for interaction-oriented planning. However, the methodological novelty, the reliability of the automatically constructed benchmark, and the validity of some experimental conclusions require further examination.

Critical strengths
  1. A useful problem formulation and structured representation. The paper formulates hierarchical scene parsing as a unified scene–object–part–affordance prediction task. Explicitly linking functional parts and affordances to their parent entities is a meaningful direction beyond evaluating object, part, and affordance predictions independently.

  2. This paper introduces a hierarchical scene parsing dataset and a corresponding evaluation framework, and develops a lightweight model (3B/4B) that establishes a strong baseline for the task.

  3. A potentially valuable benchmark and evaluation protocol. SceneParser-Bench provides a substantial collection of hierarchical annotations, and the conditional L1–L3 metrics explicitly account for cross-level dependencies. The benchmark and evaluation protocol could facilitate further research on structured, interaction-oriented visual understanding.

  4. Broad experimental coverage. The paper evaluates multiple multimodal models, includes ablation studies, examines transfer to COCO and AGD20K, and investigates the utility of parsed hierarchies for downstream planning. The overall presentation is clear and the experimental scope is reasonably broad.

Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the contribution of hierarchical generation. SceneParser is built upon the existing Rex-Omni model, while the proposed training strategy primarily combines structural-completion pseudo-labels with curriculum learning. The improvements are potentially useful, but the methodological contribution beyond structured output generation and training-schedule design remains limited. More importantly, the ablation in Section 5.2.1 and Figure 4(a) reports a substantial L1 F1 difference between flat triplets and nested hierarchies (40.7 versus 73.8), despite the claim that both variants predict the same fields. Since flat predictions are subsequently converted into hierarchies through grouping and deduplication (Appendix H), it is unclear whether this comparison isolates the effect of hierarchical representation or is confounded by output serialization and post-processing. This weakens the evidence supporting the paper's central methodological claim.

  2. Insufficient validation of benchmark reliability and evaluation validity. The benchmark is constructed largely through an automated pipeline involving GPT-5 and multiple localization or segmentation models (Section 3.3). Although Appendix B reports a human audit of 1,000 validation images and 90.4% accuracy for complete object–part–affordance chains, the paper provides limited detail on annotation agreement and the types of semantic or relational errors that remain. In addition, ParseRate (Section 3.2 and Appendix A) primarily measures whether required hierarchical expansions are present, rather than whether the predicted parts and affordances are semantically and spatially correct. Consequently, a structurally expanded but incorrect prediction may still count as complete. These limitations raise concerns about the reliability of the benchmark and the extent to which the proposed metrics substantiate the claimed improvements in coherent hierarchical understanding.

  3. The scene parsing task does not explicitly model semantic relations between entities. The proposed representation organizes visual content into a hierarchy of scenes, objects, parts, and affordances, but does not explicitly capture the semantic relations between different entities. Such relations are essential for a more complete and structured understanding of a scene, as recognizing individual entities and their hierarchical associations alone cannot fully describe interactions and contextual dependencies among them. This limitation is particularly relevant in light of recent work on hierarchical scene graph generation, such as Liu et al., HSGG: Training-Free Hierarchical Scene Graph Generation with Geometry-Guided Relation Reasoning (ICML 2026), which jointly models hierarchical entity structures and semantic relations through geometry-guided reasoning. Incorporating relational annotations and evaluating the generation of hierarchical scene graphs could make the proposed dataset and benchmark more comprehensive and better support downstream structured reasoning.

  4. The technical contribution of the proposed lightweight model appears limited, as its training strategy relies primarily on curriculum learning without introducing substantial methodological innovations. Consequently, the main contributions seem to lie in constructing the dataset and adapting an existing model to establish a baseline. However, the dataset contribution could be further strengthened, and the methodological novelty remains insufficiently demonstrated. The authors should better articulate the unique value of the dataset compared with existing resources and provide stronger justification for the technical contributions beyond the application of curriculum learning and baseline development.

Questions for authors
  1. How are synonymous, near-synonymous, or inconsistently granular object and part labels normalized across the automatically generated annotations? Could the authors provide a more detailed description of the human audit protocol, including the sampling procedure and annotation agreement, and clarify whether the reported 90.4% complete-chain accuracy is representative of the long-tailed categories?

  2. In Appendix A, an interaction point is considered valid if it lies within the ground-truth affordance box, with the parent part box used as a fallback when the affordance box is unavailable. How frequently is this fallback used, and how sensitive are the L3 results to this choice? Would the main conclusions remain unchanged under a stricter spatial criterion?

  3. Given the relatively small changes in L2 and L3 F1 in Table 2, can the authors demonstrate that structural-completion pseudo-labels and curriculum learning provide consistent gains over stronger alternative training strategies? In particular, how much of the improvement is attributable to the nested output representation itself rather than the proposed training procedure?

  4. Have the authors evaluated generalization to unseen object–part–affordance compositions or held-out semantic categories, beyond image-level separation and evaluation on VSIBench? Additionally, how do the authors distinguish the proposed functional hierarchy from a more general scene graph that explicitly represents relations among different objects?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2SpV (Part 1/3)

Dear Reviewer 2SpV, We provide below our rebuttal response addressing the questions and concerns raised in the review. We thank the reviewer for the careful and constructive assessment. We address the weaknesses (W) and questions (Q) below.

[W1、W4、Q3]:Clarifying Contributions and Isolating Representation and Training Gains

  • Method and Benchmark Contributions

SceneParser provides a unified framework for grounded object-part-affordance parsing through three complementary designs:

  1. Direct hierarchical generation jointly predicts grounded entities and explicitly binds parts to objects and affordances to functional parts.
  2. Structural-completion supervision explicitly supervises branch expansion and termination without adding semantic instances.
  3. Structural curriculum learning progressively introduces completion supervision after emphasizing visual grounding.

SceneParser-Bench provides object/part localization, action-labeled interaction points, and cross-level bindings, evaluating which object’s which part supports which action, and where the interaction occurs. Hierarchy-aware matching assesses these associations. On 200 image-task pairs, adding SceneParser hierarchies improves Complete Plan Rate from 28% to 43% for GPT-5.5 and from 29% to 71% for Claude Opus 4.7, supporting their utility for interaction-oriented planning.

  • Flat-to-Hierarchy Conversion

We group triplets with identical object names and boxes into one object node. Within each object, identical part names and boxes form one part node. Action-point pairs are attached to their corresponding parts, removing only identical pairs; different interaction points are retained. GT remains unchanged.

To test whether small box-coordinate differences affect grouping, we keep predictions fixed and merge same-name nodes whose box IoU exceeds a threshold. Each threshold is applied at both object and part levels.

Grouping RuleL1 F1L2 F1L3 F1
Exact-box grouping40.674822.265017.7788
Same name, IoU ≥ 0.9540.693922.268317.7788
Same name, IoU ≥ 0.9040.693922.268317.7788
Same name, IoU ≥ 0.7540.703422.271617.7788

Maximum changes are only 0.03/0.01/0.00 percentage points, showing that the tested grouping choices have a minimal impact on the original Flat results.

  • Separating Representation and Training Contributions

To separate training contributions from output representation, we hold the nested format fixed and compare structural-completion supervision with and without curriculum learning.

Training SettingL1 F1L2 F1L3 F1ParseRatePart RecallAffordance RecallExact Recovery
No completion, no curriculum74.0743.1329.0639.6341.3326.3520.27
Completion, no curriculum74.3043.0728.8140.5042.0727.2320.68
Ours-3B (FT), completion + curriculum73.8243.6129.4043.7544.3429.1622.30

The final row corresponds to the object-level results of Ours-3B (FT) in Table 1, using 50% structural-completion samples in Stage 3.

Completion supervision alone modestly improves coverage and exact recovery, although entity-level F1 does not uniformly increase. Adding curriculum further improves Part/Affordance Recall by 2.27/1.94 points and Exact Recovery by 1.62 points. These comparisons quantify training-related gains while keeping the output format unchanged.

  • Comparison with Alternative Training Strategies

    The completion-without-curriculum baseline introduces all completion supervision directly, providing an alternative to progressive scheduling. Compared with this baseline, our curriculum improves L2/L3 F1 by 0.54/0.59 points, ParseRate by 3.26 points, and both child-coverage and exact-recovery measures.

    Thus, the observed benefit extends beyond simply adding completion targets: progressive scheduling improves downstream correctness and hierarchical completeness. We will include these results and scope our conclusion to the evaluated training strategies.

AuthorsOpenReview ↗

Response to Reviewer 2SpV (Part 2/3)

[W2、Q1]: Annotation Reliability

Thank you for these questions. We address the audit protocol, residual annotation errors and long-tail coverage, name matching, and evaluation validity separately.

  • Residual Errors and Long-Tail Coverage

    The recorded errors include semantic mislabeling, localization errors, duplicate or nonexistent annotations, omissions, and incorrect parent-child bindings.

    We additionally stratify reviewed chains by the minimum training frequency of their object, part, and action categories:

    Frequency GroupTraining FrequencyAccuracy
    Head>10089.8%
    Medium11–10094.9%
    Tail1–1095.2%
    OverallNot applicable90.4%

Medium- and tail-category chains achieve 94.85% and 95.24% accuracy, respectively, showing no observed decline for tail-category chains.

  • Name Normalization and Synonym Matching

    The primary evaluation uses identical names after lowercasing and whitespace trimming. We additionally constructed a shared mapping using only training and validation GT vocabularies. The same mapping is applied to GT and predictions for every method, without changing spatial matching.

MethodL1 F1L2 F1L3 F1
GPT-5.521.47 → 21.731.80 → 1.900.57 → 0.67
Gemini 3.1 Pro56.29 → 56.298.07 → 8.693.92 → 5.05
Qwen2.5-VL-72B42.13 → 42.143.24 → 3.340.95 → 1.12
Ours-3B (FT)73.82 → 73.8243.61 → 44.0329.40 → 30.53

Values are percentages; arrows indicate strict → GT-vocabulary-normalized matching. Normalization recovers valid matches, while SceneParser’s substantial advantage over these baselines remains.

  • ParseRate and Correctness-Aware Metrics

ParseRate requires a correctly matched object and a correct downstream chain, including semantic, spatial, and parent-child binding correctness, rather than merely hierarchical expansion.

We additionally report three correctness-aware metrics:

  • Part Recall: the fraction of GT parts correctly recovered, averaged over GT objects with parts.
  • Affordance Recall: the fraction of GT affordance chains correctly recovered, averaged over GT objects with affordances.
  • Exact Recovery: the fraction of eligible GT objects whose entire subtree is correctly recovered, with no missing or extra parts or affordances.

Missed objects receive zero credit. All matches follow the same semantic, spatial, and binding criteria as L1–L3.

MethodParseRatePart RecallAffordance RecallExact Recovery
Qwen3.5-4B (FT)20.621.612.410.3
Rex-Omni (FT)16.512.819.83.3
Ours-3B (FT)43.844.329.222.3

SceneParser outperforms both fine-tuned baselines across all four metrics, supporting gains in correct hierarchical recovery rather than structural expansion alone. We will add these definitions and results to the revised manuscript.

  • Human Audit Protocol and Agreement

    For additional independent re-auditing, we specify the following protocol:

    The audit protocol should specify the random sampling seed and image list. Annotators independently assess object, part, and affordance labels and localization, together with object–part and part–affordance bindings, recording errors and omissions. A complete chain is correct only when all three nodes and both bindings are correct. Agreement is measured on overlapping independently reviewed samples before adjudication; disagreements are then resolved, and node precision/recall, binding accuracy, and complete-chain accuracy are reported.

AuthorsOpenReview ↗

Response to Reviewer 2SpV (Part 3/3)

[W3, Q4]: Representation Scope and Generalization

  • Functional Hierarchy vs. Inter-Object Relations

SceneParser focuses on a complementary question: which part of which object supports an action, and where the interaction should occur. For example, “a cup on a table” describes an inter-object relation, whereas “grasping the cup by its handle” requires grounding an action to a specific part and interaction point. Our contribution is explicit supervision and hierarchy-aware evaluation of these object–part–affordance bindings.

We will expand the discussion of HSGG, clarify that general inter-object relations are outside our current scope, and discuss relational annotations and joint evaluation as future extensions.

  • Generalization to Unseen Combinations and Categories

Appendix J evaluates generalization to new images from VSIBench without further training. However, new images do not necessarily contain new object–part–affordance combinations or categories absent from our fine-tuning data. We have not separately evaluated these two cases. We will clarify that our current results demonstrate cross-dataset transfer, and consider dedicated evaluations of unseen combinations and categories as future work.

[Q2]: Affordance-Box Fallback and Its Impact on L3 Evaluation

The fallback clause was inadvertently carried over from an internal diagnostic setting that retained structural-completion placeholders to assess branch expansion and termination. These placeholders may lack affordance boxes but are excluded from the reported evaluation. All 51,782 valid affordances across the 5,000 validation images have affordance boxes, so the fallback frequency is 0%. Removing it leaves all L3 scores and model rankings unchanged. We will remove this unnecessary clause from Appendix A.

Reviewer_B8HnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates interaction-oriented hierarchical scene parsing as a scene → object → part → affordance prediction task, where each object and part has a bounding box and each affordance specifies an action and interaction point. The proposed SceneParser model builds on Rex-Omni and auto-regressively generates nested JSON-style hierarchies. The paper also introduces SceneParser-Bench, derived from EgoObjects through an automated annotation pipeline using GPT-5, grounding/segmentation models, matching, and quality checks. It reports 110K training and 5K validation images, approximately 777K objects, 1.14M parts, and 1.74M affordances. The authors define conditional L1 object, L2 object–part, and L3 object–part–affordance metrics, together with ParseRate for structural expansion.

Critical strengths
  1. A meaningful problem formulation and evaluation contribution. By connecting objects, functional parts, and grounded affordances within an explicit scene-level representation, this work addresses an identifiable gap between isolated visual grounding outputs and interaction-oriented understanding.
  2. The benchmark is large, hierarchically deeper than what exists, and shaped so that others can reuse it.
Critical weaknesses
  1. Clarification is needed on metric interpretation, particularly conditional counting and ParseRate.
  2. Supervision and evaluation targets are produced by the same automated pipeline, and the audit measures plausibility rather than validity.
  3. Please report L2 conditioned on ground-truth objects and L3 conditioned on ground-truth object-part pairs.
  4. Given the definition in Appendix A, does predicting a single part for an object annotated with five parts earn full credit? Please report per-object part/affordance recall, or a coverage-aware ParseRate, alongside the current number.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer B8Hn (Part 1/2)

Dear Reviewer B8Hn, We provide below our rebuttal response addressing the questions and concerns raised in the review. We thank the reviewer for the careful and constructive assessment. We address the weaknesses (W) and questions (Q) below.

[W1, W3]: Clarification of End-to-End Metrics, ParseRate, and Matched-Parent Conditional Evaluation

  • [W1] End-to-End Counting and ParseRate

    Thank you for highlighting this ambiguity. We will explicitly distinguish parent-matching constraints from counting scope in Appendix A. An L2 true positive requires a correctly matched object–part pair, while an L3 true positive additionally requires a matching action and a spatially valid interaction point. However, all GT and predicted descendants remain in the counting scope: descendants of unmatched GT parents count as FNs, and descendants of unmatched predicted parents count as FPs. Thus, the reported L2/L3 scores are end-to-end metrics that already penalize upstream errors, rather than metrics restricted to matched parents.ParseRate requires a correctly matched object and at least one correctly recovered downstream chain of the required type, not merely generating a hierarchy expansion.

  • [W3] Downstream Quality Under Matched Parents

Our model jointly generates the complete hierarchy and has no dedicated interface for decoding children from supplied GT parents. We therefore report matched-parent conditional evaluation as a complementary diagnostic: conditional L2 counts parts only within correctly matched object pairs, while conditional L3 counts affordances only within correctly matched object–part pairs.

MethodE2E L2 F1Conditional L2 F1E2E L3 F1Conditional L3 F1
GPT-5.51.86.40.623.0
Gemini 3.1 Pro8.112.53.932.7
Qwen3.5-plus4.713.01.420.3
Qwen2.5-VL-72B3.26.30.920.8
Qwen3-VL-235B4.614.50.813.0
Qwen3.5-35B5.512.60.810.2
SEED2.05.011.20.811.5
Qwen3.5-4B (FT)27.245.813.051.5
Ours-3B (FT)43.651.529.466.9

All values are percentages, using identical predictions for each method. SceneParser achieves conditional L2/L3 F1 of 51.48/66.85, compared with end-to-end 43.61/29.40. These diagnostics characterize downstream quality when the model has correctly recovered the parents.

[W2]: Annotation Validity and Shared-Pipeline Bias

To go beyond estimating annotation quality, we corrected identified annotation errors in a 500-image validation subset and re-evaluated all models against the corrected labels, keeping predictions, prompts, and evaluation rules unchanged.

MethodL1 F1L2 F1L3 F1ParseRate
Ours-3B (FT)75.24 → 75.6042.67 → 43.1327.28 → 27.3243.76 → 44.71
Qwen3.5-4B (FT)49.62 → 50.0425.20 → 25.7414.53 → 15.0518.79 → 19.14
Rex-Omni (FT)32.09 → 32.4119.85 → 19.9524.49 → 24.5515.71 → 15.86

Values are percentages; arrows indicate original → human-corrected GT.

All three models improve slightly, while SceneParser retains its lead across all metrics. This provides evidence that its advantage on this subset is not merely due to reproducing errors from the shared annotation pipeline.

AuthorsOpenReview ↗

Response to Reviewer B8Hn (Part 2/2)

[W4] Clarifying ParseRate and Evaluating Hierarchical Coverage and Completeness

Predicting one correct part out of five does not receive full L2 credit: the remaining four GT parts count as FNs, and that object’s Part Recall is 20%. It may satisfy ParseRate, but only when the required downstream chain is correctly recovered: object/part names and boxes must match, and, when affordances are required, the action and interaction point must also satisfy the matching criteria. Merely generating children is insufficient. We will clarify these constraints in Appendix A.

To directly assess coverage and completeness, we additionally report:

  • Per-object Part/Affordance Recall: the correctly recovered fraction of parts/affordance chains within each GT object, averaged over objects containing the corresponding GT children; missed objects receive zero.
  • Complete Recovery: a correctly matched object with every GT part and affordance chain recovered; extra predicted children are allowed.
  • Exact Recovery: Complete Recovery with no extra predicted parts or affordances within that object.

Complete/Exact Recovery use all parse-eligible GT objects as their denominator and follow the same hierarchical matching and spatial criteria.

MethodParseRatePart RecallAffordance RecallComplete RecoveryExact Recovery
GPT-5.51.82.41.00.30.0
Gemini 3.1 Pro5.05.53.00.70.2
Qwen3.5-plus5.78.82.71.40.1
Qwen2.5-VL-72B1.32.20.70.30.0
Qwen3-VL-235B3.57.51.70.70.0
Qwen3.5-35B3.28.41.30.80.0
SEED2.01.53.50.70.30.0
Qwen3.5-4B (FT)20.621.612.413.210.3
Rex-Omni Assembly (GT-assisted)32.248.58.926.726.6
Rex-Omni (FT)16.512.819.84.73.3
Ours-3B (FT)43.844.329.227.722.3

All values are percentages. Assembly uses GT-assisted queries/bindings. Rex-Omni Assembly performs modular grounding using GT-provided queries at each level and parent–child bindings. This setting therefore evaluates GT-assisted grounding rather than autonomous generation of the complete hierarchy. Exact Recovery uses the matching criteria, not exact numerical coordinate equality.

These measures distinguish recovering at least one correct chain, correct child coverage, and complete recovery without extra children. SceneParser achieves Complete/Exact Recovery of 27.7%/22.3%, compared with 13.2%/10.3% for Qwen3.5-4B (FT), demonstrating more complete recovery of the reference hierarchy.

I thank the authors for the response. W1 and W4 are largely addressed through the clarification of end-to-end counting/ParseRate and the added per-object recall and Complete/Exact Recovery metrics, while W2 is only partially mitigated by the 500-image human-corrected subset. However, W3 remains unresolved because the reported “matched-parent conditional” evaluation is not conditioned on ground-truth objects/object–part pairs as requested. I also note that Reviewer 2SpV raises similar concerns about the reliability of the automatically constructed benchmark and the insufficiency of ParseRate in ensuring semantic/spatial correctness and coverage. Overall, given that several key issues remain only partially addressed and Reviewer 2SpV raises overlapping concerns, I keep my rating.

AuthorsOpenReview ↗

Clarifying GT-Parent Evaluation and Annotation Reliability

Thank you for the follow-up and for recognizing that W1 and W4 are largely addressed. We clarify the remaining concerns below.

GT-Parent Evaluation (W3). Our model jointly generates the hierarchy and does not provide a parent-conditioned decoding interface. Adding this capability through training or adaptation would evaluate a modified model, rather than the original model under different evaluation conditions. Our conclusion of stronger end-to-end hierarchy recovery is supported independently by Complete/Exact Recovery: 27.7%/22.3%, versus 13.2%/10.3% for Qwen3.5-4B (FT) and 4.7%/3.3% for Rex-Omni (FT). Both metrics require correctly recovering every GT part and affordance chain; Exact Recovery additionally disallows extra predicted children. Missed objects receive zero credit.These results demonstrate stronger end-to-end recovery of complete reference hierarchies.

Annotation Validity and Shared-Pipeline Bias (W2). Beyond estimating annotation quality, we corrected identified errors in 500 validation images and re-scored the same predictions against the original and corrected labels, keeping prompts and evaluation rules unchanged.

MethodL1 F1L2 F1L3 F1ParseRate
SceneParser75.24 → 75.6042.67 → 43.1327.28 → 27.3243.76 → 44.71
Qwen3.5-4B (FT)49.62 → 50.0425.20 → 25.7414.53 → 15.0518.79 → 19.14
Rex-Omni (FT)32.09 → 32.4119.85 → 19.9524.49 → 24.5515.71 → 15.86

Values are percentages; arrows indicate original → human-corrected GT.

All three models improve slightly, while SceneParser retains its lead across all four metrics. These results support the robustness of the observed performance advantage to annotation correction and provide evidence against attributing it solely to reproducing errors from the shared annotation pipeline.

We hope these clarifications and results address your remaining concerns, and respectfully ask you to reconsider your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To AC: Rebuttal Summary

Dear AC, SAC, and PC,

Thank you for your time and for supporting the review process. We appreciate the reviewers’ constructive feedback and summarize our rebuttal below.

The reviewers recognized the value of interaction-oriented hierarchical scene parsing, explicit cross-level bindings, and a reusable benchmark. Our responses provide additional evidence on evaluation correctness, annotation reliability, baseline fairness, and training effectiveness.

Reviewer NTek (Initial Score: 3)

  • Metric interpretation: We clarified end-to-end counting and added per-object recall and Complete/Exact Recovery, explicitly penalizing missed objects and descendants.
  • Annotation reliability: Human-corrected evaluation with fixed predictions and synonym-aware matching both preserve SceneParser’s advantage over the compared baselines.
  • Missing scene-level baselines: We added Qwen3.5-4B (FT) and Rex-Omni (FT) under the same full-scene protocol. SceneParser achieves 48.4 ParseRate, compared with 43.7 and 26.1, respectively.

Following our responses, the reviewer stated: “Most of my concerns are solved.”

Reviewer B8Hn (Initial Score: 2)

  • Correctness and coverage: SceneParser achieves 27.7% Complete Recovery and 22.3% Exact Recovery, compared with 13.2%/10.3% for Qwen3.5-4B (FT), demonstrating stronger recovery of complete reference hierarchies.
  • Annotation validity: Re-evaluating unchanged predictions against human-corrected labels preserves SceneParser’s lead across all four reported metrics.
  • Parent-conditioned evaluation: We clarified the joint-generation setting and provided matched-parent diagnostics alongside end-to-end recovery metrics.

The reviewer acknowledged that W1 and W4 are largely addressed. We subsequently provided a focused follow-up on annotation validity and the parent-conditioned evaluation setting.

Reviewer 2SpV (Initial Score: 2)

  • Contributions: We clarified the unified task, cross-level supervision, and benchmark value beyond isolated object, part, or affordance prediction.
  • Post-processing sensitivity: Alternative Flat-to-hierarchy grouping thresholds change L1/L2/L3 F1 by at most 0.03/0.01/0.00 points.
  • Training effectiveness: Holding nested outputs fixed, curriculum improves Part/Affordance Recall by 2.27/1.94 points and Exact Recovery by 1.62 points over direct completion supervision.
  • Benchmark analysis: We supplied residual-error analysis, frequency-stratified audit results, synonym-aware evaluation, and correctness-aware recovery measures.

The representation’s downstream value is further supported by the controlled planning probe: Complete Plan Rate increases from 28% to 43% for GPT-5.5 and 29% to 71% for Claude Opus 4.7.

We hope these additional experiments and clarifications help inform your assessment of our work. We sincerely thank you and the reviewers for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.