- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new task that generates the structured dependencies for scene parsing. To support evaluation, they build SceneParser-Bench, which contains 110k train/5k val images with affordance annotations generated by an automated pipeline and introduces Level-1 to Level-3 condition metrics with ParseRate. They also train SceneParser using pseudo-labels and curriculum learning to solve this task. Experiments show better performance on SceneParser-Bench than general MLLMs and demonstrate that SceneParser provides good representation for downstream tasks.
Critical strengths
-
The built datasets and introduced metrics provide a good benchmark to evaluate the structure-aware performance of models. The hierarchical data engine that generated affordance annotations with 90.4% full-chain accuracy provides an effective solution for reducing manual annotation costs.
-
The proposed hierarchical scene parsing organizes scenes into explicit structures, and it provides the interaction-relevant scene information for downstream manipulation, filling the gap between isolated predictions and compositional scene understanding.
Critical weaknesses
- The introduced metrics are flawed. Appendix A shows that L2 is only computed over matched parent objects, and L3 is computed over matched object-part pairs. Under this rule, although a model fails to detect almost all objects in a scene, it can still get a high L2 score by parsing parts on few matched objects. And the ParseRate only checks whether the structure is complete, rather than whether the child predictions are actually correct.
- Although Appendix B audits 1000 validation images with 90.4% full-chain accuracy, humans did not correct wrong samples. And Figure 6 shows spelling errors such as “obj:glaves”, “part:handel” which may happen in the datasets. With these wrong labels, models are penalized although outputting the right answers. The datasets are generated by the automated pipeline with Rex-Omni, and SceneParser is fine-tuned from Rex-Omni, which can naturally get better performance under the strict name-matching rule.
Questions for authors
-
Why are the scene-level evaluation numbers omitted (-) for Qwen3.5-4B (FT), Rex-Omni Assembly, and Rex-Omni (FT) in Table 1? Could you provide these results?
-
In L2/L3, how are the GT and predicted descendants under unmatched parent objects/parts counted?
-
Since Appendix B only reports full-chain accuracy on 1,000 images without incorrect samples, how do the baselines perform if evaluated on a human-corrected validation split?
3 comments
Response to Reviewer NTek (Part 2/2)
[W2、Q3] Human-Corrected Evaluation and Name-Matching Robustness
Thank you for highlighting potential bias from annotation errors and strict name matching. We address these concerns through two complementary evaluations.
1. Human-corrected evaluation. Beyond estimating annotation error rates, we corrected identified errors in a 500-image validation subset and re-evaluated all methods, keeping predictions, prompts, and evaluation rules unchanged.
| Method | L1 F1 | L2 F1 | L3 F1 | ParseRate |
|---|---|---|---|---|
| Ours-3B (FT) | 75.24 → 75.60 | 42.67 → 43.13 | 27.28 → 27.32 | 43.76 → 44.71 |
| Qwen3.5-4B (FT) | 49.62 → 50.04 | 25.20 → 25.74 | 14.53 → 15.05 | 18.79 → 19.14 |
| Rex-Omni (FT) | 32.09 → 32.41 | 19.85 → 19.95 | 24.49 → 24.55 | 15.71 → 15.86 |
Values are percentages; arrows indicate original → human-corrected GT. All three models improve slightly, while SceneParser retains its lead across all metrics. This provides evidence that its advantage on this subset is not merely due to reproducing automated annotation errors.
2. Name-matching robustness. We additionally evaluated three settings on the full validation set:
- Strict: identical names after lowercasing and whitespace trimming.
- GT-Vocab: a shared normalization and synonym mapping constructed exclusively from training and validation GT vocabularies.
- GT+Pred-Vocab: an expanded mapping additionally covering names generated by all evaluated models, testing aliases outside the GT vocabulary.
The mappings cover punctuation, conservative singularization, action inflections, and an auditable synonym list. In both normalized settings, the same mapping is applied to GT and predictions for every model; all other evaluation rules remain unchanged. Representative results are:
| Method | L1 F1 | L2 F1 | L3 F1 |
|---|---|---|---|
| GPT-5.5 | 21.47 / 21.73 / 22.13 | 1.80 / 1.90 / 2.05 | 0.57 / 0.67 / 0.72 |
| Gemini 3.1 Pro | 56.29 / 56.29 / 56.29 | 8.07 / 8.69 / 9.09 | 3.92 / 5.05 / 5.25 |
| Qwen2.5-VL-72B | 42.13 / 42.14 / 42.14 | 3.24 / 3.34 / 3.44 | 0.95 / 1.12 / 1.16 |
| Ours-3B (FT) | 73.82 / 73.82 / 73.82 | 43.61 / 44.03 / 44.03 | 29.40 / 30.53 / 30.61 |
Each cell reports Strict / GT-Vocab / GT+Pred-Vocab, in percentages. Normalization recovers additional valid matches, particularly at L3, while SceneParser’s substantial advantage over these baselines remains.
We will document the human-correction protocol and release the corrected subset, complete evaluation results, and canonical name mappings.
[Q1]: Scene-Level Baselines and Assembly Evaluation Protocol
Thank you for highlighting this omission. We now report both fine-tuned baselines on the same validation set under the same full-scene evaluation protocol, without GT category or box prompts.
| Method | L1 F1 | L2 F1 | L3 F1 | ParseRate |
|---|---|---|---|---|
| Qwen3.5-4B (FT) | 50.9 | 37.0 | 25.9 | 43.7 |
| Rex-Omni (FT) | 32.7 | 23.2 | 25.4 | 26.1 |
| Ours-3B (FT) | 54.6 | 37.5 | 26.3 | 48.4 |
Rex-Omni Assembly uses GT-provided queries and parent–child identities, together with GT-assisted candidate selection. It therefore measures assisted grounding rather than autonomous full-scene parsing. We will explicitly label this distinction and add the above results to Table 1.
Thanks for the author's response. Most of my concerns are solved.
Response to Reviewer NTek (Part 1/2)
Dear Reviewer NTek, We provide below our rebuttal response addressing the questions and concerns raised in the review. We thank the reviewer for the careful and constructive assessment. We address the weaknesses (W) and questions (Q) below.
[W1, Q2]: End-to-End Counting and Correctness-Based Hierarchy Recovery
Appendix A does not sufficiently distinguish matching constraints from counting scope. An L2 TP requires correctly matched objects and parts; an L3 TP additionally requires a matching action and a spatially valid interaction point. However, all GT and predicted chains remain in the denominators:
At each level, false positives equal the total number of predicted chains minus true positives, and false negatives equal the total number of GT chains minus true positives. L2 counts object–part chains, while L3 counts object–part–affordance chains.
Descendants of unmatched GT parents remain FNs, and descendants of unmatched predicted parents remain FPs. Thus, missing most objects cannot be hidden by accurately parsing a few recovered objects: the reported L2/L3 are global end-to-end chain metrics, not matched-parent conditional scores. The supplementary implementation applies these rules; we will clarify Appendix A accordingly.
Object-level ParseRate uses all parse-eligible GT objects as its denominator. Success requires a correctly matched object and at least one correctly recovered downstream chain of the required type, rather than merely generating an expansion. To additionally assess exhaustive coverage, we report:
Missed GT objects receive zero recall. Complete/Exact Recovery use the same denominator as ParseRate: all parse-eligible GT objects. All correctness decisions follow the same hierarchical matching and spatial criteria.
Compared with pseudo-only training, the curriculum setting improves Part/Affordance Recall by 2.27/1.94 points and Complete/Exact Recovery by 1.94/1.62 points. These results support improved correct child coverage and full-reference hierarchy recovery.