- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces World Scene Graph Generation (WSGG) and ActionGenome4D, extending ActionGenome with 3D boxes, camera poses, and relationships involving unobserved objects. It adapts video scene-graph baselines, proposes the WorldWise family using visual/geometric representations and masked completion, and evaluates zero-shot UWSGG-GraphRAG. In practice, the supervised setting uses complete videos and preconstructed persistent object representations with supplied temporal identities, primarily predicting person–object relationships.
Critical strengths
- Maintaining relationships involving occluded or out-of-view objects is a meaningful research problem.
- ActionGenome4D extends existing video relationship annotations to 3D geometry and unobserved objects through monocular reconstruction, geometric alignment, temporal completion, and human correction. The documented pipeline, annotation tools, and quality-control procedures provide a valuable foundation for further research.
- The WorldWise family shows advantages in class-balanced relationship prediction, suggesting potential benefits for unobserved objects and long-tail predicates. This makes the combination of temporal information and structured completion a worthwhile direction for further study.
Critical weaknesses
- The paper targets persistent 3D world understanding, but the main supervised experiments rely on preconstructed object representations and temporal identities. Joint validation of complete object discovery, 3D localization, and recovery of persistently unobserved objects remains limited. End-to-end evaluation over the complete object inventory, including more challenging occlusions and dynamic interactions, would establish whether the benefits extend beyond relationship prediction conditioned on existing objects.
- The independent contribution of the completion mechanism needs fairer controls. The proposed methods and baselines differ in feature sources, supervision, and training strategies, making it difficult to attribute the overall improvements specifically to the completion architecture. Experiments should align these conditions, directly ablate the core completion mechanism, and compare it with simpler temporal memory or information-propagation alternatives. This would clarify whether the additional model complexity provides a distinctive benefit.
- The conclusions need a more independent and consistent evaluation framework. Test-based model selection and differences in object coverage and evaluation settings limit the reliability and comparability of the reported results. An independent validation split and a unified test protocol should support fair comparisons of the main methods and the GraphRAG branch. Repeated experiments and inference-cost evaluation would further establish whether the benefits are stable and practically meaningful.
Questions for authors
See the weakness above.
4 comments
Response to Reviewer y9Am: Weakness 2
W2. The proposed methods and baselines differ in feature sources, supervision, and training strategies, making it difficult to attribute improvements specifically to the completion architecture; align these conditions, directly ablate completion, and compare against simpler temporal memory.
- Adapting strong VidSGG backbones as natural baselines for a new task. As we introduce World Scene Graph Generation (WSGG) as a new task, establishing principled baselines naturally begins with adapting strong existing methods from the closest related domain. Following this standard practice, we first adapted the leading object-centric representation learning backbones in Video Scene Graph Generation (W-STTran, W-DSGDetr, and W-USG, together with their geometry-augmented W-STTran++ and W-DSGDetr++ variants) to operate over persistent observed and unobserved world states.
- Designing the WorldWise family from adapted baseline insights. Specifically, once we observed the empirical impact and limitations of each representation, temporal-aggregation, and training component in these adapted VidSGG methods, we designed the progressive architecture of the WorldWise family (WorldWise, WorldWise+, and WorldWise++) to directly leverage those findings for persistent 4D world scene graphs.
- Entry-point baselines versus our proposed architecture. Thus, our adapted VidSGG models serve as natural entry-point baselines for WSGG, while our proposed WorldWise family significantly outperforms every adapted baseline across both PredCls and SGDet under 2D and uniform 3D evaluation (achieving gains of +15.5% / +120.9% in SGDet 2D WC R@20 / mR@20, +20.9% / +133.3% in SGDet 3D WC R@20 / mR@20, and +5.6% / +24.0% in PredCls WC R@20 / mR@20 in Table 2).
- Adapted baselines as temporal-memory propagation models. Moreover, W-STTran (W-STTran++) and W-DSGDetr (W-DSGDetr++) can indeed be viewed directly as baselines equipped with simple temporal memory and cross-frame information propagation: because persistent object slots are maintained across the video, their temporal self-attention encoders aggregate and propagate context from visible frames into occluded and out-of-view frames, and our WorldWise family demonstrates substantial improvements over these temporal-memory baselines.
Response to Reviewer y9Am: Weakness 3
W3. Test-based selection and differences in object coverage and evaluation settings limit reliability and comparability; an independent validation split, a unified test protocol across the main methods and the GraphRAG branch, repeated experiments, and inference-cost evaluation are needed.
We agree, and we addressed these concerns in our revised experiments and manuscript:
- 1. Independent validation split (Section 5). Every supervised model in Table 2 now selects its checkpoint by With Constraint (WC) R@20 on a 750-video validation split held out from the official training split (training on the remaining 6,766 videos), never accessing the 1,511-video test split.
- 2. Unified full-inventory supervised protocol and zero-shot GraphRAG positioning (Section 5; Section 5.2). Every supervised method is now trained and evaluated under a single unified protocol across all 773,236 annotated triplets (414,443 OO and 358,793 UO), with identical setting-specific inputs, 2D and uniform 3D (IoU3D ≥ 0.10) matching rules, and confidence-ranked scoring. For the zero-shot U-WSGG branch (Section 5.2, Table 3), because generative MLLMs emit discrete predicate sets rather than continuous rankings over all 26 predicates, we evaluate UWSGG-GraphRAG against caption-enriched prompting across five diverse MLLM families under both PredCls and SGDet using P / R / F1 without cross-ranking zero-shot F1 directly against supervised R@K. Moreover, we position UWSGG-GraphRAG as a strong structured-retrieval baseline that enhances zero-shot MLLM performance over raw caption transcripts across backbones by adapting GraphRAG to spatio-temporal world scene graphs.
Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):
| Method | SGDet WC 2D | SGDet WC 3D | SGDet NC 2D | SGDet NC 3D | SGDet UO (NC) 2D | SGDet UO (NC) 3D | PredCls WC | PredCls NC | PredCls UO (NC) |
|---|---|---|---|---|---|---|---|---|---|
| W-STTran | 19.8 / 8.1 | 4.0 / 2.2 | 20.0 / 8.4 | 4.1 / 2.3 | 20.8 / 5.9 | 2.8 / 1.3 | 64.1 / 38.1 | 78.4 / 55.1 | 67.2 / 24.9 |
| W-STTran++ | 20.5 / 8.6 | 4.2 / 2.4 | 20.7 / 8.9 | 4.3 / 2.5 | 20.8 / 6.5 | 2.8 / 1.5 | 64.3 / 39.1 | 78.5 / 56.3 | 66.9 / 24.7 |
| W-DSGDetr | 21.3 / 8.4 | 4.3 / 2.3 | 21.4 / 8.7 | 4.4 / 2.4 | 22.5 / 6.2 | 2.9 / 1.3 | 64.3 / 37.4 | 78.5 / 55.5 | 66.0 / 23.5 |
| W-DSGDetr++ | 21.1 / 8.3 | 4.3 / 2.2 | 21.3 / 8.6 | 4.4 / 2.4 | 22.3 / 6.3 | 2.9 / 1.4 | 64.5 / 37.3 | 78.6 / 54.5 | 66.4 / 24.4 |
| W-USG | 17.1 / 7.3 | 3.2 / 1.7 | 17.3 / 7.7 | 3.3 / 1.9 | 18.1 / 5.3 | 2.3 / 1.2 | 63.7 / 37.1 | 77.9 / 54.1 | 66.0 / 24.0 |
| WorldWise | 23.9 / 18.1 | 5.1 / 5.7 | 24.3 / 21.0 | 5.3 / 6.1 | 19.8 / 12.8 | 2.8 / 2.4 | 63.7 / 42.4 | 77.8 / 62.0 | 67.8 / 32.8 |
| WorldWise+ | 24.3 / 18.6 | 5.2 / 5.5 | 24.7 / 21.0 | 5.3 / 5.9 | 20.7 / 13.8 | 2.9 / 2.2 | 65.6 / 48.0 | 79.0 / 68.3 | 66.9 / 42.9 |
| WorldWise++ | 24.6 / 19.0 | 5.2 / 5.6 | 25.0 / 21.8 | 5.3 / 6.1 | 19.8 / 12.5 | 2.6 / 2.0 | 68.1 / 48.5 | 80.7 / 68.8 | 69.7 / 46.4 |
| Δ vs. best baseline (%) | +15.5 / +120.9 | +20.9 / +133.3 | +16.8 / +144.9 | +20.5 / +144.0 | -12.0 / +92.3 | -10.3 / +33.3 | +5.6 / +24.0 | +2.7 / +22.2 | +3.7 / +86.3 |
I thank the authors for the comprehensive rebuttal and additional results, in which my concerns have been well addressed. I would, therefore, keep my original score.
Response to Reviewer y9Am: Weakness 1
To validate end-to-end world graph recovery over the complete annotated inventory, we rebuilt the supervised evaluation protocol and retrained every method from scratch:
(object class, rank)(where rank 0 is the highest-scoring detection of that class) without relying on an external multi-object tracker.Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):
Insights and Scope Limitations (Section 6):