acceptodds
Withdrawn from ICLR 2027

Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatio-temporal scene graphs provide a principled representation for modeling evolving object interactions, yet existing methods remain fundamentally frame-centric: they operate primarily in 2D, reason about currently visible objects, and discard entities upon occlusion. We introduce World Scene Graph Generation (WSGG), which constructs a temporally persistent world scene graph encompassing observed and unobserved interacting objects. To support this task, we introduce ActionGenome4D, upgrading ActionGenome videos to 4D scenes. We study the task in two settings: supervised WSGG and zero-shot WSGG. For the supervised WSGG setting, we begin by adapting existing VidSGG methods to WSGG to establish strong baselines. Then, we propose a suite of methods (WorldWise, WorldWise+, WorldWise++) that captures joint visual and 3D object representations and treats occlusion as a natural masking signal, reconstructing missing features via camera-pose-conditioned cross-attention over an object's visible history. For zero-shot WSGG, we evaluate open-source MLLMs under strong context-based prompting (including our proposed graph retrieval framework UWSGG-GraphRAG) strategies with explicit geometric inputs and outputs. Extensive experiments demonstrate the efficacy of the proposed methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World-Centric Scene Graphs, Spatio-Temporal Scene Graphs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y9AmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces World Scene Graph Generation (WSGG) and ActionGenome4D, extending ActionGenome with 3D boxes, camera poses, and relationships involving unobserved objects. It adapts video scene-graph baselines, proposes the WorldWise family using visual/geometric representations and masked completion, and evaluates zero-shot UWSGG-GraphRAG. In practice, the supervised setting uses complete videos and preconstructed persistent object representations with supplied temporal identities, primarily predicting person–object relationships.

Critical strengths
  1. Maintaining relationships involving occluded or out-of-view objects is a meaningful research problem.
  2. ActionGenome4D extends existing video relationship annotations to 3D geometry and unobserved objects through monocular reconstruction, geometric alignment, temporal completion, and human correction. The documented pipeline, annotation tools, and quality-control procedures provide a valuable foundation for further research.
  3. The WorldWise family shows advantages in class-balanced relationship prediction, suggesting potential benefits for unobserved objects and long-tail predicates. This makes the combination of temporal information and structured completion a worthwhile direction for further study.
Critical weaknesses
  1. The paper targets persistent 3D world understanding, but the main supervised experiments rely on preconstructed object representations and temporal identities. Joint validation of complete object discovery, 3D localization, and recovery of persistently unobserved objects remains limited. End-to-end evaluation over the complete object inventory, including more challenging occlusions and dynamic interactions, would establish whether the benefits extend beyond relationship prediction conditioned on existing objects.
  2. The independent contribution of the completion mechanism needs fairer controls. The proposed methods and baselines differ in feature sources, supervision, and training strategies, making it difficult to attribute the overall improvements specifically to the completion architecture. Experiments should align these conditions, directly ablate the core completion mechanism, and compare it with simpler temporal memory or information-propagation alternatives. This would clarify whether the additional model complexity provides a distinctive benefit.
  3. The conclusions need a more independent and consistent evaluation framework. Test-based model selection and differences in object coverage and evaluation settings limit the reliability and comparability of the reported results. An independent validation split and a unified test protocol should support fair comparisons of the main methods and the GraphRAG branch. Repeated experiments and inference-cost evaluation would further establish whether the benefits are stable and practically meaningful.
Questions for authors

See the weakness above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer y9Am: Weakness 1

W1. The main supervised experiments rely on preconstructed object representations and temporal identities, leaving joint validation of complete object discovery, 3D localization, and recovery of persistently unobserved objects (including harder occlusions and dynamic interactions) limited.

To validate end-to-end world graph recovery over the complete annotated inventory, we rebuilt the supervised evaluation protocol and retrained every method from scratch:

  • Ground-truth-free SGDet object discovery and persistence. In SGDet, zero ground truth is supplied at inference: every object class detected at confidence ≥ 0.2 anywhere in the video persists as a held unobserved object across frames where it goes undetected (transforming its nearest detection's 3D box through the shared π3 world frame into the current camera frame), while never-detected classes remain missed and count directly as failures. Persistent object representations are aligned per video by (object class, rank) (where rank 0 is the highest-scoring detection of that class) without relying on an external multi-object tracker.
  • Uniform 3D localization matching. Alongside the 2D rule, our 3D rule requires every object (both observed and unobserved) to match in category and achieve oriented IoU3D ≥ 0.10 in the π3 reconstruction scale without class-only fallback. While 2D detection on Action Genome is strong (36.7% mAP50 on our DINOv3-L Faster R-CNN, matching 60.5% of visible GT objects at 2D IoU ≥ 0.5), lifting single-view 2D detections into oriented 3D boxes yields a mean IoU3D of 0.130–0.145 on visible detections (Table 14) and meets IoU3D ≥ 0.10 on 24.4% (ResNet-50) of held unobserved object-frames, making monocular 3D estimation the primary bottleneck across all methods.

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights and Scope Limitations (Section 6):

  • Evaluating end-to-end discovery and 3D matching over 100% of the inventory confirms WorldWise++'s substantial lead in overall R@20 / mR@20 (+15.5% / +120.9% SGDet 2D WC; +20.9% / +133.3% SGDet 3D WC; +5.6% / +24.0% PredCls WC) and unobserved mR@20 (+92.3% SGDet 2D NC; +33.3% SGDet 3D NC; +86.3% PredCls NC). We also explicitly note four limitations in Section 6: (1) localizing objects that move while occluded, (2) stratified subsets isolating dynamic interactions or occlusion duration, (3) causal online inference, and (4) object–object relationships beyond person–object pairs.
AuthorsOpenReview ↗

Response to Reviewer y9Am: Weakness 2

W2. The proposed methods and baselines differ in feature sources, supervision, and training strategies, making it difficult to attribute improvements specifically to the completion architecture; align these conditions, directly ablate completion, and compare against simpler temporal memory.

  1. Adapting strong VidSGG backbones as natural baselines for a new task. As we introduce World Scene Graph Generation (WSGG) as a new task, establishing principled baselines naturally begins with adapting strong existing methods from the closest related domain. Following this standard practice, we first adapted the leading object-centric representation learning backbones in Video Scene Graph Generation (W-STTran, W-DSGDetr, and W-USG, together with their geometry-augmented W-STTran++ and W-DSGDetr++ variants) to operate over persistent observed and unobserved world states.
  2. Designing the WorldWise family from adapted baseline insights. Specifically, once we observed the empirical impact and limitations of each representation, temporal-aggregation, and training component in these adapted VidSGG methods, we designed the progressive architecture of the WorldWise family (WorldWise, WorldWise+, and WorldWise++) to directly leverage those findings for persistent 4D world scene graphs.
  3. Entry-point baselines versus our proposed architecture. Thus, our adapted VidSGG models serve as natural entry-point baselines for WSGG, while our proposed WorldWise family significantly outperforms every adapted baseline across both PredCls and SGDet under 2D and uniform 3D evaluation (achieving gains of +15.5% / +120.9% in SGDet 2D WC R@20 / mR@20, +20.9% / +133.3% in SGDet 3D WC R@20 / mR@20, and +5.6% / +24.0% in PredCls WC R@20 / mR@20 in Table 2).
  4. Adapted baselines as temporal-memory propagation models. Moreover, W-STTran (W-STTran++) and W-DSGDetr (W-DSGDetr++) can indeed be viewed directly as baselines equipped with simple temporal memory and cross-frame information propagation: because persistent object slots are maintained across the video, their temporal self-attention encoders aggregate and propagate context from visible frames into occluded and out-of-view frames, and our WorldWise family demonstrates substantial improvements over these temporal-memory baselines.
AuthorsOpenReview ↗

Response to Reviewer y9Am: Weakness 3

W3. Test-based selection and differences in object coverage and evaluation settings limit reliability and comparability; an independent validation split, a unified test protocol across the main methods and the GraphRAG branch, repeated experiments, and inference-cost evaluation are needed.

We agree, and we addressed these concerns in our revised experiments and manuscript:

  • 1. Independent validation split (Section 5). Every supervised model in Table 2 now selects its checkpoint by With Constraint (WC) R@20 on a 750-video validation split held out from the official training split (training on the remaining 6,766 videos), never accessing the 1,511-video test split.
  • 2. Unified full-inventory supervised protocol and zero-shot GraphRAG positioning (Section 5; Section 5.2). Every supervised method is now trained and evaluated under a single unified protocol across all 773,236 annotated triplets (414,443 OO and 358,793 UO), with identical setting-specific inputs, 2D and uniform 3D (IoU3D ≥ 0.10) matching rules, and confidence-ranked scoring. For the zero-shot U-WSGG branch (Section 5.2, Table 3), because generative MLLMs emit discrete predicate sets rather than continuous rankings over all 26 predicates, we evaluate UWSGG-GraphRAG against caption-enriched prompting across five diverse MLLM families under both PredCls and SGDet using P / R / F1 without cross-ranking zero-shot F1 directly against supervised R@K. Moreover, we position UWSGG-GraphRAG as a strong structured-retrieval baseline that enhances zero-shot MLLM performance over raw caption transcripts across backbones by adapting GraphRAG to spatio-temporal world scene graphs.

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

I thank the authors for the comprehensive rebuttal and additional results, in which my concerns have been well addressed. I would, therefore, keep my original score.

Reviewer_wqgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to build scene graphs that keep track of objects and their relationships even when they are out of view. It introduces ActionGenome4D, which adds persistent object identities, 3D boxes, and relationships involving unobserved objects to ActionGenome. The proposed WorldWise models use visual features, geometry, and temporal context to fill in missing object representations. The paper also explores a zero-shot approach using graph retrieval. Experiments compare these methods with adapted video scene graph baselines.

Critical strengths
  1. ActionGenome4D is a useful contribution. It includes 7,516 training videos and 1,511 test videos, with annotations for relationships involving unobserved objects. The human correction process and annotation agreement provide support for its quality. This makes it a valuable resource for studying scene understanding beyond visible objects.
  2. WorldWise++ shows clear gains in class-balanced recall under the reported protocol. Overall SGDet NC mR@20 increases from 20.5 for the strongest adapted baseline to 30.9, and from 6.6 to 11.8 for unobserved objects. These gains suggest better coverage across predicate classes, though raw recall on unobserved objects is lower.
Critical weaknesses
  1. The supervised evaluation does not cover the full task. It includes only 67,347 of the 358,793 annotated relationships involving unobserved objects, or about 19%, because objects without pre-extracted features are left out. Unobserved objects are also matched by class without checking their 3D location. The methods are compared on the same subset, but the results leave a large part of world graph recovery untested.
  2. Checkpoints are selected using test-set performance, with no separate validation split. Evaluating the selected checkpoint over all test frames does not remove this issue. The authors acknowledge this, but it is still unclear how much the reported gains depend on that selection.
Questions for authors
  1. Can you rescore the existing WorldWise++ predictions against all annotated relationships, counting omitted objects as misses? Can the saved outputs also be evaluated with 3D localization matching?
  2. Do you have results for a fixed final checkpoint or a checkpoint chosen without test scores? It would help to know whether the gains over the baselines remain.
3 comments
AuthorsOpenReview ↗

Response to Reviewer wqgS: Weakness 1

We thank Reviewer wqgS for a clear, constructive review and for the concrete verification suggestions that helped us thoroughly strengthen our evaluation protocol and benchmark analysis.

W1. The supervised evaluation includes only 67,347 of the 358,793 unobserved relationships (~19%) by omitting objects without pre-extracted features, and matches unobserved objects by class without checking their 3D location, leaving a large part of world graph recovery untested.

We are extremely sorry for the confusion regarding the 67,347 unseen relations. Those numbers correspond to a controlled subset of scenes taken for one of the ablations to save computational cost, and the text was part of the Appendix. Regarding 3D localization, we partially agree with this statement and found the suggestion indeed helpful, so following the reviewers' comments, we revised our evaluation protocol to include uniform 3D IoU-based matching alongside full-inventory evaluation. To eliminate both discrepancies, we rebuilt the supervised evaluation protocol (with a 750-video held-out validation split) and retrained every supervised method from scratch across the complete annotated test set (all 773,236 triplets: 414,443 observed OO and 358,793 unobserved UO triplets), as reported below and in Table 2:

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights:

  • Full-inventory and 3D matching gains: Alongside the 2D rule, our 3D rule requires every object (both observed and unobserved) to match in category and achieve oriented IoU3D ≥ 0.10 in the π3 reconstruction scale. Evaluating over 100% of the annotated inventory confirms that the WorldWise family's gains hold across the board: WorldWise++ leads every PredCls column (+5.6% / +24.0% WC; +3.7% / +86.3% NC UO), leads overall SGDet R@20 and mR@20 in both 2D (+15.5% / +120.9% WC) and 3D (+20.9% / +133.3% WC), and substantially improves SGDet unobserved mR@20 (+92.3% in 2D NC; +33.3% in 3D NC).
  • On Strength 2 (unseen raw recall vs. mean recall): WorldWise++ leads PredCls UO R@20 (69.7 NC, +3.7%), whereas W-DSGDetr leads SGDet 2D UO R@20 (22.5 vs. 19.8 NC, −12.0%). Specifically, we attribute the higher SGDet UO raw recall of W-DSGDetr and its variants to their training on head-heavy VLM pseudo-labels (λ_vlm = 0.2) without logit adjustment (adding τ = 0.5 lowers W-DSGDetr++ WC UO R@20 from 22.4 to 19.6).
AuthorsOpenReview ↗

Response to Reviewer wqgS: Weakness 2 and Question 1

W2. Checkpoints are selected using test-set performance with no separate validation split; evaluating the selected checkpoint over all test frames does not remove this issue, leaving unclear how much the reported gains depend on that selection.

  • Prior work on Action Genome uses only its official train and test splits without a separate validation split, and our initial submission followed this convention. However, we agree completely with the reviewer that selecting checkpoints on the test split leaves it unclear how much the reported margins depend on selection. We addressed W2 by introducing a dedicated 750-video held-out validation split, retraining all models from scratch, and evaluating them on the complete test dataset:
  • In all retrained runs, we carve out 750 videos from the official training split (by a fixed hash of the video ID) as a dedicated validation split, train on the remaining 6,766 videos, derive unobserved training labels and logit-adjustment priors strictly from the training split, and select checkpoints by validation WC R@20 without ever accessing the test split.
  • Under this validation-selected full-inventory protocol, WorldWise++ achieves a +9.4-point (+24.0%) PredCls WC mR@20 lead over the strongest baseline (48.5 vs. 39.1 for W-STTran++) and a +10.4-point (+120.9%) SGDet 2D WC mR@20 lead (19.0 vs. 8.6), confirming that our gains hold without test-set checkpoint selection.

Q1. Can you rescore the existing WorldWise++ predictions against all annotated relationships, counting omitted objects as misses? Can the saved outputs also be evaluated with 3D localization matching?

We are extremely sorry for this confusion. As noted in W1, those 67,347 unseen relations correspond to a controlled subset of scenes taken for one of the ablations in the Appendix to save computational cost. To eliminate this discrepancy and evaluate world graph recovery across all annotated relationships, we retrained all the supervised methods and re-evaluated them on the complete annotated test set. Under our revised protocol (including using a 750-video validation split, supplying every annotated object in PredCls), performance across the complete inventory is:

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights:

  • Rankings in unseen class-balanced recall (UO mR@20): Every WorldWise variant outperforms every adapted baseline across all UO mR@20 columns (+86.3% NC / +98.9% WC in PredCls; +92.3% NC / +89.1% WC in SGDet 2D; +33.3% in SGDet 3D for WorldWise++).
  • Rankings in unseen raw recall (UO R@20): WorldWise++ leads PredCls UO R@20 (69.7 NC, +3.7%; 57.7 WC, +2.1%), whereas W-DSGDetr leads SGDet 2D UO R@20 (22.5 vs. 19.8 NC, −12.0%). Specifically, we attribute the high UO performance of W-DSGDetr and its variants to their training on head-heavy VLM pseudo-labels (λ_vlm = 0.2) without logit adjustment (adding τ = 0.5 lowers W-DSGDetr++ WC UO R@20 from 22.4 to 19.6).
  • 3D localization insights: Under uniform IoU3D ≥ 0.10, WorldWise++ outperforms every baseline on overall 3D R@20 / mR@20 (+20.9% / +133.3% WC; +20.5% / +144.0% NC), observed OO (+48.0% / +162.1% WC, Table 28), and UO 3D mR@20 (+33.3% for WorldWise++, +60.0% for WorldWise), while monocular 3D depth/pose estimation bounds 3D recall across all methods.
AuthorsOpenReview ↗

Response to Reviewer wqgS: Question 2

Q2. Do you have results for a fixed final checkpoint or a checkpoint chosen without test scores? It would help to know whether the gains over the baselines remain.

Yes, we now evaluate checkpoints chosen strictly without test scores (selected by With Constraint R@20 on 750 held-out training videos):

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10, and in PredCls, GT 3D box inputs make 3D reproduce 2D):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights:

  • We can clearly see that the proposed method significantly outperforms the baselines.
  • Rankings in unseen class-balanced recall (UO mR@20): Every WorldWise variant outperforms every adapted baseline across all UO mR@20 columns (+86.3% NC / +98.9% WC in PredCls; +92.3% NC / +89.1% WC in SGDet 2D; +33.3% in SGDet 3D for WorldWise++).
  • Rankings in unseen raw recall (UO R@20): WorldWise++ leads PredCls UO R@20 (69.7 NC, +3.7%; 57.7 WC, +2.1%), whereas W-DSGDetr leads SGDet 2D UO R@20 (22.5 vs. 19.8 NC, −12.0%). Specifically, we attribute the high UO performance of W-DSGDetr and its variants to their training on head-heavy VLM pseudo-labels (λ_vlm = 0.2) without logit adjustment (adding τ = 0.5 lowers W-DSGDetr++ WC UO R@20 from 22.4 to 19.6).
Reviewer_SRrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces World Scene Graph Generation (WSGG), which aims to construct scene graphs from monocular video that incorporate 3D spatial localization and temporal persistence—ensuring that objects remain in the graph even when occluded or moved out of the field of view. To support this task, the authors developed ActionGenome4D, an extension of the ActionGenome dataset that includes annotations for 3D geometry and relationships involving occluded or invisible objects, derived through monocular 3D reconstruction, geometric alignment, bounding box completion, and manual refinement. The dataset comprises 9,027 videos, split into 7,516 for training and 1,511 for testing.

Regarding methodology, the authors adapted several existing video scene graph baselines and introduced the "WorldWise" series. This approach treats missing visual information as a mask-completion problem, predicting relationships by integrating geometric data, temporal context, and supervision from 3D reconstruction. A further iteration, WorldWise++, incorporates an image-feature-driven entity decoder, detection queries, and bounding box refinement. Additionally, the authors propose UWSGG-GraphRAG to investigate the performance of large multimodal models in zero-shot relationship prediction.

Critical strengths
  1. Integrating the persistence of occluded objects, their 3D geometry, and their time-varying relationships into a unified evaluation framework is a significant endeavor. By augmenting large-scale, existing video datasets with geometric data and information on occluded relationships—complemented by a human-in-the-loop verification process—ActionGenome4D has the potential to serve as a foundational resource for research in this field.
  2. The results in Table 2 suggest that the complete system may improve predictive performance for common predicates, while also demonstrating the potential to enhance coverage of long-tail relationships.
Critical weaknesses
  1. Appendix G.2 specifies that visible objects are matched based on category and 2D IoU, while occluded (invisible) objects are matched solely by category; the primary relationship metrics do not require 3D IoU. Consequently, objects with incorrect 3D positions or scales may still receive a score for relationship prediction.
  2. Supervised evaluation overlooks a large number of unseen relations present in the complete set of annotations, which limits the validation of performance in key, challenging scenarios. As shown in Table 20, the supervised evaluation includes 67,347 instances of unseen relations, whereas the complete annotation set contains 358,793—meaning the former accounts for only about 18.8% of the total. Furthermore, the supervised evaluation relies on a subset of objects with pre-extracted features, creating a significant discrepancy compared to the complete set of objects.
Questions for authors
  1. Why were the numerous unseen relations in Table 20 excluded from the supervised evaluation? What were the specific filtering criteria? If the prediction of a relation corresponding to a missing object were counted as a failure on the full annotated dataset, how would the method rankings and the recall for unseen objects change?
  2. What are the 3D IoU values ​​for WorldWise++ final bounding boxes on visible versus occluded objects? When a uniform 3D IoU threshold is applied for object matching, do the advantages in predicting primary relationships still hold?
  3. What are the sample size, the source of reference bounding boxes, and the independent annotation process for the geometric consistency experiments? Is there an audit procedure in place for re-annotating relationships—conducted without showing the annotators the relationships proposed by the VLM? Furthermore, what rules are applied to handle relationships that are invisible and cannot be reliably determined from the video?
  4. As shown in Table 3, the performance gains from GraphRAG are heavily dependent on the underlying backbone model, with some configurations even showing a decline. More critically, Table 30—which controls for both the model and the input across a matched set of 150 videos—reveals that GraphRAG achieves a WC R@20 of 47.4 (lower than the 47.8 observed in the context-free condition) and a WC mR@20 of 28.8 (lower than the 30.6 in the context-free condition). This limits the persuasiveness of GraphRAG’s contribution as a standalone method.
5 comments
AuthorsOpenReview ↗

Response to Reviewer SRrf: Weakness 1

We thank Reviewer SRrf for a thorough, insightful review that helped us substantially strengthen our evaluation protocol and empirical analysis.

W1. Visible objects are matched by category and 2D IoU while occluded objects are matched solely by category, and the primary relationship metrics do not require 3D IoU, allowing objects with inaccurate 3D positions or scales to receive credit.

We partially agree with this statement; however, we found the suggestion indeed helpful, so following the reviewers' comments, we revised our evaluation protocol to include 3D IoU-based matching. Specifically, under the ICLR-submitted protocol (Appendix G.2), visible objects were matched by category and 2D IoU ≥ 0.5, whereas occluded or out-of-view objects were matched by category alone without verifying 3D spatial overlap. To resolve this issue, we rebuilt the supervised evaluation protocol and retrained every method from scratch; we report the results below and in Table 2:

Supervised WSGG on the complete inventory: SGDet (left) and PredCls (right) (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %; 3D uses uniform IoU3D ≥ 0.10):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights:

  • Alongside the 2D rule, our 3D rule requires every object (both observed and unobserved) to match in category and achieve oriented IoU3D ≥ 0.10 in the π3 reconstruction scale. As can be seen from the table above, the proposed WorldWise family of methods outperforms the baselines by a large margin across the 3D metrics (WC and NC 3D R@20 and mR@20).

Changes in Paper: Section 5 (Evaluation Protocol) and Section 5.1 (Table 2) elevate uniform 3D matching to a primary metric.

AuthorsOpenReview ↗

Response to Reviewer SRrf: Weakness 2

W2. The supervised evaluation includes only 67,347 of the 358,793 unseen relations (18.8% in Table 20) and relies on a subset of objects with pre-extracted features, overlooking most unseen relations in the complete annotation set.

We are extremely sorry for this confusion. Those numbers correspond to a controlled subset of scenes taken for one of the ablations to save computational cost, and the text was part of the Appendix. To eliminate this discrepancy, we retrained all the supervised methods and re-evaluated them on the complete annotated test set:

Retrained full-inventory supervised performance across all 773,236 triplets (Table 2; Recall@20 / mean Recall@20 (R@20 / mR@20), in %):

MethodSGDet WC 2DSGDet WC 3DSGDet NC 2DSGDet NC 3DSGDet UO (NC) 2DSGDet UO (NC) 3DPredCls WCPredCls NCPredCls UO (NC)
W-STTran19.8 / 8.14.0 / 2.220.0 / 8.44.1 / 2.320.8 / 5.92.8 / 1.364.1 / 38.178.4 / 55.167.2 / 24.9
W-STTran++20.5 / 8.64.2 / 2.420.7 / 8.94.3 / 2.520.8 / 6.52.8 / 1.564.3 / 39.178.5 / 56.366.9 / 24.7
W-DSGDetr21.3 / 8.44.3 / 2.321.4 / 8.74.4 / 2.422.5 / 6.22.9 / 1.364.3 / 37.478.5 / 55.566.0 / 23.5
W-DSGDetr++21.1 / 8.34.3 / 2.221.3 / 8.64.4 / 2.422.3 / 6.32.9 / 1.464.5 / 37.378.6 / 54.566.4 / 24.4
W-USG17.1 / 7.33.2 / 1.717.3 / 7.73.3 / 1.918.1 / 5.32.3 / 1.263.7 / 37.177.9 / 54.166.0 / 24.0
WorldWise23.9 / 18.15.1 / 5.724.3 / 21.05.3 / 6.119.8 / 12.82.8 / 2.463.7 / 42.477.8 / 62.067.8 / 32.8
WorldWise+24.3 / 18.65.2 / 5.524.7 / 21.05.3 / 5.920.7 / 13.82.9 / 2.265.6 / 48.079.0 / 68.366.9 / 42.9
WorldWise++24.6 / 19.05.2 / 5.625.0 / 21.85.3 / 6.119.8 / 12.52.6 / 2.068.1 / 48.580.7 / 68.869.7 / 46.4
Δ vs. best baseline (%)+15.5 / +120.9+20.9 / +133.3+16.8 / +144.9+20.5 / +144.0-12.0 / +92.3-10.3 / +33.3+5.6 / +24.0+2.7 / +22.2+3.7 / +86.3

Insights:

  • Evaluating over 100% of the annotated inventory confirms that the WorldWise family's gains hold. Specifically, WorldWise++ leads every PredCls column (including WC UO R@20 / mR@20 of 57.7 / 34.8 vs. at most 56.5 / 17.5 in Table 26, +2.1% / +98.9%), leads overall SGDet R@20 and mR@20 (+15.5% / +120.9% WC; +16.8% / +144.9% NC), and nearly doubles SGDet unobserved mR@20 (+92.3% NC; +89.1% WC in Table 27).
AuthorsOpenReview ↗

Response to Reviewer SRrf: Question 1

Q1. Why were the numerous unseen relations in Table 20 excluded from the supervised evaluation, and what were the specific filtering criteria? If missing objects were counted as failures on the full annotated dataset, how would method rankings and unseen-object recall change?

We are extremely sorry for this confusion. Those numbers correspond to a controlled subset of scenes taken for one of the ablations to save computational cost, and the text was part of the Appendix. To eliminate this discrepancy, we retrained all the supervised methods and re-evaluated them on the complete annotated test set (along with 3D IoU-based matching).

Under our revised protocol following all reviewers' suggestions (including using a 750-video validation split, supplying every annotated object in PredCls, and evaluating every class detected at ≥ 0.2 across undetected frames in SGDet), unseen-object (UO) performance across all UO triplets is:

Unseen-object (UO) performance across all 358,793 UO triplets (Tables 2, 26–28; Recall@20 / mean Recall@20 (R@20 / mR@20), in %):

MethodPredCls 2D/3D WC UOPredCls 2D/3D NC UOSGDet 2D WC UOSGDet 2D NC UOSGDet 3D NC UO
W-STTran56.4 / 17.567.2 / 24.920.8 / 5.920.8 / 5.92.8 / 1.3
W-STTran++56.1 / 17.466.9 / 24.720.8 / 6.420.8 / 6.52.8 / 1.5
W-DSGDetr56.3 / 17.466.0 / 23.522.5 / 6.222.5 / 6.22.9 / 1.3
W-DSGDetr++56.5 / 17.466.4 / 24.422.4 / 6.322.3 / 6.32.9 / 1.4
W-USG55.7 / 17.466.0 / 24.018.1 / 5.318.1 / 5.32.3 / 1.2
WorldWise55.3 / 23.167.8 / 32.820.1 / 12.319.8 / 12.82.8 / 2.4
WorldWise+53.7 / 28.366.9 / 42.921.0 / 13.620.7 / 13.82.9 / 2.2
WorldWise++57.7 / 34.869.7 / 46.420.2 / 12.119.8 / 12.52.6 / 2.0
Δ vs. best baseline (%)+2.1 / +98.9+3.7 / +86.3-10.2 / +89.1-12.0 / +92.3-10.3 / +33.3

Insights:

  • Rankings in unseen class-balanced recall (UO mR@20): Every WorldWise variant outperforms every adapted baseline across all UO mR@20 columns (+86.3% NC / +98.9% WC in PredCls; +92.3% NC / +89.1% WC in SGDet 2D; +33.3% in SGDet 3D for WorldWise++).
  • Rankings in unseen raw recall (UO R@20): WorldWise++ leads PredCls UO R@20 (69.7 NC, +3.7%; 57.7 WC, +2.1%), whereas W-DSGDetr leads SGDet 2D UO R@20 (22.5 vs. 19.8 NC, −12.0%). Specifically, we attribute the high UO performance of W-DSGDetr and its variants to their training on head-heavy VLM pseudo-labels (λ_vlm = 0.2) without logit adjustment (adding τ = 0.5 lowers W-DSGDetr++ WC UO R@20 from 22.4 to 19.6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their detailed, constructive, and insightful reviews. To address these concerns thoroughly, we rebuilt the supervised evaluation pipeline and retrained every supervised method in the main paper (five adapted Video Scene Graph Generation baselines and all three WorldWise variants). In parallel, we audited our saved zero-shot U-WSGG predictions and revised the presentation and analysis of UWSGG-GraphRAG accordingly. Below, we answer every Weakness and Question raised by each reviewer individually in self-contained comments, with all corresponding revisions highlighted in blue in the updated PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.