acceptodds
Under review as a conference paper at ICLR 2027

Integration by Parts: Compositional Multimodal Learning for Dynamic 3D Scene Graph Generation

Abstract

Three-dimensional (3D) scene understanding is essential for autonomous systems operating in dynamic environments. While RGB, LiDAR, and textual semantics provide complementary information, existing dynamic 3D scene graph methods often rely on holistic object representations that are sensitive to occlusion, viewpoint changes, sensor noise, incomplete observations, and cross-modal misalignment. We propose Integration by Parts (IBP), a compositional multimodal framework that represents objects using fine-grained latent components rather than single embeddings. Shared learnable queries decompose RGB and LiDAR observations into latent components, which are explicitly aligned across modalities and integrated through recomposition with global multimodal and textual context to form compositional object representations. A temporal module associates these representations across observations and maintains semantic consistency for spatial-temporal reasoning. Experiments on KITTI-360 demonstrate the effectiveness of IBP for multimodal alignment, temporal reasoning, and robustness to incomplete observations. The code is available at: https://anonymous.4open.science/r/ibp_anonymized-/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic 3D Scene Graph Generation, Multimodal Learning, Cross-Modal Alignment, Scene Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JYy1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Integration by Parts builds a dynamic 3D scene graph from RGB, LiDAR, and a BLIP caption on KITTI-360. Shared learnable queries pull CLIP patch tokens and PointNet tokens into K = 8 latent components. An InfoNCE loss aligns RGB and LiDAR components with the same index, and a second InfoNCE loss aligns pooled object features with the caption embedding. A gate mixes the pooled multimodal vector with the recomposed components. Objects are linked across five keyframes by an MLP on these features plus a 16-dimensional geometric descriptor, normalized with Sinkhorn matching. Spatial predicates (left of, in front of, near, overlapping, occluding) and temporal predicates (approaching, moving away, same motion direction) are predicted from the resulting tracklets. Under teacher-forced PredCLS, temporal macro-F1 is 94.14, against 91.19 for an adapted DSG-DETR and 90.83 for TEMPURA. With predicted tracklets, association accuracy is 98.76 ± 0.08 and temporal macro-F1 is 84.93 ± 0.59.

Critical strengths

1.The pipeline separates decomposition, alignment, recomposition, association, and relation prediction, and the ablation assigns a measurable share to the components. On the predicted-tracklet setting, removing component-level representation and alignment leaves spatial macro-F1 unchanged at about 93.8, but lowers temporal macro-F1 from 85.57 to 81.04 and tracklet coverage from 84.36 to 78.91 (Table 4, seed 42). The matched-component cosine similarity of 0.580 against 0.186 for non-matches shows that the shared indices do what the contrastive loss asks them to do. Three seeds on the full pipeline have small variance.

2.The PredCLS protocol holds object pairs and five-step tracklets fixed, so Table 1 is not confounded by tracking failures. The sequence-level split and the decision to keep KITTI identities out of the input features are stated clearly. The flat occlusion curve in Table 9 is also informative once it is read against the geometric inputs: it shows how much of the score survives when RGB and LiDAR are degraded.

Critical weaknesses

1.The predicates are functions of box geometry and motion, and those quantities are available to the predictors. Association uses center offsets, inter-center distance, normalized offsets, and box-size ratios. Temporal labels are approaching, moving away, and same motion direction. Removing motion geometry and temporal context drops temporal macro-F1 from 85.57 to 50.08, while destroying 60% of the RGB and LiDAR input only drops PredCLS temporal macro-F1 from 93.33 to 89.90 and spatial macro-F1 from 93.75 to 93.17 (Table 9). Spatial R@20 for IBP, DSG-DETR, and TEMPURA is 29.05, 29.06, and 29.07. The paper never reports a geometry-only MLP, and Appendix A.1.2 does not give the thresholds that create the labels. Until that control exists, the compositional multimodal story is not what these metrics measure. The similarity diagonal is likewise the training target of L_part, not evidence that the slots correspond to physical parts.

2.The gains that remain after this confound do not support the abstract. Under PredCLS, IBP’s temporal macro-F1 exceeds DSG-DETR by 2.95 and its spatial macro-F1 is lower than both DSG-DETR (94.59) and TEMPURA (96.12). Predicted-tracklet results are reported only for IBP. UNO and Scene-LLM, at temporal macro-F1 45.11 and 47.76, are not usable comparisons: one has no public code and was reimplemented, and the other places all eight predicates into a single Action Genome head (Appendix A.4). Node-classification gains under degradation are about 2–4 points at 60%, and at 20% joint degradation the whole-object ablation is higher (75.63 vs. 74.41). At 60% occlusion, DSG-DETR’s temporal macro-F1 is 90.72 against IBP’s 89.90.

Questions for authors

1.Please state the exact geometric rules and thresholds for all eight predicates, and report an MLP that sees only box trajectories and the same geometric descriptors, with RGB, LiDAR, and text removed. If that baseline reaches the temporal macro-F1 of 94.14 and the spatial R@K in Table 1, I would reject the multimodal claim.

2.Please list every coordinate in g^s_ij and g^t_ij, and confirm whether DSG-DETR and TEMPURA received the same cues. The spatial R@20 values differ by at most 0.02. Were the rankings checked for equality?

3.Define the degradation operator in Tables 3 and 9. Also evaluate DSG-DETR and TEMPURA under predicted tracklets with the same association inputs. I would discount UNO and Scene-LLM unless those runs are shown to have converged with the same object features as IBP.

Reviewer_PUMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Integration by Parts (IBP), a method for dynamic 3D scene graph generation from RGB and LiDAR. A set of shared queries pulls latent components out of each modality. The RGB and LiDAR components are aligned and then combined with global multimodal features, which include a text feature from BLIP captions. The resulting object features are used to associate objects over time and to predict spatial and temporal relations. Experiments are on a benchmark the authors build from KITTI-360. Under teacher-forced PredCLS, IBP beats the adapted baselines on temporal relations. On spatial relations its M-F1 and mAP are lower than those of DSG-DETR and TEMPURA.

Critical strengths
  1. The idea of keeping local RGB and LiDAR evidence before object-level fusion makes sense when only part of an object is visible. Using shared queries together with a component alignment loss is a sensible way to learn correspondence between two modalities without part annotations.
  2. IBP performs well on temporal predicates, with M-F1 of 94.14 compared with 91.19 for DSG-DETR and 90.83 for TEMPURA under teacher-forced PredCLS. The results in Table 9 also show that this advantage holds under moderate degradation.
Critical weaknesses
  1. From the reported results, I cannot determine how much the latent components contribute beyond the supplied geometric cues. In Table 4, removing Parts/Alignment lowers temporal M-F1, but tracklet coverage also drops. So it is hard to separate the contribution to relation prediction from the improvement in association. What’s more, the appendix does not show whether the baselines (e.g. DSG-DETR, Scene-LLM and UNO) receive the same geometric cues as IBP. More ablations and information are needed (e.g. use only geometric cues for the teacher-forced PredCLS task and more details about how the baselines are adapted) to demonstrate the effectiveness of the proposed method.
  2. The robustness claim is not well supported. The proposed method’s main motivation is that under partially observed scenarios, component-level representations could preserve useful local information, which is reasonable. However, Table 3 compares IBP against the whole-object variant only on node classification, not on relation prediction. Table 9 evaluates relation prediction under degradation, but it uses GT tracklets and does not show whether component-level representations improve object association under partial observations. And since the degradation details are not provided, it’s unclear whether this experiment reveals the realistic partial observations.
Questions for authors
  1. The relation heads already receive spatial and motion descriptors. How well would a model using only these descriptors perform under the teacher-forced PredCLS experiment setting, without the learned object features? And do the baselines compared in Table 1 (e.g. DSG-DETR, Scene-LLM and UNO) receive the same explicit spatial and motion descriptors as IBP?
  2. How is degradation applied to the RGB and LiDAR inputs? Could the authors compare IBP with the whole-object variant on relation prediction under the same degradation, using both GT tracklets and the model’s predicted tracklets to assess how association errors affect the results?
  3. What accounts for the different results in Tables 2 and 10, and in Table 1 versus the 0% setting of Table 9? Are different checkpoints or evaluation protocols used?
Reviewer_uprqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In my view, this paper proposes a compositional multimodal framework for dynamic 3D scene graph generation. I find the approach itself reasonable, but I am concerned about the experimental setup and whether the baseline comparisons are fair. In my view, these issues make it difficult for me to assess the actual contribution of the proposed method.

Critical strengths

The paper has three strong points: (1) the paper addresses an interesting problem in multimodal dynamic scene understanding; (2) the component-level RGB-LiDAR alignment provides a reasonable approach to learning object representations; (3) the paper includes temporal reasoning, ablation studies, and robustness analysis.

Critical weaknesses

In my view, the weaknesses of the paper: Dataset selection: (1) The experiments are conducted only on KITTI-360 with newly constructed relation annotations, rather than established dynamic scene graph benchmarks such as Action Genome. This limits direct comparison with existing methods and raises questions about generalization. (2) Baseline fairness: Most compared methods were originally designed for different tasks and input modalities. Although the authors adapt several baselines to RGB-LiDAR inputs, the adaptations are not consistent across methods. It is unclear whether these comparisons fairly reflect the capabilities of the original approaches. (3) Limited multimodal comparisons: The paper lacks comparisons with stronger RGB-LiDAR fusion methods under a common representation and evaluation setting. Therefore, the advantages of the proposed component-level fusion strategy are not fully established. (4) Incomplete end-to-end comparison: The main baseline comparisons use ground-truth tracklets, while the predicted-tracklet evaluation reports only the proposed method. This makes it difficult to assess its advantage in practical dynamic scene graph generation.

Questions for authors

(1) The experiments are conducted only on KITTI-360 with newly constructed relation annotations, rather than established dynamic scene graph benchmarks such as Action Genome. This limits direct comparison with existing methods and raises questions about generalization. (2) Baseline fairness: Most compared methods were originally designed for different tasks and input modalities. Although the authors adapt several baselines to RGB-LiDAR inputs, the adaptations are not consistent across methods. It is unclear whether these comparisons fairly reflect the capabilities of the original approaches. (3) Limited multimodal comparisons: The paper lacks comparisons with stronger RGB-LiDAR fusion methods under a common representation and evaluation setting. Therefore, the advantages of the proposed component-level fusion strategy are not fully established. (4) Incomplete end-to-end comparison: The main baseline comparisons use ground-truth tracklets, while the predicted-tracklet evaluation reports only the proposed method. This makes it difficult to assess its advantage in practical dynamic scene graph generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.