- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Integration by Parts builds a dynamic 3D scene graph from RGB, LiDAR, and a BLIP caption on KITTI-360. Shared learnable queries pull CLIP patch tokens and PointNet tokens into K = 8 latent components. An InfoNCE loss aligns RGB and LiDAR components with the same index, and a second InfoNCE loss aligns pooled object features with the caption embedding. A gate mixes the pooled multimodal vector with the recomposed components. Objects are linked across five keyframes by an MLP on these features plus a 16-dimensional geometric descriptor, normalized with Sinkhorn matching. Spatial predicates (left of, in front of, near, overlapping, occluding) and temporal predicates (approaching, moving away, same motion direction) are predicted from the resulting tracklets. Under teacher-forced PredCLS, temporal macro-F1 is 94.14, against 91.19 for an adapted DSG-DETR and 90.83 for TEMPURA. With predicted tracklets, association accuracy is 98.76 ± 0.08 and temporal macro-F1 is 84.93 ± 0.59.
Critical strengths
1.The pipeline separates decomposition, alignment, recomposition, association, and relation prediction, and the ablation assigns a measurable share to the components. On the predicted-tracklet setting, removing component-level representation and alignment leaves spatial macro-F1 unchanged at about 93.8, but lowers temporal macro-F1 from 85.57 to 81.04 and tracklet coverage from 84.36 to 78.91 (Table 4, seed 42). The matched-component cosine similarity of 0.580 against 0.186 for non-matches shows that the shared indices do what the contrastive loss asks them to do. Three seeds on the full pipeline have small variance.
2.The PredCLS protocol holds object pairs and five-step tracklets fixed, so Table 1 is not confounded by tracking failures. The sequence-level split and the decision to keep KITTI identities out of the input features are stated clearly. The flat occlusion curve in Table 9 is also informative once it is read against the geometric inputs: it shows how much of the score survives when RGB and LiDAR are degraded.
Critical weaknesses
1.The predicates are functions of box geometry and motion, and those quantities are available to the predictors. Association uses center offsets, inter-center distance, normalized offsets, and box-size ratios. Temporal labels are approaching, moving away, and same motion direction. Removing motion geometry and temporal context drops temporal macro-F1 from 85.57 to 50.08, while destroying 60% of the RGB and LiDAR input only drops PredCLS temporal macro-F1 from 93.33 to 89.90 and spatial macro-F1 from 93.75 to 93.17 (Table 9). Spatial R@20 for IBP, DSG-DETR, and TEMPURA is 29.05, 29.06, and 29.07. The paper never reports a geometry-only MLP, and Appendix A.1.2 does not give the thresholds that create the labels. Until that control exists, the compositional multimodal story is not what these metrics measure. The similarity diagonal is likewise the training target of L_part, not evidence that the slots correspond to physical parts.
2.The gains that remain after this confound do not support the abstract. Under PredCLS, IBP’s temporal macro-F1 exceeds DSG-DETR by 2.95 and its spatial macro-F1 is lower than both DSG-DETR (94.59) and TEMPURA (96.12). Predicted-tracklet results are reported only for IBP. UNO and Scene-LLM, at temporal macro-F1 45.11 and 47.76, are not usable comparisons: one has no public code and was reimplemented, and the other places all eight predicates into a single Action Genome head (Appendix A.4). Node-classification gains under degradation are about 2–4 points at 60%, and at 20% joint degradation the whole-object ablation is higher (75.63 vs. 74.41). At 60% occlusion, DSG-DETR’s temporal macro-F1 is 90.72 against IBP’s 89.90.
Questions for authors
1.Please state the exact geometric rules and thresholds for all eight predicates, and report an MLP that sees only box trajectories and the same geometric descriptors, with RGB, LiDAR, and text removed. If that baseline reaches the temporal macro-F1 of 94.14 and the spatial R@K in Table 1, I would reject the multimodal claim.
2.Please list every coordinate in g^s_ij and g^t_ij, and confirm whether DSG-DETR and TEMPURA received the same cues. The spatial R@20 values differ by at most 0.02. Were the rankings checked for equality?
3.Define the degradation operator in Tables 3 and 9. Also evaluate DSG-DETR and TEMPURA under predicted tracklets with the same association inputs. I would discount UNO and Scene-LLM unless those runs are shown to have converged with the same object features as IBP.