- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Particle-based Object Representation Transport (PORT), a self-supervised object-centric framework that transforms 2D object-level latent representations extracted from a single source RGB image to target viewpoints conditioned on relative camera azimuth . Building upon the Deep Latent Particles (DLPv3 / LPWM) autoencoder architecture—where each foreground object is parameterized by disentangled attributes for 2D position , scale , depth/compositing order , transparency , and appearance —PORT introduces a ParticleTransporter module. This module uses a Scene-Context Transformer to capture inter-object relational geometry across the scene, followed by decoupled transport heads for geometric/visibility states (), appearance (), and background (). Crucially, by transporting particles in-place while retaining their indices, PORT preserves persistent object correspondence across viewpoints by construction, trained end-to-end via novel-view reconstruction and an optional stop-gradient bidirectional Transport Chamfer loss ().
To evaluate cross-view object transport, the authors introduce a Kubric-based multi-view dataset generation protocol (producing TP-OBJ3D and TP-GROCERY) and object-centric correspondence metrics (2D Position Error, Scale Error, Identity Accuracy, and multi-view 3D Triangulation Error). Experiments across TP-OBJ3D, TP-GROCERY, and CLEVR3D demonstrate that PORT preserves near-perfect object identity () and interpolates smoothly to unseen azimuths (unlike slot-based OSRT, which degrades between sparse training views, or patch-anchored LPWM, which swaps particle identities under large viewpoint shifts). Finally, the authors demonstrate downstream utility in: (1) linear probing of 3D positions, (2) substituting a physical camera with "imagined" transported particle states in multi-view robotic manipulation (PandaPush with EC-Diffuser), cutting representation construction time in half while maintaining or improving success rates, and (3) feeding transported RGB and alpha-mask views into 3D Gaussian Splatting (3DGS) and 3D scene editing pipelines.
Critical strengths
-
Elegant Solution to Cross-View Object Correspondence Without 3D Supervision: A fundamental limitation of existing patch/grid-based particle models (like DLPv3 and LPWM) is that particles are anchored to local 2D image regions; under large camera rotations, an object moves across patches and its identity is handed off to different particle indices. Conversely, slot-based 3D models (like OSRT) rely on ray-conditioned decoders that overfit when trained on sparse views (e.g., 3 views in CLEVR3D, as compellingly shown in Figures 4 and 5). PORT solves both issues by transporting low-dimensional explicit geometric attributes conditioned on relative camera azimuth while keeping the decoder camera-agnostic.
-
Well-Motivated Architectural Decoupling and Thorough Ablations: The design of ParticleTransporter reflects strong domain intuition. Specifically, using a Scene-Context Transformer to model inter-particle occlusion and relative spatial layout for the geometry/visibility head , while excluding scene context from the appearance head —which instead takes source and transported 2D positions and the camera feature—prevents appearance bleeding across objects while accounting for perspective-induced viewing angle changes. Table 10 cleanly validates both choices.
-
Compelling Downstream Robotic Manipulation Utility and Occlusion Recovery: The PandaPush integration with EC-Diffuser (Table 4) is a highlight. Replacing a second physical camera (2R) with PORT-transported particles (1R+1T and 1R+2T) matches or improves manipulation success (77.1% to 79.6%) at roughly half the encoding latency (73.8 ms to 37.2 ms). Furthermore, the occlusion analysis in Appendix A.7.1 (Table 8) provides a concrete physical explanation for why imagined views help: when the robot arm occludes a cube in the front view, encoding the real front view misses the cube (85.6% localization), whereas transporting the unoccluded side-view particle preserves its state for the policy (91.8% localization).
-
Principled Evaluation Protocol: Standard NVS metrics (PSNR/SSIM) can mask complete failures in object-level identity tracking. The proposed source-anchored evaluation protocol—measuring 2D position error, log-ratio amodal scale consistency, Hungarian-matched identity accuracy, and multi-view ray triangulation—sets a strong standard for evaluating 3D-aware object-centric representations.
Critical weaknesses
-
Strict 1-DoF Camera Pose Parameterization (Azimuth-Only on a Fixed Ring): From the perspective of 3D object representation and 6D pose reasoning, the camera transformation model in PORT is highly restricted. ParticleTransporter only conditions on a 1-DoF horizontal azimuth assuming a fixed camera elevation, constant radius, and look-at fixation at the scene center. Furthermore, as described in Appendix A.3.1–A.3.2, although camera radius and elevation are randomized per scene, they remain fixed across views within an episode, yet the transporter receives no information about the source camera's elevation or distance/intrinsics. Consequently, the model must implicitly infer the scene's elevation/horizon from the 2D image just to project how an azimuthal rotation maps to 2D elliptical trajectories on the image plane. This severely limits applicability to general 6-DoF camera transformations, wrist-mounted robot cameras, or varying camera elevations.
-
Modification of the PandaPush Camera Setup to Fit the 1-DoF Assumption: In the original EC-Diffuser benchmark, the two cameras (front and side) are placed at different heights and distances. To apply PORT, the authors had to modify the simulator environment to place the cameras on a constant-height, constant-radius ring (Appendix A.3.2, lines 894–900). While fair as a controlled comparison against 2R on the same ring, this highlights that PORT could not be directly applied out-of-the-box to the standard multi-camera manipulation setup where cameras have arbitrary 6-DoF relative extrinsics .
-
Lack of Explicit 3D Orientation / 6D Pose Grounding for Asymmetric Objects: While TP-GROCERY introduces ShapeNetV2 and YCB objects (e.g., bananas, bowls, cans), each object is represented by a 2D keypoint , 2D bounding-box scale , scalar compositing depth , and an unstructured appearance vector . Any 3D rotational change (yaw/pitch/roll of an asymmetric object like a banana rotating in view) is absorbed entirely into the unstructured latent residual . Moreover, almost all objects in TP-OBJ3D, CLEVR3D, and TP-GROCERY (bowls, jars, bottles, cans, trash bins, round fruits, spheres, cylinders) are radially symmetric around the vertical axis, meaning their silhouette scale barely changes under azimuthal rotation except for perspective depth shifts. It is unclear how accurately PORT transports object extent and appearance for strongly elongated or asymmetric multi-part objects under rotation.
-
Temporal Identity Inconsistency in Dynamic Scenes: As revealed in Appendix A.9.3 (Table 11), adding explicit particle identity embeddings across views degrades PandaPush manipulation performance from 77.1% to 63.5% because the frozen DLP encoder does not maintain consistent particle indices across time steps (or between the current observation and the goal image). While feature concatenation works around this for a single frame's cross-view tokens, it underscores a tension between cross-view transport and temporal tracking in dynamic scenes.
-
Incomplete Discussion of Structured Slot Initialization and Adaptive Entity Counts in Related Work and Baselines: In Section 2 (Self-Supervised Object-Centric Representation Learning), the paper contrasts slot-based methods (which assign visual information to a predefined set of randomly initialized latent slots) with particle-based approaches (which anchor latents to keypoints/features and still require pre-specifying proposals and filtering to a fixed Top-, as shown in Table 5). However, advances in slot-based object abstraction have specifically addressed random slot initialization and fixed slot counts—for example, Enhancing Interpretable Object Abstraction via Clustering-based Slot Initialization (Gao et al., BMVC 2023), which initializes slots via perceptual feature clustering (e.g., mean-shift clustering) to automatically adapt the number of object abstractions and improve performance on downstream object discovery and novel view synthesis. Discussing this line of work in Section 2—and considering whether clustering-based slot initialization would mitigate the slot degradation observed in the OSRT baseline or help PORT avoid manual Top- particle filtering—would strengthen both the literature positioning and baseline analysis.
Questions for authors
-
Ablation on Transport Chamfer Loss: Why was the Transport Chamfer loss weight set to for TP-GROCERY and PandaPush while set to for TP-OBJ3D and CLEVR3D (Table 5)? How much does performance (Identity Accuracy, Position Error, and PSNR) degrade on TP-GROCERY if the Transport Chamfer weight is set to ? Without it, does the model suffer from local minima or particle collapse on more complex textures/geometries?
-
Conditioning on Full Extrinsics vs. Azimuth-Only: What happens if the camera conditioning vector is parameterized by the full relative camera extrinsic transformation rather than just horizontal azimuth? Specifically, in PandaPush, could PORT be trained to transport directly between the original EC-Diffuser front and side cameras (which differ in height and radius), and how does it perform?
-
Sensitivity to Randomized Source Camera Elevation and Radius: In TP-OBJ3D and TP-GROCERY, camera radius and elevation are randomized per episode, yet the camera feature only provides relative azimuth. Does the Scene-Context Transformer infer the camera elevation implicitly from the vertical distribution of particles or the background latent? Have you evaluated whether transport accuracy varies as a function of the episode's camera elevation angle?
-
Asymmetric Geometry Analysis in TP-GROCERY: Since most categories in TP-GROCERY are radially symmetric around the yaw axis (bowls, cans, jars, bottles, spherical fruits), could you provide a per-category breakdown of Scale Error and Position Error—specifically contrasting elongated/asymmetric objects (e.g., bananas or tilted objects) against radially symmetric ones?
-
Mitigating the Shadow-Induced -Axis Bias: In Appendix A.5.2 (Figure 8), you insightfully note that the DLP encoder includes cast shadows in the particle mask, shifting the 2D particle center downward and causing a systematic negative -bias during 3D triangulation. Does fine-tuning the DLP encoder jointly with the multi-view transport loss reduce this shadow-capture artifact compared to a single-view DLP encoder, since cast shadows do not transform rigidly with the 3D object centroid across azimuths?
-
Comparison/Integration with Clustering-Based Object Abstraction (Gao et al., 2023): In Table 5, PORT relies on a fixed number of keypoint proposals (), transported particles ( or ), and manual Top- filtering ( or ) per dataset. Meanwhile, prior work on interpretable object abstraction (Enhancing Interpretable Object Abstraction via Clustering-based Slot Initialization, Gao et al., BMVC 2023) demonstrated that initializing object slots via perceptual feature clustering (such as mean-shift) enables automatic determination of object counts and significantly improves slot-based novel view synthesis. Could the authors discuss this work in Section 2, and comment on: (a) whether equipping the OSRT baseline with clustering-based slot initialization improves its cross-view stability on CLEVR3D, and (b) whether clustering-based initialization could replace the fixed Top- heuristic when selecting active source particles for transport in PORT?
Note on Rating: I am willing to adjust/increase my rating if the authors can formally address the concerns and questions above—including rigorous mathematical/algorithmic clarifications, the missing related work discussion, and fixing the manuscript formatting oversights—during the rebuttal phase.
5 comments
W3 / Q4. Asymmetric objects. From near-isotropic objects (horizontal aspect ratio < 1.15) to elongated ones (≥ 1.5), position error rose from 3.41 to 4.12 px and scale error from 10.6% to 15.2%, while identity accuracy stayed above 0.991. Bananas had 3.78 px position error but 23.1% scale error; the target-view extent is hard to predict when the source view does not reveal the relevant geometry. We discuss this in the Limitations.
Q5. Shadow bias. PORT trains the encoder, transporter, and decoder jointly, and after warm-up target-view reconstruction also updates the encoder. After joint training, TP-OBJ3D still shows a vertical bias of −0.153 ± 0.042 with a ray RMS of 0.031 and an XY error of 0.045. This is consistent with rays agreeing on a point below the true center when a particle includes the cast shadow. We did not quantify the change relative to a single-view encoder. We clarified the joint training setup in Sec. 4.
W5 / Q6. Clustering. We added Gao et al. (2023) to Sec. 2. On CLEVR3D, k-means slot initialization lowered OSRT's position error at unseen angles from 44.6 to 34.8 px, but tracked slots fell from 856 to 316 and identity accuracy stayed similar (0.392 vs. 0.383).
| Method | Position error (px), training angles ↓ | Position error (px), 31–45° away ↓ | Identity accuracy, training angles ↑ | Identity accuracy, 31–45° away ↑ |
|---|---|---|---|---|
| OSRT | 6.5 | 59.4 | 0.989 | 0.193 |
| OSRT + k-means | 13.3 | 43.6 | 0.861 | 0.228 |
| PORT | 6.1 | 7.5 | 0.996 | 0.996 |
PORT's identity accuracy stays at 0.993–0.996 across all offsets. OSRT's difficulty at unseen angles may also relate to rendering view-agnostic slots with a ray-conditioned decoder. For PORT, Top- caps the number of retained particles, and several particles can represent one object. Clustering candidate locations and keeping existing particles as representatives could improve spatial coverage while preserving their indices; we see it as a practical extension of the current architecture.
W4. Temporal identity. PORT preserves indices across views generated from the same observation. The encoder processes timesteps independently, so an index can refer to different objects over time or between the current and goal images; in Appendix A.9.3, shared index embeddings reduced success from 77.1% to 63.5%. Temporal association, for example with a causal Transformer over particles, is future work.
We have also corrected typos and formatting issues throughout the revised manuscript.
Follow-Up to Your Review
Dear Reviewer 9Zsx,
Thank you again for the time and effort you put into reviewing our work!
We understand that this may be a busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted. We have also uploaded additional qualitative results to our project website(https://sites.google.com/view/particletransport) in response to the requests raised during the rebuttal.
Whenever you have a chance, we would be grateful for your feedback on whether our responses address your concerns. We would be happy to clarify anything that remains open.
Thank you!
Best regards,
The Authors of Submission 4376
We thank the reviewer for the thorough review. Changes in the revised manuscript are highlighted in blue, and new qualitative results are on our project website (https://sites.google.com/view/particletransport).
Q1. Transport Chamfer. Transport Chamfer is an auxiliary loss (weight 0.1 where used) that aligns the transported particles with the encoded target-view particles as sets; reconstruction remains the main training signal.
On TP-GROCERY, where particles are less stable early in training, removing the loss degrades both object-level accuracy and reconstruction. On TP-OBJ3D it improves PSNR, but position error rises by the same amount at Δ=0, before any transport, so we omit it there. On CLEVR3D, adding it during fine-tuning changed PSNR by only +0.04 dB and slightly raised position error (6.34 vs. 6.13 px). These metrics show degraded performance without the loss but do not directly measure particle collapse. We set the loss per dataset based on these ablations (Table 5).
W1–W2 / Q2. Relative extrinsics and the original PandaPush cameras. We kept the transport architecture and conditioned it on relative rotation and translation. On TP-GROCERY (best results per evaluation setting in bold):
With the same multi-elevation training poses, relative SE(3) lowered cross-ring position error from 3.62 to 3.16 px and scale error from 23.4% to 20.1% compared with azimuth + elevation. Training on free poses lowered SE(3) position error on free-pose pairs from 14.15 to 5.74 px. Reconstruction quality is lower for free poses (PSNR 27.09 vs. 30.26 dB within rings).
The submitted PandaPush comparison used matched ring cameras. We also trained PORT with cameras placed as in the original EC-Diffuser setup, conditioned on relative SE(3).
PORT transports object states between the original front and side cameras without rearranging them into a ring. Full TP-GROCERY results are in Appendix A.9.4 (Table 12).
W1 / Q3. Source elevation and distance. Within each scene, source and target share elevation and distance, so the source image carries cues about the camera configuration. Across 1,000 TP-GROCERY test scenes, position error ranged from 3.41 to 3.66 px across elevation bins (26–38°) and from 3.39 to 3.55 px across distance bins (8–10), with identity accuracy of at least 0.992. Correlations with position error were −0.09 and 0.01. Elevation is unsupervised, and this analysis does not show whether the model estimates it internally. We clarified the scope of the azimuth-only setting in Sec. 4.1 and the Limitations.