acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Object-Centric Cross-View Reasoning from a Single Image

Abstract

We introduce Particle-based Object Representation Transport (PORT), a self-supervised object-centric model that directly transports object-centric latent representations extracted from a single source image to target viewpoints according to relative camera pose. Rather than merely generating novel-view images, PORT directly transports each object latent to predict its position, scale, depth, and appearance under different horizontal camera viewpoints. This enables PORT to construct viewpoint-conditioned multi-view object representations from a single image while preserving object correspondence across views, enabling downstream 3D reasoning by leveraging object latents across multiple viewpoints. We provide object-level evaluation metrics and a protocol for quantitatively assessing the accuracy of cross-view latent transport. We then demonstrate the utility of the resulting multi-view object representations in robotic manipulation and Gaussian Splatting, showing their applicability to both 3D scene representation and downstream tasks. Our code will be publicly available. Videos are available at https://sites.google.com/view/particletransport.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Object-Centric Representation Learning, Multi-View Representation, Novel View Synthesis, Camera-Conditioned Transport, 3D Geometry, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_9ZsxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Particle-based Object Representation Transport (PORT), a self-supervised object-centric framework that transforms 2D object-level latent representations extracted from a single source RGB image to target viewpoints conditioned on relative camera azimuth Δθ∈(−π,π]\Delta\theta \in (-\pi, \pi]. Building upon the Deep Latent Particles (DLPv3 / LPWM) autoencoder architecture—where each foreground object is parameterized by disentangled attributes for 2D position zpz_p, scale zsz_s, depth/compositing order zdz_d, transparency ztz_t, and appearance zfz_f—PORT introduces a ParticleTransporter module. This module uses a Scene-Context Transformer to capture inter-object relational geometry across the scene, followed by decoupled transport heads for geometric/visibility states (GψG_\psi), appearance (AηA_\eta), and background (BξB_\xi). Crucially, by transporting particles in-place while retaining their indices, PORT preserves persistent object correspondence across viewpoints by construction, trained end-to-end via novel-view reconstruction and an optional stop-gradient bidirectional Transport Chamfer loss (LTCL_{TC}).

To evaluate cross-view object transport, the authors introduce a Kubric-based multi-view dataset generation protocol (producing TP-OBJ3D and TP-GROCERY) and object-centric correspondence metrics (2D Position Error, Scale Error, Identity Accuracy, and multi-view 3D Triangulation Error). Experiments across TP-OBJ3D, TP-GROCERY, and CLEVR3D demonstrate that PORT preserves near-perfect object identity (>99.5%>99.5\%) and interpolates smoothly to unseen azimuths (unlike slot-based OSRT, which degrades between sparse training views, or patch-anchored LPWM, which swaps particle identities under large viewpoint shifts). Finally, the authors demonstrate downstream utility in: (1) linear probing of 3D positions, (2) substituting a physical camera with "imagined" transported particle states in multi-view robotic manipulation (PandaPush with EC-Diffuser), cutting representation construction time in half while maintaining or improving success rates, and (3) feeding transported RGB and alpha-mask views into 3D Gaussian Splatting (3DGS) and 3D scene editing pipelines.

Critical strengths
  1. Elegant Solution to Cross-View Object Correspondence Without 3D Supervision: A fundamental limitation of existing patch/grid-based particle models (like DLPv3 and LPWM) is that particles are anchored to local 2D image regions; under large camera rotations, an object moves across patches and its identity is handed off to different particle indices. Conversely, slot-based 3D models (like OSRT) rely on ray-conditioned decoders that overfit when trained on sparse views (e.g., 3 views in CLEVR3D, as compellingly shown in Figures 4 and 5). PORT solves both issues by transporting low-dimensional explicit geometric attributes (zp,zs,zd,zt)(z_p, z_s, z_d, z_t) conditioned on relative camera azimuth while keeping the decoder camera-agnostic.

  2. Well-Motivated Architectural Decoupling and Thorough Ablations: The design of ParticleTransporter reflects strong domain intuition. Specifically, using a Scene-Context Transformer to model inter-particle occlusion and relative spatial layout for the geometry/visibility head GψG_\psi, while excluding scene context from the appearance head AηA_\eta—which instead takes source and transported 2D positions (zpm,z^pm)(z_p^m, \hat{z}_p^m) and the camera feature—prevents appearance bleeding across objects while accounting for perspective-induced viewing angle changes. Table 10 cleanly validates both choices.

  3. Compelling Downstream Robotic Manipulation Utility and Occlusion Recovery: The PandaPush integration with EC-Diffuser (Table 4) is a highlight. Replacing a second physical camera (2R) with PORT-transported particles (1R+1T and 1R+2T) matches or improves manipulation success (77.1% to 79.6%) at roughly half the encoding latency (73.8 ms to 37.2 ms). Furthermore, the occlusion analysis in Appendix A.7.1 (Table 8) provides a concrete physical explanation for why imagined views help: when the robot arm occludes a cube in the front view, encoding the real front view misses the cube (85.6% localization), whereas transporting the unoccluded side-view particle preserves its state for the policy (91.8% localization).

  4. Principled Evaluation Protocol: Standard NVS metrics (PSNR/SSIM) can mask complete failures in object-level identity tracking. The proposed source-anchored evaluation protocol—measuring 2D position error, log-ratio amodal scale consistency, Hungarian-matched identity accuracy, and multi-view ray triangulation—sets a strong standard for evaluating 3D-aware object-centric representations.

Critical weaknesses
  1. Strict 1-DoF Camera Pose Parameterization (Azimuth-Only on a Fixed Ring): From the perspective of 3D object representation and 6D pose reasoning, the camera transformation model in PORT is highly restricted. ParticleTransporter only conditions on a 1-DoF horizontal azimuth Δθ∈(−π,π]\Delta\theta \in (-\pi, \pi] assuming a fixed camera elevation, constant radius, and look-at fixation at the scene center. Furthermore, as described in Appendix A.3.1–A.3.2, although camera radius and elevation are randomized per scene, they remain fixed across views within an episode, yet the transporter receives no information about the source camera's elevation or distance/intrinsics. Consequently, the model must implicitly infer the scene's elevation/horizon from the 2D image just to project how an azimuthal rotation maps to 2D elliptical trajectories on the image plane. This severely limits applicability to general 6-DoF camera transformations, wrist-mounted robot cameras, or varying camera elevations.

  2. Modification of the PandaPush Camera Setup to Fit the 1-DoF Assumption: In the original EC-Diffuser benchmark, the two cameras (front and side) are placed at different heights and distances. To apply PORT, the authors had to modify the simulator environment to place the cameras on a constant-height, constant-radius ring (Appendix A.3.2, lines 894–900). While fair as a controlled comparison against 2R on the same ring, this highlights that PORT could not be directly applied out-of-the-box to the standard multi-camera manipulation setup where cameras have arbitrary 6-DoF relative extrinsics [R∣t]∈SE(3)[R \mid t] \in SE(3).

  3. Lack of Explicit 3D Orientation / 6D Pose Grounding for Asymmetric Objects: While TP-GROCERY introduces ShapeNetV2 and YCB objects (e.g., bananas, bowls, cans), each object is represented by a 2D keypoint zp∈R2z_p \in \mathbb{R}^2, 2D bounding-box scale zs∈R2z_s \in \mathbb{R}^2, scalar compositing depth zd∈Rz_d \in \mathbb{R}, and an unstructured appearance vector zf∈Rdz_f \in \mathbb{R}^d. Any 3D rotational change (yaw/pitch/roll of an asymmetric object like a banana rotating in view) is absorbed entirely into the unstructured latent residual Δzfm\Delta z_f^m. Moreover, almost all objects in TP-OBJ3D, CLEVR3D, and TP-GROCERY (bowls, jars, bottles, cans, trash bins, round fruits, spheres, cylinders) are radially symmetric around the vertical axis, meaning their silhouette scale barely changes under azimuthal rotation except for perspective depth shifts. It is unclear how accurately PORT transports object extent zsz_s and appearance zfz_f for strongly elongated or asymmetric multi-part objects under rotation.

  4. Temporal Identity Inconsistency in Dynamic Scenes: As revealed in Appendix A.9.3 (Table 11), adding explicit particle identity embeddings across views degrades PandaPush manipulation performance from 77.1% to 63.5% because the frozen DLP encoder does not maintain consistent particle indices across time steps tt (or between the current observation and the goal image). While feature concatenation works around this for a single frame's cross-view tokens, it underscores a tension between cross-view transport and temporal tracking in dynamic scenes.

  5. Incomplete Discussion of Structured Slot Initialization and Adaptive Entity Counts in Related Work and Baselines: In Section 2 (Self-Supervised Object-Centric Representation Learning), the paper contrasts slot-based methods (which assign visual information to a predefined set of randomly initialized latent slots) with particle-based approaches (which anchor latents to keypoints/features and still require pre-specifying M=64M=64 proposals and filtering to a fixed Top-kk, as shown in Table 5). However, advances in slot-based object abstraction have specifically addressed random slot initialization and fixed slot counts—for example, Enhancing Interpretable Object Abstraction via Clustering-based Slot Initialization (Gao et al., BMVC 2023), which initializes slots via perceptual feature clustering (e.g., mean-shift clustering) to automatically adapt the number of object abstractions and improve performance on downstream object discovery and novel view synthesis. Discussing this line of work in Section 2—and considering whether clustering-based slot initialization would mitigate the slot degradation observed in the OSRT baseline or help PORT avoid manual Top-kk particle filtering—would strengthen both the literature positioning and baseline analysis.

Questions for authors
  1. Ablation on Transport Chamfer Loss: Why was the Transport Chamfer loss weight set to 0.10.1 for TP-GROCERY and PandaPush while set to 00 for TP-OBJ3D and CLEVR3D (Table 5)? How much does performance (Identity Accuracy, Position Error, and PSNR) degrade on TP-GROCERY if the Transport Chamfer weight is set to 00? Without it, does the model suffer from local minima or particle collapse on more complex textures/geometries?

  2. Conditioning on Full SE(3)SE(3) Extrinsics vs. Azimuth-Only: What happens if the camera conditioning vector is parameterized by the full relative camera extrinsic transformation [ΔR∣Δt]∈SE(3)[\Delta R \mid \Delta t] \in SE(3) rather than just horizontal azimuth? Specifically, in PandaPush, could PORT be trained to transport directly between the original EC-Diffuser front and side cameras (which differ in height and radius), and how does it perform?

  3. Sensitivity to Randomized Source Camera Elevation and Radius: In TP-OBJ3D and TP-GROCERY, camera radius and elevation are randomized per episode, yet the camera feature only provides relative azimuth. Does the Scene-Context Transformer infer the camera elevation implicitly from the vertical distribution of particles {zpm}\{z_p^m\} or the background latent? Have you evaluated whether transport accuracy varies as a function of the episode's camera elevation angle?

  4. Asymmetric Geometry Analysis in TP-GROCERY: Since most categories in TP-GROCERY are radially symmetric around the yaw axis (bowls, cans, jars, bottles, spherical fruits), could you provide a per-category breakdown of Scale Error and Position Error—specifically contrasting elongated/asymmetric objects (e.g., bananas or tilted objects) against radially symmetric ones?

  5. Mitigating the Shadow-Induced zz-Axis Bias: In Appendix A.5.2 (Figure 8), you insightfully note that the DLP encoder includes cast shadows in the particle mask, shifting the 2D particle center downward and causing a systematic negative zz-bias during 3D triangulation. Does fine-tuning the DLP encoder jointly with the multi-view transport loss reduce this shadow-capture artifact compared to a single-view DLP encoder, since cast shadows do not transform rigidly with the 3D object centroid across azimuths?

  6. Comparison/Integration with Clustering-Based Object Abstraction (Gao et al., 2023): In Table 5, PORT relies on a fixed number of keypoint proposals (M=64M=64), transported particles (L=12L=12 or 2525), and manual Top-kk filtering (1010 or 2020) per dataset. Meanwhile, prior work on interpretable object abstraction (Enhancing Interpretable Object Abstraction via Clustering-based Slot Initialization, Gao et al., BMVC 2023) demonstrated that initializing object slots via perceptual feature clustering (such as mean-shift) enables automatic determination of object counts and significantly improves slot-based novel view synthesis. Could the authors discuss this work in Section 2, and comment on: (a) whether equipping the OSRT baseline with clustering-based slot initialization improves its cross-view stability on CLEVR3D, and (b) whether clustering-based initialization could replace the fixed Top-kk heuristic when selecting active source particles for transport in PORT?

Note on Rating: I am willing to adjust/increase my rating if the authors can formally address the concerns and questions above—including rigorous mathematical/algorithmic clarifications, the missing related work discussion, and fixing the manuscript formatting oversights—during the rebuttal phase.

5 comments

We thank the reviewer for the thorough review. Changes in the revised manuscript are highlighted in blue, and new qualitative results are on our project website (https://sites.google.com/view/particletransport).

Q1. Transport Chamfer. Transport Chamfer is an auxiliary loss (weight 0.1 where used) that aligns the transported particles with the encoded target-view particles as sets; reconstruction remains the main training signal.

DatasetTransport ChamferPosition error (px) ↓Identity accuracy ↑PSNR (dB) ↑
TP-GROCERYwith3.480.99530.16
without5.030.98929.79
TP-OBJ3Dwith3.440.99838.85
without2.560.99837.50

On TP-GROCERY, where particles are less stable early in training, removing the loss degrades both object-level accuracy and reconstruction. On TP-OBJ3D it improves PSNR, but position error rises by the same amount at Δ=0\Delta=0, before any transport, so we omit it there. On CLEVR3D, adding it during fine-tuning changed PSNR by only +0.04 dB and slightly raised position error (6.34 vs. 6.13 px). These metrics show degraded performance without the loss but do not directly measure particle collapse. We set the loss per dataset based on these ablations (Table 5).

W1–W2 / Q2. Relative extrinsics and the original PandaPush cameras. We kept the transport architecture and conditioned it on relative rotation and translation. On TP-GROCERY (best results per evaluation setting in bold):

Evaluation pairsCamera inputTraining posesPosition error (px) ↓Identity accuracy ↑
Within-ringRelative azimuthWithin-ring3.460.997
Joint azimuth and elevation changeRelative azimuthWithin-ring11.740.918
Relative azimuth + elevationMulti-elevation rings3.620.990
Relative SE(3)Multi-elevation rings3.160.983
Free poses (distance and look-at vary)Relative SE(3)Multi-elevation rings14.150.911
Relative SE(3)Free poses5.740.982

With the same multi-elevation training poses, relative SE(3) lowered cross-ring position error from 3.62 to 3.16 px and scale error from 23.4% to 20.1% compared with azimuth + elevation. Training on free poses lowered SE(3) position error on free-pose pairs from 14.15 to 5.74 px. Reconstruction quality is lower for free poses (PSNR 27.09 vs. 30.26 dB within rings).

The submitted PandaPush comparison used matched ring cameras. We also trained PORT with cameras placed as in the original EC-Diffuser setup, conditioned on relative SE(3).

PandaPush camerasCamera conditioningPosition error (px) ↓Identity accuracy ↑
Ring layout (submission)Relative azimuth0.560.984
Original EC-Diffuser layoutRelative SE(3)0.670.988

PORT transports object states between the original front and side cameras without rearranging them into a ring. Full TP-GROCERY results are in Appendix A.9.4 (Table 12).

W1 / Q3. Source elevation and distance. Within each scene, source and target share elevation and distance, so the source image carries cues about the camera configuration. Across 1,000 TP-GROCERY test scenes, position error ranged from 3.41 to 3.66 px across elevation bins (26–38°) and from 3.39 to 3.55 px across distance bins (8–10), with identity accuracy of at least 0.992. Correlations with position error were −0.09 and 0.01. Elevation is unsupervised, and this analysis does not show whether the model estimates it internally. We clarified the scope of the azimuth-only setting in Sec. 4.1 and the Limitations.

W3 / Q4. Asymmetric objects. From near-isotropic objects (horizontal aspect ratio < 1.15) to elongated ones (≥ 1.5), position error rose from 3.41 to 4.12 px and scale error from 10.6% to 15.2%, while identity accuracy stayed above 0.991. Bananas had 3.78 px position error but 23.1% scale error; the target-view extent is hard to predict when the source view does not reveal the relevant geometry. We discuss this in the Limitations.

Q5. Shadow bias. PORT trains the encoder, transporter, and decoder jointly, and after warm-up target-view reconstruction also updates the encoder. After joint training, TP-OBJ3D still shows a vertical bias of −0.153 ± 0.042 with a ray RMS of 0.031 and an XY error of 0.045. This is consistent with rays agreeing on a point below the true center when a particle includes the cast shadow. We did not quantify the change relative to a single-view encoder. We clarified the joint training setup in Sec. 4.

W5 / Q6. Clustering. We added Gao et al. (2023) to Sec. 2. On CLEVR3D, k-means slot initialization lowered OSRT's position error at unseen angles from 44.6 to 34.8 px, but tracked slots fell from 856 to 316 and identity accuracy stayed similar (0.392 vs. 0.383).

MethodPosition error (px), training angles ↓Position error (px), 31–45° away ↓Identity accuracy, training angles ↑Identity accuracy, 31–45° away ↑
OSRT6.559.40.9890.193
OSRT + k-means13.343.60.8610.228
PORT6.17.50.9960.996

PORT's identity accuracy stays at 0.993–0.996 across all offsets. OSRT's difficulty at unseen angles may also relate to rendering view-agnostic slots with a ray-conditioned decoder. For PORT, Top-LL caps the number of retained particles, and several particles can represent one object. Clustering candidate locations and keeping existing particles as representatives could improve spatial coverage while preserving their indices; we see it as a practical extension of the current architecture.

W4. Temporal identity. PORT preserves indices across views generated from the same observation. The encoder processes timesteps independently, so an index can refer to different objects over time or between the current and goal images; in Appendix A.9.3, shared index embeddings reduced success from 77.1% to 63.5%. Temporal association, for example with a causal Transformer over particles, is future work.

We have also corrected typos and formatting issues throughout the revised manuscript.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer 9Zsx,

Thank you again for the time and effort you put into reviewing our work!

We understand that this may be a busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted. We have also uploaded additional qualitative results to our project website(https://sites.google.com/view/particletransport) in response to the requests raised during the rebuttal.

Whenever you have a chance, we would be grateful for your feedback on whether our responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best regards,

The Authors of Submission 4376

Reviewer_JiMyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a self-supervised framework for learning cross-view object-centric representations. The method builds upon the existing DLP representation, introducing an additional Transformer to model scene context and several MLPs to predict changes in particle attributes, including position, depth, appearance, etc., under different camera viewpoints. The experiments demonstrate improved cross-view object correspondence and further evaluate the method on robotic manipulation and Gaussian Splatting tasks.

Critical strengths
  1. The authors conduct various experiments from different aspects to validate their method, including 3D triangulation and robotic manipulation.

  2. The overall method design is generally reasonable.

Critical weaknesses
  1. Limited technical novelty. PORT builds upon the existing DLP representation, with an additional Transformer and several MLPs to predict camera-conditioned changes in particle attributes, which is relatively straightforward. Regarding supervision, the method mainly inherits the reconstruction and KL losses from prior works, with an additional transport Chamfer distance loss. However, the effectiveness of this additional loss is questionable, as the configurations in the Appendix show that the method can still achieve high performance without it (as shown in Table 1 Identity Acc metric).

  2. Limited evaluation on realistic scenarios. Although the paper shows some promising results, the evaluation datasets mainly consist of toy-like scenes rather than realistic scenarios. As shown in the paper figures and supplementary videos, the scenes generally have clean backgrounds, sparsely distributed objects, and objects with clearly distinguishable appearances. This differs from real-world scenarios, where backgrounds can be complex and objects may be densely distributed with similar appearances. Therefore, it remains unclear whether the proposed method and the observations reported in this paper can generalize to more realistic settings.

Questions for authors
  1. In Table 5, the transport Chamfer loss is disabled for TP-OBJ3D and CLEVR3D but enabled for the other two datasets. Why is this loss only used for certain datasets? Could this loss negatively affect performance, especially when DLP produces different particle decompositions for the target view?

  2. The current experiments are mainly conducted on simple synthetic scenes with predefined camera rotations. How does the method perform when there are multiple similar-looking or same-category objects, especially under severe occlusion? Also, the current experiments only consider azimuth changes. How would the method perform under more general camera movements?

  3. Since there is no explicit geometric constraint, how does the model ensure transport consistency across multiple camera transformations? For example, would directly transporting a particle by 60 degrees produce the same result as transporting it by 30 degrees twice? Evaluating this consistency would further strengthen the claim that the model learns meaningful and consistent cross-view transformations.

4 comments

We thank the reviewer for the constructive feedback. Changes in the revised manuscript are highlighted in blue, and new qualitative results are on our project website (https://sites.google.com/view/particletransport).

W1. Technical contribution. PORT uses DLP particles and learns to transform their attributes from one image to a requested viewpoint. Keeping the source indices lets us track the same particles across views, and Q2 tests whether those indices still correspond to the correct objects when appearances are identical. In PandaPush, adding a transported view to one real view raised success from 68.0% to 77.1%, matching the two-real-view policy with about half the representation time and fewer input entities, since corresponding particles are merged into one token.

Q1. Transport Chamfer. Transport Chamfer is an auxiliary loss (weight 0.1 where used) that aligns the transported particles with the encoded target-view particles as sets; reconstruction remains the main training signal. To limit the effect of differing decompositions, we match particles without index-wise correspondence, weight matches by reference transparency, and stop gradients through the reference particles.

DatasetTransport ChamferPosition error (px) ↓Identity accuracy ↑PSNR (dB) ↑
TP-GROCERYwith3.480.99530.16
without5.030.98929.79
TP-OBJ3Dwith3.440.99838.85
without2.560.99837.50

Index preservation gives correspondence by construction, so identity accuracy stays high without the loss; the loss improves the accuracy of the transported states on TP-GROCERY, where particles are less stable early in training. On TP-OBJ3D it improves PSNR, but position error rises by the same amount at Δ=0\Delta=0, before any transport, so the change comes from the encoder. On CLEVR3D, adding it during fine-tuning changed PSNR by only +0.04 dB. This is why the loss is set per dataset (Table 5).

W2 / Q2. Scene complexity and camera motion. The added tests use synthetic scenes. With textured backgrounds (TP-GROCERY-TEX, same assets, placements, and cameras), identity accuracy and scale error were unchanged (0.995, 10.9%) and position error fell from 3.48 to 2.97 px, while PSNR fell from 30.16 to 28.86 dB.

We also built held-out scenes with identical-appearance and crowded layouts and evaluated the existing models without retraining, counting swaps between identical objects as errors.

ConditionTP-GROCERY position (px) ↓TP-GROCERY identity ↑TP-OBJ3D position (px) ↓TP-OBJ3D identity ↑
Original test set3.480.9952.560.998
Identical appearance3.330.9972.590.999
Crowded3.970.9682.520.998
Identical appearance + crowded3.900.9652.620.997

Crowding, rather than identical appearance, accounts for most of the drop: among identical grocery objects, identity accuracy was 0.998 without crowding and 0.961 with it. In PandaPush, when the arm occludes an object, localization is 85.6% from the real front view and 91.8% from a view transported from the unobstructed side view (Appendix A.7.1).

For camera motion, we trained PORT on TP-GROCERY with richer camera inputs and training poses, keeping the transport architecture. Best results per evaluation setting are in bold.

Evaluation pairsCamera inputTraining posesPosition error (px) ↓Identity accuracy ↑
Within-ringRelative azimuthWithin-ring3.460.997
Joint azimuth and elevation changeRelative azimuthWithin-ring11.740.918
Relative azimuth + elevationMulti-elevation rings3.620.990
Relative SE(3)Multi-elevation rings3.160.983
Free poses (distance and look-at vary)Relative SE(3)Multi-elevation rings14.150.911
Relative SE(3)Free poses5.740.982

With camera inputs and training poses that cover the test-time camera motion, PORT transports object states beyond the ring setting. Reconstruction quality is lower for free poses (PSNR 27.09 vs. 30.26 dB within rings).

Qualitative results for these settings are on our project website. We clarified the synthetic scope in the Limitations; camera results are in Appendix A.9.4 (Table 12).

Q3. Multi-step transport. The reconstruction objective does not enforce composition explicitly. For a 60° target, direct transport and two 30° steps gave position errors of 3.49 and 3.53 px, with predicted centers 1.09 px apart; six 10° steps raised the error to 4.92 px and the gap to 4.41 px. Two steps of unseen sizes also stayed close to direct transport (0.67 px apart for 15° × 2 and 1.57 px for 45° × 2). We use direct transport for target-view prediction, and a compositional consistency loss is a natural extension.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer JiMy,

Thank you again for the time and effort you put into reviewing our work!

We understand that this may be a busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted. We have also uploaded additional qualitative results to our project website(https://sites.google.com/view/particletransport) in response to the requests raised during the rebuttal.

Whenever you have a chance, we would be grateful for your feedback on whether our responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best regards,

The Authors of Submission 4376

Thanks for the response. However, my main concern regarding the practical significance of this work remains.

For the additional experiments, I checked the provided website. I found that the textured-background variant uses a low-contrast, nearly uniform concrete-like floor that differs little from the original clean setting, and I could not find examples of the crowded scenes. Qualitative examples of both settings should be included in the paper, together with details of how the crowded scenes were constructed (e.g., the number and spacing of objects).

Currently, all real-world visual results in the paper show scenes with clean backgrounds and sparsely distributed objects. It therefore remains unclear to me whether the proposed method and its conclusions can transfer to realistic scenarios with complex backgrounds and densely distributed objects.

Since the two other reviewers rate positively, I think that this paper should still offer some insights. However, I am still not convinced that the conclusions drawn from these highly simplified settings hold in real-world scenes. I therefore maintain my negative rating, albeit with low confidence.

Thank you for checking our project website and for your follow-up regarding cluttered scenes.

In our rebuttal, we interpreted “crowded scenes” as dense layouts with increased inter-object occlusion. That set retains the assets, cameras, lighting, and backgrounds of TP-GROCERY and TP-OBJ3D while changing the placement rules. We reduced the sampling range for object centers from ±2.8 to ±1.0 world units and the minimum gap between object footprints from 0.10 to 0.02. Scenes contain 3–6 objects on TP-GROCERY and 3–5 on TP-OBJ3D.

To also cover scenes with more objects and similar objects appearing together, we conducted an additional evaluation with 8–10 objects per scene, placed in the original range (±2.8) with the same 0.02 minimum gap and no interpenetration (100 scenes per dataset). Every TP-GROCERY scene contains multiple instances from the same category. Examples of the original crowded scenes and the additional scenes are available under “Crowded scenes with visually similar objects” and “Scenes with more objects,” respectively, on our project website.

Statistic (original → set with more objects)TP-OBJ3DTP-GROCERY
Static objects per scene3–5 (mean 4.0) → 8–10 (9.1)3–6 (mean 4.5) → 8–10 (9.0)
Nearest-object gap, median / 10th percentile (world units)0.93 / 0.32 → 0.36 / 0.110.71 / 0.34 → 0.31 / 0.10
Image area covered by objects7.6% → 15.0%6.6% → 12.8%
Object–view pairs whose mask touches another object32% → 66%26% → 52%

Without retraining, PORT achieves the following results on the scenes with more objects. Results on the original test sets are shown in parentheses. Position, identity, and scale metrics are computed over objects matched to a source particle.

Set with more objects (original test set)TP-OBJ3DTP-GROCERY
Position error (px) ↓2.60 (2.56)4.07 (3.48)
Identity accuracy ↑0.997 (0.998)0.977 (0.995)
Scale error (%) ↓15.6 (15.9)12.0 (10.9)
Objects matched to a source particle98.8%92.0%

These results show that, without additional training, PORT keeps high identity accuracy in both dense layouts and scenes with more objects. Accuracy stays close to the original test set on TP-OBJ3D and decreases moderately on TP-GROCERY, where crowding also had a larger effect in the rebuttal.

Regarding backgrounds, PORT learns object-centric representations, and our primary goal is to predict and use changes in the same objects’ states across viewpoints. Image reconstruction provides the self-supervised training signal for this objective. In the textured-background experiment, full-image reconstruction quality decreased, while identity accuracy remained at 0.995 and position and scale predictions remained stable. This shows that object-level transport performance was maintained under the evaluated background change.

We agree, however, that the tested texture does not capture the complexity of real-world backgrounds. We acknowledge the reduced background reconstruction quality as a limitation and will discuss reconstruction of more complex backgrounds and evaluation of object decomposition and transport in diverse real-world scenes as future work.

We will include qualitative examples of the crowded, more-object, and textured-background settings, together with the construction details and evaluation results, in the appendix. If a specific additional analysis would help address your remaining concern, we would appreciate your guidance.

Comments on the paper

3 comments

We thank the reviewer for the careful reading. Changes in the revised manuscript are highlighted in blue, and new qualitative results are on our project website (https://sites.google.com/view/particletransport).

Q1. Camera conditioning. The main model receives only the relative azimuth Δ\Delta, encoded as [Δ,sin⁡Δ,cos⁡Δ][\Delta, \sin\Delta, \cos\Delta], and no radius, elevation, intrinsics, or translation. It learns target-view positions and scales from the source particles, Δ\Delta, and scene context through reconstruction. In TP-OBJ3D and TP-GROCERY, elevation and radius vary across scenes but stay constant within a scene. We chose this minimal camera input to test whether learning to predict object states across viewpoints from a single image yields representations useful for 3D reasoning. We also trained PORT on TP-GROCERY with richer camera inputs and training poses. Best per setting in bold.

Evaluation pairsCamera inputTraining posesPosition error (px) ↓Identity accuracy ↑
Within-ringRelative azimuthWithin-ring3.460.997
Joint azimuth and elevation changeRelative azimuthWithin-ring11.740.918
Relative azimuth + elevationMulti-elevation rings3.620.990
Relative SE(3)Multi-elevation rings3.160.983
Free poses (distance and look-at vary)Relative SE(3)Multi-elevation rings14.150.911
Relative SE(3)Free poses5.740.982

With camera inputs and training poses that cover the test-time camera motion, PORT transports object states beyond the ring setting. Reconstruction quality is lower for free poses (PSNR 27.09 vs. 30.26 dB within rings). We clarified this scope in Sec. 4.1 and the Limitations; full results are in Appendix A.9.4 (Table 12).

W1 / Q2. Compositing order and 3D evidence. We do not claim that PORT explicitly models 3D geometry. Keeping an object's identity, position, and size consistent across views requires knowing how they appear from other viewpoints, so we hypothesize that learning to predict these changes lets the learned representation space capture 3D information, recoverable from the object states PORT predicts across views.

zdz_d is the compositing-order attribute from DLP and LPWM. It controls rendering order, has no metric-depth supervision, and is not used in triangulation. The revision uses compositing order throughout the main text.

Triangulation uses calibrated cameras only at evaluation. PORT predicts the target-view 2D centers from one image; ground truth sets the initial source assignment and the reference 3D centers, and particles are tracked by index. With the same cameras and triangulation, 3D error is 0.163 / 0.248 / 0.401 for PORT and 2.724 / 2.395 / 2.565 for tracked LPWM (TP-OBJ3D / TP-GROCERY / CLEVR3D).

A ridge probe with no camera inputs predicts each object's 3D position from frozen representations. Predicted 2D centers are taken at five fixed offsets.

Probe inputTP-OBJ3D 3D error ↓CLEVR3D 3D error ↓
PORT, source particle0.3960.316
PORT, + predicted 2D centers0.1400.066
LPWM, source particle0.4620.324
LPWM, + camera-conditioned rollouts0.4550.320

Predicted centers make 3D position linearly recoverable, while LPWM's rollouts, which do not preserve correspondence, do not. Source and predicted 2D centers alone, without zdz_d, give 0.151 / 0.099. The revised Introduction and Sec. 5.3 state this scope; probe details are in Appendix A.6.1.

Q3. Seen and unseen viewpoints. Seen and unseen refer to the relative azimuth offset between source and target views; all scenes are held out. Seen offsets are those used in training: every 60° on TP-OBJ3D, every 30° on TP-GROCERY, and nominally 0°, 120°, and 240° on CLEVR3D, with small jitter in training. Unseen offsets are the remaining points of the 2°, 10°, and 1° evaluation grids. We clarified this in the Table 1 caption and Appendix A.3.3.

W2 / Q4. Manipulation controls. Same policy training protocol, five seeds (mean ± SD).

Policy inputSuccess (%) ↑Representation time (ms) ↓
1R only68.0 ± 2.8535.6
1R + random particles26.3 ± 2.1535.6
EC-Diffuser (2R)77.1 ± 1.8273.8
PORT (1R+1T)77.1 ± 1.8837.2
PORT (1R+2T)79.6 ± 2.1237.5

The random control replaces the transported block with the same number of particles sampled from the DLP prior at training and test time, so adding particles alone does not explain the gain. One transported view adds 9.1 points over one real view and matches the two-real-view mean with 37.2 vs. 73.8 ms of representation time. Because correspondence is preserved, each particle's real-view and transported features form one token, so the policy also processes fewer entities than with two independently encoded real views. Both controls are in Table 4 and Appendix A.7.

AuthorsOpenReview ↗

Thank You for Your Reconsideration

Dear Reviewer,

Thank you for taking the time to review our rebuttal and additional experiments!

We sincerely appreciate your decision to raise your rating.

We are grateful for your thoughtful consideration and constructive feedback throughout the review process.

Thank you again!

Warm regards,

The Authors of Submission 4376

Area_Chair_kZsVOpenReview ↗

Discussions

Dear Reviewers,

Could you please engage in further discussions with the authors based on their response?

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.