- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper reconstructs synchronized robot-camera videos, including world-model rollouts, into a shared metric 4D Gaussian field. Stage 1 extends MoVieS with joint cross-view processing, robot-mesh source anchoring, and forward-kinematics-conditioned learned deformation. Stage 2 uses rendering-feedback refinement with position corrections shared across query times, preserving each Gaussian’s center displacement. Evaluation covers recorded observations and Cosmos 3 rollouts from 256 held-out DROID episodes.
Critical strengths
1)The paper addresses a useful problem: converting multi-camera robot videos into a shared metric 4D scene. Combining joint reconstruction with robot geometry and kinematics is well motivated.
2)The method shows substantial improvements over the evaluated baselines. On recorded observations, novel-view PSNR increases from 13.93 to 16.08 dB and depth AbsRel decreases from 0.373 to 0.197 relative to MoVieS. Similar gains on generated rollouts, together with a clearly documented evaluation protocol, support the system’s practical value.
Critical weaknesses
1)The system-level gains and component studies are informative, but the added value over a comparable joint multi-view model remains unclear. The main baselines reconstruct cameras independently, while Table 2 uses inference-time interventions. A comparison under matched training conditions would better establish the contribution of the robot-specific design.
2)The paper reasonably excludes unreliable non-robot motion labels and acknowledges this limitation. Still, robot EPE and whole-image rendering metrics provide limited direct evidence about manipulated-object motion. An evaluation focused on moving objects, even on a small audited subset, would strengthen the evidence for dynamic reconstruction.
Questions for authors
-
Beyond the inference-time interventions in Table 2, what evidence supports the added value of the robot-specific design over a joint multi-view model trained with comparable data and supervision?
-
Given the acknowledged limitations of the non-robot motion labels, can you provide evidence focused on manipulated objects, such as results on a small audited subset, to clarify whether the reported gains extend to object dynamics?
3 comments
Thank you for providing the two controlled studies requested in my review.
The matched joint-reconstruction comparison in Table R3-1 is the evidence I was missing. Training a joint model with the same initialization, data, schedule, and supervision, while removing only the mesh anchoring and FK-conditioning inputs, is a fairer control than inference-time interventions. Under this setting, the robot-specific design improves held-out robot-region PSNR and substantially reduces robot depth and displacement error on both recorded and Cosmos-generated inputs. This better establishes the added value of the robot-specific components beyond joint multi-view training.
The audited 30-clip object-motion evaluation in Table R3-2 also helps. Using checked object support, the method reduces object displacement EPE relative to both the DROID-adapted MoVieS baseline and a matched independent-reconstruction model, with gains on both rigid-object and cloth subsets. The subset is still small, but it provides the direct object-centric evidence that was previously lacking.
These additions address the two main weaknesses in my review. I am therefore inclined to raise my rating from 2 (Weak rejection) to 3 (Weak acceptance).
Thank you for your careful reassessment and constructive feedback.
Your suggestions helped us clarify the contribution of the robot-specific components and provide stronger evidence for manipulated-object reconstruction.
We are glad that the resulting matched joint-reconstruction comparison and audited object-motion evaluation have addressed your main concerns, and we sincerely appreciate your time and thoughtful engagement throughout the review process.
Response to Reviewer p42m
We thank the reviewer for recognizing this problem's value and requesting matched controls and object-motion evidence. RoGSW4RLD makes the predicted interactions of a pretrained video world model accessible as a shared, time-queryable metric 4D Gaussian scene. We combine the world model's visual predictions with feed-forward reconstruction that incorporates robot geometry and kinematics across moving wrist and fixed external views. This connects predicted videos to metric geometry and motion without learning a separate geometric transition model. The new tests establish robot-specific gains beyond matched joint training and extend the reconstruction evidence to manipulated objects.
[W1 & Q1] What establishes robot-specific value beyond a comparably trained joint multi-view model?
Explicit mesh/FK inputs improve robot reconstruction under matched joint training and supervision, and the final two-stage system retains this advantage. We separately trained Joint reconstruction (no mesh/FK inputs) with the same initialization, data, schedule and supervision, including FK targets. It retains joint attention, merged-field losses and calibrated camera trajectories, but omits mesh anchoring and FK-conditioning inputs.
Table R3-1. Matched robot-grounding comparison. 256 clips per input setting; recorded references. PSNR uses robot regions in held-out stereo views; depth/EPE use source-camera robot support. The first two rows isolate mesh/FK inputs at Stage 1; the last adds refinement.
(a) Recorded-video inputs
(b) Cosmos-generated inputs
Bold marks the best displayed values within each panel, with ties at reported precision.
The robot-region gains persist beyond joint processing under matched supervision. On recorded inputs, adding mesh/FK inputs at Stage 1 reduces robot depth error by approximately 54% and displacement error by 33%; held-out-view robot PSNR improves by 1.14 dB [95% paired CI: 0.97, 1.31]. All three metrics also improve with generated inputs. The gains cover rendered surface placement, displacement and novel-view appearance, directly supporting robot grounding beyond matched joint training.
The final system retains the advantage: robot depth MAE is 13.7 vs. 28.9 mm on recorded inputs and 15.5 vs. 45.8 mm on Cosmos inputs, alongside better robot-region PSNR and EPE.
[W2 & Q2] Do the gains extend to manipulated-object motion on a checked subset?
Yes. The final model improves object displacement reconstruction over both DROID-adapted and matched independent-camera baselines. For the requested object-specific evaluation, we use released PointWorld trajectories with checked object support and temporal alignment to the fixed window. We selected 30 recorded carried-object clips (16 rigid-object, 14 cloth) using recorded video/state and track consistency, without reconstruction outputs or scores. Projected support was visually checked to remove robot/cable contamination and ambiguous associations. All models share the same retained external-camera trajectories as estimated 3D references.
Table R3-2. Direct manipulated-object evaluation. Recorded inputs; displacement EPE in mm at identical external-camera reference points. The type columns partition the same 30 clips.
The object-motion gain appears in both rigid-object and cloth interactions. The final model lowers mean object EPE by approximately 28% relative to fine-tuned MoVieS and 16% relative to matched independent reconstruction. All methods use the same cameras, trajectories and scored points, providing the requested direct evidence of manipulated-object reconstruction.
Revision. We will add the trained joint control to §5.3 and the object results to §5.2/Appendix C, with cohort selection in B.1 and track alignment, support checks and metric aggregation in B.5.